Scribd
Upload
16 views15 pages

Workflow Task 2.1

The document outlines a workflow for analyzing question difficulty levels using Bloom's Taxonomy, categorizing questions as 'easy', 'moderate', or 'difficult'. It details the steps for data preparation, model creation using SVM and transformer models like BERT, and the evaluation of model performance through various metrics. The final goal is to deploy the model as APIs and ensure it remains updated with new datasets.

Uploaded by

Anusha Tripathi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
16 views15 pages

Workflow Task 2.1

The document outlines a workflow for analyzing question difficulty levels using Bloom's Taxonomy, categorizing questions as 'easy', 'moderate', or 'difficult'. It details the steps for data preparation, model creation using SVM and transformer models like BERT, and the evaluation of model performance through various metrics. The final goal is to deploy the model as APIs and ensure it remains updated with new datasets.

Uploaded by

Anusha Tripathi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 15

WORKFLOW TASK 2

QUESTION ANALYSIS USING BLOOM’S


TAXONOMY
GENERAL PROBLEM SUMMARY
Identification of question difficulty level, where the categories are “easy”, “moderate” and
“difficult”, assuming that the inputs are questions only.
We are provided with -
1. Question (from the question paper)
2. Syllabus
3. Glossary / Index of text material corresponding to syllabus
We will be using Bloom’s taxonomy to identify the difficulty levels of each individual
questions.
Step 1
Required Input Dataset
Input Data: Create a csv file dataset, containing of the various word levels based off “Bloom’s Taxonomy”.The
different levels are usually said to be: “Create”, “Evaluate”, “Analyze”,“Apply”, “Understand” and “Remember”.A
more extensive view of the difficulty levels are:

Remember Understand Apply Analyze Evaluate Create

Factual

Conceptual

Procedural

Metacognitive
Each of the 24 levels of this taxonomy can be assigned a difficulty level. Thus, we will have 24
categories of the dataset.
We also have the syllabus keyword dictionary and index/glossary from the textbook material as
our dataset.
Step 2
Classification Rules

● We are going to build a model which will categorize input words into a
few classes: “stopwords”, “syllabus-context” and “Bloom’s category
context”.
● The decision tree to be used for this model has been shown in the
previous slide.
Step 3
Classification model creation
• We will be using SVM classifiers in our case. (It can handle multi-class, non-linear
classification. It is also robust to overfitting, giving good accuracy).
• We will begin with linear kernel SVM. It is a relatively easy model with low complexity and
works well with smaller datasets. However, it might create accuracy problems and
overfitting.
• If we observe that accuracy is unreasonable for this case, we will work with low-degree
polynomial kernel. It has higher accuracy and complexity.
• We will also be using grid search to tune the kernel’s hyperparameters.
Step 4

Preprocessing of the input question


• Tokenization: Split the input question into words/tokens.
• Normalization: Convert all text to lowercase and remove punctuation and white
spaces. (Need attention - If a diagrammatic representation occurs, what are we supposed
to do?)
Step 5
Difficulty level analysis
• We perform calculations on the individual tokens from each question.
• Next we apply mathematical formula/model to calculate overall difficulty to each
question:
• Difficulty score = w1.feature1 + w2.feature2 + … +wn.featuren
• Where w1,w2…wn are the weights assigned to each feature.
• And feature1, feature2…featuren are the values of each extracted feature.
• Next we need to define the threshold which will classify each question into one of the 3
target categories.
Step 6
Transformer-Based Difficulty Prediction
• Model Selection:Use a pre-trained transformer model (e.g., BERT) to predict difficulty levels.
• Model Fine-Tuning:
• Fine-tune the pre-trained transformer model on the dataset (Note: We will require Bloom’s
wordset, syllabus and the previous year questions as datasets to fine-tune BERT).
• Train the model to learn the relationship between question features and difficulty levels (easy,
moderate, difficult).
• Prediction/Difficulty Level Classification:
• After fine-tuning, use the model to predict the difficulty level of each question.
• Apply the pre-defined thresholds from step 6 to classify each question into one of the three
categories (easy, moderate, difficult).
Step 7

Pass the result of SVM model to training data of BERT


• The question and their corresponding difficulty which is obtained using SVM
classifier is passed as the training data to BERT to fine-tune the model
according to our requirements.
Step 8

Evaluation Metrics and Accuracy


• Accuracy and Metrics: Evaluate the model's performance on both tasks (Bloom's level and
difficulty) using metrics like accuracy, precision, recall, and F1-score.
• Cross-validation: Perform cross-validation to ensure the model's robustness.
• Use the predicted Bloom's level as an additional feature to predict the difficulty level.
Step 9

Validation and Refinement


• Validation:Validate the model using a separate validation dataset
to ensure accuracy.
• Refinement: Refine the model based on validation results to
improve performance.
Step 10

Deployment
• After validating the model and increasing its accuracy to the highest possible
level, we create a csv file to store the final results.
• We deploy the model as APIs.
• Periodically, it has to be retrained with newer datasets.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy