Evaluation of Student Learning:

Test Construction and

Other Practical Strategies
 Guiding Principles for Evaluation
• Evaluation should relate directly to instructional
objectives
• Each evaluation activity should be designed to promote
student growth
– The actual activity should be useful practice in itself
– Feedback should be useable by the student
• Multiple evaluation strategies should be provided to
master achievement of the objective/competency
• Student should clearly understand the methods of
evaluation for a test or activity
Questions to Ask yourself in Designing
a Test
• What objectives will (should) I be testing?
• What types of items will be included in the
test?
• How long will the test be in terms of time and
number of items?
• How much will each objective be worth in
terms of weighting and number of items?
 Tests as Diagnostic Tools

• Students demonstrate learning

• Instructor effectiveness – modify
teaching strategies or activities
• Assignment of letter grades
 Planning a Test

• First step: Outline learning objectives or

major concepts to be covered by the test
– Test should be representative of objectives and
material covered
– Major student complaint: Tests do not fairly
cover the material that was supposed to be
included in the test.
• Second Step: Create a test blueprint
• Third Step: Create questions based on blueprint
– Match the question type with the appropriate level
of learning
• Fourth Step: For each check on the blueprint,
jot down 3-4 alternative questions on ideas and
item types which could be tested on the same
objective
• Fifth Step: Organize questions and/or ideas by
item types
• Sixth Step: Eliminate similar questions
• Seventh Step: Rest (pause) for a couple of days
• Eighth Step: Reread all of the items – try doing
this from the standpoint of a student
• Ninth Step: Organize questions logically
• Tenth Step: Time yourself actually taking the
test and then multiply that by about 4
depending on the level of students
• Eleventh Step: Analyze the results (item
analyses)
 Translating Course
Objectives/Competencies into
Test Items
• Syllabus
– Specification table- what was taught/weight
areas to be tested
• Creating a Test Blueprint
– Blueprint- this is the test plan, i.e., which
questions test what concept
– Plotting the objectives/competencies against
some hierarchy representing levels of
cognitive difficulty or depth of processing
 Thinking Skills

• What level of learning corresponds to the

course content
• Bloom’s Taxonomy of Educational Objectives
– Knowledge
– Comprehension
– Application
– Analysis
– Synthesis
– Evaluation
 Practical Considerations

• Representative sample of the course content

not random– purposeful based on blueprint
• Representative sample of skill or cognitive
levels across content
• Analyze results by level AND content area
 Question Arrangement on a Test

• Group by question type

– Common instructions will save reading time
• Limit the number of times students have to
change frame of reference
• Patterns on test must be logical
– Arrange from a content standpoint
– Keep similar concepts together
• Group by difficulty (easy to hard)
Selecting the Right Type of evaluation

• How do you know what type of

question to use and when?
• It depends on the skill you are
testing (Purpose of the test).
• Evaluation should always match
as closely as possible the actual
activity you are teaching.
 Question Types verses
Cognitive Levels of Learning

Knowledge Application Analysis

Comprehension Synthesis
Evaluation
Multiple Choice (MC) MC MC
True/False (TF) Short Answer Short Answer
Matching Problems Essay
Completion Essay
Short Answer Performance
 Constructing the Test

• Types of Test Questions:

– Multiple-Choice Items
– True-False Items
– Matching Items
– Fill-In, Completion or Short-Answer
Items
– Essay Questions
 Multiple Choice Items

• Advantages:
– Extremely versatile-can measure the higher level
mental processes (application, analysis, synthesis
and evaluation)
– A compromise between a short answer/essay and
T/F item
– Can cover a wide range of content and can be
sampled by one test
• Disadvantages
– Difficult to construct plausible alternative responses
 Types of Multiple Choice Items

• Four Basic Types

– Question Type
– Incomplete Statement Type
– Right Answer Type
– Best Answer Type
• Which Type is Best?
– Question Type vs. Incomplete Statement
– Right Answer vs. Best Answer Type
 Multiple Choice Items

1. Writing the stem first:

A. Be sure the stem asks a clear question
B. Stems phrased as questions are usually easier to
write
C. Stems should not contain a lot of irrelevant
information
D. Appropriate reading level/terms
E. Be sure the stem is grammatically correct
F. Avoid negatively stated stems
 Multiple Choice Items

• Writing the correct response

– Use same terms/reading level
– Avoid too many qualifiers
– Assign a random position in the answer sequence
• Read the stem and correct response together
• Generate the distractors/alternative responses
 Multiple Choice Items

• Other Tips for Constructing MC Items:

– Items should have 4 alternatives.
– Stem should present a single, clearly formulated
problem
– Simple, understandable, exclude extraneous words
from both stem and alternatives
– Include in the stem any word that are repeated in
each response
– Avoid all of the above (can answer based on partial
information)
– Avoid none of the above
 Multiple Choice Items

• Alternative responses/distractors should be plausible

and as homogeneous as possible
• Response alternatives should not overlap
– Two synonymous terms (arithmetic average/mean)
• Avoid double negatives
– None of the following are part of the brain except which one?
• Emphasize negative wording
 Difficulty of Construction.
• Good multiple-choice test items are
generally more difficult and time-
consuming to write than other types of
test items.
• Coming up with plausible distracters
requires a certain amount of skill.
• This skill, however, may be increased
through study, practice, and experience.
• Each item should be independent of other items in the
test

– Information in the stem of one

item should NOT help answer
another item.
 Examples - Comprehension
A pendulum consists of a sphere hanging from a string.
What will happen to the period of the
pendulum if the mass of the sphere is doubled?
(Assume that the effects of air friction and
the mass of the string are negligible, and that the
sphere traces an arc of 20° in a plane as it
swings.)
a. It will increase.
b. It will decrease.
*c. It will remain unchanged.
d. More information is needed to determine what will
happen.
Include as much of the item as possible in the stem,
but do not include irrelevant material.
• Rather than repeating redundant words or phrases in each
of the alternatives, place such material in the stem to
decrease the reading burden and more clearly define the
problem in the stem.
• Poor Example
If the pressure of a certain amount of gas is held constant,
what will happen if its volume is increased?
a. The temperature of the gas will decrease.
*b. The temperature of the gas will increase.
c. The temperature of the gas will remain the same.
d. The temperature of the gas will increase slightly and then
decreases.
• Better Example
If you increase the volume of a certain amount
of gas while holding its pressure constant, its
temperature will:
a. decrease.
*b. increase.
c. remain the same.
d. increase slightly and then decreases
 Keep the grammar of each alternative
consistent with the stem.

• Students often assume that inconsistent

grammar is the sign of a distractor, and they
are generally right.
Poor Example
• A word used to describe a noun is called an:
*a. Adjective.
b. Conjunction.
c. Pronoun.
d. Verb.
 Better Example
• A word used to describe a noun is called:
*a. An adjective.
b. A conjunction.
c. A pronoun.
d. A verb.

• The word “an” in the stem of the poor example above

serves as a clue to the correct answer,
“adjective,” because the other alternatives begin with
consonants. The problem has been corrected in the
better example by placing the appropriate article, “an”
or “a,” in each alternative.
 Try this out
Poor Example
• Which of the following would do the most to
promote the application of nuclear discoveries
to medicine?
a. Trained radioactive therapy specialists.
*b. Developing standardized techniques for
treatment of patients.
c. Do not place restrictions on the use of
radioactive substances.
d. If the average doctor is trained to apply
radioactive treatments.
 Better Example
• Which of the following would do the most to
promote the application of nuclear discoveries
to medicine?
a. Adding trained radioactive therapy specialists
to hospital staffs.
*b. Developing standardized techniques for
treatment patients.
c. Removing restrictions on the use of
radioactive substances.
d. Training the average doctor to apply
radioactive treatments.
 Avoid the use of specific
determiners.
• When words such as never, always, and only
are included in distracters in order to make them
false, they serve as flags to alert the student.
 Poor Example
• To avoid infection after receiving a puncture
wound to the hand, you should:
a. Always go to the immunization center to
receive a tetanus shot.
b. Be treated with an antibiotic only if the
wound is painful.
*c. Ensure that no foreign object has been left in
the wound.
d. Never wipe the wound with alcohol unless it is
still bleeding.
 Better Example

• To avoid infection after receiving a puncture

wound to the hand, you should always:
a. Go to the immunization center to receive a
tetanus shot.
b. Be treated with an antibiotic if the wound is
painful.
*c. Ensure that no foreign object has been left in
the wound.
d. Wipe the wound with alcohol unless it is still
bleeding.
• In the poor example above, the italicized word
in each of the distracters is a specific
determiner. These words have been removed
from the better example by rewording both the
stem and the distracters.
 Avoid including keywords in
the alternatives.
• When a word or phrase in the stem is also
found in one of the alternatives, it tips the
student off that the alternative is probably the
answer.
Keep the alternatives free from clues as to
which response is correct.
• Poorly-written items often contain clues that
help students who do not know the correct
answer eliminate incorrect alternatives and
increase their chance of guessing correctly.
Such items tend to measure how clever the
students are at finding the clues rather than
how well they have attained the objective
being measured.
• The following suggestions will help you detect
and remove many of these clues from your
items.
Poor Example Better Example
• All of the following are • All of the following are
correct procedures for correct procedures for
putting out a fire in a putting out a fire in a pan
pan on the stove except: on the stove except:
• a. Do not move the • a. Leave the pan where
pan. it is.
• *b. Pour water into the • *b. Pour water into the
pan. pan.
• c. Slide a fitted lid onto • c. Slide a fitted lid onto
the pan. the pan.
• d. Turn off the burner • d. Turn off the burner
controls. controls.
 Items of the negative variety
For most educational objectives, a student’s
achievement is more effectively measured by
having
him or her identify a correct answer rather than
an incorrect answer.

Just because the student knows an incorrect

answer does not necessarily imply that he or
she knows the correct answer.
For this reason, items of the negative variety are
not recommended for general use.
• The negative word “except” in
the poor example above is not
emphasized, and
• alternative (a) forms a double
negative with the stem. These
defects have been corrected in
the better example.
 Avoid using unnecessarily difficult
vocabulary
• If the vocabulary is somewhat difficult, the
item will likely measure reading ability in
addition to the achievement of the objective
for which the item was written. As a result,
poor readers who have achieved the objective
may receive scores indicating that they have
not.
• Use difficult and technical vocabulary only
when essential for measuring the objective.
 True-False Test Items

• Best suited for testing 3 kinds of information:

• Knowledge level learning
• Understanding of misconceptions
• When there are two logical responses
• Advantages:
– Sample a large amount of learning per unit of
student testing time
• Disadvantages:
– Tends to be very easy
– 50-50 chance of guessing
– Tends to be low in reliability
Tips for Constructing True/False Items

– Tips for constructing True-False Items

• Avoid double negatives
• Avoid long or complex sentences
• Specific determiners (always, never, only, etc.) should
be used with caution
• Include only one central idea in each statement
• Avoid emphasizing the trivial
• Exact quantitative (two, three, four) language is better
than qualitative (some, few, many)
• Avoid a pattern of answers
 Objective Test Item Analyses

• Evaluating the Effectiveness of Items..

– Why?
• Scientific way to improve the quality of tests and test items
• Identify poorly written items which mislead students
• Identify areas (competencies) of difficulty
– Item analyses provided information on:
• Item difficulty
• Item discrimination
• Effectiveness of alternatives in MC Tests
 Short-Answer Items

• Two Types: (Question and Incomplete Statement)

• Advantages:
– Easy to construct
– Excellent format for measuring who, what, when, and where
information
– Guessing in minimized
– Student must know the material- rather than simply recognize
the answer
• Disadvantages:
– Grading can be time consuming
– More than one answer can be correct
 Short Answer Items

• Tips for Constructing Short Answer Items

– Better to supply the term and require a definition
– For numerical answers, indicate the degree of
precision expected and the units in which they are
to be expressed.
– Use direct questions rather than incomplete
statements
– Try to phrase items so that there is only one possible
correct response
– When incomplete statements are used, do not use
more than one blank within an item.
 Essay Questions

Types of Essay Questions

• Extended Response Question
– Great deal of latitude on how to respond to a question.
– Example: Discuss essay and multiple-choice type tests.
• Restricted Response Question
– More specific, easier to score, improved reliability and validity
– Example: Compare and contrast the relative advantages of
disadvantages of essay and multiple choice tests with respect
to: reliability, validity, objectivity and usability.
 Essay Items

• Advantages:
– Measures higher learning levels (synthesis,
evaluation) and is easier to construct than an
objective test item
– Students are less likely to answer an essay question
by guessing
– Require superior study methods
– Offer students an opportunity to demonstrate their
abilities to:
• Organise knowledge
• Express opinions
• Foster creativity
 Essay Items

• May limit the sampling of material covered

• Tends to reduce validity of the test
• Disadvantages
– Subjective unreliable nature of scoring if a detailed
marking scheme is not prepared
• “halo effect” – good or bad student’s previous level of
performance
• Written expression
• Handwriting legibility
• Grammatical and spelling errors
• Time Consuming
 Essay Questions
• Give students a clear idea of the scope and direction
intended for the answer
– Might help to start the question with the description of the
required behaviour (e.g., compare, analyze)
• Appropriate language level for students
• Construct questions that require students to
demonstrate a command of background info, but do not
simply repeat that info.
• If question calls for an opinion, be sure that the
emphasis is not on the opinion but on the way its
presented or argued.
• Use a larger number of shorter, more specific questions
rather than one or two longer questions so that more
information can be assessed.
 Essay Questions
• You might
– Give students a pair of sample answers to a question of the
type you will give on the test.
– Sketch out a rubric (grading scheme) for each question before
reading the papers OR randomly select a few to read and make
up the grading scheme based on those answers
– Give students a writing rubric
– Detach identifying information and use code numbers instead
to avoid letting personality factors influence you.
– After grading all the papers on one item, reread the first
few to make sure you maintained consistent standards
– Be clear to student the extend to which factors other than
content (e.g., grammar, handwriting, etc.) will influence the
grade.
 Essay Questions

• Tips for constructing Essay Questions

– Provide reasonable time limits for each question
• “thinking and writing time”
– Avoid permitting students a choice of questions
• Will not necessarily get a representative sample of student
achievement. Only be requiring all students to answer all
questions can their achievement be compared
– A definite task should be put forth to the student
• Critical words: compare, contrast, analyse, evaluate,
etc.
• Avoid “to find out” –cannot be measured, why?
 Scoring Essay Items

• Write an outline of the key points (use outline to design

a rubric)
• Determine how many points are to be assigned to the
question as a whole and to the various parts within it.
• If possible, score the test without knowledge of the
student’s name
– Face Sheet
• Score all of the answers to one question before
proceeding to the next question
– Consistent standard
 Scoring Essay Exams

• If possible, score each set of answers within the

same time frame
• Handwriting, spelling & Neatness
– Two separate grades?
• Mastery of material
• Other
 Cheating

• Preventing Cheating
– Reduce the pressure (multiple evaluations)
– Make reasonable demands (length/content of exam)
– Use alternative seating
– Use alternative forms
Any Questions?