Objectives, Evaluation, and The Improvement of Education: Studies in Educational Evaluation
Objectives, Evaluation, and The Improvement of Education: Studies in Educational Evaluation
Objectives, Evaluation, and The Improvement of Education: Studies in Educational Evaluation
Educational
Evaluation
Studiesin EducationalEvaluation31 (2005) 102-113
ELSEVIER www.elsevier.com/stueduc
Lorin W. A n d e r s o n I
Abstract
For five years, froln 1995 until 2000, a group of eight educators and researchers met
twice annually in Syracuse, NY, for the purpose of revising Bloom's Taxonomy.
Based in part on the structure of educational objectives, in part on advances in
cognitive psychology, and in part on numerous other attempts to classify educational
objectives that were made since the publication of Bloom's Taxonomy, this group
produced a two-dimensional table, known simply as the Taxonomy Table. The
horizontal dimension was a modification of Bloom's Taxonomy, with verb forms
replacing the noun forms of the original category labels: Remember, Understand,
Apply, Analyze, Evaluate, and Create. The vertical dimension consisted of four types
of knowledge: Factual Knowledge, Conceptual Knowledge, Procedural Knowledge,
and Metacognitive Knowledge. The purposes of this article are to (1) describe the
major differences between the original Taxonomy and the Taxonomy Table, (2)
discuss ways in which the Taxonomy Table can be used to examine and ultimately
improve tile quality of assessment and instruction, and (3) explore how the Taxonomy
Table can be used to provide more accurate estimates of curriculum alignment and
opportunity to learn.
On the surface, it seems so reasonable and so simple. You specify the changes in
learners that should result from schooling. You evaluate the extent to which the changes
have taken place. You use the results of the evaluation to improve the schooling process.
Although it is reasonable, it is not simple. Problems arise at each step along the way.
Dick Wolf was aware of all the problems. "One frequently voiced dissatisfaction
with statements of educational objectives, both general and specific, is that they fail to say
anything about the basic structure, intentions, or framework of an educational enterprise"
(Woll; 1979, p. 44) (emphasis mine). Furthermore, although a "great deal of activity goes
on under the heading of evaluation, relatively few people are trained to do such work.
There is also. alas, a widespread lack of agreement about what evaluation is and how
evaluation studies should be conducted" (p. iii) (emphasis mine). Finally, "educational
evaluation is clearly decision-oriented, It is intended to lead to better policies and
practices. If this intention is in any way lacking, evaluation probably should be dispensed
with" (p. 6) (emphasis mine).
One of the early attempts to develop a framework that was intended to promote
agreement among educators and had the potential to lead to better practices was the
Taxonomy of Educational Objectives, which was designed by Benjamin Bloom and his
colleagues. 2 Ahnost a half century after its publication, the Handbook in which the
Taxonomy first appeared remains one of the most familiar educational books of all times
(Bloom (Ed.), Engelhard, Furst, Hill, & Krathwohl, 1956). Educators throughout the world
can recite its six major categories: Knowledge, Comprehension, Application, Analysis,
Synthesis, and Evaluation. Krathwohl (1994) has estimated that the Handbook has been
translated into twenty-one languages.
The decision to embark on a revision of such a classic text was not made easily.
Preliminary discussions of the possibility of a revision began shortly after the publication
of the volume, Bloom's Taxonon~v: A Forty-Year Retrospective, in 1994 (Anderson &
Sosniak, 1994). These informal discussions - between David Krathwohl and Lorin
Anderson - led to a decision to invite a group of educators from across the United States to
attend a two-day meeting at which the feasibility and desirability of revising the Taxonomy
was discussed. By the meeting's end, those attending were in agreement that a revision was
both needed and worth attempting.
Between 1995 and 2000 a group of educators worked on a revision of the
Taxonomy. The group included those with expertise in cognitive psychology (Richard
Mayer, Paul Pintrich, and Merle Wittrock), curriculum and instruction (Lorin Anderson,
Kate Cruikshank, and James Raths), and testing, measurement, and assessment (Peter
Airasian, David Krathwohl). Meetings were held twice a year for five years, with most o f
the writing done between meetings.
As in the case of the original work, the revision was a group effort. The results of
this effort were published as A Taxonomy for Learning, Teaching, and Assessing: A
Revision of Bloom's Taxonomy (Anderson (Ed.), Krathwohl (Ed.), Airasian, Cruikshank,
104 L. W.Anderson / Studies in Educational Evaluation 31 (2005) 102-113
Mayer, Pintrich, Raths, & Wittrock, 2001). The purposes of this article are to (I) examine
the primary differences between the original Taxonomy and the revision, (2) describe how
the revised Taxonomy can be used to examine and improve assessment and instruction, and
(3) discuss how the revised Taxonomy can be used to increase our understanding of
curriculum alignment, particularly opportunity to learn.
As mentioned earlier, the original Taxonomy contained six major categories situated
on a single dimension. Along the dimension the categories were arranged from simple to
complex and from concrete to abstract. That is, knowledge and comprehension were
believed to be more simple and concrete than synthesis and evaluation, which were more
complex and abstract. In addition, it was assumed that the dimension represented a
cumulative hierarchy. That is, mastery of each "lower" category was a prerequisite for
achieving mastery of the next "higher" category. Finally, with the exception of
Application, each category was broken into subcategories. Comprehension, for example,
was subdivided into Translation, Interpretation, and Extrapolation.
Those engaged in revising the original Taxonomy had two major resources at their
disposal. The first was a set of 19 alternative taxonomic frameworks that had been
developed during the time period from the publication of the original Taxonomy to the time
at which the revision was published. Based on his analysis of these frameworks, David
Krathwohl divided them into two general types: unidimensional, representing a single
dimension or set of categories like the original Taxonomy (11 alternative frameworks) and
multidimensional, representing two or more dimensions or sets of categories (8 alternative
frameworks).
The second resource was the curricululn standards that had been adopted by
numerous states and which were intended to be used by teachers in those states to plan and
deliver instruction and to assess learning. As the various standards were examined, two
things became evident. First, standards are simply mandated objectives. That is, they are
objectives that teachers are expected to teach regardless of how important the teachers
themselves believe thena to be. Second, and more importantly, all statements o f objectives
(standards included) have a common grammatical format, namely, subject-verb-object.
The subject is the student or the learner. The object indicates the content that the student or
learner is expected to learn. The verb indicates what the student or learner is expected to
do with or to that content.
Consider the tbllowing objective: The student will use the law of supply and
demand to estimate the costs of goods and services. The subject is "the student," the verb
is "will use," and the objective is "the law of supply and demand." The final portion o f the
objective - " t o estimate the costs of goods and services"- simply indicates the purpose for
which the law of supply and demand (the object) is to be used (verb) by the student (the
subject). When the original Taxonomic categories were examined within this context, it
became apparent that, with one exception, the categories, although stated as nouns, were
intended to function as verbs. This interpretation was supported by comments made by
Bloom and his colleagues (1956) throughout the Handbook. On page 18, for example,
flaey wrote: "We are of the opinion that although the objectives and test materials and
L. W.Anderson / Studies in Educational Evaluation 31 (2005) 102-113 105
techniques may be specified in an almost unlimited number of ways, the student behaviors
involved in these objectives can be represented by a relatively small number of classes.
Therefore, this taxonomy is designed to be a classification of the student behaviors which
represent the intended outcomes of the educational process." (p. 18). Behaviors are verbs.
The lone exception was Knowledge which had a dual function. At first blush,
Knowledge is simply the lowest level of the cognitive Taxonomy. At the same time,
however, statements made by the authors of the original Taxonomy suggest that
Knowledge is different from the other categories in the Taxonomy. Consider the following
excerpt:
A teacher, in classi~ing the goals of a teaching unit, may find that they all fall
within the taxonomy category of recalling or remembering knowledge.
Looking at the taxonomy categories may suggest [that the teacher] could
include some goals dealing with the application of this knowledge and with the
analysis of the situations in which the knowledge is used (Bloom et al., 1956,
p. 2).
A careful reading of this excerpt suggests two important points. First, "recalling or
remembering knowledge," not knowledge p e r se, is the lowest level of Bloom's Taxonomy.
The consecutively higher levels are comprehending knowledge, applying knowledge, and
so on. This is consistent with the contention that the original Taxonomic categories were
intended to function as verbs. Second, knowledge, as used in this way, is a second
dimension, one which crosses all levels of the cognitive dimension. This need for a second
dimension is also consistent with the multidimensional frameworks that David Krathwohl
reviewed.
Factual
knowledge
b.
Conceptual
knowledge
C.
Procedural
knowledge
d.
Meta-
cognitive
knowledge
Source: Anderson, Krathwohl,et al. (2001). Reprintedwith permission.
106 L. W.Anderson / Studies in Educational Evahtation 31 (2005) 102-113
As mentioned earlier, the verb is "use." "Use" suggests Apply. The object is "the law of
supply and demand." "Laws" are "generalizations" which suggests Conceptual
Knowledge. This objective, then, is a special case of the general objective: Apply
Conceptual Knowledge.
Why is important to be able to classify and categorize objectives in terms of the
Taxonomy Table? The best answer to this question may have been given by Bloom
himself ahnost 60 years ago. A taxonomy of educational objectives
could do much to bring order out of chaos in the field of education. It could
furnish the conceptual framework around which our descriptions of
educational programs and experiences could be oriented. It could furnish a
framework lbr the development of educational theories and research. It could
furnish the scheme needed for training our teachers and for orienting them to
the varied possibilities of education (1949, p. 4).
When Bloom came up with the idea of a taxonomy of educational objectives, he was
serving as the Associate Director of the Board of Examinations of the University of
Chicago. It is not surprising, then, that he saw the development of a taxonomy as a way of
reducing the labor of preparing annual comprehensive examination. In fact, the second
half of the original Handbook contained test items for the six taxonomic categories. Like
its predecessor, the Taxonomy Table has implications for the preparation of test items and
other assessment tasks (e.g., performance measures). In addition, however, as suggested by
Bloom in 1949, the Taxonomy Table has implications for planning instruction.
As shown in Figure 1, all assessment tasks are derived from the same blueprint. The three
major components are the introductory material, the stem, and the response. As shown in
Figure 1, the introductory material can take many forms: written, pictorial, or real objects
(e.g., rocks to be classified). The stem can take the form of a question, an incomplete
statement, or a directive (e.g., "Do this." "Prove that."). Finally, the expected response can
be short or long (i.e., extended). Within the short-answer format, students may have to
supply the answer or choose one from a set of response options. Within the extended
response tbrlnat, students may have to write something or perform some action (or series of
actions).
Not all assessment tasks include all three components. In fact, the components that
are included provide clues as to the nature of the objective being assessed. Assessing
Remember Factual Knowledge objectives rarely includes introductory material. Such
material is not needed. In contrast, tasks for the purpose of assessing Apply Procedural
Knowledge quite often contain introductory material. This material provides the context
within which the procedural knowledge is to be applied and provides information needed to
apply the procedural knowledge.
108 L. W.Anderson / Studies in Educational Evaluation 31 (2005) 102-113
INTRODUCTORY MATERIAL
STEM
RESPONSE
Note: To score short-answer tasks, you need a scoring key. To score extended response tasks, you need
criteria and an accompanying checklist, rating scale, or rubric. For short-answer tasks, the
standards are embedded in the task itself (introductory material, stem, response). For extended
response tasks, the standards are largely or completely embedded in the criteria used to evaluate the
response.
Just as different assessment tasks are required by objectives that are placed within
different cells of the Taxonomy Table, different objectives also require different
instructional approaches and techniques. Because Factual Knowledge is so plentiful (since,
lbr example, almost any sentence in a textbook can be designated as important by some
teacher), lbcusing students' attention on the most important terms and details is a crucial
element of effective instruction when Remembering Factual Knowledge is the objective.
The use o f repetition, mnemonic devices, acronyms, and songs are also effective ways of
helping students Remember Factual Knowledge.
1 l0 L.W. Anderson/Studies in EducationalEvaluation 31 (2005) 102-113
neither objectives, nor instructional activities and materials, nor assessments. These cells
indicate potential "missed opportunities" in designing and/or delivering the unit(s).
Using the Taxonomy Table to examine alignment provides at least three benefits
over more conventional approaches. First, because the Taxonomy Table requires
examining objectives in terms of both knowledge and cognitive processes, it provides a
more accurate estimate of alignment (Gamoran et al., 1997). Second, because objectives,
instruction activities and materials, and assessments are each examined in terms of the
Taxonomy Table rather than with each other, the Taxonomy Table emphasizes alignment
in terms of student learning and provides a more in-depth examination of alignment. Third,
because the Taxonomy Table is appropriate for use across subject matters and school or
grade levels, it enables educators to examine differences in alignment from one subject
matter to another or, perhaps more importantly, from one grade level to the next.
Concluding Colnments
Note
I. The author expresses his appreciation to David R. Krathwohl for his insightful comments and
constructive suggestions on an earlier version of this lnanuscript.
2. Although it came to be known as Bloom's Taxonorny, time design of the taxonomy was a group
project. Furthermore, the volume in which the Taxonomy appeared was written by five authors:
Bloom, George Engelhard, Edward Furst, Walker Hill, and David Krathwohl. Bloom was the lead
author and timeeditor of the volume.
The question must be a verbatim transformation of the statement if Remember Factual Knowledge
is being assessed. A paraphrase of the statement would result in an assessment task that assesses
Understand Factual Knowledge.
References
Anderson, L.W. (2002). Curricular alignment: A re-examination. Theoo, into Practice. 41,255-260.
Anderson, L.W. & Sosniak, L.O. (1994). Bloom's taxonomy: A fbrly-year retrospective. Chicago:
University of Chicago Press.
L. W.Anderson / Studies in Educational Evaluation 31 (2005) 102-113 113
Anderson, L.W. (Ed.), Krathwohl, D. R. (Ed.), Airasian, P.W., Cruikshank, K.A., Mayer, R.E.,
Pintrich, P.R., Raths, J., & Wittrock, M.C. (2001), A taxonomy/or learning, teaching, and assessing: A
revision q/'Bloom's Taxonomy qf Educational Ol?jectives. Boston: Allyn & Bacon.
Bloom, B.S. (Ed.). Englehard, M.D., Furst, E.J., Hill, W.H., & Krathwohl, D.R. (1956). Taxonomy
q/'educational objectives'." The classification of educational goals'. Handbook L" Cognitive domain, New
York: David McKay.
Bloom, B.S., Hastings, J.T., & Madaus, G.F. (1971). Handbook on formative and summative
evaluation of student learning. New York: McGraw-Hill.
Gamoran, A., Porter, A.C., Smithson, J., & White, P.A. (1997). Upgrading high school mathematics
instruction: Improving learning opportunities for low-achieving, low-income youth. Educational Evaluation
and Policy Analysis. 19, 325-338.
Krathwohl, D. R. (2004). Reflections on the taxonomy: Its past, present, and future. In L.W.
Anderson & L.O. Sosniak (Eds.), Bloom's Taxonomy. A Jbrty-year retrospective. Chicago: University of
Chicago Press.
Winfield, L.F. (1993). Investigating test content and curriculum content overlap to assess
opportunity to learn..lournal qfNegro Education, 62, 288-310.
The A u t h o r
Correspondence: <AndReGroup@sc.rr.com>