Mmitchell - Final 823 Paper
Mmitchell - Final 823 Paper
Mmitchell - Final 823 Paper
Encouraging Higher Level Questioning: Exploring the Effects of Professional Development and
Meredith Mitchell
Abstract
A single subject case study was used to explore the functional relation between
professional development and lesson co-planning on the rate of higher level thinking questions
posed by an elementary school teacher to a class of students. One participant was studied using
an ABAB reversal design by recording the first 15 minutes of direct instruction during the
mathematics block. The recordings allowed for the questions posed by the teacher to later be
coded using Bloom's Taxonomy and further reviewed by an interobserver. The study
professional development and lesson co-planning may be an effective strategy for supporting
Organizations such as the Partnership for 21st Century Learning, and many other
research based organizations, have begun to highlight the necessity of refocusing our
educational system on the significant issue of equipping students with the skillsets they need to
be productive in todays global economy (Kay & Greenhill, 2011; Voogt & Roblin, 2012).
Their research suggests that it is not rote memorization of facts and content knowledge, which
is what the majority of mandated national and state standardized assessments currently
measure, that matters; it is the ability to create, collaborate, communicate and think critically
that will allow young scholars to be equipped with what they need in todays job market
(Supovitz, 2009; Kay & Greenhill, 2011; Voogt & Roblin, 2012). As the body of research on
21st century teaching and learning gains momentum, many school districts have adopted
measures and practices to commit to teaching these educational ideals, which is an important
first step in reforming education (Gunn & Hollingsworth, 2013). While the explicit adoption
of a shared vision of 21st century learning is important, merely stating our intention of better
equipping students for the future cannot and will not suffice.
Schools that will adequately refocus their intentions on developing 21st century skills
must intentionally plan for the strategic implementation of these skillsets in the classroom.
One of these skillsets, critical thinking, has a rich and robust literature base that demonstrates
how teachers might elicit higher level thinking from their students (Ennis, 1985; Kugelman,
n.d.; Seker & Komur, 2008). One such strategy for doing so is a framework of questioning
called Blooms Taxonomy, which delineates six different levels of thinking that become
progressively more complex (Ennis, 1985; Kugelman, n.d.; Seker & Komur, 2008). The
HIGHER LEVEL QUESTIONING 4
extensive research conducted on Blooms Taxonomy has yielded a variety of training and
reference guides that can be utilized in the classroom and throughout professional development
While there is a wealth of information about critical thinking, the utility of Blooms
Taxonomy, and higher level thinking questioning, it will be important to identify which
training and teaching methods result in teachers actually implementing this research for the
suggests that leadership theory, including instructional leadership theory and distributed
leadership theory, hold utility in contributing to a culture of professional growth and sustained
instructional practices (Robinson & Timperley, 2007; Spillane, Halverson, & Diamond, 2001).
organizations within school buildings has been shown to engage educators in learning and
implementation processes that sustain a school improvement (Vescio, Ross, & Adams, 2008).
Currently, significant research would suggest creativity, communication, critical thinking, and
collaboration are skillsets that are important for all our students (Kay & Greenhill, 2011;
Jacobsen-Lundeberg, 2013), but more research needs to be done to determine how to create the
learning environments where these skillsets are practiced. Practically speaking, it will be
important to explore how we change teachers teaching behaviors in order for this vast body of
research to be realized.
improve new teachers ability to provide elementary students with greater opportunities to
answer critical thinking questions. While there is a gap in the literature regarding the utility of
HIGHER LEVEL QUESTIONING 5
these two interventions in higher level thinking questioning, the evidence the literature
intervention (Vescio, Ross, & Adams, 2008). This research will demonstrate whether such
interventions are effective strategies for implementing a district wide vision for students 21st
century learning.
Research Question
higher level thinking questioning and providing them with a lesson co-planning experience
specifically for critical thinking, increase the frequency of higher level thinking questioning
Method
This question will be addressed using a single subject research study. Single subject
research allows for a researcher to quantitatively evaluate how a participant responds to both
baseline and treatment conditions (Gast & Ledford, 2014). In this study, a novice elementary
school teacher will be recorded during her mathematics direct instruction in order to determine
the percentage of questions she asks the class that would be classified as higher level thinking
questions on Blooms Taxonomy. The percentage of higher level thinking questions recorded
during this time will represent the teachers baseline data, but for the treatment phase, the
participant will be trained on Blooms Taxonomy and the six levels of questioning and will
also participate in a lesson co-planning experience where the researcher and participant work
together to generate a set of higher level thinking questions that align to the content being
HIGHER LEVEL QUESTIONING 6
taught in subsequent math lessons. The study will take place within the novice teachers
mathematics class, which is part of a small neighborhood school in a large Mid-Atlantic school
district. It is anticipated that the Blooms Taxonomy professional development and lesson co-
planning experience will result in an increase in the number of higher level thinking questions
Participant
In this study, one novice elementary teacher was monitored during direct instruction
during the first 15 minutes of the mathematics block. This teacher was selected because she is
new to the district, and having taught in the county for less than three months, is largely
unfamiliar with the district level strategic plan and vision for critical thinking and 21st century
learning. The teacher is in her late 30s and is a career switcher who moved from California less
than a year ago. She has had some previous teaching experience in a neighboring district in the
prior school year. She was hired by the school in May 2016 and after being evaluated by an
administrator, was the recipient of some instructional support in an effort to improve the quality
of instruction in certain subject areas, namely mathematics and language arts. The teacher is in
charge of one homeroom class of 28 students and she teaches them all core curricular subject
areas. The students represent a diverse population of learners, including seven English Language
learners. She meets weekly with a collaborative team within her grade level who support each
other in planning, instruction and assessment. This collaborative team includes two other
general education teachers, one special education teacher, and one special education instructional
assistant.
The participant was selected from a pool of teachers at the study site who have less than 3
years teaching experience and are unfamiliar with the districts strategic plan for 21st century
HIGHER LEVEL QUESTIONING 7
learning. This made it more likely that the participant was not currently utilizing critical thinking
questioning techniques, which would leave more room for improvement in this regard. There
were two other new teachers that fit these selection criteria who were excluded from the study.
These two teachers both had at least one additional year of teaching experience and have also
been evaluated by an administrator who determined they were currently meeting an acceptable
standard for instructional quality, and therefore would not have as much to gain from such an
intervention. For this reason, the researcher focused intervention efforts on the single selected
participant. Because the researcher did not intend on conducting a formal study with IRB
approval, in lieu of an informed consent document, the researcher had a face to face conversation
with the participant describing that they would be a support during the mathematics block and
asked permission to utilize this time in her classroom to conduct some research on questioning
practices. While the researcher was hesitant that the participant would not accept the conditions
of the study, the participant was actually extremely gracious and eager to accept both support
within the classroom, the chance to learn about higher level thinking questioning techniques, and
also excited to reciprocate some help and support back to the researcher.
Setting
The study was conducted within a school district that will be referred to as Large Mid-
Atlantic County. Large Mid-Atlantic County is named for its actual geographic location and
demographics, and was selected not only for convenience purposes, but also because it has
recently adopted a vision and mission at the district level to provide students with more
frequent and authentic 21st century learning experiences. The school selected within this
district is a smaller neighborhood school servicing nearly 500 students from PreK to 6th grade.
This school has recently become a Local Level IV services provider, meaning it has a
HIGHER LEVEL QUESTIONING 8
specialized Advanced Academics curriculum it can offer to qualifying students within the 3rd
through 6th grade. The school is predominately comprised of students who identify as White
(56% of students), with the remaining students identifying as Asian (25%), Hispanic (8%),
Black (4%) or Other (7%). Approximately 85% of students are proficient in English, meaning
about 15% of the students are classified as English Learners and receive special services to
support their language acquisition. About 88% of students are not recipients of free or reduced
priced meals which can be used as a proxy for determining that about 11% of students identify
The classroom within the school is a general education classroom and is not one of the
classrooms receiving the specialized advanced curriculum. The grade level is considered an
upper elementary classroom and participates in the mandated state testing requirements in both
mathematics and reading. There are 28 students in the classroom and all participate in a
mathematics instructional block from 9:00- 10:00 am each morning throughout the school
week. The mathematics block follows a math workshop model, also known as guided math,
which begins with a brief opportunity for students to check their previous nights homework,
then a 15 minute direct instruction lesson about the topic, followed by two to four independent
or small group activities for students to complete. The 15 minute direct instruction lesson
served as the source of the data the researcher collected because it was during this time when
the teacher was addressing the group of students in order to build upon their understandings,
teach new content, and check for understanding. This teacher directed time is when the
teacher herself was posing the most questions for student learning. During this instructional
time, students gather at the front of the room on a large carpet and sit arranged in rows facing
the board and SMARTboard. The teacher then directs a lesson about the content for the day
HIGHER LEVEL QUESTIONING 9
using either the white board to show examples or the SMARTboard to show examples and
sometimes instructional videos used to teach the content to the students. This time also
incorporates frequent questioning of students and when posed a question, students are directed
to turn and talk to a student sitting near them or to raise their hand to respond to the teacher
directly. This direct instruction time typically begins when students leave their desk area,
where they were checking homework, and concludes when students are dismissed back to their
Independent Variables
and lesson co-planning experience on the rate of higher level thinking questions posed during
instruction. For each treatment phase, the participant received both the professional
development training and the planning experience in the school day prior to the beginning of
the treatment phase. These experiences were provided by the researcher during either the
participants planning block from 10:50 11:55am, for the first treatment phase, or at end of
the school day at 1:30pm, for the second treatment phase. The second treatment phase training
occurred at the end of an early closing day, where students were dismissed from school two
training on the six levels of questioning in Blooms Taxonomy, using the reference provided in
Appendix A. After describing the meaning of each of the six level of questioning and going
over each levels keywords and question starters, the participant was asked to generate their
own question that would be classified within each level of the taxonomy, using the reference
as a guide. These exemplars were confirmed by the researcher and assessed for
HIGHER LEVEL QUESTIONING 10
correction if any of the participants responses did not suit the six levels of questioning,
The next part of the intervention consisted of the researcher and participant planning
the math content, resources, and learning activities, as well as generating higher level thinking
questions to pose to the class for each of the subsequent five math lessons that the participant
was going to teach. The planning template in Appendix B was used to structure and guide
these planning sessions. Additionally, the Blooms Taxonomy questioning guide was readily
available for reference throughout these co-planning sessions. It is notable that the template
prompted the researcher and participant to generate up to six higher level thinking questions,
Dependent Variable
The study aimed to see how this intervention, a professional development and lesson
co-planning session, affected the percentage of higher level thinking questions asked during
mathematics direct instruction. To calculate this percentage, the researcher tallied the
frequency of lower level questions during the 15 minute observation block and also tallied the
frequency of higher level questions during the same block of time. The rate of higher level
questions posed was calculated by dividing the number of number of higher level questions
asked by the total number of questions asked, and then converting that decimal to a
percentage.
Level III (Application) on Blooms Taxonomy were categorized as lower level thinking
questions. Knowledge questions are those that ask students to recall basic facts or vocabulary
HIGHER LEVEL QUESTIONING 11
and will include keywords including: what is, who, why, when, where, which, choose, find,
select, name, recall, select, and describe. Comprehension questions are characterized as
questions that require students to interpret or relate concepts, and include key terms such as:
compare, contrast, interpret, relate, summarize, classify, show, explain, and infer. Application
questions will require students to apply knowledge or skills in a new way and will include
question frames such as: build, apply, plan, model, organize, and solve.
Analysis questions will require students to find evidence within given information in order to
support their ideas or inferences. These questions will use key terms including: analyze,
categorize, classify, compare, contrast, examine, infer, conclusion, and relationship. Synthesis
questions will require students to combine given information to propose a new solution and will
include key words such as build, choose, combine, construct, create, predict, plan, improve,
formulate, theorize, and develop. Evaluation questions will require students to make judgments
about the quality or validity based on a set of criteria. These questions may include keywords
such as determine, justify, measure, compare, prioritize, prove, disprove, assess, and interpret.
Materials
materials or support during the two baseline phases of the study. While the participant
continued to plan mathematics content with her collaborative learning team, which consisted
of her grade level teammates and a special education teacher, this support represents the status
quo of what was received by the participant in terms of instructional support. The teams
planning sessions were not generally focused on questioning techniques to use during
HIGHER LEVEL QUESTIONING 12
instruction.
The researcher used an iPhone to voice record the 15 minute mathematics direct
instruction lesson during the baseline phases. The researcher also used the recording tool in
Appendix C to record the level of lower and higher level questions posed during each recorded
session of the mathematics direction instruction shortly after the session was recorded. A
Taxonomy guide and five lesson planning templates during the two treatment phases of the
study. The participant was allowed to keep the Blooms Taxonomy guide after the initial
training, and was also allowed to use this guide throughout the lesson planning session which
immediately followed training. The Blooms Taxonomy guide is attached in Appendix A and
the co-planning lesson template is provided in Appendix B. A sample record of the co-
The researcher used an iPhone to voice record the 15 minute mathematics direct
instruction lesson during the treatment phases, just as in the baseline phases. The researcher
also used the same recording tool from the baseline phase, in Appendix C, to record the level
of lower and higher level questions posed during each recorded session of the mathematics
direction instruction shortly after the session was recorded. A sample record from the
Research Design
The design of this study is characterized as an ABAB design, or reversal design (Gast
& Baekey, 2014). This design allows the researcher to use the same participant in order to
replicate the first basic effect demonstrated within the first two conditions. This allows for
HIGHER LEVEL QUESTIONING 13
some experimental control and also holds an advantage over ABA designs, which does not
leave the participant in a treatment phase and also does not allow for another demonstration of
effect from baseline into treatment. This design is particularly appropriate for the current study
questioning behavior is likely to be reversed without the co-planning intervention, and while
ABAB designs are best suited for behaviors that are likely to be reversed at baseline (i.e. not
learned behaviors), it is reasonable to assume that a novice teacher likely wont plan for higher
level thinking questioning on their own, unless provided the structure and support to do so.
This study was designed to meet all four design standards, including: a systematic
three opportunities for a basic effect to be demonstrated, and a total of five data points in each of
the four phases (Kratochwill, Hitchcock, Horner, Levin, Odom, Rindskopf, & Shadish, 2010). In
this study, a professional development and co-planning session were employed only for the five
sessions in each treatment phase; no other supports were given to the participant during this time
or during baseline. This allowed the researcher to determine precisely how conditions differed
from baseline to treatment. At least one session in each of the four phases was planned to be
assessed by an additional observer using the permanent product of an audio recording. This
allows for 20% of all data to be assessed in each condition. Adequate training and the ability to
review the permanent product will help ensure that at least 80% agreement was met in each
phase. The reversal design allows for three basic effects to be demonstrated: between A1 and B1,
between B1 and A2, and between A2 and B2. This design also requires at least five data points
within each phase, which is also reflected in this studys procedures which includes precisely
HIGHER LEVEL QUESTIONING 14
Baseline procedures. The researcher entered the classroom at 9:00am each morning and
set the recording device, an iPhone, to voice record the events taking place in the classroom.
Shortly after 9:00am, the participant would instruct students to get out their homework, check
their answers with a partner, and upon finishing, make their way to carpet located in the front of
the classroom. Students were in the routine of transitioning to the carpet and arranging
themselves in neat rows facing the white board and SMARTboard mounted on the front wall.
When students were settled, the researcher listened for the precise moment when the participant
began direct instruction related to the content of the days math block. For example, if the
participant first asked students about how students were doing or if they had completed their
previous nights homework, these questions were ignored by the researcher. When direct
instruction began, the researcher would look at the timer on the voice recording and record the
minutes and seconds, for use as a reference for when the permanent product was later reviewed.
The participant engaged in a 15 to 20 minute lesson that was developed entirely on her own or
with the help of her grade level teammates in a previous planning session. After students were
dismissed to their seats to work on independent or group activities, the researcher stopped the
audio recording. Within seven days of each recording, the researcher reviewed the permanent
product, using the recording tool in Appendix C to record the frequency of lower and higher
level questions posed during the direct instruction time. This data was then used to calculate the
percentage of higher level questions asked during the session. The researcher also logged the
Appendix G.
HIGHER LEVEL QUESTIONING 15
Treatment procedures. During the school day prior to each treatment phase, the
researcher scheduled a time to meet with the participant and at least one other member of the
participants grade level team to conduct the Blooms Taxonomy based professional
development and engage in a lesson co-planning session for the lessons that were to be taught in
the subsequent five days of mathematics instruction. This single training session served to
prepare the participant for all lessons that would be taking place during the treatment phase.
The grade level teammates were provided the fidelity of treatment checklist in Appendix H, and
were asked to complete the checklist throughout the professional development portion of the
meeting.
The researcher presented the Blooms Taxonomy reference guide, included in Appendix
A, to the participant and focused on each of the six levels of questioning. The training included
going over the keywords and question starters used at each level and concluded with the
researcher checking for the participants understanding by asking them to generate an exemplar
question at each level. Each training lasted about 15 minutes and upon completion, the
researched and participant immediately engaged in planning for the subsequent math lessons.
The researcher and participant used the grade levels mathematics pacing guide, provided
by the county, to identify the lessons and standards that were to be taught over the next five days.
After the standards were identified, the participant and researcher planned activities and
resources to be used during each days lesson. This planning process was recorded on a lesson
planning template, included in Appendix B. The template also required the researcher and
participant to develop up to six possible higher level thinking questions that would be suited for
that individual lesson, with a possibility of two questions at levels IV, V, and VI of Blooms
Taxonomy. A sample lesson plan is included in Appendix E. After this process was completed
HIGHER LEVEL QUESTIONING 16
for all five lessons of the treatment phase, the training session concluded.
After this professional development and planning session, the data collection sessions for
the treatment phase utilized the same procedures as the baseline phase. There were instances
when disruptions during planned data collection sessions of the treatment phase resulted in the
researcher meeting an additional time to co-plan additional lessons for the subsequent teaching
days. The additional lesson co-planning session did not include the Blooms Taxonomy training,
the researcher simply worked with the participant on two additional lesson plan templates so that
the treatment phase could still include five data points and continue to meet design standards
Interobserver Agreement
The recording of one session of each baseline and each treatment phase was also shared
with an additional observer, who, as a graduate student, has experience with data coding and as a
teacher, has experience with observing instruction. The additional observer was trained using
the same procedures as the participant, and was provided the same Blooms Taxonomy guide,
attached in Appendix A. After the researcher reviewed the six levels of questioning, described
their key characteristics, and listed the common keywords and sentence starters for each level,
the interobserver was then asked to generate an exemplar question for each level of questioning.
The interobserver responded with appropriate exemplars for each level of questioning, indicating
that they had the knowledge and understanding necessary to discern whether or not the
participants questions were to be classified as a lower level or higher level question. The
observers responses represent training to 100% agreement with the researcher, which is
Because each session was voice recorded on an iPhone, the observer randomly selected a
HIGHER LEVEL QUESTIONING 17
session from each of the four phases to listen to and record data from. The selection of one
session from each phase represents 20% of the total collected data, and also represents 20% of
the data within each baseline and each treatment phase (Kratochwill, et al., 2010). Due to
scheduling conflicts and time constraints, all four of the interobservers data coding sessions
were conducted at the same time at the end of the study using the permanent products. While
this scenario is not ideal, using the total agreement formula to calculate IOA, 100,
the interobserver agreement coefficient was calculated for each of the four sessions. In the first
baseline, second baseline, and second treatment phases, there was an interobserver agreement of
100%, whereby both the participant and the researcher coded the same percentage of higher level
thinking questions posed by the participant. However, in the first treatment phase, the
interobserver agreement was calculated to be just 91%, as there was a discrepancy between the
researcher and observer as to the total number of questions posed to the group. The average of
Fidelity of Treatment
During the professional development and co-planning intervention, a range of one to two
observers were present to verify that the researcher adhered to the intervention procedures. For
the first treatment phase, which was conducted during the participants planning time, two of the
participants grade level team members were available to observe the session and utilize the
checklist in Appendix G which denoted the steps to be taken by the researcher throughout the
training. These included introducing each level of Blooms Taxonomy, sharing the key
characteristics, keywords, and question frames used within each of the six levels, and prompting
the participant to generate their own examples of questions that could be classified within each
Each of the fidelity of treatment observers were familiar with Blooms Taxonomy and
agreed to check off each step as they saw it implemented by the researcher. While both
observers participated prior to the first treatment phase, only one of the observers was available
to participate before the second treatment phase, due to the early school closing prior to a long
holiday weekend. These observers also checked over the completed co-planning templates to
ensure completion. While there no discrepancies concerning the adequate completion of each
planning template, if there had been, the researcher and participant would have gone back to the
The fidelity of treatment was calculated by looking at the percentage of check marks on
the checklist that were marked by each observer as a percentage of the total check marks that
were possible on the checklist. In the case of the first treatment, the average of these two
percentages was calculated. The fidelity of treatment for the first treatment phase was 100%, as
both observers indicated that all steps on the checklist occurred during the intervention. The
fidelity of treatment of the second treatment phase was also 100%, as the remaining observer
indicated that all steps in the intervention were carried out a second time.
Social Validity
A subjective evaluation of the social validity was collected throughout the course of the
research study using interviews with the participant and observers. Prior to the studys first
intervention phase, the researcher asked the participant and grade level team about their
perceptions of the importance of questioning in the classroom, to which the team unanimously
responded that they felt questioning was an important teacher practice, but also explained was
difficult to master on the spot. The teachers also lamented how it can be difficult to ask the kinds
of questions that require students to make broader connections across the curriculum. This
HIGHER LEVEL QUESTIONING 19
feedback validates the concept of taking the time to explicitly plan out questions for lessons, to
ensure students are subjected to rich, thought provoking questions that are often difficult to
develop on the spot, especially for teachers who are new to the field.
After the second intervention, the fidelity of treatment observer was asked about the
utility of the training tools and lesson planning template utilized throughout the intervention and
whether or not these procedures were reasonable and adequate to support the studys goals. The
observer felt that the materials effectively prompted teachers to engage with different levels of
questioning and provided a no excuses bank of questions from which the teacher could draw
upon throughout their lessons, but also noted that it would be difficult to maintain this level of
training and this level of detail throughout all subject areas for every lesson taught during the
school day. This concern about reasonability and practicality will certainly be noted as a
limitation of the study, even though the observer commented on its potential efficacy.
Data Analysis
Visual analysis. The researcher utilized the evidence standards to conduct a visual
analysis of the collected data (Kratochwill, et al., 2010). Visual analysis requires that baseline
line data be assessed to determine whether or not it is indicative of a relevant problem and
demonstrates a predictable pattern within the phase. Secondly, each phase must be assessed to
determine whether the data points demonstrate a predictable pattern within the phase. Third, a
visual analysis must include an assessment of the basic effect between phases. This is
accomplished by considering the six components of basic effect: level, trend, variability,
immediacy, consistency, and overlap of data. Three of these components are analyzed within the
phase; the level, which represents the mean of data within a phase, trend, which represents the
slope of the best-fit line within a phase, and variability, which is the deviation of the data around
HIGHER LEVEL QUESTIONING 20
the best-fit line. The remaining three components, are analyzed between phases; immediacy
demonstrates the magnitude of change between one phase and the next, consistency is a measure
that describes the extent to which data patterns are similar across like phases, and overlap
indicates the percentage of data in a treatment phase that is in the same data range as the phase
For the overlap measure, the most commonly used effect size, Percent of Non
Overlapping Data (PND) will be calculated within this study. This method identifies the highest
point in baseline, and then assesses the percent of points in treatment that exceed that level. This
is an appropriate measure for this study because it was not anticipated that there would be any
extreme outlier data, which is a limitation of PND. The researcher will use Scruggs, Mastropieri,
and Castos guidelines which indicate a PND greater than 70% means the treatment was
effective, 50 -70% means the intervention has questionable effectiveness, and less than 50%
means there was no observed effect (1987). Lastly, visual analysis includes an assessment of
experimental control, wherein studies with strong effects show three different demonstrations of
Results
Visual Analysis
During this study, the participant alternated between a baseline phase where the percent
of higher level thinking questions during the teachers mathematics direction instruction was
calculated, and a treatment phase, where the percent of higher level thinking questions was
each session, every question posed during the mathematics direct instruction was coded as either
higher level or lower level using Blooms Taxonomy, and the percent of total questions that were
HIGHER LEVEL QUESTIONING 21
classified as higher level thinking questions was calculated. The data collected from the study
indicate that the participant demonstrated a mean increase between baseline and treatment (See
Figure 1), which was the anticipated direction of change. In the baseline phases, the participant
demonstrated a mean of 0 (SD =0) for percentage of higher level questions posed to the class
during direct instruction. In treatment phases, the participant demonstrated a mean of 10.10 (SD
= 6.74) for percentage of higher level thinking questions asked, demonstrating a mean increase
of 10.10% from baseline to treatment. The overall trend was slightly negative as the participant
demonstrated a slight downward trend between treatment phases. Within each phase, variability
was nonexistent in baseline and was much higher in treatment phases, with a range of 20
percentage points between the highest and lowest data points in the treatment phases. The
participant demonstrated an immediacy of change upon the introduction of the intervention. The
overlap between phases was measured using Percentage of Non-Overlapping Data (PND;
Scruggs, Mastropieri, & Casto, 1987). The PND across all phases was 80%, which indicated that
the treatment was effective given the standard that a PND greater than 70% is considered
effective (Scruggs & Mastropieri, 1998). The data from the participant also demonstrated
consistency, as an increase in higher level thinking questioning occurred during the intervention
phase. Based on the visual inspection of data presented in Figure 1, there is evidence of a
25%
Rate of Higher Level Questions Baseline 1 Treatment 1 Baseline 2 Treatment 2
20%
15%
10%
5%
0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Sessions
Figure 1. Higher level thinking questioning. The percent of higher level thinking questions
asked during baseline () and across intervention () of one participant during direct
In the first baseline phase, it is clear that there is a predictable pattern of no higher level
thinking questions being posed to the students in the classroom. This consistent lack of higher
level thinking questions being asked yields a mean of 0 (SD =0) within the phase. The stability
of the data after the collection of five data points indicated to the researcher that it was
In the first treatment phase, there was immediate effect compared to the percentage of
higher level thinking questions asked in baseline, and an apparent increase in the level of the
treatment phase data. The mean of data in second treatment phase was 13.2 (SD= 5.89). The
trend of this data is increasing throughout the duration of the phase and a PND of 100% was
calculated, indicating that all treatment data in this phase exceeded that of the baseline data.
While the data pattern in the first treatment phase lacks a strong pattern of consistency, the other
measures of basic effect demonstrate strong evidence for the potential efficacy of the
HIGHER LEVEL QUESTIONING 23
In the second baseline phase, there is another predictable pattern of no higher level thinking
questions being posed to the students in the classroom. The data continues to be consistent
throughout the phase and results in a mean of 0 (SD =0) within the phase. The stability of the
data after the collection of five data points again justifies the initiation of the second treatment
phase.
In the second treatment phase, there was only a slight immediacy of effect compared to the
percentage of higher level thinking questions asked in the previous baseline phase, but a clearer
increase in the level of the treatment phase data. The mean of the data in the second treatment
phase was 7.0 (SD= 6.59). There is no clear trend in this data throughout the duration of the
phase and a PND of 60% was calculated, indicating that there is only a moderate demonstration
of effect compared to the previous phase. While the data pattern in the second treatment phase
also lacks a strong pattern of consistency, the other measures of basic effect demonstrate at least
moderate evidence for the potential efficacy of the intervention compared to the previous phase.
Discussion
The overall assessment of the baseline and treatment data can be analyzed as providing
moderate to strong evidence of a functional relation given the evidence standards (Kratochwill,
et al., 2010). The baseline data demonstrated a problem pertaining to the research question, in
that there is a predictable pattern of no higher level thinking questions being posed to the class
throughout phases where no intervention was in place. In each phase, there was a predictably
stable pattern of responses and a basic effect is evident between phases: the percentage of higher
level thinking questions posed to the class consistently increased when the participant was
trained and planned with the researcher, and the percentage of higher level thinking questions
HIGHER LEVEL QUESTIONING 24
dropped when no intervention was in place. Experimental control was established through the
three demonstrations of effect at different points in time, between each of the four phases, and at
no time did an intervention phase fail to produce a mean increase compared to baseline.
Practical Implications
Students will need 21st century learning skills to be successful in their futures and this
includes opportunities for students to think critically at higher levels. It will be important for
teachers in todays 21st century classrooms to understand how to pose higher level thinking
questions to their students and to regularly implement this type of questioning in their
classrooms. As the body of research related to 21st century skills grows, and the importance of
preparing todays students for new, dynamic challenges in a globalized economy becomes
more evident, it will be crucial for educational leaders and administrators to be able to identify
which teachers are successfully providing their pupils with higher level thinking opportunities
and how best to support teachers who can improve in this regard. While it may prove
impractical for teachers to co-plan every lesson with an educator skilled at posing higher level
thinking questions, it may be possible to develop these co-planning capacities within schools
While this study yielded some noteworthy findings, it is important to address the major
limitations of this study. One of the largest considerations is the fact that there was a preexisting
professional relationship between the researcher and participant prior to the beginning of the
study, which may have influenced the willingness of the participant to put forth the effort
development or training may have yielded behavioral changes that may not have emerged
HIGHER LEVEL QUESTIONING 25
otherwise. It is also possible that these changes were intentionally made in the presence of the
researcher, with little intention to continue the behavior beyond the scope of the study.
Another limitation of the study pertains to the timing of the second intervention training
and co-planning session. Due to the tight timeline of the study, this session necessarily fell after
the fifth session in the second baseline phase, which also happened to be the Wednesday before
students and teachers left on their Thanksgiving break. This timing was not ideal in terms of the
participant also divulged to the researcher that she would be resigning from her current teaching
position and would only be carrying out her professional responsibilities for another two and half
Another limitation to the study concerns the time that is necessarily expended in order to
plan each individual lesson with the participant. This level of support would not be practical for
the long term so without future research to determine whether or not the procedures resulted in
the improvement of teachers planning behaviors, or their ability to spontaneously generate higher
level thinking questions throughout the course of their instructional delivery, it is difficult to
ascertain whether or not the procedures would be worthwhile. Future research should include a
maintenance phase or focus on developing the capacity to plan for questioning within grade level
teams in a school. Future research may also employ some randomization techniques to discern
whether or not the small increases in the percentage of higher level questions posed during
Reflection
My experiences in learning about single subject research designs and planning and
HIGHER LEVEL QUESTIONING 26
conducting my own study were very valuable in demonstrating the multiple factors and
considerations one must anticipate when conducting research. The greatest implication of this
experience for me was understanding that even though I thought I had planned for and deeply
considered my methodology and procedures, the process of data collection showed me there
were many things I had in fact not factored in to my design ahead of time and needed to adjust
for throughout the research process. For example, I had planned to conduct my data collection
during the direct instruction portion of the math workshop lesson, which I realized in practice, is
perhaps too loose of a definition. Prior to my research, I tended to think of this instructional time
as the time shortly after the beginning of class where students have just completed checking their
homework in pairs and gathered on the carpet for the mini-lesson. I have been accustomed to
observing and participating in this routine, but when in the process of data collection, I realized
that this direct instruction time is not always fully utilized for new instruction; it is sometimes
used for some routine book keeping; i.e. questioning students about their partners homework
completion, going over the schedule for the day or noting upcoming assignments and quizzes,
etc. This type of questioning was not instructional in nature, and often made it difficult for me to
ascertain when the precise moment that actual instruction was beginning. Even though I was
familiar with the research setting prior to data collection, I never would have realized how much
time was devoted to these items prior to conducting this study. I ultimately had to readjust my
Another realization that only came about through the process of data collection is that
even with operationalized definitions based in a largely researched concept, it was still difficult
to categorize the level of questioning for some questions. It is true that in Blooms Taxonomy, it
HIGHER LEVEL QUESTIONING 27
is possible to use a particular keyword in multiple levels of questioning, so that ambiguity often
made it much more difficult to discern a level for each question asked. For this reason, I am
extremely glad to have gone through with collecting a permanent product, a voice recording of
each direct instruction lesson, in order to capture each question asked and be afforded more time
to deliberate on the best categorization for each. I ultimately only cared about whether or not a
question fell in one of two categories, lower level or higher level, as opposed to Blooms six
levels of questioning, which made these determinations easier, but quickly realized that it would
have been an almost insurmountable task to come up with a consistent level of interobserver
While the permanent products certainly relieved the pressure of coding data on the spot,
one of my greatest lessons was learning to not defer coding the data. I did not make time to code
any of my initial baseline data until the very end of the phase, and it was only then that I realized
how difficult it was to hear based on the position I had placed my iPhone, the recording device I
used, relative to my participant. I ultimately had to hook up the recording device to a pair of
large external speakers which proved to be very cumbersome. At the very least, I was able to
catch this issue early enough in the process to adjust for future recordings. This initial recording
issue was not my only technological faux pas, however; during one session my recording device,
my personal iPhone, was not adequately charged, ran out of battery, and resulted in me needing
to exclude the session from my data collection. This was supposed to be a session in my first
treatment phase, which then also meant I had to go back to my participant and conduct an
additional lesson co-planning session to reflect the material covered for an additional
instructional day.
While these lessons are largely a result of my own learning process as a researcher, I also
HIGHER LEVEL QUESTIONING 28
learned about how tremendously factors outside of my control have and will change my research.
While the structure of the math block is largely consistent throughout the school year, and
largely conforms to the math workshop model, there are times when the typical structure was
broken by the teachers choice or an external factor, such as school assemblies, an assessment
that day, and participant absenteeism. While I ultimately had to adjust to these schedule
disruptions, whether it meant skipping a planned session because the participant was absent or
recording a more limited direct instruction time than what usually occurred, it helped me to
realize just how difficult it is to control for the variety of variables that may impact how a
particular session might go. As a researcher, these adjustments can be quite frustrating, but as a
practitioner, I also understand that these changes in plans are sometimes unavoidable.
A final reflection I have for my research study is an issue I have grappled with
transformation of my data best illustrates how a teachers questioning practices impact students
exposure to higher level thinking opportunities? On some occasions during my study, a very rich
discussion occurred after the higher level thinking question was posed by the teacher, however,
sometimes the teacher also happened to ask a higher frequency of lower level questions in these
sessions, essentially washing out the data for what actually occurred that day in the classroom.
One well posed question can certainly result in many minutes of discussion, so in the future it
might be useful to look at the duration of time students have to consider the higher level thinking
questions posed during their instructional time. While this ultimately changes the behaviors
being observed in the teacher and necessitates a different recording tool and procedure, along
with factoring in things such as how much wait time should ideally be provided to students, it
may yield more reflective results to what students were experiencing in terms of higher level
HIGHER LEVEL QUESTIONING 29
While single subject research design is not a typical methodology used within my field of
study, this class and this experience have been a very worthwhile experience. I am glad to have
educational leader, can see the utility in understanding this methodology for use with students
who may have special learning circumstances. Additionally, the lessons I have learned through
the process of conducting a study are not necessarily specific to this research design, and will
ultimately shape my way of thinking and encourage my preparedness when I embark on future
research endeavors.
HIGHER LEVEL QUESTIONING 30
References
Ennis, R.H. (1985). A logical basis for measuring critical thinking skills. Educational
Gast, D.L. & Ledford, J.R. (2014). Applied research in education and behavioral sciences. In D.
Gast & J. Ledford (Eds.) Single case research methodology: Applications in special
education and behavioral sciences (pp. 1-18). New York, NY: Routledge
Gast, D.L. & Baekey, D.H. (2014). Withdrawal and reversal designs. In D. Gast & J. Ledford
Gunn, T. & Hollingsworth, M. (2013). The implementation and assessment of shared 21st
Kay, K. & Greenhill, V. (2011). Twenty-first century students need 21st century skills. In G. Wan
& D. Gut (Eds.), Bringing schools into the 21st century (pp. 41-65). Netherlands:
Springer. http://dx.doi.org/10.1007/978-94-007-0268-4_3
Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M &
http://www.bloomstaxonomy.org/Blooms%20Taxonomy%20questions.pdf.
HIGHER LEVEL QUESTIONING 31
Robinson, V. & Timperley, H. (2007). The leadership of the improvement of teaching and
learning: Lessons from initiatives with positive outcomes for students. Australian Journal
Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative synthesis of single
subject research: Methodology and validation. Remedial and Special Education, 8, 24-33.
Scruggs, T. E., & Mastropieri, M. A. (1998). Summarizing single-subject research: Issues and
Seker, J. & Komur, S. (2008). The relationship between critical thinking skills and in class
Spillane, J. P., Halverson, R., & Diamond, J. B. (2001). Investigating school leadership practice:
Supovitz, J. (2009). Can high stakes testing leverage educational improvement? Prospects from
the last decade of testing and accountability reform. Journal of Educational Change, 10,
211-227.
Vescio , V. Ross, D. & Adams, A. (2008). A review of research on the impact of professional
learning communities on teaching practice and student learning. Teaching and Teacher
Voogt, J. & Roblin, N. (2012). A comparative analysis of international frameworks for 21st
Appendix A
The following reference was printed out in hard copy and used at the beginning of both
treatment phases as an introduction and training tool on Blooms Taxonomy. It was adapted
from an online resource to provide both the participant and the interobserver a succinct guide
with useful keywords and question starters for each of the six levels of questioning (Kugelman,
F., n.d). After the initial training, both the participant and interobserver were given the reference
BLOOMS TAXONOMY
higher order thinking. By providing a hierarchy of levels, this taxonomy can assist teachers in
designing performance tasks, crafting questions for conferring with students, and providing
This resource is divided into different levels each with Keywords that exemplify the level
and questions that focus on that same critical thinking level. Questions for Critical Thinking can
be used in the classroom to develop all levels of thinking within the cognitive domain. The
results will be improved attention to detail, increased comprehension and expanded problem
solving skills. Use the keywords as guides to structuring questions and tasks. Finish the
Level I Knowledge
Level II Comprehension
HIGHER LEVEL QUESTIONING 33
Level IV Analysis
Level V Synthesis
Level VI Evaluation
Exhibits memory of previously learned material by recalling fundamental facts, terms, basic
Keywords: who, what, why, when, omit, where, which, choose, find, how, define, label, show,
Questions:
What is? Can you select? Where is? When did ____ happen?
Who were the main? Which one? Why did? How would you describe?
When did? Can you recall? Who was? How would you explain?
How did ___happen? Can you list the three..? How is?
Keywords: compare, contrast, demonstrate, interpret, explain, extend, illustrate, infer, outline,
Questions:
How would you classify the type of? How would you compare? Contrast?
Solve problems in new situations by applying acquired knowledge, facts, techniques and rules in
Keywords: apply. build, choose, construct, develop, interview, make use of, organize,
Questions:
How would you use? How would you solve ___ using what youve learned?
What examples can you find to? How would you show your understanding of?
What approach would you use to? What other way would you plan to?
What would result if? Can you make use of the facts to?
What elements would you use to change? What facts would you select to show?
Examine and break information into parts by identifying motives or causes. Make inferences and
Keywords: analyze, categorize, classify, compare, contrast, discover, dissect, divide, examine,
HIGHER LEVEL QUESTIONING 35
inspect, simplify, survey, test for, distinguish, list, distinction, theme, relationships, function,
Questions:
Can you list the parts . . . ? What inference can you make . . . ?
How would you categorize . . . ? Can you identify the different parts . . . ?
Keywords: build, choose, combine, compile, compose, construct, create, design, develop,
estimate, formulate, imagine, invent, make up, originate, plan, predict, propose, solve, solution,
suppose, discuss, modify, change, original, improve, adapt, minimize, maximize, theorize,
Questions:
What changes would you make to solve? How would you improve?
How could you change (modify) the plot (plan)? What facts can you compile?
What way would you design? What could be combined to improve (change)?
Suppose you could _____what would you do? How would you test?
Can you formulate a theory for? Can you predict the outcome if?
How would you estimate the results for? What could be done to minimize (maximize)?
Can you construct a model that would change? How is _____ related to?
Can you think for an original way for the? What are the parts or features of?
Can you list the parts? What inference can you make? ? What ideas justify?
How would you categorize? Can you identify the different parts?
Can you make the distinction between? What is the function of?
Present and defend opinions by making judgments about information, validity of ideas or quality
Keywords: award, choose, conclude, criticize, decide, defend, determine, dispute, evaluate,
judge, justify, measure, compare, mark, rate, recommend, rule on, select, agree, appraise,
prioritize, opinion, interpret, explain, support importance, criteria, prove, disprove, assess,
Questions:
How would you prove/ disprove? Can you assess the value or importance of?
HIGHER LEVEL QUESTIONING 37
How would you evaluate? How would you compare the ideas? the people?
How could you determine? What choice would you have made?
What would you select? How would you prioritize? How would you justify?
What judgment would you make about? Why was it better that?
How would you prioritize the facts? What would you cite to defend the actions?
Appendix B
Date of Lesson:
Standard:
Key Vocabulary:
Key Understandings:
- - -
Independent Activities:
Appendix C
Recording Tool
Phase (circle one): A B A (Baseline): teacher plans and B (treatment): teacher implements
implements lesson without support lesson after Blooms Taxonomy
Session #: training and lesson co-planning
session
Appendix D
Appendix E
Appendix F
Appendix G
INTEROBSERVER
Tot
al
Total # # Qu
Sessi Pha # of # of Questi Session of of esti
Date on # se LL HL ons % Notes Date # Phase LL HL ons % IOA
F F 0
10/28 1 B1 16 0 16 0% 10/28 1 B1 16 0 16 % 100%
M
10/31 2 B1 10 0 10 0%
T
11/1 3 B1 7 0 7 0%
W
11/2 4 B1 12 0 12 0%
Treatment
training
during Th
R planning
11/3 5 B1 8 0 8 0% block
F 10 ##
11/4 6 T1 9 1 10 % F 11/4 6 T1 8 1 9 # 91%
M/T school
holidays, W
R recording
11/10 7 T1 19 1 20 5% issue
F 20
11/11 8 T1 8 2 10 %
Tues-
M 17 Teacher
11/14 9 T1 5 1 6 % Absent
W 14
11/16 10 T1 12 2 14 %
R R 0
11/17 11 B2 13 0 13 0% 11/17 11 B2 13 0 13 % 100%
F
11/18 12 B2 6 0 6 0%
M
11/21 13 B2 12 0 12 0%
T
11/22 14 B2 16 0 16 0%
Treatment
training after
school at
1:20 before
W Tgiving
11/23 15 B2 14 0 14 0% holiday
M M 0
11/28 16 T2 8 0 8 0% 11/28 16 T2 8 0 8 % 100%
T 13 AVG:
11/29 17 T2 7 1 8 % 98%
W
11/30 18 T2 10 1 11 9%
R
12/1 19 T2 11 0 11 0%
F 13
12/2 20 T2 13 2 15 %
HIGHER LEVEL QUESTIONING 44
Appendix H
Fidelity of Treatment Checklist
DIRECTIONS: please put a check mark by each bulleted item as you observe it being
completed. If you do not observe the item taking place, do not check the bulleted point.
o The teacher was informed that they would be learning about different levels of
o All six levels of Blooms Taxonomy were read aloud to the teacher
o Sentence starters at each of the six levels were read to the teacher
o Teacher asked to generate an exemplar question for each of the six levels
o Researcher affirms or corrects the teachers exemplar question, pointing out why or why
o Blooms Taxonomy summary pages were given to teacher and it was explained that these
materials could be used as a reference or guide for their use at any time.