Experimental Designs

N TA L
R IM E
E X P E S
S IG N
DE
Presented by
MARLON CARLO T. LLAMERA
An experimental design is the traditional approach to
conducting quantitative research. This chapter defines
experimental research, identifies when you use it, assesses the key
characteristics of it, and advances the steps in conducting and
evaluating this design.
By the end of this chapter, you should be able to:

Define experimental research, and describe when to use it, and
how it developed.
Identify the key characteristics of experiments.
State the types of experimental designs.
Recognize potential ethical issues in experimental research.
Describe the steps in conducting an experiment.
Evaluate the quality of an experimental study.
What is an experiment?
• In an experiment, you test an idea (or practice or
procedure) to determine whether it influences an
outcome or dependent variable.
• You first decide on an idea with which to “experiment,”
assign individuals to experience it (and have some
individuals experience something different), and then
determine whether those who experienced the idea
(or practice or procedure) performed better on some
outcome than those who did not experience it.
When Do You Use an Experiment?
• You use an experiment when you want to establish
possible cause and effect between your independent
and dependent variables.
• This means that you attempt to control all variables that
influence the outcome except for the independent
variable. Then, when the independent variable influences
the dependent variable, we can say the independent
variable “caused” or “probably caused” the dependent
variable.
• Because experiments are controlled, they are the best of
the quantitative designs to use to establish probable cause
and effect.
When Did Experiments Develop?
• Experimental research began in the late 19th and early
20th centuries, with psychological experiments.
1903
 Schuyler used experimental and control groups, and his use
became so commonplace that he felt no need to provide a
rationale for them.
1916
 McCall advanced the idea of randomly assigning individuals to
groups ( Campbell & Stanley, 1963 )
1925
 Authoring a major book, “How to Conduct an Experiment,” McCall
firmly established the procedure of comparing groups.
1936
 Fisher’s book Statistical Methods for Research Workers discussed
statistical procedures useful in experiments in psychology and
agriculture. In this book, he advanced the concept of randomly
assigning individuals to groups before starting an experiment.
 Other developments in statistical procedures at this time (e.g., chi-
square goodness of fit and critical values) and the testing of the
significance of differences (e.g., Fisher’s 1935 The Design of
Experiments) enhanced experimental research in education.
1963
 Campbell and Stanley had identified the major types of experimental designs. They
specified 15 different types and evaluated each design in terms of potential threats
to validity.
1979
 Cook and Campbell elaborated on the types of designs, expanding
the discussion about validity threats.
2002
 Shadish, Cook, and Campbell had refi ned the discussions about
the major experimental designs. These books established the
basic designs, the notation, the visual representation, the
potential threats to designs, and the statistical procedures of
educational experiments.
 Since the 1980s, experiments have grown in sophistication and
complexity, largely because of computers and improved statistical
procedures. Researchers now employ multiple independent and
dependent variables, compare more than two groups, and study
different types of experimental units of analysis, such as entire
organizations, groups, and individuals (Boruch, 1998 ; Neuman,
2000).
 Procedural refinements represent the latest development in
experiments, and a number of “how to” books (e.g., Bausell,
1994 ) are available for the educational researcher.
 Also, books that link statistical procedures with experimental
design in terms of designing sensitive experiments (e.g., Lipsey,
1990 ) represent new ideas about strengthening procedures in
experimental studies.
Key Characteristics of Experiments
Manipulation
Control over
Random of the
extraneous
assignment treatment
variables
conditions
Outcome Group Threats to

measures comparisons validity
Random
Random Assignment assignment
The process of assigning individuals at random to groups or

to different groups in an experiment. The random
assignment of individuals to groups (or conditions within a
group) distinguishes a rigorous, “true” experiment from an
adequate, but less-than-rigorous, “quasi-experiment.”
It is used so that any bias in the personal characteristics of
individuals in the experiment is distributed equally among
the groups.
By randomization, you provide control for extraneous
characteristics of the participants that might influence the
outcome (e.g., student ability, attention span, motivation).
The experimental term for this process is “equating” the
groups.
Random
Equating the groups

means that the researcher randomly assigns individuals to
groups and equally distributes any variability of individuals
between or among the groups or conditions in the
experiment.
In practice, personal factors that participants bring to an
experiment can never be totally controlled—some bias or
error will always affect the outcome of a study. However,
by systematically distributing this potential error among
groups, the researcher theoretically distributes the bias
randomly.
Random
Random Selection
Quantitative researchers randomly select a sample from a
population. In this way, the sample is representative of the
population and you can generalize results obtained during
the study to the population.
Random
Experiments often do not include random selection of

participants for several reasons.
Participants often are individuals who are available to take
part in the experiment or who volunteer to participate.
Although random selection is important in experiments, it
may not be logistically possible.
However, the most sophisticated type of experiment
involves random assignment.
Control over
Control Over Extraneous Variables extraneous
variables
Extraneous factors
are any influences in the selection of participants, the
procedures, the statistics, or the design likely to affect the
outcome and provide an alternative explanation for our
results than what we expected.
Control over
variables
All experiments have some random error (where the
scores do not reflect the “true” scores of the population)
that you cannot control, but you can try to control
extraneous factors as much as possible.
Random assignment is a decision made by the investigator
before the experiment begins.
Other control procedures you can use both before and
during the experiment are pretests, covariates, matching
of participants, homogeneous samples, and blocking
variables.
Control over
variables
Pretests and Posttests
Pretest:
provides a measure on some attribute or characteristic
that you assess for participants in an experiment
before they receive a treatment.
Posttest
is a measure on some attribute or characteristic that is
assessed for participants in an experiment after a
treatment.
Control over
variables
Pretests and Posttests
Disadvantages of Pretest:
Take time and effort to administer
Can also raise the participants’ expectations about the

outcome
May influence the experimental treatment
May also affect posttest scores because participants

can anticipate the questions on the posttest based on
their experiences with the pretest.
Control over
variables
Covariates
are variables that the researcher controls for using
statistics and that relate to the dependent variable but
that do not relate to the independent variable. The
researcher needs to control for these variables, which
have the potential to co-vary with the dependent
variable. Often, these variables are scores on a pretest,
but they might be any variables correlated with the
dependent variable.
Control over
variables
Covariates
The statistical procedure of analysis of covariance
adjusts the scores on the dependent variable to
account for the covariance. This procedure becomes
another means for equating the groups and controlling
for potential influences that might affect the dependent
variable.
Control over
variables
Covariates
Control over
variables
Matching of Participants
Matching
is the process of identifying one or more personal
characteristics that influence the outcome and
assigning individuals with that characteristic equally to
the experimental and control groups.
Typically, experimental researchers match on one or
two of the following characteristics: gender, pretest
scores, or individual abilities.
Control over
variables
Matching of Participants
Control over
variables
Homogeneous Samples
Homogeneous samples
are people who vary little in their personal
characteristics.
 For example, we might assume that the students in the two
civics classes (one receives the lecture on “health hazards” and
the second does not) are similar in terms of characteristics that
students bring to the experiment, such as their academic grade
point average, gender, racial group (e.g., Caucasian, African
American), or prior abilities in civics. When the experimenter
assigns students to the two classes, the more similar they are in
personal haracteristics or attributes, the more these
characteristics or attributes are controlled in the experiment.
Control over
variables
Blocking Variables
Blocking Variable
is a variable the researcher controls before the
experiment starts by dividing (or “blocking”) the
participants into subgroups (or categories) and
analyzing the impact of each subgroup on the
outcome.
Control over
variables
Blocking Variables
The variable (e.g., gender) can be blocked into males
and females; similarly, high school grade level can be
blocked into four categories: freshmen,
sophomores, juniors, and seniors.
In this procedure, the researcher forms homogeneous
subgroups by choosing a characteristic common to all
participants in the study (e.g., gender or different age
categories). Then the investigator randomly assigns
individuals to the control and experimental groups
using each category of the variable.
Control over
variables
Blocking Variables
For example, if the students who participate in
the experiment are 15 and 16 years old, you
assign an equal number of 15- and 16-year-olds to
the control and experimental groups.
Manipulation
Manipulating Treatment Conditions of the
treatment
conditions
 In experimental treatment, the researcher

physically intervenes to alter the conditions
experienced by the experimental unit (e.g., a
reward for good spelling performance or a special
type of classroom instruction, such as small
group discussion).
Manipulation
treatment
conditions
Treatment Variables
 are independent variables that the researcher manipulates
to determine their effect on the outcome, or dependent
variable.
 are categorical variables measured using categorical
scales.
 For example, treatment independent variables used in
educational experiments might be:
Type of instruction (small group, large group)
Type of reading group (phonics readers, whole-language
readers)
Manipulation
treatment
conditions
Conditions
 In experiments, treatment variables need to have two or
more categories, or levels. In an experiment, levels are
categories of a treatment variable.
 For example, you might divide type of instruction into:
a. standard civics lecture;
b. standard civics lecture plus discussion about health hazards;
and
c. standard civics lecture plus discussion about health hazards
and slides of damaged lungs.
In this example, we have a three-level treatment variable.
Manipulation
treatment
conditions
Intervening in the Treatment Conditions

 The experimental researcher manipulates one or
more of the treatment variable conditions.
 In other words, in an experiment, the researcher
physically intervenes (or manipulates with
interventions) in one or more condition so that
individuals experience something different in the
experimental conditions than in the control
conditions.
Manipulation
treatment
conditions

 to conduct an experiment, you need to be able to
manipulate at least one condition of an independent
variable. It is easy to identify some situations in which
you might measure an independent variable and
obtain categorical data but not be able to manipulate
one of the conditions.
 In summary, experimental researchers manipulate or
intervene with one or more conditions of a treatment
variable.
Manipulation
treatment
conditions

Outcome
Outcome Measures measures
Outcome
 Outcome (or response, criterion, or posttest) is the
dependent variable that is the presumed effect of the
treatment variable. It is also the effect predicted in a
hypothesis in the cause-and-effect equation.
 In all experimental situations, you assess whether a
treatment condition influences an outcome or
dependent variable, such as a reduced rate of
smoking or achievement on tests.
Outcome
Outcome Measures measures
 Good outcome measures are sensitive to treatments

in that they respond to the smallest amount of
intervention.
 Outcome measures (as well as treatment variables)
also need to be valid so that experimental
researchers can draw valid inferences from them.
Group
Group Comparisons comparisons
 the process of a researcher obtaining scores for

individuals or groups on the dependent variable and
comparing the means and variance both within the
group and between the groups.
Group
Group
Threats to
Threats to Validity validity
 refer to specific reasons for why we can be wrong

when we make an inference in an experiment
because of covariance, causation constructs, or
whether the causal relationship holds over variations
in persons, setting, treatments, and outcomes
(Shadish, Cook, & Campbell, 2002).
Threats to
Four Types of Validity

(Shadish, Cook, & Campbell, 2002)
Statistical Conclusion Validity
 refers to the appropriate use of statistics (e.g.,
violating statistical assumptions, restricted
range on a variable, low power) to infer
whether the presumed independent and
dependent variables covary in the experiment.
Threats to

 Construct Validity
 means the validity of inferences about the
constructs (or variables) in the study.
 Internal Validity
 relates to the validity of inferences drawn about
the cause and effect relationship between the
independent and dependent variables .
Threats to

 External validity
 refers to the validity of the cause-and-effect
relationship being generalizable to other persons,
settings, treatment variables, and measures.
Threats to
Threats to Internal Validity
related to participants
related to treatments
related to the procedures

Threats to
History
Maturation
Regression
Selection
Mortality
Interactions with
Selection
Threats to

 Time passes between the beginning of
the experiment and the end, and events
may occur (e.g., additional discussions
about the hazards of smoking besides
History the treatment lecture) between the
pretest and posttest that influence the
Maturation outcome. In educational experiments, it
is impossible to have a tightly controlled
Regression environment and monitor all events.
However, the researcher can have the
Selection control and experimental groups
experience the same activities (except
Mortality for the treatment) during the experiment.
Interactions with
Selection
Threats to

 Individuals develop or change during the
related to participants experiment (i.e., become older, wiser,
stronger, and more experienced), and
History these changes may affect their scores
between the pretest and posttest. A
Maturation careful selection of participants who
mature or develop in a similar way (e.g.,
Regression individuals at the same grade level) for
oth the control and experimental groups
Selection helps guard against this problem.
Mortality
Interactions with
Selection
Threats to
Threats to Internal Validity  When researchers select individuals for

a group based on extreme scores, they
related to participants will naturally do better (or worse) on the
posttest than the pretest regardless of
the treatment. Scores from individuals,
History
over time, regress toward the mean. For
example, the selection of heavy
Maturation
smokers for an experiment will probably
contribute to lower rates of smoking
Regression
after treatment because the teens
selected started with high rates at the
Selection
beginning of the experiment. The
selection of individuals who do not have
Mortality extreme scores on entering
Interactions with characteristics (e.g., moderate smokers
Selection or average scores on pretests) may help
solve this problem.
Threats to
History
 “People factors” may introduce threats
Maturation that influence the outcome, such as
selecting individuals who are brighter,
Regression more receptive to a treatment, or more
familiar with a treatment (e.g., teen
Selection smokers ready to quit) for the
experimental group. Random selection
Mortality may partly address this threat.
Interactions with
Selection
Threats to
 When individuals drop out during the
History experiment for any number of reasons
(e.g., time, interest, money, friends,
Maturation parents who do not want them
participating in an experiment about
Regression smoking), drawing conclusions from
scores may be difficult. Researchers
Selection need to choose a large sample and
compare those who drop out with those
Mortality who remain in the experiment on the
Interactions with outcome measure.
Selection
Threats to
Threats to Internal Validity  Several of the threats mentioned thus far can interact
(or relate) with the selection of participants to add
related to participants additional threats to an experiment. Individuals
selected may mature at different rates (e.g., 16-year-
old boys and girls may mature at different rates
History during the study). Historical events may interact with
selection because individuals in different groups
Maturation come from different settings. For instance, vastly
different socioeconomic backgrounds of students in
Regression the teen smoking experiment may introduce
uncontrolled historical factors into the selection of
student participants. The selection of participants
Selection may also influence the instrument scores, especially
when different groups score at different mean
Mortality positions on a test whose intervals are not equal. If
Interactions with the scale for measuring number of cigarettes is
ambiguous (e.g., number of cigarettes per week or
Selection
per day?), groups are likely to interpret the scale
differently.
Threats to
Diffusion of treatments
Compensatory equalization
Compensatory rivalry
Resentful demoralization

Threats to
Threats to Internal Validity  When the experimental and control

groups can communicate with each other,
related to participants the control group may learn from the
experimental group information about the
treatment and create a threat to internal
related to treatments validity. The diffusion of treatments
(experimental and nonexperimental) for
Diffusion of treatments the control and experimental groups
needs to be different. As much as
Compensatory equalization possible, experimental researchers need
to keep the two groups separate in an
experiment (e.g., have two different civic
Compensatory rivalry classes participate in the experiment).
This may be difficult when, for example,
Resentful demoralization two civic classes of students in the same
grade in the same high school are
related to the procedures involved in an experiment about teen
smoking.
Threats to
Threats to Internal Validity  When only the experimental group

receives a treatment, an inequality
related to participants exists that may threaten the validity
of the study. The benefits (i.e., the
related to treatments goods or services believed to be
desirable) of the experimental
Diffusion of treatments treatment need to be equally
distributed among the groups in the
Compensatory equalization study. To counter this problem,
researchers use comparison groups
Compensatory rivalry (e.g., one group receives the health
hazards lecture, whereas the other
Resentful demoralization receives a handout about the
problems of teen smoking) so that all
related to the procedures groups receive some benefi ts during
an experiment.
Threats to
 If you publicly announce assignments
related to treatments to the control and experimental
groups, compensatory rivalry may
Diffusion of treatments develop between the groups because
the control group feels that it is the
Compensatory equalization “underdog.” Researchers can try to
avoid this threat by attempting to
Compensatory rivalry reduce the awareness and
expectations of the presumed
Resentful demoralization benefits of the experimental
treatment.
Threats to
Threats to Internal Validity  When a control group is used,

individuals in this group may become
related to participants resentful and demoralized because
they perceive that they receive a less
desirable treatment than other groups.
related to treatments One remedy to this threat is for
experimental researchers to provide a
Diffusion of treatments treatment to this group after the
experiment has concluded (e.g., after
Compensatory equalization the experiment, all classes receive the
lecture on the health hazards of
Compensatory rivalry smoking). Researchers may also
provide services equally attractive to
Resentful demoralization the experimental treatment but not
directed toward the same outcome as
related to the procedures the treatment (e.g., a class discussion
about the hazards of teen driving with
friends).
Threats to

Testing
Instrumentation
Threats to
Threats to Internal Validity  A potential threat to internal validity is

that participants may become familiar
related to participants with the outcome measures and
remember responses for later testing.
related to treatments During some experiments, the
outcome is measured more than one
time, such as in pretests (e.g.,
related to the procedures repeated measures of number of
cigarettes smoked). To remedy this
Testing situation, experimental researchers
measure the outcome less frequently
Instrumentation and use different items on the
posttest than those used during
earlier testing
Threats to
Threats to Internal Validity  Between the administration of a pretest

and a posttest, the instrument may
related to participants change, introducing a potential threat to
the internal validity of he experiment.
For example, observers may become
related to treatments more experienced during the time
between a pretest and the posttest and
related to the procedures change their scoring procedures (e.g.,
observers change the location to
Testing observe teen smoking). Less frequently,
the measuring instrument may change
Instrumentation so that the scales used on a pretest and
a posttest are dissimilar. To correct for
this potential problem, you standardize
procedures so that you use the same
observational scales or instrument
throughout the experiment.
Threats to
Threats to External Validity
Interaction of selection and treatment
Interaction of setting and treatment
Interaction of history and treatment:

Threats to
• This threat to external validity involves the inability to

generalize beyond the groups in the experiment, such
as other racial, social, geographical, age, gender, or
personality groups. One strategy researchers use to
increase generalizability is to make participation in the
experiment as convenient as possible for all individuals
in a population.

Threats to

• This threat to external validity arises from the inability to generalize from the setting
where the experiment occurred to another setting. For example, private high schools
may be different from public high schools, and the results from our civics experiment
on smoking may not apply outside the public high school where the researcher
conducts the experiment. This threat may also result from trying to generalize results
from one level in an organization to another. For example, you cannot generalize
treatment effects you obtain from studying entire school districts to specific high
schools. The practical solution to an interaction of setting and treatment is for the
researcher to analyze the effect of a treatment for each type of setting.

Threats to

• This threat to external validity develops when the researcher tries to generalize
findings to past and future situations. Experiments may take place at a special
time (e.g., at the beginning of the school year) and may not produce similar
results if conducted earlier (e.g., students attending school in the summer may
be different from students attending school during the regular year) or later (e.g.,
during semester break). One solution is to replicate the study at a later time
rather than trying to generalize results to other times.
Types of Experimental Designs
Differentiating
Experimental Designs
Threats to Internal
Validity in Types of
Experimental Designs
Between Groups Within Groups or
Designs Individual Designs
Time series
True experiments
experiments
Repeated
Quasi-
measures
experiments
experiments
Single subject
Factorial designs
experiments
True experiments
In true experiments, the researcher randomly assigns

participants to different conditions of the experimental
variable.
Individuals in the experimental group receive the
experimental treatment, whereas those in the control
group do not.
After investigators administer the treatment, they compile
average (or mean) scores on a posttest.
One variation on this design is to obtain pretest as well as
posttest measures or observations. When experimenters
collect pretest scores, they may compare net scores (the
differences between the pre- and posttests).
True experiments
Alternatively, investigators may relate the pretest

scores for the control and experimental groups to see
if they are statistically similar, and then compare the
two posttest group scores. In many experiments, the
pretest is a covariate and is statistically controlled by
the researcher.
Because you randomly assign individuals to the
groups, most of the threats to internal validity do not
arise. Randomization or equating of the groups
minimizes the possibility of history, maturation,
selection, and the interactions between selection and
other threats.
True experiments
When true experiments include only a posttest, it

reduces the threats of testing, instrumentation, and
regression because you do not use a pretest.
If a pretest is used, it introduces all of these factors
as possible threats to validity.
Instrumentation exists as a potential threat in most
experiments, but if researchers use the same or
similarinstrument for the pre- and posttest or enact
standard procedures during the study, you hold
instrumentation threats to a minimum.
Quasi-experiments
Quasi-experiments include assignment, but not
random assignment of participants to groups. This is
because the experimenter cannot artificially create
groups for the experiment.
The researcher assigns intact groups the experimental
and control treatments, administers a pretest to both
groups, conducts experimental treatment activities
with the experimental group only, and then
administers a posttest to assess the differences
between the two groups.
A variation on this approach, similar to the true
experiment, uses only a posttest in the design.
Quasi-experiments
The quasi-experimental approach introduces
considerably more threats to internal validity than the
true experiment.
Because the investigator does not randomly assign
participants to groups, the potential threats of
maturation, selection, mortality, and the interaction of
selection with other threats are possibilities.
Individuals assigned to the two groups may have
selection factors that go uncontrolled in the
experiment.
Quasi-experiments
Because we compare two groups, the treatment
threats may also be present.
When the pretest–posttest design is used, additional
threats of history, testing, instrumentation, and
regression also may occur.
While the quasi-experimental design has the
advantage of utilizing existing groups in educational
settings, it introduces many threats that you need to
address in the design of the experiment.
Factorial designs
Factorial designs represent a modification of the
between group design in which the researcher studies
two or more categorical, independent variables, each
examined at two or more levels (Vogt, 2005).
The purpose of this design is to study the
independent and simultaneous effects of two or more
independent treatment variables on an outcome.
This design has the advantage of a high level of
control in the experiment. It allows the investigator to
examine the combination or interaction of
independent variables to better understand the
results of the experiment.
Factorial designs
If only a posttest is used, internal validity threats of
testing and instrumentation do not exist.
If you randomly assign individuals to groups, you
minimize the threats related to participants and their
experiences (history, maturation, regression,
selection, mortality, and interaction of selection and
other factors).
However, with multiple independent variables in a
factorial design, the statistical procedures become
more complex and the actual results become more
difficult to understand.
Factorial designs
As researchers manipulate additional independent
variables, more participants are needed in each group
for statistical tests, and the interpretation of results
becomes more complex. Because of this complexity,
factorial designs typically include at most three
independent variables manipulated by the researcher.
Factorial designs
Factorial designs
• Main effects are the influence
of each independent variable
(e.g., type of instruction or
extent of depression) on the
outcome (e.g., the dependent
variable, rate of smoking) in an
experiment.
• Interaction effects exist when
the influence on one
independent variable depends
on (or co-varies with) the other
independent variable in an
experiment.
Factorial designs
Time series experiments
A time series design consists of studying one group,
over time, with multiple pretest and posttest
measures or observations made by the researcher.
This design does not require access to large numbers
of participants, and it requires only one group for the
study.
It is ideal for examining change in an entire system
(e.g., a school district) where it would be difficult to
find a control group or system willing to cooperate.
However, this design is labor intensive because the
researcher needs to gather multiple measures.
Interrupted Time Series
Equivalent Time Series
TABLE 10.5
This procedure consists of studying
one group, obtaining multiple pretest
Interrupted Time Series measures for a period of time,
administering an intervention (or
interrupting the activities), and then
Equivalent Time Series measuring outcomes (or posttests)
several times. Data analysis in this
example consists of examining
difference scores between the
pretests and posttests or posttest-only
scores and using the pretests as
TABLE 10.5 covariates.
Interrupted Time Series

In this design, the investigator
alternates a treatment with a posttest
Equivalent Time Series measure. The data analysis then
consists of comparing posttest
measures or plotting them to discern
patterns in the data over time.
TABLE 10.5
 The time series design permits significant control over
threats to internal validity. The effects of history are not
always clear-cut. History effects are minimized by the short
time intervals between measures and observations.
 However, threats to validity may occur because of the
overall length of data collection in this design.
 The maturation of participants may be a problem, although
the researcher can estimate changes in maturation by
studying them and removing them statistically in the
design.
 To control for statistical regression, researchers can also
observe the scores on the pretests and control for
unusually high or low scores.
Because only one group is studied, the issues of
selection and treatment are not relevant, although
individuals can choose to drop out of the study.
Testing may be a problem, but the repeated
measures or observations over time may diminish the
effects of testing.
When researchers change the instrument during
multiple testing administrations, they may also
introduce threats to validity.
Repeated measures experiments
In a repeated measures design, all participants in a

single group participate in all experimental treatments,
with each group becoming its own control.
The researcher compares a group’s performance under
one experimental treatment with its performance under
another experimental treatment.
The experimenter decides on multiple treatments (as in
factorial designs) but administers each separately to
only one group.
After each administration, the researcher obtains a
measure or observation.
 After selecting participants, the researcher decides on different experimental

treatments to determine the effect of each on one or more outcomes.
 An outcome measure or observation follows the first experimental treatment,
and then a second outcome measure or observation is taken following the
second experimental treatment.
 Variations in outcome measures are then assessed for differences from
treatment to treatment.
 In terms of threats to internal validity, this design is not

affected by threats related to comparing groups (i.e.,
selection, treatments, regression, mortality, maturation,
or interactions with selection).
 Without use of a pretest, testing and instrumentation
are not threats in this design.
 History is a potential problem in that events may occur
during the experiment that raise the potential for
extraneous influences to affect the outcome measure.
 One experimental treatment may influence the next
treatment, and researchers need to make the
treatments as distinct as possible.
Single subject experiments
Single-subject research (also called N of 1 research,

behavior analysis, or within-subjects research)
involves the study of single individuals, their
observation over a baseline period, and the
administration of an intervention. This is followed by
another observation after the intervention to
determine if the treatment affects the outcome.
For example, in one single-subject study, the
researcher tested whether elementary students with
learning disabilities achieve better if they monitor
their own on-task behavior (Kellogg, 1997).
Without random assignment, this design is a quasi-

experimental rather than an experimental design.
The researcher studies the behaviors of single
individuals (one or more) rather than a group of
subjects, with the subject becoming its own control in
the experiment (see Cooper, Heron, & Heward, 1987;
Neuman & McCormick, 1995).
Key Characteristics
 Prior to administering the intervention, the
researcher establishes a stable baseline of
information about the individual’s behavior. A stable
baseline means that behavior for an individual varies
little over several sessions or days. A behavior is
stable if (a) variability over time is minimal, and (b)
there is no upward or downward trend in
performance over time (Poling & Grossett, 1986).
Key Characteristics
 The researcher repeatedly and frequently measures
behavior (i.e., the outcome) throughout the
experiment based on making observations and
recording scores for each individual.
Key Characteristics
 After administering the intervention, the researcher
notes the patterns of behavior and plots them on a
graph. This pattern may be ascending, descending,
flat, or variable. Data are typically analyzed by
visually inspecting the data rather than by using
statistical analysis. In particular, the researcher notes
how the behavior of the individual has changed after
the intervention, after withdrawing the intervention,
or during multiple interventions.
Key Characteristics
 In a graphic analysis of the data, the single-subject
researcher plots behaviors for specific individuals on
a graph. On this graph, the vertical axis records
percentages or counts of the behavior being studied.
Alternatively, the horizontal axis displays the days or
sessions in which the observations occur. The plot
can show data for several individuals or multiple
dependent variables for a single individual.
 Single-subject research has the advantage of providing data on

single individuals, such as the learning and behaviors of
children with disabilities, where a person-by-person analysis is
needed.
 It also controls for many threats to internal validity. Because
only one individual is studied at a time, groups are not involved
and the threats to selection, treatments, mortality, maturation,
regression, and interactions with selection are not relevant.
 Assuming that observers use the same standard procedures,
instrumentation may not be a problem.
 When multiple treatments are used, the learning from one
intervention may affect the second intervention, and history
may be an issue since the experiment takes place over time
A/B Design
Multiple Baseline Design
Alternating Treatments
Design
• An A/B design consists of observing and
Single subject experiments measuring behavior during a trial period
(A), administering an intervention, and
observing and measuring the behavior
after the intervention (B).
• A variation on this design is an A/B/A, or a
A/B Design reversal design, in which the researcher
establishes a baseline behavior,
administers an intervention, and then
Multiple Baseline Design withdraws the intervention and
determines if the behavior returned to the
Alternating Treatments baseline level.
• Another variation is an A/B/A withdrawal
Design design. In this design, researchers may
implement one or more treatments. The
disadvantage of this type of design is that
in some studies, the withdrawing of the
intervention may have serious effects on
the participants in the study, raising an
ethical issue for the researcher
Single subject experiments • In this design, each participant receives an
experimental treatment at a different time
(hence, multiple baselines exist) so that
treatment diffusion will not occur among
participants.
A/B Design • Researchers choose this design when the
treatment (e.g., skill or strategy being
taught) cannot be reversed and doing so
Multiple Baseline Design would be unethical or injurious to
participants.
• Variations on this approach could involve
Alternating Treatments different types of behaviors for the
Design participants or behaviors for participants
in different settings.
• The results of this design may be less
convincing than the reversal design and it
may introduce negative consequences if
the treatment is withheld for an extended
period.
• An alternating treatment design is a

single-subject design in which the
A/B Design researcher examines the relative effects of
two or more interventions and determines
which intervention is the more effective
Multiple Baseline Design treatment on the outcome.
• In this type of design, potential problems
Alternating Treatments with threats to internal validity from
treatment diffusion may result, but the
Design design permits a test of multiple
treatments simultaneously to determine
their effect on outcomes.
Potential Ethical Issues
 Withholding treatments from individuals in
control groups when these individuals might be
disadvantaged by not receiving the beneficial
treatment.
 Randomized assignment may not be ethical.
 An experiment needs to be concluded and it
continues on.
 Experimental methods may give the best answer
to a problem and the need to consider how high
the stakes may be in conducting the experiments.
STEPS in Conducting ER
1. Decide if an Experiment Addresses Your Research
Problem
2. Form Hypotheses to Test Cause-and-Effect Relationships
3. Select an Experimental Unit and Identify Study
Participants
4. Select an Experimental Treatment and Introduce It
5. Choose a Type of Experimental Design
6. Conduct the Experiment
7. Develop an Experimental Research Report
How to Evaluate ER?
The following list, adapted from Bausell (1994) , presents criteria
useful in this evaluation. For a good experiment, here are some
criteria:
• The experiment has a powerful intervention.
• The treatment groups are few in number.
• Participants will gain from the intervention.
• The researcher derives the number of participants per group in
some systematic way.
• An adequate number of participants were used in the study.
• The researcher uses measures and observations that are valid,
reliable, and sensitive.
• The researcher controls for extraneous factors that might
influence the outcome.
• The researcher addresses threats to internal and external validity.
o u f o r
ha n k y
T i ng !
l i st e n
Reporter:
MARLON CARLO T. LLAMERA

Experimental Designs

Uploaded by

Copyright:

Available Formats

Experimental Designs

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Experimental Designs

Uploaded by

Copyright:

Available Formats

N TA L

By the end of this chapter, you should be able to:

Outcome Group Threats to

The process of assigning individuals at random to groups or

Equating the groups

Experiments often do not include random selection of

Can also raise the participants’ expectations about the

May also affect posttest scores because participants

 In experimental treatment, the researcher

Intervening in the Treatment Conditions

Intervening in the Treatment Conditions

Intervening in the Treatment Conditions

 Good outcome measures are sensitive to treatments

 the process of a researcher obtaining scores for

 refer to specific reasons for why we can be wrong

Four Types of Validity

Four Types of Validity

Four Types of Validity

Threats to Internal Validity

related to the procedures

Threats to Internal Validity

Threats to Internal Validity

Threats to Internal Validity

Threats to Internal Validity  When researchers select individuals for

Threats to Internal Validity

Threats to Internal Validity

Threats to Internal Validity

related to the procedures

Threats to Internal Validity  When the experimental and control

Threats to Internal Validity  When only the experimental group

Threats to Internal Validity

Threats to Internal Validity  When a control group is used,

Threats to Internal Validity

related to the procedures

Threats to Internal Validity  A potential threat to internal validity is

Threats to Internal Validity  Between the administration of a pretest

Threats to External Validity

Interaction of selection and treatment

Interaction of setting and treatment

Interaction of history and treatment:

Threats to External Validity

Interaction of selection and treatment

• This threat to external validity involves the inability to

Interaction of setting and treatment

Interaction of history and treatment:

Threats to External Validity

Interaction of selection and treatment

Interaction of setting and treatment

Interaction of history and treatment:

Threats to External Validity

Interaction of selection and treatment

Interaction of setting and treatment

Interaction of history and treatment:

In true experiments, the researcher randomly assigns

Alternatively, investigators may relate the pretest

When true experiments include only a posttest, it

Interrupted Time Series

Equivalent Time Series

Interrupted Time Series

In a repeated measures design, all participants in a