Principles of Experimental Design: 

 
1. Control: The minimizing of outside variable effects on the independent variable. This makes the 
experiment more reliable. 
2. Randomize: Reduces bias within the experimental design because it equalizes the effect of 
non-explicit factors.  
3. Replication: Repeating the experiment on a large scale to see if the outcome is still within the 
margin of error.  
4. Block: The arranging of experimental variables to eliminate unwanted sources of variability. 
 
*The information on this page is a summary of the page found in the link. It is for reference purposes* 
Problem: A statement that defines the topic of an experiment and identifies the relationship between two 
variables. The problem statement generalizes the factors being tested. 
Hypothesis: A statement that predicts the outcome of testing the relationship between the independent 
and dependent variable as specified in the problem statement. You must include your rationale.  
Key Terms:  
Control Group - A group of subjects assigned to an experiment, but without being subjected to a certain 
treatment Treatment Group - Often called an experiment group, they receive a specified treatment. 
Variable - Anything liable to change Extraneous Variable - NOT INDEPENDENT VARIABLES that 
could still change the outcome of the experiment Randomness - completely chance based events which 
are unpredictable. Randomness is preferred when assigned individuals to groups. Random Assignment - 
The preferred method of assigning subjects to remove the tendency of an outcome of any one group. 
Computers are typically used to create random tables of groups. Ex Post Facto - “After the Fact”. The 
event of interest has already occurred and the data studied is the outcome of the experiment. Variance - 
the change in the outcome or path of any event. No event is perfectly recreated, but can have very similar 
outcomes. Internal Validity - The logic of the experiment. Are the assumptions made reasonable? 
External Validity - The validity of an experiment in relation to groups outside of the experiment. For 
example, will this covid vaccine work for a majority of the population or groups with different heredity? 
Blocks - The arranging of experimental units to reveal possible interactions between certain variables of 
the treatment. Interaction - variables which may interact with each other. Hawthorne Effect - the 
noticeable shift in behavior as a result of an individual being aware of an observer. Eg. A test subject is 
less likely to steal a pen when there's a camera watching them. Blind Experiment - An experiment where 
the subject is unaware of whether or not they will be receiving a treatment or placebo. Double Blind 
Experiment - an experiment in which the administrator nor the subject knows if treatment/placebo was 
used in the experiment. RESPONSE VARIABLE (DV)- More commonly known as the dependent 
variable, it is the result of changes in the explanatory variable. EXPLANATORY VARIABLE - Also 
known as the independent variable, it is the variable that the administrator changes to evaluate the effect 
of the treatment. REPLICATION - The ability to create consistent outcomes which reduces the 
variability in an experiment. Constants : All other factors that are changed in the course of the 
experiment. They are potential independent variables for future experiments. Hypothesis - A statement 
that predicts the outcome of testing the relationship between the independent variable and the dependent 
variable as specified in the problem. Always include the rationale for this prediction. Confounding 
Variable : An Error variable which has effects that are consistent with independent variables and have 
the same impact on the dependent variable. Basically, once you confound one variable to another you 
can’t determine their individual effects on the dependent variable. Say you have a block that has two 
treatments and then an observation, you can’t determine what either treatment had on the experiment, 
only what they did together. Null hypothesis - A statement that says the independent variable has no 
effect on the dependent variable and that the results found in the group had been a chance occurrence. 
ERROR - A deviation from our model. The outside world is always right, it is our model that is wrong. 
OUTLIERS - An outlier is an observation outside the overall pattern of a distribution. Usually, the 
presence of an outlier indicates some sort of problem. This can be a case which does not fit the model 
under study, or an error in measurement. They can be easily spotted when plotted on a scatter diagram.  
Scatter Diagram: A tool used for analyzing the relationships between two variables based on how 
closely two variables are related.  
SIX MAJOR CLASSES OF INFORMATION THAT EXPERIMENT DESIGNERS MUST COPE 
WITH:  
1. Post-treatment behavior of physical measurement [P1] 
2. Pre-treatment behavior or physical measurement [P2] 
3. Internal threats to validity [I] 
4. Comparable groups [C] 
5. Experiment errors [E] 
6. Relationship to treatment [R] 
 
[P1] Post-Treatment Behavior or Physical Measurement 

This is the information of primary interest. What was the physical or behavior of the subject after 
treatment? All experimental designs used this principle to some effect. There are five categories of 
post-treatment behavior that can be identified:  
1. P1-1: behavior or measurement immediately after treatment 
2. P1-2: a comparison of post-treatment behavior between experimental and control groups 
3. P1-3: a comparison of the post-treatment behavior between experimental groups or blocks 
4. P1-4: long-term effects with continuing treatment and periodic observations 
5. P1-5: long-term effects without continuing treatment but with observation(s) 

[P2] Pre-Treatment Behavior or Physical Measurement 

Experiments require observations of the patient/subject prior to experimental manipulation. A 

lack of pre-treatment observation will not answer any questions about the result of the treatment because 
there will be no observations to compare with.  
 
[I] Internal Threats to Validity 
 
  The existence of rivaling hypotheses can threaten the validity of an experiment. A rival 
hypothesis is when the two different hypotheses both explain the outcome of the experiment. Rival 
hypotheses can be typically resolved by adding more control groups to the study.  
 
1. I-1: the subjects exhibited behavior because of some event other than the treatment 
○ some other drug or process caused the change 
○ the subject changed naturally [just improved] 
○ the subject had a massive change in attitude or emotion 
 ○ some other physical change occurred 
2. I-2: the subject could or would perform the behavior, or would have exhibited the measurement 
without the treatment 

[C] Comparable Groups  

A class of information that only exists if there are more than one groups to compare. It classifies 
whether or not certain relevant attributes are present in a group prior to treatment. The experimenter must 
admit if certain relevant attributes are not present in a certain group. There are two types of compatibility 
information:  
1. C-1: were the groups (either experimental or control) comparable before the treatment? 
2. C-2: did the groups receive a comparable degree of experiences during the time of the study 
(except for differences in treatment?) 

[E] - Experiment Errors 

This refers to any unwanted side effects caused by something other than the treatment itself. The 
Hawthorne effect is one example of an experiment error. There are ways to medicate the Hawthorne 
effect: 

1. E-1: provide for a placebo treatment group which gets the attention, but not the "real" treatment 
and use blind and double blind strategies as needed 
2. E-2: continue the treatment over a longer period of time; research shows that the Hawthorne 
effect tends to be short-lived. 

[R] - Relationship to Treatment  

This class of information relates to the effects that combined treatments may have on a subject. 
There are many complicated factors that impact the performance of a subject. This includes: Intensity of 
the treatment, repeated applications of the treatment, different sequences of application, and different 
kinds of subjects all impact the performance of a treatment(s).  

1. R-1: did the treatment interact with subject characteristics so that subjects with different 
characteristics behaved or reacted differently? 
2. R-2: how does the treatment interact when combined with other sorts of treatment? 
3. R-3: does the treatment contain different factors which may operate differentially on the subjects? 
4. R-4: what is the effect of different levels or degrees of the treatment? 
5. R-5: what is the effect of different orders or sequences of various treatments? 

Describing Experimental Designs: 

The following letters will be used to describe the various experimental design activities: 

Activity  Letter(s) 
 Selection of the group or  GP 
experimental unit 

Random assignment to a group  R 

Blocking subjects, or other  BLK 

variables, into sets 

Administering a treatment to a  T 
group 

Observing (measuring) results  O 

Basic Experiment Design: There are eleven typical experimental designs. + 

ONE SHOT EXPERIMENT: GP--T--O 

A One-Shot Experiment is an experiment where a unit is administered a treatment and then measured 
(Observed). The subjects are not randomly assigned in this experimental design and are only administered 
one treatment. 

ONE GROUP, Pre-Post: GP--O--T--O 

This design involves using pre-treatment observations and measurements prior to treatment. This design 
is very similar to one-shot, except for the inclusion of an extra data-type. 

STATIC GROUP: GP--T--O  

GP------O 

This design involves the use of two separate groups in which only one is administered the treatment. This 
design can provide very useful information on rival hypotheses. Ensuring the comparability of these 
groups is essential to ruling out rival hypotheses.  

RANDOM GROUP: R--GP--T--O 

  R--GP------O 

This design is very similar to the static group but includes a randomized selection of subjects. It 
works to ensure that the two groups are comparable and reduces similar subjects in a single group 
(You don’t want half of your group to have had a history of allergic reactions).  

PRE-POST RANDOMIZED GROUP: R--GP--O--T--O 

 R--GP--O------O 

This experimental design adds observations prior to treatment to ensure that the groups have proper 
comparability.  

SOLOMON FOUR GROUP: R--GP--O--T--O 

R--GP--O------O 

R--GP-------T--O 

R--GP-----------O 

This experimental design is used for mitigating “Sensitizing” effects of a treatment. They are 
typically used for studies involving education and in medical studies involving the physical 
activities of a patient. 

RANDOMIZED BLOCK: ----------BLK--T--O 

R--GP 

----------BLK--T--O 

----------BLK------O 

  R--GP 

----------BLK------O 

This experimental design uses selected groups which have similar characteristics to construct 
groups. Say you want to do a drug test and think that gender has an impact on the effects of the 
treatment. You would first group the subjects into male and female blocks which are separated and 
then randomized prior to treatment. The design handles questions relating to comparability, 
pre-post observations, and internal validity.  

Factorial: R--GP--T-------O 

A1 B1 

  R--GP--T-------O 

A1 B2 

R--GP--T-------O 

A2 B1 

  R--GP--T-------O 

A2 B2 
This experimental design is similar to the randomized black design but groups subjects after they 
are given a treatment. It is useful for grouping effects of combining different treatments and 
gathering observations from certain groups.  

One-Shot Repeated Measures: GP--T--O--T--O--T--O 

This experimental design is used to test the effects of a treatment over a period of time. 
Observations are made repeatedly to test the effects of the treatment. It is a highly effective method 
as a trial phase and combined with data from other experiments.  

Randomized Groups Repeated Measures: R--GP--T---O--T----O--T----O 

R--GP--T2--O--T2--O--T2--O 

This is very similar to One-Shot repeated measures design except for the inclusion of randomized 
groups as well as the inclusion of another experiment method. This design fails to handle questions 
on internal validity or P-1-2, P1 -3, P1- 4 questions.  

Latin Square: R--GP--TA--O--TB--O--TC--O 

R--GP--TB--O--TC--O--TA--O 

R--GP--TC--O--TA--O--TB--O 

This experimental design is used to test all combinations of a set of different treatments. This 
design can grow in cost very fast so researchers have to consider the permutations of groupings to 
ensure the lowest possible cost. The Latin Square design answers questions P1-1, P1-3, C-1, C-2, 
and R-5. It will not provide information on P1-2, P1-4, P1-5 nor any questions in class I or class E. 
Questions R-1 through R-4 are not addressed as well.  

The Question of External Validity: 

Do the observations made in the experiment map the “real word’? External validity is the extent to 
which you can generalize the findings of a study to other situations, people, settings and measures. 
The classes of questions about external validity include:  

● did some of the early procedures in the research affect the subjects so that their later 
measurements were, in part a result of that? 
● Were the subjects themselves a representative sample of the general population of people to 
which it is desired to extend the research findings? 
● Was there something in the research or setting that would cause or influence the measurement of 
the variable of interest? 
● Was the treatment accompanied by any personal interaction that may be somewhat peculiar to the 
research or to the subjects or the experimenter involved? 

Statistical Analysis of Data: 

Measure of central tendency: The typical value of a probability distribution, it is often called 
the average. It summarizes the entire distribution in a single variable.  
 Measure of Variation: This will tell you far the data set is spread. It is useful for finding the 
standard deviation. It is the average of the squared differences from the Mean.  

How to Calculate:  

1.First find the mean (Average) 

2.Create a new list where each element is: x - avg, where x is an element in the data set. 

3. Then take each element and square it. 

4. Find the average of the new squared data set. 

5. Take the square root of the average. That is your standard deviation.  

The equation in a visual form is:     

Variance 

σ2  =  2062 + 762 + (−224)2 + 362 + (−94)25 

   =  42436 + 5776 + 50176 + 1296 + 88365 

   =  1085205 

   =  21704 
And the Standard Deviation is just the square root of Variance, so:  
 
Standard Deviation 

σ  =  √21704 

   =  147.32... 

   =  147 (to the nearest unit) 

 

Measurements: A measurement must include a magnitude and a unit. Measurements must be 
Must be made with the correct tools for the units selected. I.E. You can’t use an oscilloscope to 
measure the height of a building. 
Data Tables: Data tables are used to organize the data you collect. Be sure that all raw data is 
given and that correct significant figures are included. Include a condensed table with the most 
important data.   
Graphs: When graphing, remember that the independent variable goes on the horizontal or X - 
axis while the dependent variable goes on the vertical or Y-axis. To scale the axis of a graph so it will 
always fit the grid, use the following formula: High value - low value (use zero if you plan to start 
numbering by zero) divided by the number of spaces on that axis. ALWAYS ROUND UP.  
Coefficient of Variation (CV): 
  

σ – the standard deviation μ – the mean 

General Linear Model: Data = model + error → known as Y = MX + B.  

It is the foundation for regression analysis. It is used for comparing how several variables affect different 
continuous variables. IN its simplest form it is described as: Data = Model + Error.  

Linear Regression - A method for predicting the relationship between the XY variables by using linear 
lines (lines of best fit). A linear regression line with a slope of zero implies a null distribution. 

How to draw the line of best fit: 

1. Begin by plotting all your data. ... 

2. Draw a shape that encloses all of the data, (try to make it smooth and relatively even). ... 
3. Draw a line that divides the area that encloses the data in two even sized areas.  
4. Congratulations. 

F-TEST : Helps us quantify how well we think our data fit a distribution, like the null distribution. 

F = (Observed Model - Model if the Null is True) / Average Variation.  

STANDARD TEST STATISTIC: (statistic - parameter) / (standard deviation of the statistic).

X is the sample mean   μ – the mean of population 

σ – the standard deviation sqrt(n) - sample size 

This is a z-test which is an example of a statistical test to determine whether two population means are 
different when the variances are known and the sample size is large. 

Degrees of Freedom: The maximum number of logically independent values, which are values that 
agave the freedom to vary, in the data sample. 

Regression Analysis: A set of statistical processes for estimating the relationship between a dependent 
variable and one or more independent variables. You can calculate the regression in a scatterplot by 
finding the variance, and running the variance as the average variation in the F-Test Statistic.  

 
Chi - Square Statistic:  

x^2 = chi squared Oi - Observed value 

Ei = expected value.  

This is a test to check when the test statistic is chi - squared under the null hypothesis. It is simply another 
way of comparing the model to real data when the data is random, raw, mutually exclusive, drawn from 
independent variables, and drawn from a large enough sample.  

Mutually Exclusive Events - These are events that cannot occur at the same time. A coin can be head or 
tales, but never both.  

Calculating Probability:  

Calculating Bayes Theorem :  

 
 
 
 
ERROR ANALYSIS:  
Percent Error Formula:  

 
This is used to determine the offset of the hypothesized results and the real results. Always ask if the 
errors affect the results as this can lead to problems of internal validity later on.  
Systematic Error: These errors are caused by shifts in all measurements as a result of things such as 
incorrect calibration of equipment, consistently improper use of equipment, or failure to properly account 
for a certain effect. Large blunders can be avoided, but small systematic errors will always be present.  
Random Error - These are errors that fluctuate from one measurement to another. They yield results 
distributed among some mean value.  
● They may occur due to lack of sensitivity. For a sufficiently small change an instrument may not 
be able to respond to it or to indicate it or the observer may not be able to discern it. 
 
● They may occur due to noise. There may be extraneous disturbances which cannot be taken into 
account. 
 
● They may be due to imprecise definitions. 
 
● They may also occur due to statistical processes such as the roll of dice. 

Maximum Error : The maximum and minimum values of the data set can be specified as :  

σ – the standard deviation sqrt(n) - sample size 

Z - standard test statistic.  
Propagation of Uncertainty - The effect of variables’ uncertainty (errors) on the uncertainty of a 
function based on them. Basically, the uncertainty of variables will add up to the uncertainty of a function.