Lesson 15: TESTS FOR SIGNIFICANCE

What Are Tests for Significance

  Two questions arise about any hypothesized relationship between two

variables:
1) what is the probability that the relationship exists;
2) if it does, how strong is the relationship
  There are two types of tools that are used to address these questions: the
first is addressed by tests for statistical significance; and the second is
addressed by Measures of Association.

    Tests for statistical significance are used to address the question: what is
the probability that what we think is a relationship between two variables is
really just a chance occurrence?

    If we selected many samples from the same population, would we still
find the same relationship between these two variables in every sample? If we
could do a census of the population, would we also find that this relationship
exists in the population from which the sample was drawn? Or is our finding
due only to random chance?

    Tests for statistical significance tell us what the probability is that the
relationship we think we have found is due only to random chance. They tell us
what the probability is that we would be making an error if we assume that we
have found that a relationship exists.

    We can never be completely 100% certain that a relationship exists

between two variables. There are too many sources of error to be controlled, for
example, sampling error, researcher bias, problems with reliability and validity,
simple mistakes, etc.

    But using probability theory and the normal curve, we can estimate the
probability of being wrong, if we assume that our finding a relationship is true.
If the probability of being wrong is small, then we say that our observation of
the relationship is a statistically significant finding.

    Statistical significance means that there is a good chance that we are
right in finding that a relationship exists between two variables. But statistical
significance is not the same as practical significance. We can have a
statistically significant finding, but the implications of that finding may have no
practical application. The researcher must always examine both the statistical
and the practical significance of any research finding.

    For example, we may find that there is a statistically significant

relationship between a citizen's age and satisfaction with city recreation
services. It may be that older citizens are 5% less satisfied than younger
citizens with city recreation services. But is 5% a large enough difference to be
concerned about?

    Often times, when differences are small but statistically significant, it is
due to a very large sample size; in a sample of a smaller size, the differences
would not be enough to be statistically significant.
 

Steps in Testing for Statistical Significance

1) State the Research Hypothesis

2) State the Null Hypothesis
3) Select a probability of error level (alpha level)
4) Select and compute the test for statistical significance
5) Interpret the results
 

1) State the Research Hypothesis

    A research hypothesis states the expected relationship between two

variables. It may be stated in general terms, or it may include dimensions of
direction and magnitude. For example,
1. General: The length of the job training program is related to the
rate of job placement of trainees.
2. Direction: The longer the training program, the higher the rate of
job placement of trainees.
3. Magnitude: Longer training programs will place twice as many
trainees into jobs as shorter programs.
4. General: Graduate Assistant pay is influenced by gender.
5. Direction: Male graduate assistants are paid more than female
graduate assistants.
6. Magnitude: Female graduate assistants are paid less than 75% of
what male graduate assistants are paid.

2) State the Null Hypothesis

    A null hypothesis usually states that there is no relationship between

the two variables. For example,

There is no relationship between the length of the job training program

and the rate of job placement of trainees.

Graduate assistant pay is not influenced by gender.

   
 A null hypothesis may also state that the relationship proposed in the
research hypothesis is not true. For example,

Longer training programs will place the same number or fewer trainees
into jobs as shorter programs.

Female graduate assistants are paid at least 75% or more of what male
graduate assistants are paid.
   

Researchers use a null hypothesis in research because it is easier to

disprove a null hypothesis than it is to prove a research hypothesis. The null
hypothesis is the researcher's "straw man." That is, it is easier to show that
something is false once than to show that something is always true. It is easier
to find disconfirming evidence against the null hypothesis than to find
confirming evidence for the research hypothesis.
 

3) TYPE I AND TYPE II ERRORS

    Even in the best research project, there is always a possibility (hopefully
a small one) that the researcher will make a mistake regarding the relationship
between the two variables. There are two possible mistakes or errors.

    The first is called a Type I error. This occurs when the researcher
assumes that a relationship exists when in fact the evidence is that it does not.
In a Type I error, the researcher should accept the null hypothesis and reject
the research hypothesis, but the opposite occurs. The probability of committing
a Type I error is called alpha.

    The second is called a Type II error. This occurs when the researcher
assumes that a relationship does not exist when in fact the evidence is that it
does. In a Type II error, the researcher should reject the null hypothesis and
accept the research hypothesis, but the opposite occurs. The probability of
committing a Type II error is called beta.

    Generally, reducing the possibility of committing a Type I error increases

the possibility of committing a Type II error and vice versa, reducing the
possibility of committing a Type II error increases the possibility of committing
a Type I error.

    Researchers generally try to minimize Type I errors, because when a

researcher assumes a relationship exists when one really does not, things may
be worse off than before. In Type II errors, the researcher misses an
opportunity to confirm that a relationship exists, but is no worse off than
before.
 In this example, which type of error would you prefer to commit?

1. Research Hypothesis: El Nino has reduced crop yields in County X,

making it eligible for government disaster relief.
2. Null Hypothesis: El Nino has not reduced crop yields in County X,
making it ineligible for government disaster relief.

    If a Type I error is committed, then the County is assumed to be eligible

for disaster relief, when it really is not (the null hypothesis should be accepted,
but it is rejected). The government may be spending disaster relief funds when
it should not, and taxes may be raised.

    If a Type II error is committed, then the County is assumed to be

ineligible for disaster relief, when it really is eligible (the null hypothesis should
be accepted, but it is rejected). The government may not spend disaster relief
funds when it should, and farmers may go into bankruptcy.

In this example, which type of error would you prefer to commit?

1. Research Hypothesis: The new drug is better at treating heart

attacks than the old drug
2. Null Hypothesis: The new drug is no better at treating heart
attacks than the old drug

    If a Type I error is committed, then the new drug is assumed to be better
when it really is not (the null hypothesis should be accepted, but it is rejected).
People may be treated with the new drug, when they would have been better off
with the old one.

    If a Type II error is committed, then the new drug is assumed to be no

better when it really is better (the null hypothesis should be rejected, but it is
accepted). People may not be treated with the new drug, although they would
be better off than with the old one.
 

SELECT A PROBABILITY OF ERROR LEVEL (ALPHA LEVEL)

    Researchers generally specify the probability of committing a Type I

error that they are willing to accept, i.e., the value of alpha. In the social
sciences, most researchers select an alpha=.05. This means that they are
willing to accept a probability of 5% of making a Type I error, of assuming a
relationship between two variables exists when it really does not. In research
involving public health, however, an alpha of .01 is not unusual. Researchers
do not want to have a probability of being wrong more than 0.1% of the time, or
one time in a thousand.
    If the relationship between the two variables is strong (as assessed by a
Measure of Association), and the level chosen for alpha is .05, then moderate or
small sample sizes will detect it. As relationships get weaker, however, and/or
as the level of alpha gets smaller, larger sample sizes will be needed for the
research to reach statistical significance.
 

4) The Chi Square Test

    For nominal and ordinal data, Chi Square is used as a test for statistical
significance. For example, we hypothesize that there is a relationship between
the type of training program attended and the job placement success of
trainees. We gather the following data:
 

 
Type of Training Attended: Number attending Training
Vocational Education 200
Work Skills Training 250
Total 450
 
 
Placed in a Job? Number of Trainees
Yes 300
No 150
Total 450

    To compute Chi Square, a table showing the joint distribution of the two
variables is needed:

Table 1. Job Placement by Type of Training (Observed Frequencies)

 

Placed in a Job? Type of Training

Vocational Work Skills Total
Education Training
Yes 175 125 300
No 25 125 150
Total 200 250 450

    Chi Square is computed by looking at the different parts of the table. The
"cells" of the table are the squares in the middle of the table containing
numbers that are completely enclosed. The cells contain the frequencies that
occur in the joint distribution of the two variables. The frequencies that we
actually find in the data are called the "observed" frequencies.

    In this table, the cells contain the frequencies for vocational education
trainees who got a job (n=175) and who didn't get a job (n=25), and the
frequencies for work skills trainees who got a job (n=125) and who didn't get a
job (n=125).

    The "Total" columns and rows of the table show the marginal
frequencies. The marginal frequencies are the frequencies that we would find if
we looked at each variable separately by itself. For example, we can see in the
"Total" column that there were 300 people who got a job and 150 people who
didn't. We can see in the "Total" row that there were 200 people in vocational
education training and 250 people in job skills training. Finally, there is the
total number of observations in the whole table, called N. In this table, N=450.
 

Calculate Chi Square

1) display observed frequencies for each cell

2) calculate expected frequencies for each cell
3) calculate, for each cell, the expected minus observed frequency squared,
divided by the expected frequency
4) all up the results for all the cells

    To find the value of Chi Square, we first assume that there is no
relationship between the type of training program attended and whether the
trainee was placed in a job. If we look at the column total, we can see that 300
of 450 people found a job, or 66.7% of the total people in training found a job.
We can also see that 150 of 450 people did not find a job, or 33.3% of the total
people in training did not find a job.

    If there was no relationship between the type of program attended and
success in finding a job, then we would expect 66.7% of trainees of both types
of training programs to get a job, and 33.3% of both types of training programs
to not get a job.

    The first thing that Chi Square does is to calculate "expected" frequencies
for each cell. The expected frequency is the frequency that we would have
expected to appear in each cell if there was no relationship between type of
training program and job placement.

    The way to calculate the expected cell frequency is to multiply the
column total for that cell, by the row total for that cell, and divide by the total
number of observations for the whole table.
 For the upper left hand corner cell, multiply 200 by 300 and divide by
450=133.3. For the lower left hand corner cell, multiply 200 by 150 and divide
by 450=66.7 . For the upper right hand corner cell, multiply 250 by 300 and
divide by 450=166.7. For the lower right hand corner cell, multiply 250 by 150
and divide by 450=83.3

Table 2. Job Placement by Type of Training (Expected Frequencies)

 

Placed in a Job? Type of Training

Vocational Work Skills Total
Education Training
Yes 133.3 166.7 300
No 66.7 83.3 150
Total 200 250 450

    This table shows the distribution of "expected" frequencies, that is, the
cell frequencies we would expect to find if there was no relationship between
type of training and job placement.

    Note that Chi Square is not reliable if any cell in the contingency table
has an expected frequency of less than 5.

To calculate Chi Square, we need to compare the original, observed

frequencies with the new, expected frequencies. For each cell, we perform the
following calculations:

a) Subtract the value of the observed frequency from the value of the expected
frequency
b) square the result
c) divide the result by the value of the expected frequency

For each cell above,

 

fe - fo (fe - fo)2   [(fe - fo)2] / fe  Result

(133.3 - 175) (133.3 - 175)2 [(133.3 - 175)2] / 133.3  13.04
(66.7 - 25)  (66.7 - 25)2  [(66.7 - 25)2] / 66.7 26.07
(166.7 - 125) (166.7 - 125)2   [(166.7 - 125)2] / 166.7 10.43
(83.3 - 125) (83.3 - 125)2 [(83.3 - 135)2] / 83.3  20.88

To calculate the value of Chi Square, add up the results for each cell--
Total=70.42
 

DEGREES OF FREEDOM

    We cannot interpret the value of the Chi Square statistics by itself.
Instead, we must put it into a context.

    In theory, the value of the Chi Square statistic is normally distributed;
that is, the value of the Chi Square statistics looks like a normal (bell-shaped)
curve. Thus we can use the properties of the normal curve to interpret the
value obtained from our calculation of the Chi Square statistic.

    If the value we obtain for Chi Square is large enough, then we can say
that it indicates the level of statistical significance at which the relationship
between the two variables can be presumed to exist.

    However, whether the value is large enough depends on two things: the
size of the contingency table from which the Chi Square statistic has been
computed; and the level of alpha that we have selected.

    The larger the size of the contingency table, the larger the value of Chi
Square will need to be in order to reach statistical significance, if other things
are equal. Similarly, the more stringent the level of alpha, the larger the value
of Chi Square will need to be, in order to reach statistical significance, if other
things are equal. The term "degrees of freedom" is used to refer to the size of
the contingency table on which the value of the Chi Square statistic has been
computed. The degrees of freedom are calculated as the product of (the number
of rows in the table minus 1) times (the number of columns in the table
minus).
 

For a table with two rows of cells and two columns of cells, the formula
is:

df = (2 - 1) x (2 - 1) = (1) x (1) = 1
 For a table with two rows of cells and three columns of cells, the formula
is:

df = (3 - 1) x (2 - 1) = (2) x (1) = 2

For a table with three rows of cells and three columns of cells, the
formula is:

df = (3 - 1) x (3 - 1) = (2) x (2) = 4
   
The level of alpha can vary, but the smaller the value, the more stringent
the requirement for reaching statistical significance becomes. Alpha levels are
often written as the "p-value", or "p=.05." Usual levels are p=.05 (or the chance
of one in 20 of making an error), or p=.01 (or the chance of one in 100 of
making an error), or p=.001 (or the chance of one in 1,000 of making an error).

    When reporting the level of alpha, it is usually reported as being "less
than" some level, using the "less than" sign or <.  Thus, it is reported as p<.05,
or p<.01; unless you are reporting the exact p-value, such as p=.04 or p=.22.
 

DISTRIBUTION TABLES

    Once we have the calculated value of the Chi Square statistic, and the
degrees of freedom for the contingency table, and the desired level for alpha, we
can look up the normal distribution for Chi Square in a table. There are many
tables available in statistics texts for this purpose.

    In the table, find the degrees of freedom (usually listed in a column down
the side of the page). Next find the desired level of alpha (usually listed in a row
across the top of the page). Find the intersection of the degrees of freedom and
the level of alpha, and that is the value which the computed Chi Square must
equal or exceed to reach statistical significance.

    For example, for df=2 and p=.05, Chi Square must equal or exceed 5.99
to indicate that the relationship between the two variables is probably not due
to chance. For df=4 and p=.05, Chi Square must equal or exceed 9.49.
 

INTERPRET THE RESULTS

    If the computed value for Chi Square equals or exceeds the value
indicated in the table for the given level of alpha and degrees of freedom, then
the researcher can assume that the observed relationship between the two
variables exists (at the specified level of probability of error, or alpha), and
reject the null hypothesis. This gives support to the research hypothesis. The
computed value of Chi Square, at a given level of alpha and with a given degree
of freedom, is a type of "pass-fail" measurement. It is not like a measure of
association, which can vary from 0.0 to (plus or minus) 1.0, and which can be
interpreted at every point along the distribution. Either the computed value of
Chi Square reaches the required level for statistical significance or it does not.
 
It is important to note that Chi Square, like other tests for statistical
significance:

1) does not indicate the strength of an association between two variables

2) does not indicate the direction of an association between two variables
3) does not indicate the probability of a Type I error
4) does not take into account the reliability and validity of the research
5) does not provide absolute, conclusive proof of a relationship

To recap, for the example above:

1) state the research hypothesis:

1) There is a relationship between the type of training program attended

and the job placement success of trainees
2) state the null hypothesis:
3) There is no relationship between the type of training program
attended and the job placement success of trainees
4) calculate the test for statistical significance
5) Chi Square=70.42
6) 4) calculate the degrees of freedom of the contingency table
7) df=1
6) select the level of alpha
8) p=.05
7) look up the Chi Square value in the table at p=.05 and df=1
9) Chi Square=3.84
8) interpret the result
10) The computed value of Chi Square (70.42) exceeds the value in the
table for p=.05 and df=1 (Chi Square=3.84). Therefore, we can reject
the null hypothesis (with a 5% probability of error) and accept the
research hypothesis that a relationship exists between type of training
program attended and the job placement success of trainees.

Using T-Tests
    T-Tests are tests for statistical significance that are used with interval
and ratio level data. T-tests can be used in several different types of statistical
tests:

1) to test whether there are differences between two groups on the same
variable, based on the mean (average) value of that variable for each
group; for example, do students at private schools score higher on the
SAT test than students at public schools?
2) to test whether a group's mean (average) value is greater or less than
some standard; for example, is the average speed of cars on freeways in
California higher than 65 mph?
3) to test whether the same group has different mean (average) scores on
different variables; for example, are the same clerks more productive on
IBM or Macintosh computers?

To calculate a value of t,

a) state the research hypothesis;

b) state the null hypothesis;
c) stipulate whether the t-test will be a one-tailed test or a two-tailed test for
significance
d) select the level of alpha
e) calculate t

To calculate a value of t,

a) state the research hypothesis;

The average salary of male graduate assistants is higher than the average
salary of female graduate assistants at CSULB.
b) state the null hypothesis;
There is no difference in the average salary of male and female graduate
assistants at CSULB.
c) select the level of alpha
select a value for alpha, such as p=.05, p=.01, or p=.001
d) stipulate whether the t-test will be a one-tailed test or a two-tailed test for
significance

    Like other statistics, the t-test has a distribution that approaches the
normal distribution, especially if the sample size is greater than 30. Since we
know the properties of the normal curve, we can it to tell us how far away from
the mean of the distribution our calculated t-score is.

    The normal curve is distributed about a mean of zero, with a standard
deviation of one. A t-score can fall along the normal curve either above or below
the mean; that is, either plus or minus some standard deviation units from the
mean.
    A t-score must fall far from the mean in order to achieve statistical
significance. That is, it must be quite different from the value of the mean of
the distribution, something that has only a low probability of occurring by
chance if there is no relationship between the two variables. If we have chosen
a value of p=.05 for alpha, we look for a value of t that falls into the extreme 5%
of the distribution.

    If we have a hypothesis that states the expected direction of the results,
e.g., those male graduate assistant salaries are higher than female graduate
assistant salaries, and then we expect the calculated t-score to fall into only
one end of the normal distribution. We expect the calculated t-score to fall into
the extreme 5% of the distribution.

    If we have a hypothesis, however, that only states that there is some
difference between two groups, but does not state which group is expected to
have the higher score, then the calculated t-score can fall into either end of the
normal distribution. For example, our hypothesis could be that we expect to
find a difference between the average salaries of male and female graduate
assistant members (but we do not know which is going to be higher, or which
is going to be lower).

    For a hypothesis which states no direction, we need to use a "two-tailed"

t-test. That is, we must look for a value of t that falls into either one of the
extreme ends ("tails") of the distribution. But since t can fall into either tail, if
we select p=.05 for alpha, we must divide the 5% into two parts of 2-1/2%
each. So a two-tailed test requires t to take on a more extreme value to reach
statistical significance than a one-tailed test of t.

e) Calculate t

    A t-score is calculated by comparing the average value on some variable

obtained for two groups; the calculation also involves the variance of each
group and the number of observations in each group.  For example,

Table 3. Male and Female Graduate Assistant Salaries at CSULB

 
 MaleGraduate Female Graduate
Assistants Assistants
Number of  403 132
observations
Mean $17,095 $14,885
Standard  6329 4676
Deviation
Variance 40045241 21864976

To calculate t,
1) subtract the mean of the second group from the mean of the first group
2) calculate, for each group, the variance divided by the number of
observations minus 1
3) add the results obtained for each group in step two together
4) take the square root of the results of step three
5) divide the results of step one by the results of step four

For example,

1) subtract the mean of the second group from the mean of the first group
17095-14885=2210
2) calculate, for each group, the variance divided by the number of
observations minus 1
Male graduate assistants:
[40056241 / (403-1)] = [40056241 / (402)] = 99642
Female graduate assistants:
[21864976 / (132-1)] = [21864976 / (131)] = 166908
3) add the results obtained for each group in step two together
99642+166908=266550
4) take the square root of the results of step three
square root of 266550=516.28
5) divide the results of step one by the results of step four
2210/516.28=4.28
To interpret the results,
f) calculate the degrees of freedom
g) look up the value in the table
h) interpret the value of t

Degrees of freedom
 The degrees of freedom for the t-test is calculated by adding up the
number of observations for each group, and then subtracting the number two
(because there are two groups).  For example, (403 + 132 - 2) = 533

Distribution of T

    The values of t are printed in tables in most statistics texts. The values of
the degrees of freedom are listed in a column down the side, and the values of
alpha (p-value) are listed in a row across the top. There are different tables for
one-tailed and two-tailed tests of t. Find the correct table for the number of
tails. Then find the intersection of the degrees of freedom and the value of
alpha in the table. That value is the value that the calculated t-score must
equal or exceed to indicate statistical significance.
For a one-tailed test of t, with df=533 and p=.05, t must equal or exceed
1.645.
For a two-tailed test of t, with df=533 and p=.05, t must equal or exceed
1.960.

Interpret the value of t

    If the computed t-score equals or exceeds the value of t indicated in the
table, then the researcher can conclude that there is a statistically significant
probability that the relationship between the two variables exists and is not
due to chance, and reject the null hypothesis. This lends support to the
research hypothesis.

    In this example, the computed t-score of 4.28 exceeds the table value of
t, so we can reject the null hypothesis of no relationship between graduate
assistant gender and graduate assistant pay, and instead accept the research
hypothesis and conclude that there is a relationship between graduate
assistant gender and graduate assistant pay.

    Remember, however, that this is only one statistic, based on just one
sample, at one point in time, from one research project. It is not absolute,
conclusive proof that a relationship exists, but rather support for the research
hypothesis. It is only one piece of evidence, that must be considered along with
many other pieces of evidence on the same subject.
 

REPORTING TESTS OF STATISTICAL SIGNIFICANCE

    In research reports, tests of statistical significance are reported in three

ways. First, the results of the test may be reported in the textual discussion of
the results. Include:
1) the hypothesis
2) the test statistic used and its value
 3) the degrees of freedom
4) the value for alpha (p-value)
For example,
   
Workers in organizations with non-authoritarian management styles
were found to have higher levels of job satisfaction than workers in
organizations with authoritarian management styles (Chi Square=50.57, df=4,
p<.05).
The average salary of male graduate assistants is higher than that for
female graduate assistants (t=4.28, df=533, p<.05).
No differences were found in job placement rates between vocational
education programs and work skills programs (Chi Square=1.2, df=1,
p>.05).

    A second method of reporting the results of tests for statistical

significance is to report the test and its value, the degrees of freedom, and the
p-value at the bottom of the contingency table or printout showing the data on
which the calculations were based.

Table 1. Job Placement by Type of Training (Observed Frequencies)

 

Placed in a Job? Type of Training

Vocational Work Skills Total
Education Training
Yes 175 125 300
No 25 125 150
Total 200 250 450

Chi Square=70.42, df=1, p<.05

Table 3. Male and Female Graduate Assistant Salaries at CSULB

 

Male Graduate Female Graduate

Assistants Assistants
Number of  403 132
observations
Mean $17,095 $14,885
Standard  6329 4676
Deviation
Variance 40045241 21864976

t=4.28, df=533, p<.05

    The third way to report tests of statistical significance is to include them
in tables showing the results of an extended analysis of the data, including a
number of variables. For example, here are some results from a study of older
Hispanic women in El Paso, TX, and Long Beach, CA.

Table 4. Characteristics of Workshop Participants Age 40 and Older

 

Characteristics El Paso Long Beach value of

(N=83) (N=131) t
Mean age 60.5 years 68.7 years 2.1*
Ethnic self-identification  97.2 89.7 0.9
Mexican-American
Language preference 68.5 52.3 3.2**
Spanish only

*t significant at p<.05
**t significant at p<.01
 

Final Comments

    Tests for statistical significance are used to estimate the probability that
a relationship observed in the data occurred only by chance; the probability
that the variables are really unrelated in the population. They can be used to
filter out unpromising hypotheses.

    Tests for statistical significance are used because they constitute a
common yardstick that can be understood by a great many people, and they
communicate essential information about a research project that can be
compared to the findings of other projects.

    However, they do not assure that the research has been carefully
designed and executed. In fact, tests for statistical significance may be
misleading, because they are precise numbers. But they have no relationship
to the practical significance of the findings of the research.

    Finally, one must always use measures of association along with tests for
statistical significance. The latter estimate the probability that the relationship
exists; while the former estimate the strength (and sometimes the direction) of
the relationship. Each has its use, and they are best when used together.