MULTIVARIATE ANALYSIS AND

STATISTICAL SIGNIFICANCE 19
INTRODUCTION 421
MULTIVARIATE ANALYSIS 421
Regression Analysis 421
Path Analysis 424
Other Multivariate Techniques 425
STATISTICAL INFERENCE 426
Tests of Statistical Significance 426
The Misuse of Tests of Significance 428
SUMMARY 429
KEY TERMS 430
EXERCISES 430
SUGGESTED READINGS 431
REFERENCES 432

INTRODUCTION

significance to make inferences about a pop-

The first part of this chapter introduces ulation (or universe) based on the evidence
regres- sion analysis, one of the most widely obtained from a sample selected from that
used mul- tivariate statistical techniques. pop- ulation. Many social researchers put a
Although a com- prehensive treatment of this great deal of confidence in these tests of
topic is beyond the scope of a first course in signifi- cance, and it is important that both
research meth- ods, a brief introduction to researchers and consumers of social research
regression analysis is essential because it understand their uses and misuses. Although
appears so frequently in quantitative social they can help researchers to answer
research literature. The aim of the present legitimately a wide range of questions, tests of
discussion is to give the reader a basic significance simply cannot answer some
overview and some suggestions for further questions.
reading. One reason that regression analysis is
so
MULTIVARIATE ANALYSIS
widely used is that it lends itself to causal mod-
eling. Although there are many types of
causal modeling, Regression Analysis
wediscusspathanalysisherebecause it is one of LINEAR REGRESSION is a statistical procedure
the most commonly used types. used to estimate the amount of change in a
The second section of this chapter covers sta- dependent variable that can be expected for a
tistical inference. Social researchers use tests of given change in an independent variable. We
will begin by considering SIMPLE REGRESSION,
Thischaptermaybedifficultwithoutapriorcourseinstatistics.
421

Downloaded from https://www.cambridge.org/core. Macquarie University, on 07 Dec 2019 at 15:16:07, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/CBO9780511819391.020
422 Multivariate Analysis and Statistical Significance

Y  a  bX

Y

X
b  Y
X

scattergram, as
Figure 19.1. The equation for a straight line.

which involves one dependent variable and one

independent variable (or predictor). We will
then consider MULTIPLE REGRESSION, which
involves one dependent variable and two or more
predictors. Recall from elementary algebra that
the equation for a straight line is

Y =a+
bX.
Figure 19.1 is an illustration of the interpreta-
tion of the constants a and b in this equation.
We find that a is the value Y takes when X is
equal to zero. It is referred to as the Y-
intercept because it is the value of Y at the
point where the straight line crosses the Y-axis.
The constant b is equal to the slope of this
line. If we move an arbitrary distance along
the line described by this equation, recording
the amount that Y has changed(callit Y)
andtheamount X haschanged (call it X), and
then divide the change in Y by the change in X,
the result is the slope of the line (i.e., b =
Y/X).
In this example from elementary algebra,
the Y values refer to points along the straight
line defined by the equation Y = a + bX. This
formula
does not account for any Y values that do not
fall
on this line. For any arbitrary value of X, we
can find the corresponding Y value that
satisfies the equation by locating the Y value
on the straight line that falls directly over the
specified X value (i.e., we would determine
that point at which a line constructed
perpendicular to the X-axis from the specified
X value intersects the straight line given by the
equation Y = a + bX).
Simple regression is a procedure for fitting
a straight line to a set of points in a
 X

illustrated in Figure 19.2. The regression

line is that line through the set of points
for which the sum of the squares of the
deviations from the line is a minimum.
These deviations are shown in Figure
19.2. For any line other than the regres-
sion line through the same set of points,
the sum of the squares of the deviations is
greater.
The major distinction between the
regression line equation and the basic
algebra linear equa- tion is that in
regression values of Y rarely fall along
the actual line, whereas in basic algebra
values of Y always fall along the line.
By con- structing a vertical line from
each observed Y value to the regression
line, as illustrated in Fig-
ure 19.2, we can locate a set of Yˆ (called
Y-
predicted) values that do fall along the
regres- sion line. Thus, the equation for
the regression line becomes

Yˆ = a
+ bX.

Suppose that Y is “annual income” and

that X is “years of education.” In simple
regression, the slope (b) is referred to as
the REGRESSION COEF- fiCIENT. If the
regression coefficient has a value of 500,
we would estimate that for a one-year
increase in level of education there
would be a
$500 increase in annual income. In
general, the regression coefficient gives
the number of units of change in Y (in
whatever units Y is measured) that can be
expected for a one-unit change in X (in
whatever units X is measured).
In regression analysis, it is important
to dis- tinguish between the actual Y
values that do not fall on the regression
line and the correspond- ing Yˆ values
that we would estimate based on
 Multivariate Analysis 423

Y  a  bX

YYi

X
Yi b  Y
X

Figure 19.2. Fitting a least squares regression line to a set of points in a scattergram.

a given respondent’s X value. The discrepancy between Y and Yˆ represents error in our
between the actual Y value and the estimated pre- diction. If we have selected a set of
Yˆ value represents prediction error. When the predictors that yield accurate estimates of Y,
Y values tend to cluster very close to the then the dif- ference between Y and Yˆ values
regression line, Yˆ and Y values will be very will be small, andthemultiplecorrelationwill
similar, and the error in prediction will be low. behigh. If, how- ever, we have selected a set
However, when the Y values tend to deviate of predictors that yields poor estimates of Y,
markedly from the regression line, the Y and then the difference between Yˆ and Y values
Yˆ values will be quite different, will tend to be larger, and the multiple
andtheerrorinpredictionwill behigh. Multiple correlation will be low. The multiple
regression is an extension of simple correlation ranges from .00 (when the
regression: Instead of one predictor, we include independent variables in no way help to
two or more predictors in a single regression predict Y) to 1.00 (when the independent
equation. When there are four predictors, the variables pre- dict Y with complete accuracy).
equation is as follows:
The multiple- correlation coefficient squared
Yˆ = a + b1x1 + b2x2, + b3x3 + b4x4. (R2) gives the proportion of the variance in
the dependent variable that is accounted for
The b values in multiple regression are by the set of pre- dictors included in the
referred toaspartial-regressioncoefficients. 1 regression equation. If R = .50, then R2 = .
Thesecoef- ficients give the change in the 25; we would conclude that the predictors
dependent vari- able (in whatever units the being considered account for 25 percent of
dependent variable is measured) the variance in the dependent variable.
thatwewouldestimateforaone-unit change in Let us assume that our goal is to predict
the specified predictor (in whatever units the the grade-point average for 1,000 seniors who
predictor is measured). have just graduated from college. Suppose we
The MULTIPLE CORRELATION COEFfiCIENT decide to use the following four predictors: high
(R) is used to summarize the accuracy of our
prediction equation.2 Recall that the difference script 1 refers to the dependent variable X1 and the sub-
scripts 2, 3, 4, and 5 refer to the predictorsX2, X3, X4, and
1 It is equal to the Pearson correlation between Y and the X5. There will be as many numbers following the period in
Yˆ values. The b values are referred to as partial- the subscript as there are predictors. The notation system
regression coefficients because they are estimates of has been changed here so that the dependent variable
the change in the dependent variable that is estimated referred tointhetestas Y isreferredtohereas X1. For
for a one- unit change in the specified predictor after thesubscripted multiple-correlation coefficient, as for
we statisti- cally control for the effects of the other several other multi- variate statistics (e.g., the partial-
predictors in the equation. correlation coefficient), the notion is simpler if we refer
2 The subscripted version of the multiple-correlation coeffi- to our variables as X1, X2, X3, and so on, rather than as Y,
cient is designated symbolically as “R1.2345” where the sub- X1, and X2.
 424 Multivariate Analysis and Statistical Significance

.30 Respondent’s Education X3

Mother’s Education X5

.40
.30

.05Respondent’s Present Occupation X1

.10
.05
.45 .30

Mother’s Occupation X4 Respondent’s First Job X2

.20

Figure 19.3. A model of the process of socioeconomic achievement.

school grade-point average (X1), mother’s

is estimated to change (in whatever units it is
edu- cation (X2), verbal SAT (Scholastic Aptitude
measured) for a one-unit change in the inde-
Test) score (X3), and mother’s occupational
pendent variable (in whatever units it is
status (X4). These variables are all measured in
mea- sured). For this reason, the units in
differ- ent units; consequently, we cannot
which the variables are measured make a
make direct comparisons among these partial-
difference. If income were one of our
regression coefficients (b1, b2, b3, and b4) to
predictors, we would have a choice of units for
determine their relative strength as predictors
measuring it. We might decide on yen, dollars,
of grade-point average.
or lire. Depending on which units we
Fortunately, there is a way to manage this selected, we would get a dif- ferent
problem. It calls for computing
unstandardized partial-regression coef- ficient
STANDARDIZED
for income. However, because the stan-
PARTIAL-REGRESSION COEFfiCIENTS (these dardized partial-regression coefficient is not
coefficients are commonly referred to as BETA influenced by the unit of measurement, it
wEIGHTS) for each of these predictors. The beta would be the same for each of these three
weight is a partial-regression coefficient that has alternatives.
been adjusted in such a way that the unit of
mea- sure does not influence its value . All
Path Analysis
units are changedtostandarddeviationunits.
Thus, when a beta weight equals .50, our Now that we have considered multiple regres-
interpretation is thattherewill bea.50 sion, it is appropriate to discuss PATH ANALY-
standarddeviationchange in the dependent SIS, a form of causal modeling based on
variable (grade-point average) for a 1 standard multi- ple regression. Path analysis can be
deviation change in the speci- fied predictor. viewed as a procedure for presenting the
Because each of the coefficients is now stated results of a series of multiple regressions, or
in standard deviation units, it is pos- sible to as a procedure for doing causal modeling with
compare the relative strength of each predictor. multiple regression. To be more concrete, we
The statistic referred to earlier as the will consider a model of socioeconomic
partial-regression coefficient is also called achievement.
the UNSTANDARDIZED PARTIAL-REGRESSION The model in Figure 19.3 includes four pre-
COEFfiCIENT. As we recall, this statistic dictors and the “respondent’s present occupa-
indi- cates how many units the dependent tion,” which is the main dependent variable.
variable The arrows in the model specify a seemingly
plausible causal order among these variables

Downloaded from https://www.cambridge.org/core. Macquarie University, on 07 Dec 2019 at 15:16:07, subject to the Cambridge Core terms of use, available at
https://www.cambridge.org/core/terms. https://doi.org/10.1017/CBO9780511819391.020
Multivariate Analysis 425

before looking at the data. The selection of pre-

dictors and the assumed causal ordering called direct effect of “respondent’s
among these predictors is based on prior education” on “respondent’s present
theory, past research, and common sense. In occupation” is .40; that is, for every increase
the present model, we assume that “mother’s of 1 standard deviation in “respondent’s
education” (x5) influences “mother’s education,” we would estimate an increase of .
occupation” (x4), that “mother’s education” 40 standard deviation in the score for
and “mother’s occupa- tion” influence “respondent’s present occupation.” On the
“respondent’s education” (x3), that all three basis of the path coefficients in Figure 19.3, we
influence “respondent’s first job” (x2), and would conclude that the direct effect of
that all four influence “respondent’s present “respon- dent’s education” on “respondent’s
occupation” (x1). This model is then used to first job” (.45) is much greater than is the
set up a series of multiple-regression direct effect of “mother’s education” on
equations. In this case, four separate “respondent’s first job” (.05). It is also
equations would be called for, one for each possible to use path analy- sis to estimate the
variable in the model that is used as a indirect effect of one variable on another
dependent variable. (Any variable that has an through an intervening variable, by
arrow coming toward it is being used as a multiplying the appropriate path coefficients.3
dependent variable.) For equa- Because of space limitations, the
tion(1),“respondent’spresentoccupation” isthe presentation of path analysis has been
dependent variable, and it is predicted by the simplified in many ways. We have not
other four variables. In equation (2), “respon- discussed residual paths or the
dent’s first job” is the dependent variable, and it decomposition of the total relationship
is predicted by the three variables in the between variables into causal and noncausal
model that are causally prior to it. In equation components. We have only briefly touched on
(3), “respondent’s education” is the dependent thedecompositionofthecausalcomponentinto
vari- able, and it is predicted by the two direct and indirect components.4
variables that are causally prior. Finally, in
equation (4), “mother’s occupation” is the
dependent vari- able, and it is predicted by Other Multivariate Techniques
the one variable, “mother’s education,” that is
Although this chapter focuses on multiple re-
causally prior to it. These four equations may
gression as an important example of the logic of
be summarized as follows:
multivariate analysis, researchers have a range
of multivariate techniques from which to select.
Many of these techniques are similar to mul-

Xˆ1 = a1 + b1 x1 + tiple regression in that they involve one

+ b3x3 + (1) depen- dent variable and two or more
b2 x2
b4x4 predictors, but do not necessarily require
both the indepen-
Xˆ2 = a2 + b1 x1 + b2 x2 + b3 x3 (2) coefficient for the so-

Xˆ3 = a3 + b1 x1 + b2 x2 (3)

Xˆ4 = a4 + b1 x1 (4)

These equations yield a set of unstandardized

partial-regression coefficients that are some-
times used in path analysis, but it is more
com- mon to use the corresponding
standardized partial-regression coefficients, or
beta weights. In path analysis, the beta weights
are called PATH COEFfiCIENTS and are often
presented along the corresponding arrows as
we have done in Figure 19.3. The path
dent and dependent variables to be interval
level. Techniques such as logistic regression and
discriminant analysis involve a nominal-level
dependent variable; n-way ANOvA (analysis of
variance) and ANCOvA (analysis of covariance)
involve mostly nominal-level independent vari-
ables. Survival or failure analysis is used when
3 In Figure 19.3, we may obtain the indirect effect of “respon-
dent’s education” on “respondent’s present occupation,”
which occurs through the intervening variable, “respon-
dent’s first job.” We multiply the direct effect of X3 on X2
(which is .45) by the direct effect of X2 on X1 (which is .30)
and obtain an indirect effect of X3 on X1, through the inter-
vening variable X2, equal to .135, which we might round
to .14.
4 For a more complete introduction to path models, see
Ullman (2001).
426 Multivariate Analysis and Statistical Significance

the dependent variable is the time until an

the mean income for Democratic respondents
event, such as getting married or getting
is $45,000. On the basis of this evidence, we
divorced, occurs. These techniques, along
know that there is a difference in mean
with many oth- ers, seek to predict the values
income between Republicans and Democrats
of the dependent variable based on the
for the sample. We are probably more
independent variables. Other multivariate
interested, how- ever, in knowing whether we
techniques do not explic- itly deal with
can infer from this that there is a tendency in
independent and dependent vari- ables. For
the community as a whole for Republicans to
instance, log-linear analysis examines the
have higher incomes than do Democrats.
association between three or more nominal-
Estimates based on a sample rarely corre-
level variables. Other techniques, such as
spond to the exact value for the population.
cluster analysis and factor analysis, look for As we learned in Chapter 6, probability theory
clusters of related cases or variables.5
assures us that there will be SAMPLING ERROR
(the deviation of the mean for our sample from
STATISTICAL INFERENCE
the true population mean) in our estimates of
In quantitative research, we are rarely the mean; that is, sample estimates will
interested in our sample per se. Rather, we fluctu- ate around the true population value.
want to gener- alize about a larger population Because some sample estimates will be too
based on what we know about the sample high and oth- ers will be too low, we do not
drawn from it. When our goal is simply to know whether our estimate of $70,000 is
describe the characteris- tics of the sample or above or below the actual mean income of
to describe the relationship between variables Republicans in the com- munity. The mean
for the sample, we engage in descriptive income for Democrats might be higher than
statistical analysis. However, we often want the mean income for Republi- cans. This
to infer characteristics of the pop- ulation would be the case if, owing to sam- pling
based on a sample’s characteristics or make error, the sample estimate for Republican
inferencesabouttherelationshipbetween respondents overstates their actual income in
variables in the population based on the rela- the community by $15,000 and the sample esti-
tionship between these variables in the mate for Democratic respondents understates
sample. When we want to infer population their actual income by $11,000. In that case,
character- istics based on information from a the actual population mean for Republicans
sample, we engage in STATISTICAL INFERENCE. would be $55,000, whereas the actual mean
Tests of sig- nificance are often used as a basis for Democrats would be $56,000.
for statistical inference. Suppose that before looking at our data, we
have the idea that in the community the mean
Tests of Statistical Significance income for Republicans is higher than the
mean income for Democrats. How might we
TESTS OF SIGNIfiCANCE are widely used in test this research hypothesis? As a first step,
both descriptive and explanatory research. we might compute the mean incomes for
Researchers often use these tests to help decide, Republicans ($70,000) and Democrats
based on the relationship between two vari- ($45,000) in our sam- ple. Nevertheless, our
ables in a sample, whether to infer a relation- hypothesis refers to the community, not to our
ship between these variables in the population sample. We can ask how likely it is that we
from which the sample was drawn . Suppose we would find a difference as great that in our
interview arepresentative cross section of a sample as the result of sampling error alone,
local community and find that the sample if the population mean incomes of
estimate of the mean income for Republican Republicans and Democrats were equal. More
respon- dents is $70,000, whereas the sample generally, whenever we want to test the hypoth-
estimate of esis that one group is different from another,
we
5
See Mertler and Vannatta (2001) for introductions to most canaskwhetherthedifferencefoundinthesam-
of these techniques. ple could be expected on the basis of
sampling error (chance) alone. To this end, we
formulate
Statistical Inference 427

what is referred to as a nULL HYPOTHESIS

that there is indeed no difference. Our null is, we reject the null hypothesis that the incomes
hypothesis in this case is that in the of Republicans and Democrats are equal for
community the mean income of Republicans the community. However, we realize that we
is equal to the mean income of Democrats. could be wrong. We realize that there are 5
Along with the null hypothesis, we always chances in 100 that a difference as great as
formulate a research hypothesis. In the pre- that found in our sample could have resulted
ceding example, we specified a DIRECTIONAL from sampling error alone. Although we have
HYPOTHESIS, that is, that the mean income of decided to classify the
Republicans is higher than the mean income differenceinincomeasstatisticallysignificant, it
of Democrats. Sometimes we do not specify a is still possible that the incomes for Republicans
direction in our research hypothesis. A and Democrats in the community are equal.
nONDI- RECTIONAL HYPOTHESIS appropriate Whatifwehadcarriedoutatestofsignificance
to the example would be that the mean andfound thatwe were notable to reject
income for Republicans is not equal to that thenull hypothesis that the mean incomes for
for Democrats. The income for Republicans Repub- licans and Democrats are exactly
could be either higher or lower than that for equal? Does
Democrats and still be consistent with this thismeanthatweacceptthenullhypothesisand
nondirectional researchhypothesis. conclude that the incomes are identical? No, we
Whenourresearchhypoth- esis is directional,
do not. There is a difference between failing
we use what is referred
to reject the null hypothesis and actually
accepting it. It is very unlikely that the mean
income for Republicans will be exactly equal
to the mean
to as ONE-TAILED TESTS SIGNIfiCANCE income for Democrats even if we have failed to
OF
because we have specified the nature of the rela- tailed” tests of significance is too technical for present
treatment; a
tionship between variables in our hypothesis . fulldiscussionoftheissuemaybefoundinanyintroducto
Whenourresearchhypothesisisnondirectional, ry statistics text.
we use TWO-TAILED TESTS OF SIGNIfiCANCE
because we have not specified the nature of the
relationship.6
When carrying out a significance test, we
always specify a SIGNIfiCANCE LEvEL. It is a
common practice to select in advance one of
the conventional significance levels, such as .05,
.0l, or .001. These levels refer to there being a 5
percent, 1 percent, and .1 percent
probability, respectively, of getting a
relationship as strong as that in our sample
when there is in fact no relationship between
the variables in the popu- lation. If, for
example, we select the .05 level, this says that
we are going to classify the differ- ence in
means as statistically significant if there are
fewer than 5 chances in 100 that the differ-
ence is due to sampling error alone. Suppose we
carry out a test of significance on the difference
between the mean incomes of Republicans
and Democratsforoursample.
Ifthedifferenceissig- nificant at the .05 level, we
refer to the relation- shipasbeing
STATISTICALLY SIGNIfiCANT; that

6
The rationale for the names “one-tailed” and “two-
reject the null hypothesis. There is a distinction
between the conclusion that the difference in
means found in our sample could result from
chance (sampling error) alone if the population
means were exactly equal and the conclusion
that the population means are identical.
A test of significance is a procedure for decid-
ing how likely it is that the relationship we have
found in the sample is due to sampling error
when there is no relationship between variables
inthepopulation. Itcannotbeusedtoprovethat there
actually is a relationship in the population. In
addition, it cannot prove that there actually is no
relationship in the population. A test of sig-
nificance can be used only to indicate how likely
we would be to obtain the relationship we find in
the sample if there were no relationship in the
population.
The t-TEST is a test of significance that we use
when we are interested in comparing the means
for two samples or two categories of the same
sample. The null hypothesis for a t-test is that the
population means are equal (and therefore, in the
example above, that the mean incomes for the
Democrats and Republicans are equal). When we
want to compare the means for more than two
groups in the same test of significance, we use the
F-TEST. The null hypothesis for the F-test is that
the means for all the groups being
428 Multivariate Analysis and Statistical Significance

compared are equal (for example, the means sample.

for the Democrats, Independents, and
Republican incomes). The research hypothesis
for the F-test is that the means are not equal;
there is no direc- tional (one-tailed) option
when more than two groups are being
considered.
Tests of significance are often used in contin-
gencytableanalysis. Thetestmostoftenusedwith
contingency tables, when at least one of the
vari- ables concerned is at the nominal level ,
is the CHI-SqUARE(χ 2). The null hypothesis for
the chi-square test is that there is no
relationship between the two variables in the
table; that is, the respondents are distributed
among the table cells as would be expected by
chance.
The PEARSON CORRELATION COEFfiCIENT
(r), a well-used measure of association, can
be tested for statistical significance. The null
hypothesis here is that the correlation
between the two variables in the population is
zero. If we have grounds for predicting the
direction of the correlation, we choose an
alternative research hypothesis that states that
there is a positive or negative correlation
between the variables and we use a one-tailed
test. If we do not have a prior hypothesis as to
the direction of the correlation, the research
hypothesis states that in the popu- lation the
correlation between the two variables is not
equal to zero, and we use a two-tailed test. In
this case, if the correlation proves to be
statistically significant, we accept the
research hypothesis that the correlation in the
popula- tion is not zero. Note that the
significance test does not say the correlation
in the population is equal to the correlation
for the sample. In fact, it
saysnothingabouttheactualstrengthofthecor-
relation in the population. More generally, the
strength of a correlation and the statistical
sig- nificance of a correlation are conceptually
inde- pendent. In general, for a given sample
size, cor- relation that are statistically
significant will tend to be larger than
correlations that are not. How- ever, with a
large enough sample, a weak corre- lation will
be statistically significant; and with a small
enough sample, a strong correlation will not
be statistically significant.
The reasonthatthesame-strengthcorrelation
is more likely to be classified as statistically
sig- nificant if it is based on a larger sample is
that there is less sampling error for a large
If we were to draw a sample of 5 to educationalexpenditures.” Supposealsothatwe
estimate the mean income for a are working with data for all fifty states. In such
community, our estimate would tend to
7 For a complete discussion of the logic of probability sam-
be less accurate than if it were based on
pling, see Chapter 6.
a sample of 500. In general, a sam- ple 8 See Ritchey (2000) for a brief discussion of statistical
statistic based on a larger sample will power.

more closely approximate the

corresponding popula- tion parameter
thanwill thesame statistic based on a
smaller sample.7 Thus, a nonzero
correla- tion based on a large sample is
less likely to be duetochancethanis
thesamecorrelationbased
onasmallersample.
Thesamelogicappliestoall measures of
association. For a given strength of
association, the larger the sample size,
the more likely it is that the association
will prove to be statistically significant.8

The Misuse of Tests of Significance

Social scientists have debated a great deal
about theuse of tests of significance. Of
particular con-
cernistheevidencethatsuchtestsarefreque
ntly used in situations for which the
requirements underlying the test have
not been met. Some argue that the
problem has been made worse by
pressure from journal editors and
reviewers who encourage the use of
such tests, despite the violation of
theassumptions thatunderlie them.
Others argue that tests of significance
do have their uses even when these
assumptions are not completely
satisfied.
All tests of significance require a
probability sample,
andmostassumeasimplerandomsam- ple.
However, it is common to find tests used
in studies where there is not even a
remote approx- imation to a simple
random sample. Most social research is
based on cluster, quota, or accidental
samples,
forwhichtheerrorismuchgreaterthan in a
simple random sample. This failing can
lead to inflated estimates of statistical
significance.
A related problem is the use of tests
of sig- nificance when the researcher is
working with the entire population.
Suppose we are using state-level data,
and we find a correlation of .30 between
“median income” and “level of state
Summary 429

a situation, our sample (N = 50) is the popu- reports based only on the statistically significant
lation, and so it is inappropriate to compute a relationships. Such researchers are
test of statistical significance. If the capitalizing on sampling error and may
correlation between these variable is .30, then . prepare an entire reportarounda set of
30 is the cor- relation in the population, and it correlations thatcould not be replicated.
is meaningless to compute a test of Suppose a researcher computes 1,000
significance to test the null hypothesis that in correlation coefficients and tests each for
the population the correla- tion is zero. In statistical significance. Even if all of these
short, if we already have the pop- ulation, there cor- relations in the population are zero, we
is no need to make the inferences that tests of would expect on the basis of sampling error
significance are designed to help us make. that 50 (or 5 of every 100) of these
Researchers sometimes use a test of signifi- correlations would be significant at the .05
cance to generalize beyond the population level. Thus, if the researcher looks through the
from which the sample was drawn. Suppose we 1,000 correlations and bases thereportonthe 50
have a orsothatarestatistically sig- nificant, the findings
simplerandomsampleoftheseniorsatacollege, reported run a risk of being highly unreliable.
and we find that the Jews in the sample are Most of these correlations will have resulted
more likely to support gun control than the from sampling error alone. For this reason, it
Catholics. If this difference turns out to be will not be possible to replicate the findings
statistically sig- nificant, we can generalize to of the study.
all the seniors at the college. It might seem Therearedifferenttestsof significanceforvar-
plausible that a simi- lar ious types of data. Some tests are appropriate
trendwouldholdforseniorsatothercolleges, for for nominal-level data, some are appropriate
all college students, or for the adult popula- for ordinal-level data, and still others are
tion in general. However, we have no grounds appropri- ate for interval- and ratio-level data.
for making such a generalization based on the A common error is to use a test of significance
data we have. appropri- ate for interval-level data when the
Statistical significance is often confused with data are only ordinal level. A typical example
SUBSTANTIvE SIGNIfiCANCE, that is, whether of this error is the computation of a test of
some piece of data is important to us . It is com- significance for a Pearson correlation
mon for researchers to suggest that a finding between two ordinal-level variables.
is important because it is statistically signifi-
cant. While it is generally reasonable to
SUMMARY
discount findings that are not statistically
significant, statistical significance per se does Regression is one of the most commonly used
not make a relationship important. We can statistical procedures in social research. In sim-
often find sta- tistically significant ple regression analysis, we consider only two
relationships between vari- ables that are variables, one dependentvariable andone
causally unrelated, variables that are inde- pendent variable (the predictor). The
alternative measures of the same thing (for regression line is the line through the set of
instance, the relationship between age and data points being considered that minimizes
year of birth), and variables that are the sum of the square of the deviations from
sociologically uninteresting (such as the the line. For any other line through this same
relationship between weight and waist set of points, the sum is greater.
measurement). There can also be relationships Multiple regression, an extension of simple
based on very large samples that are regression to include two or more predictors,
statistically significant but so weak as to be is themostwidelyusedformofregressionanalysis.
substantively unimportant. A correlation can The coefficients that result when we do multi-
be very close to zero (even .01) and still be ple regression are called partial-regression
statistically significant if the sample size is coef- ficients, or unstandardized partial-
large enough. regression coefficients. These specify the
Another misuse of significance tests is number of units of difference in the dependent
illus- trated by researchers who compute a variable we would
very large number of tests and then prepare
research
430 Multivariate Analysis and Statistical Significance

estimate for a one-unit difference in the

multiple regression
predic- tor being considered when we multiple-correlation
statistically con- trol all the other predictors nondirectional research hypothesis
in the equation. For some purposes, such as null hypothesis
path analysis, it is useful toobtainasetof one-tailed test of significance
standardizedpartial-regression coefficients, partial-regression coefficient
which are called beta weights in the context of path analysis
multiple-regression analysis, and path path coefficient
coefficients in the context of path analysis. The Pearson correlation coefficient
beta weight gives the number of standard regression coefficient
deviations difference in the dependent sampling error
variable we would estimate for a one standard significance level
deviation unit difference in the predictor simple regression
being consid- ered. When using beta weights, standardized partial-regression coefficient
it is possible to compare the “effects” of the statistical inference
statistical significance
various predictors because they are all being
substantive significance
measured in the same standard deviation
t-test
units.
two-tailed test of significance
Tests of statistical significance are
unstandardized partial-regression coefficient
extensively used in explanatory research. A
test of signifi- cance cannot be used to prove
that there actu- ally is a relationship in the EXERCISES
population or that there actually is no 1. If we consider a simple regression in which
relationship in the popula- tion. Such a test the dependent variable is “monthly income
can, however, be used to indi- cate how likely measured in yen” and the predictor is “time at
we are to obtain the relation- ship we find in work mea- sured in hours,” what would be the
the sample, if there is no rela- tionship in the units for the regression coefficient? If the
population. Tests of significance are used numerical value of the regression coefficient were
incorrectly by many social researchers. One of to be 1,000, how much of a difference in income
the most common errors is the use of such would we expect between two persons who differ
tests when there are flagrant violations of the by six hours with respect to hours worked?
assumption of a simple random sample. In 2. Consideramultipleregressioninwhich X1, isthe
such situations, levels of statistical signifi- dependent variable, with X2, and X3, as predictors.
cance are often greatly inflated. They are also That is, where X1 = a + b2x2 + b3x3. Assume also
inflated when the researcher uses a test that that
assumes interval-level data on measures that
are ordinal. Another error is explicitly or X1 = monthly income in dollars
implic- itly to convey the impression that X2 = time worked in hours
statistically significant relationships are
X3 = years of education completed
substantively sig- nificant when in fact they
are not. When the sample is large, a very a. Set up the appropriate regression equation
weak relationship, one that may account for for the above example. Be sure to use subscripts
less than 1 percent of the variance, may turn for the partial-regression coefficients.
out to be statistically significant. b. If you wanted to compare the relative
strength of time worked (X2) and years of
education completed (X3,) as predictors, which
KEY TERMS would you use– theunstandardizedpartial-
beta weight regressioncoef- ficients or the standardized
chi-square partial-regression coefficients? Defend your
directional research hypothesis choice.
F-test
linear regression
Suggested Readings 431

on X1 through the intervening variable X2. Which

is larger?
X2
6. Using adatasetprovided by yourinstructor,
per- form some basic statistical tests, such as
one-tailed and two-tailed t-tests and one-way
ANOVAs.
7. From the section on the misuse of tests of
X4 X1 signif- icance, we know that sometimes people
incorrectly generalize results from one
population to another. Usingthe
SPSSdatasetsoffemaleandmalerespon- dents in the
X3 1998 General Social Survey, avail- able at
http://webapp.icpsr.umich.edu/GSS/, run the
same regression analysis using each data set. Use
Figure 19.4. Comparing path coefficients. respondent’s education (EDUC) as the depen- dent
variable, and mother’s education (MAEDUC),
c. If we are considering an equation based on father’s education (PAEDUC), and age of
unstandardized partial-regression coefficients, respon- dent (AGE) as independent variables.
how much of a difference in X1 would we esti- How are the results similar or different for men
and for women?
mate for a one-unit difference in X2? If we are
considering an equation based on standardized
partial-regression coefficients, how much of a
difference in X1 values would we estimate for
SUGGESTED READINGS
a one-unit difference in X2? Assume that b2 = Kuh, George D. 2006. The National Survey of
1,000 and b3 = 50,000 and that the Student Engagement: Conceptual Framework and
corresponding stan- dardized coefficients are . Overview of Psychometric Properties. Indiana
63 and .47. Uni- versity.
3. Construct the series of regression equations This Web site http://nsse.iub.edu/redirect.cfm?
that would be used to estimate the path target contains data from, and description of,
coefficients in Figure 19.4. Would you prefer to a regression analysis performed on American
use the unstan- dardizedpartial- col- lege students. In surveys, students are
regressioncoefficientsorthestan- dardized partial- asked to report the frequency with which they
regression coefficients to estimate the path engage in dozens of activities that represent
coefficients? Defend your choice. good educa- tional practice, such as using the
4. In whatwaysisastandardizedpartial-regression institution’s humanresources,
coefficient similar to a partial-correlation coeffi- curricularprograms, andother opportunities for
cient? In what ways is it different? learning and development that the college
provides. Additional items assess the amount of
5. For the path model in Figure 19.5, compare
reading and writing students did dur- ing the
the direct effect of X3 on X1 with the indirect effect
current school year, the number of hours per
of X3
week they devoted to schoolwork, extracur-
ricular activities, employment, and family
matters.
X2
Levin, Jack, and James Alan Fox. 2006.
Elementary Statistics in Social Research. Boston:
.20 .30 Allyn & Bacon.
This book is an introduction to descriptive
statis- tics, hypothesis testing, one-tailed and
X3 X1 two-tailed tests, and significance tests such as
.10 analysis of variance, chi-square tests,
correlation, and sim- ple regression.
Figure 19.5. Comparing direct and indirect effects.
Mertler, Craig A., and Rachel A. Vannatta. 2001.
Advanced and Multivariate Statistical Methods:
432 Multivariate Analysis and Statistical Significance

Practical Application and Interpretation. Los

Tabachnick, Barbara G., and Linda S. Fidell. 2001.
Angeles: Pryrczak.
Using Multivariate Statistics. 4th ed. Boston:
This text introduces a range of multivariate Allyn & Bacon.
statistics, including analysis of variance and
An alternative to Mertler and Vannatta (2001)
covariance, multivariate analysis of variance
that provides a more in-depth treatment of tech-
and covariance, multiple regression, path anal-
niquessuchasmultipleregression, canonicalcor-
ysis, factor analysis, discriminant analysis, and
relation, analysisofvarianceandcovariance, log-
logistic regression. Instructions for these tech-
linear analysis, discriminant function analysis,
niques using SPSS for Windows are included,
logistic regression, factor analysis, and structural
along with suggestions for interpreting the SPSS
equation modeling.
output.
Ritchey, Ferris. 2000. The Statistical Imagination: REFERENCES
Elementary Statisticsforthe Social Sciences. Boston:
McGraw-Hill. Mertler, Craig A., and Rachel A. Vannatta. 2001.
Advanced and Multivariate Statistical Methods:
Thisisanotherintroductiontobasicstatisticsthat
Practical Application and Interpretation. Los
includes more treatment of “statistical
Angeles: Pryrczak.
fallacies” such as treating sample estimates as
if they were true of the population and assuming Ritchey, Ferris. 2000. The Statistical Imagination:
that point estimates are true of the population. Elementary Statisticsforthe Social Sciences. Boston:
McGraw-Hill.
Spicer, John. 2004. Making Sense of Multivariate
Data Analysis. Thousand Oaks, CA: Sage. Ullman, Jodie B. 2001. “Structural Equation
Model- ing.” In Using Multivariate Statistics,
Thisbookoffersanintroductiontotheapproaches
653–771. 4th ed. Barbara G. Tabachnick and
to data discussed in this chapter.
Linda S. Fidell, eds. Boston: Allyn & Bacon.