EC771: Econometrics, Spring 2012

Greene, Econometric Analysis (7th ed, 2012)

Chapters 4, 8:
Properties of LS and IV estimators

We now consider the least squares estimator

from the statistical viewpoint, rather than as
merely an algebraic curve-fitting tool, return-
ing to the assumptions that we spelled out in
earlier presentations of the least squares ap-
proach.

The least squares method can be motivated by

several of its advantages: including, of course,
its ease of computation. Least squares, like
many of the methods we use in econometrics,
is a generalized method of moments (GMM)
estimator, motivated by a set of orthogonal-
ity conditions on population moments. Let x
be the vector of independent variables in the
population regression function, which may be
stochastic or nonstochastic. We assume that
these variables are exogenous: in statistical
terms, that the population errors are orthogo-
nal to the independent variables: E[|x] = 0. If
this conditional mean is zero, the covariance is
also zero, and we have a vector of K moment
conditions: one for each independent variable.
Those moment conditions may be written as
E[x] = E[x(y − x0β)] = 0,
or
E[xy] = E[xx0]β.
This is a population relationship on the mo-
ments of the variables. A method of moments
estimator replaces population moments with
consistent estimates derived from the sample.
The normal equations of least squares, (X 0X)b =
X 0y may be written as
   
n n
1 X 1 X
 xiyi =  xix0i b
n i=1 n i=1
in which we estimate the population moments
as the average of the sample moments over
the n observations of the sample. Therefore,
the least squares estimator may be considered
as a method of moments estimator, in which
the population relationship is reflected by that
methodology applied to the sample.

We can also motivate the use of least squares

as a solution to the problem of finding an op-
timal linear predictor: x0γ. The mean squared
error of this predictor is

M SE = E[y − x0γ]2
which may be decomposed as

M SE = E[y − E[y|x]]2 + E[E[y|x] − x0γ]2

in which the first term does not contain γ.
Minimizing MSE thus only involves the sec-
ond term. The first order condition for this
problem will lead us to the same population
expression as above:

E[xy] = E[xx0]γ
which leads to the same least squares esti-
mator as that for β above. Thus, the least
squares estimator solves the optimal linear pre-
dictor problem: even without specification of
the form of the conditional mean function E[y|x],
since we did not use the assumption of linearity
in the above derivation.

Unbiasedness of least squares

The LS estimator is unbiased:

b = (X 0X)−1X 0y
b = (X 0X)−1X 0(Xβ + )
b = β + (X 0X)−1X 0
Taking expectations of this expression, we find
that E[b|X] = E[b] = β, since by the assump-
tion of orthogonal regressors (exogeneity), the
expectation of X 0 is zero. For any particular
set of observations X the least squares estima-
tor has expectation β. If we average over the
possible values of X (in the case where X is
stochastic) the unconditional mean of b is β as
well.

Best linear unbiased estimation

To discuss the precision of the LS estimators,

we may consider X as a matrix of fixed con-
stants in order to derive the sampling variance
of b. Alternatively, we can perform the analysis
conditional on X, and average over the possi-
ble values of X as above. Returning to the
definition of the LS estimator,

b = (X 0X)−1X 0(Xβ + ) = β + (X 0X)−1X 0.

Since we can write b = β + A, where A =
(X 0X)−1X 0, b is a linear function of the dis-
turbances, and is thus a linear estimator; from
the above, we may also write b = Ay, so that
the LS estimator is a linear function of the de-
pendent variable. Since we have established
unbiasedness, we may state that the LS b is
a linear, unbiased estimator. What about its
covariance matrix? Given exogeneity of the X
variables and our assumption on spherical dis-
turbances,

V ar[b|X] = E[(b − β)(b − β)0|X]

= E[(X 0X)−1X 00X(X 0X)−1|X]
= (X 0X)−1X 0E[0|X]X(X 0X)−1
= (X 0X)−1X 0(σ 2I)X(X 0X)−1
= σ 2(X 0X)−1.
Let us consider a more general approach to
computing a minimum–variance linear unbiased
estimator of β. Let b0 = Cy be another linear,
unbiased estimator of β, which implies

E[Cy|X] = E[(CXβ + C)|X) = β,

which implies that CX = I. The covariance
matrix of b0 can be found, using the above
derivation, as σ 2CC 0. Let

D = C − (X 0X)−1X 0,
so Dy = (b0 − b). Then
0 −1 0 0 −1 0 0
h i
2
V ar[b0|X] = σ (D + (X X) X )(D + (X X) X )
Since

CX = DX + (X 0X)−1(X 0X) = I,
DX = 0, and

V ar[b0|X] = σ 2(X 0X)−1 + σ 2DD0,

we find that the variance of any linear, unbi-
ased estimator of β equals the variance of the
LS estimator, V ar[b], plus a positive semidef-
inite matrix. The matrix D contains the dif-
ferences between the elements of the arbitrary
estimator b0 and the LS estimator b. Iff those
differences are uniformly zero, the variance of
b0 will be minimized; but if that is so, b0 = b.
For all other choices of b0, the estimator will
be less precise than the corresponding LS es-
timator. Thus, b is the minimum variance lin-
ear unbiased estimator of β: the Gauss-Markov
theorem, stating that b is “BLUE”: the Best
Linear Unbiased Estimator. The theorem also
states that if we seek to estimate w0β, where w
is an arbitrary vector of constants, the BLUE
of w0β will be w0b.

Estimating σ 2(X 0X)−1

In order to test hypotheses about β or to form

confidence intervals, we need a sample esti-
mate of the covariance matrix

V ar[b|X] = σ 2(X 0X)−1,

in particular, the population parameter σ 2. In
the least squares approach, unlike the maxi-
mum likelihood approach to the problem, we
do not generate an estimate of σ 2 in the op-
timization. Since σ 2 is the expected value of
2
i and ei is an estimate of i , by analogy we
might calculate
n
2 1 X
σ̂ = e2
n i=1 i
which is, indeed, the maximum likelihood es-
timator of σ 2. But the least squares residu-
als, although consistent estimates of the cor-
responding i, are imperfect estimates of their
population counterparts in that we have re-
placed β with its unbiased estimate, b:
ei = yi − x0ib = i − x0i(b − β)
Although the expected value of ei is i, that
does not necessarily hold for the expected value
of the squares. In matrix form,
e = M y = M [Xβ + ] = M ,
since M X = 0. An estimator based on the
residual sum of squares is then
e0e = 0M 
with expected value

E[e0e|X] = E[0M |X]

The quantity 0M  is a scalar, thus equal to its
trace, and

E[tr(0M )|X] = E[tr(M 0)|X]

Since M is a function of X, this can be written
as

tr(M E[0|X]) = tr(M σ 2I) = σ 2tr(M )

Since we know that

tr(I − X(X 0X)−1X 0)

= tr(In) − tr[(X 0X)−1(X 0X)]
=n−K
We find that E[e0e|X] = (n − K)σ 2, so that an
unbiased estimator of σ 2 will be
e0e
s2 =
n−K
and the standard error of regression, its posi-
tive square root, is s. We can now also calcu-
late the estimated variance–covariance matrix
of the estimated parameters:

Est.V ar[b|X] = s2(X 0X)−1,

and the positive square roots of the diagonal
elements of this matrix are the estimated stan-
dard errors si of the regression parameters bi.

Normality and simple hypothesis testing

Thus far, we have not used the assumption

of normality: even in deriving estimates of the
precision of the LS coefficients above, we did
not invoke normality. But to make use of those
estimated standard errors, we must impose a
specific distributional assumption: that the el-
ements of b are distributed multivariate Nor-
mal.
How might we test an hypothesis about a spe-
cific coefficient? Assuming normality,
b − βk
z = pk
σ 2S kk
where S kk is the kth element on the main diag-
onal of (X 0X)−1. If we could compute this ex-
pression, z would have a standard Normal dis-
tribution under H0 : bk = βk0. Using s2 rather
than σ 2, we can derive a feasible test statistic:
(n − K)s2 e0e  0 
   
= = M
σ2 σ2 σ σ
is an idempotent quadratic form in a standard
normal vector σ , which will have a χ2 distribu-
tion with tr(M ) = (n − K) degrees of freedom.
We may prove that this quadratic form is in-
dependent of the expression in s2. That is,
if  is normally distributed, the least squares
coefficient vector b is statistically independent
of the residual vector e and all functions of e,
including s2.
Since a Student t–distributed variable is the ra-
tio of a standard normal variate to the square
root of a χ2 random variable which has been
divided by its degrees of freedom, a bit of al-
gebra will lead us to
bk − βk
tn−K = p
s2S kk
in which we have essentially replaced σ 2 with
its consistent estimate s2. The resulting test
statistic is distributed Student t with (n − K)
degrees of freedom under H0. In particular,
the ratio of the estimated coefficient to its es-
timated standard error is distributed t under
the null hypothesis that the population coeffi-
cient equals zero. Likewise, we may construct
a α–percent confidence interval for βk as

P r(bk − tα/2sbk ≤ βk ≤ bk + tα/2sbk ) = 1 − α

where (1 − α) is the desired level of confidence,
and tα/2 is the appropriate critical value from
the t distribution with (n − K) degrees of free-
dom.

How might we test the hypothesis that the re-

gression as a whole is significant: in a model
with a constant term, a joint test of the hy-
pothesis that all regressors’ slopes are zero?
The ANOVA F statistic may be written as a
transformation of the R2 value:

K−1 R2/(K − 1)
Fn−K =
(1 − R2)/(n − K)
which will have a Fisher F distribution under
that null hypothesis. In later discussions, we
will consider alternative joint hypotheses on
combinations or subsets of the regression pa-
rameters.
Collinearity

Although the OLS estimator is minimum vari-

ance in the class of linear, unbiased estima-
tors of the population parameters, that vari-
ance may still be unacceptably large under cer-
tain circumstances: such as a “high degree”
of collinearity. If two regressors are perfectly
correlated, of course, their sampling variances
go to infinity, since the inverse matrix that en-
ters the expression for their sampling variances
cannot be calculated. What if that correla-
tion is high, or in general, what if there are
near–linear dependencies in the regressor ma-
trix? When this is encountered, we may find
that some data points have a great deal of
“leverage”: small changes in the data matrix
may cause large changes in the parameter es-
timates. Although the overall fit of the regres-
sion (as measured by R2 or R¯2) may be very
good, the coefficients may have very high stan-
dard errors, and perhaps even incorrect signs
or implausibly large magnitudes.

These are all understandable consequences of

near–linear dependencies in the regressor ma-
trix. If we consider a k–variable regression
model containing a constant and (k−1) regres-
sors, we may write the kth diagonal element of
(X 0X)−1 as:
1
(1 − Rk2)Skk
where Rk2 is the R2 from the regression of vari-
able k on all other variables in the model, and
Skk is the variation in the kth variable about
its mean. The estimated variance of the kth
coefficient estimate is s2 times this quantity.
Observations about this expression:

• The greater the correlation of xk with the

other regressors (including the constant term),
 cet.par., the higher will be the estimated
variance;

• The greater the variation in xk about its

mean, cet. par., the lower will be the esti-
mated variance;

• The better the overall fit of the regression,

the lower will be the estimated variance.

This expression is the rationale for the so–

called V IF , or variance inflation factor, (1 −
Rk2)−1. The V IFk measures the degree to which
the variance has been inflated due to the non–
orthogonality of regressor k. Another measure,
a summary for the regression equation, is the
condition number of X 0X, which can be ex-
pressed as the positive square root of the ra-
tio of the largest to the smallest eigenvalue
of the matrix. Since a matrix which is near–
computationally singular will have at least one
very small eigenvalue, the condition number of
such a matrix will be large (relative to the value
of unity that would apply for I). Belsley, Kuh
and Welsch (BKW) have written the seminal
work on such regression diagnostics, and they
suggest that the condition number should be
calculated from a transformed data matrix in
which each regressor has unit length. A rule of
thumb would suggest that a condition number
in excess of 20 might be cause for concern.
But just as there is no objective measure of
how small the determinant of X 0X might be to
trigger instability in the estimates, it is difficult
to come up with a particular value that would
indicate a problem. One conclusion should be
clear: the statement that “these estimates are
flawed by collinearity among the regressors” is
hardly sensible if the model fits well and its
coefficient estimates are acceptably precise.
Some illustrations of the effects of contrived
collinearity may be viewed in the accompany-
ing handout. A number of these techniques,
as well as several of BKW’s proposed measures
of “influential observations”, may be found in
the Stata documentation under “regression di-
agnostics.”

Large–sample properties of OLS and IV esti-

mators

To consider the asymptotic properties of least

squares estimators, we leave the DGP for X
unspecified: it may include any mixture of con-
stants and random variables generated inde-
pendently of the DGP producing . Two cru-
cial assumptions:

• (xi, i), i = 1, . . . , n is a sequence of inde-

pendent observations;
 X 0X
• plim n = Q, a positive definite matrix

The OLS estimator may then be written as

!−1
0 X 0
!
XX
b=β+ .
n n
Presuming the existence of Q−1,
X 0
!
plim b = β + Q−1 plim
n
The plim of the last term may be written as
the sample average of xii = wi, and each term
in that average has expectation zero. For non-
stochastic x, this follows from the marginal
distribution of ; for stochastic x, the indepen-
dence of the two DGPs provides the result.
Thus, we may write

plim b = β + Q−1 plim w̄.

Likewise, we may use the spherical distribution
of  to derive the conditional variance of this
expression:
σ2 X 0X
! !
V ar[w̄|X] = E[w̄w̄0|X] = ,
n n
or
σ2 X 0X
! !
V ar[w̄] = E
n n
The variance will collapse to zero if the ex-
pectation converges to a constant matrix, so
that the leading scalar will dominate as n in-
creases. Under this condition—on the “well–
behavedness” of the regressor matrix—the limit
of the variance of w̄ is zero. Since the mean
of w̄ is identically zero and its variance con-
verges to zero, we may state that w̄ converges
in mean square to zero:
x0 
plim = 0,
n
or
plim b = β.
The assumptions we have used above are often
too restrictive in the case of time–series data
with trending variables and polynomials in the
regressors. In that case, a weaker set of as-
sumptions about X, the so–called Grenander
conditions, state that the regressors are suit-
ably “well-behaved” to lead to consistency of
the OLS estimator. Those conditions are likely
to be satisfied in empirical datasets.

Asymptotic normality of the OLS estimator

Our result on the plim of b allows us to write

!−1
0
!
√ XX 1
n(b − β) = √ X 0
n n
Since this inverse matrix has a plim equal to
Q−1, the limiting distribution
  of the above quan-
tity is that of Q−1 √1n X 0; we need to con-
√
sider the limiting distribution of n(w̄ − E[w̄]),
where the expectation is zero. What then is
 √
the limiting distribution of nw̄? Assuming in-
dependence of the observations, w̄ is the aver-
age of n independent random vectors wi = xii
with means zero and variances (given spheri-
cal disturbances) of σ 2Qi. Thus the variance of
√ 2
nw̄ is σ 2Q̄n = σn [Q1 + Q2 + · · · + Qn]. As long
as this sum is not dominated by any term and
the regressors are “well behaved” as discussed
above, the limit of this quantity is σ 2Q. Thus if
xii are n independent vectors distributed with
mean zero and finite variance σ 2Q, we may
write
!
1 d
√ X 0 → N [0, σ 2Q].
n
We may premultiply this quantity by Q−1, which
then leads to the result that
√ d
n(b − β) → N [0, σ 2Q−1],
or in terms of b itself,
σ2
" #
a
b ∼ N β, Q−1 .
n
The importance of this result is that if the
regressors are “well behaved” and the obser-
vations are independent, then the asymptotic
normality of the OLS estimator does not de-
pend on normality of the disturbances, but
rather on the central limit theorems used in the
derivation. It will, of course, follow if the dis-
turbances themselves are distributed normally.

To make this operational, we must estimate

the two quantities in the covariance matrix:
e0 e
σ 2 by (n−K) and (1/n)Q−1 by (X 0X)−1. The
former estimator can be demonstrated to be
consistent, since it can be written
1
2
s = [0 − 0X(X 0X)−1X 0]
(n − K)
 !−1 !
0 0 0 0
!
n   X XX X 
= − .
n−K n n n n

Since the leading constant has a plim of one,

and the second term in the brackets converges
to zero, we are concerned about the conver-
gence of the first term. Under the weak con-
ditions of finite moments of  (two if they
are identically distributed), we have that plim
s2 = σ 2, giving us the appropriate estimator
for the asymptotic covariance matrix of b.

The delta method

The delta method may be used to generate

estimated variances and covariances for func-
tions of b. Let f (b) be a set of J continuous
and continuously differentiable functions of b,
and define
∂f (b)
C(b) =
∂b0
be the J ×K matrix whose j th row is the vector
of derivatives of the j th function with respect
to b0. We can then write

plim f (b) = f (β)

and
∂f (β)
plim C(b) = 0
= Γ.
∂β
Using a linear Taylor series,

f (b) = f (β) + Γ(b − β) + . . .

If plim b = β, the higher-order terms are negli-
gible as n → ∞. So we can write
σ2
" #
a
f (b) ∼ N f (β), Γ Q−1Γ0 .
n
so that the operational estimated asymptotic
covariance matrix will be

C[s2(X 0X)−1]C 0.
If any of the J functions are nonlinear, the un-
biasedness of b may not carry over to f (b).
Nevertheless, f (b) will be a consistent estima-
tor of f (β), with a consistent estimate of the
asymptotic covariance matrix. This is the ra-
tionale for the widely–employed delta method,
implemented in Stata as the testnl command.
Asymptotic efficiency

What about asymptotic efficiency, the large–

sample counterpart of the Gauss–Markov re-
sult? In finite samples, we can prove that OLS
is BLUE under a set of assumptions on X and
 (which do not require normality of the ). As
we noted in examining maximum likelihood es-
timators, OLS is also a MLE if  is distributed
multivariate normal. Since MLEs are asy. effi-
cient among consistent and asy. normally dis-
tributed estimators, we can state that OLS
estimators are asy. efficient in the presence of
normally distributed . Conversely, if the error
distribution is not normal, this result of asy.
efficiency does not follow.

Heterogeneity in xi and dependent observa-

tions

The assumptions made to establish these asymp-

totic results include exogeneity of the regres-
sors (violations of which we will discuss in the
next section), spherically distributed errors, and
independence of the observations. The latter
assumption is often called into question in the
context of a panel, or longitudinal data set,
where we have multiple observations on each
of a number of units. It is surely likely that the
xs will be correlated across observations within
individuals: indeed, some may be constant for
each individual. The regressors are likely to in-
clude both stochastic terms (such as family in-
come, or firms’ revenues) and non–stochastic
regressors such as an individual “fixed effect”.
The asymptotics for such a setup are often
described as “large n, small T ”: that is, very
commonly we have a large number of obser-
vations (individuals, families, or firms) with a
time–series of modest length on each individ-
ual. In this case, we hold T fixed and let n in-
crease, treating each individual’s observations
as a unit. The conditions necessary to estab-
lish convergence are those related to n → ∞.
This is the setup, for instance, underlying the
“dynamic panel data” estimator of Arellano
and Bond, which is commonly applied to panels
of firm–level balance sheet and income state-
ment data. There are, of course, instances
of “small n, large T ” panels, for which the
asymptotics consider holding n fixed and let-
ting T increase; and in the econometric theory
underlying a number of panel unit–root tests,
there are asymptotics in which both n and T
are allowed to increase.

A second cause for violation of the assumption

of independence is that of regressors which are
prior values of the dependent variable: that
is, lagged dependent variables. We continue
to assume that the disturbances are i.i.d., but
even with this assumption, it is obvious that
the regressor vectors are correlated across ob-
servations. Since every observation yt is de-
pendent on the history of the disturbances,
we cannot assume strict exogeneity: by con-
struction, current  is correlated with future x.
The finite–sample results presented earlier no
longer hold in this case of stochastic regres-
sors which cannot be considered independent
of the error process, past, present and future;
only asymptotic results remain.

To generate asymptotic results for this case,

we must modify the assumption of strict exo-
geneity with

E[t|xt−s] = 0 ∀s ≥ 0.
This states that the disturbance at period t
is an innovation, uncorrelated with the past
history of the x process. It cannot be uncor-
related with the future of the process, since it
will become part of those future values. We
further must assume that the series in x are
stationary (at least in terms of covariance sta-
tionarity), which assumes that they have finite,
non–time–varying second moments which de-
pend only on the temporal displacement be-
tween their values; and that the autocorrela-
tion of the series is damped (so that the de-
pendence between observations declines with
the temporal displacement, and sample esti-
mates of the autocovariance function will be
suitable estimates of their population counter-
parts). The combination of these conditions is
equivalent to stating that the regressors are
stationary and ergodic. Under these condi-
tions, consistency of the OLS estimator can
be proven.

Next, we consider the appropriate strategy if

the assumption of exogeneity of the regressors
is untenable: for instance, in the context of a
simultaneous equations model.
The method of instrumental variables

The equation to be estimated is, in matrix no-

tation,

y = Xβ + u, E(uu0) = Ω (1)
with typical row

yi = Xiβ + ui (2)

The matrix of regressors X is n × K, where

n is the number of observations. The error
term u is distributed with mean zero and the
covariance matrix Ω is n×n. Four special cases
for Ω that we will consider are:

Homoskedasticity:Ω = σ 2I
  
σ12 0

 ... 

σi2
 
Heteroskedasticity:Ω =  
...
 
 
 
0 2
σn

σ2
 

 σ21 σ 2
 

σ2
 
Serial correlation:Ω = 
 σ31 σ32

 .. ... ... ...

 .


σn1 σn2 . . . σ2

 
Σ1 0

 ... 

 
Clustering:Ω =  Σm 
...
 
 
 
0 ΣM

where Σm indicates an intra-cluster covariance

matrix. For cluster m with t observations, Σm
will be t × t. Zero covariance between obser-
vations in the M different clusters gives the
covariance matrix Ω, in this case, a block-
diagonal form.

Some of the regressors are endogenous, so that

E(Xiui) 6= 0. We partition the set of regres-
sors into [X1 X2], with the K1 regressors X1
assumed under the null to be endogenous, and
the (K − K1) remaining regressors X2 assumed
exogenous.

The set of instrumental variables is Z and is

n × L; this is the full set of variables that
are assumed to be exogenous, i.e., E(Ziui) =
0. We partition the instruments into [Z1 Z2],
where the L1 instruments Z1 are excluded in-
struments, and the remaining (L − L1) instru-
ments Z2 ≡ X2 are the included instruments /
exogenous regressors:
Regr X = [X1 X2] = [X1 Z2] = [Endog Exog]

Inst Z = [Z1 Z2] = [Excluded Included]

The order condition for identification of the

equation is L ≥ K; there must be at least as
many excluded instruments as there are en-
dogenous regressors. If L = K, the equation
is said to be exactly identified; if L > K, the
equation is overidentified.

Denote by PZ the projection matrix Z(Z 0Z)−1Z 0.

The instrumental variables or two-stage least
squares (2SLS) estimator of β is

β̂IV = (X 0Z(Z 0Z)−1Z 0X)−1X 0Z(Z 0Z)−1Z 0y =

(X 0PZ X)−1X 0PZ y
The asymptotic distribution of the IV estima-
tor under the assumption of conditional ho-
moskedasticity can be written as follows. Let
QXZ = E(Xi0Zi)

QZZ = E(Zi0Zi)
and let û denote the IV residuals,
û ≡ y − X β̂IV
Then the IV estimator is asymptotically dis-
A
tributed as β̂IV ∼ N (β, V (β̂IV )) where
1 2 0
V (β̂IV ) = σ (QXZ Q−1
ZZ Q XZ ) −1
n
Replacing QXZ , QZZ and σ 2 with their sample
estimates
1
QXZ = X 0Z
n
1 0
QZZ = ZZ
n

2 û0û
σ̂ =
n
we obtain the estimated asymptotic variance–
covariance matrix of the IV estimator:

V (β̂IV ) = σ̂ 2(X 0Z(Z 0Z)−1Z 0X)−1 = σ̂ 2(X 0PZ X)−1

Note that some packages, including Stata’s

ivreg, include a degrees–of–freedom correction
to the estimate of σ̂ 2 by replacing n with n − L.
This correction is not necessary, however, since
the estimate of σ̂ 2 would not be unbiased any-
way (cf. Greene, 2000, p. 373.)

The Generalized Method of Moments

The standard IV estimator is a special case

of a Generalized Method of Moments (GMM)
estimator. The assumption that the instru-
ments Z are exogenous can be expressed as
E(Ziui) = 0. The L instruments give us a set
of L moments,

gi(β̂) = Zi0ûi = Zi0(yi − Xiβ̂)

where gi is L × 1. The exogeneity of the instru-
ments means that there are L moment condi-
tions, or orthogonality conditions, that will be
satisfied at the true value of β:

E(gi(β)) = 0
Each of the L moment equations corresponds
to a sample moment, and we write these L
sample moments as
n n
1 X 1 X 0 1 0
g(β̂) = gi(β̂) = Zi (yi − Xiβ̂) = Z û
n i=1 n i=1 n
The intuition behind GMM is to choose an es-
timator for β that solves g(β̂) = 0.

If the equation to be estimated is exactly iden-

tified, so that L = K, then we have as many
equations—the L moment conditions—as we
do unknowns—the K coefficients in β̂. In this
case it is possible to find a β̂ that solves g(β) =
0, and this GMM estimator is in fact the IV es-
timator.
If the equation is overidentified, however, so
that L > K, then we have more equations than
we do unknowns, and in general it will not be
possible to find a β̂ that will set all L sam-
ple moment conditions to exactly zero. In this
case, we take an L × L weighting matrix W
and use it to construct a quadratic form in the
moment conditions. This gives us the GMM
objective function:

J(β̂) = ng(β̂)0W g(β̂)

A GMM estimator for β is the β̂ that minimizes
J(β̂). Deriving and solving the K first order
conditions
∂J(β̂)
=0
∂ β̂
yields the GMM estimator:

β̂GM M = (X 0ZW Z 0X)−1X 0ZW Z 0y (3)

Note that the results of the minimization, and

hence the GMM estimator, will be the same for
weighting matrices that differ by a constant of
proportionality (we will make use of this fact
below). Beyond this, however, there are as
many GMM estimators as there are choices of
weighting matrix W .

What is the optimal choice of weighting ma-

trix? Denote by S the covariance matrix of the
moment conditions g:
1 1
S = E(Z uu Z) = E(Z 0ΩZ)
0 0
n n
where S is an L × L matrix. The general for-
mula for the distribution of a GMM estimator
is

V (β̂GM M ) =
1 0
(QXZ W QXZ )−1(Q0XZ W SW QXZ )(Q0XZ W QXZ )−1 (4)
n
The efficient GMM estimator is the GMM es-
timator with an optimal weighting matrix W ,
one which minimizes the asymptotic variance
of the estimator. This is achieved by choosing
W = S −1. Substitute this into Equation (3)
and Equation (4) and we obtain the efficient
GMM estimator

β̂EGM M = (X 0ZS −1Z 0X)−1X 0ZS −1Z 0y

with asymptotic variance
1 0
V (β̂EGM M ) = (QXZ S −1QXZ )−1
n

Note the generality (the “G” of GMM) of the

treatment thus far; we have not yet made any
assumptions about Ω, the covariance matrix of
the disturbance term. But the efficient GMM
estimator is not yet a feasible estimator, be-
cause the matrix S is not known. To be able
to implement the estimator, we need to esti-
mate S, and to do this, we need to make some
assumptions about Ω.
GMM and heteroskedastic errors

Let us start with one of the most commonly

encountered cases in cross–section analysis: het-
eroskedasticity of unknown form, but no clus-
tering. We need a heteroskedasticity–consistent
estimator of S. Such an Ŝ is available by using
the standard “sandwich” approach to robust
covariance estimation. Denote by Ω̂ the diag-
onal matrix of squared residuals:
 
û2 0
 1 ... 
 
û2
 
Ω̂ =  i

...
 
 
 
0 û2
n
where ûi is a consistent estimate of ui. Then
a consistent estimator of S is
1 0
Ŝ = (Z Ω̂Z)
n
This works because, although we cannot hope
to estimate the n diagonal elements of Ω with
only n observations, they are sufficient to en-
able us to obtain a consistent estimate of the
L × L matrix S.

The û used for the matrix can come from any

consistent estimator of β; efficiency is not re-
quired. In practice, the most common choice
for estimating û is the IV residuals. This gives
us the algorithm for the feasible efficient two-
step GMM estimator, as implemented in ivreg2,gmm2s.

1. Estimate the equation using IV.

2. Form the residuals û. Use these to form

the optimal weighting matrix Ŵ = Ŝ −1 =
1 (Z 0 Ω̂Z) −1 .
 
n

3. Calculate the efficient GMM estimator β̂EGM M

and its variance-covariance matrix using the
 estimated optimal weighting matrix.This yields

β̂EGM M = (X 0Z(Z 0Ω̂Z)−1Z 0X)−1 ×

X 0Z(Z 0Ω̂Z)−1Z 0y
with asymptotic variance

V (β̂EGM M ) = (X 0Z(Z 0Ω̂Z)−1Z 0X)−1

GMM, IV and homoskedastic vs. heteroskedas-

tic errors

Let us now see what happens if we impose the

more restrictive assumption of conditional ho-
moskedasticity on Ω. This means the S matrix
simplifies:
1 1
S = E(Z ΩZ) = σ E(Z 0Z)
0 2
n n
1 Z 0 Z,
The expectation term can be estimated by n
but what about σ 2? As we noted above, the
GMM estimator will be the same for weight-
ing matrices that differ by a constant of pro-
portionality. We can therefore obtain the ef-
ficient GMM estimator under conditional ho-
moskedasticity if we simply ignore σ 2 and use
as our weighting matrix
1 0 −1
 
Ŵ = ZZ
n
Substituting into Equation (3), we find that it
reduces to the formula for the IV estimator in
Equation (3). To obtain the variance of the
estimator, however, we do need an estimate of
σ 2. If we use the residuals of the IV estimator
to calculate σ̂ 2 = n1 û0 û, we obtain

2 1 0
Ŝ = σ̂ Z Z
n
Finally, if we now set
1 0 −1
 
Ŵ = Ŝ −1 2
= σ̂ Z Z
n
and substitute into the formula for the asymp-
totic variance of the efficient GMM estimator
we find that it reduces to the formula for the
asymptotic variance of the IV estimator. In ef-
fect, under the assumption of conditional ho-
moskedasticity, the (efficient) iterated GMM
estimator is the IV estimator, and the itera-
tions converge after one step. It is worth not-
ing that the IV estimator is not the only such
efficient GMM estimator under conditional ho-
moskedasticity. Instead of treating σ̂ 2 as a pa-
rameter to be estimated in a second stage,
what if we return to the GMM criterion func-
tion and minimize by simultaneously choosing
β̂ and σ̂ 2? The estimator that solves this min-
imization problem is in fact the Limited Infor-
mation Maximum Likelihood estimator (LIML).
In effect, under conditional homoskedasticity,
the continuously updated GMM estimator is
the LIML estimator. Calculating the LIML es-
timator does not require numerical optimiza-
tion methods; it can be calculated as the so-
lution to an eigenvalue problem. The latest
version of ivreg2 (Baum, Schaffer and Still-
man) supports LIML and k–class estimation
methods.

What are the implications of heteroskedastic-

ity for the IV estimator? Recall that in the
presence of heteroskedasticity, the IV estima-
tor is inefficient but consistent, whereas the
standard estimated IV covariance matrix is in-
consistent. Asymptotically correct inference is
still possible, however. In these circumstances
the IV estimator is a GMM estimator with a
sub–optimal weighting matrix, and hence the
general formula for the asymptotic variance of
a general GMM estimator The IV weighting
matrix Ŵ remains, what we need is a consis-
tent estimate of Ŝ. This is easily done, using
exactly the same method employed in two–step
efficient GMM. First, form the “hat” matrix
Ω̂ using the IV residuals, and use this matrix
to form the Ŝ matrix. Substitute this Ŝ, the
(sub–optimal) IV weighting matrix Ŵ and the
sample estimates of QXZ and QZZ into the
general formula for the asymptotic variance of
a GMM estimator (4), and we obtain an es-
timated variance–covariance matrix for the IV
estimator that is robust to the presence of het-
eroskedasticity:

Robust V (β̂IV ) = (X 0PZ X)−1 ×

(X 0Z(Z 0Z)−1(Z 0Ω̂Z)(Z 0Z)−1Z 0X)(X 0PZ X)−1
This is in fact the usual Eicker–Huber–White
“sandwich” robust variance–covariance matrix
for the IV estimator, available from ivreg or
ivreg2 with the robust option.

We may also deal with non–independence in

the error distribution by using a kernel esti-
mator to produce autocorrelation–consistent
standard errors. The “Newey–West” standard
errors are HAC, that is, heteroskedasticity- and
autocorrelation-consistent. If one does not doubt
the homoskedasticity assumption, but wants to
deal with autocorrelation, one should use the
“AC” correction without the “White” piece.
Thus, the latest ivreg2 allows selection of H,
AC, or HAC standard errors by combining the
robust, bandwidth and kernel options. One may
use the Bartlett kernel, as do Newey and West,
but a number of alternative kernel estimators
are available that will likewise produce a posi-
tive definite estimated covariance matrix. See
help ivreg2 for details.

Testing overidentifying restrictions in 2SLS and

GMM

The parameters of a structural equation are

said to be identified when we have sufficient
instruments to produce unique estimates from
the IV estimator. The identification status of
an equation is determined by two conditions.
When econometricians speak of the identifica-
tion of a structural equation, it is shorthand
for being able to uniquely identify the param-
eters in that equation. The first, the order
condition, requires that we count endogenous
regressors and available instruments. In order
to use the 2SLS estimator the matrix (Z 0X)
must be square and invertible. If Z is to have
k columns, we must have sufficient variables
to fill in the columns corresponding to each
endogenous regressor. The order condition
is necessary but not sufficient for identifica-
tion. The second, the rank condition, involves
the rank of the resulting instrument matrix
Z 0Z.That matrix must be of full rank `. The
rank condition is both necessary and sufficient
for identification. We first focus on the order
condition.

An exactly identified equation can be directly

estimated by instrumental variables. An overi-
dentified equation must be estimated with 2SLS,
while an underidentified equation cannot be es-
timated by any technique. Although it might
sound like overidentification is a nuisance to
be avoided, it is actually preferable to work-
ing with an exactly-identified equation. If the
equation is exactly identified, its point and in-
terval estimates may be quite dependent on
the specific instrument used. The estimates
might change considerably if a different valid
instrument was employed. Furthermore, re-
call that the first essential property of an in-
strument is statistical independence from the
disturbance process. Although we cannot test
the validity of that assumption directly, we can
assess the adequacy of instruments in an overi-
dentified context with a test of overidentifying
restrictions.

In such a test, the residuals from an instru-

mental variables regression are regressed on
all exogenous variables: both included exoge-
nous regressors and excluded instruments. Un-
der the null hypothesis that all instruments
are uncorrelated with u, a Lagrange Multiplier
(LM) statistic of the N × R2 form will not ex-
ceed the critical point on a χ2 (r) distribution,
where r is the number of overidentifying re-
strictions: the number of excess instruments.
If we reject this hypothesis, then we cast doubt
on the suitability of the instrument set. One
or more of the instruments do not appear to
be uncorrelated with the disturbance process.
This Sargan or Basmann test (Sargan (1958),
Basmann (1960)) is available in Stata as the
overid command (Baum, Schaffer, Stiillman
(2003)). This command can be installed from
ssc for use after estimation with ivreg.

Just as in the case of 2SLS the validity of the

overidentifying restrictions imposed on a GMM
estimator can be tested. The test, which can
and should be performed as a standard diag-
nostic in any overidentified instrumental vari-
ables model, has a null hypothesis of correct
model specification and the overidentifying re-
strictions. correct model specification and the
orthogonality conditions. A rejection calls ei-
ther or both of those hypotheses into question.

In the context of GMM, the overidentifying

restrictions may be tested by the commonly
employed J statistic of Hansen (1982). This
statistic is merely the value of the GMM ob-
jective function evaluated at the efficient GMM
estimator bEGM M . Under the null,
A
J(bEGM M ) = N g(b)0Ŝ −1g(b) ∼ χ2
`−k (5)
In the case of non-i.i.d. errors, the matrix Ŝ
is estimated using the two-step methods de-
scribed above and the J statistic becomes
A
J(bF EGM M ) = eZ 0(Z 0Ω̂Z)−1Ze0 ∼ χ2
`−k (6)
The J statistic is asymptotically distributed as
χ2 with degrees of freedom equal to the num-
ber of overidentifying restrictions ` − k rather
than the total number of moment conditions
`. In effect, k degrees of freedom are spent in
estimating the coefficients β. Hansen’s J is the
most common diagnostic utilized in GMM esti-
mation to evaluate the suitability of the model.
A rejection of the null hypothesis implies that
the instruments do not satisfy the orthogonal-
ity conditions required for their employment.
This may be either because they are not truly
exogenous, or because they are being incor-
rectly excluded from the regression. The J
statistic is calculated and displayed by ivreg2
when the gmm2s or robust options are specified.

Just as IV is a special case of GMM, Sargan’s

statistic is a special case of Hansen’s J under
the assumption of conditional homoskedastic-
ity. Thus if we use the IV optimal weighting
matrix together with the expression for J we
obtain
1
Sargan’s statistic = 2 û0Z(Z 0Z)−1Z 0û =
σ̂
û0Z(Z 0Z)−1Z 0û û0PZ û
0
= 0
û û/n û û/n

The Hansen–Sargan tests for overidentifica-

tion presented above evaluate the entire set
of overidentifying restrictions. In a model con-
taining a very large set of excluded instruments,
such a test may have very little power. Another
common problem arises when the researcher
has prior suspicions about the validity of a sub-
set of instruments, and wishes to test them.

In these contexts, a GMM distance or “difference–

in–Sargan” statistic, sometimes called the C
statistic, may usefully be employed.The C test
allows us to test a subset of the original set of
orthogonality conditions. The statistic is com-
puted as the difference between two Sargan
statistics (or, for efficient GMM, two J statis-
tics): that for the (restricted, fully efficient) re-
gression using the entire set of overidentifying
restrictions, versus that for the (unrestricted,
inefficient but consistent) regression using a
smaller set of restrictions, in which a specified
set of instruments are removed from the set.
For excluded instruments, this is equivalent to
dropping them from the instrument list. For
included instruments, the C test hypothecates
placing them in the list of included endoge-
nous variables: in essence, treating them as en-
dogenous regressors. The C test, distributed
χ2 with degrees of freedom equal to the loss
of overidentifying restrictions (i.e., the number
of suspect instruments being tested), has the
null hypothesis that the specified variables are
proper instruments.
Although the C statistic can be calculated as
the simple difference between the Hansen–Sargan
statistics for two regressions, this procedure
can generate a negative test statistic in finite
samples. In the IV context this problem can
be avoided and the C statistic guaranteed to
be non-negative if the estimate of the error
variance σ̂ 2 from the original (restricted, more
efficient) IV regression is used to calculate the
Sargan statistic for the unrestricted IV regres-
sion as well. The equivalent procedure in the
GMM context is to use the Ŝ matrix from the
original estimation to calculate both J statis-
tics. More precisely, Ŝ from the restricted esti-
mation is used to form the restricted J statis-
tic, and the submatrix of Ŝ with rows/columns
corresponding to the unrestricted estimation is
used to form the J statistic for the unrestricted
estimation.

The C test is conducted in ivreg2 by specifying

the endog option, and listing the endogenous
variables to be considered as potentially exoge-
nous (that is, now considered as instruments).
The equation as specified is compared with
an equation with a larger set of instruments,
augmented by those listed as endog(), which
imposes additional orthogonality conditions on
the estimation. The C test evaluates whether
those additional conditions are rejected by the
data. The Hansen–Sargan overidentification
test is an “omnibus” test for the failure of any
of the instruments to satisfy the orthogonality
conditions, but at the same time requires that
the investigator believe that at least some of
the instruments are valid.

Alternatively, a C test may be constructed by

specifying the orthog option, and listing the in-
struments (either included or excluded) to be
challenged. The equation must still be identi-
fied with these instruments either removed or
reconsidered as endogenous if the C statistic is
to be calculated. Note that if the unrestricted
equation is exactly identified, the Hansen–Sargan
statistic for the unrestricted equation will be
zero and the C statistic will coincide with the
Hansen–Sargan statistic for the original (re-
stricted) equation, and this will be true irre-
spective of the instruments used to identify the
unrestricted estimation.

Testing the relevance of instruments

An instrumental variable must not be corre-

lated with the equation’s disturbance process
and it must be highly correlated with the in-
cluded endogenous regressors. We may test
the latter condition by examining the fit of the
first stage regressions. The first stage regres-
sions are reduced form regressions of the en-
dogenous regressors X1 on the full set of in-
struments Z. The relevant test statistics here
relate to the explanatory power of the excluded
instruments Z1 in these regressions. A statistic
commonly used, as recommended by Bound,
Jaeger, Baker (1995) is the R2 of the first-
stage regression with the included instruments
partialled out. Alternatively, this may be ex-
pressed as the F -test of the joint significance
of the Z1 instruments in the first-stage regres-
sion. However, for models with multiple en-
dogenous variables, these indicators may not
be sufficiently informative.

To illustrate the pitfalls facing empirical re-

searchers here, consider the following simple
example. You have a model with two endoge-
nous regressors and two excluded instruments.
One of the two excluded instruments is highly
correlated with each of the two endogenous re-
gressors, but the other excluded instrument is
just noise. Your model is basically underiden-
tified. You have one valid instrument but two
endogenous regressors. The Bound et al. F -
statistics and partial R2 measures from the two
first-stage regressions will not reveal this weak-
ness. Indeed, the F -statistics will be statisti-
cally significant, and without further investiga-
tion you may not realize that the model cannot
be estimated in this form. To deal with this
problem of instrument irrelevance, either addi-
tional relevant instruments are needed or one
of the endogenous regressors must be dropped
from the model. The statistics proposed by
Bound et al. are able to diagnose instrument
relevance only in the presence of a single en-
dogenous regressor. When multiple endoge-
nous regressors are used, other statistics are
required.

One such statistic has been proposed by Shea

(1997): a partial R2 measure that takes the
intercorrelations among the instruments into
account.For a model containing a single en-
dogenous regressor, the two R2 measures are
equivalent. The distribution of Shea’s par-
tial R2 statistic has not been derived, but it
may be interpreted like any R2. As a rule of
thumb, if an estimated equation yields a large
value of the standard (Bound et al.) partial
R2 and a small value of the Shea measure, you
should conclude that the instruments lack suf-
ficient relevance to explain all the endogenous
regressors. Your model may be essentially un-
deridentified. The Bound et al. measures and
the Shea partial R2 statistic are provided by
the first or ffirst options of the ivreg2 com-
mand.

A more general approach to the problem of

instrument relevance was proposed by Ander-
son (1984) and discussed in Hall, Rudebusch,
Wilcox (1996). Anderson’s approach consid-
ers the canonical correlations of the X and Z
matrices. These measures, ri, i = 1, . . . . , k
represent the correlations between linear com-
binations of the k columns of X and linear com-
binations of the ` columns of Z. The squared
canonical correlations may be calculated as the
eigenvalues of (X 0X)−1(X 0Z)(Z 0Z)−1(Z 0X). If
an equation to be estimated by instrumental
variables is identified from a numerical stand-
point, all k of the canonical correlations must
be significantly different from zero. Ander-
son’s likelihood ratio test has the null hypoth-
esis that the smallest canonical correlation is
zero, and assumes that the regressors are dis-
tributed multivariate Normal. Under the null,
the test statistic is distributed χ2 with (`−k+1)
degrees of freedom, so that it may be calcu-
lated even for an exactly-identified equation.
A failure to reject the null hypothesis calls the
identification status of the estimated equation
into question. The Anderson statistic is dis-
played in ivreg2’s standard output.

The canonical correlations between X and Z

may also be used to test a set of instruments
for redundancy following Hall and Peixe (2000).
In an overidentified context with ` ≥ k, if some
of the instruments are redundant then the large-
sample efficiency of the estimation is not im-
proved by including them. The test statistic is
a likelihood ratio statistic based on the canon-
ical correlations with and without the instru-
ments being tested. Under the null hypothesis
that the specified instruments are redundant,
the statistic is distributed as χ2 with degrees
of freedom equal to the number of endoge-
nous regressors times the number of instru-
ments being tested. Like the Anderson test,
the redundancy test assumes that the regres-
sors are distributed multivariate Normal. This
test is available in ivreg2 with the redundant()
option.
Durbin–Wu–Hausman tests for endogeneity in
IV estimation

Many econometrics texts discuss the issue of

“OLS vs. IV” in the context of the Durbin–
Wu–Hausman (DWH) tests, which involve es-
timating the model via both OLS and IV ap-
proaches and comparing the resulting coeffi-
cient vectors. In the Hausman form of the test,
a quadratic form in the differences between the
two coefficient vectors, scaled by the precision
matrix, gives rise to a test statistic for the null
hypothesis that the OLS estimator is consis-
tent and fully efficient.

Denote by β̂ c the estimator that is consistent

under both the null and the alternative hy-
potheses, and by β̂ e the estimator that is fully
efficient under the null but inconsistent if the
null is not true. The Hausman (1978) specifi-
cation test takes the quadratic form
 H = n(β̂ c − β̂ e)0D−(β̂ c − β̂ e)
where  
c e
D = V (β̂ ) − V (β̂ )
(7)
and where V (β̂) denotes a consistent estimate
of the asymptotic variance of β, and the oper-
ator − denotes a generalized inverse.

A Hausman statistic for a test of endogeneity

in an IV regression is formed by choosing OLS
as the efficient estimator β̂ e and IV as the in-
efficient but consistent estimator β̂ c. The test
statistic is distributed as χ2 with K1 degrees
of freedom, this being the number of regres-
sors being tested for endogeneity. The test
is perhaps best interpreted not as a test for
the endogeneity or exogeneity of regressors per
se, but rather as a test of the consequence of
employing different estimation methods on the
same equation. Under the null hypothesis that
OLS is an appropriate estimation technique,
only efficiency should be lost by turning to IV;
the point estimates should be qualitatively un-
affected.

There are a variety of ways of conducting a

DWH endogeneity test in Stata for the stan-
dard IV case with conditional homoskedastic-
ity. Three equivalent ways of obtaining the
Durbin flavor of the Durbin–Wu–Hausman statis-
tic are:

1. Estimate the less efficient but consistent

model using IV, followed by the command
estimates store iv. Then estimate the fully
efficient model by OLS (or by IV if only a
subset of regressors is being tested for en-
dogeneity), followed by hausman, sigmamore.

2. Estimate the fully efficient model using ivreg2,

specifying the endogenous regressors to be
tested in the endog option.
3. Estimate the less efficient but consistent
model using ivreg, then use ivendog to con-
duct an endogeneity test. This program
will take as its argument a varlist consist-
ing of the subset of regressors to be tested
for endogeneity; if the varlist is empty, the
full set of endogenous regressors is tested.

The latter two methods are of course more

convenient than the first, as the test can be
done in one step.