Et

Econometric Theory
John Stachurski
May 22, 2014
Contents
Preface
Background Material
Probability
1.1
Probability Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2
Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
1.3
Dependence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
1.4
Asymptotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
1.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
1.6
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
Linear Algebra
50
2.1
Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
2.2
Span, Dimension and Independence . . . . . . . . . . . . . . . . . . .
60
2.3
Matrices and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
2.4
Random Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . .
72
2.5
Convergence of Random Matrices . . . . . . . . . . . . . . . . . . . .
75
2.6
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
80
2.7
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
81
CONTENTS
3
II
4
ii
Projections
86
3.1
Orthogonality and Projection . . . . . . . . . . . . . . . . . . . . . . .
86
3.2
Overdetermined Systems of Equations . . . . . . . . . . . . . . . . . .
92
3.3
Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
95
3.4
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
3.5
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Foundations of Statistics
Statistical Learning
109
110
4.1
Inductive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.2
Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.3
Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.4
Parametric vs Nonparametric Estimation . . . . . . . . . . . . . . . . 129
4.5
Empirical Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.6
Empirical Risk Minimization . . . . . . . . . . . . . . . . . . . . . . . 143
4.7
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4.8
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Methods of Inference
158
5.1
Making Inference about Theory . . . . . . . . . . . . . . . . . . . . . . 158
5.2
Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
5.3
Hypothesis Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5.4
Use and Misuse of Testing . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
5.6
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
CONTENTS
6
III
7
Linear Least Squares
iii
181
6.1
Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
6.2
Transformations of the Data . . . . . . . . . . . . . . . . . . . . . . . . 185
6.3
Goodness of Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.4
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
6.5
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Econometric Models
Classical OLS
201
202
7.1
The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
7.2
Variance and Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
7.3
The FWL Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
7.4
Normal Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
7.5
When the Assumptions Fail . . . . . . . . . . . . . . . . . . . . . . . . 222
7.6
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
7.7
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Time Series Models
233
8.1
Some Common Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
8.2
Dynamic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
8.3
Maximum Likelihood for Markov Processes . . . . . . . . . . . . . . . 256
8.4
Models with Latent Variables . . . . . . . . . . . . . . . . . . . . . . . 263
8.5
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
8.6
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
CONTENTS
9
Large Sample OLS
iv
273
9.1
Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
9.2
Asymptotic Normality . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
9.3
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
9.4
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
10 Further Topics
286
10.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286

10.2 Method of Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
10.3 Bayesian Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
10.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
10.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
IV
Appendices
305
11 Appendix A: Analysis
306
11.1 Sets and Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

11.2 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
11.3 Logical Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
Preface
This is a quick course on modern econometric and statistical theory, focusing on
fundamental ideas and general principles. The course is intended to be concise
suitable for learning conceptsrather than an encyclopedic treatment or a reference
manual. It includes a significant amount of foundational material in probability,
statistics and linear algebra that budding econometricians should know. The level is
aimed at the first year of graduate school, although I have taught the same material
successfully to undergraduates.
Most of the topics covered here are standard, and I have borrowed ideas and results from many sources. There are, however, some innovations in the material and
presentation. For example, some of the proofs concerning large sample theory are
newa slight remix old ideas. They have been developed to illuminate more clearly
the kinds of conditions that allow the law of large numbers and central limit theorem to function, and to make large sample results accessible to students without
knowledge of measure theory.
The other source of originality is in the presentation of otherwise standard material.
First, I have tried hard to unify the various concepts in the book and present the theory of econometrics as revolving around several core ideasrather than a jumble of
many separate methods. For example, the orthogonal projection theorem bestrides
much of our material, from least squares to conditional expectation. A great deal
can be learned from studying this elegant theorem and its many implications. Second, in my humble opinion, the mathematical arguments found here are sometimes
more precise and more carefully put together than some econometrics texts aimed
at a similar level.
Although it was originally written to teach from, there are many solved exercises
in the text, making it well suited to self-study. This also reflects my preferred style
of teaching for this kind of theoretical material: exercises and worked examples are
vital to reinforce ideas and deepen understanding. A lot of effort has been put in to
v
PREFACE
vi
providing a range of what I hope are fun and interesting exercises, along with their
solutions.
Finally, the course also teaches a bit of programming by providing sample code
in R. It goes without saying in this day and age that good programming skills are
essential for econometricseven for studing econometric theory. In fact one of the
best ways to understand a result in econometric theory is to first work your way
through the proof, and then run a simulation which shows the theory in action.
R is adopted because it is a common statistical language that does the job we need
. . . and its free. If you are not a fan of R thats fineyou will find it relatively easy
to translate these pieces of code into your favorate programming language. Either
way, in case it should prove helpful, a primer on programming in R can be found at
http://johnstachurski.net/personal/emet book.html. It will walk you through
the fundamentals of the language, graphics, elementary regressions and so on.
These notes have benefited from the input of many students. In particular, I wish to
thank Blair Alexander, Frank Cai, Yiyong Cai, Patrick Carvalho, Paul Kitney, Bikramaditya Datta, Alex Olssen, Stefan Webb and Varang Wiriyawit.
Part I
Background Material
Chapter 1
Probability
Probability theory forms the foundation stones of statistics and econometrics. If you
want to be a first class statistician/econometrician, then every extra detail of probability theory that you can grasp and internalize will prove an excellent investment.
1.1
Probability Models
We begin with the basic foundations of probability theory. What follows will involve
a few set operations, and you might like to glance over the results on set operations
(and the definition of functions) in 11.1.
1.1.1
Sample Spaces
In setting up a model of probability, we usually start with the notion of a sample

space, which, in general, can be any nonempty set, and is typically denoted . We
can think of as being the collection of all possible outcomes in a random experiment. A typical element of is denoted . The general idea is that a realization of
uncertainty will lead to the selection of a particular .
Example 1.1.1. Let := {1, . . . , 6} represent the six different faces of a dice. A
realization of uncertainty corresponds to a roll of the dice, with the outcome being
an integer in the set {1, . . . , 6}.
CHAPTER 1. PROBABILITY
The specification of all possible outcomes is one part of our probability model.
The other thing we need to do is to assign probabilities to outcomes. The obvious
thing to do here is to assign a probability to every in , but it turns out, for
technical reasons beyond the scope of this text (see, e.g., Williams, 1991), that is is
not the right way forward. Instead, the standard approach is to assign probabilities
to subsets of . In the language of probability theory, subsets of are called events.
The set of all events is usually denoted by F , and we follow this convention.1
Example 1.1.2. Let be any sample space. Two events we always find in F are
itself and the empty set . (The empty set is regarded as being a subset of every set,
and hence ). In this context, is called the certain event because it always
occurs (regardless of which outcome is selected, is true by definition). The
empty set is called the impossible event.
In all of what follows, if B is an event (i.e., B F ), then the notation P( B) will
represent the probability that event B occurs. The way you should think about it is
this:
P( B) represents the probability that when uncertainty is resolved and

some is selected by nature, the statement B is true.
Example 1.1.3. Continuing example 1.1.1, let B be the event {1, 2}. The number
P( B) represents the probability that the face selected by the roll is either 1 or 2.
The second stage of our model construction is to assign probabilities to elements
of F . In order to make sure our model of probability is well behaved, its best to
put certain restrictions on P. (For example, we wouldnt want to have a B with
P( B) = 93, as negative probabilities dont make much sense.) These restrictions
are imposed in the next definition:
A probability P on (, F ) is a function that associates to each event in F a number
in [0, 1], and, in addition, satisfies
1. P() = 1, and
2. Additivity: P( A B) = P( A) + P( B) whenever A, B F with A B = .
1 Im
skirting technical details here. In many common situations, we take F to be a proper subset
of the set of all subsets of . In particular, we exclude a few troublesome subsets of from F ,
because they are so messy that assigning probabilities to these sets cause problems for the theory.
See 1.1.2 for more discussion.
Together, the triple (, F , P) is called a probability space. It describes a set of

events and their probabilities for a given experiment. Confession: Ive simplified the
standard definition of a probability space slightlyin ways that will never matter to us in
this courseto avoid technical discussions that we dont need to go into. See 1.1.2 if youd
like to know more.
Remark 1.1.1. Additivity of P was defined for two sets, but this implies additivity
over any disjoint finite union. In particular, if A1 , . . . , A J are disjoint in the sense
that Ai A j = whenever i 6= j, then P( jJ=1 A j ) = jJ=1 P( A j ) also holds. See
exercise 1.6.1.
The axioms in the definition of P are fairly sensible. For starters, we are restricting
the probability of any event to be between zero and one. Second, it is clear why
we require P() = 1, since the realization will always be chosen from by
its definition. Third, the additivity property is natural: To find the probability of
a given event, we can determine all the different (i.e., disjoint) ways that the event
could occur, and then sum their probabilities.
Example 1.1.4. Continuing example 1.1.1, let := {1, . . . , 6} represent the six different faces of a dice, and, for A F , let
P( A) := #A/6, where #A := number of elements in A
(1.1)
For example, given this definition of P, we see that P{2, 4, 6} = 3/6 = 1/2. It is
simple to check that P is a probability on (, F ). Lets check additivity. Suppose
that A and B are two disjoint subsets of {1, . . . , 6}. In this case we must have #( A
B) = #A + #B, since, by disjointness, the number of elements in the union is just the
number contributed by A plus the number contributed by B. As a result,
P( A B) = #( A B)/6 = (#A + #B)/6 = #A/6 + #B/6 = P( A) + P( B)

The additivity property is very intuitive in this setting. For example, if we roll the
dice, the probability of getting an even number is the probability of getting a 2 plus
that of getting a 4 plus that of getting a 6. Formally,
P{2, 4, 6} = P[{2} {4} {6}]

= P{2} + P{4} + P{6} = 1/6 + 1/6 + 1/6 = 1/2
To finish off the proof that P is a probability on (, F ), it only remains to check that
0 P( B) 1 for any B F , and that P() = 1. These details are left to you.
Example 1.1.5. Consider a memory chip in a computer, made up of billions of tiny

switches. Imagine that a random number generator accesses a subset of N switches,
setting each one to on or off at random. One sample space for this experiment
is
0 := {(b1 , . . . , b N ) : bn {on, off} for each n}
Letting zero represent off and one represent on, we can also use the more practical
space
:= {(b1 , . . . , b N ) : bn {0, 1} for each n}
Thus, is the set of all binary sequences of length N. As our probability, we define
P( A) := 2 N (#A)
To see that this is indeed a probability on (, F ) we need to check that 0 P( A) 1
for all A , that P() = 1, and that P is additive. Exercise 1.6.7 asks you to
confirm that P is additive. That P() = 1 follows from the fact that the number of
binary sequences of length N is 2 N .
Now lets go back to the general case, where (, F , P) is an arbitrary probability
space. From the axioms above, we can derive a suprising number of properties.
Lets list the key ones, starting with the next fact.
Fact 1.1.1. Let (, F , P) be a probability space, and let A, B F . If A B, then
1.
2.
3.
4.
P( B \ A ) = P( B ) P( A );
P( A ) P( B );
P( Ac ) := P( \ A) = 1 P( A); and
P() = 0.
These claims are not hard to prove. For example, regarding the part 1, if A B,
then we have B = ( B \ A) A. (Sketching the Venn diagram will help confirm this
equality in your mind.) Since B \ A and A are disjoint, additivity of P now gives
P( B ) = P( B \ A ) + P( A )
(whenever A B)
This equality implies parts 14 of fact 1.1.1. Rearranging gives part 1, while nonnegativity of P gives part 2. Specializing to B = gives part 3, and setting B = A gives
part 4.
The property that if A B, then P( A) P( B) is called monotonicity, and is
fundamental. If A B, then we know that B occurs whenever A occurs (because if
lands in A, then it also lands in B). Hence, the probability of B should be larger.
Many crucial ideas in probability boil down to this one point.
Fact 1.1.2. If A and B are any (not necessarily disjoint) events, then
P( A B ) = P( A ) + P( B ) P( A B )
In particular, for any A, B F , we have P( A B) P( A) + P( B).
If A and B are events, then the conditional probability of A given B is
P( A | B ) : = P( A B ) /P( B )
(1.2)
It represents the probability that A will occur, given the information that B has occurred. For the definition to make sense, it requires that P( B) > 0. Events A and B
are called independent if P( A B) = P( A)P( B). If A and B are independent, then
the conditional probability of A given B is just the probability of A.
Example 1.1.6. Consider an experiment where we roll a dice twice. A suitable sample space is the set of pairs (i, j), where i and j are between 1 and 6. The first element
i represents the outcome of the first roll, while the second element j represents the
outcome of the second roll. Formally,
:= {(i, j) : i, j {1, . . . , 6}}
For our probability, lets define P( E) := #E/36, where #E is the number of elements
in E . (In this case, elements are pairs, so #E is the number of pairs in E.) Now
consider the events
A := {(i, j) : i is even}
and
B := {(i, j) : j is even}
In this case we have

A B = {(i, j) : i and j are even}
With a bit of work we can verify that P( A B) = P( A)P( B), indicating that A and B
are independent under the probability P. To check this, we need to be able to count
the number of elements in A, B and A B. The basic principle for counting ordered
tuples is that the total number of possible tuples is the product of the number of
possibilities for each tuple. For example, the number of distinct tuples
(i, j, k) where i I, j J and k K

is (#I ) (#J ) (#K ). Hence, the number of elements in A is 3 6 = 18, the number
of elements in B is 6 3 = 18, and the number of elements in A B is 3 3 = 9. As
a result,
P( A B) = 9/36 = 1/4 = (18/36) (18/36) = P( A)P( B)

Thus, A and B are independent, as claimed.
A very useful result is the law of total probability, which says that if A F and
B1 , . . . , B M is a partition of (i.e., Bm F for each m, the Bm s are mutually disjoint
M B = ) with P( B ) > 0
in the sense that Bj Bk is empty when j 6= k, and m
m
=1 m
for all m, then
M
P( A) = P( A | Bm ) P( Bm )
m =1
The proof is quite straightforward, although you should check that the manipulations of intersections and unions work if you have not seen them before:
P( A) = P[ A (mM=1 Bm )] = P[mM=1 ( A Bm )]
=
m =1
m =1
P( A Bm ) = P( A | Bm ) P( Bm )
Example 1.1.7. Heres an informal example of the law of total probability: Suppose
I flip a coin to decide whether to take part in a poker game. Being a bad player,
the chance of losing money when I play is 2/3. The overall chance of losing money
(LM) that evening is
P{LM } = P{LM | play}P{play} + P{LM | dont play}P{dont play}

which is (2/3) (1/2) + 0 (1/2) = 1/3.
1.1.2
Technical Details
Confession time: As alluded to above, in my presentation of probability spaces Ive

swept some technical details under the carpet to make the presentation smooth.
These details wont affect anything that follows, and this whole course can be completed successfully without knowing anything about them. Hence you can skip this
section on first reading. Nevertheless, if you intend to keep going deeper into probability and statistics, eventually you will have to work your way through them. So
lets note them for the record.
First, it turns out that assigning probabilities to all subsets of an arbitrary set
in a consistent way can be quite a difficult task. The reason is that if is relatively
largea continuum, saythen it contains an awful lot of subsets, and some of them
can be manipulated to exhibit very strange phenomena. (Look up the Banach-Tarski
paradox for a hint of what I mean.) Because of this we usually take our set of events
F to be a sub-collection of the subsets of , and only assign probabilities to elements
of F . Thus, the first stage of our model construction is to choose (i) a sample space
, and (ii) a collection of its subsets F that we want to assign probabilities to.
When we choose F , usually we dont just choose freely, because doing so will make
it hard to form a consistent theory. One restriction we always put on F is to require
that it contains the empty set and the whole set . (In the definition of P, we
require that P() = 1. Hence we need to be an element of F , the events we
assign probability to.)
Another sensible restriction concerns complements. For example, lets suppose that
A F , so that P( A) is well defined, and represents the probability of event A.
Now, given that we can assign a probability to the event A, it would be a bit unfortunate if we couldnt assign a probability to the event not A, which corresponds
to Ac . So normally we require that if A F , then Ac F . When this is true, we say
that F is closed under the taking of complements.
Also, lets suppose that A and B are both in F , so we assign probabilities to these
events. In this case, it would be natural to think about the probability of the event
A and B, which corresponds to A B. So we also require that if A and B are in F ,
then A B is also in F . We say that F is closed under the taking of intersections.
Perhaps we should also require that if A and B are in F , then A B is also in F ?
Actually, we dont have to, because (see fact 11.1.1 on page 308),
A B = ( Ac Bc )c
Thus, if F is closed under the taking of complements and intersections, then F is
automatically closed under the taking of unions.
There is one more restriction thats typically placed on F , which is the property
of being closed under countable unions. This just means that if A1 , A2 , . . . is a
sequence of sets in F , then its union is likewise in F . Since the details dont matter here we wont discuss it further. Suffice to say that when F satisfies all these
properties, it is called a -algebra.
Finally, in standard probability theory, there is another restriction placed on P that
I have not mentioned, called countable additivity. The definition of countable additivity is that if A1 , A2 , . . . is a disjoint sequence of sets in F (disjoint means that
Ai A j = for any i 6= j), then
P(i Ai ) := P{ : Ai for some i} = P( Ai )

i =1
Why strengthen additivity to countable additivity? Countable additivity works behind the scenes to make probability theory run smoothly (expectations operators are
suitably continuous, and so on). None of these details will concern us in this course.
If you wish, you can learn all about -algebras and countable additivity in any text
on measure theory. There are many beautiful books on this subject. One of my
favorites at the introductory level is Williams (1991).
1.1.3
Random Variables
If youve done an elementary probability course, you might have been taught that a
random variable is a value that changes randomly, or something to that effect. To
work more deeply with these objects, however, we need a sounder definition. For
this reason, mathematicians define random variables to be functions from into R.
Thus, in this formal model, random variables convert outcomes in sample space into numerical outcomes. This is useful, because numerical outcomes are easy to manipulate
and interpret.2
To visualize the definition, consider a random variable x, and imagine that nature
picks out an element in according to some probability. The random variable
now sends this into x ( ) R. In terms of example 1.1.5, a random number generator picks out an , which is a binary sequence. A random variable x converts this
sequence into a real number. Depending on the conversion rule (i.e., depending on
the definition of x), the outcome x ( ) might simulate a Bernoulli random variable,
a uniform random variable, a normal random variable, etc.
Example 1.1.8. Recall example 1.1.5, with sample space
:= {(b1 , . . . , b N ) : bn {0, 1} for each n}
The set of events and probability were defined as follows:
F := all subsets of and P( A) := 2 N (#A)

skipping some technical details again. The definition of random variables on infinite
is actually a bit more subtle. In practice, when identifying random variables with the class of all
functions from to R, we typically exclude some particularly nasty (i.e., complicated and generally
badly behaved) functions. The remaining nice functions (called measurable functions) are our
random variables. In this course we will never meet the nasty functions, and theres no need to go
into further details. Those who want to know more should consult any text on measure theory (e.g.,
Williams, 1991).
2 Im
10
Consider a random variable x on that returns the first element of any given sequence. That is,
x ( ) = x (b1 , . . . , b N ) = b1
Then x is a Bernoulli random variable (i.e., x takes only the values zero and one).
The probability that x = 1 is 1/2. Indeed,
P{ x = 1} : = P{ : x ( ) = 1}
= P{(b1 , . . . , b N ) : b1 = 1}
= 2 N #{(b1 , . . . , b N ) : b1 = 1}
The number of length N binary sequences with b1 = 1 is 2 N 1 , so P{ x = 1} = 1/2.
Example 1.1.9. Consider the sample space
:= {(b1 , b2 , . . .) : bn {0, 1} for each n}
is called the set of all infinite binary sequences. (This is an infinite version of
the sample space in example 1.1.5. Imagine a computer with an infinite amount of
memory.) Consider an experiment where we flip a coin until we get a heads. We
let 0 represent tails and 1 represent heads. The experiment of flipping until we get a
heads can be modeled with the random variable
x ( ) = x (b1 , b2 , . . .) = min{n : bn = 1}
As per the definition, x is a well-defined function from into R.3
Lets go back to the general case, with arbitrary probability space (, F , P), and talk
a bit more about Bernoulli (i.e., binary) random variables. There is a generic way to
create Bernoulli random variables, using indicator functions. If Q is a statement,
such as on the planet Uranus, there exists a tribe of three-headed monkeys, then
1{Q} is considered as equal to one when the statement Q is true, and zero when the
statement Q is false. Hence, 1{ Q} is a binary indicator of the truth of the statement
Q.
In general (in fact always), a Bernoulli random variable has the form
x ( ) = 1{ C }
3 Actually,
thats not strictly true. What if = 0 is an infinite sequence containing only zeros?
Then {n : bn = 1} = . The convention here is to set x (0 ) = min{n : bn = 1} = min = . But
then x is not a map into R, because it can take the value . However, it turns out that this event has
probability zero, and hence we can ignore it. For example, we can set x (0 ) = 0 without changing
anything significant. Now were back to a well-defined function from to R.
11
Figure 1.1: Simple function x ( ) = s1{ A} + t1{ B}
where C is some subset of . Thus, x is a binary random variable indicating whether

or not the event C occurs (one means yes and zero means no).
From Bernoulli random variables we can create discrete random variables. A discrete random variable is a random variable with finite range.4 We can create discrete random variables by taking linear combinations of Bernoulli random variables. (A linear combination of certain elements is formed by multiplying these
elements by scalars and then summing the result.) For example, let A and B be
disjoint subsets of . The random variable
x ( ) = s1{ A } + t1{ B }
(1.3)
is a discrete random variable taking the value s when falls in A, t when falls in
B, and zero otherwise. (Check it!) Figure 1.1 shows a graph of x when = R.
It turns out that any discrete random variable can be created by taking linear combinations of Bernoulli random variables. In particular, we can also define a discrete
random variable as a random variable having the form
J
x ( ) =
s j 1{ A j }
(1.4)
j =1
4 See
11.1.1 for the definition of range. Our usage is not entirely standard, in that many texts
call random variables with countably infinite range discrete as well.
12
We will work with this expression quite a lot. In doing so, we will always assume
that
the scalars s1 , . . . , s J are distinct, and
the sets A1 , . . . , A J form partition of .5
Given these assumptions, we then have
x ( ) = s j if and only if A j .
{x = sj } = Aj.
P{ x = s j } = P( A j ).
The second two statements follow from the first. Convince yourself of these results
before continuing.
Before finishing this section, lets clarify a common notational convention with random variables that weve adopted above and that will be used below. With a random variable x, we often write
{ x has some property}

as a shorthand for
{ : x ( ) has some property}

Well follow this convention, but you should translated it backwards and forwards
in your mind if youre not yet familiar with the notation. To give you an example,
consider the claim that, for any random variable x,
P{ x a} P{ x b} whenever a b
(1.5)
This is intuitively obvious. The mathematical argument goes as follows: Observe

that
{ x a} := { : x ( ) a} { : x ( ) b} =: { x b}
(The inclusion must hold, because if is such that x ( ) a, then, since a b,
we also have x ( ) b. Hence any in the left-hand side is also in the right-hand
side.) The result in (1.5) now follows from monotonicity of P (fact 1.1.1 on page 5).
5 That
is, Ai A j = when i 6= j, and j A j = .
1.1.4
13
Expectations
Our next task is to define expectations for an arbitrary random variable x on probability space (, F , P). Roughly speaking, E [ x ] is defined as the sum of all possible values of x, weighted by their probabilities. (Here sum is in quotes because
there may be an infinite number of possibilities.) The expectation E [ x ] of x also represents the average value of x over a very large sample. (This is a theorem, not
a definitionsee the law of large numbers below.)
Lets start with the definition when x is a discrete random variable. Let x be the
discrete random variable x ( ) = jJ=1 s j 1{ A j }, as defined in (1.4). In this case,
the expectation of x is defined as
J
E [ x ] : = s j P( A j )
(1.6)
j =1
The definition is completely intuitive: For this x, given our assumption that the sets
A j s are a partition of and the s j s are distinct, we have
A j = { x = s j } := { : x ( ) = s j }
Hence, (1.6) tells us that
J
E [ x ] = s j P{ x = s j }
j =1
Thus, the expectation is the sum of the different values that x may take, weighted
by their probabilities.
How about arbitrary random variables, with possibly infinite range? Unfortunately,
the full definition of expectation for these random variables involves measure theory, and we cant treat in in detail. But the short story is that any arbitrary random
variable x can be approximated by a sequence of discrete variables xn . The expectation of discrete random variables was defined in (1.6). The expectation of the limit x
is then defined as
E [ x] := lim E [ xn ]
(1.7)
n
When things are done carefully (details omitted), this value doesnt depend on the
particular approximating sequence { xn }, and hence the value E [ x ] is well defined.
Lets list some facts about expectation, and then discuss them one by one.
14
Fact 1.1.3. Expectation of indicators equals probability of event: For any A F ,
E [1{ A}] = P( A)
(1.8)
Fact 1.1.4. Expectation of a constant is the constant: If R, then E [] = .

Fact 1.1.5. Linearity: If x and y are random variables and and are constants, then
E [x + y] = E [ x] + E [y]
Fact 1.1.6. Monotonicity: If x, y are random variables and x y,6 then E [ x ] E [y].
Fact 1.1.3 follows from the definition in (1.6). We have
1{ A } = 1 1{ A } + 0 1{ A c }
Applying (1.6), we get
E [1{ A}] = 1 P( A) + 0 P( Ac ) = P( A)
Fact 1.1.4 say that the expectation of a constant is just the value of the constant.
The idea here is that the constant should be understood in this context as the
constant random variable 1{ }. From our definition (1.6), the expectation of
this constant is indeed equal to its value :
E [] := E [1{ }] = P() =
Now lets think about linearity (fact 1.1.5). The way this is proved, is to first prove
linearity for discrete random variables, and then extend the proof to arbitrary random variables via (1.7). Well omit the last step, which involves measure theory.
Well also omit the full proof for discrete random variables, since its rather long.
Instead, lets cover a quick sketch of the argument that still provides most of the
intuition.
Suppose we take the random variable x in (1.4) and double it, producing the new
random variable y = 2x. More precisely, for each , we set y( ) = 2x ( ). (Whatever happens with x, were going to double it and y will return that value.) In that
case, we have E [y] = 2E [ x ]. To see this, observe that
"
#
J
y( ) = 2x ( ) = 2
s j 1{ A j }
j =1
6 The
2s j 1{ A j }
j =1
statement x y means that x is less than y for any realization of uncertainty. Formally, it
means that x ( ) y( ) for all .
15
Hence, applying (1.6),

"
E [y] = 2s j P( A j ) = 2 s j P( A j ) = 2E [ x]
j =1
j =1
What we have shown is that E [2x ] = 2E [ x ]. Looking back over our argument, we
can see that there is nothing special about the number 2 herewe could have used
any constant. In other words,
For any constant , we have E [x ] = E [ x ]
Another aspect of linearity of expectations is additivity, which says that given random variables x and y, the statement E [ x + y] = E [ x ] + E [y] is always true. Instead
of giving the full proof, lets show this for the Bernoulli random variables
x ( ) = 1{ A }
and
y ( ) = 1{ B }
(1.9)
Consider the sum x + y. By this, we mean the random variable z( ) = x ( ) + y( ).

More succinctly, ( x + y)( ) := x ( ) + y( ). We claim that E [ x + y] = E [ x ] + E [y].
To see that this is the case, note first that
( x + y)( ) = 1{ A \ B} + 1{ B \ A} + 21{ A B}
(To check this, just go through the different cases for , and verify that the right
hand side of this expression agrees with x ( ) + y( ). Sketching a Venn diagram
will help.) Therefore, by the definition of expectation,
E [ x + y ] = P ( A \ B ) + P ( B \ A ) + 2P ( A B )
(1.10)
Now observe that

A = ( A \ B) ( A B)
It follows (why?) that
E [ x ] : = P( A ) = P( A \ B ) + P( A B )
Performing a similar calculation with y produces
E [ y ] : = P( B ) = P( B \ A ) + P( A B )
Adding these two produces the value on the right-hand side of (1.10), and we have
now confirmed that E [ x + y] = E [ x ] + E [y].
16
0.0
0.2
0.4
F(s)
0.6
0.8
1.0
15
10
10
15
Figure 1.2: Cauchy cdf
1.2
Distributions
All random variables have distributions. Distributions summarize the probabilities

of different outcomes for the random variable in question, and allow us to compute
expectations. In this section, we describe the link between random variables and
distributions.
1.2.1
CDFs
A cumulative distribution function (cdf) on R is a right-continuous, monotone increasing function F : R [0, 1] satisfying lims F (s) = 0 and lims F (s) = 1.
(F is monotone increasing if F (s) F (s0 ) whenever s s0 , and right continuous if
F (sn ) F (s) whenever sn s.)
Example 1.2.1. The function F (s) = arctan(s)/ + 1/2 is a cdfone variant of the
Cauchy distribution. A plot is given in figure 1.2.
17
Let x be a random variable on some probability space (, F , P), and consider the
function
Fx (s) := P{ x s} := P{ : x ( ) s}
( s R)
(1.11)
It turns out that this function is always a cdf.7 We say that Fx is the cdf of x, or,
alternatively, that Fx is the distribution of x, and write x Fx .
We wont go through the proof that the function Fx defined by (1.11) is a cdf. Note
however that monotonicity is immediate from (1.5) on page 12.
Fact 1.2.1. If x F, then P{ a < x b} = F (b) F ( a) for any a b.
Proof: If a b, then { a < x b} = { x b} \ { x a} and { x a} { x b}.
Applying fact 1.1.1 on page 5 gives the desired result.
A cdf F is called symmetric if F (s) = 1 F (s) for all s R.8 The proof of the next
fact is an exercise (exercise 1.6.12).
Fact 1.2.2. Let F be a cdf and let x F. If F is symmetric and P{ x = s} = 0 for all
s R, then the distribution F| x| of the absolute value | x | is given by
F| x| (s) := P{| x | s} = 2F (s) 1
1.2.2
( s 0)
Densities and Probability Mass Functions
Cdfs are important because every random variable has a well-defined cdf via (1.11).
However, they can be awkward to manipulate mathematically, and plotting cdfs
is not a very good way to convey information about probabilities. For example,
consider figure 1.2. The amount of probability mass in different regions of the xaxis is determined by the slope of the cdf. Research shows that humans are poor at
extracting quantitative information from slopes. They do much better with heights,
which leads us into our discussion of densities and probability mass functions.
Densities and probability mass functions correspond to two different, mutually exclusive cases. The first (density) case arises when the increase of the cdf in question is smooth, and contains no jumps. The second (probability mass function) case
following is also true: For every cdf F, there exists a probability space (, F , P) and a random variable x : R such that the distribution of x is F. Exercise 1.6.14 gives some hints on how
the construction works.
8 Thus, the probability that x s is equal to the probability that x > s. Centered normal distributions and t-distributions have this property.
7 The
18
arises when the increase consists of jumps alone. Lets have a look at these two
situations, starting with the second case.
The pure jump case occurs when the cdf represents a discrete random variable. To
understand this, suppose that x takes values s1 , . . . , s J . Let p j := P{ x = s j }. We then
have 0 p j 1 for each j, and jJ=1 p j = 1 (exercise 1.6.13). A finite collection of
numbers p1 , . . . , p J such that 0 p j 1 and p1 + + p J = 1 is called a probability
mass function (pmf). The cdf corresponding to this random variable is
J
Fx (s) =
1{ s j s } p j
(1.12)
j =1
How do we arrive at this expression? Because, for this random variable,
J
P{ x s } = P
{ x = s j } = P{ x = s j } = 1{ s j s } p j
j s.t. s s
j =1
j s.t. s j s
Visually, Fx is a step function, with a jump up of size p j at point s j . Figure 1.3 gives
an example with J = 2.
The other case of interest is the density case. A density is a nonnegative function
p that integrates to 1. For example, suppose that F is a smooth cdf, so that the
derivative F 0 exists. Let p := F 0 . By the fundamental theorem of calculus, we then
have
Z s
Z s
p(t)dt =
F 0 (t)dt = F (s) F (r )
r
R +
From the definition of cdfs, we can see that p is nonnegative and p(s)ds = 1.
In other words, p is a density. Also, taking the limit as r we obtain
F (s) =
Z s
p(t)dt
which tells us that F can be recovered from p.

More generally, if F is the cdf of random variable x and p : R [0, ) satisfies
F (s) =
Z s
p(t)dt
for all s R
then p is called the density of x.

Not every random variable has a density. The exact necessary and sufficient condition
for a density to exist is that F is absolutely continuous. The most important special
19
s1
s2
0.15
0.10
0.05
0.00
p(s)
0.20
0.25
0.30
Figure 1.3: Discrete cdf
15
10
10
Figure 1.4: Cauchy density
15
20
case of absolute continuity is when F is differentiable. (See any text on measure

theory for details.) For our purposes, you can think of absolute continuity meaning
that F does not have any jumps. In particular, discrete random variables are out, as
is any random variable putting positive probability mass on a single point.9
Fact 1.2.3. If x has a density, then P{ x = s} = 0 for all s R.
As discussed above, cdfs are useful because every random variable has one, but
pmfs and densities are nicer to work with, and visually more informative. For example, consider figure 1.4, which shows the density corresponding to the Cauchy
cdf in figure 1.2. Information about probability mass is now conveyed by height
rather than slope, which is easier for us humans to digest.
1.2.3
The Quantile Function
Let F be any cdf on R. Suppose that F is strictly increasing, so that the inverse
function F 1 exists:
F 1 (q) := the unique s such that F (s) = q
(0 < q < 1)
(1.13)
The inverse of the cdf is called the quantile function, and has many applications in
probability and statistics.
Example 1.2.2. The quantile function associated with the Cauchy cdf in example 1.2.1
is F 1 (q) = tan[ (q 1/2)]. See figure 1.5.
Things are a bit more complicated when F is not strictly increasing, as the inverse
F 1 is not well defined. (If F is not strictly increasing, then there exists at least two
distinct points s and s0 such that F (s) = F (s0 ).) This problem is negotiated by setting
F 1 (q) := inf{s R : F (s) q}
(0 < q < 1)
This expression is a bit more complicated, but in the case where F is strictly increasing, it reduces to (1.13).
The value F 1 (1/2) is called the median of F.
9 In
elementary texts, random variables with densities are often called continuous random variables. The notation isnt great, because continuous here has nothing to do with the usual definition of continuity of functions.
21
0
5
0.2
0.4
0.6
0.8
Figure 1.5: Cauchy quantile function
The quantile function features in hypothesis testing, where it can be used to define
critical values (see 5.3). An abstract version of the problem is as follows: Let x F,
where F is strictly increasing, differentiable (so that a density exists and x puts no
probability mass on any one point) and symmetric. Given (0, 1), we want to
find the c such that P{c x c} = 1 (see figure 1.6). The solution is given
by c := F 1 (1 /2). That is,
c = F 1 (1 /2) = P{| x | c} = 1
(1.14)
To see this, fix (0, 1). From fact 1.2.2, we have
P{| x| c} = 2F(c) 1 = 2F[ F1 (1 /2)] 1 = 1

In the case where F is the standard normal cdf , this value c is usually denoted by
z/2 . We will adopt the same notation:
z/2 := 1 (1 /2)
(1.15)
22
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Figure 1.6: Finding critical values
1.2.4
Expectations from Distributions
Until now, weve been calculating expectations using the expectation operator E ,
which was defined from a given probability P in 1.1.4. One of the most useful
facts about distributions is that one need not know about x or E to calculate E [ x ]
knowledge of the distribution of x is sufficient.
Mathematically, this is an interesting topic, and a proper treatment requires measure
theory (see, e.g., Williams, 1991). Here Ill just tell you what you need to know for
the course, and then follow that up with a bit of intuition.
In all of what follows, h is an arbitrary function from R to R.
Fact 1.2.4. If x is a discrete random variable taking values s1 , . . . , s J with probabilities
p1 , . . . , p J , then
J
E [h( x)] = h(s j ) p j
(1.16)
j =1
Fact 1.2.5. If x is a continuous random variable with density p, then
E [h( x)] =
h(s) p(s)ds
(1.17)
23
Its convenient to have a piece of notation that captures both of these cases. As a
result, if x F, then we will write
E [h( x)] =
h(s) F (ds)
The way you should understand this expression is that when F is differentiable with
R
R
derivative p = F 0 , then h(s) F (ds) is defined as h(s) p(s)ds. If, on the other
hand, F is the step function F (s) = jJ=1 1{s j s} p j corresponding to the discrete
R
random variable in fact 1.2.4, then h(s) F (ds) is defined as jJ=1 h(s j ) p j .
Example 1.2.3. Suppose that h( x ) = x2 . In this case, E [h( x )] is the second moment
of x. If we know the density p of x, then fact 1.2.5 can be used to evaluate that second
moment by solving the integral on the right-hand side of (1.17).
Just for the record, let me note that if you learn measure theory you will come to
R
understand that, for a given cdf F, the expression h(s) F (ds) has its own precise
definition, as the Lebesgue-Stieltjes integral of h with respect to F. In the speR
cial case where F is differentiable with p = F 0 , one can prove that h(s) F (ds) =
R
h ( s ) p ( s ) ds, where the left hand side is the Lebesgue-Stieltjes integral, and the
right hand side is the ordinary (Riemann) integral you learned in high school. An
similar statement holds for the discrete case. However, this is not the right place
for a full presentation of the Lebesgue-Stieltjes integral. We want to move on to
statistics.
Although were skipping a lot of technical details here, we can at least prove fact 1.2.4.
This is the discrete case, where x is of the form x ( ) = jJ=1 s j 1{ A j }, and
p j := P{ x = s j } = P( A j ). As usual, the values {s j } are distinct and the sets { A j }
are a partition of . As we saw in 1.1.4, the expectation is
J
j =1
j =1
j =1
E [ x ] = s j P( A j ) = s j P{ x = s j } = s j p j
Now let h : R R. You should be able to convince yourself that
J
h( x ( )) =
h ( s j )1{ A j }
j =1
(Pick an arbitrary A j and check that the left- and right-hand sides are equal when
A j .) This is a discrete random variable, which we can take the expectation of
24
using (1.6) (page 13). We get

J
j =1
j =1
E [h( x)] = h(s j )P( A j ) = h(s j ) p j

Fact 1.2.4 is now confirmed.
1.2.5
Common Distributions
Lets list a few well-known distributions that will come up in this course.
Let a < b. The uniform distribution on interval [ a, b] is the distribution associated
with the density
1
( a s b)
p(s; a, b) :=
ba
(If s < a or s > b, then p(s; a, b) := 0.) The mean is
Z b
a
s p(s; a, b)ds =
a+b
2
The univariate normal density or Gaussian density is a function p of the form

1
2 1/2
2 2
p(s) := p(s; , ) := (2 )
exp (s )
2
for some R and > 0. We represent this distribution symbolically by N (, 2 ).
The distribution N (0, 1) is called the standard normal distribution
It is well-known that if x N (, 2 ), then E [ x ] = , and var[ x ] = 2 . Hence the
two parameters separately define the mean and the variance (or standard deviation),
and this is one of many attractive features of the distribution.
Fact 1.2.6. If x1 , . . . , x N are normally distributed and 0 , . . . , N are any constants,
then 0 + nN=1 n xn is also normally distributed.
The chi-squared distribution with k degrees of freedom is the distribution with
density
1
p(s; k ) := k/2
sk/21 es/2
( s 0)
2 (k/2)
where is the Gamma function (details omitted). If x has a distribution described
by this density, then we write x 2 (k ).
25
Students t-distribution with k degrees of freedom, or, more simply, the t-distribution
with k degrees of freedom, is the distribution on R with density
p(s; k ) :=
( k+2 1 )
(k )1/2 ( 2k )
s2
1+
k
(k+1)/2
The F-distribution with parameters k1 , k2 is the distribution with the unlikely looking density
q
(k1 s)k1 kk22 /[k1 s + kk21 +k2 ]
p(s; k1 , k2 ) :=
( s 0)
sB(k1 /2, k2 /2)
where B is the Beta function (details omitted). The F-distribution arises in certain
hypothesis tests, some of which we will examine later.
1.3
Dependence
[roadmap]
1.3.1
Joint Distributions
Consider a collection of N random variables x1 , . . . , x N . For each individual random

variable xn : R, the distribution Fn of xn is
Fn (s) := P{ xn s}
( < s < )
(1.18)
This distribution tells us about the random properties of xn viewed as a single entity.
But we often want to know about the relationships between the variables x1 , . . . , x N ,
and outcomes for the group of variables as a whole. To quantify these things, we
define the joint distribution of x1 , . . . , x N to be
F ( s1 , . . . , s N ) : = P{ x1 s1 , . . . , x N s N }
( < sn < ; n = 1, . . . , N )
In this setting, the distribution Fn of xn is sometimes called the marginal distribution, in order to distinguish it from the joint distribution.
The joint density of x1 , . . . , x N , if it exists, is a function p : R N [0, ) satisfying
Z tN
Z t1
p(s1 , . . . , s N )ds1 ds N = F (t1 , . . . , t N )
(1.19)
26
for all tn R, n = 1, . . . , N.
Typically, the joint distribution cannot be determined from the N marginal distributions alone, since the marginals do not tell us about the interactions between the
different variables. Once special case where we can tell the joint from the marginals
is when there is no interaction. This is called independence, and we treat it in the
next section.
From joint densities we can construct conditional densities. The conditional density
of xk+1 , . . . , x N given x1 = s1 , . . . , xk = sk is defined by
p ( s k +1 , . . . , s N | s 1 , . . . , s k ) : =
p ( s1 , . . . , s N )
p ( s1 , . . . , s k )
(1.20)
Rearranging this expression we obtain a decomposition of the joint density:

p ( s 1 , . . . , s N ) = p ( s k +1 , . . . , s N | s 1 , . . . , s k ) p ( s 1 , . . . , s k )
(1.21)
This decomposition is useful in many situations.
1.3.2
Independence
Let x1 , . . . , x N be a collection of random variables with xn Fn , where Fn is a cdf.

The variables x1 , . . . , x N are called identically distributed Fn = Fm for all n, m. They
are called independent if, given any s1 , . . . , s N , we have
P{ x1 s1 , . . . , x N s N } = P{ x1 s1 } P{ x N s N }
(1.22)
Equivalently, if F is the joint distribution of x1 , . . . , x N and Fn is the marginal distribution of xn , then independence states that
N
F (s1 , . . . , s N ) = F1 (s1 ) FN (s N ) =
Fn (sn )
n =1
We use the abbreviation IID for collections of random variables that are both independent and identically distributed.
Example 1.3.1. Consider a monkey throwing darts at a dartboard. Let x denote the
horizontal location of the dart relative to the center of the board, and let y denote
the vertical location. (For example, if x = 1 and y = 3, then the dart is 1cm to the
left of the center, and 3cm above.) At first pass, we might suppose that x and y are
independent and identically distributed.
27
Fact 1.3.1. If x1 , . . . , x M are independent and E | xm | is finite for each m, then

"
#
M
m =1
m =1
E xm = E [ xm ]
We wont prove the last fact in the general case, as this involves measure theory.
However, we can illustrate the idea by showing that E [ xy] = E [ x ]E [y] when x and
y are independent and defined by (1.9). In this case, it can be shown (details omitted)
that the random variables x and y are independent precisely when the events A and
B are independent. Now observe that
( xy)( ) := x ( )y( ) = s1{ A}t1{ B} = st1{ A B}

Hence, by the definition of expectations, we have
E [ xy] = stP( A B) = stP( A)P( B) = sP( A)tP( B) = E [ x]E [y]

Fact 1.3.2. If x and y are independent and g and f are any functions, then f ( x ) and
g(y) are independent.
An important special case of the independence means multiply rule is as follows.
Fact 1.3.3. If random variables x1 , . . . , x N are independent, and each has density pn ,
then the joint density p exists, and is the product of the marginal densities:
N
p ( s1 , . . . , s N ) =
pn (sn )
n =1
Here are some useful facts relating independence and certain common distributions.
IID
Fact 1.3.4. If x1 , . . . , xk N (0, 1), then

k
Q :=
xi2 2 (k)
i =1
Fact 1.3.5. If Q1 , . . . , Q J are independent with Q j 2 (k j ), then jJ=1 Q j 2 ( j k j ).

Fact 1.3.6. If Z and Q are two random variables such that
1. Z N (0, 1),
28
2. Q 2 (k), and
3. Z and Q are independent,
then Z (k/Q)1/2 has the t-distribution with k degrees of freedom.
Fact 1.3.7. If Q1 2 (k1 ) and Q2 2 (k2 ) are independent, then
Q1 /k1
Q2 /k2
is distributed as F (k1 , k2 ).
1.3.3
Variance and Covariance
R
Let x F. For k N, the k-th moment of x is defined as E [ x k ] = sk F (ds). If
E [| x|k ] < then the k-th moment is said to exist. For a random variable with the
Cauchy distribution, even the first moment does not exist. For the normal distribution, every moment exists.
Fact 1.3.8. If the k-th moment of x exists, then so does the j-th moment for all j k.
The variance of random variable x is defined as
var[ x ] := E [( x E [ x ])2 ]
This gives a measure of the dispersion of x. (Not all random variables have a well
defined variance, but in general well just talk about the variance of a given random
variable
pwithout adding the caveat assuming it exists.) The standard deviation
of x is var[ x ].
The covariance of random variables x and y is defined as
cov[ x, y] := E [( x E [ x ])(y E [y])]
Fact 1.3.9. If x1 , . . . , x N are random variables and 1 , . . . , N are constant scalars,
then
"
#
N
var
n xn
n =1
2n var[xn ] + 2
n =1
n<m
n m cov[ xn , xm ]
29
In particular, if and are real numbers and x and y are random variables, then
var[] = 0,10 var[ + x ] = 2 var[ x ], and
var[x + y] = 2 var[ x ] + 2 var[y] + 2 cov[ x, y]
Given two random variables x and y with finite variances x2 and y2 respectively, the
correlation of x and y is defined as
corr[ x, y] :=
cov[ x, y]
x y
If corr[ x, y] = 0, we say that x and y are uncorrelated. For this to occur, it is necessary and sufficient that cov[ x, y] = 0. Positive correlation means that corr[ x, y] is
positive, while negative correlation means that corr[ x, y] is negative.
Fact 1.3.10. Given any two random variables x, y and positive constants , , we
have
1 corr[ x, y] 1 and corr[x, y] = corr[ x, y]
Fact 1.3.11. If x and y are independent, then cov[ x, y] = corr[ x, y] = 0.
Note that the converse is not true: One can construct examples of dependent random variables with zero covariance.
1.3.4
Best Linear Predictors
As a little exercise that starts moving us in the direction of statistics, lets consider
the problem of predicting the value of a random variable y given knowledge of the
value of a second random variable x. Thus, we seek a function f such that f ( x ) is
close to y on average. To measure the average distance between f ( x ) and y, we
will use the mean squared deviation between f ( x ) and y, which is
E [(y f ( x))2 ]
As we will learn in chapter 3, the minimizer of the mean squared deviation over all
functions of x is obtained by choosing f ( x ) = E [y | x ], where the right-hand size is
the conditional expectation of y given x. However, the conditional expectation may
var[] should be understood as var[1{ }], as was the case when we discussed
fact 1.1.4 on page 14.
10 Here
30
be nonlinear and complicated, so lets now consider the simpler problem of finding
a good predictor of y within a small and well-behaved class of functions. The class
of functions we will consider is the set of linear functions
L := { all functions of the form `( x ) = + x }

(While elementary courses refer to these functions as linear, in fact they are not linear
unless = 0 (see 2.1.3). The class of functions L is more correctly known as the set
of affine functions.) Thus, we consider the problem
min E [(y `( x ))2 ] = min E [(y x )2 ]
,
` L
(1.23)
Expanding the square on the right-hand side and using linearity of E , the objective
function becomes
(, ) := E [y2 ] 2E [y] 2E [ xy] + 2E [ x ] + 2 + 2 E [ x2 ]
Computing the derivatives and solving the equations
(, )
=0
and
(, )
=0
We obtain (exercise 1.6.25) the minimizers

:=
cov[ x, y]
var[ x ]
and
:= E [y] E [ x ]
(1.24)
The best linear predictor is therefore
` ( x ) := + x
If youve studied elementary linear least squares regression before, you will realize
that and are the population counterparts for the coefficient estimates in the
regression setting. Well talk more about the connections in the next chapter.
1.4
Asymptotics
In statistics, we often want to know how our tests and procedures will perform as
the amount of data we have at hand becomes large. To this end, we now investigate
the limiting properties of sequences of random variables. We begin by discussing
three modes of convergence for random variables, all of which are used routinely in
econometrics.
1.4.1
31
Modes of Convergence
Let { xn }
n=1 be a sequence of random variables. We say that { xn }n=1 converges to
random variable x in probability if
for any > 0,
P{| xn x| > } 0 as n
p
In symbols, this convergence is represented by xn x. In almost all the applications

we consider, the limit x will be a constant. The next example illustrates the definition
for this case.
p
Example 1.4.1. If xn N (, 1/n), then xn . That is, for any > 0, we have
P{| xn | > } 0. Fixing > 0, the probability P{| xn | > } is shown
in figure 1.7 for two different values of n, where it corresponds to the size of the
shaded areas. This probability collapses to zero as n , decreasing the variance
and causing the density to become more peaked.
A full proof of the convergence result in example 1.4.1 can be found by looking at the
normal density and bounding tail probabilities. However, a much simpler proof can
also be obtained by exploiting the connection between convergence in probability
and convergence in mean squared error. The details are below.
Fact 1.4.1. Regarding convergence in probability, the following statements are true:
p
1. If g : R R is continuous at x and xn x, then g( xn ) g( x ).

p
2. If xn x and yn y, then xn + yn x + y and xn yn xy.

p
3. If xn x and n , then xn + n x + and xn n x.11

We say that { xn } converges to x in mean square if
E [( xn x)2 ] 0 as n
ms
In symbols, this convergence is represented by xn x.

Fact 1.4.2. Regarding convergence in mean square, the following statements are
true:
11 Here
{n } is a nonrandom scalar sequence.
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
32
1.0
(a) n = 10
(b) n = 20
Figure 1.7: P{| xn | > } for xn N (, 1/n)

ms
1. If xn x, then xn x.
ms
2. If is constant, then xn if and only if E [ xn ] and var[ xn ] 0.

Part 1 of fact 1.4.2 follows from Chebychevs inequality, which states that for any
random variable y with finite second moment and any > 0, we have
P{|y| }
E [ y2 ]
(1.25)
(See exercise 1.6.29.) Using monotonicity of P and then applying (1.25) to y = xn x,

we obtain
E [( xn x)2 ]
P{| xn x| > } P{| xn x| }
2
Part 1 of fact 1.4.2 follows. Part 2 is implied by the equality
E [( xn )2 ] = var[ xn ] + (E [ xn ] )2
Verification of this equality is an exercise.
p
Example 1.4.2. In example 1.4.1, we stated that if xn N (, 1/n), then xn . This

follows from parts 1 and 2 of fact 1.4.2, since E [ xn ] = and var[ xn ] = 1/n 0.
33
1.0
0.0
0.2
0.4
0.6
0.8
df = 1
df = 10
N(0,1)
Figure 1.8: t-distribution with k df converges to N (0, 1) as k

Let { Fn }
n=1 be a sequence of cdfs, and let F be a cdf. We say that Fn converges
weakly to F if, for any s such that F is continuous at s, we have
Fn (s) F (s)
as
Example 1.4.3. It is well-known that the cdf of the t-distribution with k degrees
of freedom converges to the standard normal cdf as k . This convergence is
illustrated in figure 1.8.
Sometimes densities are easier to work with than cdfs. In this connection, note that
pointwise convergence of densities implies weak convergence of the corresponding
distribution functions:
Fact 1.4.3. Let { Fn }
n=1 be a sequence of cdfs, and let F be a cdf. Suppose that all these
cdfs are differentiable, and let pn and p be the densities of Fn and F respectively. If
pn (s) p(s) for all s R, then Fn converges weakly to F.
Let { xn }
n=1 and x be random variables, where xn Fn and x F. We say that
xn converges in distribution to x if Fn converges weakly to F. In symbols, this
d
convergence is represented by xn x.
34
Fact 1.4.4. Regarding convergence in distribution, the following statements are true:
d
1. If g : R R is continuous at x and xn x, then g( xn ) g( x ).

p
2. If xn x, then xn x.
p
3. If is constant and xn , then xn .

The next result is sometimes known as Slutskys theorem.
p
Fact 1.4.5. If is constant, xn and yn y, then xn + yn + y and xn yn y.

An immediate but useful conseqence is that
p
Fact 1.4.6. If xn 0 and yn y, then xn yn 0.

d
Indeed, by Slutskys theorem (fact 1.4.5) we have xn yn 0. Since the limit is constant, 1.4.4 then tells us that convergence is in probability as well.
1.4.2
The Law of Large Numbers
Two of the most important theorems in both probability and statistics are the law
of large numbers and the central limit theorem. In their simplest forms, these theorems deal with averages of independent and identically distributed (IID) sequences.
The law of large numbers tells us that these averages converge in probability to the
mean of the distribution in question. The central limit theorem tells us that a simple
transform of the average converges to a normal distribution.
Lets start with the law of large numbers, which relates to the sample mean
x N :=
1
N
xn
n =1
of a given sample x1 , . . . , x N
Theorem 1.4.1. Let { xn } be an IID sequence of random variables with common distribution
R
F. If the first moment |s| F (ds) is finite, then
p
x N E [ xn ] =
sF (ds)
as
(1.26)
35
To prove theorem 1.4.1, we can use fact 1.4.2 on page 31. In view of this fact, it
R
suffices to show that E [ x N ] sF (ds) and var[ x N ] 0 as N . These steps
are left as an exercise (exercise 1.6.31). When you do the exercise, note to yourself
exactly where independence bites.12
Example 1.4.4. To illustrate the law of large numbers, consider flipping a coin until
10 heads have occurred. The coin is not fair: The probability of heads is 0.4. Let x
be the number of tails observed in the process. It is known that such an x has the
negative binomial distribution, and, with a little bit of googling, we find that the
mean E [ x ] is 15. This means that if we simulate many observations of x and take
the sample mean, we should get a value close to 15. Code to do this is provided in
listing 1. Can you see how this program works?13 An improved implementation
is given in listing 2. The generation of a single observation has been wrapped in a
function called f. To generate multiple observations, we have used the R function
replicate, which is handy for simulations.
Listing 1 Illustrates the LLN
num . repetitions <- 10000
outcomes <- numeric ( num . repetitions )
for ( i in 1: num . repetitions ) {
num . tails <- 0
num . heads <- 0
while ( num . heads < 10) {
b <- runif (1)
num . heads <- num . heads + ( b < 0.4)
num . tails <- num . tails + ( b >= 0.4)
}
outcomes [ i ] <- num . tails
}
print ( mean ( outcomes ) )
At first glance, the law of large numbers (1.26) appears to only be a statement about
the sample mean, but actually it can be applied to functions of the random variable
12 The proof involves a bit of cheating,
because it assumes that the variance of each xn is finite. This

second moment assumption is not necessary for the result, but it helps to simplify the proof.
13 Hint: If u is uniform on [0, 1] and q [0, 1], then P{ u q } = q. This fact is used to simulate the
coin flips. Also recall that the logical values TRUE and FALSE are treated as 1 and 0 respectively in
algebraic expressions.
36
Listing 2 Second version of listing 1

f <- function ( q ) {
num . tails <- 0
num . heads <- 0
while ( num . heads < 10) {
b <- runif (1)
num . heads <- num . heads + ( b < q )
num . tails <- num . tails + ( b >= q )
}
return ( num . tails )
}
outcomes <- replicate (10000 , f (0.4) )
print ( mean ( outcomes ) )
as well. For example, if h : R R is such that
1
N
|h(s)| F (ds) is finite, then
h(xn ) E [h(xn )] =
h(s) F (ds)
(1.27)
n =1
This can be confirmed by letting yn := h( xn ) and then applying theorem 1.4.1.

Also, the law of large numbers applies to probabilities as well as expectations. To
see this, let x F, fix B R, and consider the probability P{ x B}. Let h be
the function defined by h(s) = 1{s B} for all s R. Using the principle that
expectations of indicator functions equal probabilities of events (page 14), we have
E [h( x)] = E [1{ x B}] = P{ x B}

It now follows from (1.27) that if { xn } is an IID sample from F, then
1
N
1{ x n B } P{ x n B }
(1.28)
n =1
The left hand side is the fraction of the sample that falls in the set B, and (1.28) tells
us that this fraction converges to the probability that xn B.
1.4.3
The Central Limit Theorem
The central limit theorem is another classical result from probability theory. It is
arguably one of the most beautiful and important results in all of mathematics. Rel-
37
ative to the LLN, it requires an additional second moment condition.

Theorem 1.4.2. Assume the conditions of theorem 1.4.1. If, in addition, the second moment
R 2
s F (ds) is finite, then
N ( x N ) y N (0, 2 ) as N
R
R
where := sF (ds) = E [ xn ] and 2 := (s )2 F (ds) = var[ xn ].
(1.29)
Another common statement of the central limit theorem is as follows: If all the conditions of theorem 1.4.2 are satisfied, then
x N d
z N := N
z N (0, 1) as N
(1.30)
Exercise 1.6.32 asks you to confirm this via theorem 1.4.2 and fact 1.4.4.
The central limit theorem tells us about the distribution of the sample mean when
N is large. Arguing informally, for N large we have
N ( x N ) y N (0, 2 )

y
2
x N + N ,
N
N
Here means that the distributions are approximately equal. We see that x N is
approximately normal, with mean equal to := E [ x1 ] and variance converging to
zero at a rate proportional to 1/N.
The convergence in (1.30) is illustrated by listing 3, the output of which is given
in figure 1.9. The listing generates 5,000 observations of the random variable z N
defined in (1.30), where each xn is 2 (5). (The mean of this distribution is 5, and the
variance is 2 5 = 10.) The observations of z N are stored in the vector outcomes,
and then histogrammed. The last line of the listing superimposes the density of the
standard normal distribution over the histogram. As expected, the fit is pretty good.
Before finishing this section, we briefly note the following asymptotic result, which
is frequently used in conjunction with the central limit theorem:
Theorem 1.4.3. Let {tn } be a sequence of random numbers and let be a constant. Sup
d
pose that n(tn ) N (0, 2 ) for some > 0. Suppose further that g : R R is
differentiable at and g0 ( ) 6= 0. Under these conditions we have
n{ g(tn ) g( )} N (0, g0 ( )2 2 )
as
(1.31)
38
Listing 3 Illustrates the CLT

num . replications <- 5000
outcomes <- numeric ( num . replications )
N <- 1000
k <- 5
# Degrees of freedom
for ( i in 1: num . replications ) {
xvec <- rchisq (N , k )
outcomes [ i ] <- sqrt ( N / (2 * k ) ) * ( mean ( xvec ) - k )
}
hist ( outcomes , breaks =50 , freq = FALSE )
curve ( dnorm , add = TRUE , lw =2 , col = " blue " )
0.2
0.1
0.0
Density
0.3
0.4
Histogram of outcomes
outcomes
Figure 1.9: Illustration of the CLT
39
The technique illustrated in theorem 1.4.3 is referred to as the delta method. The
delta method is extremely useful, particularly when one seeks the asymptotic distribution of certain kinds estimators. We will see its importance in some applications
later on. The proof of theorem 1.4.3 is based on a Taylor expansion of g around
the point , and can be found in almost any text on mathematical statistics. Exercise 1.6.39 walks you throught the most important ideas.
Instead of giving a full proof here, we will cover some parts of the proof of the
following corollary: If the conditions of theorem 1.4.2 are satisfied and g : R R is
differentiable at with g0 () 6= 0, then
1.5
N { g( x N ) g()} N (0, g0 ()2 2 )
as
(1.32)
Further Reading
To be written
1.6
Exercises
Ex. 1.6.1. Suppose that P is a probability on (, F ), so that P( A B) = P( A) +

P( B) whenever A and B are disjoint. Show that if A, B and C are disjoint, then
P( A B C ) = P( A ) + P( B ) + P( C ).
Ex. 1.6.2. Prove fact 1.1.2: P( A B) = P( A) + P( B) P( A B) for any A, B.14
Ex. 1.6.3. Given sample space := {1, 2, 3}, let A := {1}, B := {2} and C := {3}.
Let P( A) = P( B) = 1/3. Compute P(C ), P( A B), P( A B), P( Ac ), P( Ac Bc )
and P( A | B). Are A and C independent?
Ex. 1.6.4. A dice is designed so that the probability of getting face m is qm, where
m {1, . . . , 6} and q is a constant. Compute q.
Ex. 1.6.5. Let be a nonempty finite set, and let 0 be a fixed element of . For
each A , define P( A) := 1{0 A}. Is P a probability on ? Why or why not?
14 Hint:
Sketching the Venn diagram, convince yourself that A = [( A B) \ B] ( A B). Finish

the proof using the definition of a probability and fact 1.1.1 (page 5).
40
Ex. 1.6.6. Let be any sample space, and let P be a probability on the subsets F . Let
A F . Show that if P( A) = 0 or P( A) = 1, then A is independent of every other
event in F . Show that if A is independent of itself, then either P( A) = 0 or P( A) =
1. Show that if A and B are independent, then Ac and Bc are also independent.
Ex. 1.6.7. Let P and be defined as in example 1.1.5. Show that P is additive, in
the sense that if A and B are disjoint events, then P( A B) = P( A) + P( B).
Ex. 1.6.8. Let P and be defined as in example 1.1.5. Let A be the event that the
first switch is on, and let B be the event that the second switch is on. Show that A
and B are independent under P.
Ex. 1.6.9. Show that when is finite, a random variable x on can only take on a
finite set of values (i.e., has finite range).15
Ex. 1.6.10. Recall Fx defined in (1.11). We claimed that Fx is a cdf, which implies that
lims Fx (s) = 1. Verify this when x is the finite-valued random variable in (1.4).
Ex. 1.6.11. Recall Fx defined in (1.11). Suppose that x is the finite-valued random
variable in (1.4). Show that lims Fx (s) = 0. If you can, show that F is rightcontinuous.
Ex. 1.6.12. Prove the claim in fact 1.2.2 on page 17.
Ex. 1.6.13. Let x be a discrete random variable taking values s1 , . . . , s J , and let p j :=
P{ x = s j }. Show that 0 p j 1 for each j, and jJ=1 p j = 1.
Ex. 1.6.14. This exercise describes the inverse transform method for generating
random variables with arbitrary distribution from uniform random variables. The
uniform cdf on [0, 1] is given by F (s) = 0 if s < 0, F (s) = s if 0 s 1, and F (s) = 1
if s > 1. Let G be another cdf on R. Suppose that G is strictly increasing, and let
G 1 be the inverse (quantile). Show that if u F, then G 1 (u) G.
Ex. 1.6.15. Let x F where F is the uniform cdf on [0, 1]. Give an expression for the
cdf G of the random variable y = x2 .
Ex. 1.6.16. Let y F, where F is a cdf. Show that F (s) = E [1{y s}] for any s.
Ex. 1.6.17. Confirm monotonicity of expectations (fact 1.1.6 on page 14) for the special case where x and y are the random variables in (1.9).
15 Hint:
Have a look at the definition of a function in 11.1.1.
41
Ex. 1.6.18. Prove fact 1.3.8. (Existence of k-th moment implies existence of j-th moment for all j k.)
Ex. 1.6.19. Confirm the expression for variance of linear combinations in fact 1.3.9.
Ex. 1.6.20. Let x and y be scalar random variables. With reference to fact 1.3.10 on
page 29, is it true that corr[x, y] = corr[ x, y] for any constant scalars and ? Why
or why not?
Ex. 1.6.21. Confirm the claim in fact 1.3.11: If x and y are independent, then cov[ x, y] =
corr[ x, y] = 0.
Ex. 1.6.22. Let x1 and x2 be random variables with densities p1 and p2 . Let q be their
joint density. Show that x1 and x2 are independent whenever q(s, s0 ) = p1 (s) p2 (s0 )
for every s, s0 R.
Ex. 1.6.23. Fact 1.3.2 tells us that if x and y are independent random variables and g
and f are any two functions, then f ( x ) and g(y) are independent. Prove this for the
case where f ( x ) = 2x and g(y) = 3y 1.
Ex. 1.6.24. Let x and y be independent uniform random variables on [0, 1]. Let
z := max{ x, y}. Compute the cdf, density and mean of z.16 In addition, compute
the cdf of w := min{ x, y}.
Ex. 1.6.25. Confirm the solutions in (1.24).
Ex. 1.6.26. Consider the setting of 1.3.4. Let , and ` be as defined there. Let
the prediction error u be defined as u := y ` ( x ). Show that
1. E [` ( x )] = E [y]
2. var[` ( x )] = corr[ x, y]2 var[y]
3. var[u] = (1 corr[ x, y]2 ) var[y]
Ex. 1.6.27. Continuing on from exercise 1.6.26, show that cov[` ( x ), u] = 0.
Ex. 1.6.28. Let { xn } be a sequence of random variables satisfying xn = y for all n,
where y is a single random variable. Show that if P{y = 1} = P{y = 1} = 0.5,
p
then xn 0 fails. Show that if P{y = 0} = 1, then xn 0 holds.

Fix s R and compare the sets {z s} and { x s} {y s}. What is the relationship
between these two sets?
16 Hint:
42
Ex. 1.6.29. Prove Chebychevs inequality (1.25). In particular, show that if x is a

random variable with finite second moment and > 0, then P{| x | }
p
E [ x2 ] .
2
Ex. 1.6.30. We saw in fact 1.4.4 that if xn x, then xn x. Show that the converse
is not generally true. In other words, give an example of a sequence of random
variables { xn } and random variable x such that xn converges to x in distribution,
but not in probability.
Ex. 1.6.31. In this exercise, we complete the proof of the LLN on page 34. Let { xn }
be an IID sequence of random variables with common distribution F. Show that
R
E [ x N ] sF(ds) and var[ x N ] 0 as N .
Ex. 1.6.32. Confirm (1.30) via theorem 1.4.2 and fact 1.4.4.
Ex. 1.6.33 (Computational). Provided that we can at least generate uniform random
variables, the inverse transform method (see exercise 1.6.14) can be (and is) used to
generate random variables with arbitrary distribution G. Pick three different continuous distributions G1 , G2 and G3 available in R. Using Q-Q plots,17 examine for each
Gi whether the random variables generated via inverse transform do appear equally
distributed to the random variables generated from Gi using Rs built in algorithms
(accessed through rname, where name is one of norm, lnorm, etc.).
Ex. 1.6.34 (Computational). Using numerical integration, show that the 8th moment
of the standard normal density is approximately 105.
Ex. 1.6.35 (Computational). Using numerical integration and a for loop, compute
the first 10 moments of the exponential distribution with mean 1. (The exponential
distribution has one parameter. If the mean is 1, the value of the parameter is pinned
down to what value?)
Ex. 1.6.36 (Computational). Using a for loop, plot the chi-squared density for k =
1, 2, 3, 4, 5, all on the same figure. Use different colors for different k, and include a
legend.
Ex. 1.6.37 (Computational). Replicate the simulation performed in listing 3, but this
time for N = 2. Why is the fit not as good?
Ex. 1.6.38 (Computational). Replicate the simulation performed in listing 3, but this
time for uniformly distributed random variables on [1, 1]. Compare histograms
and normal density plots in the manner of figure 1.9. Use the appropriate mean
and variance in the normal density. Produce plots for N = 1, N = 2, N = 5 and
N = 200.
17 Look
them up if you dont know what they are. In R, see the documentation on qqplot.
43
Ex. 1.6.39. This exercise covers some of the proof behind theorem 1.4.3 on page 37.
Suppose that {tn } is a sequence of random variables, is a constant, and
n(tn ) N (0, 2 )
as
Let g : R R be differentiable at with g0 ( ) 6= 0. Taking a first order Taylor

expansion of g around , we can write g(tn ) = g( ) + g0 ( )(tn ) + R(tn ),
where R(tn ) is a remainder term. It turns out that under these conditions we
p
have nR(tn ) 0. The details are omitted. Using this fact, prove carefully that
d
n{ g(tn ) g( )} N (0, g0 ( )2 2 ).
1.6.1
Solutions to Selected Exercises
Solution to Exercise 1.6.1. If A, B and C are disjoint, then A B and C are also
disjoint, and A B C = ( A B) C. As a result, using additivity over pairs,
P( A B C) = P(( A B) C) = P( A B) + P(C) = P( A) + P( B) + P(C)

This result can be extended to an arbitrary number of sets by using induction.
Solution to Exercise 1.6.2. Pick any sets A, B F . To show that
P( A B ) = P( A ) + P( B ) P( A B )
we start by decomposing A into the union of two disjoint sets: A = [( A B) \ B]
( A B). Using additivity of P, we then have
P( A) = P[( A B) \ B] + P( A B)
Since B ( A B), we can apply part 1 of fact 1.1.1 (page 5) to obtain
P( A ) = P( A B ) P( B ) + P( A B )
Rearranging this expression gives the result that we are seeking.
Solution to Exercise 1.6.3. First, P(C ) = 1/3 as 1 = P() = P( A B C ) =
P( A) + P( B) + P(C) = 1/3 + 1/3 + P(C), and hence P(C) = 1/3. In addition,
P( A B) = 2/3, P( A B) = 0, P( Ac ) = 2/3, P( Ac Bc ) = P(( A B)c ) =
P() = 1, and P( A C) = 0 6= 1/9 = P( A)P(C). Therefore A is not independent
of C.
44
Solution to Exercise 1.6.4. When the dice is rolled one face must come up, so the
sum of the probabilities is one. More formally, letting = {1, . . . , 6} be the sample
space, we have
P{1, . . . , 6} =
P 6m=1
{m} =
m =1
m =1
P{ m } =
qm = 1
Solving the last equality for q, we get q = 1/21.

Solution to Exercise 1.6.5. To show that P is a probability on we need to check
that
1. 1{0 A} [0, 1] for every A .
2. 1{0 } = 1
3. If A B = , then 1{0 A B} = 1{0 A} + 1{0 B}
1 is immediate from the definition of an indicator function. 2 holds because 0 .
Regarding 3, pick any disjoint A and B. If 0 A, then 0
/ B, and we have
1 { 0 A B } = 1 = 1 { 0 A } + 1 { 0 B }
If 0 B, then 0
/ A, and once again we have
1 { 0 A B } = 1 = 1 { 0 A } + 1 { 0 B }
Finally, if 0 is in neither A nor B, then
1 { 0 A B } = 0 = 1 { 0 A } + 1 { 0 B }
We have shown that 13 hold, and hence P is a probability on .
Solution to Exercise 1.6.6. Suppose that P( A) = 0 and that B F . We claim that
P( A B) = P( A)P( B), or, in this case, P( A B) = 0. Using nonnegativity and
monotonicity of P (fact 1.1.1), we obtain
0 P( A B ) P ( A ) = 0
Therefore P( A B) = 0 as claimed.
45
Now suppose that P( A) = 1. We claim that P( A B) = P( A)P( B), or, in this case,
P( A B) = P( B). In view of fact 1.1.2 on page 6, we have
P( A B ) = P( A ) + P( B ) P( A B )
Since P( A) = 1, it suffices to show that P( A B) = 1. This last equality is implied
by monotonicity of P, because 1 = P( A) P( A B) 1.
Next, suppose that A is independent of itself. Then P( A) = P( A A) = P( A)P( A) =
P( A)2 . If a = a2 , then either a = 0 or a = 1.
Finally, let A and B be independent. We have
P( Ac Bc ) = P(( A B)c ) = 1 P( A B)
Applying fact 1.1.2 and independence, we can transform the right-hand side to obtain
P( Ac Bc ) = (1 P( A))(1 P( B)) = P( Ac )P( Bc )
In other words, Ac and Bc are independent.
Solution to Exercise 1.6.7. The proof is almost identical to the proof of additivity in
example 1.1.4 (page 4).
Solution to Exercise 1.6.8. The proof of independence is essentially the same as the
proof of independence of A and B in example 1.1.6 (page 6).
Solution to Exercise 1.6.10. We are assuming that x has finite range, and hence takes
only finitely many different values. Let m be the largest such value. For this m, we
have
lim Fx (s) Fx (m) = P{ : x ( ) m} = P() = 1
s
(The inequality is due to the fact that Fx is increasing.) On the other hand,
lim Fx (s) = lim P{ x s} lim P() = 1
From these two inequalities we get 1 lims Fx (s) 1, which is equivalent to

lims Fx (s) = 1.
46
Solution to Exercise 1.6.12. Fix s 0. Using additivity over disjoint sets, we have
F| x| (s) := P{| x | s} = P{s x s} = P{ x = s} + P{s < x s}
By assumption, P{ x = s} = 0. Applying fact 1.2.1 on page 17 then yields
F| x| (s) = P{s < x s} = F (s) F (s)
The claim F| x| (s) = 2F (s) 1 now follows from the definition of symmetry.
Solution to Exercise 1.6.13. That 0 p j 1 for each j follows immediately from
the definition of P. In addition, using additivity of P, we have
J
j =1
j =1
p j = P { x = s j } = P j =1 { x = s j } = P ( ) = 1
J
(1.33)
(We are using the fact that the sets { x = s j } disjoint. Why is this always true? Look
carefully at the definition of a function given in 11.1.1.)
Solution to Exercise 1.6.14. Let z := G 1 (u). We want to show that z G. Since G
is monotone increasing we have G ( a) G (b) whenever a b. As a result, for any
s R,
P{z s} = P{G1 (u) s} = P{G(G1 (u)) G (s)} = P{u G(s)} = G(s)

We have shown that z G as claimed.
Solution to Exercise 1.6.15. Evidently G (s) = 0 when s < 0. For s 0 we have
P{ x2 s} = P{| x|
s } = P{ x
s} = F ( s)
Thus, G (s) = F ( s)1{s 0}.

Solution to Exercise 1.6.17. If x ( ) := 1{ A} 1{ B} =: y( ) for any
, then A B. (If A, then x ( ) = 1. Since x ( ) y( ) 1, we then
have y( ) = 1, and hence B.) Using (1.8) and monotonicity of P, we then have
E [ x] = E [1{ A}] = P( A) P( B) = E [1{ B}] = E [y]

as was to be shown.
47
Solution to Exercise 1.6.18. Let a be any nonnegative number, and let j k. If a

1, then a j ak . If a < 1, then a j 1. Thus, for any a 0, we have a j ak + 1, and for
any random variable x we have | x | j | x |k + 1. Using monotonicity of expectations
(fact 1.1.6 on page 14) and E [1] = 1, we then have E [| x | j ] E [| x k |] + 1. Hence the
j-th moment exists whenever the k-th moment exists.
Solution to Exercise 1.6.23. Let u := f ( x ) = 2x and v := g(y) = 3y 1, where
x and y are independent. Independence of u and v can be confirmed via (1.22) on
page 26. Fixing s1 and s2 in R, we have
P{u s1 , v s2 } = P{ x s1 /2, y (s2 + 1)/3}

= P{ x s1 /2}P{y (s2 + 1)/3} = P{u s1 }P{v s2 }
Thus u and v are independent as claimed.
Solution to Exercise 1.6.24. As in the statement of the exercise, x and y are independent uniform random variables on [0, 1], z := max{ x, y} and w := min{ x, y}. As
a first step to the proofs, you should convince yourself that if a, b and c are three
numbers, then
max{ a, b} c if and only if a c and b c
min{ a, b} c if and only if a c or b c
Using these facts, next convince yourself that, for any s R,
{z s} = { x s} {y s}
{w s} = { x s} {y s}
(For each, equality, show that if is in the right-hand side, then is in the left-hand
side, and vice versa.) Now, for s [0, 1], we have
P{z s} = P[{ x s} {y s}] = P{ x s}P{y s} = s2

By differentiating we get the density p(s) = 2s, and by integrating
E [z] = 2/3. Finally, regarding the cdf of w, for s [0, 1] we have
R1
0
sp(s)ds we get
P{w s} = P[{ x s} {y s}]

= P{ x s} + P{y s} P[{ x s} {y s}]
Hence P{w s} = 2s s2 .
48
Solution to Exercise 1.6.27. Using y = ` ( x ) + u and the results form exercise 1.6.26,
we have
var[` ( x ) + u] = var[y]
= corr[ x, y]2 var[y] + (1 corr[ x, y]2 ) var[y]

= var[` ( x )] + var[u]
It follows (why?) that cov[` ( x ), u] = 0 as claimed.
Solution to Exercise 1.6.28. From the definition of convergence in probability (see
p
1.4.1), the statement xn 0 means that, given any > 0, we have P{| xn | > }
0. Consider first the case where P{y = 1} = P{y = 1} = 0.5. Take = 0.5. Then,
since xn = y for all n,
P{| xn | > } = P{|y| > 0.5} = 1

p
Thus, the sequence does not converge to zero. Hence xn 0 fails. On the other
hand, if P{y = 0} = 1, then for any > 0 we have
P{| xn | > } = P{|y| > } = 0

p
This sequence does converge to zero (in fact its constant at zero), and xn 0 holds.
Solution to Exercise 1.6.29. Pick any random variable x and > 0. By considering
what happens at an arbitrary , you should be able to convince yourself that
x2 = 1{| x | } x2 + 1{| x | < } x2 1{| x | }2
Using fact 1.1.6 (page 14), fact 1.1.3 (page 14) and rearranging completes the proof
that P{| x | }
E [ x2 ] .
2
Solution to Exercise 1.6.30. We want to give an example of a sequence of random

variables { xn } and random variable x such that xn converges to x in distribution,
but not in probability. Many examples can be found by using IID sequences. For
example, if { xn }
n=1 and x are IID standard normal random variables, then xn and
x have the same distribution for all n, and hence xn converges in distribution to x.
However, zn := xn x has distribution N (0, 2) for all n. Letting z be any random
variable with distribution N (0, 2) and be any strictly positive constant, we have
P{| xn x| } = P{|z| } > 0. Thus, P{| xn x| } does not converge to
zero.
49
Solution to Exercise 1.6.31. By linearity of expectations,

1
E [ x N ] =
N
N
E [ xn ] = N
n =1
sF (ds) =
sF (ds)
R
This confirms that E [ x N ] sF (ds) as claimed. To see that var[ x N ] 0 as N ,
let 2 be the common variance of each xn . Using fact 1.3.9, we obtain
#
"
1 N 2
1 N
2
x
=
var
+ 2 cov[ xn , xm ]
n
2
N n =1
N n =1
N n<m
By independence, this reduces to var[ x N ] = 2 /N, which converges to zero.
Chapter 2
Linear Algebra
The first part of this chapter is mainly about solving systems of linear equations,
while the second deals with random matrices. We start our story from the beginning, with the notions vectors and matrices.
2.1
Vectors and Matrices
[roadmap]
2.1.1
Vectors
An important set for us will be, for arbitrary N N, the set of all N-vectors, or
vectors of length N. This set is denoted by R N , and a typical element is of the form
x1
x2
x = . where xn R for each n

..
xN
Here x has been written vertically, as a column of numbers. We could also write
x horizontally, like so: x = ( x1 , . . . , x N ). At this stage, we are viewing vectors just
as sequences of numbers, so it makes no difference whether they are written vertically or horizontally. Later, when we come to deal with matrix algebra, we will
distinguish between column (vertical) and row (horizontal) vectors.
50
CHAPTER 2. LINEAR ALGEBRA
51
The vector of ones will be denoted 1, while the vector of zeros will be denoted 0:

1
0
..
..
1 := .
0 := .
1
For elements of R N there are two fundamental algebraic operations: addition and
scalar multiplication. If x R N and y R N , then the sum is defined by
x1 + y1
x1
y1
x2 y2
x2 + y2
x + y :=: . + . :=
.
..
.. ..
xN
xN + yN
yN
If R, then the scalar product of and x is defined to be
x1
x2
x := .
..
x N
Thus, addition and scalar multiplication are defined in terms of ordinary addition
and multiplication in R, and computed element-by-element, by adding and multiplying respectively. Figures 2.1 and 2.1 show examples of vector addition and scalar
multiplication in the case N = 2. In the figure, vectors are represented as arrows,
starting at the origin and ending at the location in R2 defined by the vector.
We have defined addition and scalar multiplication of vectors, but not subtraction.
Subtraction is performed element by element, analogous to addition. The definition
can be given in terms of addition and scalar multiplication. x y := x + (1)y. An
illustration of this operation is given in figure 2.3. The way to remember this is to
draw a line from y to x, and then shift it to the origin.
The inner product of two vectors x and y in R N is denoted by x0 y, and defined as
the sum of the products of their elements:
x0 y :=
xn yn = y0 x
n =1
The (euclidean) norm of a vector x R N is defined as

!1/2
N
kxk := x0 x := xn2
n =1
(2.1)
Figure 2.1: Vector addition
Figure 2.2: Scalar multiplication
52
53
Figure 2.3: Difference between vectors
and represents the length of the vector x. (In the arrow representation of vectors in
figures 2.12.3, the norm of the vector is equal to the length of the arrow.)
Fact 2.1.1. For any R and any x, y R N , the following properties are satisfied
by the norm:
1. kxk 0 and kxk = 0 if and only if x = 0.
2. kxk = ||kxk.
3. kx + yk kxk + kyk.
4. |x0 y| kxkkyk.
The third property is called the triangle inequality, while the fourth is called the
Cauchy-Schwartz inequality.
Given two vectors x and y, the value kx yk has the interpretation of being the
distance between these points. To see why, consult figure 2.3 again.
2.1.2
54
Matrices
A N K matrix is a rectangular array A of real numbers with N rows and K columns,

written in the following way:
A=
a1K
a2K
..
.
a N1 a N2
a NK
a11
a21
..
.
a12
a22
..
.
Often, the values ank in the matrix represent coefficients in a system of linear equations, such as
a11 x1 + a12 x2 + + a1K xK = b1
a21 x1 + a22 x2 + + a2K xK = b2
..
.
a N1 x1 + a N2 x2 + + a NK xK = b N
Well explore this relationship further after some more definitions.
In matrix A, the symbol ank stands for the element in the n-th row of the k-th column.
For obvious reasons, the matrix A is also called a vector if either N = 1 or K = 1. In
the former case, A is called a row vector, while in the latter case it is called a column
vector. If A is N K and N = K, then A is called square. If, in addition ank = akn
for every k and n, then A is called symmetric.
When convenient, we will use the notation rown (A) to refer to the n-th row of A,
and colk (A) to refer to its k-th column.
For a square matrix A, the N elements of the form ann for n = 1, . . . , N are called the
principal diagonal:
a11 a12 a1N

a
21 a22 a2N
..
..
..
.
.
.
a N1 a N2
a NN
A is called diagonal if the only nonzero entries are on the principal diagonal.
(Clearly every diagonal matrix is symmetric.) If, in addition to being diagonal, each
element ann along the principal diagonal is equal to 1, then A is called the identity
55
matrix, and denoted by I:
I :=
0
1
..
.
0
0
..
.
0 0
1
0
..
.
Just as was the case for vectors, a number of algebraic operations are defined for
matrices. The first two, scalar multiplication and addition, are immediate generalizations of the vector case: For R, we let
a11 a12 a1K

a11 a12 a1K
a
a
a22 a2K
21
21 a22 a2K
:
=
.
..
..
..
..
..
..
.
.
.
.
.
a N1 a N2
while
a11
a
21
..
.
..
.
a1K
a2K
..
.
a N1
a NK
a N1 a N2
a NK
..
.
b1K
b2K
..
.
b N1
b NK
b11
b21
..
.
:=
a NK
..
.
a1K + b1K
a2K + b2K
..
.
a N1 + b N1
a NK + b NK
a11 + b11
a21 + b21
..
.
In the latter case, the matrices have to have the same number of rows and columns
in order for the definition to make sense.
Now lets look at multiplication of matrices. If A and B are two matrices, then their
product AB is formed by taking as its i, j-th element the inner product of the i-th
row of A and the j-th column of B. For example, consider the following product.
a11 a1K
b11 b1J
c11 c1J
a
21 a2K b21 b2J c21 c2J

=
..
..
.. ..
..
.. ..
..
..
.
.
.
.
.
.
.
.
.
a N1 a NK
bK1 bK J
c N1 c N J
Here c11 is computed as
c11 = row1 (A)0 col1 (B) =
a1k bk1
k =1
There are many good tutorials for multiplying matrices on the web (try Wikipedia,
for example), so Ill leave it to you to get a feeling for this operation.
56
Since inner products are only defined for vectors of equal length, this requires that
the length of the rows of A is equal to the length of the columns of B. Put differently,
the number of columns of A is equal to the number of rows of B. In other words,
if A is N K and B is J M, then we require K = J. The resulting matrix AB is
N M. Heres the rule to remember:
product of N K and K M is N M
From the definition, it is clear that multiplication is not commutative, in that AB and
BA are not generally the same thing. Indeed BA is not well-defined unless N = M
also holds. Even in this case, the two are not generally equal.
Other than that, multiplication behaves pretty much as wed expect. In particular,
for conformable matrices A, B and C, we have
A(BC) = (AB)C
A(B + C) = AB + AC
(A + B)C = AC + BC
(Here, we are using the word conformable to indicate dimensions are such that
the operation in question makes sense. For example, well say for two conformable
matrices A and B, the product AB satisfies xyz if the dimensions of A and B are
such that the product is well defined; and similarly for addition, etc.)
2.1.3
Linear Functions
One way to view matrices is as objects representing coefficients in linear systems

of equations. Another way to view matrices is as functions, or mappings from one
space to another. We will see that these two perspectives are very closely related.
Lets begin with a definition: A function f : R N R M is called linear if
f (x + y) = f (x) + f (y) for any x, y R N and , R
Example 2.1.1. The function f : R R defined by f ( x ) = x2 is nonlinear, because
if we take = = x = y = 1, then f (x + y) = f (2) = 4, while f ( x ) + f (y) =
1 + 1 = 2.
57
Example 2.1.2. The function f : R R defined by f ( x ) = 2x is linear, because if we

take any , , x, y in R, then
f (x + y) = 2(x + y) = 2x + 2y = f ( x ) + f (y)
Example 2.1.3. The affine function f : R R defined by f ( x ) = 1 + 2x is nonlinear,
because if we take = = x = y = 1, then f (x + y) = f (2) = 5, while
f ( x ) + f (y) = 3 + 3 = 6.
Now lets return to matrices. When we think of an N K matrix A as a mapping, we
are considering the operation of sending a vector x RK into a new vector y = Ax
in R N . In this sense, A defines a function from RK to R N . Among the collection
of all functions from RK to R N , these functions defined by matrices have a special
property: they are all linear. Moreover, it turns out that the functions defined by
matrices are the only linear functions. In other words, the set of linear functions
from RK to R N and the set of N K matrices are essentially the same thing.
Lets look at some of the details. Take a fixed N K matrix A and consider the
function f : RK R N defined by f (x) = Ax. To see that f is a linear function, pick
any x, y in RK , and any scalars and . The rules of matrix addition and scalar
multiplication tell us that
f (x + y) := A(x + y) = Ax + Ay = Ax + Ay =: f (x) + f (y)
In other words, f is linear.
How about some more examples of a linear functions, to help you grasp the intuition? As I mentioned just above, I cant give you any more examples because there
arent any more. The next theorem states this result.
Theorem 2.1.1. If A is an N K matrix and f : RK R N is defined by f (x) = Ax, then
the function f is linear. Conversely, if f : RK R N is linear, then there exists an N K
matrix A such that f (x) = Ax for all x RK .
The proof of the second part of theorem 2.1.1 is an exercise (exercise 2.7.3).
2.1.4
Maps and Linear Equations
In 2.1.3, we started to think about matrices as maps. That is, given an N K matrix
A, we can identify A with the function f (x) = Ax. Now lets think about linear
58
equations. The canonical problem is as follows: Given A and a fixed vector b, we

want to find a vector x such that Ax = b. The best way to think about this is that
we want to invert the mapping f (x) = Ax, because what we want to do is find an
x such that f (x) = b. That is, we want to find f 1 (b), the preimage of b under f .
Now if you know anything about inverting functions, you will know that there are
various potential problems here. For one, there may be no x such that f (x) = b.
Secondly, there may be multiple x with f (x) = b. These problems concern existence
and uniqueness of solutions respectively.
Before tackling this problem directly, lets go back to the general problem of inverting functions in the one-dimensional case, the advantage being that we can graph
the function and gain visual intuition. Consider figure 2.4, which graphs a onedimensional function f : [0, 1] R. The set [0, 1] over which f is defined is called
the domain of f . The red interval is called the range of f , and consists of all y such
that f ( x ) = y for some x in the domain [0, 1]. More generally, for an arbitrary function f : X Y, the range of f is
rng( f ) := {y Y : f ( x ) = y for some x X }
Returning to the problems of existence and uniqueness of solutions, have another
look at the function f in figure 2.4. Evidently, the equation f ( x ) = b has a solution if
and only if b rng( f ). In figure 2.5, b falls outside rng( f ) and there is no solution.
The other issue we must consider is uniqueness. Even if the equation f ( x ) = b has
a solution, the solution may not be unique. Figure 2.6 gives an example. Here both
x1 and x2 solve f ( x ) = b.
Lets return now to the matrix setting. Let f (x) = Ax, where A is a given matrix,
and consider the equation Ax = b for some fixed b. When will this equation have
a solution? In view of our preceding discussion, the answer is that there will be a
solution if and only if b is in rng(A) := rng( f ). This is more likely if the range of
f is large. One way to check this by looking at something called the rank of A,
which is a measure of the size of its range. In turn, the rank is related to whether or
not the columns of A are linearly independent or not. Conveniently, this last question
turns out to be closely connected to the issue of uniqueness of solutions. To grasp
these ideas takes a bit of effort but is certainly worthwhile. Lets get started.
59
Figure 2.4: Preimage of b under f
Figure 2.5: No solution
60
x1
x2 1
Figure 2.6: Multiple solutions
2.2
Span, Dimension and Independence
Motivated by the preceding discussion, we now introduce the important notions of

linear subspaces, spans, linear independence, basis and dimension.
2.2.1
Spans and Linear Subspaces
Given K vectors x1 , . . . , xK in R N , we can form linear combinations, which are vectors of the form
K
y=
k x k = 1 x1 + + K x K
k =1
for some collection 1 , . . . , K of K real numbers.

Fact 2.2.1. Inner products of linear combinations satisfy the following rule:
!0
k =1
k xk
j yj
j =1
k j x0k y j
k =1 j =1
61
The set of all linear combinations of X := {x1 , . . . , xK } is called the span of X, and
denoted by span( X ):
(
)
K
span( X ) :=
all vectors
k xk such that := (1, . . . , K ) RK
k =1
Let Y be any subset of R N , and let X be as above. If Y span( X ), we say that the
vectors X := {x1 , . . . , xK } span the set Y, or that X is a spanning set for Y. This is a
particularly nice situation when Y is large but X is small, because it means that all
the vectors in the large set Y are described by the small number of vectors in X.
Example 2.2.1. Let X = {1} = {(1, 1)} R2 . The span of X is all vectors of the
form (, ) with R. This constitutes a line in the plane. Since we can take = 0,
it follows that the origin 0 is in span( X ). In fact span( X ) is the unique line in the
plane that passes through both 0 and the vector 1 = (1, 1).
Example 2.2.2. Consider the vectors {e1 , . . . , e N } R N , where en has all zeros except for a 1 as the n-th element. The case of R2 , where e1 := (1, 0) and e2 := (0, 1), is
illustrated in figure 2.7. The vectors e1 , . . . , e N are called the canonical basis vectors
of R N well see why later on. One reason is that {e1 , . . . , e N } spans all of R N . To
see this in the case of N = 2 (check general N yourself), observe that for any y R2 ,
we have

y1
y1
0
1
0
=
+
+ y2
= y1 e1 + y2 e2
y :=
= y1
y2
0
y1
0
1
Thus, y span{e1 , e2 } as claimed. Since y is just an arbitrary vector in R2 , we have
shown that {e1 , e2 } spans R2 .
Example 2.2.3. Consider the set P := {( x1 , x2 , 0) R3 : x1 , x2 R}. Graphically,
P corresponds to the flat plane in R3 , where the height coordinate is always zero.
If we take e1 = (1, 0, 0) and e2 = (0, 1, 0), then given y = (y1 , y2 , 0) P we have
y = y1 e1 + y2 e2 . In other words, any y P can be expressed as a linear combination
of e1 and e2 , and {e1 , e2 } is a spanning set for P.
Fact 2.2.2. Let X and Y be any two finite subsets of R N . If X Y, then we have
span( X ) span(Y ).
One of the key features of the span of a set X is that it is closed under the linear
operations of vector addition and scalar multiplication, in the sense that if we take
62
Figure 2.7: Canonical basis vectors in R2
elements of the span and combine them using these operations, the resulting vectors
are still in the span. For example, to see that the span is closed under vector addition,
observe that if X = {x1 , . . . , xK } and y, z are both in span( X ), then we can write
them as
K
y=
k xk
k =1
and
z=
k xk
k =1
for a suitable scalars k , k R. It then follows that

K
y+z =
(k + k )xk span(X )
k =1
Hence span( X ) is closed under vector addition as claimed. Another easy argument
shows that span( X ) is closed under scalar multiplication.
The notion of a set being closed under scalar multiplication and vector addition is
important enough to have its own name: A set S R N with this property is called
a linear subspace of R N . More succinctly, a nonempty subset S of R N is called a
linear subspace if, for any x and y in S, and any and in R, the linear combination
x + y is also in S.
Example 2.2.4. It follows immediately from the proceeding discussion that if X is
any finite nonempty subset of R N , then span( X ) is a linear subspace of R N . For this
reason, span( X ) is often called the linear subspace spanned by X.
63
Figure 2.8: The vectors e1 and x

Example 2.2.5. In R3 , lines and planes that pass through the origin are linear subspaces. Other linear subspaces of R3 are the singleton set containing the zero element 0, and the set R3 itself.
Fact 2.2.3. Let S be a linear subspace of R N . The following statements are true:
1. The origin 0 is an element of S.
2. If X is a finite subset of S, then span( X ) S.
2.2.2
Linear Independence
In some sense, the span of X := {x1 , . . . , xK } is a measure of the diversity of the

vectors in Xthe more diverse are the elements of X, the greater is the set of vectors
that can be represented as linear combinations of its elements. In fact, if X is not very
diverse, then some similar elements may be redundant, in the sense that one can
remove an element xi from the collection X without reducing its span.
Lets consider two extremes. First consider the vectors e1 := (1, 0) and e2 := (0, 1)
in R2 (figure 2.7). As we saw in example 2.2.2, the span {e1 , e2 } is all of R2 . With just
these two vectors, we can span the whole plane. In algebraic terms, these vectors
are relatively diverse. We can also see their diversity in the fact that if we remove
one of the vectors from {e1 , e2 }, the span is no longer all of R2 . In fact it is just a line
in R2 . Hence both vectors have their own role to play in forming the span.
Now consider the pair e1 and x := 2e1 = (2, 0), as shown in figure 2.8. This pair
is not very diverse. In fact, if y span{e1 , x}, then, for some 1 and 2 ,
y = 1 e1 + 2 x = 1 e1 + 2 (2)e1 = (1 22 )e1 span{e1 }
64
In other words, any element of span{e1 , x} is also an element of span{e1 }. We can

kick x out of the set {e1 , x} without reducing the span.
Lets translate these ideas into formal definitions. In general, the set of vectors X :=
{x1 , . . . , xK } in R N is called linearly dependent if one (or more) vector(s) can be
removed without changing span( X ). We call X linearly independent if it is not
linearly dependent.
To see this definition in a slightly different light, suppose that X := {x1 , . . . , xK } is
linearly dependent, with
span{x1 , . . . , xK } = span{x2 , . . . , xK }
Since x1 span{x1 , . . . , xK } certainly holds, this equality implies that
x1 span{x2 , . . . , xK }
Hence, there exist constants 2 , . . . , K with
x1 = 2 x2 + K x K
In other words, x1 can be expressed as a linear combination of the other elements in
X. This is a general rule: Linear dependence means that at least one vector in the
set can be written as a linear combination of the others. Linear independence means
the opposite is true. The following fact clarifies matters further:
Fact 2.2.4 (Definitions of linear independence). The following statements are all
equivalent:
1. The set X := {x1 , . . . , xK } is linearly independent.
2. If X0 is a proper subset of X, then span( X0 ) is a proper subset of span( X ).1
3. No vector in X can be written as a linear combination of the others.
4. If kK=1 k xk = 0, then 1 = 2 = = K = 0.
5. If j 6= 0 for some j, then kK=1 k xk 6= 0.
1A
is a proper subset of B if A B and A 6= B.
65
Part 5 is just the contrapositive of part 4, and hence the two are equivalent. (See 11.3
if you dont know what a contrapositive is.) The equivalence of part 4 and part 3
might not be immediately obvious, but the connection is clear when you think about
it. To say that (1 , . . . , K ) = 0 whenever kK=1 k xk = 0 means precisely that no xk
can be written as a linear combination of the other vectors. For example, if there
does exist some j 6= 0 with kK=1 k xk = 0, then x j = k6= j (k / j )xk .
Example 2.2.6. The set of canonical basis vectors in example 2.2.2 is linearly independent. Indeed, if j 6= 0 for some j, then kK=1 k ek = (1 , . . . , K ) 6= 0.
One reason for our interest in the concept of linear independence lies in the following problem: We know when a point in R N can be expressed as a linear combination
of some fixed set of vectors X. This is true precisely when that point is in the span
of X. What we do not know is when that representation is unique. It turns out that
the relevant condition is independence:
Theorem 2.2.1. Let X := {x1 , . . . , xK } be any collection of vectors in R N , and let y be any
vector in span( X ). If X is linearly independent, then there exists one and only one set of
scalars 1 , . . . , K such that y = kK=1 k xk .
Proof. Since y is in the span of X, we know that there exists at least one such set of
scalars. Suppose now that there are two. In particular, suppose that
y=
k =1
k =1
k xk = k xk
It follows from the second equality that kK=1 (k k )xk = 0. Using fact 2.2.4, we
conclude that k = k for all k. In other words, the representation is unique.
2.2.3
Dimension
In essence, the dimension of a linear subspace is the minimum number of vectors

needed to span it. To understand this idea more clearly, lets look at an example.
Consider the plane
P := {( x1 , x2 , 0) R3 : x1 , x2 R}
(2.2)
from example 2.2.3. Intuitively, this plane is a two-dimensional subset of R3 . This
intuition agrees with the definition above. Indeed, P cannot be spanned by one
vector, for if we take a single vector in R3 , then the span created by that singleton is
66
only a line in R3 , not a plane. On the other hand, P can be spanned by two vectors,
as we saw in example 2.2.3.
While P can also be spanned by three or more vectors, it turns out that one of the
vectors will always be redundantit does not change the span. In other words, any
collection of 3 or more vectors from P will be linearly dependent. The following
theorem contains the general statement of this idea:
Theorem 2.2.2. If S is a linear subspace of R N spanned by K vectors, then every linearly
independent subset of S has at most K vectors.
Put differently, if S is spanned by K vectors, then any subset of S with more than
K vectors will be linearly dependent. This result is sometimes called the exchange
theorem. The proof is not overly hard, but it is a little long. Readers keen to learn
more will find it in most texts on linear algebra.
We now come to a key definition. If S is a linear subspace of R N and B is some finite
subset of R N , then B is called a basis of S if B spans S and is linearly independent.
Example 2.2.7. The pair {e1 , e2 } is a basis for the set P defined in (2.2).
Example 2.2.8. Consider the set of canonical basis vectors {e1 , . . . , e N } R N described in example 2.2.8. This set is linearly independent, and its span is equal to all
of R N . As a result, {e1 , . . . , e N } is a basis for R N as anticipated by the name.
Theorem 2.2.3. If S is a linear subspace of R N , then every basis of S has the same number
of elements.
Proof. Let B1 and B2 be two bases of S, with K1 and K2 elements respectively. By
definition, B2 is a linearly independent subset of S. Moreover, S is spanned by the
set B1 , which has K1 elements. Applying theorem 2.2.2, we see that B2 has at most
K1 elements. That is, K2 K1 . Repeating the same argument while reversing the
roles of B1 and B2 we obtain K1 K2 . Hence K1 = K2 .
Theorem 2.2.3 states that if S is a linear subspace of R N , then every basis of S has the
same number of elements. This common number is called the dimension of S, and
written as dim(S). For example, if P is the plane in (2.2), then dim( P) = 2, because
the set {e1 , e2 } is a basis, and this set contains two elements. The whole space R N
is N dimensional, because the canonical basis vectors form a basis, and there are N
canonical basis vectors.
In R3 , a line through the origin is a one-dimensional subspace, while a plane through
the origin is a two-dimensional subspace.
67
Fact 2.2.5. The only N-dimensional linear subspace of R N is R N .

If we take a set of K vectors, then how large will its span be in terms of dimension?
The next lemma answers this question.
Lemma 2.2.1. Let X := {x1 , . . . , xK } R N . Then
1. dim(span( X )) K.
2. dim(span( X )) = K if and only if X is linearly independent.
Proof. Regarding part 1, let B be a basis of span( X ). By definition, B is a linearly independent subset of span( X ). Since span( X ) is spanned by K vectors, theorem 2.2.2
implies that B has no more than K elements. Hence, dim(span( X )) K.
Regarding part 2, suppose first that X is linearly independent. Then X is a basis for
span( X ). Since X has K elements, we conclude that dim(span( X )) = K.
Conversely, if dim(span( X )) = K then X must be linearly independent. For if X is
not linearly independent, then exists a proper subset X0 of X such that span( X0 ) =
span( X ). By part 1 of this theorem, we then have dim(span( X0 )) #X0 K 1.
Therefore, dim(span( X )) K 1. Contradiction.
Part 2 of lemma 2.2.1 is important in what follows, and also rather intuitive. It says
that the span of a set will be large when its elements are algebraically diverse.
2.3
Matrices and Equations
[Roadmap]
2.3.1
Rank
Lets now connect matrices to our discussion of span, linear independence and dimension. We will be particularly interested in solving equations of the form Ax = b
for unknown x. We take A to be an N K matrix. As discussed in 2.1.32.1.4, we
can view this matrix as a mapping f (x) = Ax from RK to R N . If b R N is given
and we are looking for an x to solve Ax = b, then we know at least one such x will
68
exist if b is in the range of f . Since A and f are essentially the same thing, we will
denote the range by rng(A) instead of rng( f ). That is,
rng(A) := {Ax : x RK }
Just a little bit of thought will convince you that this is precisely the span of the
columns of A:
rng(A) = span(col1 (A), . . . , colK (A))
For obvious reasons, this set is sometimes called the column space of A. Being
defined as a span, it is obviously a linear subspace of R N .
As stated above, for the system Ax = b to have a solution, we require that b
rng(A). If we want to check that this is true, well probably be wanting to check
that rng(A) is suitably large. The obvious measure of size for a linear subspace
such as rng(A) is its dimension. The dimension of rng(A) is known as the rank of
A. That is,
rank(A) := dim(rng(A))
Furthermore, A is said to have full column rank if rank(A) is equal to K, the number
of its columns. Why do we say full rank here? Because, by definition, rng(A) is
the span by K vectors, and hence, by part 1 of lemma 2.2.1, we have dim(rng(A))
K. In other words, the rank of A is less than or equal to K. A is said to have full
column rank when this maximum is achieved.
When is this maximum achieved? By part 2 of lemma 2.2.1, this while be the case
precisely when the columns of A are linearly independent. Thus, the matrix A is
of full column rank if and only if the columns of A are linearly independent. By
fact 2.2.4 on page 64, the next characterization is also equivalent.
Fact 2.3.1. A is of full column rank if and only if the only x satisfying Ax = 0 is
x = 0.
Lets return to the problem of solving the system Ax = b for some fixed b R N .
For existence of a solution we need b rng(A), and this range will be large when A
is full column rank. So the property of A being full column rank will be connected
to the problem of existence. Even better, the full column rank condition is exactly
what we need for uniqueness as well, as follows immediately from theorem 2.2.1.
In matrix terminology, theorem 2.2.1 translates to the following result:
Fact 2.3.2. If A has full column rank and b rng(A), then the system of equations
Ax = b has a unique solution.
2.3.2
69
Square Matrices
Now let N = K, so that A is a square N N matrix. Suppose that A is full column

rank, so that its columns are independent. In that case, fact 2.2.5 on page 67 implies
that rng(A) is equal to R N , because rng(A) is a linear subspace of R N by definition,
and its dimension is N by the full column rank assumption.
Since rng(A) = R N , it follows immediately that, for any b R N , the system Ax = b
has a solution. Moreover, by fact 2.3.2, the solution is unique. To repeat, if A is
square and full column rank, then for any b R N , the there is a unique x R N
such that Ax = b.
Since this problem is so important, there are several different ways of describing it.
First, a square matrix A is called invertible if there exists a second square matrix B
such that AB = BA = I, where I is the identity matrix. If this is the case, then B is
called the inverse of A, and written as A1 . Invertibility of A is exactly equivalent
to the existence of a unique solution x to the system Ax = b for all b R N . Indeed,
the existence of an inverse A1 means that we can take our solution as
x b = A 1 b
(2.3)
since Axb = AA1 b = Ib = b.

In addition, to each square matrix A, we can associate a unique number det(A)
called the determinant of A. The determinant crops up in many ways, but the general definition is a bit fiddly to state and hence we omit it. (Look it up if you are
interested, but note that it wont be required in what follows.) If det(A) = 0, then A
is called singular. Otherwise A is called nonsingular.
It turns out that nonsingularity is also equivalent to invertibility. Lets summarize
our discussion:
Fact 2.3.3. For N N matrix A, the following are equivalent:
1. A is of full column rank.
2. The columns of A are linearly independent.
3. A is invertible.
4. A is nonsingular.
5. For each b R N , the system Ax = b has unique solution A1 b.
70
The next fact collects useful results about the inverse.

Fact 2.3.4. If A and B are invertible and 6= 0, then
1. det(A1 ) = (det(A))1 ,
2. (A1 )1 = A,
3. (A)1 = 1 A1 , and
4. (AB)1 = B1 A1 .
2.3.3
Other Properties of Matrices
The transpose of N K matrix A is a K N matrix A0 such that the first column of

A0 is the first row of A, the second column of A0 is the second row of A, and so on.
For example, given
10 40
A := 20 50
30 60

B :=
1 3 5
2 4 6

(2.4)
the transposes are

A0 =
10 20 30
40 50 60
1 2
B0 := 3 4
5 6
Fact 2.3.5. For conformable matrices A and B, transposition satisfies the following:
1. (AB)0 = B0 A0
2. (A + B)0 = A0 + B0
3. (cA)0 = cA0 for any constant c.
Note that a square matrix C is symmetric precisely when C0 = C. Note also that
A0 A and AA0 are well-defined and symmetric.
Fact 2.3.6. For each square matrix A, we have
71
1. det(A0 ) = det(A), and

2. (A1 )0 = (A0 )1 whenever the inverse exists.
The trace of a square matrix is the sum of the elements on its principal diagonal.
That is,
a11 a12 a1N

a
N
21 a22 a2N
trace .
= ann
.
.
..
..
..
n =1
a N1 a N2 a NN
Fact 2.3.7. Transposition does not alter trace: trace(A) = trace(A0 ).
Fact 2.3.8. If A and B are N N matrices and and are two scalars, then
trace(A + B) = trace(A) + trace(B)
Moreover, if A is N M and B is M N, then trace(AB) = trace(BA).
The rank of a matrix can be difficult to determine. One case where it is easy is where
the matrix is idempotent. A square matrix A is called idempotent if AA = A.
Fact 2.3.9. If A is idempotent, then rank(A) = trace(A).
2.3.4
Quadratic Forms
Let A be N N and symmetric, and let x be N 1. The expression

x0 Ax =
aij xi x j
j =1 i =1
is called a quadratic form in x. Notice that if A = I, then this reduces to kxk2 , which
is positive whenever x is nonzero. The next two definitions generalize this idea: An
N N symmetric matrix A is called
nonnegative definite if x0 Ax 0 for all x R N , and
positive definite if x0 Ax > 0 for all x R N with x 6= 0.
As we have just seen, the identity matrix is positive definite.
72
Fact 2.3.10. If A is nonnegative definite, then each element ann on the principal diagonal is nonnegative.
Fact 2.3.11. If A is positive definite, then:
1. Each element ann on the principal diagonal is positive.
2. A is full column rank and invertible, with det(A) > 0.
To see that positive definiteness implies full column rank, consider the following
argument: If A is positive definite, then A must be full column rank, for if not there
exists a x 6= 0 with Ax = 0 (fact 2.3.1). But then x0 Ax = 0 for nonzero x. This
contradicts the definition of positive definiteness.
2.4
Random Vectors and Matrices
A random vector x is just a sequence of K random variables ( x1 , . . . , xK ). Each realization of x is an element of RK . The distribution (or cdf) of x is the joint distribution
F of ( x1 , . . . , xK ). That is,
F ( s ) : = : F ( s1 , . . . , s K ) : = P{ x1 s1 , . . . , x K s K } : = : P{ x s }
(2.5)
for each s in RK . (Here and in what follows, the statement x s means that xn sn
for n = 1, . . . , K.)
Just as some but not all distributions on R have a density representation (see 1.2.2),
some but not all distributions on RK can be represented by a density. We say that
f : RK R is the density of random vector x := ( x1 , . . . , xK ) if
Z
B
f (s) ds = P{x B}
(2.6)
for every subset B of RK .2 Most of the distributions we work with in this course
have density representations.
some subsets of RK are so messy that its not possible to integrate over them, so we
only require (2.6) to hold for a large but suitably well-behaved class of sets called the Borel sets. See
any text on measure theory for details.
2 Actually,
73
For random vectors, the definition of independence mirrors the scalar case. In particular, a collection of random vectors x1 , . . . , x N is called independent if, given any
s1 , . . . , s N , we have
P{ x1 s1 , . . . , x N s N } = P{ x1 s1 } P{ x N s N }
We note the following multivariate version of fact 1.3.2:
Fact 2.4.1. If x and y are independent and g and f are any functions, then f (x) and
g(y) are also independent.
A random N K matrix X is a rectangular N K array of random variables. In this
section, we briefly review some properties of random vectors and matrices.
2.4.1
Expectations for Vectors and Matrices
Let x = ( x1 , . . . , xK ) be a random vector taking values in RK with k := E [ xk ]

for all k = 1, . . . , K. The expectation E [x] of vector x is defined as the vector of
expectations:
1
E [ x1 ]
E [ x2 ] 2

E [x] :=
= .. =:
.
.
.
.
E [ xK ]
More generally, if X is a random N K matrix, then its expectation E [X] is the

matrix of the expectations:
E [ x11 ] E [ x12 ] E [ x1K ]

E [ x ] E [ x22 ] E [ x2K ]
21
E [X] :=
..
..
.
.
E [ x N1 ] E [ x N2 ]
..
.
E [ x NK ]
Expectation of vectors and matrices maintains the linearity of scalar expectations:

Fact 2.4.2. If X and Y are random and A, B and C are conformable constant matrices,
then
E [A + BX + CY] = A + BE [X] + CE [Y]
74
The covariance between random N 1 vectors x and y is

cov[x, y] := E [(x E [x])(y E [y])0 ]
The variance-covariance matrix of random vector x with := E [x] is defined as
var[x] := cov(x, x) = E [(x E [x])(x E [x])0 ] = E [(x )(x )0 ]
Expanding this out, we get
E [( x1 1 )( x1 1 )]
E [( x2 2 )( x )]
1
1
var[x] =
.
..
..
.
E [( x N N )( x1 1 )]
E [( x1 1 )( x N N )]
E [( x2 2 )( x N N )]
..
.
E [( x N N )( x N N )]
The j, k-th term is the scalar covariance between x j and xk . As a result, the principle
diagonal contains the variance of each xn .
Some simple algebra yields the alternative expressions
cov[x, y] = E [xy0 ] E [x]E [y]0
and
var[x] = E [xx0 ] E [x]E [x]0
Fact 2.4.3. For any random vector x, the variance-covariance matrix var[x] is square,
symmetric and nonnegative definite.
Fact 2.4.4. For any random vector x, any constant conformable matrix A and any
constant conformable vector a, we have
var[a + Ax] = A var[x]A0
2.4.2
Multivariate Gaussians
The multivariate normal density or Gaussian density in R N is a function p of the

form

1
0 1
N/2
1/2
p(s) = (2 )
det()
exp (s ) (s )
2
where is any N 1 vector and is a symmetric, positive definite N N matrix. In
symbols, we represent this distribution by N (, ). Although we omit the derivations, it can be shown that if x N (, ), then
E [x] = and var[x] =

We say that x is normally distributed if x N (, ) for some N 1 vector and
symmetric, positive definite N N matrix . We say that x is standard normal if
= 0 and = I.
75
Fact 2.4.5. N 1 random vector x is normally distributed if and only if a0 x is normally distributed in R for every constant N 1 vector a.3
Fact 2.4.6. If x N (, ), then a + Ax N (a + A, AA0 ).
Here, the fact that a + Ax has mean a + A and variance-covariance matrix AA0 is
not surprising. What is important is that normality is preserved .
Fact 2.4.7. Normally distributed random variables are independent if and only if
they are uncorrelated. In particular, if both x and y are normally distributed and
cov[ x, y] = 0, then x and y are independent.
Fact 2.4.8. If x N (, ), then (x )0 1 (x ) 2 (k ), where k := length of x.
Fact 2.4.9. If x N (0, I) and A is a conformable idempotent and symmetric matrix
with rank(A) = j, then x0 Ax 2 ( j). (In view of fact 2.3.9, when using this result it
is sufficient to show that trace(A) = j.)
2.5
Convergence of Random Matrices
As a precursor to time series analyais, we extend the probabilistic notions of convergence discussed in 1.4.1 to random vectors and matrices.
2.5.1
Modes of Convergence
We already have a notion of scalar convergence in probability (see 1.4.1). Extending

this to the matrix case, let {Xn }
n=1 be a sequence of random I J matrices. We say
p
that Xn converges to a random I J matrix X in probability (and write Xn X) if

every element of Xn converges to the corresponding element of X in probability in
the scalar sense. That is,
p
Xn X whenever xijn xij for all i and j

Here xijn is the i, j-th element of Xn and xij is the i, j-th element of X.
3 If
a = 0 then we can interpret a0 x as a normal random variable with zero variance.
76
If we are dealing with vectors (I = 1 or J = 1 in the previous definition), then the

condition for convergence has the form
n
x1
x1
p
p
xn := ... ... =: x whenever xkn xk for all k

xKn
xK
With vectors, we can also consider norm deviation. In this connection, we have the
following result.
Fact 2.5.1. If {xn } is a sequence of random vectors in RK and x is a random vector
in RK , then
p
p
xn x if and only if kxn xk 0
In other words, each element of xn converges in probability to the corresponding
element of x if and only if the norm distance between the vectors goes to zero in
probability. Although fact 2.5.1 is stated in terms of vectors, the same result is in
fact true for matrices if we regard matrices as vectors. In other words, if we take
an N K matrix A, we can think of it as a vector in R N K = R NK by stacking all
the columns into one long column, or rows into one long rowit doesnt matter
p
which. Thinking of matrices this way, fact 2.5.1 is applicable: Xn X if and only if
p
kXn Xk 0.4
Now lets extend the notion of convergence in distribution to random vectors. The
definition is almost identical to the scalar case, with only the obvious modifications.
K
K
Let { Fn }
n=1 be a sequence of cdfs on R , and let F be a cdf on R . We say that Fn
converges to F weakly if, for any s such that F is continuous at s, we have
Fn (s) F (s)
as n
K
Let {xn }
n=1 and x be random vectors in R , where xn Fn and x F. We say
that xn converges in distribution to x if Fn converges weakly to F. In symbols, this
d
convergence is represented by xn x.
As discussed above, convergence of xn to x in probability simply requires that the
elements of xn converge in probability (in the scalar sense) to the corresponding
elements of x. For convergence in distribution this is not generally true:
n
x1
x1
d
xkn xk for all k does not imply xn := ... ... =: x

xKn
4 There
xK
are various notions of matrix norms. The one defined here is called the Frobenius norm.
77
Put differently, convergence of the marginals does not necessarily imply convergence of the joint distribution. (As you might have guessed, one setting where convergence of the marginals implies convergence of the joint is when the elements of
the vectors are independent, and the joint is just the product of the marginals.)
The fact that elementwise convergence in distribution does not necessarily imply
convergence of the vectors is problematic, because vector convergence is harder
to work with than scalar convergence. Fortunately, we have the following results,
which provide a link from scalar to vector convergence:
Fact 2.5.2. Let xn and x be random vectors in RK .
d
1. If a0 xn a0 x for any a RK , then xn x.

2. If a0 xn a0 x for any a RK , then xn x.
The second of these results is quite straightforward to prove (exercise 8.6.2). The
first is more difficult (the standard argument uses characteristic functions). It is
often referred as the Cramer-Wold device.
2.5.2
Further Properties
We noted a variety of useful results pertaining to convergence in probability and

distribution for sums and products in the scalar case. Most of these carry over to the
vector case essentially unchanged. For example,
Fact 2.5.3. Assuming conformability, the following statements are true:
p
1
1
1. If Xn X and Xn and X are invertible, then X
n X .
p
2. If Xn X and Yn Y, then
p
Xn + Yn X + Y,
Xn Yn XY
and
Yn Xn YX
and
An Xn AX
3. If Xn X and An A, then
p
Xn + An X + A,
Xn An XA
78
In part 3 of fact 2.5.3, the matrices An and A are nonrandom. The convergence
An A means that each element of An converges in the usual scalar sense to the
corresponding element of A:
An A means aijn aij for all i and j
Alternatively, we can stack the matrices into vectors and take the norms, as discussed above. Then we say that An A if kAn Ak 0. The two definitions can
be shown to be equivalent.
Example 2.5.1. To see how fact 2.5.3 can be used, lets establish convergence of the
quadratic form
p
a0 Xn a a0 Xa whenever a is a conformable constant vector and Xn X
(2.7)
This follows from two applications of fact 2.5.3. Applying fact 2.5.3 once we get
p
a0 Xn a0 X. Applying it a second time yields the convergence in (2.7).

As in the scalar case, convergence in probability and convergence in distribution are
both preserved under continuous transformations:
Fact 2.5.4 (Continuous mapping theorem, vector case). Let xn and x be random vectors in RK , and let g : RK R J be continuous at x. In this setting,
d
1. If xn x, then g(xn ) g(x).

2. If xn x, then g(xn ) g(x).
Another result used routinely in econometric theory is the vector version of Slutskys theorem:
Fact 2.5.5 (Slutskys theorem, vector case). Let xn and x be random vectors in RK , let
p
Yn be random matrices, and let C be a constant matrix. If Yn C and xn x, then

d
Yn xn Cx
and
whenever the matrices are conformable.
Yn + xn C + x
79
The delta method from theorem 1.4.3 on page 37 extends to random vectors. For
example, let g : RK R be differentiable at a vector RK , in the sense that the
gradiant vector
g()
1
.
.
g() :=
.
g()
K
is well defined (i.e., the limit defining each of the partial derivatives exists). In this
context,
d
Fact 2.5.6. If {tn } is a sequence of random vectors in RK with n(tn ) N (0, )
for some RK and positive definite K K matrix , then
n{ g(tn ) g()} N (0, g()0 g())
as
(2.8)
whenever g()0 g() is positive. This last assumption will be satisfied if, for
example, at least one of the partial derivatives in g() is nonzero (why?).
2.5.3
Vector LLN and CLT
With the above definitions of convergence in hand, we can move on to the next topic:
Vector LLN and CLT. The scalar LLN and CLT that we discussed in 1.4 extend to
the vector case in a natural way. For example, we have the following result:
Theorem 2.5.1. Let {xn } be an IID sequence of random vectors in RK with E [kxn k2 ] < .
Let := E [xn ] and let := var[xn ]. For this sequence we have
x N :=
1
N
xn
and
N (x N ) N (0, )
(2.9)
n =1
Figure 2.9 illustrates the LLN in two dimensions. The green dot is the point 0 =
(0, 0) in R2 . The black dots are IID observations x1 , . . . , x N of a random vector with
mean = 0. The red dot is the sample mean N1 nN=1 xn . (Remember that we are
working with vectors here, so the summation and scalar multiplication in the sample mean x N is done elementwisein this case for two elements. In particular, the
sample mean is a linear combination of the observations x1 , . . . , x N .) By the vector
LLN, the red dot converges to the green dot.
The vector LLN in theorem 2.5.1 follows from the scalar LLN. To see this, let xn be as
in theorem 2.5.1, let a be any constant vector in RK and consider the scalar sequence
80
Figure 2.9: LLN, vector case
{yn } defined by yn = a0 xn . The sequence {yn } inherets the IID property from {xn }.5
By the scalar LLN (theorem 1.4.1) we have
1
N
But
1
N
yn E [yn ] = E [a0 xn ] = a0 E [xn ] = a0
n =1
yn =
n =1
1
N
a0 xn = a0
"
n =1
1
N
xn
= a0 x N
n =1
Since a was arbitrary, we have shown that

p
a0 x N a0 for any a RK
p
The claim x N now follows from fact 2.5.2.

The vector CLT in theorem 2.5.1 also follows from the scalar case. The proof is rather
similar to the vector LLN proof we have just completed. See exercise 8.6.5.
2.6
Further Reading
To be written.
5 Functions
of independent random variables are themselves independent (fact 1.3.2, page 27).
2.7
81
Exercises
Ex. 2.7.1. Given two vectors x and y, show that |kxk kyk| kx yk.6
Ex. 2.7.2. Use the first property in fact 2.1.1 to show that if y R N is such that
y0 x = 0 for every x R N , then y = 0.
Ex. 2.7.3. Prove the second part of theorem 2.1.1. In particular, show that if f : RK
R N is linear, then there exists an N K matrix A such that f (x) = Ax for all x RK .7
Ex. 2.7.4. Show that if S and S0 are two linear subspaces of R N , then S S0 is also a
linear subspace.
Ex. 2.7.5. Show that every linear subspace of R N contains the origin 0.
Ex. 2.7.6. Show that the vectors (1, 1) and (1, 2) are linearly independent.8
Ex. 2.7.7. Find two unit vectors (i.e., vectors with norm equal to one) that are orthogonal to (1, 2).
Ex. 2.7.8. Let a R N and let A := {x R N : a0 x = 0}. Show that A is a linear
subspace of R N .
Ex. 2.7.9. Let Q be the subset of R3 defined by
Q := {( x1 , x2 , x3 ) R3 : x2 = x1 + x3 }
Is Q a linear subspace of R3 ? Why or why not?
Ex. 2.7.10. Let Q be the subset of R3 defined by
Q := {( x1 , x2 , x3 ) R3 : x2 = 1}
Is Q a linear subspace of R3 ? Why or why not?
Ex. 2.7.11. Let X := {x1 , . . . , xK } be a linearly independent subset of R N . Is it
possible that 0 X? Why or why not?
Ex. 2.7.12. Prove facts 2.3.1 and 2.3.10.
6 Hint:
Use the triangle inequality.

colk (A) = f (ek ).
8 Hint: Look at the different definitions of linear independence. Choose the one thats easiest to
work with in terms of algebra.
7 Hint:
82
Ex. 2.7.13. Show that for any two conformable matrices A and B, we have (AB)1 =
B 1 A 1 .9
Ex. 2.7.14. Let A be a constant N N matrix. Assuming existence of the inverse
A1 , show that (A0 )1 = (A1 )0 .
Ex. 2.7.15. Show that if ei and e j are the i-th and j-th canonical basis vectors of R N
respectively, and A is an N N matrix, then ei0 Ae j = aij , the i, j-th element of A.
Ex. 2.7.16. Let

A :=
1 1
1 1
Show that A is nonnegative definite.

Ex. 2.7.17. Show that for any matrix A, the matrix A0 A is well-defined (i.e., multiplication is possible), square, and nonnegative definite.
Ex. 2.7.18. Show that if A and B are positive definite and A + B is well defined, then
it is also positive definite.
Ex. 2.7.19. Let A be N K. Show that if Ax = 0 for all K 1 vectors x, then A = 0
(i.e., every element of A is zero).
Ex. 2.7.20. Let I N be the N N identity matrix.
1. Explain briefly why I N is full column rank.
2. Show that I N is the inverse of itself.
3. Let A := I N . Give a condition on such that A is positive definite.
Ex. 2.7.21. Let X := I N 2uu0 , where u is an N 1 vector with kuk = 1. Show that
X is symmetric and XX = I N .
Ex. 2.7.22. Let 1 be an N 1 vector of ones. Consider the matrix
Z :=
1 0
11
N
1. Show that if x is any N 1 vector, then Zx is a vector with all elements equal
to the sample mean of the elements of x.
9 Hint:
Look at the definition of the inverse! Always look at the definition, and then show that the
object in question has the stated property.
83
2. Show that Z is idempotent.

Ex. 2.7.23. Show that if x is a random vector with E [xx0 ] = I and A is a conformable
constant matrix, then E [x0 Ax] = trace(A).
Ex. 2.7.24. Let x be random and let a be constant. Show that if E [x] = and
var[x] = , then E [a0 x] = a0 and var[a0 x] = a0 a.
Ex. 2.7.25. Let x be a random K 1 vector. Show that E [xx0 ] is nonnegative definite.
Ex. 2.7.26. Let x = ( x1 , . . . , x N ) N (0, I N ).
1. Are x1 and x2 independent? Why or why not?
2. What is the distribution of x12 ? Why?
3. What is the distribution of x12 /x22 ? Why?
4. What is the distribution of x1 [2/( x22 + x32 )]1/2 ? Why?
5. What is the distribution of kxk2 ? Why?
6. If a is an N 1 constant vector, what is the distribution of a0 x?
Ex. 2.7.27 (Computational). Write a function in R called innerprod that computes
the inner product of any two vectors using sum. Write another function called norm,
also using sum, that computes the norm of any vector. Choose two vectors and a
scalar, and check that the properties in fact 2.1.1 all hold.
Ex. 2.7.28 (Computational). Form a 4 5 matrix A, the elements of which are chosen
randomly via uniform sampling from {1, 2, 3, 4} with replacement (look up the online documentation on the sample function). Compute the row sums and column
sums by either pre- or postmultiplying by a vector of ones. (Which is which?) Check
against the built-in functions colSums and rowSums.
Ex. 2.7.29 (Computational). Using the matrix A in exercise 2.7.28, create a new matrix B such that the j-th column of B is the j-th column of A minus the column mean
of the j-th column of A. The built-in function colMeans can be used to obtain the
column means.
2.7.1
84
Solution to Exercise 2.7.8. Let x, y A and let , R. We must show that z :=

x + y A, or, equivalently, a0 z = a0 (x + y) = 0. This is immediate, because
a0 (x + y) = a0 x + a0 y = 0 + 0 = 0.
Solution to Exericse 2.7.9. If a := (1, 1, 1), then Q is all x with a0 x = 0. This set is
a linear subspace of R3 , as shown in exercise 2.7.8.
Solution to Exercise 2.7.11. If X has more than one element, then it is not possible.
To see this, suppose (without any loss of generality) that x1 = 0 X. Then
K
x1 = 0 =
0xk
k =2
In other words, x1 is a linear combination of the elements of X. This contradicts

linear independence.
Solution to Exercise 2.7.20. The solutions are as follows: (1) I N is full column rank
because its columns are the canonical basis vectors, which are independent. (2) By
definition, B is the inverse of A if BA = AB = I N . It follows immediately that I N
is the inverse of itself. (3) A sufficient condition is > 0. If this holds, then given
x 6= 0, we have x0 I N x = kxk2 > 0.
Solution to Exercise 2.7.21. First, X is symmetric because
X0 = (I N 2uu0 )0 = I0N 2(uu0 )0 = I N 2(u0 )0 u0 = I N 2uu0 = X
Second, XX = I N , because
XX = (I N 2uu0 )(I0N 2uu0 ) = I N I N 2I N 2uu0 + (2uu0 )(2uu0 )
= I N 4uu0 + 4uu0 uu0 = I N 4uu0 + 4uu0 = I N

The second last equality is due to the assumption that u0 u = kuk2 = 1.
Solution to Exercise 2.7.26. First note that since x N (0, I N ) we have cov[ xi , x j ] =
0 for all i 6= j. Since uncorrelated normal random variables are independent, we
85
IID
then have x1 , . . . , x N N (0, 1). Since sums of squares of independent standard

normals are chi-squared, we have in particular that
k
xn2 2 (k)
(2.10)
n =1
for any k N. The solutions to the exercise can now be given:

1. Yes, for the reason just described.
2. x12 2 (1) by (2.10)
3. x12 /x22 F (1, 1), because if Q1 2 (k1 ) and Q2 2 (k2 ) and Q1 and Q2 and
independent, then ( Q1 /k1 )/( Q2 /k2 ) F (k1 , k2 ).
4. x1 [2/( x22 + x32 )]1/2 t(2), because if Z N (0, 1) and Q 2 (k ) and Z and Q
are independent, then Z (k/Q)1/2 t(k ).
5. kxk2 = nN=1 x2N 2 ( N ) by (2.10).
6. Linear combinations of normals are normal, so y := a0 x is normal. Evidently
E [y] = E [a0 x] = a0 E [x] = 0. Using independence, we obtain
N
var[y] =
n =1
Hence y N (0, nN=1 a2n ).
a2n var[ xn ]
a2n
n =1
Chapter 3
Projections
This chapter provides further background in linear algebra for studying OLS with
multiple regressors. At the heart of the chapter is the orthogonal projection theorem,
which lies behind many of the key results in OLS theory. The theory of projections
also allows us to define conditional expectations, and determine the properties of
the conditioning operation.
3.1
Orthogonality and Projection
[roadmap]
3.1.1
Orthogonality
Let x and z be two vectors in R N . If x0 z = 0, then x and z are said to be orthogonal,

and we write x z. In R2 , x and z are orthogonal when they are perpendicular to
one another (figure 3.1). If x is a vector and S is a set, then we say that x is orthogonal
to S if x z for all z S. In this case we write x S. Figure 3.2 illustrates.
The first thing you need to know about orthogonal vectors is the Pythagorean Law:
Theorem 3.1.1. If x1 , . . . , xK are vectors in R N and xi x j whenever i 6= j, then
k x1 + + x K k2 = k x1 k2 + + k x K k2
Orthogonality and linear independence are related. For example,
86
CHAPTER 3. PROJECTIONS
87
Figure 3.1: x z
Figure 3.2: x S
88
Fact 3.1.1. If V is a finite set with x y for all distinct pairs x, y V, and, moreover,
0
/ V, then V is linearly independent.
3.1.2
Projections
One problem that comes up in many different contexts is approximation of an element y of R N by an element of a given subspace S of R N . Stated more precisely,
the problem is, given y and S, to find the closest element y of S to y. Closeness is in
terms of euclidean norm, so y is the minimizer of ky zk over all z S:
y := argmin ky zk
zS
Existence of a minimizer is not immediately obvious, suggesting that y may not be

well-defined. However, it turns out that we need not be concerned, as y always
exists (given any S and y). The next theorem states this fact, as well as providing a
way to identify y.
Theorem 3.1.2 (Orthogonal Projection Theorem, Part 1). Let y R N and let S be a
subspace of R N . The closest point in S to y is the unique vector y S such that y y S.
The vector y in theorem 3.1.2 is called the orthogonal projection of y onto S. Although we do not prove the theorem here, the intuition is easy to grasp from a
graphical presentation. Figure 3.3 illustrates. Looking at the figure, we can see that
the closest point y to y within S is indeed the one and only point in S such that y y
is orthogonal to S.
Holding S fixed, we can think of the operation
y 7 the orthogonal projection of y onto S
as a function from R N to R N .1 The function is typically denoted by P, so that P(y)
or Py represents the orthogonal projection y.
In general, P is called the orthogonal
projection onto S. Figure 3.4 illustrates the action of P on two different vectors.
Using this notation, we can restate the orthogonal projection theorem, as well as
adding some properties of P:
Theorem 3.1.3 (Orthogonal Projection Theorem 2). Let S be any linear subspace, and
let P : R N R N be the orthogonal projection onto S. The function P is linear. Moreover,
for any y R N , we have
1 Confirm
in your mind that we are describing a functional relationship, as defined in 11.1.1.
Figure 3.3: Orthogonal projection
Figure 3.4: Orthogonal projection under P
89
90
1. Py S,
2. y Py S,
3. kyk2 = kPyk2 + ky Pyk2 ,
4. kPyk kyk, and
5. Py = y if and only if y S.
These results are not difficult to prove, given theorem 3.1.2. Linearity of P is left as
an exercise (exercise 3.5.6). Parts 1 and 2 follow directly from theorem 3.1.2. To see
part 3, observe that y can be decomposed as
y = Py + y Py
Part 3 now follows from parts 12 and the Pythagorean law. (Why?) Part 4 follows
from part 3. (Why?) Part 5 is obvious from the definition of Py as the closest point
to y in S.
Theres one more very important property of P that we need to make note of: Suppose we have two linear subspaces S1 and S2 of R N , where S1 S2 . What then is
the difference between (a) first projecting a point onto the bigger subspace S2 , and
then projecting the result onto the smaller subspace S1 , and (b) projecting directly to
the smaller subspace S1 ? The answer is nonewe get the same result.
Fact 3.1.2. Let S1 and S2 be two subspaces of R N , and let y R N . Let P1 and P2 be
the projections onto S1 and S2 respectively. If S1 S2 , then
P1 P2 y = P2 P1 y = P1 y
Theres yet another way of stating the orthogonal projection theorem, which is also
informative. Given S R N , the orthogonal complement of S is defined as
S := {x R N : x S}
In other words, S is the set of all vectors that are orthogonal to S. Figure 3.5 gives
an example in R2 .
Fact 3.1.3. Given any S, the orthogonal complement S is always a linear subspace.
91
Figure 3.5: Orthogonal complement of S
This is easy enough to confirm: Looking back at the definition of linear subspaces,
we see that the following statement must be verified: Given x, y S and , R,
the vector that x + y is also in S . Clearly this is the case, because if z S, then
(x + y)0 z = x0 z + y0 z
( linearity of inner products)
= 0+0 = 0
( x, y S and z S)
We have shown that x + y z for any z S, thus confirming that x + y S .

Fact 3.1.4. For S R N , we have S S = {0}.
Now, lets look at the orthogonal projection theorem again. Our interest was in
projecting y onto S. However, we have just learned that S is itself a linear subspace,
so we can also project y onto S . Just as we used P to denote the function sending
y into its projection onto S, so well use M to denote the function sending y into
so that u := My. Figure 3.6
its projection onto S . The result well denote by u,
and indeed that is the
illustrates. The figure suggests that we will have y = y + u,
case. The next theorem states this somewhat more mathematically.
Theorem 3.1.4 (Orthogonal Projection Theorem 3). Let S be a linear subspace of R N . If
P is the orthogonal projection onto S and M is the orthogonal projection onto S , then Py
and My are orthogonal, and
y = Py + My
If S1 and S2 are two subspaces of R N with S1 S2 , then S2 S1 . This means that
the result in fact 3.1.2 is reversed for M.
92
Figure 3.6: Orthogonal projection
Fact 3.1.5. Let S1 and S2 be two subspaces of R N and let y R N . Let M1 and M2 be
the projections onto S1 and S2 respectively. If S1 S2 , then,
M1 M2 y = M2 M1 y = M2 y
Fact 3.1.6. Py = 0 if and only if y S , and My = 0 if and only if y S.2
3.2
Overdetermined Systems of Equations
When we get to multivariate linear regression, we will see that, mathematically

speaking, the problem we are presented with is one of solving what is called an
overdetermined system of equations. In turn, overdetermined systems of equations
are usually solved using orthogonal projection. Lets have a quick look at how this
is done. We begin with a system of equations such as X = y, where X is N K,
is K 1, and y is N 1. We regard the matrix X and the vector y as given, and
seek a RK that solves this equation. Throughout this section, we maintain the
assumption that X is full column rank.
If K = N, then the full column rank assumption and fact 2.3.3 imply that this system
has precisely one solution. However, we are going to study the case when N > K. In
this case, the system of equations is said to be overdetermined. This corresponds to
2 For
example, if Py = 0, then y = Py + My = My. Hence M does not shift y. If an orthogonal

projection onto a subspace doesnt shift a point, thats because the point is already in that subspace
(see, e.g., theorem 3.1.3). In this case the subspace is S , and we conclude that y S .
93
the situation where the number of equations (equal to N) is larger than the number
of unknowns (the K elements of ). Intuitively, in such a situation, we may not be
able find a that satisfies all N equations.
To understand this problem, recall from 2.3.1 that X can be viewed as a mapping
from RK to R N , and its range is the linear subspace of R N spanned by the columns
of X:
rng(X) := {all vectors X with RK } :=: column space of X
As discussed in 2.3.1, a solution to X = y exists precisely when y lies in rng(X).
In general, given our assumption that K < N, this outcome is unlikely.3 As a result,
the standard approach is to admit that an exact solution may not exist, and instead
focus on finding a RK such that X is as close to y as possible. Closeness is
defined in the euclidean sense, so the problem is to minimize ky Xk over the set
of all RK . Using the orthogonal projection theorem, the minimizer is easy to
identify:
Theorem 3.2.1. The minimizer of ky Xk over all RK is := (X0 X)1 X0 y.
Proof. If we can show that X is the closest point in rng(X) to y, we then have
ky X k ky Xk for any RK
which is all we need to prove. To verify that y := X is in fact the closest point in
rng(X) to y, recall the orthogonal projection theorem (page 88). By this theorem,
y := X is the closest point in rng(X) to y when
1. y rng(X), and
2. y y rng(X)
Here 1 is true by construction, and 2 translates to claim
y X(X0 X)1 X0 y X
3 Why
for all RK
is it unlikely that y lies in the range of X? Since X is assumed to be full column rank, the
range of X is a K-dimensional subspace of R N , while y is any point in R N . In a sense, for K < N, all
K-dimensional subspaces of R N are small, and the chance of y happening to lie in this subspace
is likewise small. For example, consider the case where N = 3 and K = 2. Then the column space
of X forms a 2 dimensional plane in R3 . Intuitively, this set has no volume because planes have no
thickness, and hence the chance of a randomly chosen y lying in this plane is near zero. More
formally, if y is drawn from a continuous distribution over R3 , then the probability that it falls in this
plane is zero, due to the fact that planes in R3 are always of Lebesgue measure zero.
94
This is true, because if RK , then
(X)0 [y X(X0 X)1 X0 y] = 0 [X0 y X0 X(X0 X)1 X0 y] = 0 [X0 y X0 y] = 0

The proof of theorem 3.2.1 is done.
Notice that theorem 3.2.1 implicitly assumes that X0 X is invertible. This is justified,
however, because X is assumed to be full column rank. (Exercise 3.5.9.)
Remark 3.2.1. On an intuitive level, why do we need full column rank for X? Full
rank means that the columns of X are linearly independent. Lets drop this assumption and consider what happens. The set rng(X) is still a linear subspace, and the
orthogonal projection theorem still gives us a closest point y to y in rng(X). Since
The problems is that now
y rng(X), there still exists a vector such that y = X .
there may exist two such vectorsor even an infinity.
Lets tie this discussion in to theorem 3.1.4 on page 91. We define the projection
matrix P associated with X as
P : = X ( X 0 X ) 1 X 0
(3.1)
We also define the annihilator M associated with X as

M := I P
(3.2)
where I is, as usual, the identity matrix (in this case N N). Given these definitions,
we then have
Py = X(X0 X)1 X0 y = X = y
and
My = (I P)y = y Py = y y
The projection matrix and the annihilator correspond to the two projections P and
M in theorem 3.1.4. P projects onto rng(X), while M projects onto the orthogonal
complement of rng(X). In particular, to find the closest element of rng(X) to a given
vector y in R N , we can just premultiply y by P := X(X0 X)1 X0 .
Fact 3.2.1. Both P and M are symmetric and idempotent.
The proof is an exercise (exercise 3.5.10). Idempotence is rather intuitive here, because both P and M represent orthogonal projections onto linear subspaces. Such
projections map vectors into their respective subspaces. Applying the mapping a
second time has no effect, because the vector is already in the subspace.
95
Fact 3.2.2. The annihilator M associated with X satisfies MX = 0.

The proof is an exercise. (Exercise 3.5.11.) The intuition is as follows: The j-th column of MX is Mx j , where x j is the j-th column of X. Since x j is in rng(X), it gets
mapped into the zero vector by M. This follows from fact 3.1.6 on page 92, but its
also quite intuitive in light of figure 3.6 (where S corresponds to rng(X)).
3.3
Conditioning
The main purpose of this section is to introduce conditional expectations and study
their properties. The definition of conditional expectations given in elementary
probability texts is often cumbersome to work with, and fails to provide the big
picture. In advanced texts, there are several different approaches to presenting conditional expectations. The one I present here is less common than the plain vanila
treatment, but it is, to my mind, by far the most intuitive. As you might expect given
the location of this discussion, the presentation involves orthogonal projection.
3.3.1
The Space L2
Suppose we want to predict the value of a random variable u using another variable
v. In this case wed want u and v to be similar to each other in some sense. Since it
helps to think geometrically, we usually talk about closeness instead of similarity,
but the meaning is the same. A natural measure of closeness is mean squared error
(MSE). The mean squared error of v as a predictor of u is defined as E [(u v)2 ]. For
the purposes of this section, it will be more convenient if we make a slight adjustment, replacing the mean squared error with the root mean squared error (RMSE),
which is, as the name suggests, the square root of the MSE. Since well be using it a
lot, lets give the RMSE its own notation:
q
9u v9 := E [(u v)2 ]
More generally, if we define
9 z9 :=
E [ z2 ]
(3.3)
and regard this as the norm of the random variable z, then the RMSE between u
and v is the norm of the random variable u v.
96
In fact the random variable norm 9 9 defined in (3.3) behaves very much like
the euclidean norm k k over vectors defined in (2.1) on page 51. If z is a vector in
R N and z is a random variable with density f , then the definitions of the two norms
written side by side look pretty similar:
N
kzk =
n =1
!1/2
z2n
and
9 z9 =
Z
1/2
s f (s)ds
More importantly, all the properties of the euclidean norm k k given in fact 2.1.1
(page 53) carry over to then norm 9 9 if we replace vectors with random variables. So lets stop calling 9 9 a norm, and just start calling it a norm.4
Unlike the situation with the euclidean norm, there is a risk here that 9z9 may not
be defined because E [z2 ] = . So for the purposes of this section, lets restrict
attention to random variables with finite second moment. The standard name of
this set of random variables is L2 . That is,
L2 := { all random variables x with E [ x2 ] < }
We can draw another parallel with the euclidean norm. As we saw in 2.1.1, the
euclidean norm is defined in terms of the inner product on R N . If x and y aretwo
vectors in R N , then the inner product is x0 y, and the norm of vector x is kxk = x0 x.
Similarly, for random variables x and y, we define
inner product of x and y := E [ xy]
As for the euclidean case, you can see here that the norm 9x9 of x is precisely the
square root of the inner product of x with itself.
As in the euclidean case, if the inner product of x and y is zero, then we say that
x and y are orthogonal, and write x y. This terminology is used frequently in
econometrics (often by people who arent actually sure why the term orthogonal
is usedwhich puts you one step ahead of them). Clearly, if either x or y is zero
mean, then orthogonality of x and y is equivalent to cov[ x, y] = 0.
4 One
caveat is that while kxk = 0 implies that x = 0, it is not true that 9z9 = 0 implies z is the
zero random variable (i.e., z( ) = 0 for all ). However, we can say that if 9z9 = 0, then the set
E := { : |z( )| > 0} satisfies P( E) = 0. In this sense, z differs from the zero random variable
only in a trivial way.
3.3.2
97
Measurability
Whats the main point of the discussion in the previous section? By providing the
collection of random variables L2 with a norm, weve made it look rather similar to
euclidean vector space R N . The advantage of this is that we have a lot of geometric
intuition about the vector space R N . Since L2 with its norm 9 9 behaves a lot
like R N with its norm k k, that same geometric intuition concerning vectors can
be applied to the study of random variables. For example, we will see that the
orthogonal projection theorem carries over to L2 , and this is precisely how we will
study conditional expectation.
Recall that, in the case of R N , orthogonal projection starts with a linear subspace
S of R N . Once we have this subspace, we think about how to project onto it. In
fact S is the crucial component here, because once we select S, we implictly define
the orthogonal projection mapping P that projects onto S (see theorems 3.1.2 and
3.1.3). So when I tell you that conditional expectation is characterized by orthogonal
projection, you will understand that the first thing we need to think about is the
linear subspaces that we want to project onto. It is to this topic that we now turn.
The first step is a definition at the very heart of probability theory: measurability. Let
x1 , . . . , x p be some collection of random variables, and let G := { x1 , . . . , x p }. Thus,
G is a set of random variables, often referred to in what follows as the information
set. We will say that another random variable z is G -measurable if there exists a
(nonrandom) function g : R p R such that
z = g ( x1 , . . . , x p )
Informally, what this means is that once the values of the random variables x1 , . . . , x p
have been realized, the variable z is completely determined (i.e., no longer random)
and its realized value can be calculated (assuming that we can calculate the functional form g). You might like to imagine it like this: Uncertainty is realized, in the
sense that some is selected from the sample space . Suppose that we dont get
to view itself, but we do get to view certain random outcomes. For example, we
might get to observe the realized values x1 ( ), . . . , x p ( ). If z is G -measurable, we
can now calculate the realized value z( ) of z, even without knowning , because
we can compute z( ) = g( x1 ( ), . . . , x p ( )).5
5A
technical note: In the definition of measurability above, where we speak of existence of the
function g, it is additional required that the function g is Borel measurable. For the purposes of
this course, we can regard non-Borel measurable functions as a mere theoretical curiosity. As such,
the distinction will be ignored. See any text on measure theory for further details.
98
As a matter of notation, if G = { x } and y is G -measurable, then we will also say that

y is x-measurable.
Example 3.3.1. Let x and z be two random variables. If z = 2x + 3, then z is xmeasurable. To see this formally, we can write z = g( x ) when g( x ) = 2x + 3. Less
formally, when x is realized, the value of z can be calculated.
Example 3.3.2. Let x1 , . . . , x N be random variables and let x N be their sample mean.
If G = { x1 , . . . , x N }, then x N := N 1 nN=1 xn is clearly G -measurable.
Example 3.3.3. If x and y are independent, then y is not x-measurable. Indeed,
if y was x-measurable, then we would have y = g( x ) for some function g. This
contradicts independence of x and y.
Example 3.3.4. Let x, y and z be three random variables with z = x + y. Suppose that
x and y are independent. Then z is not x-measurable. Intuitively, even if we know
the realized value of x, the realization of z cannot be computed until we know the
realized value of y. Formally, if z is x-measurable then z = g( x ) for some function
g. But then y = g( x ) x, so y is x-measurable. This contradicts independence of x
and y.
Example 3.3.5. Let y = , where is a constant. This degenerate random variable
is G -measurable for any information set G , because y is already deterministic. For
p
example, if G = { x1 , . . . , x p }, then we can take y = g( x1 , . . . , x p ) = + i=1 0xi .
If x and y are known given the information in G , then a third random variable that
depends on only on x and y is likewise known given G . Hence G -measurability is
preserved under the taking of sums, products, etc. In particular,
Fact 3.3.1. Let , be any scalars, and let x and y be random variables. If x and y are
both G -measurable, then u := xy and v := x + y are also G -measurable.
Let G and H be two information sets with G H. In this case, if random variable z is
G measurable, then it is also H-measurable. This follows from our intuitive definition
of measurability: If the value z is known once the variables in G are known, then it
is certainly known when the extra information provided by H is available. The next
example helps to clarify.
Example 3.3.6. Let x, y and z be three random variables, let G = { x }, and let
H = { x, y}. Suppose that z = 2x + 3, so that z is G -measurable. Then z is also
H-measurable. Informally, we can see that z is deterministic once the variables in H
are realized. Formally, we can write z = g( x, y), where g( x, y) = 2x + 3 + 0y. Hence
z is also H-measurable as claimed.
99
Lets note this idea as a fact:

Fact 3.3.2. If G H and z is G -measurable, then z is H-measurable.
We started off this section by talking about projecting onto linear subspaces. Recall
that S R N is called a linear subspace of R N if, given arbitrary scalars , and
vectors x, y in S, the linear combination x + y is again in S. Similarly S L2 is
called a linear subspace of L2 if, given arbitrary scalars , and random variables
x, y in S, the random variable x + y is also in S.
For conditional expectations, the subspaces of interest are the subspaces of measurable random variables. In particular, given G L2 , we define
L2 (G) := {the set of all G -measurable random variables in L2 }
In view of fact 3.3.1, we have the following important result:
Fact 3.3.3. For any G L2 , the set L2 (G) is a linear subspace of L2 .
From fact 3.3.2 we see that, in the sense of set inclusion, the linear subspace is increasing with respect to the information set.
Fact 3.3.4. If G H, then L2 (G) L2 (H).
3.3.3
Conditional Expectation
Now its time to define conditional expectations. Let G L2 and y be some random variable in L2 . The conditional expectation of y given G is written as E [y | G]
or E G [y], and defined as the closest G -measurable random variable to y.6 More
formally,
E [y | G] := argmin 9y z9
(3.4)
z L2 (G)
This definition makes a lot of sense. Our intuitive understanding of the conditional
expectation E [y | G] is that it is the best predictor of y given the information contained in G . The definition in (3.4) says the same thing. It simultaneously restricts
E [y | G] to be G -measurable, so we can actually compute it once the variables in G
are realized, and selects E [y | G] as the closest such variable to y in terms of RMSE.
prefer the notation E G [y] to E [y | G] because, as we will see, E G is a function (an orthogonal
projection) from L2 to L2 , and the former notation complements this view. However, the notation
E [y | G] is a bit more standard, so thats the one well use.
6I
100
While the definition makes sense, it still leaves many open questions. For example,
there are many situations where minimizers dont exist, or, if the do exist, there are
lots of them. So is our definition really a definition? Moreover, even assuming we
do have a proper definition, how do we actually go about computing conditional expectations in practical situations? And what properties do conditional expectations
have?
These look like tricky questions, but fortunately the orthogonal projection theorem
comes to the rescue. The orthogonal projection theorem in L2 is almost identical to
the orthogonal projection theorem we gave for R N . Given a linear subspace S of L2
and a random variable y in L2 , there is a unique y S such that
9y z 9 for all z S
9y y9
The variable y S is called the orthogonal projection of y onto S.7 Just as for the
R N case, the projection is characterized by two properties:
y is the orthogonal projection of y onto S if and only if y S and y y S
As for R N , we can think of y 7 y as a function, which we denote by P, so that Py is
the orthogonal projection of y onto S for arbitrary y L2 . Moreover, P satisfies all
the properties in theorem 3.1.3 (page 88). Lets state this as a theorem for the record.
Theorem 3.3.1. Given a linear subspace S of L2 , the function
Py := argmin 9y z9
(3.5)
zS
is a well-defined linear function from L2 to S. Given any y L2 , we have

1. Py S,
2. y Py S, and
3. Py = y if and only if y S.
7 There
are two small caveats I should mention. First, we actually require that S is a closed
linear subspace of L2 , which means that if { xn } S, x L2 and 9xn x9 0, then x S. For the
subspaces we consider here, this condition is always true. Second, when we talk about uniqueness in
L2 , we do not distinguish between elements x and x 0 of L2 such that P{ x = x 0 } = 1. A nice treatment
of orthogonal projection in Hilbert spaces (of which L2 is one example) is provided in Cheney (2001,
chapter 2). Most other books covering Hilbert space will provide some discussion.
101
Comparing (3.4) and (3.5), we see that y 7 E [y | G] is exactly the orthogonal projection function P in the special case where the subspace S is the G -measurable functions L2 (G).
Okay, so E [y | G] is the orthogonal projection of y onto L2 (G). Thats kind of neat,
but what does it actually tell us? Well, it tells us quite a lot. For starters, theorem 3.3.1 implies that E [y | G] is always well defined and unique. Second, it gives
us a useful characterization of E [y | G], because we now know that E [y | G] is the
unique point in L2 such that E [y | G] L2 (G) and y E [y | G] z for all z L2 (G).
Rewriting these conditions in a slightly different way, we can give an alternative
(and equivalent) definition of conditional expectation: E [y | G] L2 is the conditional expectation of y given G if
1. E [y | G] is G -measurable, and
2. E [E [y | G] z] = E [yz] for all G -measurable z L2 .
This definition seems a bit formidable, but its not too hard to use. Before giving an
application, lets bow to common notation and define
E [y | x1 , . . . , x p ] := E [y | G]
Also, lets note the following obvious fact:
Fact 3.3.5. Given { x1 , . . . , x p } and y in L2 , there exists a function g : R p R such
that E [y | x1 , . . . , x p ] = g( x1 , . . . , x p ).
This is obvious because, by definition, E [y | G] is G -measurable. At the same time,
its worth keeping in mind: A conditional expectation with respect to a collection of
random variables is some function of those random variables.
Example 3.3.7. If x and w are independent and y = x + w, then E [y | x ] = x + E [w].
Lets check this using the second definition of conditional expectations given above.
To check that x + E [w] is indeed the conditional expectation of y given G = { x }, we
need to show that x + E [w] is x-measurable and that E [( x + E [w]) z] = E [yz] for
all x-measurable z. The first claim is clearly true, because x + E [w] is a deterministic
function of x. The second claim translates to the claim that
E [( x + E [w]) g( x)] = E [( x + w) g( x)]
(3.6)
102
for any function g. Verifying this equality is left as an exercise (exercise 3.5.12)
The next example shows that when x and y are linked by a conditional density (remember: densities dont always exist), then our definition of conditional expectation
reduces to the one seen in elementary probability texts. The proof of the claim in the
example is the topic of exercise 3.5.17.
Example 3.3.8. If x and y are random variables and p(y | x ) is the conditional density
of y given x, then
Z
E [y | x ] =
tp(t | x )dt
There are some additional goodies we can harvest using the fact that conditional
expectation is an orthogonal projection.
Fact 3.3.6. Let x and y be random variables in L2 , let and be scalars, and let G
and H be subsets of L2 . The following properties hold.
1. Linearity: E [x + y | G] = E [ x | G] + E [y | G].
2. If G H, then E [E [y | H] | G] = E [y | G] and E [E [y | G]] = E [y].
3. If y is independent of the variables in G , then E [y | G] = E [y].
4. If y is G -measurable, then E [y | G] = y.
5. If x is G -measurable, then E [ xy | G] = xE [y | G].
Checking of these facts is mainly left to the exercises. Most are fairly straightforward. For example, consider the claim that if y is G -measurable, then E [y | G] = y.
In other words, we are saying that if y L2 (G), then y is projected into itself. This
is immediate from the last statement in theorem 3.3.1.
The fact that if G H, then E [E [y | H] | G] = E [y | G] is called the tower property
of conditional expectations (by mathematicians), or the law of iterated expectations
(by econometricians). The law follows from the property of orthogonal projections
given in fact 3.1.2 on page 90: Projecting onto the bigger subspace L2 (H) and from
there onto L2 (G) is the same as projecting directly onto the smaller subspace L2 (G).
3.3.4
103
The Vector/Matrix Case
Conditional expectations of random matrices are defined using the notion of conditional expectations for scalar random variables. For example, given random matrices X and Y, we set
E [y11 | X] E [y12 | X] E [y1K | X]

E [y | X] E [y22 | X] E [y2K | X]
21
E [Y | X] :=
..
..
..
.
.
.
E [y N1 | X] E [y N2 | X] E [y NK | X]
where
E [ynk | X] := E [ynk | x11 , . . . , x`m , . . . , x LM ]
We also define
and
cov[x, y | Z] := E [xy0 | Z] E [x | Z]E [y | Z]0

var[x | Z] := E [xx0 | Z] E [x | Z]E [x | Z]0
Using the definitions, one can show that all of the results on conditional expectations
in fact 3.3.6 continue to hold in the current setting, replacing scalars with vectors and
matrices. We state necessary results for convenience:
Fact 3.3.7. Let X, Y and Z be random matrices, and let A and B be constant matrices.
Assuming conformability of matrix operations, the following results hold:
1. E [Y | Z]0 = E [Y0 | Z].
2. E [AX + BY | Z] = AE [X | Z] + BE [Y | Z].
3. E [E [Y | X]] = E [Y] and E [E [Y | X, Z] | X] = E [Y | X].
4. If X and Y are independent, then E [Y | X] = E [Y].
5. If g is a (nonrandom) function, so that g(X) is a matrix depending only on X,
then
E [ g(X) | X] = g(X)
E [ g ( X ) Y | X ] = g ( X )E [ Y | X ]
E [Y g(X) | X] = E [Y | X] g(X)
3.3.5
104
An Exercise in Conditional Expectations
Let x and y be two random variables. We saw that E [y | x ] is a function f of x

such that f ( x ) is the best predictor of y in terms of root mean squared error. Since
monotone increasing transformations do not affect minimizers, f also minimizes the
mean squared error. In other words, f solves
min E [(y g( x ))2 ]
g G
(3.7)
where G is the set of functions from R to R. From this definition of conditional

expectations, we employed the orthogonal projection theorem to deduce various
properties of conditional expectations. We can also reverse this process, showing
directly that f ( x ) := E [y | x ] solves (3.7), given the various properties of conditional
expectations listed in fact 3.3.6. To begin, suppose that the properties in fact 3.3.6
hold, and fix an arbitrary g G. We have
(y g( x ))2 = (y f ( x ) + f ( x ) g( x ))2
= (y f ( x ))2 + 2(y f ( x ))( f ( x ) g( x )) + ( f ( x ) g( x ))2
Lets consider the expectation of the cross-product term. From the law of iterated
expectations (fact 3.3.6), we obtain
E {(y f ( x))( f ( x) g( x))} = E { E [(y f ( x))( f ( x) g( x)) | x] }
(3.8)
We can re-write the term inside the curly brackets on the right-hand side of (3.8) as
( f ( x ) g( x ))E [(y f ( x )) | x ]
(Which part of fact 3.3.6 are we using here?) Regarding the second term in this
product, we have (by which facts?) the result
E [y f ( x ) | x ] = E [y | x ] E [ f ( x ) | x ] = E [y | x ] f ( x ) = E [y | x ] E [y | x ] = 0
We conclude that the expectation in (3.8) is E [0] = 0. It then follows that
E [(y g( x))2 ] = E [(y f ( x))2 + 2(y f ( x))( f ( x) g( x)) + ( f ( x) g( x))2 ]

= E [(y f ( x ))2 ] + E [( f ( x ) g( x ))2 ]
Since ( f ( x ) g( x ))2 0 we have E [( f ( x ) g( x ))2 ] 0, and we conclude that
E [(y g( x))2 ] E [(y f ( x))2 ] :=: E [(y E [y | x])2 ]

Since g was an arbitrary element of G, we conclude that
f = argmin E [(y g( x ))2 ]
g G
3.4
105
Further Reading
To be written.
3.5
Exercises
Ex. 3.5.1. Prove the Pythagorean law in theorem 3.1.1.8

Ex. 3.5.2. Prove theorem 3.1.4 using theorems 3.1.23.1.3.
Ex. 3.5.3. Prove fact 3.1.4: If S R N , then S S = {0}.
Ex. 3.5.4. Prove fact 3.1.2.
Ex. 3.5.5. Let x and y be any two N 1 vectors.
1. Show that kx + yk2 = kxk2 + kyk2 + 2x0 y
2. Explain the connection between this equality and the Pythagorean Law.
Ex. 3.5.6. Let P be the orthogonal projection described in theorem 3.1.3 (page 88).
Confirm that P is a linear function from R N to R N , as defined in 2.1.3.
Ex. 3.5.7. Let S := {( x1 , x2 , x3 ) R3 : x3 = 0}, and let y := 1 := (1, 1, 1). Using the
orthogonal projection theorem, find the closest point in S to y.
Ex. 3.5.8. Let P be the orthogonal projection described in theorem 3.1.3 (page 88). Is
it true that Px 6= Py whenever x 6= y? Why or why not?9
Ex. 3.5.9. Show that when N K matrix X is full column rank, the matrix X0 X is
invertible.10
Ex. 3.5.10. Show by direct computation that P and M in (3.1) and (3.2) are both
symmetric and idempotent.
Ex. 3.5.11. Verify fact 3.2.2 (i.e., MX = 0) directly using matrix algebra.
8 Hint:
See fact 2.2.1.

Sketch the graph and think about it visually.
10 Hint: This is non-trivial. In view of fact 2.3.11, it suffices to show that X0 X is positive definite.
Make use of the full column rank assumption. Look at the different equivalent conditions for linear
independence of a set of vectors.
9 Hint:
106
Ex. 3.5.12. Show that the equality in (3.6) holds when x and w are independent.
Ex. 3.5.13. In fact 3.3.6, it is stated that if y is independent of the variables in G ,
then E [y | G] = E [y]. Prove this using the (second) definition of the conditional
expectation E [y | G]. To make the proof a bit simpler, you can take G = { x }.
Ex. 3.5.14. Confirm the claim in fact 3.3.6 that if x is G -measurable, then E [ xy | G] =
xE [y | G].
Ex. 3.5.15. Let var[y | x ] := E [y2 | x ] (E [y | x ])2 . Show that
var[y] = E [var[y | x ]] + var[E [y | x ]]
Ex. 3.5.16. Show that the conditional expectation of a constant is . In particular,
using the results in fact 3.3.6 (page 102) as appropriate, show that if is a constant
and G is any information set, then E [ | G] = .
Ex. 3.5.17. Prove the claim in example 3.3.8. (Warning: The proof is a little advanced
and you should be comfortable manipulating double integrals.)
3.5.1
Solution to Exercise 3.5.3. Let S R N . We aim to show that S S = {0}. Fix

a S S . Since a S , we know that a0 s = 0 for any s S. Since a S, we have
in particular, a0 a = kak2 = 0. As we saw in fact 2.1.1, the only such vector is 0.
Solution to Exercise 3.5.6. Fix , R and x, y R N . The claim is that
P(x + y) = Px + Py
To verify this equality, we need to show that the right-hand side is the orthogonal
projection of x + y onto S. Going back to theorem 3.1.2, we need to show that (i)
Px + Py S and (ii) for any z S, we have
(x + y (Px + Py))0 z = 0
Here (i) is immediate, because Px and Py are in S by definition; and, moreover S is
a linear subspace. To see that (ii) holds, just note that
(x + y (Px + Py))0 z = (x Px)0 z + (y Py)0 z

By definition, the projections of x and y are orthogonal to S, so we have (x Px)0 z =
(y Py)0 z = 0. We are done.
107
Solution to Exercise 3.5.7. Let x = ( x1 , x2 , x3 ) be the closest point in S to y. Note

that e1 S and e2 S. By the orthogonal projection theorem we have (i) x S, and
(ii) y x S. From (i) we have x3 = 0. From (ii) we have
hy x, e1 i = 0 and hy x, e2 i = 0
These equations can be expressed more simply as 1 x1 = 0 and 1 x2 = 0. We
conclude that x = (1, 1, 0).
Solution to Exercise 3.5.8. It is false to say that Px 6= Py whenever x 6= y: We can
find examples of vectors x and y such that x 6= y but Px = Py. Indeed, if we fix any
y and then set x = Py + My for some constant , you should be able to confirm
that Px = Py, and also that x 6= y when 6= 1.
Solution to Exercise 3.5.9. Let A = X0 X. It suffices to show that A is positive definite, since this implies that its determinant is strictly positive, and any matrix with
nonzero determinant is invertible. To see that A is positive definite, pick any b 6= 0.
We must show that b0 Ab > 0. To see this, observe that
b0 Ab = b0 X0 Xb = (Xb)0 Xb = kXbk2
By the properties of norms, this last term is zero only when Xb = 0. But this is not
true, because b 6= 0 and X is full column rank (see fact 2.2.4, part 5).
Solution to Exercise 3.5.12. Let g be any function from R R. Given independence of x and w (and applying fact 1.3.2 on page 27), we have
E [( x + E [w]) g( x)] = E [ xg( x)] + E [w]E [ g( x)]

= E [ xg( x )] + E [wg( x )]
= E [( x + w) g( x )]
This confirms (3.6).
Solution to Exercise 3.5.13. Let y be independent of x. From the (second) definition
of conditional expectation, to show that E [y | x ] = E [y] we need to show that
1. E [y] is G -measurable, and
2. E [E [y] g( x )] = E [yg( x )] for any function g : R R.
108
Part 1 is immediate, because E [y] is constant (see example 3.3.5 on page 98). Regarding part 2, if g is any function, then by facts 1.3.1 and 1.3.2 (see page 27) we have
E [yg( x)] = E [y]E [ g( x)]. By linearity of expectations, E [y]E [ g( x)] = E [E [y] g( x)].
Solution to Exercise 3.5.14. We need to show that if x is G -measurable, then E [ xy | G] =
xE [y | G]. To confirm this, we must show that
1. xE [y | G] is G -measurable, and
2. E [ xE [y | G]z] = E [ xyz] for any z L2 (G).
Regarding part 1, E [y | G] is G -measurable by definition, and x is G -measurable by
assumption, so xE [y | G] is G -measurable by fact 3.3.1 on page 98. Regarding part
2, fix z L2 (G), and let u := xz. Since x L2 (G), we have u L2 (G). We need to
show that
E [E [y | G]u] = E [yu]
Since u L2 (G), this is immediate from the definition of E [y | G].
Solution to Exercise 3.5.16. By fact 3.3.6 (page 102), we know that if is G -measurable,
then E [ | G] = . Example 3.3.5 on page 98 tells us that is indeed G -measurable.
Solution to Exercise 3.5.17. As in example 3.3.8, let x and y be random variables

R
where p(y | x ) is the conditional density of y given x. Let g( x ) := tp(t | x )dt.
The claim is that E [y | x ] = g( x ). To prove this, we need to show that g( x ) is xmeasurable, and that
E [ g( x)h( x)] = E [yh( x)] for any function h : R R
(3.9)
The first claim is obvious. Regarding (3.9), let h be any such function. Using the
notation in (1.20) on page 26, we can write
Z

E [ g( x)h( x)] = E
tp(t | x )dt h( x )
=
=
=
Z Z
Z Z
Z Z
tp(t | s)dt h(s) p(s)ds

t
p(s, t)
dt h(s) p(s)ds
p(s)
t h(s) p(s, t)dtds
This is equal to the right-hand side of (3.9), and the proof is done.
Part II
Foundations of Statistics
109
Chapter 4
Statistical Learning
Econometrics is just statistics applied to economic problemsnothing more and
nothing less. We should probably call it statistical economics, but I guess people feel that the term econometrics has a better ring to it. The only cost of using
the term econometrics is that we are sometimes fooled into thinking that we work
on a distinct discipline, separate from statistics. This is not true.
The next two chapters provides a short, concise review of the foundations of modern statistics, including parametric and nonparametric methods, empirical distributions, hypothesis testing and confidence intervals.
4.1
Inductive Learning
In the modern world we have lots of data, but still lack fundamental knowledge
on how many systems work, or how different economic variables are related to one
another. What then is the process of extracting knowledge from data? Under what
conditions will this process be successful?
4.1.1
Generalization
The fundamental problem of statistics is learning from data. Learning from data
concerns generalization. A finite set of data is observed, and, on the basis of this data,
one seeks to make more general statements. For example, suppose that a certain
drug is tested on 1,000 volunteers, and found to produce the desired effect in 95%
110
CHAPTER 4. STATISTICAL LEARNING
111
of cases. On the basis of this study, the drug company claims that the drug is highly
effective. The implication of their claim is that we can generalize to the wider population. The interest is not so much in what happened to the volunteers themselves,
but rather on what the outcome for the volunteers implies for other people.
Another word for generalization is induction. Inductive learning is where reasoning
proceeds from the specific to the generalas opposed to deductive learning, which
proceeds from general to specific.
Example 4.1.1. You show a child pictures of dogs in a book and say dog. After a
while, the child sees a dog on the street and says dog. The child has generalized
from specific examples. Hence, the learning is inductive. If, on the other hand, you
had told the child that dogs are hairy, four legged animals that stick their tongues
out when hot, and the child determined the creature was a dog on this basis, then
the nature of the learning process could be called deductive.
Here are some typical statistical problems, phrased in more mathematical language:
Example 4.1.2. N random values x1 , . . . , x N are drawn from a given but unknown
cdf F. We wish to learn about F from this sample.
Example 4.1.3. Same as example 4.1.2, but now we only care about learning the
mean of For the standard deviation, or the median, etc.
Example 4.1.4 (Regression). We observe inputs x1 , . . . , x N to some system, as
well as the corresponding outputs y1 , . . . , y N . Given this data, we wish to compute
a function f such that, given a new input/output pair ( x, y), the value f ( x ) will be
a good guess of the corresponding output y. (Here we imagine that y is observed
after x or not at all, and hence the need to predict y from x.)
In these examples, the problem lies in the fact that we do not know the underlying
distributions. If, in example 4.1.4, we knew the joint distribution of ( x, y) pairs, then
we could work out the conditional expectation E [y | x ]. As well see in 3.3.3, there
is a natural sense in which the conditional expectation is the best predictor of y given
x. In statistical applications, however, we dont know the distributions. All we have
is the observations. We must do the best we can given the information contained in
this sample.
112
output
0.65
0.70
0.75
0.60
0.55
0.50
0.0
0.2
0.4
0.6
0.8
1.0
input
Figure 4.1: Generalization requires knowledge
4.1.2
Data is not Enough
As a rule, statistical learning requires more than just data. Ideally, data is combined with a theoretical model that encapsulates our knowledge of the system we
are studying. The data is often used to pin down parameter values for the model.
This is called fitting the model to the data. If our model is good, then combining
model with data allows to gain an understanding of how the system works.
Even when we have no formal model of how the system works, we still need to
combine the data with some assumptions in order to generalize. Figure 4.1 helps to
illustrate this idea. Consider the regression setting of example 4.1.4, and suppose
we observe the blue dots as our data. Now make a subjective guess as to the likely
value of the output, given that the input value is 0.8. Was your guess something like
the red dot in Figure 4.2? It looks reasonable to me too.
But why does it look reasonable? Because our brain picks up a pattern: The blue
dots lie roughly on a straight line. We instictively predict that the red dot will lie
on the same line, or at least we feel it would be natural for that to occur. One way
or another, our brains have been trained (or are hard-wired?) to think in straight
lines. And even though this thought process is subconscious, in the end what we
are doing is bringing our own assumptions into play.
113
0.70
0.75
output
0.65
0.60
0.55
0.50
0.0
0.2
0.4
0.6
0.8
1.0
input
Figure 4.2: Generalization requires knowledge
Obviously our assumption about the linear relationship could be completely wrong.
After all, we havent even talked about the kind of system we are observing here.
Maybe the functional relationship between inputs and outputs is totally different
to what we perceived from these few data points. Ideally, our assumptions should
be based on sound theory and understanding of the system we are studying, rather
than some subconscious feeling that straight lines are most likely.1
Either way, regardless of the process that led to our assumptions, the point is that
we cannot forecast the new observation from the data alone. We have to make some assumptions as to the functional relationship in order to come up with a guess of
likely output given input 0.8. Those assumptions may come from knowledge of the
system, or they may come from subconscious preference for straight lines. Either
way, we are adding something to the data in order to make inference about likely
outcomes.
If the assumptions we add to the data are to some extent correct, this injection of
prior knowledge into the learning process allows us to generalize from the observed
1 In 1929, the economist Irving Fisher famously declared that Stocks have reached what looks like
a permanently high plateau. Perhaps Dr Fisher based his projection on subconscious attraction to
straight lines, rather than some deeper understanding of the underlying forces generating the time
series of equity prices he observed.
114
data points. Thus, roughly speaking, the rule is

statistical learning = prior knowledge + data
4.2
Statistics
Statistics sounds like an odd name for a section. Isnt this whole course about
statistics? Yes, sure it is, but here were using the term statistic with a special meaning. Specifically, a statistic is any function of a given data set. To repeat:
A statistic is an observable function of the sample data.
For example, suppose that we have data x1 , . . . , x N , which might represent the price
of a Big Mac in N different countries, or the closing price of one share in Google over
N consecutive days. Common statistics used to summarize the data are the sample
mean
1 N
x N :=: x :=
xn
N n
=1
the sample variance
s2N
N
1
:=: s :=
( xn x )2
N 1 n =1
2
(4.1)
and the sample standard deviation
s N :=: s :=
"
s2 =
N
1
( xn x )2
N 1 n
=1
#1/2
(4.2)
For positive integer k, the sample k-th moment is given by

1
N
xnk
n =1
If we have bivariate data ( x1 , y1 ), . . . , ( x N , y N ), then the sample covariance is

N
1
( xn x )(yn y )
N 1 n
=1
(4.3)
115
and the sample correlation is the sample covariance divided by the product of the
two sample standard deviations. With some rearranging, this becomes
nN=1 ( xn x )(yn y )
q
nN=1 ( xn
x )2 nN=1 (yn
(4.4)
y )2
R has functions for all of these common statistics. The sample mean, sample variance and sample standard deviation are calculated using the functions mean, var
and sd respectively. Sample covariance and sample correlation can be computed
using cov and cor:
> x <- rnorm (10)
> y <- rnorm (10)
> cov (x , y )
[1] 0.001906421
> cor (x , y )
[1] 0.004054976
Perhaps the most important thing to remember about statistics is that, being functions of the sample, they are also random variables. This might not be clear, since, we
tend to think of the data as a fixed set of numbers in a file on our hard disk, determined by some previous historical outcome. Statistics are deterministic functions of
these numbers, and we only observe one value of any particular statisticone sample mean, one sample variance, etc. However, the way that statisticians think about
it is that they imagine designing the statistical exercise prior to observing the data. At
this stage, the data is regarded as a collection of random variableseven though
these variables may have been previously determined in some historical data set.
Hence, each statistic is also a random quantity (i.e., random variable).
More formally, if we look at the sample mean, for example, when we write x N :=
N 1 nN=1 xn , what we actually mean is
x N ( ) :=
1
N
xn ( )
( )
(4.5)
n =1
Hence x N is a function from R. Put differently, x N is a random variable.

Being random variables, statistics have expectations, variances and so on. For example, consider the sample mean x := N 1 n xn of an identically distributed sample.
Each observation xn is drawn from some fixed distribution F with unknown mean
116
. In that case, the mean of x is also , because, from linearity of expectations,

"
#
1 N
1 N
E [ x ] = E
xn =
E [ xn ] =
(4.6)
N n
N n
=1
=1
Since we dont actually know what is, this result might not seem very helpful, but
it is: It tells us that x is a useful predictor of the unknown quantity , in the sense
that its most likely outcome is this unknown quantity. We say that x is unbiased
for . The next session discusses this and other properties of estimators.
4.2.1
Vector Valued Statistics
Statistics can be vector valued, or even matrix valued. For example, if x1 , . . . , x N are
random vectors of equal length, then the sample mean is the random vector defined
by
1 N
x :=
xn
N n
=1
and the sample variance-covariance matrix is defined as
Q :=
N
1
[(xn x )(xn x )0 ]
N 1 n =1
(4.7)
In R, the sample variance-covariance matrix is obtained using cov, which acts on

matrices. In relation to Q in (4.7), the observed vectors x1 , . . . , x N are treated as row
vectors of the matrix. This is a standard convention in statistics: rows of a matrix are
observations of a random vector. Hence, to obtain the sample variance covariance
matrix Q of observations x1 , x2 , x3 , we stack them into rows and use cov:
> cov ( rbind ( x1 , x3 , x2 ) )
More commonly, we will be working with a data matrix X where the rows are observations, and we can just use cov(X).
4.2.2
Estimators and their Properties
Estimators are just statisticsthat is, functions of the data. However, when we
talk about estimators, we have in mind the idea of estimating a specific quantity
117
of interest. In other words, to discuss an estimator, we need to also specify what

it is were trying to estimate. For example, suppose we wish to estimate the mean
R
:= s F (ds) of F given observations x1 , . . . , x N from F. A common way to do
this is to use the sample mean x of the observations x1 , . . . , x N . In this setting, x is
regarded as an estimator of .
In any given problem, there are always many estimators we can use. For example,
when estimating the mean in the preceding problem we can also use the so-called
mid-range estimator
minn xn + maxn xn
m N :=
2
Which is better, the sample mean or the mid-range estimator?
More generally, let us consider the problem of what makes a good estimator. Not
surprisingly, that depends on how you define good. To begin the discussion, first
we collect some terminology. Let be an estimator of . The bias of is defined as
bias[] := E []
(4.8)
The estimator is called unbiased for if its bias is zero, or E [] = . The mean
squared error of a given estimator of some fixed quantity is
mse[] := E [( )2 ]
(4.9)
Low mean squared error means that probability mass is concentrated around .2
Example 4.2.1. As we saw in (4.6), if x1 , . . . , x N is identically distributed, then the
sample mean x is always an unbiased estimator of the mean. As a result, the mean
squared error of x is equal to the variance. If, in addition, the random variables
x1 , . . . , x N in question are uncorrelated, then the variance of x is
"
#
1 N
2
xn =
(4.10)
var[ x ] = var
N n =1
N
where 2 :=
(s )2 F (ds) is the common variance of each xn .
Example 4.2.2. For an IID sample, the sample variance s2 is an unbiased estimator
of the variance (exercise 4.8.2).
2 In
the definition of mean squared error, we are implicitly assuming that the expression on the
right hand side of (4.9) is finite. This may not be true for certain estimators, simply because the
second moment of the estimator = .
118
2
Figure 4.3: Unbiased estimators

Example 4.2.3. The mid-range estimator m N may be biased as an estimator of the
mean of a distribution, depending on the distribution in question. Bias in the lognormal case is illustrated in the following simulation:
> mr <- function ( x ) return (( min ( x ) + max ( x ) ) / 2)
> observations <- replicate (5000 , mr ( rlnorm (20) ) )
> mean ( observations )
# Sample mean of mid - range
[1] 3.800108
> exp (1 / 2)
# Mean of lognormal
[1] 1.648721
By the LLN, the sample mean of m N is close to E [m N ]. This is clearly a long way
from the mean of the lognormal density.
low variance is desirable. Low variance means that
For an unbiased estimator ,
probability mass for the random variable is concentrated around its mean (see
figure 4.3). This is precisely what we want, since the mean of an unbiased estimator
is the quantity that we wish to estimate.
One issue here is that low variance is a bit hard to quantify. For example, consider
the variance of the sample mean, as given in (4.10). Is that low or is it not? One
way to approach this kind of question is to take the class of unbiased estimators of
a given quantity , and find the estimator in the class with the lowest variance. For
given and given data x1 , . . . , x N , the estimator in the set of unbiased estimators
U := {all statistics with E [] = }
that has the lowest variance within this class (i.e., set) is called the minimum variance unbiased estimator.
119
While minimum variance unbiased estimators are nice in theory, its certainly possible that no minimizer exists. Second, even if such an estimator does exist in this
class, it may be hard to determine in practice. Hence, there is a tendency to focus on
smaller classes than U , and find the best estimator in that class. For example, the
estimator in the set of linear unbiased estimators
U` := {all linear statistics with E [] = }
with the lowest varianceif it existsis called the best linear unbiased estimator, or BLUE. Here linear means that is a linear function of the data x1 , . . . , x N .
Linearity will be defined formally in 2.1.3. For now lets just look at an example.
IID
Example 4.2.4. Let x1 , . . . , x N F, where F has finite mean 6= 0 and variance 2 .

Take it on trust for now that the set of linear estimators of is given by
(
)
N
all statistics of the form =
n xn , where n R for n = 1, . . . , N
n =1
Hence, the set linear unbiased estimators of is given by

(
U` :=
all =
n xn with n R,
n = 1, . . . , N and E
"
n =1
n xn
n =1
Using linearity of expectations, we see that this set can be re-written as

(
)
N
U` :=
all =
n xn with
n =1
n = 1
n =1
By fact 1.3.9 on page 28, the variance of an element of this class is given by
"
#
N
var
n =1
n xn =
2n var[ xn ] + 2
n =1
n m cov[ xn , xm ] = 2
n<m
2n
n =1
where the last equality is due to independence. To find the BLUE, we need to solve
minimize 2
2n
over all 1 , . . . , N with
n =1
n = 1
n =1
To solve this constrained optimization problem, we can use the Lagrangian, setting
"
#
L ( 1 , . . . , N ; ) : = 2
n =1
2n
n 1
n =1
120
E [1 ] =
E [2 ]
Figure 4.4: Biased and unbiased estimators

where is the Lagrange multiplier. Differentiating with respect to n and setting
the result equal to zero, the minimizer n satisfies
n = /(22 )
n = 1, . . . , N
In particular, each n takes the same value, and hence, from the constraint n n = 1,
we have n = 1/N. Using these values, our estimator becomes
N
n =1
n xn =
(1/N )xn = x
n =1
We conclude that, under our assumptions, the sample mean is the best linear unbiased estimator of .
Returning to the general case, note that while classical statistics puts much emphasis
on unbiased estimators, in recent years the use of biased estimators has become
very common. To understand why, its important to bear in mind that what we
seek is an estimator of that is close to with high probability, and far away with
low probability. In this sense, an unbiased estimator is not necessarily better than
a biased one. For example, consider the two estimators 1 and 2 of depicted
figure 4.4. The unbiased estimator 1 has higher variance than the biased estimator
2 , and it is not clear that its performance will be better.
An overall measure of performance that takes into account both bias and variance is
mean squared error, as defined in (4.9). Indeed, we can (exercise 4.8.1) decompose
mean squared error into the sum of variance and squared bias:
mse[] = var[] + bias[]2
(4.11)
In many situations we find a trade-off between bias and variance: We can lower
variance at the cost of extra bias and vice-versa.
4.2.3
121
Asymptotic Properties
Notice in (4.10) how, under the IID assumption, the variance of the sample mean
converges to zero in N. Since the sample mean is unbiased for the mean, this suggests that in the limit, all probability mass concentrates on the meanwhich is the
value that the sample mean seeks to estimate. This is a useful piece of information.
In order to formalize it, as well as generalize to other estimators, lets now consider
asymptotic properties.
Let { N } RK be a vector valued sequence of estimators, designed to estimate
vector RK . Here N denotes the size of the sample from which N is constructed.
We say that N is
asymptotically unbiased for if E [ N ] as N
p
consistent for if N as N
d
asymptotically normal if N ( N ) N (0, ) for some positive definite
matrix .
In the last definition, is called the asymptotic variance-covariance matrix of N .

Of the preceding three definitions, consistency and asymptotic normality are by far
the most important.
Note that asymptotic normality implies consistency. Exercise 4.8.10 asks you to confirm this. In fact asymptotic normality is considerably stronger, since it gives us not
only convergence of N to , but also some idea of the rate at which this
occurs. To
understand the idea, observe that

if N is asymptotically normal, then N ( N )
does not diverge to infinity. Since N , this means that not only does the term
N go to zero, but in addition it goes fast enough to offset the diverging term
N. To emphasize
this point, we sometimes say that an asymptotically normal
estimator N is N-consistent.
In addition, asymptotic normality provides a means of performing inferenceforming
confidence intervals, carrying out hypothesis tests, and so on. Chapter 5 discusses
these ideas at length.
Example 4.2.5. The sample mean x N of any identically distributed sample is asymptotically unbiased for the common mean because it is unbiased. If the random
variables x1 , . . . , x N in the sample are also independent, then we can apply the law
122
of large numbers, which implies that x N is consistent for (see (1.26) on page 34).
If, in addition, E [ xn2 ] < , then we can also apply the central limit theorem, which
implies that x N is asymptotically normal (see (1.29) on page 37).
Example 4.2.6. As another example of consistency, lets consider the sample standard deviation s N = s defined in (4.2). Let x1 , . . . , x N be an IID sample
as before,
2
with each xn having mean , variance and standard deviation = 2 . Fact 1.4.1
p
on page 31 tells us that if g is continuous and s2N

2 , then g(s2N ) g(2 ). Taking
q
p
p
g( x ) = x, we see that if s2N 2 , then s N = s2N 2 = , which is what we
p
want to show. Hence it suffices to prove that s2N 2 . To see that this is the case,
note that
s2N
N
N
1
1
( xn x N )2 =
=
N 1 n =1
N1 N
(xn x N )2
n =1
N
1
N1 N
[(xn ) (x N )]2
n =1
Expanding out the square, we get

s2N
N
=
N1
"
1
N
1
( xn ) 2 N
n =1
2
(xn )(x N ) + (x N )
#
2
n =1
N
=
N1
"
1
N
( xn )
#
2
( x N )
n =1
By the law of large numbers,

1
N
( x n )2 2
and
( x N ) 0
n =1
Applying the various results in fact 1.4.1 (page 31), we then have
"
#
N
p
N
1
s2N =
( xn )2 ( x N )2 1 [2 0] = 2
N 1 N n =1
(4.12)
Hence the sample variance and sample standard deviation are consistent estimators
of the variance and standard deviation respectively.
Example 4.2.7. The sample variance s2N of an IID sample is asymptotically normal provided that sufficiently high moments exist. To see that this is the case, let
123
x1 , . . . , x N be an IID sample with each xn having mean and variance 2 . Suppose

that the fourth moment exists. Let N := N/( N 1). We can modify the expression
for s2N in (4.12) to obtain
N (s2N 2 ) = N
"
N
1
N
( x n )2 2
n =1
N N ( x N )2 + ( N 1) N2 (4.13)
Consider the three terms on the right hand side of (4.13). That the right-most term
converges to zero can be shown by basic analysis. Also, the middle term converges
to zero in probability, since N 1 and
N ( x N )2 = an bn where an := N ( x N ), bn := x N
p
Appealing to the CLT, the LLN and fact 1.4.6 on page 34, we have an bn 0.
Hence, by (4.13) and fact 1.4.5 on page 34, to establish the asymptotic normality
d
result N (s2N 2 ) N (0, v) for some v, we need only show that
#
"
1 N
d
N
( xn )2 2 N (0, v)
N n =1
2
To see that this is so, let
Yn := ( xn ) . The left hand side of the last expression
can then be written as N (Yn E [Yn ]). We can now apply the CLT, and obtain v as
var[Yn ]. That is,
v = E [(Yn 2 )2 ] = E [( xn )4 2( xn )2 2 + 4 ]
In summary,
N (s2N 2 ) N (0, m4 4 )
where
m4 := E [( xn )4 ]
(4.14)
Example 4.2.8. Under the same assumptions as example 4.2.7, the sample standard
deviation is also asymptotically normal, with
N (s N ) N
Exercise 4.8.3 asks you to confirm this.
m 4
0, 4 2
4

(4.15)
124
N ( x1 , 1)
x1
N (, 1)
Figure 4.5: Maximizing the likelihood
4.3
Maximum Likelihood
How does one come up with an estimator having nice properties, such as unbiasedness, consistency, etc.? Sometimes we can just use intuition. For example, its
natural to use the sample mean to estimate the mean of a random variable. In more
complicated settings, however, more systematic approaches are required. One such
approach is the celebrated principle of maximum likelihood.
4.3.1
The Idea
To motivate the methodology, suppose I present you with a single draw x1 from
distribution N (, 1), where the is unknown, and the standard deviation is known
and equal to one for simplicity. Your job is to make a guess of the mean of
the distribution, given the observation x1 . Since a guess of also pins down
1), we could equivalently say that your job is to guess the
the distribution N (,
distribution that generated the observation x1 .
In guessing this distribution, if we centered it around some number much larger
than x1 , then our observed data point x1 would be an unlikely outcome for this
distribution. See figure 4.5. The same logic would apply if we centered the density at
a point much smaller than x1 .3 In fact, in the absence of any additional information,
the most obvious guess would be that the normal density is centered on x1 . To center
the density on x1 , we must choose the mean to be x1 . In other words, our guess of
is = x1 .
that our distribution can be represented by a density, all individual outcomes s R have
probability zero, and in this sense, all outcomes are equally unlikely. Whats meant by the statement
about x1 being relatively unlikely is that there is little probability mass in the neighborhood of x1 .
3 Given
125
Maximum likelihood leads to the same conclusion. The density of x1 is

p(s; ) := (2 )
1/2
( s )2
exp
2

( s R)
Consider plugging the observed value x1 into this density:

p( x1 ; ) = (2 )
1/2
( x )2
exp 1
2

Even though were dealing with a continuous random variable here, lets think of
p( x1 ; ) as representing the probability of realizing our sample point x1 . The principle of maximum likelihood suggests that we take as our guess of the value that
maximizes this probability. In some sense, this is the most likely given the sample. A little thought will convince you that = x1 is the maximizer. That is,
= x1 = argmax p( x1 ; )
<<
This coincides with our intuitive discussion regarding figure 4.5.

IID
The same principle applies when we have x1 , . . . , x N N (, 1), where is unknown. By independence, the joint density of this sample is obtained as the product
of the marginal densities. Plugging the sample values into the joint density, one then
maximizes the joint density with respect to . The maximizer
:= argmax (2 )
<<
N/2
( x n )2
exp 2
n =1

(4.16)
is called the maximum likelihood estimate. As you are asked to show in exercise 4.8.5, the maximizer is precisely the sample mean of x1 , . . . , x N .
We can generalize these ideas in several ways. Lets suppose now that the data
x1 , . . . , x N has joint density p in the sense of (1.19). We will assume that p = p(; )
is known up to a vector of parameters RK . In other words, the functional
form of p is known, and each choice of pins down a particular density p = p(; ),
but the value of in the density p(; ) that generated the data is unknown. In
this setting, the likelihood function is p evaluated at the sample x1 , . . . , x N , and
regarded as a function of :
L ( ) : = p ( x1 , . . . , x N ; )
( )
(4.17)
126
The principle of maximum likelihood tells us to estimate using the maximizer of

L( ) over . Alternatively, we can maximize the log likelihood function (see
11.2), defined as the log of L:
`( ) := ln( L( ))
( )
The maximum likelihood estimate (MLE) is the maximizer of L( ), or, equivalently, of `( ):

:= argmax L( ) = argmax `( )

(4.18)
(In the preceding discussion, p was a density function, but it can be a probability
mass function as well. Exercise 4.8.7 treats one example.)
To implement this method, we obviously need to know the joint density p of the
data. As we saw in (4.16), when the data points are independent this is easy because
the joint density p is the product of the marginals. More generally, if each xn is
drawn independently from fixed arbitrary (marginal) density pn (; ) on R, then
N
L( ) =
pn ( xn ; )
and
`( ) =
n =1
4.3.2
ln pn (xn ; )
(4.19)
n =1
Conditional Maximum Likelihood
In many statistical problems we wish to learn about the relationship between one
or more input variables and an output or response variable y. (See example 4.1.4 on page 111.) In the case of scalar input, we observe inputs x1 , . . . , x N and
corresponding outputs y1 , . . . , y N . Given this data, we wish to estimate the relationship between x and y. For the following theoretical discussion, we suppose that the
pairs ( xn , yn ) are independent of each other and share a common density p:
P{ xn s, yn t} =
Z t Z s

p(s, t) dsdt
for all s, t R
Lets suppose that in investigating the relationship between x and y, we have decided that the conditional density of y given x has the form f (y| x ), where is
a vector of parameters. How can we choose by maximum likelihood?
127
The principle of maximum likelihood tells us to maximize the log likelihood function formed from the joint density of the sample, which in this case is
N
`( ) =
ln p(xn , yn )
n =1
Letting g be the marginal density of x, we can use the decomposition (1.21) on

page 26 to write p(s, t) = f (t | s) g(s), and re-express the log likelihood as
N
`( ) =
ln[ f (yn |xn ) g(xn )]
n =1
Since the function g enters into this expression, it might seem like we need to specify
the marginal distribution of x in order to maximize `. However, if g is not a function
of then this is unnecessary, since
N
n =1
n =1
n =1
ln[ f (yn |xn ) g(xn )] = ln f (yn |xn ) + ln g(xn )
By assumption, the second term is independent of , and as such it does not affect
the maximizer. As a result, the MLE is
N
argmax

ln f (yn |xn )
n =1
The term nN=1 ln f (yn | xn ) is formally referred to as the conditional log likelihood,
but in applications most people just call it the log likelihood.
Example 4.3.1. Consider a binary response model with scalar input x. To be concrete, we will imagine that binary (i.e., Bernoulli) random variables y1 , . . . , y N indicate whether or not a sample of married women participate in the labor force. We
believe that the decision yn of the n-th individual is influenced by a variable xn measuring income from the rest of the household (e.g., the salary of their spouse). Let
q(s) be the probability that y = 1 (indicates participation) when x = s. Often this is
modelled by taking q(s) = F (s), where is an unknown parameter and F is a cdf.
We can then write
P{y = t | x = s} = F(s)t (1 F(s))1t for s R and t {0, 1}
128
Taking this expression as the conditional density of y given x, the (conditional) log
likelihood is therefore
N
`( ) =
ln[ F(xn )yn (1 F(xn ))1yn ]
n =1
N
n =1
yn ln F (xn ) +
(1 yn ) ln(1 F(xn ))
n =1
If F is the standard normal cdf, then the binary response model is called the probit
model. If F is the logistic cdf F (s) = 1/(1 es ), then its called the logit model.4
4.3.3
Comments on Maximum Likelihood
Lets finish with some general comments on maximum likelihood. Maximum likelihood theory formed the cornerstone of early to mid 20th Century statistics. Analyzed by a series of brilliant statisticians, maximum likelihood estimators were
shown to be good estimators under a variety of different criteria. For example, under a bunch of regularity conditions that we wont delve into, MLEs are
1. consistent,
2. asymptotically normal, and
3. have small variance, at least asymptotically.
These results are genuinely remarkable. For details, see, for example, Dasgupta
(2008, chapter 16).
In the last several decades, however, many statisticians have become increasingly
dissatisfied with the limitations of maximum likelihood, and other approaches have
become more popular. The most important criticism of maximum likelihood is that
the statistician must bring a lot of knowledge to the table in order to form an estimator. If we look at (4.18), we see that to determine the MLE we must first specify
the likelihood function L, which in turn requires the joint distribution of the sample
as a function of . Thus, to pin down the MLE we need to know the parametric
4 To
find the MLE, we can differentiate ` with respect to to obtain the first order condition, but
there is no analytical solution for either the probit or logit case. Instead, numerical optimization
is required. However, ` can be shown to be concave on R, which means that most hill climbing
algorithms will converge to the global maximum. Some discussion of numerical optimization is
given in 8.3.3.
129
class of the density from which the sample was drawn. All of the nice properties of
the MLE mentioned above are entirely dependent correct specification of the joint
distribution. This is a very big caveat indeed.
Discussion of these issues continues in 4.4.
4.4
Parametric vs Nonparametric Estimation
[roadmap]
4.4.1
Classes of Distributions
We used the terminology parametric class in the preceding discussion. Consider,

for example, the set D of all normal densities. That is,

( s )2
2 1/2
, R, > 0
D := all p s.t. p(s; , ) = (2 )
exp
22
The set D is an example of a parametric class. In this case the parametric class is
the set of all normal densities p that can be formed by different choices of and .
The parameters are and , and a particular choice of the parameters determines
(parameterizes) an element of the class D .
More generally, a parametric class of densities
D := { p } :=: { p : }
is a set of densities p indexed by a vector of parameters RK . In the
previous example, = (, ), and R2 . Not all classes of densities are parametric, however. For example, consider the set D 0 of all densities p with finite second
moment. In other words,

Z
Z
0
2
D := all p : R R s.t. p 0,
p(s)ds = 1,
s p(s)ds <
This is a large set of densities that cannot be expressed as a parametric class. In such
cases, we say that the class of densities is nonparametric.
Classical methods of inference such as maximum likelihood are parametric in nature. In this setting, we typically assume that:
130
0.00
0.05
0.10
0.15
0.20
0.25
Figure 4.6: The true density f
The data is generated by an unknown density.

The density belongs to parametric class D = { p } .
We know the class, but is unknown.
Nonparametric statistical techniques assume instead that the unknown density is in
some nonparametric class of densities. Often this class will be very broad. It may
even be the set of all densities.
4.4.2
Parametric Estimation
To understand the difference between parametric and nonparametric estimation,

lets look at an extended example. In the example, we let f be a density that is a
mixture of two normals. In particular,

1
( s + 1)2
1
( s 1)2
1/2
1/2
f (s) := (2 )
exp
+ (2 )
exp
(4.20)
2
2
2
2
To understand the density f , suppose that we flip a fair coin. If we get heads then
we draw x from N (1, 1). If we get tails then we draw x from N (1, 1). The random
variable x then has density f . A plot of f is given in figure 4.6.
Now consider an econometrician who does not know f , but has instead access to an
IID sample x1 , . . . , x N from f . His (or her) objective is to estimate the whole density
131
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Figure 4.7: Estimate f (black line) and true f (blue line)
f , based on the sample. Assuming that he is trained in traditional econometrics/statistics, his instinct will be to choose a particular parametric class for f , and then
estimate the parameters in order to pin down a particular density in that class.
Lets suppose that, not having any particular pointers from theory as to the nature
of the distribution, our econometrician decides in the end to model the density as
being normally distributed. In other words, he makes the assumption that f D ,
where D is the class of normal densities, as defined above.
He now wants to form an estimator f D of f , based on the data. This involves determining two parameters, and . He does this in the obvious way: He estimates
the sample mean, and via := s, the sample standard deviation.
via := x,
), where p is
Plugging these into the density, his estimate becomes f(s) := p(s; ,
the normal density.
Lets generate 200 independent observations x1 , . . . , x200 from f and see how this
procedure goes. The estimator f of f proposed by our econometrician is the black
density in figure 4.7. The density is superimposed over the histogram and the original true density f (in blue).
Lets now consider whether f a is good estimator of f . On balance, one would
probably have to say no. Even though the fit in figure 4.7 (i.e., the deviation between
the black and blue lines) might be thought of as reasonable, the estimator f does not
have good properties. In particular, f will not converge to f as the sample size
132
0.00
0.05
0.10
0.15
0.20
0.25
0.30
Figure 4.8: Sample size = 2,000
goes to infinity.5 The problem is the mistaken original assumption that f D .

There is no element of D that can approximate f well. We can see this by rerunning the
simulation with ten times more observations (2,000 instead of 200). The result is
given in figure 4.8. As expected, the fit is not much better.
4.4.3
Nonparametric Kernel Density Estimation
Lets now look at a standard nonparametric approach to the same problem. Our
next econometrician is in the same position as the econometrician in the previous
section: IID data x1 , . . . , x N is generated from the density f in (4.20) and presented
to her. She does not have knowledge of f , and seeks to construct a estimate f of f
on the basis of the data. Unlike the previous econometrician, however, she does not
presume to know the parametric class that the density f belongs to. How can she
proceed?
One approach to estimating f without making assumptions about the parametric
class is to use a kernel density estimator. Let K : R R be a density function, and
f is not consistent. I havent used this terminology, because consistency was defined for real-valued estimators, not function-valued estimators like f. However, one can
define a notion of convergence in probability for functions, and then give a definition of consistency
that applies here. See any advanced text on density estimation.
5 In general, one would say that
x1
133
x2
Figure 4.9: Function f when N = 2
let be a positive real number. From the sample, we then define

1 N
K
fN (s) :=: f(s) :=
N n
=1
s xn

(4.21)
Here K is called the kernel function of the estimator, and is called the bandwidth.
Exercise 4.8.8 asks you to confirm that f is indeed a density.6
A common choice for K is the standard normal density. For this choice of K, the
function f in (4.21) is illustrated in figure 4.9 for the case of two sample points, x1
and x2 . Centered on sample point xn we place a smooth bump drawn in red,
which is the function

1
s xn
gn ( s ) : =
K
(n = 1, 2)
N
Summing these two bumps gives f = g1 + g2 , drawn in black.

6 Although
we have not specified a parametric class, our choice of K and are associated with
some assumptions about the shape and form of f . For example, if K is taken to be Gaussian, then fN
will have exponentially decreasing tails.
134
In R, nonparametric kernel density estimates can be produced using the function

density. Try, for example,
> plot ( density ( runif (200) ) )
To learn more, type ?density.
Returning to our estimation problem, figure 4.10 shows a kernel density estimate
of f from 200 sample points, using the default settings in R. Figure 4.11 shows the
estimate with 2,000 sample points. (The code for generating figure 4.11 is given
in listing 4.) The fit in figure 4.11 is much better than the comparable parametric
fit in figure 4.8, and, unlike the parametric case, further increases in sample size
continue to improve the fit. Indeed, it can be proved in that, for any density f , the
nonparametric kernel density estimator converges to f in the following sense:
IID
Theorem 4.4.1. Let f and K be any densities on R, and let { xn }

n=1 f . Let f N be as
defined in (4.21). If the bandwidth N is a sequence depending on N and satisfying N 0
and NN as N , then
Z

E
| f N (s) f (s)| ds 0
as N
For a proof, see Devroye and Lugosi (2001, theorem 9.2).

On the other hand, if you experiment with the code in listing 4, you will see that,
for small sample sizes (try 10), the nonparametric estimate is actually poorer than
the parametric alternative. The good theoretical results we have discussed are all
asymptotic, and nonparametric methods generally need large sample sizes. This
stands to reason: Nonparametric methods have little structure in the form of prior
knowledge, and hence require abundant data.
4.4.4
Commentary
In some fields of science, researchers have considerable knowledge about parametric classes and specific functional forms. For example, the theory of Brownian motion describes how the location of a tiny particle in liquid is approximately normally
distributed. Hence, the underlying theory provides the exact parametric class of the
density.
Heres another example, from a regression perspective: An engineer is interested in
studying the effect of a certain load on the length of a spring. Classical physics tells
135
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.25
Figure 4.10: Nonparametric, sample size = 200
Figure 4.11: Nonparametric, sample size = 2,000
136
Listing 4 The source code for figure 4.11

set . seed (1234)
fden <- function ( x ) { # The density function of f

return (0.5 * dnorm (x , mean = -1) + 0.5 * dnorm (x , mean =1) )
}
fsamp <- function ( N ) { # Generates N draws from f

observations <- numeric ( N )
u <- runif ( N )
for ( i in 1: N ) {
if ( u [ i ] < 0.5) {
observations [ i ] <- rnorm (1 , mean = -1)
}
else observations [ i ] <- rnorm (1 , mean =1)
}
return ( observations )
}
observations <- fsamp (2000)
xgrid <- seq ( -4.5 , 4.5 , length =200)
plot ( density ( observations ) , main = " " , xlab = " " , ylab = " " )
lines ( xgrid , fden ( xgrid ) , col = " blue " , lwd =2)
137
her that the relationship is approximately proportional. This provides a functional

form that the engineer can use to estimate a regression function.
When underlying theory provides us with knowledge of functional forms, as in
the two examples above, the parametric paradigm is excellent. Classical statistics
yields many well behaved estimators. Moreover, all the knowledge embodied in the
functional forms helps us to conduct inference: Generalization is about combining
knowledge and data, and the more knowledge we have the better.
What about econometrics?
Unfortunately, no such elegant equivalent of the theory of Brownian motion exists
for the movement of economic variables such as stock prices, or exchange rates.
Similarly, an econometrician trying to estimate the effect of human capital on GDP
does not have the same information as the engineer studying load on a spring, who
knows that her relationship is proportional. What does classical economics tell the
econometrician about the precise functional form relating human capital to GDP?
Not a lot, to be honest.
Overall, economics and other social sciences are generally messier than the physical
sciences, and the econometrician usually comes to the table with much less knowledge of parametric classes and functional forms. This suggests that nonparametric
techniques will become increasingly popular in econometrics. At the same time,
there is no firm dividing line between parametric and nonparametric methods. Often, a flexible parametric class of densities with many parameters can do a great
job of approximating a given nonparametric class, and be more convenient to work
with. For this reason, modern statistics is something of a mix, blending both parametric and nonparametric methods. These notes follow this mixed approach.
4.5
Empirical Distributions
Recall from (1.12) on page 18 that if x is a discrete random variable taking values
s1 , . . . , s J with probabilities p1 , . . . , p J , then the cdf of x is
F ( s ) = P{ x s } =
1{ s j s } p j
(4.22)
j =1
In this section, were going to look at an important kind of discrete distribution. To

describe it, let x1 , . . . , x N be draws from some unknown distribution F. The empirical distribution of the sample is the discrete distribution that puts equal probability
138
on each sample point. Since there are N sample points, that means that probability
1/N is placed on each point xn . The concept of the empirical distribution is a bit
slippery because its a random distribution, depending on the sample. Nevertheless,
its a very useful object.
Throughout this section, well work with the same fixed observations x1 , . . . , x N
F, and x e will denote a random variable with the corresponding empirical distribution. That is, x e is a random variable taking each of the values x1 , . . . , x N with
uniform probability 1/N.
The cdf for the empirical distribution is called the empirical cumulative distribution function, or ecdf. Throughout the text, we will denote it by FN . Invoking (4.22),
we see that the ecdf can be written as
FN (s) = P{ x e s} =
1{ x n s } N
n =1
Its more common to put the 1/N term at the start, so lets use this as our definition:
FN (s) :=
1
N
1{ x n s }
( s R)
n =1
If you think about it, you will see that we can also write
FN (s) = fraction of the sample less than or equal to s
Graphically, FN is a step function, with an upward jump of 1/N at each data point.
Figure 4.12 shows an example with N = 10 and each data point drawn independently from a Beta(5, 5) distribution.
In R, the ecdf is implemented by the function ecdf. Try this example:
plot ( ecdf ( rnorm (20) ) )
4.5.1
Plug in Estimators
Continuing the proceeding discussion with the same notation, note that if h is a
function from R into R, then by (1.16) on page 22, its expectation with respect to the
empirical distribution is
Z
1
1
h(s) FN (ds) :=: E [h( x )] = h( xn ) =
N
N
n =1
e
h( xn )
n =1
(4.23)
1.0
139
F
FN
0.8
0.6
0.4
0.2
0.00.0
0.2
0.4
0.6
0.8
1.0
Figure 4.12: FN and F with N = 10
For example, the mean of the empirical distribution is the sample mean of x1 , . . . , x N :
Z
sFN (ds) :=: E [ x e ] =
1
N
xn =: x N
n =1
If the sample is IID, then by the law of large numbers, the value of the expression
R
(4.23) converges in probability to the expectation E [h( x1 )] = h(s) F (ds). In other
words,
for h : R R and large N we have
h(s) FN (ds)
h(s) F (ds)
This suggests an approach for producing estimators: Whenever we want to estimate

a quantity such that
1. =
h(s) F (ds) for some h : R R, and
2. the function h is known,

then we replace the cdf F with the ecdf FN and use the resulting statistic
N :=
1
h(s) FN (ds) =
N
h( xn )
n =1
140
R
The estimator N is called the plug in estimator of = h(s) F (ds), because FN
is plugged in to the expression in place of F. Notice that plug in estimators are
nonparametric, in the sense that we need no parametric class assumption in order
to form the estimator.
R
Example 4.5.1. The plug in estimator of the k-th moment sk F (ds) of F is the sample
k-th moment
Z
1 N k
xn
sk FN (ds) =
N n
=1
Example 4.5.2. The plug in estimator of the variance
Z
is
Z
2
sF (ds)
2
sFN (ds)
FN (dt) =
F (dt)
1
N
(xn x N )2
n =1
This differs slightly from the sample variance s2N defined on page 114. However, the
deviation is negligible when N is large.
Remark 4.5.1. Although we have defined the plug in estimator as an estimator of
quantities that can be expressed as integrals using F, the term plug in estimator
is often used more generally for any estimator produced by replacing F with FN .
For example, in this terminology, the plug in estimator of the median F 1 (0.5) is
FN1 (0.5).
4.5.2
Properties of the ecdf
Once again, let x1 , . . . , x N be IID draws from some fixed underlying distribution F,
and let FN be the corresponding ecdf. Perhaps the most important single fact about
the ecdf FN is that it converges to the cdf F as N . Indeed, from (1.28) on
page 36, we have
FN (s) :=
1
N
1{ x n s } P{ x n s } = : F ( s )
n =1
In fact, a stronger statement is true. The following theorem is sometimes called the
fundamental theorem of statistics, or the Glivenko-Cantelli theorem:
1.0
141
0.8
0.6
0.4
0.2
0.00.0
0.2
0.4
0.6
0.8
1.0
Figure 4.13: Realizations of FN with N = 10
Theorem 4.5.1 (FTS). If x1 , . . . , x N are

sponding ecdf, then
IID
draws from some cdf F, and FN is the correp
sup | FN (s) F (s)| 0

s
(Here sup is roughly equivalent to maxsee the appendix for more discussion.)
Thus, we see that the maximal deviation between the two functions goes to zero in
probability.7 Figures 4.134.15 illustrate the idea. Each picture shows 10 observations of FN , depending on 10 different observations of the data x1 , . . . , x N .
The theorem tells us that, at least in the IID setting, if we have an infinite amount of
data, then we can learn the underlying distribution without having to impose any
assumptions. This is certainly a nice result. However, we should bear in mind that
in reality we only ever have a finite amount of data. As such, assumptions are still
required to generalize from this data.
7 In
fact, the theorem tells us that convergence occurs almost surely, which is a stronger notion
that in probability. The details are omitted.
1.0
142
0.8
0.6
0.4
0.2
0.00.0
0.2
0.4
0.6
0.8
1.0
1.0
0.8
0.6
0.4
0.2
0.00.0
0.2
0.4
0.6
0.8
1.0
4.6
143
Empirical Risk Minimization
Empirical risk minimization is an inductive principle that is essentially nonparametric in nature. Except in special cases, it does not require specification of the
parametric form of the underlying density in order to form the estimator. Instead,
it starts with loss function, which states the subjective loss (opposite of utility) from
incorrect prediction.
4.6.1
The ERM Principle
To understand the principle, consider a setting where we repeatedly observe an

input x to a system, followed by an output y. Both are random variables, and we
believe there is some stable relation between them. In particular, we assume that the
input-output pairs ( x1 , y1 ), . . . , ( x N , y N ) we observe are IID draws from joint density
p. Suppose our aim is to predict new output values from observed input values.
A natural way to treat this problem is to choose a function f such that f ( x ) is our
prediction of y once x is observed. Incorrect prediction incurs a loss. Letting y be the
actual outcome, the size of this subjective loss is taken to be L(y, f ( x )). The function
L is called the loss function. Common choices for the loss function include:
The quadratic loss function L(y, f ( x )) = (y f ( x ))2
The absolute deviation L(y, f ( x )) = |y f ( x )|
The discrete loss function L(y, f ( x )) = 1{y 6= f ( x )}
The discrete loss function is typically used when y takes only finitely many possible
values. A unit loss is incurred if our guess is incorrect. No loss is incurred otherwise.
Returning to the general problem, with arbitrary loss function L, one might consider
choosing f so as to minimize the expected loss
R( f ) := E [ L(y, f ( x ))] :=:
Z Z
L(t, f (s)) p(s, t)dsdt
(4.24)
In (4.24), the expected loss is called the risk, and R is called the risk function. If
we knew the joint density p, then, at least in principle, we could evaluate R( f ) for
any f by calculating the double integral in (4.24). By repeating this calculation for
different f , we could search for a minimizer.
144
However, in the statistical setting, there is an obvious difficultly: We dont know p.

Hence R( f ) cannot be evaluated, let alone minimized.
All is not lost. While we dont know p, we do have at hand the observed inputoutput pairs ( x1 , y1 ), . . . , ( x N , y N ), which are independent draws from p. Draws
from p give us information about p. For example, the law of large numbers tells us
for large N we have
R( f ) := E [ L(y, f ( x ))]
1
N
L(yn , f (xn ))
n =1
This motivates us to replace the risk function R with the empirical risk function
1
R ( f ) :=
N
L(yn , f (xn ))
(4.25)
n =1
In particular, we obtain an estimate f of the true minimizer argmin f R( f ) by solving

1
f = argmin R ( f ) = argmin
N
f F
f F
L(yn , f (xn ))
(4.26)
n =1
This inductive principle, which produces an estimate of the risk-minimizing function by minimizing the empirical risk, is called empirical risk minimization (ERM).
Notice that in (4.26) we are minimizing over a set of functions F . This set of functions is called the hypothesis space, and is a class of candidate functions chosen by
the econometrician or researcher. At first pass, it might seem that we should F to be
the set of all functions f : R R, or at least take it to be as large as possible. After
all, if the risk minimizing function f := argmin f R( f ) is not in F , as visualized
in figure 4.16, then the solution to (4.26) is not equal to f , and we are making a
sub-optimal choice.
Although this reasoning seems logical, it turns out that setting F to be the set of all
functions from R R is a bad idea. In fact, we want to be quite restrictive in our
choice of F . These ideas are explored in detail in 4.6.2.
4.6.2
ERM and Least Squares
Specializing to the quadratic loss function L(y, y ) = (y y )2 , and observing that the
term N1 makes no difference to the solution f (see 11.2), the ERM problem becomes
N
min
(yn f (xn ))2
f F n=1
(4.27)
145
all f : R R
f := argmin f R( f )
Figure 4.16: Choosing the hypothesis space
For obvious reasons, this optimization problem is called the least squares problem.
If we specialize F to be the set of affine functions
L := { all functions of the form `( x ) = + x }
(4.28)
then the problem reduces to the linear least squares problem

N
min
(yn `(xn ))2 = min

(yn xn )2
,
` L n=1
(4.29)
n =1
This is the empirical risk counterpart to the risk minimization problem (1.23) on
page 30. Direct differentiation and simple manipulations show that the minimizers
of the empirical risk are
N ( xn x )(yn y )
= n=1 N
n=1 ( xn x )2
and
1
N
yn N xn
n =1
(4.30)
n =1
Comparing with (1.24) on page 30, which gives the minimizers of the risk, we see
that in this case the minimizers of the empirical risk are the sample analogues of the
minimizers of the risk.
Now lets return to the issue of hypothesis space mentioned above: Why would
we want to minimize the empirical risk over a restricted hypothesis space such as
L, rather than the entire set of functions from R to R? After all, minimizing the
empirical risk over a bigger set of functions makes the empirical risk smaller.8 Isnt
that desirable?
8 Intuitively,
if we expand the set of candidates, then we can find a smaller value. Formally, if A is
any set, g : A R, and D D 0 A, then infa D0 g( a) infa D g( a) always holds.
146
The answer is: not necessarily. The reason is that, while the function f obtained by
minimizing empirical risk over a large set of functions will make the empirical risk
R ( f) small, the actual risk R( f) may not be. The underlying problem is that we are
attempting to minimize expected loss on the basis of a sample mean, rather using
the expectation from the actual distribution. We need to be careful about reading
too much into this particular sample.
Lets illustrate this point by way of an example, where empirical risk is minimized
over progressively larger hypothesis spaces. In the example, the model we will
consider is one that generates input-output pairs via
x U [1, 1] and then y = cos(x ) + u where u N (0, 1)
(4.31)
where U [1, 1] is the uniform distribution on the interval [1, 1]. Our hypothesis
spaces for predicting y from x will be sets of polynomial functions. To fix notation,
let Pd be the set of all polynomials of degree d. That is,
Pd := { all functions f d ( x ) = c0 x0 + c1 x1 + cd x d where each ci R}

Clearly
P1 P2 P3
because if f is a polynomial of degree d, then f can be represented as a polynomial
of degree d + 1 just by setting the last coefficient cd+1 to zero:
P d 3 f d ( x ) = c0 x 0 + c1 x 1 + c d x d
= c0 x0 + c1 x1 + cd x d + 0x d+1 Pd+1
Also, the set of linear functions L defined in (4.28) is equal to P1 .
If we seek to predict y from x using quadratic loss and the set Pd as our candidate
functions, the risk minimization problem is
min R( f )
f Pd
where
R( f ) = E [(y f ( x ))2 ]
(4.32)
while the empirical risk minimization problem is

min R ( f )
f Pd
where
1
R ( f ) =
N
(yn f (xn ))2
(4.33)
n =1
To illustrate the difference between risk and empirical risk, we first generate N = 25
data points from the model (4.31). Taking this as our data set, we then solve (4.33)
147
Empirical risk
Risk
risk
4
0.4
0.3
empirical risk
0.5
0.6
0.7
0.2
0.1
10
degree
15
20
10
15
20
degree
Figure 4.17: Risk and empirical risk as a function of d
repeatedly, once for each d in 1, 2, . . . , 15. The solution to the d-th minimization
problem is denoted fd , and is, by construction, a polynomial of degree d. Finally, we
compare the risk R( fd ) and empirical risk R ( fd ).9 The results are in figure 4.17.
Analysing the figure, we see that, as expected, empirical risk falls monotonically
with d. This must be the case because minimizing a function over larger and larger
domains produces smaller and smaller values. On the other hand, the risk decreases
slightly and then increases rapidly. For large d, the minimizer fd of the empirical risk
is associated with high risk in the sense of large expected loss.
We can get a feeling for what is happening by plotting the data and the functions. In
figures 4.184.21, the N data points are plotted alongside the function y = cos(x )
from the true model (4.31) in black, and fitted polynomial fd in red. The function
y = cos(x ) is the risk minimizer, and represents the ideal prediction function. In
figure 4.18 we have d = 1, and the fitted polynomial f1 is the linear regression line.
In figures 4.19, 4.20 and 4.21 we have d = 3, d = 11 and d = 14 respectively, and the
fitted polynomials are f3 , f11 and f14 .
risk R( fd ) is evaluated by substituting fd into the expression for R in (4.32) and calculating
the expectation numerically.
9 The
148
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
degree = 1
Figure 4.18: Fitted polynomial, d = 1
When d = 1, the hypothesis space Pd = P1 is quite small. There is no function in

this class that can do a good job of fitting the underlying model. This situation is
called under-fitting, and is reflected in the poor fit of the red line to the black line
in figure 4.18. When d = 3, the class of functions Pd = P3 is considerably larger.
Given that the data is relatively noisy, and that we only have 25 observations, the
fit of the function is fairly good (figure 4.19). If we look at the risk for d = 3 on the
right-hand side of figure 4.17, we see that it is lower than for d = 1.
On the other hand, if we take d up to 11 or even 14, the fit to the underlying model
is poor, and the risk is high. Examining the corresponding figures (figures 4.20 and
4.21), we see that the fitted polynomial has been able to fit the observed data closely,
passing near to many of the data points. In a sense it is paying too much attention to
this particular realization of the data. When a new input x is drawn, the prediction
f14 ( x ) is likely to be a poor predictor of y, and the risk is correspondingly high. This
situation is called over-fitting.
What can we learn from this discussion? The main lesson is that the choice of the
hypothesis space F in the empirical risk minimization problem (4.27) is crucial. This
is the problem of model selection.
149
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
degree = 3
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
degree = 11
150
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
degree = 14
In real statistical applications, we do not have the luxury of knowing the true model
when we choose F . In response, many researchers simply choose F = L, the set of
linear functions. This may or may not be a good choice. Ideally, the hypothesis space
should be carefully chosen on the basis of economic theory: F should be the set of
reasonable candidate descriptions of the relationship between x and y, given our
knowledge of the economic system we are modelling. Once again, the message is
that statistical learning equals prior knowledge plus data.
The problem of model selection is discussed in more depth in chapter 10.
4.6.3
Other Applications of ERM
In our discussion of ERM so far, we have talked about finding functions to predict
y given x. A simpler situation is where we observe only y and seek to predict it. In
this case the object we seek to calculate is just a constant (a prediction of y) rather
than a function (that predicts y from any given x). This makes the learning problem
simpler.
151
IID
Suppose, for example, that we observe y1 , . . . , y N F, where F is an unknown cdf.

For the sake of concreteness, lets imagine that each observation is the monetary
payoff of a particular game at a casino. We want to predict the payoff of the next
draw. Letting be our prediction and L be the loss function, the corresponding risk
is E [ L(, y)]. If we specialize to the quadratic loss, this becomes
R() = E [( y)2 ] =
( s)2 F (ds)
The empirical risk is

1
R () =
N
( y n )2 =
( s)2 FN (ds)
(4.34)
n =1
Here FN is the empirical distribution of the sample. Minimizing R () with respect

to , we obtain our prediction of y as
1
:= argmin R () =
N
yn
n =1
Thus, at least with quadratic loss, ERM leads to the sample mean, which is the most
natural predictor of y.
As this last example helps to clarify, the ERM principle is essentially nonparametric in nature. The empirical risk is determined only by the loss function and the
empirical distribution. Unlike maximum likelihood, say, we usually dont have to
specify the parametric class of the unknown distributions in order to solve the ERM
problem.
At the same time, we can recover many parametric techniques as special cases of
empirical risk minimization. One is maximum likelihood. To see this, suppose that
x is drawn from unknown density q. We wish to learn the density q by observing
draws from this density. We take our loss function to be L( p, x ) = ln p( x ). In
other words, if our guess of q is p and the value x is drawn, then our loss is ln p( x ).
Loosely speaking, if p puts small probabilities on regions where x is realized, then
we suffer large loss. Hence, the loss function encourages us to choose p close to q.
Our choice of loss leads to the risk function
R( p) = E [ L( p, x )] =
ln[ p(s)]q(s)ds
152
Although it may not be obvious at first glance, minimizing this risk function yields
the unknown density q. To see this, let us first transform our expression for the risk
function to

Z
Z
q(s)
q(s)ds ln[q(s)]q(s)ds
R( p) = ln
p(s)
(Can you see why the two expressions for R( p) are equal?) The term on the far right
is called the entropy of the density q, and does not involve p. Hence, minimization
of the risk comes down to minimization of

Z
q(s)
D (q, p) := ln
q(s)ds
p(s)
This quantity is called the Kullback-Leibler (KL) deviation between q and p. The
KL deviation is possibly infinite, always nonnegative, and zero if and only if p = q.10
It follows that the unique minimizer of the risk is the true density q.
Now suppose that we observe IID draws x1 , . . . , x N from q. To estimate q, the ERM
principle indicates we should solve
(
)
(
)
N
N
1
p := argmin R ( p) = argmin
ln p( xn ) = argmax ln p( xn )
N n
p
p
p
=1
n =1
To make the connection with maximum likelihood, lets now add the assumption
that the unknown density lies in some parametric class { p(; )} . Suppose that
we know the parametric class, but the true value generating the data is unknown.
Choosing our estimate p of q now reduces to choosing an estimate of . Re-writing
our optimization problem for this case, we obtain
)
(
N
= argmax ln p( xn ; ) = argmax `( )
n =1
where ` is the log-likelihood. It follows from this expression that the ERM estimator
is precisely the maximum likelihood estimator.
4.7
Further Reading
The ERM principle is a very general principle for solving statistical problems and
producing estimators. For such a general method it is difficult to give a set of strong
10 More
precisely, D (q, p) = 0 if and only if p = q almost everywhere. Equality almost everywhere

is a basic concept from measure theory that is (only just) beyond the scope of these notes. Note that
if two density are equal almost everywhere then they share the same cdf, and hence represent the
same distribution.
153
results showing that ERM produces good estimators. Indeed, there will be instances
when ERM produces poor estimators, as discussed in 4.6.2. Having said that, some
rather general consistency results have been obtained. The details are beyond the
level of these notes. Some discussion can be found in [26].
4.8
Exercises
Ex. 4.8.1. Confirm (4.11): Show that, for any estimator of , we have mse[] =
var[] + bias[]2 .
Ex. 4.8.2. Confirm that for an IID sample x1 , . . . , x N with variance 2 , the sample
variance s2N defined in (4.1) is unbiased for 2 .11
Ex. 4.8.3. Assuming the conditions of examples 4.2.74.2.8, show that (4.15) on
page 123 is valid.
Ex. 4.8.4. Let x1 , . . . , x N be IID with mean and variance 2 . Let x N be the sample
mean, and let N be a consistent estimator of . What is the limiting distribution of

y N := N
x N
N
2
Ex. 4.8.5. Confirm that the maximizer of (4.16) is the sample mean of x1 , . . . , x N .
IID
Ex. 4.8.6. Let x1 , . . . , x N F, where F is a cdf. Let m4 < be the 4-th moment.
That is,
Z
m4 := s4 F (ds)
Define the plug in estimator of m4 . Is the estimator consistent? Why or why not?
Ex. 4.8.7. Suppose we are playing a slot machine (one-armed bandit) that either
pays one dollar or nothing, with each payoff independent of the previous outcome.
Let be the probability of winning (i.e., receiving one dollar). Having observed 100
plays x1 , . . . , x100 , where xn {0, 1}, a natural estimator of is the fraction of wins,
Use the principle of maximum likelihood to
which is just the sample mean = x.
12
obtain the same conclusion.
11 The sample standard deviation
s (4.2) is typically biased. To see why, look up Jensens inequality.

Each xn is a Bernoulli random variable, the probability mass function for which can be
written as p(s; ) := s (1 )1s for s {0, 1}.
12 Hint:
154
Ex. 4.8.8. Show that f in (4.21) is a density for every N, every > 0 and every
realization of the sample.13
Ex. 4.8.9. Let x be a random variable with := E [ x ]. Consider the risk function
given by R( ) = E [( x )2 ]. Show that is the minimizer of R( ) over all R,
without using differentiation.14
Ex. 4.8.10. Let N be an estimator of . Recalling the definitions on page 121, show
that if N is asymptotically normal, then N is consistent for . (Note: You can restrict
attention to the scalar case. Even so, this exercise requires more experience with
analysis than some of the others.)
Ex. 4.8.11. Consider the following estimation problem. Let { p } be a parametric class of densities. Now let a particular 0 be fixed, and suppose that an
asymptotically normal sequence of estimators { N } exists for it, in the sense that
d
N ( N 0 ) N (0, )
for some positive definite . Fix a point y and consider the problem of estimating the
value p(y, 0 ). This is a density estimation problem, similar to what we discussed in
4.4.2. A natural suggestion is to use p(y, N ). State conditions under which this estimator is consistent and asymptotically normal for p(y, 0 ). Derive the asymptotic
distribution under these conditions.
Ex. 4.8.12 (Computational). Let x1 , . . . , x N be IID and uniformly distributed on the
interval [0, 1]. Let x N be the sample mean. What is the expectation and variance
of x N ? For N = 1, 2, 10, 500, simulate 10,000 observations of the random variable
x N . Histogram the observations, using one histogram for each value of N. (For
example, the first histogram should be of 10,000 observations of x1 .) What do you
observe about these four distributions? What interpretation can you give?
Ex. 4.8.13 (Computational). Extending your results in exercise 1.6.24, determine the
cdf of z := max{u1 , . . . , u N }, where u1 , . . . , u N are N independent random variables
uniformly distributed on [0, 1]. Check this by generating 1,000 draws of y and plotting the ecdf, along with your expression for the cdf. The ecdf should be close to the
cdf. In the simulation, set N = 5.
Ex. 4.8.14 (Computational). Implement the ecdf as your own user-defined function
in R, based on the definition (i.e., that it reports the fraction of the sample falling
below a given point).
You need to show that f is nonnegative and integrates to one. Showing that
is the hard part. Try a change-of-variables argument.
14 Hint: Use the add and subtract strategy.
13 Hint:
f(s)ds = 1
4.8.1
155
Solution to Exercise 4.8.1. Adding and subtracting E [], we get

mse[] := E [( )2 ] = E [( E [] + E [] )2 ]
Expanding the square and minor manipulations yield the desired result.
Solution to Exercise 4.8.2. We showed in 4.2.3 that
"
#
N
N
1
s2 =
( xn )2 ( x )2
N 1 N n
=1
Taking expectations, applying the result var[ x ] = E [( x )2 ] = 2 /N from (4.10)
and rearranging gives E [s2 ] = 2 as claimed.
Solution to Exercise 4.8.3. The aim is to verify the claim

m4 4
d
N (s N ) N 0,
42
under the conditions of example 4.2.8 on page 123. Letting g( x ) = x and applying
theorem 1.4.3 on page 37 along with (4.15), we have

d
N (s N ) = N ( g(s2N ) g(2 )) N 0, g0 (2 )2 (m4 4 )
The result follows.
Solution to Exercise 4.8.4. Let
w N :=
p
x N
x N
N
=
N
N
N
Since N by assumption, fact 1.4.1 on page 31 yields /N / = 1. Applying the central limit theorem and Slutskys theorem (fact 1.4.5 on page 34) together,
d
we then have w N z N (0, 1). By the continuous mapping theorem (fact 1.4.4
on page 34), y N = w2N converges in distribution to z2 . By fact 1.3.4 on page 1.3.4, the
distribution of z2 is 2 (1).
Solution to Exercise 4.8.6. The plug in estimator of m4 is the sample fourth moment. The sample fourth moment is consistent for m4 under the IID assumption
by the law of large numbers, given the stated assumption that m4 < .
156
Solution to Exercise 4.8.7. Each xn is a Bernoulli random variable, with pmf given
by p(s; ) := s (1 )1s for s {0, 1}. By independence, the joint distribution is
the product of the marginals, and hence the log likelihood is
N
`() =
log p( xn ; ) =
n =1
[xn log + (1 xn ) log(1 )]
n =1
Differentiating with respect to and setting the result equal to zero yields = x as
claimed.
Solution to Exercise 4.8.8. The nonnegativity of f is obvious. To show that
1, its enough to show that

Z
sa
ds =
K
f(s)ds =
for any given number a. This equality can be obtained by the change of variable
u := (s a)/, which leads to

Z
Z
Z
sa
K
ds = K (u)du = K (u)du
Since K is a density, the proof is done.

Solution to Exercise 4.8.9. Adding and subtracting , we can express R( ) as
R( ) = E {[( ) + ( x )]2 }
Expanding this out and using E [ x ] = , we obtain R( ) = ( )2 + var[ x ]. Evidently a minimum is obtained when = .
Solution to Exercise 4.8.10. Fix > 0. It suffices to show that for any positive number e we have
lim P{|N | > } e
(4.35)
N
(If a 0 and a e for any e > 0, then a = 0.) To establish (4.35), fix e > 0. Let
zbe standard normal, and choose M such that P{|z| M } e. For N such that
N M we have
P{|N | > } = P{ N |N | > N} P{ N |N | > M}

Taking N , applying asymptotic normality, the continuous mapping theorem
(fact 1.4.4 on page 34) and the definition of M gives (4.35).
157
Solution to Exercise 4.8.11. First observe that, under the conditions of the exercise,
p(y, N ) is consistent for p(y, 0 ) whenever 7 p(y, ) is continuous at 0 . Indeed,
since N is asymptotically normal it is also consistent (see page 121). Under this continuity assumption, in probability convergence of p(y, N ) to p(y, 0 ) now follows
from fact 2.5.4 on page 78.
Regarding asymptotic normality, suppose that 7 p(y, ) is differentiable at 0 ,
and that the vector of partial derivatives p(y, 0 ) is not the zero vector. Applying
(2.8) from page 79, we then have
d
n{ p(y, N ) p(y, 0 )} N (0, p(y, 0 )0 p(y, 0 ))
Chapter 5
Methods of Inference
[roadmap]
5.1
Making Inference about Theory
[roadmap]
5.1.1
Sampling Distributions
In 4.2 we emphasized the fact that statistics, and hence estimators, are random
being a
variables. For example, if is an estimator of some quantity , then ,
statistic, must be an observable function of the data. If x1 , . . . , x N is the data, and
= T ( x1 , . . . , x N ) for known function T, then is the random variable
( ) = T ( x1 ( ), . . . , x N ( ))
( )
(See, for example, (4.5) on page 115.)

Being a random variable, has a distribution, which is the cdf G (s) = P{ s}.
The distribution G of is sometimes called its sampling distribution. Usually this
IID
distribution will depend on unknown quantities. For example, if x1 , . . . , x N F
for some unknown cdf F, then the distribution of will depend on F and T. Thus,
the sampling distribution depends partly on the known object T, and partly on the
uknown object F.
158
CHAPTER 5. METHODS OF INFERENCE
159
Lets look at two examples. First, suppose that x1 , . . . , x N is an IID sample from
the normal distribution N (, 2 ), where both and are unknown. Consider the
sample mean x as an estimator of the mean . Combining fact 1.2.6 on page 24, (4.6)
on page 116 and (4.10) on page 117, we obtain
IID
x1 , . . . , x N N (, 2 ) = x N (, 2 /N )
(5.1)
Although this expression

The right-hand side is the sampling distribution of x.
provides some information, the sampling distribution still depends on the unknown
quantities and .
As a second example, suppose that we are interested in the average length of time
between incoming telephone calls at a call center during business hours. We model
the duration x between two calls as having the exponential distribution, with density f (s) = 1 exp(s). The parameter is unknown. We propose to monitor
consecutive calls until observations x1 , . . . x N are recorded. Using these observations, we will estimate mean duration using the sample mean x = N 1 n xn .
Since sums of independent exponential random variables are known to have the
gamma distribution, nN=1 xn must be gamma. Since scalar multiples of gammas are
again gamma, x = N 1 nN=1 xn must also be gamma. Thus, under our assumptions,
when the data is collected and x is evaluated, its value will be a draw from a gamma
distribution. Which particular gamma distribution depends on the unknown quantity .
5.1.2
Comparing Theory with Outcomes
In chapter 4, we were interested in estimating and predicting. We looked at ways

to find estimators, and at the properties of these estimators. In this chapter, we
are going to consider a different style of problem. The problem is one where we
hold a belief or theory concerning the probabilities generating the data, and we are
interested in whether the observed data provides evidence for or against that theory.
IID
To illustrate, suppose again that we have x1 , . . . , x N N (, 2 ), where both and

are unknown. Suppose that some economic theory that implies a specific value 0
for the unknown parameter (prices should be equal to marginal cost, excess profits
should be equal to zero, etc.) In this case, our interest in observing will be: What
light does this realization shed on our theory?
160
Figure 5.1: Theoretical and realized values
N (0 , s2 /N )
Figure 5.2: Estimated sampling distribution when the theory is correct
The naive answer is: The realization appears to contradict our theory when its a
long way from our hypothesized value 0 . But this is not really an answer until we
specify what a long way is. For example, consider figure 5.1. Is 0 a long way
from ?
To determine what a long way means, we can look at the sampling distribution of
In the present case, this distribution is N (, 2 /N ), as shown in (5.1). Although
.
the parameters and are not known, our theory specifies that should be equal
to 0 , and the second parameter 2 can be estimated consistently by the sample
variance s2 . Plugging the hypothesized mean 0 and the estimated variance s2 into
the sampling distribution gives the density in figure 5.2. Looking at this figure, we
can see that 0 and can indeed be regarded as a long way apart, in the sense that if
our theory was correct, then would be a realization from way out in the tail of its
own distribution. Thus, the realization is unlikely when our theory is true, and
this fact can be construed as evidence against the theory.
In the following sections we formalize these ideas.
5.2
Confidence Sets
The idea of confidence sets is to provide a set of parameter values, distributions, or

models that are plausible given the observed outcome of a statistic.
5.2.1
161
Parametric Examples
Suppose that we have a parametric class of models { M } indexed by parameter

R. For example, the models might describe a set of cdfs { F }, or a family
of proposed regression functions { f } in a regression problem. We anticipate observing a sample x := ( x1 , . . . , x N ) generated by one of the models M . Let F be the
joint distribution of the sample vector x when the data is generated by M . We use
the notation P to refer to probabilities for x. For example, given B R N , we let
P {x B} be the probability that x B given x F .1
Fix (0, 1). A random set C (x) is called a 1 confidence set for if
P { C(x)} 1 for all

Remember, it is the set that is random here, not the parameter . The for all statement is necessary since we dont actually know what the true value of is, and
hence the experiment should be designed such that P { C (x)} 1 regardless of which is generating the data. The interpretation of the confidence set is
that, if we conduct all our statistical experiments with a fixed value of , then our
confidence sets will contain the true parameter about (1 ) 100% of the time.
If C (x) is an interval in R, then C (x) is also called a confidence interval.
Example 5.2.1. Let x1 , . . . , x N N (, 2 ), where = R is unknown. Suppose
for the moment that is known. We wish to form a confidence interval for . By
(5.1), we have
( x N )
N (0, 1)
(5.2)
N
and hence, applying (1.14) on page 21,

(
)
N
P
| x N | z/2 = 1 when z/2 := 1 (1 /2)
IID
Here is the standard normal cdf. Some rearranging gives

P x N z/2 x N + z/2 = 1
N
N
Since this
argument is true regardless of the value of , we conclude that if en :=
z/2 / N, then C (x) := ( x N en , x N + en ) is a 1 confidence interval for
1 For
those readers who prefer more formal definitions, let P {x B} :=
1{s B} F (ds).
162
Example 5.2.2. Continuing on from example 5.2.1, a more realistic situation is that
is also unknown. In that case, a natural approach is to replace with the sample
standard deviation s N . In this case, (5.2) becomes
( x N )
FN 1
N
sN
(5.3)
where FN 1 is the cdf of the t-distribution with N 1 degrees of freedom.2 The

reasoning in example 5.2.1 now goes through when z/2 is replaced by t/2 :=
1
FN
1 (1 /2), and we obtain

sN
sN
P x N t/2 x N + t/2 = 1
N
N
In example
5.2.2,
note
that
since
the
standard
deviation
of
x
is
/
N, the term
N
It is often
s N / N is a sample estimate of the standard deviation of the estimator x.
called the standard error. More generally, if is an estimator of some quantity ,
then the standard error is
se() := a sample estimate of the standard deviation of
Of course this is not really a formal definition because we havent specified which
estimate of the standard deviation we are talking about, but nevertheless the terminology is very common.3 Using our new notation, we can write the confidence
interval in example 5.2.2 as
C (x) := ( x N se( x N )t/2 , x N + se( x N )t/2 )
5.2.2
(5.4)
Asymptotic Confidence Sets
As the degrees of freedom increases, quantiles of the t-distribution converge to those

of the normal distribution. Hence, for large N, we can replace t/2 with z/2 in (5.4)
and the approximation will be accurate. This is part of a more general phenomenon
related to the central limit theorem. Recall from 4.2.3 that an estimator N of R
is called asymptotically normal if
q
N ( N ) v( ) Z as N
(5.5)
2 The
proof of this statement is clearly related to fact 1.3.6 on page 27. The details are a bit fiddly.
We omit them because more general results are established in chapter 7.
3 Sometimes the term standard error is used to refer to the standard deviation of the estimator,
rather than an estimate of the standard deviation.
163
where v( ) is some positive constant and Z is standard normal. The constant v( )

is called the asymptotic variance of N . Many estimators have this property. For
example, most maximum likelihood estimators are asymptotically normal, as are
smooth transformations of sample means (theorem 1.4.3 on page 37).
Now suppose we have a sequence of statistics se(N ) such that
q
N se( N ) v( ) as N
(5.6)
As exercise 5.6.1 asks you to show, (5.5) and (5.6) imply that
N d
Z
se(N )
as
(5.7)
As a result, for large N, we can write

N + se(N ) Z
(5.8)
where the approximate equality is in terms of distribution. We see that se(N ) is

an approximation to the standard deviation of N , which explains our choice of
notation. As before, se(N ) is referred to as the standard error of the estimator.
A sequence of random sets CN (x) is said to form an asymptotic 1 confidence set for if
lim P { CN (x)} 1
for all
For our asymptotically normal estimator N , the sequence

CN (x) := (N se(N )z/2 , N + se(N )z/2 )
(5.9)
can be used because, rearranging, taking the limit and applying (5.7),

lim P { CN (x)} = lim P N se(N )z/2 N + se(N )z/2
N
N
(
)
N
= lim P z/2
z/2
N
se(N )
= 1
Looking at (5.9) gives a good indication of why standard errors are normally reported along with the point estimate. For example, we have the following useful
rule of thumb: If = 0.05, then z/2 2, so there is a 95% likelihood that the true
parameter lies within 2 standard deviations of the observed value of our estimator.
5.2.3
164
A Nonparametric Example
Confidence sets can be obtained in nonparametric settings too. A nice example is

confidence sets for the ecdf. In 4.5.2, we learned that if x1 , . . . , x N are IID draws
p
from some cdf F, and FN is the corresponding ecdf, then sups | FN (s) F (s)| 0.
This is the fundamental theorem of statistics (theorem 4.5.1). In 1933, A.N. Kolmogorov used an extension of the central limit theorem to obtain an asymptotic
distribution for this term. In particular, he showed that
N sup | FN (s) F (s)| K

s
(5.10)
where K is the Kolmogorov distribution

2
(2i 1)2 2
K (s) :=
exp
s i
8s2
=1
( s 0)
(5.11)
Notice that, like in the CLT, the limiting distribution K is independent of the cdf F
that generates the data.
We can use this result to produce an asymptotic 1 confidence set for F. To do so,
let F be the set of all cdfs on R, let k1 := K 1 (1 ), and let

k 1
k 1
for all s R
CN (x) := F F : FN (s) G (s) FN (s) +
N
N
The set CN (x) F is an asymptotic 1 confidence set for F. Indeed, rearranging
the expression, we get
n
o
{ F CN (x)} = k1 N ( FN (s) F (s)) k1 for all s

n
o
=
N | FN (s) F (s)| k1 for all s

= sup N | FN (s) F (s)| k1

s
Hopefully the last equality is clear.4 Applying (5.10) now confirms our claim:

lim P{ F CN (x)} = lim P sup N | FN (s) F (s)| k1 = 1

N
g : D R and g(s) M for all s D, then sups D g(s) M. Conversely, if sups D g(s) M,
then g(s) M for all s D.
4 If
165
Given our data x1 , . . . , x N and the corresponding ecdf FN , we can present the con
fidence set CN (x) visually by plotting the lower bound
function
F
(
s
)
k
/
N
N
1
and the the upper bound function FN (s) + k1 / N. This is done in figure 5.3 for
= 0.05. The data is generated using an arbitrary distribution F (the t-distribtion
with 2 degrees of freedom). In figures (a) and (b), the true function F is not shown.
In (c) and (d), F is shown in red. The preceding theory tells us that realizations of
the confidence set will catch the true function F about 95 times in 100.
The code for producing (d) of figure 5.3 is shown in listing 5. In the code, you
will see that the value we used for k1 = k0.95 := K 1 (0.95) was 1.36. This value
was obtained numerically from the definition of K in (5.11). The technique used
was to truncate the infinite sum in (5.11) at 20 to provide an approximation to K,
and then search for an s satisfying K (s) = 1 , or, equivalently, f (s) = 0 when
f (s) := K (s) 1 + . The root of f was found using the R univariate root-finding
function uniroot. See listing 6.
Listing 5 The source code for figure 5.3
samp _ size <- 1000
grid _ size <- 400
xgrid <- seq ( -3 , 3 , length = grid _ size )
FN <- function (s , X ) return ( mean ( X <= s ) )
# ecdf
X <- rt ( samp _ size , 2) # RVs from t - dist with 2 DF

Y <- numeric ( length ( xgrid ) )
for ( i in 1: length ( xgrid ) ) Y [ i ] <- FN ( xgrid [ i ] , X )
Y _ upper <- Y + 1.36 / sqrt ( samp _ size )
Y _ lower <- Y - 1.36 / sqrt ( samp _ size )
plot ( xgrid , Y , type = " l " , col = " blue " , xlab = " " , ylab = " " )
lines ( xgrid , Y _ upper )
lines ( xgrid , Y _ lower )
lines ( xgrid , pt ( xgrid , 2) , col = " red " )
5.3
Hypothesis Tests
[roadmap]
166
0.2
0.2
0.4
0.4
0.6
0.6
0.8
0.8
(b) N = 1000
0.2
0.2
0.4
0.4
0.6
0.6
0.8
0.8
(a) N = 200
(c) N = 200, true F shown in red
(d) N = 1000, true F shown in red
Figure 5.3: Confidence sets for the ecdf
167
Listing 6 Finding the value of k1

K <- function ( s ) { # Kolmogorov cdf
a <- sqrt (2 * pi ) / s
b <- 0
for ( i in 1:20) {
b <- b + exp ( - (2 * i - 1) ^2 * ( pi ^2 / (8 * s ^2) ) )
}
return ( a * b )
}
alpha <- 0.05
f <- function ( s ) return ( K ( s ) - 1 + alpha )
ur <- uniroot (f , lower =1.2 , upper =1.5)
print ( ur )
5.3.1
The Framework
Hypothesis testing begins with the specification of a null hypothesis, which is a

statement that the observed data is being generated by a certain model, or one of a
certain class of models. For example, suppose we observe data pairs ( xn , yn ). Our
null hypothesis might be that all pairs are generated by a particular functional relationship y = f ( x ) + u, where f is a specific function and u has a specific distribution.
Alternatively, our null hypothesis might be that the distribution of u belongs to a
certain parametric class of distributions, and that the function f lies in some particular set of functions H, such as the increasing functions, or the twice differentiable
functions, or the set of 3rd order polynomials.
One might imagine that the standard procedure in statistics is to show the validity of
the null hypothesis, but it is not. Rather, a hypothesis test is an attempt reject the null.
Karl Popper (19021994) was a major originator and proponent of this approach. To
illustrate why we should focus on rejection, Popper used the example of testing the
theory that all swans are white. (Consider this to be our null hypothesis.) It is futile
to attempt to prove this theory correct: Regardless of how many white swans we
find, no amount can ever confirm the claim that that all swans on planet earth are
white. On the other hand, a single black swan can show the claim to be false. In this
sense, attempting to falsify a theory (i.e., reject the null) is more constructive than
attempting to confirm it.
168
Although we attempt to reject the null hypothesis, we do so only if strong evidence

against it is observed. To understand how this convention came about, suppose
that we have a collection of theories about how the economy works. The procedure
would then be to step through the theories, at each stage taking correctness of the
theory as the null hypothesis and attempting to reject. If the theory is rejected then
we can discard it, and go on to the remaining theories. This is a useful process of
elimination. However, we dont want to mistakenly discard a good theory. Hence,
we only reject when there is strong evidence against the null hypothesis.
5.3.2
Constructing Tests
Lets look at testing in the parametric setting of 5.2.1. We have a parametric class of
models { M } indexed by parameter R. We let F be the joint distribution
of the sample vector x when the data is generated by M . We use the notation P
to refer to probabilities for x. For example, given B R N , we let P {x B} be the
probability that x B given x F . A null hypothesis is a specification of the set of
models { M } that we believe generated x. This amounts to specifying a subset 0
of . The null hypothesis is often written as
H0 : 0
If 0 is a singleton, then the null hypothesis is called a simple hypothesis. If not,
then the null hypothesis is called a composite hypothesis.
A test of the null hypothesis amounts to a test of whether the observed data was
generated by M for some 0 . Formally, a test is a binary function mapping
the observed data x into {0, 1}. The decision rule is5
if (x) = 1, then reject H0
if (x) = 0, then do not reject H0
Remark 5.3.1. Note that, prior to implementation of the test, (x) is to be considered
as a random variable, the distribution of which depends on the distribution of x
and the function . Note also that failing to reject H0 should not be confused with
accepting H0 ! More on this below.
texts identify tests with a rejection region, which is a subset R of R N . (R N is the set of Nvectorssee chapter 2 for more.) The null is rejected if x R. This is equivalent to our formulation:
If a rejection region R is specified, then we take to be defined by (x) := 1{x R}. Conversely, if
is specified, then we take R as equal to {s R N : (s) = 1}.
5 Some
169
Remark 5.3.2. Although we are using the language of parametric hypothesis testing,
this is only for convenience. We can also think of as an arbitrary index over a
(possibly nonparametric) class of models.
The outcome of our test depends on the random sample x, and, being random, its
realization can be misleading. There are two different ways in which the realization
can mislead us. First, we can mistakenly reject the null hypothesis when it is in fact
true. This is called type I error. Second, we can fail to reject the null hypothesis
when it is false. This is called type II error.
The power function associated with the test is the function
( ) = P { ( x ) = 1 }
( )
In other words, ( ) is the probability that the test rejects when the data is generated
by M . Ideally, we would like ( ) = 0 when 0 , and ( ) = 1 when
/ 0 .
In practice, this is usually difficult to achieve.
As discussed above, we tend to be conservative in rejecting the null, because we
dont want to discard good theories. For this reason, it is traditional to keep the
probability of type I error small. Then, if our test tells us to reject the null, its
unlikely the null is true. Because of this, the standard procedure is to choose a small
number such as 0.05 or 0.01, and then adjust the test such that
( )
for all 0
(5.12)
If (5.12) holds, then the test is said to be of size .

In constructing tests, a common (but not universal) set up is to define a real-valued
test statistic T and a critical value c, and then set6
( x ) : = 1{ T ( x ) > c }
(5.13)
The pair ( T, c) then defines the test, and the rule becomes:
Reject H0 if and only if T (x) > c
6A
minor point: A test statistic is a kind of statistic, in the sense that it is a function of the data.
Usually, statistics are thought of as being computable given the data. This means that they do not
depend on any unknown quantities. In the case of a test statistic, it is common to allow the test
statistic to depend on unknown parameters (a subset of ), with the caveat that the values of these
unknown parameters are all pinned down by the null hypothesis. (Otherwise the test cannot be
implemented.)
170
Example 5.3.1. Suppose we know that x1 , . . . , x N N (, 1) for some R,

but the value of is unknown. The null hypothesis is 0, or 0 = (, 0]. To
construct a test, we seek a suitable test statistic. Since we want to make inference
about the mean, a natural choice for our statistic is the sample mean, so that
IID
T (x) :=: T ( x1 , . . . , x N ) := x N
Each c R now gives us a test via (5.13), with power function ( ) = P { x N > c}.
We can obtain a clearer expression for the power function by observing that x N
N (, 1/N ). As a result, if is the cdf of the standard normal distribution on R and
z , then
P { x N c} = P{ + N 1/2 z c} = P{z N 1/2 (c )} = [ N 1/2 (c )]
( ) = 1 [ N 1/2 (c )]
(5.14)
Given c, the power function is increasing in , because higher pushes up the mean
of x N , making the event { x N > c} more likely. Given , the function is decreasing
in c, because higher c makes the event { x N > c} less likely. A plot of is presented
in figure 5.4 for two different values of c.7
5.3.3
Choosing Critical Values
Lets think a bit more about the test in (5.13). Typically, the choice of T is suggested
by the problem. For example, if our hypothesis is a statement about the second moment of a random variable, then we might take T to be the sample second moment.
Once T is fixed, we need to adjust c such that ( T, c) attains the appropriate size.
Thus, the standard procedure is to:
1. choose a desired size according to our tolerance for type I error,
2. identify a suitable test statistic T, and
3. choose a critical value c so that ( T, c) is of size .
In performing the last step, its common to choose c such that (5.12) holds with
equality. In this case, the problem is to choose c to solve
= sup P { T (x) > c}
0
7N
is fixed at 10.
(5.15)
171
1.0
0.6
0.4
0.0
0.2
power
0.8
c = 0.2
c = 0.8
Figure 5.4: The power function
Figure 5.5 gives an illustration. In the figure, weve taken 0 to be the two element
set { a , b }. The blue line gives an imaginary distribution of T (x) when x Fa ,
represented as a density. The black line gives the same for b . Assuming that a
value of be prescribed, the next step is to determine the value of c such that (5.15)
holds. Here, we choose c such that the largest of the two shaded areas is equal to .
IID
Example 5.3.2. Lets look again at example 5.3.1, where x1 , . . . , x N N (, 1) for

unknown, and our null hypothesis is 0. Given , our task is to find the
appropriate critical value c so that the test ( T, c) is of size . To solve for c given
we use (5.15). Applying the expression for the power function in (5.14), this becomes
= sup{1 [ N 1/2 (c )]}
0
The right-hand side is increasing in , so the supremum is obtained by setting = 0.

Setting = 0 and solving for c, we obtain
c() := N 1/2 1 (1 )
where 1 is the quantile function of the standard normal distribution. In R, this
function can be evaluated using qnorm. For example,
172
a
b
Figure 5.5: Determining the critical value
> alpha = 0.05

> qnorm (1 - alpha )
[1] 1.644854
Since 1 is increasing, we see that smaller corresponds to larger c()we reduce
the probability of type I error by increasing the critical value that the mean x N must
obtain for rejection to occur. Also, higher N brings down c(): More data allows us
to reduce the critical value without increasing the probability of rejecting a true null.
5.3.4
p-Values
Typically, a test that rejects at size 0.05 will also reject at size 0.1, but may not reject at
size 0.01. Lower means less tolerance for type I error, and forces the critical value
to become larger. Hence, for a fixed value of the test statistic, the result of the test
may switch from reject to accept. A natural question, then, is: What is the smallest
value of at which we can still reject a given test statistic? This value is called the
p-value.
173
Lets give a formal definition. Consider again the general parametric setting of 5.3.
We have a null hypothesis is H0 : 0 and, for each (0, 1), a test ( T, c()) of
size . We assume here that c() is determined via the relationship in (5.15). In this
setting, the p-value of the test is defined as
p(x) := inf{ (0, 1) : c() < T (x)}
Roughly speaking, this is the at which the test switches from accept to reject. Typically 7 c() is continuous, and in this case the expression for p(x) reduces to
p(x) := the such that c() = T (x)
(5.16)
Example 5.3.3. Recall the test (5.21). Here c() := 1 (1 /2), and c() is continuous in , so we can apply the definition of p(x) in (5.16). To solve for p(x), then,
we need to solve for in the expression 1 (1 /2) = |t N (x)|. With a bit of
rearranging and an application of symmetry (page 17), we obtain
p(x) = 2(|t N (x)|)
5.3.5
(5.17)
Asymptotic Tests
As we saw in examples 5.3.1 and 5.3.2, constructing appropriate tests requires knowledge of the distribution of the test statistic. In many cases, however, we know relatively little about the distribution of the test statistic. Often we dont want to make
parametric assumptions about the distribution of the underlying data x. And even if
we make such assumptions, it may still be hard to work out the implied distribution
of a given statistic T (x).
Fortunately, for many problems, the central limit theorem and its consequences provide an elegant solution: Even if we dont know the distribution of the test statistic,
we may still be able to determine its asymptotic distribution via the CLT. Once we
know the asymptotic distribution, we have an idea of the power of the test, at least
for large samples, and can hopefully choose critical values to obtain an appropriate
size.
To go down this path we need to switch to a notion of asymptotic size, rather than
finite sample size. Writing N instead of to emphasize the fact that the power
function typically depends on sample size, a test is called asymptotically of size
if
lim N ( ) for all 0
(5.18)
N
174
To illustrate how we might go about constructing a test which is asymptotically

of size , consider the data presented in figure 5.6. The histogram is of standardized daily returns on the Nikkei 225 index from January 1984 until May 2009. Here
standardized means that we have subtracted the sample mean and divided by the
sample deviation. If the original returns were normally distributed, then the standardized returns would be approximately standard normal. The code for producing
the histogram is in the first part of listing 7. The data file can be downloaded at
http://johnstachurski.net/emet/nikkei_daily.txt
The standard normal density has been superimposed on the histogram in blue. We
can see that the fit is not particularly good. The histogram suggests that the density
of returns is more peaked and has heavier tails than the normal density. This is a
common observation for asset price returns.
Lets think about making this analysis more precise by constructing an asymptotic
test. The construction of the test essentially inverts the confidence set in 5.2.3.
To begin, let be the standard normal cdf, and let the null hypothesis be that standardized returns are IID draws from . Let be given, and let k1 = K 1 (1 ) be
the 1 quantile of the Kolmogorov distribution K, as defined in (5.11). Finally, let
FN be the ecdf of the data. If the null hypothesis is true, then, by (5.10) on page 164,
we have
d
N sup | FN (s) (s)| K
(5.19)
s
For the test

N (x) := 1
)
N sup | FN (s) (s)| > k1
s
let N () be the value of the power function when the null hypothesis is true. By
(5.19), we have
(
)
lim N () = lim P
N sup | FN (s) (s)| > k1 =
N
Hence (5.18) is verified, and the test is asymptotically of size .
The value of the statistic N supsR | FN (s) (s)| produced by listing 7 is 5.67. If
= 0.05, then, as shown in 5.2.3, the critical value k1 is 1.36. The test statistic
exceeds the critical value, and the null hypothesis is rejected.
175
0.0
0.1
0.2
0.3
0.4
0.5
0.6
returns
Figure 5.6: Standardized daily returns, Nikkei 225
Looking back on what weve done, theres an obvious weakness in our approach.
Our null hypothesis was that standardized returns are IID draws from the standard
normal density. The rejection suggests this null is false, but it may be the IID assumption rather than the normality assumption that makes our null a poor fit to
the data. The test weve implemented can be modified to tackle the case of dependent data, but such a discussion is beyond the scope of these notes. See Negri and
Nishiyama (2010) for one such test and many references.
5.4
Use and Misuse of Testing
[roadmap]
5.4.1
Testing and Model Selection
Although the mechanics of hypothesis testing are relatively straightforward, in practice the scientific paradigm based around hypothesis testing is the subject of contin-
176
Listing 7 Testing normality

nikkei <- read . table ( " nikkei _ daily . txt " , header =T , sep = " ," )
dates <- as . Date ( rev ( nikkei $ Date ) )
close <- rev ( nikkei $ Close )
returns <- diff ( log ( close ) )
data <- ( returns - mean ( returns ) ) / sd ( returns )
hist ( data , breaks =100 , prob =T , xlab = " returns " ,
ylab = " " , main = " " , xlim = c ( -6 ,6) )
xgrid <- seq ( min ( data ) , max ( data ) , length =400)
lines ( xgrid , dnorm ( xgrid ) , col = " blue " )
FN <- function ( s ) return ( mean ( data <= s ) )
FN <- Vectorize ( FN )
m <- max ( abs ( FN ( xgrid ) - pnorm ( xgrid ) ) )
print ( sqrt ( length ( data ) ) * m )
# ecdf
ual debate. For example, consider the current state of hypothesis testing in macroeconomics in particular. In the 1970s, the rational expectations revolution ushered in
a brand new class of macroeconomic models. One of the first things the proponents
of rational expectations did was to test these models against data. The results were
disappointing. Thomas Sargents account runs as follows (Evans and Honkapohja,
2005):
My recollection is that Bob Lucas and Ed Prescott were initially very enthusiastic about rational expectations econometrics. After all, it simply
involved imposing on ourselves the same high standards we had criticized the Keynesians for failing to live up to. But after about five years of
doing likelihood ratio tests on rational expectations models, I recall Bob
Lucas and Ed Prescott telling me that those tests were rejecting too many
good models.
As a result, many proponents of these good models have moved away from formal hypothesis testing in favor of so-called calibration. The consensus of this school
of though can be paraphrased as No model is a true description of the real world.
Hence, I already know that my model is wrong. Rejection of my model using standard
inference tells me nothing new.
177
This line of argument runs contrary to the standard paradigm under which most of
science and statistical testing takes place. The standard paradigm recognizes that all
models are wrong. Given that all models are wrong, we try to cull old ones that perform poorly and produce new ones that perform better. The process of culling takes
place by statistical rejection. Some models are easily rejected. Others are harder
to reject. The overall outcome resembles survival of the fittest. Those models that
explain observed phenomena effectively, and, at the same time, are most difficult to
reject, will survive.8
On the other hand, perhaps we need to acknowledge that social science is not physics,
and our models tend to be in some sense (in many senses?) more imperfect than
those used in the natural sciences. Often they capture a phenomenon of interest
along one dimension, but fail to match with reality when viewed in other ways.
In other words, for any given model, a poor fit to the data can usually be found
from at least one of its implications, thus enabling rejection. But perhaps some of
these models are still useful in guiding our thinking, and should not be so readily
discarded.
[To be completed. Importance of prediction to prove that we really understand. It
should be out of sample. Also, Bayesian approach. Summary.]
5.4.2
Accepting the Null?
Power and lack of power. Failing to reject doesnt mean the null is right.
Example 1: Unit root tests. Example 2: Stochastic dominance tests (null is lack
of dominance. See Davidson and Duclos, 2013. Testing for Restricted Stochastic
Dominance, Econometric Review.) Write it up. Run a simulation. Also, how about
diagnostic tests for regression?
8 Incidentally,
given this description of model selection, it might appear there is an easy way for
us to produce a model that survives forever in the pool of currently acceptable theories, without ever
being culled via rejection: Just make sure that the model has no testable implications. However, this
strategy does not work, because a model with no testable implications cannot be considered as a
theory of anything. Indeed, the most common definition of a scientific theory, as proposed by Karl
Popper, is that the theory has one or more testable implications. In other words, the theory can be
falsified.
5.5
178
Further Reading
To be written.
5.6
Exercises
Ex. 5.6.1. Show that (5.7) is valid when (5.5) and (5.6) hold.
d
Ex. 5.6.2. Let (0, 1) and set z/2 := 1 (1 /2). Show that if t N (x)
N (0, 1) whenever H0 is true, and TN (x) = |t N (x)|, the sequence of tests N (x) :=
1{ TN (x) > z/2 } is asymptotically of size .
Ex. 5.6.3. Let x1 , . . . , x N be an IID sample with mean and variance 2 . Assume that
both and are unknown. We wish to test the hypothesis H0 : = 0 . Consider the
statistic
x N 0
(5.20)
t N := N
sN
With reference to exercise 5.6.2, show that the sequence of tests
N (x) := 1{|t N | > z/2 }
(5.21)
is asymptotically of size .
Ex. 5.6.4. If a test is well designed, then the rejection probability will converge to
one as N whenever H0 is false:
N ( ) 1 as N
whenever
Such a test is said to be consistent. Show that the test in exercise 5.6.3 is consistent
whenever (0, 1).9
Ex. 5.6.5 (Computational). The chi-squared goodness of fit test is used to test whether
or not a given data set x1 , . . . , x N was generated from a particular discrete distribution, described by a pmf p1 , . . . , p J over values 1, . . . , J. More precisely, let x1 , . . . , x N
be N random variables, each xn taking integer values between 1 and J. The null
9 Hint:
In essence, you need to show that the absolute value of the test statistic |t N | in (5.20)
diverges to infinity when 6= 0 . Try the add and subtract strategy, replacing the expresion x N 0
in (5.20) with ( x N ) + ( 0 ).
179
hypothesis of the test is that the sample x1 , . . . , x N is IID with P{ xn = j} = p j for

n {1, . . . , N } and j {1, . . . , J }. The test statistic is given by
J
( q j p j )2
X := N
pj
j =1
(5.22)
where q j is the fraction of the sample x1 , . . . , x N taking the value j. Write a function in
R called chsqts that takes two arguments observations and p, where observations
is a vector storing the sample x1 , . . . , x N , and p is a vector storing the values p1 , . . . , p J .
The function call chsqts(observations, p) should return the value X in equation
(5.22).10
Ex. 5.6.6 (Computational). This exercise continues on from exercise 5.6.5. Under
the null hypothesis, X in (5.22) is asymptotically chi-squared with J 1 degrees
of freedom. Let J = 3 with p1 = 0.2, p2 = 0.2 and p3 = 0.6. Let N = 20. By
repeatedly simulating 20 IID observations x1 , . . . , x20 from this pmf,11 generate 5,000
independent observations of the statistic X, and store them in a vector called obsX.
Plot the ecdf of obsX. In the same figure, plot the chi-squared cdf with 2 degrees of
freedom. The functions should be close.12
5.6.1
Solution to Exercise 5.6.1. We aim to show that (N )/ se(N ) converges in dis
p
d
N )
tribution
to
a
standard
normal
when
N
(
N
(
0,
v
(
))
and
N se(N )
p
v( ). To see this, observe that
p
v( )
N
N (N )
p
= N N where N :=
and N :=
se( N )
v( )
N se(N )
Using the various rules for converges in probabilty and distribution (check them
p
carefullysee facts 1.4.1 and 1.4.4) we obtain N Z N (0, 1) and N 1.

d
Applying Slutskys theorem (page 34) now gives N N Z, as was to be shown.

10 R
has a built-in function called chisq.test for implementing the chi-squared goodness of fit
test. Do not use this built-in function in your solution.
11 In particular, draw each x such that P{ x = j } = p for j = 1, 2, 3.
n
n
j
12 You can improve the fit further by taking N larger. The reason is that the fit is only asymptotic,
rather than exact, and N = 20 is not a large sample.
180
Solution to Exercise 5.6.2. Fix (0, 1) and let z N (0, 1). In view of (1.14) on
d
page 21, we have P{|z| > z/2 } = . If H0 is true, then t N (x) z by assumption.
d
Since g(s) := |s| is continuous, fact 1.4.4 on page 34 implies that |t N (x)| |z|. As a
result, we have
lim N ( ) = lim P {|t N (x)| > z/2 } = P{|z| > z/2 } =
This confirms (5.18), and the exercise is done.

Solution to Exercise 5.6.3. In 4.2.2 we showed that the sample standard deviation
s N defined in (4.2) is consistent for the standard deviation . Appealing to (1.30) on
page 37 and fact 1.4.5 on page 34, we can see that

x N 0
x N 0
t N := N
=
N
sN
sN
is asymptotically standard normal whenever H0 is true (i.e., 0 = ). It follows that

exercise 5.6.2 can be applied. In particular, we can state that for z/2 := 1 (1
/2), the test is asymptotically of size .
Chapter 6
Linear Least Squares
[roadmap]
6.1
Least Squares
[roadmap]
6.1.1
Multivariate Least Squares and ERM
Lets consider a multivariate version of the regression problem we studied in 4.6.

Suppose we repeatedly observe a vector input x to a system, followed by a scalar
output y. Both are random, and x takes values in RK . We assume that the inputoutput pairs (x1 , y1 ), . . . , (x N , y N ) we observe are all draws from some common joint
distribution on RK +1 . This distribution is unknown to us. Our aim is to predict
new output values from observed input values. In particular, our problem for this
chapter is to
choose a function f : RK R such that f (x) is a good predictor of y
Throughout this chapter, we will be using quadratic loss as our measure of goodness, so our expected loss (risk) from given function f is
R( f ) := E [(y f (x))2 ]
181
(6.1)
CHAPTER 6. LINEAR LEAST SQUARES
182
As we learned in chapter 3, the minimizer of the risk is f (x) = E [y | x]. Since the
underlying distribution is not known, we cannot compute this conditional expectation. Instead, we will use the principle of empirical risk minimization instead, replacing the risk function with the empirical risk before minimizing. In other words,
we solve
1 N
min R( f ) where R( f ) :=
(yn f (xn ))2
N n =1
f F
This is the general least squares problem. The function f is chosen from a set of
candidate functions F mapping RK into R. As before, F is called the hypothesis
space.
When we presented the theory of empirical risk minimization in 4.6, we assumed
that the input-output pairs (x1 , y1 ), . . . , (x N , y N ) were independent of each other.
If this is true, then, for fixed f , the scalar random variables (yn f (xn ))2 are also
independent (fact 2.4.1 on page 73), and, by the law of large numbers,
1
R ( f ) :=
N
(yn f (xn ))2 E [(y f (x))2 ] = R( f )
(6.2)
n =1
This gives a fundamental justification for the empirical risk minimization principle:
for large N, the empirical risk and the true risk are close.
Let us note at this stage that (6.2) can hold under much weaker assumptions than
independence. For example, (6.2) can hold when the input-output pairs form a time
series (each n is a point in time), and correlation between the input-output pairs is
present, provided that this correlation dies out sufficiently quickly over time. An
extensive discussion of this ideas is given in chapter 8. For now, we will not make
any particular assumptions. Just keep in mind that validity of (6.2) is a minimal
requirement to justify the approach that we are taking.
Lets now turn to the hypothesis space F . If we take F to be the set of all functions
from RK to R, we will usually be able to make the empirical risk R ( f ) arbitrarily
small by choosing a function f such that yn f (xn ) is very small for all n. However,
as we discussed extensively in 4.6.2, this is not the same think as making the risk
small, which is what we actually want to minimize. Thus, F must be restricted, and
in this chapter we consider the case F = L, where L is the linear functions from RK
to R. That is,
L := { all functions ` : RK R, where `(x) = b0 x for some b RK }
(6.3)
183
The problem we need to solve is then min` L nN=1 (yn `(xn ))2 , or, more simply,
N
( y n b 0 x n )2
b R n =1
min
(6.4)
(The constant 1/N has been dropped since it does not affect the minimizer.) This is
the multivariate version of (4.29) on page 145. Although our presentation is rather
modern, the idea of choosing b to minimize (6.4) is intuitive, and this optimization
problem has a very long tradition. It dates back at least as far as Carl Gausss work
on the orbital position of Celes, published in 1801.
One might well ask whether the choice F = L is suitable for most problems we
encounter. This is an excellent question. It may not be. However, setting F = L
allows us to obtain an analytical expression for the minimizer, which greatly simplifies computations. The derivation is in 6.1.2 below. Moreover, the technique
has a very natural extension from L to a very broad class of functions, as described
in 6.2.1.
6.1.2
The Least Squares Estimator
The next step is to solve (6.4). In fact, armed with our knowledge of overdetermined
systems (see 3.2), we already have all the necessary tools. This will be more obvious
after we switch to matrix notation. To do this, let
xn1
y1
xn2
y2
(6.5)
y := . , xn := . = n-th observation of all regressors
.
.
.
.
yN
and
xnK
X :=
x10
x20
..
.
x0N
:=:
x1K
x2K
..
.
x N1 x N2
x NK
x11
x21
..
.
x12
x22
..
.
(6.6)
We will use the notational convention colk (X) := the k-th column of X. In other
words, colk (X) is all observations of the k-th regressor. Throughout this chapter, we
will always maintain the following assumption on X, which is usually satisfied in
applications unless youre doing something silly.
184
Assumption 6.1.1. N > K and the matrix X is full column rank.

Lets transform the minimization problem (6.4) into matrix form. Note that, for any
b RK , we have
0 0
x1 b
b x1
x 0 b b 0 x2
2
Xb = . = .
.
.
. .
b0 x N
x0N b
Regarding the objective function in (6.4), with a little bit of effort, you will be able to
verify that
N
(yn b0 xn )2 = ky Xbk2
n =1
Moreover, since increasing transforms dont affect minimizers (see 11.2), we have
argmin ky Xbk2 = argmin ky Xbk
b
RK
RK
(6.7)
In summary, any solution to the right-hand side of (6.7) is a minimizer of (6.4) and
vice versa. The significance is that we already know now to solve for the minimizer
on the right-hand side of (6.7). By theorem 3.2.1 (page 93), the solution is
:= (X0 X)1 X0 y
(6.8)
Traditionally, this random vector is called the least squares estimator, or the OLS
estimator. (Right now, this terminology doesnt fit well with our presentation, since
we havent really assumed that is an estimator of anything in particular. In chapter 7 we will add some parametric structure to the underlying model, and will
become an estimator of an unknown parameter vector .)
6.1.3
Standard Notation
Theres a fair bit of notation associated with linear least squares estimation. Lets try
to collect it in one place. First, let P and M be the projection matrix and annihilator
associated with X, as defined on page 94. The vector Py = X is often denoted y,
and called the vector of fitted values:

y := X = Py
185
The n-th fitted value y n is the prediction x0n associated with OLS estimate and the
n-th observation xn of the input vector.
and called the vector of residuals:
The vector My is often denoted u,
u := My = y y
The vector of residuals corresponds to the error that occurs when y is approximated
by its orthogonal projection Py. From theorem 3.1.4 on page 91 we have
My Py
and
y = Py + My
(6.9)
In other words, y can be decomposed into two orthogonal vectors Py and My,
where the first represents the best approximation to y in rng(X), and the second
represents the error.
Related to the fitted values and residuals, we have some standard definitions:
Total sum of squares :=:
TSS
: = k y k2 .
Sum of squared residuals :=:
SSR
:= kMyk2 .
Explained sum of squares :=:
ESS
:= kPyk2 .
By (6.9) and the Pythagorean law (page 86) we have the following fundamental
relation:
TSS = ESS + SSR
(6.10)
6.2
Transformations of the Data
[roadmap]
6.2.1
Basis Functions
Lets now revisit the decision to set F = L made in 6.1.1. As we saw in 4.6.2, the
choice of F is crucial. In that section, we considered data generated by the model
y = cos(x ) + u
where
u N (0, 1)
(6.11)
186
We imagined that this model was unknown to us, and attempted to minimize risk
(expected quadratic loss) by minimizing empirical risk with different choices of hypothesis space F . We saw that if F is too small, then no function in F provides a
good fit to the model, and both the empirical risk and the risk are large (figure 4.18
on page 148). This is called underfitting. Conversely, if F is too big, then the empirical risk can be made small, but this risk itself is large (figure 4.21 on page 150). We
are paying too much attention to one particular data set, causing overfitting.
As we learned in 3.3, in our quadratic loss setting, the minimizer of the risk is
the conditional expectation of y given x. Since our goal is to make the risk small,
it would be best if F contained the conditional expectation. As our information
is limited by the quantity of data, we would also like F to be small, so that the
minimizer of the empirical risk has to be close to the risk minimizer E [y | x]. Of
course choosing F in this way is not easy since E [y | x] is unknown. The ideal case
is where theory guides us, providing information about E [y | x].
From such theory or perhaps from more primitive intuition, we may suspect that,
for the problem at hand, the conditional expectation E [y | x] is nonlinear. For example, many macroeconomic phenomena have a distinctly self-reinforcing flavor
(poverty traps, dynamic network effects, deleveraging-induced debt deflation, etc.),
and self-reinforcing dynamics are inherently nonlinear. This seems to suggest that
setting F = L is probably not appropriate.
Fortunately, it is easy to extend our previous analysis to a broad class of nonlinear functions. To do so, we first transform the data using some arbitrary nonlinear
function : RK R J . The action of on x RK is
1 (x)
2 (x)
x 7 (x) = . R J
.
.
J (x)
In this context, the individual functions 1 , . . . , J mapping RK into R are referred to
as basis functions. Now we apply linear least squares estimation to the transformed
data. Formally, we choosing the hypothesis space to be
F := {all functions ` , where ` is a linear function from R J to R}

The empirical risk minimization problem is then
N
min
`
{yn `((xn ))}
n =1
(yn 0 (xn ))2
R n =1
= min
(6.12)

Switching to matrix notation, let
n := (xn )
and
:=
10
20
..
.
0N
187
1 (x1 )
1 (x2 )
..
.
1 (x N )
J ( x1 )
J ( x2 )
..
.
RN J
J (x N )
In transforming (6.12) into matrix form, the objective function can be expressed as
N
(yn 0 n )2 = ky k2
n =1
Once again, increasing functions dont affect minimizers, and the problem (6.12)
becomes
argmin ky k
(6.13)
RJ
Assuming that is full column rank, the solution is

:= (0 )1 0 y
Example 6.2.1. Suppose that K = 1, and xn is scalar valued. Consider the mononomial basis functions j ( x ) = x j1 , so that
( xn ) = n =
xn0
xn1
..
.
xnJ 1
j 1
j xn
(6.14)
j =1
This case corresponds to univariate polynomial regression, as previously discussed

in 4.6.2. There is a theorem proved by Karl Weierstrass in 1885 which states that, for
any given continuous function f on a closed interval of R, there exists a polynomial
function g such that g is arbitrarily close to f . On an intuitive level, this means that
if we take J large enough, the relationship in (6.14) is capable of representing pretty
much any (one-dimensional) nonlinear relationship we want.
In most of this chapter, we return to the elementary case of regressing y on x, rather
than on some nonlinear transformation (x). However, no loss of generality is entailed, as we can just imagine that the data has already been transformed, and x is
the result. Similarly, well use X to denote the data matrix instead of , and to
denote the least squares estimator (X0 X)1 X0 y.
6.2.2
188
The Intercept
Theres one special transformation that is worth treating in more detail: Adding an
intercept to our regression. To add an intercept, we use the transformation

(x) =
1
x
1
x1
..
.
xK
Well call the resulting matrix X instead of . (As mentioned at the end of the last
section, were going to use X to denote the data matrix from now on, even though
the data it contains may have been subject to some transformations. What this
means in the present case is that the first column of X is now 1, and each of the
remaining columns of X contains N observations on one of the non-constant regressors.) In practice, adding an intercept means fitting an extra parameter, and this
extra degree of freedom allows a more flexible fit in our regression.
One consequence of adding an intercept is that the vector of residuals must sum to
zero. To see why this is the case, observe that
10 My = 10 (I P)y = 10 y 10 Py = 10 y (P0 1)0 y = 10 y (P1)0 y
where the last equality uses the fact the P is symmetric (exercise 3.5.10). Moreover,
as exercise 6.5.5 asks you to confirm, we have P1 = 1 whenever 1 rng(X). Clearly
1 rng(X) holds, since 1 is a column vector of X.1 Therefore,
10 My = 10 y (P1)0 y = 10 y 10 y = 0
In other words, the vector of residuals sums to zero.
Its also the case that if the regression contains the intercept, then the mean of the
fitted values y = Py is equal to the mean of y. This follows from the previous
argument, because we now have
1
N
1 Remember
n =1
y n =
1 0
1
1
1
1 y = 10 Py = 10 y =
N
N
N
N
yn
n =1
that rng(X) is the span of the columns of X, and clearly each column is in the span.
6.3
189
Goodness of Fit
In traditional econometrics, goodness of fit refers to the in-sample fit of the model
to the data (i.e., how well the model fits the observed data, as opposed to the potentially more important question of how well the model would predict new y from
new x). The most common measure of goodness of fit is the coefficient of determination. It is usually represented by the symbol R2 (read R squared), and defined
as
ESS
kPyk2
R2 : =
=
TSS
k y k2
For any X and y we have 0 R2 1. The nontrivial inequality R2 1 follows from
the fact that for P and any point y, we always have kPyk kyk. This was discussed
in theorem 3.1.3 (page 88), and the geometric intuition can be seen in figure 3.4. The
closer is y to the subspace rng(X) that P projects onto, the closer kPyk/kyk will be
to one. An extreme case is when R2 = 1, a so-called perfect fit. If R2 = 1, then, as
exercise 6.5.7 asks you to verify, we must have Py = y and y rng(X).
Historically, R2 has often been viewed as a one-number summary of the success of a
regression model. As many people have noted, there are all sorts of problems with
viewing R2 in this way. In this section we cover some of the issues.
6.3.1
R Squared and Empirical Risk
One issue that arises when we equate high R2 with successful regression is that we
can usually make R2 arbitrarily close to one in a way that nobody would consider
good science: By putting as many regressors as we can think of into our regression.
Intuitively, as we add regressors we increase the column space of X, expanding it out
towards y, and hence increasings R2 . Put differently, the larger the column space X,
the better we can approximate a given vector y with an element of that column
space.
To see this more formally, consider two groups of regressors, X a and Xb . We assume
that Xb is larger, in the sense that every column of X a is also a column of Xb . Let P a
and Pb be the projection matrices corresponding to X a and Xb respectively. Let y be
given, and let R2a and R2b be the respective R squareds:
R2a
k P a y k2
:=
k y k2
and
R2b
k P b y k2
:=
k y k2
190
Since Xb is larger than X a , it follows that the column space of X a is contained in the
column space of Xb (exercise 6.5.11). It then follows from fact 3.1.2 on page 90 that
P a Pb y = P a y. Using this fact, and setting yb := Pb y, we obtain
R2a
=
R2b
kP a yk
kPb yk
2
kP a Pb yk
kPb yk
2
kP a yb k
kyb k
2
where the final inequality follows from theorem 3.1.3 on page 88. Hence R2b R2a ,
and regressing with Xb produces (weakly) larger R2 .
Lets look at this phenomenon from a more statistical perspective. There is a close
connection between R2 and empirical risk. (See (4.25) on page 144 for the definition
of the latter.) The R2 of a regression of y on X can be written as R2 = 1 SSR /kyk2 . If
we alter the regressors, we change SSR = kMyk2 while leaving other terms constant.
In particular, if we raise R2 by including more regressors, then the increase in R2
occurs because SSR is falling. Since
SSR
= kMyk =
(yn xn )2
n =1
we see that SSR is proportional to the empirical risk

1
R ( f ) =
N
(yn f (xn ))2
n =1
0
of our fitted function f (x) = x. Thus, the increase in R2 is due to a fall in empirical
risk. If we can drive the empirical risk to zero, then SSR = 0, R2 = 1 and we have a
perfect fit.
We can understand what is happening by recalling our discussion of empirical risk

minimization in 4.6.2. As we discussed, when we use empirical risk minimization,
our true goal is to minimize risk, rather than empirical risk. Empirical risk is just
proxy for risk, the latter being unobservable. Moreover, if we allow ourselves unlimited flexibility in fitting functional relationships, we can make the empirical risk
arbitrarily small, but this does not guarantee small risk. In fact, this excess of attention to the data set we have in hand often causes the risk to explode. Such an
outcome was presented in figure 4.17 on page 147.
Lets look at a small simulation that illustrates the idea. Suppose that xn and yn are
draws from a uniform distribution on [0, 1]. We will draw xn and yn independently,
191
0.8
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Figure 6.1: Minimizing empirical risk on independent data
so there is no relationship at all between these two variables. We will fit the polynomial 1 + 2 x + 3 x2 + K x K 1 to the data for successively larger K. In other
words, we will regress y on X = ( xnk1 )k,n where n = 1, . . . , N and k = 1, . . . , K.
Incrementing K by one corresponds to one additional regressor.
A simulation and plot is shown in figure 6.1. Here K = 8, so we are fitting a relatively flexible polynomial. Since x and y are independent, the best guess of y given
x is just the mean of y, which is 0.5. Nevertheless, the polynomial minimizes empirical risk by getting close to the sample points in this particular draw of observations.
This reduces SSR and increases the R2 . Indeed, for this regression, the R2 was 0.87.
Increasing K to 25, I obtained an R2 of 0.95. (The code is given in listing 8.) By this
measure, the regression is very successful, even though we know there is actually
no relationship whatsoever between x and y.
6.3.2
Centered R squared
Another issue with R2 is that it is not invariant to certain kinds of changes of units.
This problem is easily rectified, by using the so-called centered R2 in place of R2 .
192
Listing 8 Generating the sequence of R2

set . seed (1234)
N <- 25
y <- runif ( N )
x <- runif ( N )
X <- rep (1 , N )
KMAX <- 25
for ( K in 1: KMAX ) {
X <- cbind (X , x ^ K )
results <- lm ( y ~ 0 + X )
Py2 <- sum ( results $ fitted . values ^2)
y2 <- sum ( y ^2)
cat ( " K = " , K , " R ^2 = " , Py2 / y2 , " \ n " )
}
The main ideas are presented below.

First, R2 is invariant to changes of units that involve rescaling of the regressand y
(dollars versus cents, kilometers versus miles, etc.) because if y is scaled by R,
then
kPyk2
2 kPyk2
kPyk2
kPyk2
=
=
=
kyk2
kyk2
2 k y k2
k y k2
On the other hand, when the regression contains an intercept, R2 is not invariant to
changes of units that involve addition or subtraction (actual inflation versus inflation in excess of a certain level, income versus income over a certain threshold, etc.).
To see this, lets compare the R2 associated with y with that associated with y + 1,
where R. The R2 in the latter case is
kP(y + 1)k2
kPy + P1k2
kPy + 1k2
2 kPy/ + 1k2
kPy/ + 1k2
=
=
=
=
ky + 1k2
ky + 1k2
ky + 1k2
2 ky/ + 1k2
ky/ + 1k2
where the second inequality follows from the fact that 1 rng(X). Taking the limit
as , we find that the R squared converges to one. In other words, we can
make the R squared as large as we like, just by a change of units.
For this reason, many economists and statisticians use the centered R squared rather
than the R squared, at least when the regression contains an intercept. For the
193
purposes of this section, lets assume that this is the case (or, more generally, that
1 rng(X)). The centered R squared is defined as
R2c :=
kMc Pyk2
kPMc yk2
=
k M c y k2
k M c y k2
(6.15)
where
1 0
11
(6.16)
N
is the annihilator associated with 1. The equality of the two expressions for R2c is left
as an exercise (exercise 6.5.12). Hopefully it is clear that adding a constant to each
element of y will have no effect on R2c .
M c : = I 1 ( 1 0 1 ) 1 1 0 = I
The centered R squared can be re-written (exercise 6.5.13) as

R2c =
nN=1 (y n y )2
nN=1 (yn y )2
(6.17)
It is a further exercise (exercise 6.5.16) to show that, in the case of the simple regression, the centered R squared is equal to the square of the sample correlation
between the regressor and regressand, as defined in (4.4) on page 115. Thus, centered R squared can be thought of as a measure of correlation. As discussed below,
correlation should not be confused with causation.
6.3.3
A Note on Causality
You have probably heard R2 interpreted as measuring the explanatory power of

the regressors in a particular regression. The idea is that regression amounts to
decomposing y into the sum of two orthogonal parts, the fitted values Py and the
residuals My. By the Pythagorean theorem, the squared norm kyk2 =: TSS of y can
then be represented kPyk2 =: ESS plus kMyk2 =: SSR, as in (6.10). This is sometimes
paraphrased as the total variation in y is the sum of explained and unexplained
variation. The value of R2 is then claimed to be the fraction of the variation in y
explained by the regressors.
This is a poor choice of terminology, because the notion that the regressors explain variation in y suggests causation, and R2 says nothing about causation per
se. Instead, R2 is better thought of as a measure of correlation (see 6.3.2). As has
been observed on many occasions, correlation and causation are not the same thing.
Some informal examples are as follows:
194
We often see car crashes and ambulances together (correlation). This does not
imply that ambulances cause crashes.
It has been observed that motorcycles fitted with ABS are less likely to be involved in accidents than those without ABS. Does that mean fitting ABS to a
given motorcycle will reduce the probability that bike is involved in an accident? Perhaps, but another likely explanation is selection bias in the sample
cautious motorcyclists choose bikes with ABS, while crazy motorcyclists dont.
Suppose we observe that people sleeping with their shoes on often wake up
with headaches. One possible explanation is that that wearing shoes to bed
causes headaches. A more likely one is that both phenomena are caused by
too many pints at the pub the night before.2
Identifying causality in statistical studies can be an extremely difficult problem. This
is especially so in the social sciences, where properly controlled experiments are
often costly or impossible to implement. (If we stand someone on a bridge and tell
them to jump, are they more likely to do so? Try asking your national research body
to fund that experiment.) An excellent starting point for learning more is Freedman
(2009).
6.4
Further Reading
To be written.
6.5
Exercises
Ex. 6.5.1. Argue that the sample mean of a random sample y1 , . . . , y N from a given
distribution F can be viewed as a least squares estimator of the mean of F.
Ex. 6.5.2. Lets show that solves the least squares problem in a slightly different
way: Let b be any K 1 vector, and let := (X0 X)1 X0 y.
1. Show that ky Xbk2 = ky X k2 + kX( b)k2 .
2I
usually just leave my contacts in.
195
2. Using this equality, argue that is the minimizer of ky Xbk2 over all K 1
vectors b.
Ex. 6.5.3. Verify that nN=1 (yn b0 xn )2 = ky Xbk2 .
Ex. 6.5.4. Show carefully that any solution to minbRK ky Xbk2 is also a solution
to minbRK ky Xbk, and vice versa.3
Ex. 6.5.5. Explain why P1 = 1 whenever 1 rng(X).4
Ex. 6.5.6. Show that PM = MP = 0. Using this fact (and not the orthogonal projection theorem), show that the vector of fitted values and the vector of residuals are
orthogonal.
Ex. 6.5.7. Show that if R2 = 1, then Py = y and y rng(X).
Ex. 6.5.8. Suppose that the regression contains an intercept, so that the first column
of X is 1. Let y be the sample mean of y, and let x be a 1 K row vector such that
the k-th element of x is the sample mean of the k-th column of X. Show that y = x .
Ex. 6.5.9. Show that if R2 = 1, then every element of the vector of residuals is zero.
Ex. 6.5.10. Suppose the regression contains an intercept. Let Mc be as defined in
(6.16). Show that
kMyk2 = kMc yk2 kPMc yk2
(6.18)
always holds.5
Ex. 6.5.11. Let X a and Xb be N Ka and N Kb respectively. Suppose that every
column of X a is also a column of Xb . Show that rng(X a ) rng(Xb ).
Ex. 6.5.12. Confirm the equality of the two alternative expressions for R2c in (6.15).
Ex. 6.5.13. Verify the expression for R2c in (6.17).
Ex. 6.5.14. Show that the coefficient of determination R2 is invariant to a rescaling
of the regressors (where all elements of the data matrix X are scaled by the same
constant).
3 You
can use the ideas on optimization in the appendix, but provide your own careful argument.
Make sure you use the definition of a minimizer in your argument.
4 Hint: Refresh your memory of theorem 3.1.3 on page 88.
5 Hints: See fact 3.1.5 on page 92 and the Pythagorean law (page 86).
196
Ex. 6.5.15. Let x := ( x1 , . . . , x N ) and y := (y1 , . . . , y N ) be sequences of scalar random

variables. Show that the sample correlation $ between x and y (defined in (4.4) on
page 115) can be written as
(Mc x)0 (Mc y)
$ =
kMc xkkMc yk
Ex. 6.5.16. (Quite hard) Show that, in the case of the simple regression model
(see 7.3.3), R2c is equal to the square of the sample correlation between x and y.
Ex. 6.5.17 (Computational). Build an arbitrary data matrix X by simulation. Construct M and calculate MX. The resulting matrix should have all entries very close
to (but not exactly) zero. Test the orthogonality of My and Py. The inner product
should be very close to (but not exactly) zero.
Ex. 6.5.18 (Computational). Build an arbitrary data set X, y by simulation. Run
a regression with the intercept, and record the values of the estimated coefficients
of the non-constant (i.e., k 2) regressors. Confirm that these values are equal to
the estimated coefficients of the no-intercept regression after all variables have been
centered around their mean.
Ex. 6.5.19 (Computational). To be written: [R squared increasessee file rsquared.R]
Ex. 6.5.20 (Computational). The direct method of computing the OLS estimate via
the expression = (X0 X)1 X0 y may be problematic in some settings. The difficulty
is inverting the matrix (X0 X)1 when it is large and almost singular.6 To see this
in action, calculate the coefficients of (6.14) for successively higher and higher values of K, where each regression uses the same set of observations x = ( x1 , . . . , x N )
and y = (y1 , . . . , y N ). (Example 6.2.1 explains how to estimate the coefficients using multiple regression.) Let N = 20 and let x be an evenly spaced grid on [0, 1].
Generate y as yn = xn + N (0, 1). In each round of the loop, calculate the regression coefficients using first lm and then the direct method (computing (X0 X)1 via
the function solve). Set the program running in an infinite loop,7 where each iteration, print the current degree of the polynomial and the coefficients from the two
methods. You should find that the direct method fails first.
6 In
this setting, the inversion routine involves calculating many very small numbers. Since the
amount of memory allocated to storing each of these numbers is fixed, the result of the calculations
may be imprecise.
7 To set up an infinite loop, start with while(TRUE). To exist from an infinite loop running in the
terminal, use control-C.
6.5.1
197
Solution to Exercise 6.5.1. Letting := E [yn ] we can write yn = + un when un :=

yn . In other words, y = 1 + u. The OLS estimate of is
:= (10 1)1 10 y =
1
1 0
1y=
N
N
yn = y N
n =1
Reading right to left, the sample mean of y is the OLS estimate of the mean.
Solution to Exercise 6.5.2. Part 2 follows immediately from part 1. Regarding part
1, observe that
ky Xbk2 = ky X + X( b)k2
By the Pythagorean law, the claim
ky Xbk2 = ky X k2 + kX( b)k2
will be confirmed if y X X( b). This follows from the definition of ,

because for arbitrary a RK we have
(aX)0 (y X ) = a0 (X0 y X0 X(X0 X)1 X0 y) = a0 (X0 y X0 y) = 0
Solution to Exercise 6.5.4. Let be a solution to

min ky Xbk2
RK
which is to say that
ky X k2 ky Xbk2 for any b RK
If a and b and nonnegative constants with a b, then a b, and hence

ky X k ky Xbk for any b RK
In other words, is a solution to
min ky Xbk
RK
The vice versa argument follows along similar lines.
198
Solution to Exercise 6.5.7. If R2 = 1, then, by (6.10) on page 185, we have kMyk2 =

0, and hence have My = 0. Since My + Py = y, this implies that Py = y. But then
y rng(X) by 5 of theorem 3.1.3.
Solution to Exercise 6.5.9. If R2 = 1, then kPyk2 = kyk2 , and hence
kyk2 = kPyk2 + kMyk2 = kyk2 + kMyk2

Therefore kMyk2 = 0, and, by fact 2.1.1 on page 53, My = 0.
Solution to Exercise 6.5.10. Theorem 3.1.4 tells us that for any conformable vector z
we have z = Pz + Mz. where the two vectors on the right-hand side are orthogonal.
Letting z = Mc y, we obtain
Mc y = PMc y + MMc y
From fact 3.1.5 we have MMc y = My. Using this result, orthogonality and the
Pythagorean law, we obtain
kMc yk2 = kPMc yk2 + kMyk2

Rearranging gives (6.18)
Solution to Exercise 6.5.11. It suffices to show that if z rng(X a ), then z rng(Xb ).
Let x1 , . . . , x J be the columns of X a and let x1 , . . . , x J + M be the columns of Xb . If
z rng(X a ), then
J
z=
j xj
j =1
for some scalars 1 , . . . , J . But then

J
z=
j xj +
j =1
J+M
0 xj
j = J +1
In other words, z rng(Xb ).

Solution to Exercise 6.5.12. It is sufficient to show that
PMc = Mc P
199
Since 1 rng(X) by assumption, we have P1 = 1, and 10 P = (P0 1)0 = (P1)0 = 10 .

Therefore P110 = 110 P, and
PMc = P
1
1
P110 = P 110 P = Mc P
N
N
Solution to Exercise 6.5.13. It suffices to show that
kPMc yk2 =
(yn y)2
n =1
This is the case because

N
(yn y)2 = kPy 1yk2 = kPy P1yk2 = kP(y 1y)k2 = kPMc yk2
n =1
Solution to Exercise 6.5.14. This follows immediately from the definition of R2 , and
the fact that, for any 6= 0,
P = X ( X 0 X ) 1 X =
2
X(X0 X)1 X = (X)((X)0 (X))1 (X)
2
Solution to Exercise 6.5.16. From exercise 6.5.15, the squared sample correlation between x and y can be written as
$2 =
[(Mc x)0 (Mc y)]2

k M c x k2 k M c y k2
On the other hand, we have

R2c =
kMc Pyk2
k M c y k2
Therefore it suffices to show that, for the simple linear regression model in 7.3.3,
we have
|(Mc x)0 (Mc y)|
kMc Pyk =
(6.19)
kMc xk
200
Let X = (1, x) be the data matrix, where the first column is 1 and the second column
is x. Let
(Mc x)0 (Mc y)
1 := y 2 x and 2 :=
k M c x k2
be the OLS estimators of 1 and 2 respectively (see 7.3.3). We then have
Py = X = 1 1 + x 2
Mc Py = Mc X = Mc x 2
|(Mc x)0 (Mc y)|

kMc Pyk = kMc x 2 k = | 2 |kMc xk =
kMc xk
k M c x k2
Cancelling kMc xk we get (6.19). This completes the proof.
Part III
Econometric Models
201
Chapter 7
Classical OLS
In this chapter we continue our study of linear least squares and multivariate regression, as begun in chapter 6. To say more about whether linear least squares estimation is a good procedure or otherwise, we need to make more assumptions on
the process that generates our data. Not surprisingly, for some sets of assumptions
on the data generating process, the performance of linear least squares estimation
is good, while for other assumptions the performance is poor. The main purpose of
this chapter is to describe the performance of linear least squares estimation under
the classical OLS assumptions, where OLS stands for ordinary least squares.
To some people (like me), the standard OLS assumptions are somewhat difficult to
swallow. At the same time, the results obtained from these assumptions form the
bread and butter of econometrics. As such, they need to be understood.
7.1
The Model
[roadmap]
7.1.1
The OLS Assumptions
In chapter 6 we assumed that the observed input-output pairs (x1 , y1 ), . . . , (x N , y N )

were draws from some common joint distribution on RK +1 . To work with the classical OLS model, we have to impose (much!) more structure. For starters, we will
202
CHAPTER 7. CLASSICAL OLS
203
assume that the input-output pairs all satisfy

yn = 0 xn + un
(7.1)
where is an unknown K 1 vector of parameters, and u1 , . . . , u N are unobservable

random variables. The model is called linear because the deterministic part of the
output value 0 x is linear as a function of x. Letting u := (u1 , u2 , . . . , u N )0 be the
column vector formed by the N realizations of the shock, the N equations in (7.1)
can be expressed as
y = X + u
(7.2)
Here are three very traditional examples:
Example 7.1.1. The Cobb-Douglas production function relates capital and labor inputs with output via y = Ak ` where A is a random, firm-specific productivity
term and and are parameters. Taking logs yields the linear regression model
ln y = + ln k + ln ` + u
where the random term ln A is represented by + u.
Example 7.1.2. The elementary Keynesian consumption function is of the form C =
1 + 2 D, where C is consumption and D is household disposable income. Letting
:= ( 1 , 2 ) and x := (1, D ) and adding an error term u, we obtain the linear
regression model C = 0 x + u.
Example 7.1.3. Okuns law is an empirical rule of thumb relating output to unemployment. A commonly estimated form of Okuns law is U = 1 + 2 G + u, where
G is the growth rate of GDP over a given period, and U is the change in the unemployment rate over the same period (i.e., end value minus starting value). The
parameter 2 is expected to be negative. The value 1 / 2 is thought of as the rate
of GDP growth necessary to maintain a stable unemployment rate.
Let M be the annihilator associated with X. The proof of the next fact is an exercise
(exercise 7.7.1).
Fact 7.1.1. When (7.2) holds we have My = Mu and SSR = u0 Mu.
Regarding the shocks u1 , . . . , u N , the classical OLS model makes two assumptions,
one concerning the first moments, and the other concerning the variance. The first
assumption is as follows:
204
Assumption 7.1.1. Together, u and X satisfy E [u | X] = 0.

This assumption expresses a number of ideas in very compact form. Important
details are recorded in the next fact, the proof of which is an exercise (exercise 7.7.2).
Fact 7.1.2. Given assumption 7.1.1, we have:
1. E [u] = 0.
2. E [um | xnk ] = 0 for any m, n, k.
3. E [um xnk ] = 0 for any m, n, k.
4. cov[um , xnk ] = 0 for any m, n, k.
we will need an assumption on second moments of
To consider the variance of ,
the shock. The most standard assumption is as follows:
Assumption 7.1.2. Together, u and X satisfy E [uu0 | X] = 2 I, where is a positive
constant.
The value of the parameter is unknown, in the same sense that the vector of coefficients is unknown. To understand assumption 7.1.2, note that
var[u | X] := E [uu0 | X] E [u | X]E [u0 | X] = E [uu0 | X]
where the second equality is due to assumption 7.1.1. Hence, assumption 7.1.2 implies that the conditional variance-covariance matrix is the constant matrix 2 I.
Fact 7.1.3. Given assumption 7.1.2, we have the following results:
1. var[u] = E [uu0 ] = 2 I.
2. Shocks are homoskedastic: E [u2i | X] = E [u2j | X] = 2 for any i, j in 1, . . . , N.
3. Distinct shocks are uncorrelated: E [ui u j | X] = 0 whenever i 6= j.
7.1.2
205
The OLS Estimators
The standard estimator of the unknown parameter vector is the estimator defined in (6.8). To repeat:
:= (X0 X)1 X0 y
(7.3)
The standard estimator of the parameter 2 introduced in assumption 7.1.2 is
2 :=
SSR
NK
We will show below that, under the classical OLS assumptions given in 7.1.1, both
estimators have nice properties. As a precursor to the arguments, note that, applying (7.2), we obtain
:= (X0 X)1 X0 y = (X0 X)1 X0 (X + u) = (X0 X)1 X0 u
Adding to both sides yields
This deviation is known as the sampling error of .
the useful expression
= + (X0 X)1 X0 u
(7.4)
7.2
Variance and Bias
In this section we will investigate the properties of the estimators and 2 under
the standard OLS assumptions.
7.2.1
Bias
Under the linearity assumption y = X + u and the standard assumptions on the

shock, is an unbiased estimator of , and 2 is an unbiased estimator of 2 :
Theorem 7.2.1. Under assumption 7.1.1, we have E [ ] = E [ | X] = .
Theorem 7.2.2. Under assumptions 7.1.17.1.2, we have E [ 2 ] = E [ 2 | X] = 2 .
Proof of theorem 7.2.1. By (7.4) and assumption 7.1.1, we have
E [ | X] = + E [(X0 X)1 X0 u | X] = + (X0 X)1 X0 E [u | X] =

This proves E [ | X] = , and hence E [ ] = E [E [ | X]] = E [ ] = .
206
Proof of theorem 7.2.2. For reasons that will become apparent, we start the proof by
showing that trace(M) = N K. To see that this is so, observe that, recalling
fact 2.3.8,
trace(P) = trace[X(X0 X)1 X0 ] = trace[(X0 X)1 X0 X] = trace[IK ] = K
trace(M) = trace(I N P) = trace(I N ) trace(P) = N K
Now let mij (X) be the i, j-th element of M. Applying fact 7.1.1 on page 203, we have
"
#
N
E [ SSR | X] = E [u0 Mu | X] = E ui u j mij (X) | X = mij (X)E [ui u j | X]

i =1 j =1
i =1 j =1
In view of assumption 7.1.2, this reduces to

N
mnn (X)2 = trace(M)2 = ( N K)2
n =1
Hence E [ 2 | X] = 2 , and E [ 2 ] = E [E [ 2 | X]] = E [2 ] = 2 .
7.2.2
Variance of
Now that is known to be unbiased, we want to say something about the variance.
Theorem 7.2.3. Under assumptions 7.1.17.1.2, we have var[ | X] = 2 (X0 X)1 .
Proof. If A := (X0 X)1 X0 , then = + Au, and
var[ | X] = var[ + Au | X] = var[Au | X]
Since A is a function of X, we can treat it as non-random given X, and hence, by
fact 2.4.4 on page 74, we have
var[Au | X] = A var[u | X]A0 = A(2 I)A0
Moreover,
A(2 I)A0 = 2 AA0 = 2 (X0 X)1 X0 X(X0 X)1 = 2 (X0 X)1
var[ | X] = var[Au | X] = 2 (X0 X)1
7.2.3
207
The Gauss-Markov Theorem
Now that is known to be unbiased for under the OLS assumptions, the next
step is to show that has low variance. Although we obtained an expression for the
variance in theorem 7.2.3, its not clear whether this is low or high. The natural way
to answer this question is to compare the variance of with that of other unbiased
estimators. This leads us to the famous Gauss-Markov theorem.
Theorem 7.2.4 (Gauss-Markov). If b is any other linear unbiased estimator of , then
var[b | X] var[ | X], in the sense that var[b | X] var[ | X] is nonnegative definite.
The theorem is often summarized by stating that is BLUE. BLUE stands for Best
Linear Unbiased Estimator, and was discussed previously in 4.2.2.
There are a couple of points to clarify. First, the meaning of linearity: Although its
not immediately clear from the statement of the theorem, here linearity of b means
that b is linear as a function of y (taking X as fixed). In view of theorem 2.1.1 on
page 57, this is equivalent to requiring that b = Cy for some matrix C. The matrix
C is allowed to depend on X (i.e., be a function of X), but not y.
Second, how to interpret the statement that var[b | X] var[ | X] is positive definite? Matrices have no standard ordering, and hence its hard to say when one
random vector has larger variance than another. But nonnegative definiteness of
the difference is a natural criterion. In particular, all elements of the principle diagonal of a nonnegative definite matrix are themselves nonnegative, so the implication
is that var[bk | X] var[ k | X] for all k.
Third, the meaning of unbiasedness: In this theorem, it means that, regardless of the
value of (i.e., for any RK ), we have E [b | X] = E [Cy | X] = .
Proof of theorem 7.2.4. Let b = Cy, as described above, and let D := C A, where
A := (X0 X)1 X0 . Then
b = Cy = Dy + Ay = D(X + u) + = DX + Du +
(7.5)
Taking conditional expectations and using the fact that D is a function of X, we

obtain
E [b | X] = E [DX | X] + E [Du | X] + E [ | X]
= DX E [ | X] + D E [u | X] + E [ | X] = DX + 0 +
208
In light of the fact that b is unbiased, and, in particular, E [b | X] = for any given
, we have
= DX + for all RK
0 = DX
for all RK
In light of exercise 2.7.19 on page 82, we conclude that DX = 0. Combining this

Roughly speaking, this says
result with (7.5), we obtain the expression b = Du + .
that b is equal to the OLS estimator plus some zero mean noise.
To complete the proof, observe that
var[b | X] = var[Du + | X] = var[(D + A)u | X] = (D + A) var[u | X](D + A)0
Using assumption 7.1.2 and fact 2.3.5, the right-hand side of this expression becomes
2 (D + A)(D0 + A0 ) = 2 (DD0 + DA0 + AD0 + AA0 )
Since
DA0 = DX(X0 X)1 = 0(X0 X)1 = 0
and since
AA0 = (X0 X)1 X0 X(X0 X)1 = (X0 X)1
we now conclude that
var[b | X] = 2 [DD0 + (X0 X)1 ] = 2 DD0 + var[ | X]
Since 2 DD0 is nonnegative definite (why?), the proof is now done.
7.3
The FWL Theorem
The Frisch-Waugh-Lovell (FWL) Theorem yields, among other things, an explicit

While that might not
expression for an arbitrary sub-vector of the OLS estimator .
sound terribly exciting, it turns out to have many useful applications.
7.3.1
Statement and Proof
Continuing to work with our linear regression model, lets take y and X as given,
implying an OLS estimate = (X0 X)1 X0 y. Recall that y can be decomposed as
y = Py + My = X + My
(7.6)
209
We can write (7.6) in a slightly different way by partitioning the regressors and the
estimated coefficients into two classes: Let X1 be a matrix consisting of the first K1
columns of X, and let X2 be a matrix consisting of the remaining K2 := K K1
columns of X. Similarly, let
and
1. 1 be the K1 1 vector consisting of the first K1 elements of ,
2. 2 be the K2 1 vector consisting of the remaining K2 elements of .

We can then rewrite (7.6) as
y = X1 1 + X2 2 + My
(7.7)
Let P1 := X1 (X10 X1 )1 X10 be the projection onto the column space of X1 , and let
M1 := I P1 be the corresponding annihilator, projecting onto the orthogonal complement of the column space of X1 . With this notation we have the following result:
Theorem 7.3.1 (FWL theorem). The K2 1 vector 2 can be expressed as
2 = (X20 M1 X2 )1 X20 M1 y
(7.8)
The theorem gives an explicit analytical expression for our arbitrarily chosen subset
Before discussing its implications, lets present the proof.
2 of the OLS estimate .
Proof of theorem 7.3.1. Premultiplying both sides of (7.7) by X20 M1 , we obtain
X20 M1 y = X20 M1 X1 1 + X20 M1 X2 2 + X20 M1 My
(7.9)
The first and last terms on the right-hand side are zero. This is clear for the first term,
because M1 is the annihilator associated with X1 . Hence M1 X1 = 0. Regarding the
last term, it suffices to show that the transpose of the term is 00 . To see this, observe
that
(X20 M1 My)0 = y0 M0 M10 X2 = y0 MM1 X2 = y0 MX2 = 00
In the first equality we used the usual property of transposes (fact 2.3.5), in the
second we used symmetry of M and M1 (exercise 3.5.10 or direct calculation), in the
third we used fact 3.1.5 on page 92, and in the fourth we used the fact that M is the
annihilator for X, and hence MX2 = 0.
In light of the above, (7.9) becomes
X20 M1 y = X20 M1 X2 2
To go from this equation to (7.8), we just need to check that X20 M1 X2 is invertible.
The proof of this last fact is left as an exercise (exercise 7.7.10).
7.3.2
210
Intuition
As exercise 7.7.8 asks you to show, the expression for 2 in theorem 7.3.1 can be
rewritten as
2 = [(M1 X2 )0 M1 X2 ]1 (M1 X2 )0 M1 y
(7.10)
Close inspection of this formula confirms the following claim: There is another way
to obtain 2 besides just regressing y on X and then extracting the last K2 elements:
We can also regress M1 y on M1 X2 to produce the same result.
To get some feeling for what this means, lets look at a special case, where X2 is the
single column colK (X), containing the observations on the K-th regressor. In view
of the preceding discussion, the OLS estimate K can be found by regressing
y := M1 y = residuals of regressing y on X1
on
x K := M1 colK (X) = residuals of regressing colK (X) on X1
Loosely speaking, these two residual terms y and x K can be thought of as the parts
of y and colK (X) that are not explained by X1 . Thus, on an intuitive level, the
process for obtaining the OLS estimate K is:
1. Remove effects of all other regressors from y and colK (X), producing y and x K .
2. Regress y on x K .
This is obviously different from the process for obtaining the coefficient of the vector
colK (X) in a simple univariate regression, the latter being just
1. Regress y on colK (X).
In words, the difference between the univariate least squares estimated coefficient
of the K-th regressor and the multiple regression OLS coefficient is that the multiple
regression coefficient K measures the isolated relationship between xK and y, without
taking into account indirect channels involving other variables.
We can illustrate this idea further with a small simulation. Suppose that
y = x1 + x2 + u
where
IID
u N (0, 1)
211
If we generate N independent observations from this model and regress y the observations of ( x1 , x2 ), then, provided that N is sufficiently large, the coefficients for
x1 and x2 will both be close to unity.1 However, if we regress y on x1 alone, then the
coefficient for x1 will depend on the relationship between x1 and x2 . For example:
>
>
>
>
>
>
>
>
N <- 1000
beta <- c (1 , 1)
X1 <- runif ( N )
X2 <- 10 * exp ( X1 ) + rnorm ( N )
X <- cbind ( X1 , X2 )
y <- X % * % beta + rnorm ( N )
results <- lm ( y ~ 0 + X1 )
results $ coefficients
X1
30.76840
Here the coefficient for x1 is much larger than unity, because an increase in x1 tends
to have a large positive effect on x2 , which in turn increases y. The coefficient in the
univariate regression reflects this total effect.
7.3.3
Simple Regression
As an application of the FWL theorem, lets derive the familiar expression for the
slope coefficient in simple regression. Simple regression is a special case of multivariate regression, where the intercept is included (i.e., 1 is the first column of X)
and K = 2. For simplicity, the second column of X will be denoted simply by x. As
we saw in (4.30) on page 145, the OLS estimates are
N ( xn x )(yn y )
2 = n=1 N
n=1 ( xn x )2
and
1 = y 2 x
where x is the sample mean of x and y is the sample mean of y. The coefficient 2 is
known as the slope coefficient, while 1 is called the intercept coefficient.
We can rewrite 2 more succinctly as
)0 (x x1
)]1 (x x1
)0 (y y1
)
2 = [(x x1
1 The
(7.11)
reason is that, in this setting, the OLS estimator is consistent for the coefficients. A proof can
be found in chapter 7.
212
By the FWL theorem (equation 7.10), we also have

2 = [(Mc x)0 Mc x]1 (Mc x)0 Mc y
(7.12)
where Mc is the annihilator associated with the single regressor 1, as defined in

(6.16). It is straightforward to show that for this annihilator Mc and any z, we have
In other words, the annihilator associated with 1 converts vectors
Mc z = z z1.
into deviations around their mean. (The c in Mc reminds us that Mc centers vectors around their mean.)
It is now easy to see that the right-hand sides of (7.11) and (7.12) coincide.
7.3.4
Centered Observations
Lets generalize the discussion in the preceding section to the case where there are
multiple non-constant regressors. The only difference to the preceding case is that
instead of one column x of observations on a single non-constant regressor, we have
a matrix X2 containing multiple columns, each a vector of observations on a nonconstant regressor.
If the OLS estimate = (X0 X)1 X0 y is partitioned into ( 1 , 2 ), then we can write
X = 1 1 + X2 2
Applying the FWL theorem (equation 7.10) once more, we can write 2 as
2 = [(Mc X2 )0 Mc X2 ]1 (Mc X2 )0 Mc y
where Mc is the annihilator in (6.16). As we saw in the last section, Mc y is y centered
around its mean. Similarly, Mc X2 is a matrix formed by taking each column of X2
and centering it around its mean.
What we have shown is this: In an OLS regression with an intercept, the estimated
coefficients of the non-constant (i.e., non-intercept) regressors are equal to the estimated coefficients of a zero-intercept regression performed after all variables have
been centered around their mean.
7.3.5
Precision of the OLS Estimates
Lets return to the regression problem, with assumptions 7.1.17.1.2 in force. In theorem 7.2.3, we showed that the variance-covariance matrix of the OLS estimate
213
given X is 2 (X0 X)1 . The scalar variances of the individual OLS coefficient estimates 1 , . . . , K are given by the principle diagonal of this matrix. Since any one
of these OLS estimates k is unbiased (theorem 7.2.1), small variance of k corresponds to probability mass concentrated around the true parameter k . In this case,
we say that the estimator has high precision. (Precision of an estimator is sometimes
defined as the inverse of the variance, although definitions do vary.)
The Gauss-Markov theorem tells us that, at least as far as unbiased linear estimators
go, the OLS estimates will have low variance. Put differently, if we fix the regression
problem and vary the estimators, the OLS estimators will have the most precision.
However, we want to think about precision a different way: If we hold the estimation technique fixed (use only OLS) and consider different regression problems,
which problems will have high precision estimates, and which will have low precision estimates?
To answer this question, lets focus on the variance of a fixed coefficient k . We can
write the regression model y = X + u as
y = X1 1 + colk (X) k + u
(7.13)
where colk (X) is the vector of observations of the k-th regressor, X1 contains as its
columns the observations of the other regressors, and 1 is the OLS estimates of the
corresponding coefficients. From the FWL theorem, we can then express k as
k = (colk (X)0 M1 colk (X))1 colk (X)0 M1 y
(7.14)
where M1 is the annihilator corresponding to X1 . That is, M1 := I P1 where P1 is

the matrix X1 (X10 X1 )1 X10 projecting onto the column space of X1 . Applying M1 to
both sides of (7.13), we obtain
M1 y = M1 colk (X) k + M1 u
Substituting this into (7.14), we obtain a second expression for k in terms of the
shock vector u:
k = k + (colk (X)0 M1 colk (X))1 colk (X)0 M1 u
(7.15)
Some calculations then show (exercise 7.7.11) that

var[ k | X] = 2 (colk (X)0 M1 colk (X))1 = 2 kM1 colk (X)k2
(7.16)
Thus, the variance of k depends on two components, the variance 2 of the shock
u, and the norm of the vector M1 colk (X).
214
The variance in 2 is in some sense unavoidable: Some data is noisier that other data.
The larger the variance in the unobservable shock, the harder it will be to estimate
k with good precession. The term kM1 colk (X)k2 is more interesting. The vector
M1 colk (X) is the residuals from regressing colk (X) on X1 , and kM1 colk (X)k is the
norm of this vector. If this norm is small, then the variance of k will be large.
When will this norm be small (and the variance of k correspondingly large)? This
will be the case when colk (X) is almost a linear combination of the other regressors. To see this, suppose that colk (X) is indeed almost a linear combination of the
other regressors. This implies that
P1 colk (X) colk (X)
because P1 projects into the column space of the other regressors X1 , so we are
saying that colk (X) is almost in that span. Now if P1 colk (X) colk (X), then
kP1 colk (X) colk (X)k 0, and hence
kM1 colk (X)k = k colk (X) P1 colk (X)k 0

In other words, the norm of M1 colk (X) is small, and hence the variance of k is
large.
This situation is sometimes referred to as multicollinearity. As we have just seen,
multicollinearity is associated with poor precision in estimates of the coefficients.
7.4
Normal Errors
In this section, were going to strengthen and augment our previous assumptions by
specifying the parametric class of the error vector u. Once this class is specified, we
can determine the distribution of the OLS estimate up to the unknown parameters
2 , 1 , . . . , K . (In other words, if values for these parameters are specified, then the
distribution of is fully specified.) This will allow us to test hypotheses about the
coefficients.
Because of its many attractive properties, the normal distribution is our go-to distribution, at least for the case where we have no information that suggests another
distribution, or contradicts the normality assumption. Following this grand tradition, we will assume that u is a normally distributed element of R N .
A normal distribution in R N is fully specified by its mean and variance-covariance
matrix. In this case, given that were strengthening our previous assumptions, we
215
have no choice here. From assumption 7.1.1, the mean is E [u] = 0, and, from
assumption 7.1.2, the variance-covariance matrix is E [uu0 ] = 2 I. We must then
have u N (0, 2 I).
Furthermore, assumption 7.1.1 implies that shocks and regressors are uncorrelated
(see fact 7.1.2 on page 204). To make life a bit easier for ourselves, well go a step
further and assume they are independent.
Assumption 7.4.1. X and u are independent, and u N (0, 2 I).
Notice that assumption 7.4.1 implies both assumption 7.1.1 and assumption 7.1.2.
7.4.1
Preliminary Results
Assumption 7.4.1 also implies that the conditional distribution of given X is normal, since = + (X0 X)1 X0 u, and linear combinations of normals are normal. The
next theorem records this result.
Theorem 7.4.1. Under assumption 7.4.1, the distribution of given X is N ( , 2 (X0 X)1 ).
It follows from theorem 7.4.1 that the distribution of individual coefficient k given
X is also normal. This can be established directly from (7.15), and the variance is
given in (7.16). However, we will use theorem 7.4.1 instead, since it gives an expression for the variance which is easier to compute. To do this, let ek be the k-th
canonical basis vector, and observe that
e0k N (e0k , 2 e0k (X0 X)1 ek )
In other words,
k N ( k , 2 e0k (X0 X)1 ek )
(7.17)
Note that e0k (X0 X)1 ek is the (k, k )-th element of the matrix (X0 X)1 .2 It then follows
that
k
zk := q k
N (0, 1)
(7.18)
e0k (X0 X)1 ek
Our second preliminary result concerns the distribution of 2 , or more precisely, of
Q := ( N K )
2 See
exercise 2.7.15 on page 82.
2
2
(7.19)
216
Theorem 7.4.2. Under assumption 7.4.1, the distribution of Q given X is 2 ( N K ).

Proof. To see that Q 2 ( N K ) given X, observe that
Q=
(Mu)0 (Mu)
u0 Mu
=
= ( 1 u ) 0 M ( 1 u )
2
2
Since 1 u N (0, I), the expression on the far right is 2 ( N K ), as follows from
fact 2.4.9 and our previous result that trace(M) = rank(M) = N K (recall that for
idempotent matrices, trace and rank are equalsee fact 2.3.9).
7.4.2
The t-test
Lets consider the problem of testing a hypothesis about an individual coefficient

k . Specifically, we consider the null hypothesis
H0 : k = 0k
against
H1 : k 6= 0k
where 0k is any number. If we knew 2 , we could test H0 via (7.18). Since we dont,
the standard methodology is to replace 2 with its estimator 2 , and determine the
distribution of the resulting test statistic. Our next result implements this idea. In
doing so, we will make use of the following notation:
q
se( k ) := 2 e0k (X0 X)1 ek
The term se( k ) is called the standard error of k . It can be regarded as the sample
estimate of the standard deviation of k . Replacing this standard deviation with its
sample estimate se( k ) and k with 0k in (7.18), we obtain the t-statistic
tk :=
k 0k
se( k )
(7.20)
The distribution of this statistic under the null is described in the next theorem.
Theorem 7.4.3. Let assumption 7.4.1 hold. If the null hypothesis H0 is true, then, conditional on X, the distribution of the t-statistic in (7.20) is Students t with N K degrees of
freedom.
217
Proof. Suppose that the null hypothesis is true. We then have

r
r
k k
k k
1
1
=
z
tk =
=q
k
2
2
2 /2
se( k )
2 e0 (X0 X)1 ek /
k
where zk is defined in (7.18). Multiplying and dividing by the square root of N K,

we can write this as
s
s
NK
NK
tk = zk
= zk
2
2
Q
( N K ) /
where Q is defined in (7.19). In view of fact 1.3.6 on page 27, we know that tk is
Students t with N K degrees of freedom if zk is standard normal, Q is 2 ( N K )
and zk and Q are independent. The first two results were established in 7.4.1, so it
remains only to show that zk and Q are independent.
To see this, note that if a and b are independent random vectors and f and g are
two functions, then f (a) and g(b) are likewise independent. Since we can write
it suffices to show that and u are
zk as a function of and Q as a function of u,
independent. Since both are normally distributed given X, fact 2.4.7 on page 75
implies that they will be independent whenever their covariance is zero. This is
exercise 7.7.5, which completes the proof of theorem 7.4.3.
Recall that a test is a test statistic T and a critical value c, with the rule: Reject H0 if
T > c. For T we take T = |tk |. Let a desired size be given. In view of (5.15) on
page 170, we choose c = c to solve = P { T > c}, or
1 = P {|tk | c}
From (1.14) on page 21, we know that the solution is c = F 1 (1 /2), where F
is the Students t cdf with N K degrees of freedom. In view of example 5.3.3, the
corresponding p-value is 2F (|tk |).
Lets look at an example. The most common implementation of the t-test is the test
that a given coefficent is equal to zero. For the k-th coefficient k , this leads to the
statistic
k
tk :=
se( k )
This statistic is sometimes called the Z-score. To illustrate further, an application
with simulated data is given in listing 9. If you run the program, you will find that
218
the Z-scores calculated by the zscore function agree with the t value column of
the summary table produced by Rs summary function (the last line of listing 9). It is
left as an exercise to check that the p-values in the same table agree with the formula
2F (|tk |) given in the last paragraph.
Listing 9 Calculating Z-scores
set . seed (1234)

N <- 50; K <- 3
beta <- rep (1 , K )
X <- cbind ( runif ( N ) , runif ( N ) , runif ( N ) )
u <- rnorm ( N )
y <- X % * % beta + u
betahat <- solve ( t ( X ) % * % X ) % * % t ( X ) % * % y
residuals <- y - X % * % betahat
sigmahat <- sqrt ( sum ( residuals ^2) / ( N - K ) )
# Compute t - stat (Z - score ) for k - th regressor
zscore <- function ( k ) {
se <- sigmahat * sqrt ( solve ( t ( X ) % * % X ) [k , k ])
return ( betahat [ k ] / se )
}
# Print t - stats
for ( k in 1:3) {
cat ( "t - stat , k = " , k , " : " , zscore ( k ) , " \ n " )
}
# For comparison :
print ( summary ( lm ( y ~ X - 1) ) )
7.4.3
The F-test
The t-test is used to test hypotheses about individual regressors. For hypotheses
concerning multiple regressors, the most common test is the F-test. The F-test can
test quite general hypotheses, but for simplicity we will focus on null hypotheses
that restrict a subset of the coefficients to be zero.
219
In what follows, we let X1 , X2 , 1 , 2 , P1 and M1 be as defined in 7.3.1. As our null

hypothesis we take
H0 : 2 = 0 against H1 : 2 6= 0
Since
y = X + u = X1 1 + X2 2 + u
(7.21)
it follows that under the null hypothesis we have

y = X1 1 + u
(7.22)
Letting
USSR
:= kMyk2
and
RSSR
: = k M1 y k2
be the sums of squared residuals for the unrestricted regression (7.21) and restricted
regression (7.22) respectively, the standard test statistic for our null hypothesis is
F :=
( RSSR USSR )/K2

USSR / ( N K )
(7.23)
Large residuals in the restricted regression (7.22) relative to those in (7.21) result in
large values for F, which translates to evidence against the null hypothesis.
Theorem 7.4.4. Let assumption 7.4.1 hold. If the null hypothesis is true, then, conditional
on X, the statistic F defined in (7.23) has the F distribution, with parameters (K2 , N K ).
Proof. Let Q1 := ( RSSR USSR )/2 and let Q2 :=
F=
USSR /2 ,
so that
Q1 /K2
Q2 / ( N K )
In view of fact 1.3.7 on page 28, it now suffices to show that, under the null hypothesis,
(a) Q1 is chi-squared with K2 degrees of freedom.
(b) Q2 is chi-squared with N K degrees of freedom.
(c) Q1 and Q2 are independent.
Part (b) was established in theorem 7.4.2. Regarding part (a), observe that, under
the null hypothesis,
USSR
= kMyk2 = kM(X1 1 + u)k2 = kMuk2 = u0 Mu, and
RSSR
220
= kM1 yk2 = kM1 (X1 1 + u)k2 = kM1 uk2 = u0 M1 u
It follows that
RSSR USSR
= u0 M1 u u0 Mu = u0 (M1 M)u
Using the definitions of M and M1 , we then obtain

Q1 =
RSSR USSR
u 0 ( I P1 I + P ) u
= (1 u)0 (P P1 )(1 u)
2
It is an exercise to show that (P P1 ) is symmetric and idempotent.3 Applying the

techniques in the proof of theorem 7.2.2, we see that
rank(P P1 ) = trace(P P1 ) = trace(P) trace(P1 ) = K K1 = K2
Via fact 2.4.9, we conclude that Q1 2 (K2 ), as was to be shown.
To complete the proof, it remains to show that, under the null hypothesis and taking
X as given, Q1 and Q2 are independent. To see this, observe that Q1 is a function of
(P P1 )u, while Q2 is a function of Mu. Since both (P P1 )u and Mu are normal
given X, it suffices to show that their covariance is zero. This is the case, because
cov[(P P1 )u, Mu | X] = E [(P P1 )u(Mu)0 | X] = E [(P P1 )uu0 M | X]
Since P, P1 and M are just functions of X, this becomes
(P P1 )E [uu0 | X]M = 2 (P P1 )M = 2 (P P1 )(I P)

Using idempotence and fact 3.1.2, the matrix product on the right is
(P P1 )(I P) = P P2 P1 + P1 P = P P P1 + P1 = 0
This completes the proof of independence, and hence of theorem 7.4.4.
The most common implementation of the F test is the test that all coefficients of
non-constant regressors are zero. In this case (7.21) becomes
y = 1 1 + X2 2 + u
(7.24)
where 2 is the vector of coefficients corresponding to the non-constant regressors.

Since X1 = 1, we then have M1 = Mc , where the latter is defined in (6.16) on
Listing 10 Calculating the F statistic
set . seed (1234)

N <- 50; K <- 3
beta <- rep (1 , K )
x2 <- runif ( N ) ; x3 <- runif ( N )
X <- cbind ( rep (1 , N ) , x2 , x3 )
u <- rnorm ( N )
y <- X % * % beta + u
betahat <- solve ( t ( X ) % * % X ) % * % t ( X ) % * % y
residuals <- y - X % * % betahat
ussr <- sum ( residuals ^2)
rssr <- sum (( y - mean ( y ) ) ^2)
Fa <- ( rssr - ussr ) / 2
Fb <- ussr / ( N - K )
cat ( " F = " , Fa / Fb , " \ n " )
# For comparison :
print ( summary ( lm ( y ~ x2 + x3 ) ) )
221
222
An application with simpage 193, and hence RSSR is the squared norm of y y1.
ulated data is given in listing 10. If you run the program, you will find that the F
statistic calculated by the theoretical formula agrees with the F statistic produced by
Rs summary function.
It is an exercise to show that in the case of (7.24), the F statistic in (7.23) can be
rewritten as
R2c N K
F=
(7.25)
1 R2c K2
where R2c is the centered R squared defined in 6.3.2. Can you provide some intuition as to why large F is evidence against the null?
7.5
When the Assumptions Fail
The standard OLS assumptions are very strict, and the results we have obtained are
sensitive to their failure. For example, if our basic assumption y = X + u is not
true, then pretty much all the results discussed in this chapter are invalid. In what
follows, lets be polite and consider situations where the standard OLS assumptions
are only slightly wrong.
7.5.1
Endogeneity Bias
Even if the model is correctly specified, the OLS estimates can be biased when assumption 7.1.1 (i.e., E [u | X] = 0) fails. Assumption 7.1.1 is sometimes called an
exogeneity assumption. When it fails, the bias is called endogeneity bias. There are
many sources of endogeneity bias. We will look at two examples.
As a first example, consider again the Cobb-Douglas example on page 203, which
yields the regression model
ln yn = + ln k n + ln `n + un
Here y is output, k is capital, ` is labor, and subscript n indicates observation on the
n-th firm. The term un is a firm specific productivity shock. A likely problem here is
that the productivity shocks are positively correlated, and, moreover, the firm will
3 Hint:
See fact 3.1.2 on page 90.
223
choose higher levels of both capital and labor when it anticipates high productivity
in the current period. This will lead to endogeneity bias.
To illustrate this, suppose that un,1 is the productivity shock received by firm n last
period, and this value is observable to the firm. Suppose that productivity follows a
random walk, with un = un,1 + n , where n is zero mean white noise. As a result,
the firm forecasts period n productivity as E [un | un,1 ] = un,1 . Finally, suppose
that the firm increases labor input when productivity is anticipated to be high, with
the specific relationship `n = a + bE [un | un,1 ] for b > 0. When all shocks are zero
mean we then have
E [`n un ] = E [( a + bun,1 )(un,1 + n )] = E [bu2n,1 ]

This term will be strictly positive whenever un,1 has positive variance. Thus, the
conditions of fact 7.1.2 (page 204) fail, and therefore assumption 7.1.1 does not hold
(because assumption 7.1.1 implies fact 7.1.2).
This source of endogeneity bias in estimating production functions has been discussed many times in the literature. The best solution is better modeling. For an
illustration of careful modeling (and discussion of other potential problems with
estimating production functions) see the paper of Olley and Pakes (1996).
As a second example of endogeneity bias, suppose next that we have in hand data
that is generated according to the simple AR(1) model
x0 = 0
and
xn = xn1 + un
for n = 1, . . . , N
(7.26)
IID
Here we assume that {un }nN=1 N (0, 2 ). The unknown parameters are and 2 .
Letting
y : = ( x1 , . . . , x N ),
x : = ( x 0 , . . . , x N 1 )
and
u : = ( u1 , . . . , u N )
we can write the N equations in (7.26) as

y = x + u
Suppose that we now estimate by regressing y on x, obtaining the OLS estimate
x0 y
:= (x0 x)1 x0 y = 0
xx
It turns out that is a biased estimate of . The source of the bias is failure of
the exogeneity assumption. For example, fact 7.1.2 tells us that if assumption 7.1.1
224
holds, then we must have E [um xn+1 ] = 0 for any m and n. In the current set-up this
fails. For example, suppose that 6= 0 and n m. Observe that (exercise 7.7.15) we
can write xn as
n 1
xn =
j un j
(7.27)
j =0
and, therefore,
n 1
E [ xn um ] = j E [un j um ] = nm 2 whenever n m
(7.28)
j =0
It follows that when 6= 0, assumption 7.1.1 must fail.

To help illustrate the bias in our estimate of , lets generate the data 10,000 times,
compute on each occasion, and then take the sample mean. The code to do this is
given in Listing 11, with N = 20 and = 0.9. The resulting sample mean was 0.82.
Since the number of replications is very large (i.e., 10,000), this will be very close
to E [ ]. In fact, the asymptotic 95% confidence interval for our estimate of E [ ] is
(0.818, 0.824).
This bias towards small values is reinforced by the histogram of the observations
given in figure 7.1, which is skewed to the left.
7.5.2
Misspecification and Bias
The linearity assumption y = X + u can fail in many ways, and when it does the
estimator will typically be biased. Lets look at one possible failure, where the
model is still linear, but some variables are omitted. In particular, lets suppose that
the data generating process is in fact
y = X + Z + u
(7.29)
Well also assume that 6= 0, and that E [u | X, Z] = 0.

Suppose that we are aware of the relationship between y and X, but unaware of
the relationship between y and Z. This will lead us to mistakenly ignore Z, and
simply regress y on X. Our OLS estimator will then be given by the usual expression
= (X0 X)1 X0 y. Substituting in (7.29), we get
= (X0 X)1 X0 (X + Z + u) = + (X0 X)1 X0 Z + (X0 X)1 X0 u
Listing 11 Generates observations of

N <- 20
x <- numeric ( N )
x _ zero <- 0
beta <- 0.9
num . reps <- 10000
betahat . obs <- numeric ( num . reps )
for ( j in 1: num . reps ) {
u <- rnorm ( N )
x [1] <- beta * x _ zero + u [1]
for ( t in 1:( N -1) ) {
x [ t +1] <- beta * x [ t ] + u [ t +1]
}
y <- x # x _ 1 ,... , x _ N
x _ vec <- c ( x _ zero , x [ - N ]) # x _ 0 , ... , x _ {N -1}
betahat . obs [ j ] <- sum ( x _ vec * y ) / sum ( x _ vec ^2)
}
print ( mean ( betahat . obs ) )
225
226
1.5
0.0
0.5
1.0
Density
2.0
2.5
3.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Figure 7.1: Observations of
We now have
E [ ] = E [E [ | X, Z]] = + E [(X0 X)1 X0 Z]
If the columns of X and Z are orthogonal, then X0 Z = 0, the last term on the righthand side drops out, and is unbiased. If this is not the case (typically it wont be),
then is a biased estimator of .
7.5.3
Heteroskedasticity
[to be written]
7.6
Further Reading
To be written.
7.7
227
Exercises
Throughout these exercises, we maintain assumptions 7.1.1 and 7.1.2. In particular,

we assume that y = X + u, E [u | X] = 0, and E [uu0 | X] = 2 I, where is a
positive constant.
A general hint for the exercises is to remember that, as stated in fact 3.3.7, when
computing expectations conditional on X, you can treat X or any function of X as
constant. For example, matrices and vectors that depend only on X and non-random
vectors/matrices, such as X or P = X(X0 X)1 X0 , can be regarded as constant.
Ex. 7.7.1. Show that My = Mu and SSR = u0 Mu.
Ex. 7.7.2. Prove the claims in fact 7.1.2.
Ex. 7.7.3. Confirm part 1 of fact 7.1.3: Show that var[u] = E [uu0 ] = 2 I.
Ex. 7.7.4. Show that cov[Py, My | X] = 0.
u | X] = 0.
Ex. 7.7.5. Show that, under assumptions 7.1.17.1.2, we have cov[ ,
Ex. 7.7.6. Show that E [Py | X] = X and var[Py | X] = 2 P.
Ex. 7.7.7. Show that E [My | X] = 0 and var[My | X] = 2 M.
Ex. 7.7.8. Show that the two expressions for 2 in (7.8) and (7.10) are equal.4
Ex. 7.7.9. In the proof of theorem 7.2.2 we used a clever trick with the trace to show
that trace( P) = K. An alternative way to obtain the same result is to observe that,
since P is idempotent, the trace is equal to the rank (fact 2.3.9 on page 71). Prove
directly that the rank of P equals K.
Ex. 7.7.10. (Hard) At the end of the proof of theorem 7.3.1, it was claimed that the
matrix X20 M1 X2 is invertible. Verify this claim.
Ex. 7.7.11. Confirm (7.16) on page 213.
Ex. 7.7.12. Using (7.16), show that for the simple OLS model y = 1 1 + 2 x + u, the
variance of 2 given x is 2 / nN=1 ( xn x )2 .
Ex. 7.7.13. Show that in the case of (7.24), the F statistic in (7.23) can be rewritten as
(7.25).5
4 Hint:
5 Hint:
Use the symmetry and idempotence of the matrix M1 .

Note that My = MMc y by fact 3.1.5 on page 92.
228
Ex. 7.7.14. Suppose that assumption 7.4.1 holds, so that X and u are independent,
and u N (0, 2 I). Show that, conditional on X,
1. Py and My are normally distributed, and
2. Py and My are independent.
Ex. 7.7.15. Verify the expression for yn in (7.27).
7.7.1
Solution to Exercise 7.7.1. We have

My = M(X + u) = Mu
SSR
:= kMyk2 = kMuk2 = (Mu)0 (Mu) = u0 M0 Mu = u0 MMu
by symmetry of M. The result now follows from idempotence of M.

Solution to Exercise 7.7.2. The respective proofs for claims 14 are as follows:
1. E [u] = E [E [u | X]] = E [0] = 0
2. E [um | xnk ] = E [E [um | X] | xnk ] = E [0 | xnk ] = 0
3. E [um xnk ] = E [E [um xnk | xnk ]] = E [ xnk E [um | xnk ]] = 0
4. cov[um , xnk ] = E [um xnk ] E [um ]E [ xnk ] = 0
Solution to Exercise 7.7.3. By definition,

var[u] = E [uu0 ] E [u]E [u0 ]
Since E [u] = E [E [u | X]] = 0, this reduces to var[u] = E [uu0 ]. Moreover,
E [uu0 ] = E [E [uu0 | X]] = E [2 I] = 2 I
229
Solution to Exercise 7.7.4. Note that

My = M(X + u) = Mu
Py = P(X + u) = X + Pu
E [My | X] = E [Mu | X] = ME [u | X] = 0
Using these facts, we obtain
cov[Py, My | X] = cov[X + Pu, Mu | X] = E [(X + Pu)(Mu)0 | X]
From linearity of expectations and symmetry of M, this becomes
cov[Py, My | X] = E [Xu0 M | X] + E [Puu0 M | X]
Regarding the first term on the right-hand side, we have
E [Xu0 M | X] = XE [u0 | X]M = 0

Regarding the second term on the right-hand side, we have
E [Puu0 M | X] = PE [uu0 | X]M = P2 IM = 2 PM = 0
cov[Py, My | X] = 0
Solution to Exercise 7.7.5. Since M is a function of X, we have E [Mu | X] = ME [u | X] =

0. As a result,
u | X] = E [ u 0 | X] E [ | X]E [u | X]0
cov[ ,
= E [ (Mu)0 | X]
= E [ (Mu)0 + (X0 X)1 X0 u (Mu)0 | X]

= E [(X0 X)1 X0 u (Mu)0 | X]
= E [(X0 X)1 X0 uu0 M | X]
= 2 ( X 0 X ) 1 X 0 M
u | X] = 0, and the proof is done.
Since X0 M = (MX)0 = 00 we have cov[ ,
230
Solution to Exercise 7.7.6. Regarding the claim that E [Py | X] = X, our previous
results and linearity of expectations gives
E [Py | X] = E [X + Pu | X] = X + PE [u | X] = X
Regarding the claim that var[Py | X] = 2 P, our rules for manipulating variances
yield
var[Py | X] = var[X + Pu | X] = var[Pu | X] = P var[u | X]P0 = P2 IP0
Using symmetry and idempotence of P, we obtain var[Py | X] = 2 P.
Solution to Exercise 7.7.7. Similar to the solution of exercise 7.7.6.
Solution to Exercise 7.7.9. Since rank(X) = dim(rng(X)) = K, to establish that
rank(P) = dim(rng(P)) = K, it suffices to show that rng(X) = rng(P). To see
this, suppose first that z rng(X). Since P is the projection onto rng(X), we then
have z = Pz (see theorem 3.1.3 on page 88), and hence z rng(P). Conversely, if
z rng(P), then, z = Pa for some a R N . By definition, P maps every point into
rng(X), so we conclude that z rng(X).
Solution to Exercise 7.7.10. To see that the matrix X20 M1 X2 is invertible, note that,
in view of idempotence and symmetry of M1 ,
X20 M1 X2 = X20 M1 M1 X2 = X20 M10 M1 X2 = (M1 X2 )0 M1 X2
In view of fact 2.3.11, to show that this matrix is invertible, it suffices to show that
the matrix is positive definite. So take any a 6= 0. We need to show that
a 0 ( M1 X2 ) 0 M1 X2 a = ( M1 X2 a ) 0 M1 X2 a = k M1 X2 a k2 > 0
Since the only vector with zero norm is the zero vector, it now suffices to show that
M1 X2 a is non-zero. From fact 3.1.6 on page 92, we see that M1 X2 a = 0 only when
X2 a is in the column span of X1 . Thus, the proof will be complete if we can show
that X2 a is not in the column span of X1 .
Indeed, X2 a is not in the column span of X1 . For if it were, then we could write
X1 b = X2 a for some b RK1 . Rearranging, we get Xc = 0 for some non-zero c
(recall a 6= 0). This contradicts linear independence of the columns of X.
231
Solution to Exercise 7.7.11. Repeating (7.15) we have

k = k + (colk (X)0 M1 colk (X))1 colk (X)0 M1 u
(7.30)
Since k is constant, taking the variance of (7.30) conditional on X we obtain

var[ k | X] = var[Au | X]
where A := (colk (X)0 M1 colk (X))1 colk (X)0 M1
Since A is a function of X, we can treat it as constant given X, and we obtain

var[ k | X] = A var[u | X]A0 = A2 IA0 = 2 AA0
To complete the proof, we just observe that
AA0 = (colk (X)0 M1 colk (X))1 colk (X)0 M1 M10 colk (X)(colk (X)0 M1 colk (X))1
= (colk (X)0 M1 colk (X))1

where the last equality is due to symmetry and idempotence of M1 (recall that M1
is an annihilator). We conclude that (7.16) is valid.
Solution to Exercise 7.7.12. For the model y = 1 1 + 2 x + u, the matrix M1 is the
centering annihilator Mc associated with 1, and colk (X) is just x. Hence, from (7.16),
we have
var[ 2 | X] = 2 (x0 Mc x)1
Using symmetry and idempotence of Mc , this becomes
var[ 2 | X] = 2 [(Mc x)0 Mc x]1 = 2 /
(xn x )2
n =1
Finally, since the only random variables in X are the random variables in x, we can
write
var[ 2 | x] = 2 /
(xn x )2
n =1
as was to be shown.
Solution to Exercise 7.7.13. We need to show that in the special case (7.24) we have
( RSSR USSR )/K2

R2c N K
=
USSR / ( N K )
1 R2c K2
232
or, equivalently,
RSSR USSR
USSR
R2c
1 R2c
(7.31)
Consider first the left-hand side of (7.31). In the case of (7.24), this becomes
RSSR USSR
USSR
kMc yk2 kMyk2

=
kMyk2
On the other hand, regarding the right-hand side of (7.31), the definition of R2c and
some minor manipulation gives
R2c
kPMc yk2
=
1 R2c
kMc yk2 kPMc yk2
Hence, to establish (7.31), we need to show that
kMc yk2 kMyk2

kPMc yk2
=
kMyk2
kMc yk2 kPMc yk2
This is can be established using (6.18).
Solution to Exercise 7.7.14. We have shown previously that My = Mu and Py =
X + Pu. Since we are conditioning on X we can treat it as constant. When X is
constant, P, M and X are all constant. Since linear (or affine) transformations of
normal random vectors are normal, Mu and X + Pu are both normally distributed.
It remains to show that Py and My are independent given X. Since they are normally
distributed given X, we need only show that they are uncorrelated given X. This was
already proved in exercise 7.7.4.
Chapter 8
Time Series Models
The purpose of this chapter is to move away from the IID restriction and towards
the study of data that has some dependence structure over time. Such data is very
common in economics and finance. Our first step is to introduce and study common
time series models. Next, we use the techniques developed in this process to move
from finite sample OLS to large sample OLS. Large sample OLS theory is in many
ways more attractive and convincing than its finite sample counterpart.
8.1
Some Common Models
In this section we introduce some of the most common time series models, working
from specific to more general.
8.1.1
Linear Models
In time series as in other fields, the easiest models to analyze are the linear models.
Of the linear time series models, the friendliest is the scalar Gaussian AR(1) model,
which takes the form
xt+1 = + $xt + wt+1
IID
with {wt } N (0, 2 ) and x0 given
(8.1)
Here , $ and are parameters. The random variable xt is called the state variable.
Note that (8.1) fully defines the time t state xt as a random variable for each t. (Well
spell out how this works in some detail below.) Since xt is a well-defined random
233
CHAPTER 8. TIME SERIES MODELS
234
variable, it has a distribution t defined by t (s) := P{ xt s}. This distribution

is often called the marginal distribution of xt . In what follows, we will be interested in the dynamics both of the state process { xt } and also of the corresponding
distribution sequence {t }.
The scalar Gaussian AR(1) model (8.1) generalizes in serveral different directions.
For example, we can remove the assumption that the shocks are normally distributed,
in which case it is simply called the scalar AR(1) model. The scalar AR(1) model further generalizes to RK , yielding the vector AR(1) model, or VAR(1). The dynamics
of the process are given by
xt+1 = a + xt + wt+1
IID
with {wt } and x0 given
(8.2)
where a is a K 1 column vector, is a K K matrix and is some distribution

on RK . If = N (0, ) for some a symmetric, positive definite K K matrix , then
(8.2) is called the Gaussian VAR(1). The vector xt is called the state vector, and the
space RK in which it takes values is called the state space.
Another common generalization of the scalar AR(1) model is the scalar AR(p) model,
where the next state xt+1 is a (linear) function not just of the current state xt , but of
the last p previous states. For example, the AR(2) process has dynamics
xt+1 = + $xt + xt1 + wt+1
Although xt+1 is a function of two lagged states, xt and xt1 , we can reformulate it as
a first order model. To begin, we define an additional state variable yt via yt = xt1 .
The dynamics can then be expressed as
xt+1 = + $xt + yt + wt+1
y t +1 = x t
We can write this in matrix form as

1
$
xt
1
x t +1
=
+
+
w t +1
y t +1
0
1 0
yt
0
Notice that this is a special case of the VAR(1) model in (8.2). The message is that
higher order processes can be reduced to first order processes by increasing the
number of state variables. For this reason, in what follows we concentrate primarily
on first order models.
8.1.2
235
Nonlinear Models
The previous examples are all linear models. Linear models are simple, but this
is not always a good thing. Their simplicity means they cannot always capture
the kinds of dynamics we observe in data. Moreover, many theoretical modeling
exercises produce models that are not linear. In this section we introduce several
popular nonlinear models.
One well-known nonlinear model is the p-th order autoregressive conditional heteroskedasticity model (ARCH(p) model), the ARCH(1) version of which has dynamics
IID
{wt } N (0, 1)
(8.3)
xt+1 = (0 + 1 xt2 )1/2 wt+1 ,
The model arose as an effort to model evolution of returns on a given asset. In the
model, returns xt are first written as the product of an IID shock wt and a timevarying volatility component t . That is, xt = t wt . The evolution of t is specified
by t2+1 = 0 + 1 xt2 . Combining these equations gives the dynamics for xt displayed
in (8.3).
In recent years, econometricians studying asset prices have moved away from the
ARCH model in favor of a generalized ARCH (GARCH) model, the simplest of
which is the GARCH(1,1) process
xt = t wt
t2+1 = 0 + 1 xt2 + 2 t2
Another popular nonlinear model is the smooth transition threshold autoregression
(STAR) model
x t +1 = g ( x t ) + w t +1
(8.4)
where g is of the form
g(s) := (0 + $0 s)(1 (s)) + (1 + $1 s) (s)
Here : R [0, 1] is an increasing function satisfying lims (s) = 0 and
lims (s) = 1. When s is small we have (s) 0, and g(s) 0 + $0 s. When s is
large we have (s) 1, and g(s) 1 + $1 s. Thus, the dynamics transition between
two different linear models, with the smoothness of the transition depending on the
shape of .
8.1.3
236
Markov Models
All of the previous examples in this section are special cases of a general class of
process called Markov processes. The general formulation for a (first order, time
homogeneous) Markov process looks as follows:
x t +1 = G ( x t , w t +1 )
with
x0 0
(8.5)
Here we assume that {wt }t1 is an IID sequence of R M -valued shocks with common
density , and that G is a given function mapping the current state xt RK and
shock wt+1 R M into the new state xt+1 RK . The initial condition x0 and the
shocks {wt }t1 are also assumed to be independent of each other. The density 0 is
the distribution of x0 . As before, well let
t ( s ) : = P{ x t s }
represent the marginal distribution of the state xt . Where necessary, t will represent
the corresponding density.
By repeated use of (8.5), we obtain the sequence of expressions
x1 = G ( x0 , w1 )
x2 = G ( G ( x0 , w1 ), w2 )
x3 = G ( G ( G ( x0 , w1 ), w2 ), w3 )
..
.
Continuing in this fashion, we see that, for any t, the state vector xt can be written
as a function of x0 and the shocks w1 , . . . , wt . In other words, for each t, there exists
a function Ht such that
xt = Ht (x0 , w1 , w2 , . . . , wt )
(8.6)
Although there may be no neat expression for the function Ht , equation (8.6) clarifies
the fact that (8.5) pins down each xt as a well-defined random variable, depending
on the initial condition and the shocks up until date t.
Example 8.1.1. A simple example is the scalar linear AR(1) process
xt+1 = + $xt + wt+1
In this case, there is a neat expression for Ht in (8.6). Indeed, for all t 0 we have
t 1
xt =
k =0
The proof is an exercise.
$k +
t 1
$ k w t k + $ t x0
k =0
(8.7)
237
Fact 8.1.1. For the Markov process (8.5), the current state xt and future shocks wt+ j
are independent for every t and every j > 0.
Fact 8.1.1 follows from fact 2.4.1 on page 73. In particular, xt is a function of the
random variables x0 , w1 , w2 , . . . , wt , and these are all, by the IID assumption, independent of wt+ j .
Its worth mentioning that, although its often not made explicit, behind the all random vectors {xt } lies a single sample space and a probability P. The idea is that
an element of is selected by nature at the start of the experiment (with the
probability that lies in E equal to P( E)). This determines the initial condition
x0 and the shocks wt as
x0 ( ), w1 ( ), w2 ( ), w3 ( ), . . .
From these, each state vector xt is determined via
xt ( ) = Ht (x0 ( ), w1 ( ), w2 ( ), . . . , wt ( ))
where Ht is the function in (8.6).
An important object in Markov process theory is the transition density, or stochastic
kernel, which is the conditional density of the next period state given the current
state, and will be denoted by p.1 In particular,
p( | s) := the conditional density of xt+1 given xt equals s
(8.8)
We can usually derive an expression for the transition density in terms of the model.
For example, suppose that the shock is additive, so that
x t +1 = G ( x t , w t +1 ) = g ( x t ) + w t +1
with
IID
{wt }
(8.9)
for some function g and density . In this case, the transition density has the form
p(s0 | s) = (s0 g(s))
(8.10)
How does one arrive at expression (8.10)? Lets go through the argument for the
scalar case, where the model is xt+1 = g( xt ) + wt+1 with wt+1 . Let be the
1 Im
being a little careless here, because this density may not in fact exist. (For example, take
the process xt+1 = G ( xt , wt+1 ) where G (s, w) = 0 for all s and w. In this case the random variable
xt+1 is equal to zero with probability one. Such a random variable does not have a density. See
the discussion in 1.2.2.) However, the density will exist in most applications, and in all cases we
consider.
238
cdf corresponding to , so that the derivative of is . Now, we claim that the

density of xt+1 when xt equals constant s is given by p(s0 | s) := (s0 g(s)). This
is equivalent to the claim that the density of x 0 = g(s) + w is equal to (s0 g(s)).
Letting F be the cdf of x 0 = g(s) + w, we have
F (s0 ) := P{ x 0 s0 } = P{ g(s) + w s0 } = P{w s0 g(s)} = (s0 g(s))
The density we seek is the derivative of this expression with respect to s0 , which is
(s0 g(s)). In other words, p(s0 | s) = (s0 g(s)) as claimed.
8.1.4
Martingales
Loosely speaking, a martingale is a stochastic process evolving over time such that
the best guess of the next value given the current value is the current value. Martingales arise naturally in many kinds of economic and financial models. Moreover,
since the mid 20th Century, martingales have contributed to much progress in the
foundations of probability theory.
To give a more formal definition, we need first to introduce the notion of a filtration,
which is an increasing sequence of information sets. Recall from 3.3.2 that an information set is just a set of random variables or vectors. Let {Ft } be a sequence of
information sets. That is, Ft is an information set for each t. The sequence {Ft } is
called a filtration if, in addition, it satisfies Ft Ft+1 for all t. Intuitively, Ft contains the information available at time t, and the requirement that the sequence be
increasing reflects the idea that more and more information is revealed over time.2
Example 8.1.2. Let {xt } be a sequence of random vectors, and let
F0 := , F1 := {x1 }, F2 := {x1 , x2 }, F3 := {x1 , x2 , x3 },

Then {Ft } is a filtration. In fact this is the canonical example of a filtration.
Now let {mt } be a scalar stochastic process (i.e., a sequence of random variables)
and let {Ft } be a filtration. We say that {mt } is adapted to the filtration {Ft } if
mt is Ft -measurable for every t. (For the definition of measurability see 3.3.2.) In
many applications, Ft represents the variables we know as of time t, and if {mt } is
adapted to Ft , then we can compute mt at time t as well.
2 If
you learn measure theory, you will learn that {Ft } is actually best thought of as an increasing
sequence of -algebras. A presentation along these lines is beyond the scope of these notes. However,
the underlying meaning is almost identical.
239
Example 8.1.3. If {Ft } is the filtration defined by
F0 := , F1 := { x1 }, F2 := { x1 , x2 }, F3 := { x1 , x2 , x3 },
and mt := t1 tj=1 x j , then {mt } is adapted to {Ft }.
Fact 8.1.2. If {mt } is adapted to {Ft }, then E [mt | Ft+ j ] = mt for any j 0.
Hopefully the reason is clear: By adaptedness, we know that mt is Ft -measurable.
From the definition of a filtration and fact 3.3.2 on page 99 it follows that mt is Ft+ j measurable. The result in fact 8.1.2 now follows from fact 3.3.6 on page 102.
To define martingales we let {mt } be a sequence of random variables adapted to a
filtration {Ft }, and satisfying E [|mt |] < for all t. In this setting, we say that {mt }
is a martingale with respect to {Ft } if
E [mt+1 | Ft ] = mt for all t

We say that {mt } is a martingale difference sequence with respect to {Ft } if
E [mt+1 | Ft ] = 0 for all t

A martingale difference sequence is so named because if {mt } is a martingale with
respect to {Ft }, then dt = mt mt1 is a martingale difference sequence with respect
to {Ft }. See exercise 8.6.8.
Note that the unconditional mean of a martingale difference sequence is always
zero, because
E [mt ] = E [E [mt | Ft1 ]] = E [0] = 0
Example 8.1.4. The classic example of a martingale is a random walk. Let {t } be
an IID sequence of random variables with E [1 ] = 0, and let mt := tj=1 j . For
example, t might be the payoff on the t-th round of a game (e.g., poker), and mt
is the wealth of a gambler after the t-th round. (We are assuming that wealth starts
at zero and may take arbitrarily large negative values without the gambler getting
ejected from the game and knee-capped by the mafia.) In this case,
t
m t = m t 1 + t =
j =1
240
is a martingale with respect to Ft := {1 , . . . , t }. That {mt } is adapted to {Ft }

follows immediately from the definition of mt and Ft . Moreover,
E [ m t + 1 | F t ] = E [ 1 + t + t + 1 | F t ]
= E [ 1 | F t ] + E [ t | F t ] + E [ t + 1 | F t ]
= 1 + t + E [ t + 1 | F t ]
= 1 + t + E [ t + 1 ]
= 1 + t = m t
Example 8.1.5. A famous example of a martingale in economic theory is Robert
Halls hypothesis that consumption is a martingale (Hall, 1978). To understand his
hypothesis, consider an Euler equation of the form

1 + r t +1 0
0
u ( c t +1 )
u (ct ) = E t
1+$
where u0 is the derivative of a utility function u, rt is an interest rate and $ is a
discount factor. The time t expectation E t [] can be thought of as a conditional expectation E [ | Ft ], where Ft contains all variables observable at time t. Specializing
to the case rt+1 = $ and u(c) = c ac2 /2, the Euler equation reduces to
c t = E t [ c t +1 ] = : E [ c t +1 | F t ]
Thus, under the theory, consumption is a martingale with respect to {Ft }.
8.2
Dynamic Properties
The time series models discussed above can display very different dynamics from
the simple IID data processes considered earlier in this text. This has profound implications for asymptotic theory, such as the law of large numbers or central limit
theorem. In this section we try to unravel some of the mysteries, starting with a very
simple case.
8.2.1
A Simple Example: Linear AR(1)
Consider again the scalar Gaussian AR(1) process

xt+1 = + $xt + wt+1
with
IID
{wt } N (0, 1)
241
For simplicity, the variance of the shock wt has been set to one. In order to learn
about the dynamics of this process, lets begin with some simulated time series and
see what we observe. In all simulations, we take = 1.
Six individual time series { xt } are shown in figure 8.1, each generated using a different value of $. The code for generating the figure is given in listing 12. As suggested
by the figure (experiment with the code to verify it for yourself), the simulated time
paths are quite sensitive to the value of the coefficient $. Whenever $ is outside the
interval (1, 1), the series tend to diverge. If, on the other hand, |$| < 1, then the
process does not diverge. For example, if you look at the time series for $ = 0.9 in
figure 8.1, you will see that, after an initial burn in period where the series is affected
by the initial condition x0 , the process settles down to random motion within a band
(between about 5 and 15 in this case).
Listing 12 Code for figure 8.1
# Generates an AR (1) time series starting from x = init
ar1ts <- function ( init , n , alpha , rho ) {
x <- numeric ( n )
x [1] <- init
w <- rnorm (n -1)
for ( t in 1:( n -1) ) {
x [ t +1] <- alpha + rho * x [ t ] + w [ t ]
}
return ( x )
}
rhos <- c (0.1 , -0.1 , 0.9 , -0.9 , 1.1 , -1.1)
N <- 200
par ( mfrow = c (3 ,2) ) # Arrangement of figures
for ( rho in rhos ) {
plot ( ar1ts (0 , N , 1 , rho ) , type = " l " ,
xlab = paste ( " rho = " , rho ) , ylab = " " )
}
We can investigate this phenomenon analytically by looking at expression (8.7).
Since the shocks {wt } are assumed to be normal, it follows from this expression
and fact 1.2.6 on page 24 that xt will be normally distributed whenever x0 is either
normal or constant. Lets assume that this is the case. In particular, lets assume that
242
1 0
50
100
150
200
50
150
200
150
200
150
200
rho = 0.1
10
15
rho = 0.1
100
50
100
150
200
50
rho = 0.9
0e+00
6e+07
0.0e+00
1.0e+09
6e+07
rho = 0.9
100
50
100
rho = 1.1
150
200
50
100
rho = 1.1
Figure 8.1: Dynamics of the linear AR(1) model
243
x0 N (0 , 02 ), where 0 and 0 are given constants. Applying our usual rules for
expectation and variance to (8.7), we can also see that
t := E [ xt ] =
t 1
+ $ 0 and
t2
t 1
:= var[ xt ] =
k =0
$2k + $2t 02
k =0
Since xt is normal and weve now found the mean and variance, weve pinned down
the marginal distribution t for xt . In particular, we have shown that
!
t = N (t , t2 ) = N
t 1
t 1
k =0
k =0
$k + $t 0, $2k + $2t 02
Notice that if |$| 1, then the mean and variance diverge. If, on the other hand,
|$| < 1, then
1
2
and t2
t :=
:=
1$
1 $2
In this case, it seems likely that the marginal distribution t = N (t , t2 ) of xt con2 ). Using fact 1.4.3 on
verges weakly (see the definition in 2.5.1) to := N ( ,
page 33, one can then show that this is indeed the case. That is,

1
2 d
2
,
(8.11)
t = N (t , t ) := N ( , ) := N
1 $ 1 $2
Observe that this limit does not depend on the starting values 0 and 02 . In other
words, does not depend on 0 .
Figures 8.2 and 8.3 illustrate convergence of t to , and of the corresponding
densities t to , when = 0 and $ = 0.9. The initial distribution in the figure is
0 := N (0 , 02 ) with arbitrarily chosen constants 0 = 6 and 02 = 4.2. For both
the sequence of cdfs and the sequence of densities, convergence is from left to right.
The code is given in listing 13, and if you experiment with different choices of 0
and 0 , you will see that convergence to the same distribution always occurs.
The fact that t for any choice of 0 is called global stability, or ergodicity.
A more formal definition is given below.3
Besides being the limiting distribution of the sequence {t }, the distribution
has another special property: If we start with 0 = , then we will have t =
3 The
term ergodicity is sometimes used to signify that the process satisfies the law of large numbers, as described in the next section. However, as will be discussed at length there, global stability
and the law of large numbers are closely related.
244
0.0
0.2
0.4
0.6
0.8
1.0
10
0.00
0.05
0.10
0.15
0.20
Figure 8.2: Convergence of cdfs
10
Figure 8.3: Convergence of densities
245

rho <- 0.9
N <- 30 # Number of densities to plot
mu <- -6; sigma2 <- 0.8 / (1 - rho ^2) # mu _ 0 and sigma ^2 _ 0
xgrid <- seq ( -10 , 7 , length =200)
plot ( xgrid , dnorm ( xgrid , mean = mu , sd = sqrt ( sigma2 ) ) ,
type = " l " , xlab = " " , ylab = " " , main = " " )
for ( i in 2: N ) {
mu <- rho * mu
sigma2 <- rho ^2 * sigma2 + 1
lines ( xgrid , dnorm ( xgrid , mean = mu , sd = sqrt ( sigma2 ) ) )
}
for all t. For example, if, in figure 8.2 we had started at 0 = , then we would see
only one curve, which corresponds to . The sequence of distributions is constant.
For this reason, is called the stationary distribution of the process. Note also
that for our model, we have only one stationary distribution. In particular, if we start
at any other cdf 0 6= , then we will see motion in the figure as the sequence t
converges to .
The fact that if 0 = , then t = for all t is a very important point, and
as such its worth checking analytically as well. The way to do this is by induction, showing that if t = , then t+1 = is also true.4 To verify the latter,
one can use the relation xt+1 = + $xt + wt+1 . The details are left as an exercise
(exercise 8.6.7).
Thus, taking 0 = implies that xt has the same marginal distribution for every t. In other words, the sequence of random variables { xt } is identically distributed.
It is not, however, IID, because xt and xt+ j are not independent (unless $ = 0). Well
say more about this in just a moment.
logic is as follows: Suppose we know that (a) 0 = , and (b) t = implies t+1 =
. Then (a) and (b) together imply that 1 = . Next, using (b) again, we get 2 = . Using
(b) one more time we get 3 = , and so on. Hence t = for all t, as was to be shown.
4 The
8.2.2
246
Linear AR(1) Continued: The LLN
For consistency of statistical procedures, some version of the LLN is almost always
necessary. The scalar and vector LLNs we have considered so far used the IID assumption. In particular, they required zero correlation between elements of the sequence. If we admit nonzero correlation, then, as well see below, the LLN can easily
fail.
Fortunately, the global stability concept we have just investigated provides one way
to obtain the LLN without IID data. Lets think about this how this connection might
work, starting with the AR(1) model
xt+1 = + $xt + wt+1
with
IID
{wt } N (0, 2 )
Suppose for now that = 0, $ = 1 and = 0. Setting = 0 means that wt

N (0, 0), or, in other words, wt is identically equal to zero. In this case, the dynamics
reduce to xt+1 = xt , and hence
x t = x t 1 = x t 2 = = x 1
If x0 has some given distribution , then clearly
t ( s ) : = P{ x t s } = P{ x1 s } = ( s )
for all
This tells us that { xt } is identically distributed, with common distribution . (The

distribution is stationary for this process, which is why I chose the symbol .
In fact every distribution is stationary for this process, because if we fix any distribution and let x0 have that distribution, then xt will have that same distribution for
all t, as we just established.) However, the sequence { xt } does not satisfy the LLN.
Indeed,
1 T
1 T
x T := xt = x1 = x1
T t =1
T t =1
and x1 , being a random variable, does not generally converge to anything.5
So under what conditions does the sample mean of the AR(1) process converge to
the common mean of xt ? One necessary condition is that the sequence { xt } does
indeed have a common mean. So lets restrict attention to the case where the parameters , $ and are such that some stationary distribution does exist,6 and
5 The
only exception is if x0 is a degenerate random variable, putting all its probability mass on a
single point. If this is not clear, go back to exercise 1.6.28.
6 For example, if | $ | 1 and > 0, then no stationary distribution exists. If | $ | < 1, then a unique
stationary distribution always exists, for any values of and .
247
start the process off with 0 = , so that t = for all t. Hence { xt } is identically distributed. We want to know when
p
x T E [ xt ] = :=
s (ds)
(8.12)
We saw in the previous example ( = 0, $ = 1, = 0) that the mere fact that { xt }

is identically distributed is not enough. We need something more. To investigate
this, lets recall the proof of the LLN for IID sequences in theorem 1.4.1 (page 34). In
the theorem, we saw that when { xn } is an IID sequence of random variables with
common mean and variance 2 , we have
x N :=
1
N
xn E [ xn ] =
n =1
For the proof, we just observe that since E [ x N ] = , fact 1.4.2 on page 31 implies
that the result will hold whenever var[ x N ] 0 as N . In view of fact 1.3.9 on
page 28, we have
"
#
1 N
2
1 N
var
cov[ xn , xm ]
var
[
x
]
+
x
=
n
n
N n =1
N 2 n =1
N 2 n
<m
2
2
+ 2
N
N
cov[ xn , xm ]
n<m
In the IID case, cov[ xn , xm ] = 0 for all n < m, and hence the convergence var[ x N ] 0
does indeed hold.
Now lets weaken the assumption of zero correlation, and try to think about whether
the LLN can be salvaged. When correlation is non-zero, the question of whether
or not var[ x N ] 0 depends whether or not most of the terms cov[ xn , xm ] are
small. This will be the case if the covariances die out relatively quickly, so that
cov[ xn , xn+ j ] 0 when j is large. Furthermore, the property that correlations die
out over time is closely related with global stability. For example, lets take = 0
and = 1, so the dynamics are
xt+1 = $xt + wt+1
with
IID
{wt } N (0, 1)
(8.13)
We saw in (8.11) that, under the assumption 1 < $ < 1, the model has a unique,
globally stable stationary distribution given by

1
(8.14)
:= N 0,
1 $2
248
It turns out that the stability condition 1 < $ < 1 is precisely what we need for the
covariances to die out. Indeed, fixing j 1 and iterating with (8.13), we obtain
j
xt+ j = $ xt +
$ jk wt+k
k =1
we now have
"
cov[ xt+ j , xt ] = E [ xt+ j xt ] = E
$ j xt +
$ jk wt+k
!
xt
k =1
= $ j E [ xt2 ] +
= $ j E [ xt2 ] +
$ jk E [ wt+k xt ]
k =1
j
$ j k E [ w t + k ]E [ x t ]
k =1
=$
E [ xt2 ]
(In the second last equality, we used the fact that xt depends only on current and
lagged shocks (see (8.7) on page 236), and hence xt and wt+k are independent.) Since
|$| < 1 we now have
cov[ xt+ j , xt ] = $ j E [ xt2 ] =
$j
0
1 $2
as
To summarize our discussion, to make the LLN work we need covariances to go to

zero as we compare elements of the sequence that are more and more separated in
time. The condition for covariances going to zero is exactly the condition we require
for global stability (i.e., |$| < 1).
The LLN result (8.12) in the AR(1) model (8.13) is illustrated in figure 8.4. In the
R
simulation, we set $ = 0.8. Note that for this model, the mean s (ds) of is
zero. The grey line is a simulated time series of the process. The blue line is the plot
of x T against T. As anticipated by the LLN, we see that x T converges to zero. The
code for producing figure 8.4 is given in listing 14.
8.2.3
Markov Dynamics
Studying the dynamics of the AR(1) process has helped us build intuition, but now
we need to look at more general (and complex) cases. Lets look again at the Markov
249
500
1000
1500
2000
Index
Figure 8.4: LLN in the AR(1) case

rho = 0.8
N <- 2000
x <- numeric ( N )
sample _ mean <- numeric ( N )
x [1] <- 1
for ( t in 2: N ) x [ t +1] <- rho * x [ t ] + rnorm (1)
for ( T in 1: N ) sample _ mean [ T ] <- sum ( x [1: T ] / T )
plot (x , col = " gray " , type = " l " )
lines ( sample _ mean , col = " blue " , lwd =2)
abline (0 , 0)
250
process (8.5) on page 236, which includes the AR(1) model as a special case. For each
t 1, let t denote the density of xt . In particular, for B RK , we have
Z
B
t (s) ds = P{xt B} = P{ Ht (x0 , w1 , w2 , . . . , wt ) B}
where Ht is defined on page 236.7 Lets work towards finding conditions for global
stability, so that this sequence of densities will converge to a unique limit for all
initial 0 . First we need some definitions. These definitions include and formalize
the stability-related definitions given for the scalar AR(1) model.
Lets start with the definition of stationary distributions. Let p( | s) be the transition
density of the process, as defined in (8.8). Given p, a density on RK is called
stationary for the process (8.16) if
0
(s ) =
p(s0 | s) (s) ds
for all s0 RK
(8.15)
A stationary density is called globally stable if the sequence of marginal densities {t }

t=0 converges to for any choice of initial density 0 .
The interpretation of (8.15) is as follows. Suppose that xt . Informally, the
probability that xt+1 = s0 should be equal to the probability that xt+1 = s0 given
xt = s, summed over all possible s and weighted by the probability that xt = s.
This is the right-hand side of (8.15). The equality in (8.15) says that this probability
is (s0 ). In other words, xt+1 whenever xt . This is the reason is
called a stationary distribution.
As a consequence of the preceding discussion, we have the following result:
Fact 8.2.1. Let {xt } be a Markov process with stationary distribution . If x0 ,
then {xt } is identically distributed, with common marginal distribution .
While stability or instability of linear models like the AR(1) process is relatively easy
to study, for the more general Markov model we are studying now, the analysis can
be somewhat tricky. Here is a useful result pertaining to the additive shock model
x t +1 = g ( x t ) + w t +1
(8.16)
Here {wt }t1 is an IID sequence of RK -valued shocks with common density , and
x0 has density 0 .
7 In
the presentation below were going to work with densities rather than cdfs because the presentation is a little easier.
251
Theorem 8.2.1. If the density has finite mean and is strictly positive everywhere on RK ,
the function g is continuous and there exist positive constants and L such that < 1 and
k g(s)k ksk + L
for all s RK
(8.17)
then (8.16) has a unique stationary distribution that is globally stable.

Theorem 8.2.1 is a very handy result, although it is but one example of a condition
ensuring global stability.8 Well look at how to check the conditions of theorem 8.2.1
in a moment, but before that lets consider the connection between this stability
theorem and the LLN and CLT. We saw in the scalar AR(1) case that this connection
is rather close, at least for the LLN. This carries over to the general Markov case
treated here. Indeed, the stability conditions in theorem 8.2.1 are sufficient for the
LLN, and for the CLT under an additional moment condition. The details follow.
Theorem 8.2.2. Let the conditions of theorem 8.2.1 hold. Let be the unique stationary
density. Let {xt } be an generated by (8.16) with x0 . If h : RK R is any function
R
such that |h(s)| (s)ds < , then9
1
T
h(xt ) E [h(xt )] =
h(s) (s)ds
as
t =1
In fact, under the conditions of theorem 8.2.2, if, in addition, the second moment
R
condition |h(s)|2+ (s)ds < holds for some > 0, then the following CLT is
valid:
(
)
1 T
d
T
h(xt ) E [h(xt )] N (0, h2 ) as T
(8.18)
T t
=1
where
h2
:= var[h(x0 )] + 2 cov[h(x0 ), h(xt )]
(8.19)
t =1
For a proof of this last result see corollary 2 of Jones (2004).

To apply theorems 8.2.1 and 8.2.2, we need to be able to check the conditions of
theorem 8.2.1. The best way to learn to do this is by looking at examples.
8 In
fact the conditions of theorem 8.2.1 are rather strong (in order to make the statement of the
theorem straightforward). There are many other conditions for this kind of stability, based on a
variety of different criteria. For further discussion of the Markov case see Stachurski (2009) and the
references therein.
R
9 The condition that x is actually unnecessary, but it means that E [ h ( x )] =
h(s) (s)ds
t
0
for all t, which makes the result a little easier to digest. See Stachurski (2009) for a more formal
discussion of this LLN, and references containing proofs.
252
Example 8.2.1. Let { xt } be a scalar AR(1) process xt+1 = $xt + + wt+1 , where
{wt } is IID with finite mean and density = N (, 2 ) Assume that 1 < $ < 1.
The conditions of theorem 8.2.1 are satisfied. To see this, we can rewrite the model
as xt+1 = g( xt ) + wt+1 where g(s) := $s + . By the triangle inequality, | g( x )|
|s| + L where := |$| and L := ||. The conditions on are also satisfied.
Example 8.2.2. Heres a variation on the scalar threshold autoregressive model:
xt+1 = $| xt | + (1 $2 )1/2 wt+1
with
IID
1 < $ < 1 and {wt } N (0, 1)
The conditions of theorem 8.2.1 are satisfied. To see this, we can rewrite the model
as
IID
xt+1 = g( xt ) + vt+1 , g(s) = $|s| and {vt } N (0, 1 $2 )
Clearly the distribution N (0, 1 $2 ) of vt has finite mean and a density that is everywhere positive on R. Moreover, | g(s)| = |$||s|, so that (8.17) is satisfied with
= |$| and L = 0. By assumption, we have < 1, and hence all the conditions
of theorem 8.2.1 are satisfied, and a unique, globally stable stationary density exists. While for many Markov processes the stationary density has no known closed
form solution, in the present case the stationary density is known to have the form
(s) = 2(s)(qs), where q := $(1 $2 )1/2 , is the standard normal density
and is the standard normal cdf.
Example 8.2.3. Consider again the VAR(1) process from (8.2), with
xt+1 = a + xt + wt+1
(8.20)
To study the dynamics of this process, its useful to recall the definition of the spectral norm of , which is given by10
$() := max
s 6 =0
ksk
ksk
Let us consider the drift condition (8.17) applied to the law of motion (8.20). In this
case, g(s) = a + s, and, using the triangle inequality for the norm (fact 2.1.1 on
page 53), we have
k g(s)k = ka + sk kak + ksk
10 Readers familiar with the notion of eigenvalues might have seen the spectral norm defined as the
square root of the largest eigenvalue of . The two definitions are equivalent. The second definition
is the most useful for when it comes to numerical computation.
253
Let := $() and let L := kak. We then have
ksk
ksk + L $()ksk + L =: ksk + L
ksk
We can now see that the drift condition (8.17) will be satisfied whenever the spectral
norm of is less than one.
k g(s)k ksk + L =
Example 8.2.4. Consider the STAR model introduced in 8.1.2, where the function g
is given by
g(s) := (0 + $0 s)(1 (s)) + (1 + $1 s) (s)
and : R [0, 1]. Applying the triangle inequality | a + b| | a| + |b|, we obtain
| g(s)| |(0 + $0 s)(1 (s))| + |(1 + $1 s) (s)|

|0 | + |1 | + |$0 | |s(1 (s))| + |$1 | |s (s)|
Letting L := |0 | + |1 | and := max{|$0 |, |$1 |}, we then have
| g(s)| |s| + L
If both |$0 | and |$1 | are strictly less than one, the condition (8.17) is satisfied. If, in
addition, is continuous, then g is continuous. If the distribution of the shock has,
say, a normal distribution, then it has an everywhere positive density on R, and all
the conditions of theorem 8.2.1 are satisfied. Hence the process is globally stable.
One interesting special case of theorem 8.2.2 is if we take h(xt ) = 1{xt s} for
some fixed s RK . By (1.8) on page 14, we have
E [1{xt s}] = P{xt s} = (s)

so in this case theorem 8.2.2 yields
1
T
1{ x t s } ( s )
t =1
In other words, the ecdf converges to the stationary cdf , just as for the IID case.
The ecdf is a nonparametric estimator of the cdf. Provided that the cdf has
a density, the same idea works for the standard nonparametric density estimator
discussed in 4.4.3. Lets test this for the model in example 8.2.2. For this model, the
stationary density is known to be (s) = 2(s)(qs), where q := $(1 $2 )1/2 ,
is the standard normal density and is the standard normal cdf. This is the blue
line in figure 8.5. (The value of $ is 0.95.) Next, we generate a time series, and plot
s x
1
the nonparametric kernel density estimate T
tT=1 K ( t ) as a black line, using Rs
default choice of K and . Here T = 5000, and the fit is pretty good. The code for
producing figure 8.5 is given in listing 15.
254
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Figure 8.5: Stationary density and nonparametric estimate

rho <- 0.95
delta <- rho / sqrt (1 - rho ^2)
T <- 5000
x <- numeric ( T )
x [1] = 0
for ( t in 2: T ) {
x [ t ] = rho * abs ( x [t -1]) + sqrt (1 - rho ^2) * rnorm (1)
}
plot ( density ( x ) , xlab = " " , ylab = " " , main = " " )
xgrid <- seq ( -4 , 4 , length =200)
pistar <- function ( s ) {
return (2 * dnorm ( s ) * pnorm ( s * delta ) )
}
lines ( xgrid , pistar ( xgrid ) , col = " blue " )
8.2.4
255
Martingale Difference LLN and CLT
Next we consider asymptotics for martingale difference sequences. Martingale difference sequences are important to us because they are good candidates for the LLN
and CLT. To see this, suppose that {mt } is a martingale difference sequence with
respect to filtration {Ft }. Suppose further that {mt } is identically distributed, and
E [m21 ] < . If the variables {mt } are also independent, then the classical LLN and
CLT apply (theorems 1.4.1 and 1.4.2 respectively). Here we do not wish to assume
independence, but with martingale difference sequences we do at least have zero
correlation. To see this, fix t 0 and j 1. We have
cov[mt+ j , mt ] = E [mt+ j mt ] = E [E [mt+ j mt | Ft+ j1 ]]
Since t + j 1 t and {Ft } is a filtration, we know that mt is Ft+ j1 -measurable,
and hence
E [E [mt+ j mt | Ft+ j1 ]] = E [mt E [mt+ j | Ft+ j1 ]] = E [mt 0] = 0

This confirms that martingale difference sequences are uncorrelated, as claimed.
Given the lack of correlation, we might hope that the LLN and CLT will still hold in
some form. Indeed, the following result is true:
Theorem 8.2.3. Let {mt } be identically distributed. If {mt } is a martingale difference
sequence with respect to some filtration {Ft }, then
1
T
mt 0
as
(8.21)
t =1
If, in addition, 2 := E [m2t ] is positive and finite, and

1
T
then
"
1
T
T
E [m2t | Ft1 ] 2
#
mt = T 1/2
t =1
as
t =1
mt N (0, 2 )
as
(8.22)
t =1
The LLN result in (8.21) can be proved in exactly the the same way we proved the
classical LLN in theorem 1.4.1. Theorem 8.2.3 is a consequence of a martingale CLT
proved in Durrett (1996, theorem 7.4).11 We will use theorem 8.2.3 in our large sample OLS theory below.
11 Deriving
theorem 8.2.3 from the result in Durrett requires some measure theory, and is beyond
the scope of these notes. If you know measure theory then you should be able to work out the proof.
8.3
256
Maximum Likelihood for Markov Processes
[roadmap]
8.3.1
The Likelihood Function
If we have IID scalar observations x1 , . . . , x N from common density p , then, as we

saw in 4.3.1, the joint density is the product of the marginals, and the maximum
likelihood estimate (MLE) of is
:= argmax L( )
where
L( ) =
p ( xn )
(8.23)
n =1
If, on the other hand, our data x1 , . . . , x T is a time series where the independence
assumption does not hold, then the joint density is no longer the product of the
marginals. To obtain a convenient expression for the joint density in this general
case, lets begin by constructing a joint density for the first three data points x1 , x2 , x3 .
Using (1.21) on page 26, we can write the joint density as
p ( s1 , s2 , s3 ) = p ( s3 | s1 , s2 ) p ( s1 , s2 )
Applying (1.21) again, this time to p(s1 , s2 ), we get
p ( s1 , s2 , s3 ) = p ( s3 | s1 , s2 ) p ( s2 | s1 ) p ( s1 )
Extending this from T = 3 to general T we get12
T 1
p ( s1 , . . . , s T ) = p ( s1 )
p ( s t +1 | s 1 , . . . , s t )
t =1
We can specialize further if we are dealing with a Markov process. Suppose that
x1 , . . . , x T are observations of a globally stable Markov process with transition density p(st+1 | st ) and stationary density . If the process we are observing has been
running for a while, then, given global stability, it is not unreasonable to assume
that x1 . In this case, our expression for the joint density becomes
T 1
p ( s1 , . . . , s T ) = ( s1 )
p ( s t +1 | s t )
t =1
where we are using the fact that, for a (first order, time homogeneous) Markov process, p(st+1 | s1 , . . . , st ) = p(st+1 | st ).13 Finally, nothing in this expression changes if
12 Check
it by induction if you wish.

is pretty much the defining property of a first order Markov process, and follows from the
general expression xt+1 = G ( xt , wt+1 ) given on page 236.
13 This
257
we shift to the vector case, so the joint density of a Markov process x1 , . . . , x T with
transition density p(st+1 | st ) and x1 has the form
T 1
p ( s1 , . . . , s T ) = ( s1 )
p ( s t +1 | s t )
(8.24)
t =1
Turning to the likelihood function, lets suppose now that p depends on an unknown parameter vector , and write p . Since the stationary density is
determined by p (see (8.15) on page 250) we indicate this dependence by writing it
. The log-likelihood function is then given by
as
T 1
`( ) =
ln
( x1 ) +
ln p (xt+1 | xt )
t =1
In practice it is common to drop the first term in this expression, particularly when
the data size is large. There are two reasons. First, if the data size is large, then there
are many elements in the sum, and the influence of a single element is likely to be
is formally defined by
negligible. Second, even though the stationary density
(8.15), for a great many processes there is no known analytical expression for this
density.14 Here well follow this convention and, abusing notation slightly, write
T 1
`( ) =
ln p (xt+1 | xt )
(8.25)
t =1
8.3.2
Example: The ARCH Case
Recall the ARCH process

xt+1 = ( a + bxt2 )1/2 wt+1 ,
IID
{wt } N (0, 1)
(8.26)
The transition density for this model is the density of xt+1 given xt = s. If xt = s,
then xt+1 N (0, a + bs2 ), and hence

( s 0 )2
0
2 1/2
p(s | s) = (2 ( a + bs ))
exp
2( a + bs2 )
14 In
this situation it is still possible to compute the density numerically, using simulation. The
discussion surrounding figure 8.5 gives some idea. A better technique is discussed in Stachurski and
Martin (2008).
258
Since a + bs2 is the conditional variance of xt+1 , the parameters a and b are restricted
to be nonnegative. Moreover, if b < 1, then one can show that the process is globally
stable.15 From (8.25), the log-likelihood function is
)
(
2
T 1
x
1
t +1
(8.27)
`( a, b) = ln(2 ( a + bxt2 ))
2)
2
2
(
a
+
bx
t
t =1
Rearranging, dropping terms that do not depend on a or b, and multiplying by 2 (an
increasing transformation), we can rewrite this (abusing notation again) as
(
)
T 1
xt2+1
`( a, b) = ln zt +
where zt := a + bxt2
(8.28)
z
t
t =1
Lets run some simulations to see what this function looks like. In the simulations
we will set T = 500 and a = b = 0.5. Thus, we imagine the situation where,
unbeknownst to us, the true parameter values are a = b = 0.5, and we observe a
time series x1 , . . . , x500 generated by these parameters. In order to estimate a and b,
we form the likelihood function (8.28), and obtain the MLEs a and b as the vector
( a , b ) that maximizes `( a, b).
Four different simulations of ` are given in figure 8.6. In each figure, a separate data
set x1 , . . . , x500 is generated using the true parameter values a = b = 0.5, and the
function ` in (8.28) is then plotted. Since the graph of the function is three dimensional (i.e, the function has two arguments), we have plotted it using contour lines
and a color map. Lighter colors refer to larger values. The horizontal axis is a values, and the vertical axis is b values. The code for producing one of these figures
(modulo randomness) is given in listing 16. The function arch_like(theta, data)
represents ` in (8.28), with theta corresponding to ( a, b) and data corresponding to
the time series x1 , . . . , x T .
In each of the four simulations, a rough guess of the MLEs can be obtained just by
looking for maximizers in the figures. For example, in simulation (a), the MLEs look
to be around a = 0.44 and b = 0.61. To get more accurate estimates, we can use some
form of analytical or numerical optimization. For this problem, we dont have any
analytical expressions for the MLEs because setting the two partial derivatives of `
On the other hand, there
in (8.28) to zero does not yield neat expressions for a and b.
are many numerical routines we can use to obtain the MLEs for a given data set.
15 Unfortunately,
theorem 8.2.1 (page 251) cannot be used to check global stability, because the
shock is not additive. If you wish to verify global stability then have a look at the techniques in
Chapter 8 of Stachurski (2009).
259
The simplest approach is to use one of Rs inbuilt optimization routines. For example, given the definition of arch_like in listing 16 and a sequence of observations
x1 , . . . , x T stored in a vector xdata, the function arch_like can be optimized numerically via the commands
start _ theta <- c (0.65 , 0.35) # An initial guess of (a , b )
neg _ like <- function ( theta ) {
return ( - arch _ like ( theta , xdata ) ) # xdata is the data
}
opt <- optim ( start _ theta , neg _ like , method = " BFGS " )
Here optim is an built-in R function for numerical optimization of multivariate functions. Most built-in functions in most languages perform minimization rather than
maximization, and optim is no exception. For this reason, the function that we pass
to optim is neg_like, which is 1 times `. The first argument to optim is a vector of
starting values (a guess of the MLEs). The last argument tells optim to use the BFGS
routine, which is variation on the Newton-Raphson algorithm. The return value of
optim is a list, and the approximate minimizing vector is one element of this list
(called par).
In this particular set up, for most realizations of the data and starting values, you
will find that the algorithm converges to a good approximation to the global optimizer. However, theres no guarantee that it will. In case of problems, its useful to
know how these kinds of algorithms work, and how to code up simple implementations on your own. The next section will get you started.
8.3.3
The Newton-Raphson Algorithm
The Newton-Raphson algorithm is a root-finding algorithm. In other words, given a

function g : R R, the algorithm searches for points s R such that g(s) = 0. Any
root-finding algorithm can be used to optimize differentiable functions because, for
differentiable functions, interior optimizers are always roots of the objective functions first derivative.
To describe the algorithm, lets begin with the root-finding problem and then specialize to optimization. To begin, let g : R R, and let s0 be some initial point in R
that we think (hope) is somewhere near a root. We dont know how to jump from s0
straight to a root of g (otherwise there would be no problem to solve), but what we
can do is move to the root of the function which forms the tangent line to g at s0 . In
6
45
46
54
4
46
2
48
47
6
46
49
0
53
4
50
6
51
0.3
458
48
0
47
8
47
472
508
0
52
49
480
464
490
0.4
500
512
476
468
484
506
494
434
0.5
498
3
9
488
3
92
0.6
438
0.6
0.7
0.7
68
4
66
4
64
4
45
50
0.8
0.8
0.3
26
22
4
18
4
08
4 4
14 12
0.3
0.4
0.4
0.5
0.5
b
42
8
41
42
42
0.5
0.6
0.6
0.7
0.7
34
6
41 4
62
4
60
4
58
54
4
5
4
42
0
442
40
0.4
36
30
26
06
1
4
8
1
4
26
4
4
3
4
56
4
4
4
42
0
41
02
08
4
4
26
16
4
38
0.3
28
41
2
41 0
41 8
40
04
00
4
4
08
14
2
4
8
42
6
43
44
0
43
6
42
41
40
8
41
2
42
446
4
41
40
96
3
04
0
18
1
4
4
8
42
Figure 8.6: Simulations of the function (8.28) with T = 500
32
4
52
0.8
0.7
06
0.8
0
4
43
42
8
43
0.8
0.7
0.6
0.5
0.4
00
41
41
42
2
43
a
a
50
448
6
41
0
42
6
40
2
41
384
444
2
40 0
40 8
39 6
39
4
41
0
40
382
0.3
0.8
8
42
26
80
0
4
2
43
43
43
8
43
6
42
0
43
16
4
44
34
4
(d) Simulation 4
(c) Simulation 3
(b) Simulation 2
(a) Simulation 1
a
a
39
2
386
0.6
0.5
0.4
0.3
0.8
0.7
396
0.8
42
12
2
3
4
18
24
4
4
396
394
4
394
22
4
388
0.7
0.6
0.5
0.4
0.3
92
39
390
94
3
98
3
386
6
41
392
02
0
40
404
8
0.6
0.5
0.4
0.3
398
398
42
390
1
4
402
06
40
14
20
400
4
4
1
260
388
440
436
261

arch _ like <- function ( theta , data ) {
Y <- data [ -1]
# All but first element
X <- data [ - length ( data ) ] # All but last element
Z <- theta [1] + theta [2] * X ^2
return ( - sum ( log ( Z ) + Y ^2 / Z ) )
}
sim _ data <- function (a , b , n =500) {
x <- numeric ( n )
x [1] = 0
w = rnorm ( n )
for ( t in 1:( n -1) ) {
x [ t +1] = sqrt ( a + b * x [ t ]^2) * w [ t ]
}
return ( x )
}
xdata <- sim _ data (0.5 , 0.5)
True parameters
K <- 50
a <- seq (0.3 , 0.8 , length = K )
b <- seq (0.3 , 0.8 , length = K )
M <- matrix ( nrow =K , ncol = K )
for ( i in 1: K ) {
for ( j in 1: K ) {
theta <- c ( a [ i ] , b [ j ])
M [i , j ] <- arch _ like ( theta , xdata )
}
}
image (a , b , M , col = topo . colors (12) )
contour (a , b , M , nlevels =40 , add = T )
262
s1 s0
Figure 8.7: The first step of the NR algorithm
other words, we replace g with its linear approximation around s0 , which is given
by
g (s) := g(s0 ) + g0 (s0 )(s s0 )
( s R)
This point is represented as s1 in figure 8.7, and the value
and solve for the root of g.
is easily seen to be s1 := s0 g(s0 )/g0 (s0 ). The point s1 is taken as our next guess of
the root, and the procedure is repeated, taking the tangent of g at s1 , solving for the
root, and so on. This generates a sequence of points {sk } satisfying
s k +1 = s k
g(sk )
g0 (sk )
There are various results telling us that when g is suitably well-behaved and s0 is
sufficiently close to a given root s, then sequence {sk } will converge to s.16
To move from general root-finding to the specific problem of optimization, suppose
now that g : R R is a differentiable function we wish to maximize. We know that
if s is a maximizer of g, then g0 (s ) = 0. Hence it is natural to begin our search for
maximizers by looking for roots to this equation. This can be done by applying the
Newton-Raphson algorithm to g0 , which yields the sequence
s k +1
g0 (sk )
= sk 00
g (sk )
(8.29)
We can extend this algorithm to the multivariate case as well. Lets suppose that g
is a function of two arguments. In particular, suppose that g is twice differentiable
and g : R2 R. The gradient vector and Hessian of g at ( x, y) R2 are defined as
0

00

00 ( x, y )
g1 ( x, y)
g11 ( x, y) g12
2
g( x, y) :=
and g( x, y) :=
00 ( x, y ) g00 ( x, y )
g20 ( x, y)
g21
22
Here gi0 is the first partial of g with respect to its i-th argument, gij00 second derivative
cross-partial, and so on.
16 In practical situations we often have no way of knowing whether the conditions are satisfied, and
there have been many attempts to make the procedure more robust. The R function optim described
above is a child of this process.
263
By analogy with (8.29), the Newton-Raphson algorithm for this two dimensional
case is the algorithm that generates the sequence {( xk , yk )} defined by
( xk+1 , yk+1 ) = ( xk , yk ) [2 g( xk , yk )]1 g( xk , yk )
(8.30)
from some initial guess ( x0 , y0 ).17

For the sake of the exercise, lets apply this to maximization of (8.28). Let zt be as
defined in (8.28). The first partials are then
#
#
"
"
2
T 1 x 2
T 1
x
`
1
1
`
t +1
( a, b) :=
( a, b) := xt2 t+2 1
,
2
a
zt
b
zt
zt
zt
t =1
t =1
while the second partials are
"
#
T 1
xt2+1
1
2 `
( a, b) := 2 2 3 ,
a2
zt
t =1 z t
2 `
( a, b) :=
b2
T 1
xt4
t =1
"
xt2+1
1
2 3
z2t
zt
The cross-partial is
2 `
( a, b) :=
ab
T 1
t =1
"
xt2
xt2+1
1
2 3
z2t
zt
From these expressions we can easily form the gradient vector and the Hessian,
pick an initial guess, and iterate according to (8.30). Figure 8.8 show four iterations
of this procedure, starting from ( a0 , b0 ) = (0.65, 0.35).18 In this case the convergence
is quick, and we are already close to the global optimum.
Replication of this figure (modulo randomness) is left as an exercise.
8.4
Models with Latent Variables
To be written. Max likelihood with latent variables. GARCH, HMM, Markov switching, factor models?
8.5
Further Reading
To be written.
17 We
18 As
are assuming that the Hessian matrix is invertible.

before, the simulation uses a = b = 0.5 and T = 500.
264
8
4
40
39
41
41
40
41
39
2
39
8
38
6
38 4
38
2
38 0
38
39
0.7
41
40
41
40
40
39
0.8
0.5
0.6
370
88
40
39
40
0.4
4
39
368
80
86
78
376
0.5
78
92
0.4
374
3
8
84
98
06
24
0.3
14
0.3
90
3
00
372
0.6
0.7
0.8
Figure 8.8: Newton-Raphson iterates
8.6
Exercises
Ex. 8.6.1. Using fact 1.4.1 (page 31) as appropriate, prove the following part of
p
fact 2.5.3: If Xn X and Yn Y, then Xn Yn XY whenever the matrices are

conformable.
p
Ex. 8.6.2. Confirm the following claim in fact 2.5.2: If a0 xn a0 x for every a RK ,
p
then xn x.
Ex. 8.6.3. Let {xn } be a sequence of vectors in R2 , where xn := ( xn , yn ) for each n.
p
Suppose that xn 0 (i.e., xn 0 and yn 0). Show that kxn k 0.

Ex. 8.6.4. Verify fact 2.5.1 on page 76. (Note that exercise 8.6.3 is a warm up to this
exercise.)
d
Ex. 8.6.5. Confirm the claim N (x N ) N (0, ) in theorem 2.5.1.
Ex. 8.6.6. Let {xn } be an IID sequence of random vectors in RK with E [xn ] = 0 and
var[xn ] = IK . Let
1 N
x N :=
xn and y N := N kx N k2
N n =1
265
What is the asymptotic distribution of {y N }?

Ex. 8.6.7. Suppose that xt+1 = + $xt + wt+1 , where

1
and wt+1 N (0, 1)
xt := N
,
1 $ 1 $2
Show that xt+1 also holds.
Ex. 8.6.8. Let {Ft } be a filtration. Show that if {mt } is a martingale with respect to
{Ft }, then dt = mt mt1 is a martingale difference sequence with respect to {Ft }.
Ex. 8.6.9. Let {Ft } be a filtration, and let {mt } be a martingale with respect to {Ft }.
Let t := mt + 1, let t := 2mt , and let t := m2t .
1. Is {t } a martingale with respect to {Ft }?
If yes, give a proof. If not, give a counterexample.19
Ex. 8.6.10. Let {t } be an IID sequence of scalar random variables with E [1 ] = 0
and var[1 ] = 2 > 0. Let {Ft } be the filtration defined by Ft := {1 , . . . , t }, and
let zt := t t for each t.
1. Is {zt } IID? Why or why not?
2. Is {zt } a martingale difference sequence with respect to {Ft }? Why or why
not?
Ex. 8.6.11. Let {t } be an IID sequence of scalar random variables with
P{1 = 1} = P{1 = 1} = 0.5

and let {Ft } be the filtration defined by Ft := {1 , . . . , t }. Let
t
mt :=
and
t := m2t t
j =1
Show that {t } is a martingale with respect to {Ft }.

19 To
give a counterexample, you need to give a specific example of the pair {mt } and {Ft } where
the stated property fails. Look in the course notes for specific examples of martingales.
266
IID
Ex. 8.6.12. Consider the scalar sequence xt+1 = $xt + wt+1 , where {wt } N (0, 1)
and x0 = 0. Let Ft := {w1 , . . . , wt }. Give conditions on $ such that
1. { xt } is a martingale with respect to {Ft }.
2. { xt } is a martingale difference sequence with respect to {Ft }.
Ex. 8.6.13. Consider again the scalar Markov sequence xt+1 = $xt + wt+1 . Assume
that {wt } is IID, having Students t-distribution with 2 degrees of freedom, and that
1 < $ < 1. Prove that this process has a unique, globally stable stationary distribution using theorem 8.2.1.
Ex. 8.6.14. Let { xt } be the scalar AR(1) process xt+1 = $xt + wt+1 where |$| < 1 and
{wt } is IID and standard normal. Let x0 be drawn from the stationary distribution
= N (0, 1/(1 $2 )). (Recall (8.14) on page 247.) Show that
1 T
1
d
xt N (0, v) where v :=
(1 $ )2
T t =1
Incidentally, the variance v increases in $ in this expression because greater correlation (|$| closer to 1) makes it harder to estimate the mean of xt via the sample mean.
Why is this so? The intuition is that dependent data are in some sense less informative than independent data. To see this, consider an extreme case of dependence:
The first observation y0 of some sequence is drawn, and then all subsequent observations yt are set equal to y0 . Now observing all {yt } is no more informative than
just observing y0 . Conversely, if all draws are independent, then each is just as informative as the last. In summary, when we try to learn about a given distribution,
independent draws are the gold standard.
Ex. 8.6.15. Recall that the sample variance is a consistent and asymptotically normal
estimator of the variance of a distribution when the data are IID. A proof was given
in example 4.2.7 on page 122. Now assume the conditions of exercise 8.6.14. While
the data sequence { xt } is no longer IID, it is a stable Markov process. Using the proof
in example 4.2.7 as a starting point, prove that asymptotic normality still holds. In
particular, show that
o
n 2
1 + $2
d
T s T var[ xt ] N (0, 2 ) where 2 := 2
(1 $2 )3
Note that this exercise involves a fair bit of calculationkeep working and youll
get there.
8.6.1
267

p
Solution to Exercise 8.6.1. Let Xn X and Yn Y. To prove that Xn Yn XY, we

need to show that the i, j-th element of Xn Yn converges in probability to the i, j-th
element of XY. By hypothesis, we have
p
n
xik
xik
and
ynkj ykj
for all
Applying fact 1.4.1 on page 31 twice, we obtain

p
n n
xik
ykj xik ykj
and then
for all
xikn ynkj xik ykj

k
In other words, the i, j-th element of Xn Yn converges in probability to the i, j-th element of XY.
p
Solution to Exercise 8.6.2. If a0 xn a0 x for every a RK , then we know in particular that this convergence holds for the canonical basis vectors. Hence
p
e0k xn e0k x
xnk x k
for every k
xn x
for every k
(elementwise convergence)
(vector convergence, by definition)
Solution to Exercise 8.6.3. From fact 1.4.1 on page 31, we know that if g : R R
p
is continuous and {un } is a scalar sequence of random variables with un u, then

p
g(un ) g(u). We also know that if un u and vn v, then un + vn u + v. By

assumption, we have
p
xn 0
p
xn2 02 = 0
and
yn 0
p
y2n 02 = 0
and
p
kxn k2 = xn2 + y2n 0 + 0 = 0

q
p
k x n k = k x n k2 0 = 0
268
Solution to Exercise 8.6.4. Let {xn } be a sequence of random vectors in RK and x

be a random vector in RK . We need to show that
p
xkn xk for all k kxn xk 0

A special case of this argument can be found in the solution to exercise 8.6.3. The
p
general case is similar: Suppose first that xkn xk for all k. Combining the various
results about scalar convergence in probability in fact 1.4.1 (page 31), one can then
verify (details left to you) that
v
u K
u
p
kxn xk := t ( xkn xk )2 0
(n )
k =1
Regarding the converse, suppose now that kxn xk 0. Fix e > 0 and arbitrary k.
From the definition of the norm we see that | xkn xk | kxn xk is always true, and
hence
| xkn xk | > e = kxn xk > e
{| xkn xk | > e} {kxn xk > e}
0 P{| xkn xk | > e} P{kxn xk > e} 0
The proof is done.

Solution to Exercise 8.6.5. Define
zn := N (x N )
and
z N (0, )
We need to show that zn z. To do this, we apply the Cramer-Wold device

(fact 2.5.2, page 77) and the scalar CLT (theorem 1.4.2, page 37). To begin, fix a RK .
Observe that
a0 zn := N (y n E [yn ])
where yn := a0 xn . Since yn is IID (in particular, functions of independent random
variables are independent) and
var[yn ] = var[a0 xn ] = a0 var[xn ]a = a0 a
the scalar CLT yields
d
a0 zn N (0, a0 a)
d
Since a0 z N (0, a0 a), we have shown that a0 zn a0 z. Since a was arbitrary, the
Cramer-Wold device tells us that zn converges in distribution to z.
269
Solution to Exercise 8.6.6. By assumption, {xn } is an IID sequence in RK with E [xn ] =

0 and var[xn ] = IK . It follows from the vector central limit theorem that
N x N z N (0, IK )
Letting g(s) := ksk2 and applying the continuous mapping theorem (fact 2.5.4 on
page 78), we obtain
K
d
y N = k N x N k2 kzk2 = z2k
k =1
From fact 1.3.4 on page 27 we conclude that y N 2 (K ).

Solution to Exercise 8.6.7. To simplify the algebra, Ill solve the case where = 0,
and leave further details to you. Since linear combinations of normals are normal,
we know that xt+1 is normal. Thus, it remains only to show that E [ xt+1 ] = 0 and
var[ xt+1 ] = 1/(1 $2 ). The first claim is true because, by linearity of expectations,
E [ xt+1 ] = E [$xt + wt+1 ] = $E [ xt ] + E [u] = 0

The second claim also holds, because, from the rule for variance of linear combinations,
var[ xt+1 ] = var[$xt + wt+1 ] = $2 var[ xt ] + var[wt+1 ] + 2$ cov[ xt , wt+1 ]
Since xt and wt+1 are independent (xt is a function of current and lagged shocks
only), the final term is zero, and we get
var[ xt+1 ] = $2
1
1
+1 =
2
1$
1 $2
Solution to Exercise 8.6.8. We need to show that

1. {dt } is adapted to {Ft }, and
2. E [dt+1 | Ft ] = 0
To prove that {dt } is adapted to {Ft }, we need to show that dt is a function of
variables in Ft . To see this, observe first that mt is a function of variables in Ft .
Second, mt1 is a function of variables in Ft1 , and, by the definition of filtrations,
270
every variable in Ft1 is also in Ft . Hence, mt1 is also is a function of variables in

Ft . Since both mt and mt1 are functions of variables in Ft , clearly dt = mt mt1
is also a function of variables in Ft .20
Finally, since E [mt+1 | Ft ] = mt (recall that {mt } is a martingale) and E [mt | Ft ] =
mt (by fact 8.1.2), we have
E [ d t +1 | F t ] = E [ m t +1 m t | F t ] = E [ m t +1 | F t ] E [ m t | F t ] = m t m t = 0
Solution to Exercise 8.6.9. Regarding part 1, {t } a martingale with respect to {Ft }.

Firstly, {t } adapted to {Ft }, because {mt } adapted to {Ft } by assumption, so mt
is a function of variables in Ft . Hence t = mt + 1 is a function of variables in Ft .21
Moreover, using the fact that {mt } is a martingale with respect to {Ft }, we have
E [ t +1 | F t ] = E [ m t +1 + 1 | F t ] = E [ m t +1 | F t ] + 1 = m t + 1 = t
Regarding part 2, {t } a martingale with respect to {Ft }. The proof is similar to part
1, and hence omitted.
Regarding part 3, {t } is not generally a martingale with respect to {Ft }. For example, we saw in the course notes that if { t } is an IID sequence of random variables
with E [ 1 ] = 0, mt := tj=1 j and Ft := { 1 , . . . , t }, then {mt } is a martingale with
respect to {Ft }. However, the process {t } given by t = m2t is not a martingale
whenever 2 := E [ 12 ] is strictly positive. To see this, observe that
!2
t + 1 =
m2t+1
j + t +1
= (mt + t+1 )2 = m2t + mt t+1 + t2+1
j =1
Since
and
20 The
E [t+1 | Ft ] = E [m2t | Ft ] + E [mt t+1 | Ft ] + E [ t2+1 | Ft ]

E [ m t t +1 | F t ] = m t E [ t +1 | F t ] = m t E [ t +1 ] = 0
E [ t2+1 | Ft ] = E [ t2+1 ] = 2
way to think about this intuitively is to think about whether or not dt can be computed on
the basis of information available at time t. Since both mt and mt1 can be computed at time t, their
difference dt = mt mt1 can also be computed.
21 Once again, the way to remember this is to recognize that since the value of m can be computed
t
at time t (by assumption), the value of t = mt + 1 can also be computed.
271
we now have
E [t+1 | Ft ] = E [m2t | Ft ] + 2 = m2t + 2 = t + 2 > t

Hence {t } is not a martingale with respect to {Ft }.
Solution to Exercise 8.6.12. It is clear from (8.6) on page 236 that { xt } is adapted to
the filtration for all values of $.
Regarding part 1, if $ = 1, then { xt } is the random walk in example 8.1.4. Hence
{ xt } is a martingale with respect to {Ft }. Regarding part 2, if $ = 0, then xt = wt ,
and
E [ x t +1 | F t ] = E [ w t +1 | F t ] = E [ w t +1 ] = 0
Hence { xt } is a martingale difference sequence with respect to {Ft }.
Solution to Exercise 8.6.14. As in the statement of the problem, we take { xt } to follow xt+1 = $xt + wt+1 with |$| < 1 and {wt } is IID and standard normal. Note that
x0 is assumed to be drawn from the stationary distribution = N (0, 1/(1 $2 ),
and hence xt has this distribution for all t. The claim is that
1 T
1
d
xt N (0, v) where v :=
(1 $ )2
T t =1
To see that this is so, we apply (8.18) on page 251. Note here that all the relevant
conditions are satisfied. See in particular example 8.2.1 on page 252. Observe also
that the stationary distribution = N (0, 1/(1 $2 )) has finite moments of all
orders.
Applying (8.18), we have
"
1
1 T
xt = T
T
T t =1
xt
t =1
where, from (8.19),
"
1
T
T
xt
s (s)ds N (0, v)
t =1
v = var[ x0 ] + 2 cov[ x0 , xt ]
t =1
Recall here that x0

v=
N (0, 1/(1 $2 ),
so this becomes
1
1
2
+
2
$t E [ x02 ] =
+
2
2
1$
1$
1 $2
t =1
2$
$t = 1 $2 + (1 $)(1 $2 )
t =1
Here the first equality follows from the discussion of correlation in 8.2.2. Simplifying the right hand side gives v = 1/(1 $)2 as claimed.
272
Solution to Exercise 8.6.15. In this exercise we are assuming that xt+1 = $xt + wt+1
where |$| < 1, {wt } is IID and standard normal, and x0 = N (0, 1/(1 $2 )).
(The stationary distribution was obtained on page 247.) The aim is to show that

2
1 + $2
1
d
2
2
N
(
0,
)
where
:
=
2
(8.31)
T sT
1 $2
(1 $2 )3
Letting v := 1/(1 $2 ), note from the manipulations in example 4.2.7 that it suffices
to prove that
(
)
1 T 2
d
T
xt v N (0, 2 )
(8.32)
T t =1
All of the conditions of theorem 8.2.2 on page 251 are satisfied, and we apply the
resulting CLT in (8.18) with h( x ) = x2 to get (8.32) with
2 := E [( x02 v)2 ] + 2 E [( x02 v)( xt2 v)]
(8.33)
t =1
To check (8.31) it only remains to show that this expression for 2 matches that in
(8.31). To see this, consider the first term on the right hand side of (8.33). We have
E [( x02 v)2 ] = E [ x04 ] + 2vE [ x02 ] + v2 = E [ x04 ] 2vE [ x02 ] + v2 = 2v2

where weve used the fact that the forth moment of x N (0, 2 ) is 34 . For the
second term on the right hand side of (8.33), we have
n
o
2
2
2 2
2
2 t
2
E [( x0 v)( xt v)] = E [ x0 xt ] v = E x0 [$ x0 + s(w)] v2
for s(w) := it=1 $ti wi . Note that s(w) and x0 is independent. (For a Markov
process, the initial condition is always assumed to be independent of the shocks.)
Working through all the algebra we get
E [( x02 v)( xt2 v)] = 2$2t v2

Returning to (8.33) we have
2 = 2v2 + 2 2$2t v2 = 2v2 + 4

t =1
$2
v2
2
1$
(8.34)
Using v = 1/(1 $2 ) and working through the remaining algebra will get you to
the expression for 2 in (8.31).
Chapter 9
Large Sample OLS
[roadmap]
9.1
Consistency
Lets return to the linear multivariate regression problem studied in chapters 6 and
7. Although the large sample theory we develop here has applications in a crosssectional environment with no correlation between observations, for additional generality we will imagine ourselves to be in a time series setting. To remind us of this,
observations will be indexed by t rather than n, and the sample size will be denoted
by T rather than N.
The only assumption we will retain from chapter 7 is the linear model assumption
(see 7.1.1). In particular, we assume that our data (y1 , x1 ), . . . , (y T , x T ) is generated
by the linear model
yt = x0t + ut
(9.1)
where is a K-vector of unknown coefficients, and ut is an unobservable shock. We
let y be the T-vector of observed outputs, so that yt is the t-th element of the T 1
vector y, and u be the vector of shocks, so that ut is the t-th element of the T 1
vector u. We let X be the T K matrix
x11 x12 x1K
x
21 x22 x2K
X := .
..
..
..
.
.
x T1 x T2
273
x TK
CHAPTER 9. LARGE SAMPLE OLS
274
We continue to assume that X is full column rank (i.e., rank(X) = K).

We will estimate the parameter vector via least squares. The expression for the
OLS estimate is unchanged:
T := (X0 X)1 X0 y
Notice also that we are subscripting with T to make the dependence of the estimator on T explicit. We can make this dependence clearer by rewriting the estimator
in a different way. Multiplying and dividing by T, we get

1
1
1
0
T =
XX
X0 y
T
T
Expanding out the matrix products (exercise 9.4.1), we obtain
"
# 1
T
1
1 T
0
T =
x
x
xt yt
t t
T t
T t
=1
=1
(9.2)
Also, taking our usual expression = (X0 X)1 X0 u for the sampling error and
performing a similar manipulation, we get
"
# 1
T
1
1 T
0
T =
x
x
(9.3)
t t
xt ut
T t
T
=1
t =1
9.1.1
Assumptions
Lets now study the properties of this estimator in the time series setting. In this setting, we abandon assumption 7.1.1, which is the exogeneity assumption E [u | X] =
0. The reason is that this assumption excludes too many models. For example, we
showed in 7.5.1 that the assumption fails when we try to estimate the simple AR(1)
model yt+1 = yt + ut+1 by setting xt = yt1 , thereby producing the regression
model
yt = xt + ut ,
t = 1, . . . , T
(9.4)
The problem is that for this specification of (9.1), the regressor is correlated with
lagged values of the shock.
We know that under assumption 7.1.1, the OLS estimator is unbiased for (theorem 7.2.1). In fact assumption 7.1.1 is close to the minimum requirement for unbiasedness, and without it there is little chance of establishing this property. Instead
To this end, we make the
we will aim for a large sample property: consistency of .
following assumptions:
275
Assumption 9.1.1 (Ergodic regressors). The sequence x1 , . . . , x T is identically distributed, and xx := E [x1 x10 ] is positive definite. Moreover,
1
T
xt x0t xx
as
(9.5)
t =1
Two remarks: First, although the observations x1 , . . . , x T in assumption 9.1.1 are

required to be identically distributed, they are not assumed to be IIDsome correlation is allowed. Second, we are implicity assuming that xx := E [x1 x10 ] exists. This
is a second moment assumption.
Example 9.1.1. Lets look at a scalar example. Let { xt } be the Markov process in
example 8.2.2 on page 252. To repeat,
xt+1 = $| xt | + (1 $2 )1/2 wt+1
with
IID
1 < $ < 1 and {wt } N (0, 1)
As discussed in example 8.2.2, the model has a unique, globally stable stationary
distribution, given by (s) = 2(s)(qs), where q := $(1 $2 )1/2 , is the standard normal density and is the standard normal cdf. Lets assume that x0 has
density .1 In this case, all of the conditions in assumption 9.1.1 are satisfied. Exercise 9.4.2 asks you to step through the details.
Assumption 9.1.2 (Weak exogeneity). The shocks {ut } are IID with E [ut ] = 0 and
E [u2t ] = 2 . Moreover, the shocks are independent of contemporaneous and lagged
regressors:
ut is independent of x1 , x2 , . . . , xt for all t
Remark: Assumption 9.1.2 permits dependence between current shocks and future
regressors. It is desirable to admit this possibility in a time series setting, because
current shocks usually feed into future state variables.
Example 9.1.2. For example, in the AR(1) regression (9.4), this will be the case whenever the shock process {ut } is IID, because the contemporaneous and lagged regressors x1 , . . . , xt are equal to the lagged state variables y0 , . . . , yt1 , which in turn are
functions of only y0 and u1 , . . . , ut1 , and therefore independent of ut .
1 In
the econometric setting, it is standard to assume that the first data point is drawn from the
stationary distribution. This seems justified when the process has been running for a long time, and
hence the distribtion of the state has converged to the stationary distribution by the time the first
data point is observed.

One consequence of assumption 9.1.2 is that we have
(
2 if s = t
E [ u s u t | x1 , . . . , x t ] =
0
if s < t
276
(9.6)
The proof is an exercise (exercise 9.4.3)

One of the most important consequences of assumptions 9.1.1 and 9.1.2 for us is that
linear functions of {xt ut } now form a martingale difference sequence. This allows
us to apply the LLN and CLT results in theorem 8.2.3 (page 255). From this LLN and
CLT, we will be able to show that the OLS estimator is consistent and asymptotically
normal.
Lemma 9.1.1. If assumptions 9.1.1 and 9.1.2 both hold, then, for any constant vector a
RK , the sequence {mt } defined by mt = a0 xt ut is
1. identically distributed with E [m21 ] = 2 a0 xx a, and
2. a martingale difference sequence with respect to the filtration defined by
F t : = { x 1 , . . . , x t , x t +1 , u 1 , . . . , u t }
(9.7)
Proof. First lets check part 1. That {mt } is identically distributed follows from the
assumption that {ut } and {xt } are identically distributed, and that xt and ut are
independent.2 Regarding the second moment E [m21 ], we have
E [m21 ] = E [E [u21 (a0 x1 )2 | x1 ]] = E [(a0 x1 )2 E [u21 | x1 ]]

From independence of u1 and x1 , the inner expectation is 2 . Moreover,
(a0 x1 )2 = a0 x1 a0 x1 = a0 x1 x10 a
E [m21 ] = E [a0 x1 x10 a 2 ] = 2 a0 E [x1 x10 ]a = 2 a0 xx a
To check part 2, note that {mt } is adapted to {Ft }, since mt := ut a0 xt is a function of

variables in Ft . Moreover, we have
E [ m t +1 | F t ] = E [ u t +1 a 0 x t +1 | F t ] = a 0 x t +1 E [ u t +1 | F t ] = a 0 x t +1 E [ u t +1 ] = 0
(Here the second equality follows from the fact that xt+1 Ft , while the third follows from the independence in assumption 9.1.2.) This confirms that {mt } is a martingale difference sequence with respect to {Ft }.
distribution of mt depends on the joint distribution of a0 xt and ut . Since a0 xt and ut are
independent, their joint distribution is just the product of their marginal distributions. Since {ut }
and {xt } are identically distributed, these marginal distributions do not dependend on t.
2 The
9.1.2
277
Consistency of T
Under the assumptions of the previous section, the OLS estimator is consistent for
the parameter vector . In particular, we have the following result:
p
Theorem 9.1.1. If assumptions 9.1.1 and 9.1.2 both hold, then T as T .
Proof. It suffices to show that the expression on the right-hand side of (9.3) converges
in probability to 0. As a first step, lets show that
1
T
xt ut 0
(9.8)
t =1
is true. In view of fact 2.5.2 on page 77, it suffices to show that, for any a RK , we
have
#
"
T
p
1
xt ut a0 0 = 0
(9.9)
a0
T t =1
If we define mt := a0 xt ut , then (9.9) can be written as T 1 tT=1 mt . Since {mt }
is an identically distributed martingale difference sequence (see lemma 9.1.1 on
p
page 276), the convergence T 1 tT=1 mt 0 follows from theorem 8.2.3 (page 255).
We have now verified (9.8).
Now let us return to the expression on the right-hand side of (9.3). By assumption 9.1.1 and fact 2.5.3, we see that
# 1
"
p
1 T
0
xt xt
xx 1 as T
T t =1
Appealing to fact 2.5.3 once more, we obtain
"
# 1
T
1
1
T =
xt x0t
T t =1
T
ut xt xx 1 0 = 0
t =1
The proof of theorem 9.1.1 is now done.
9.1.3
Consistency of T2
To estimate the variance 2 of the error terms, we previously used the expression
2 := SSR /( N K ), where N was the sample size. In the current setting T is the
278
sample size, and, since T is assumed to be large relative to K, we have 1/( T K )

1/T. Hence for our new expression we just take T2 = SSR /T. (None of the following
theory is affected if we use SSR /( T K ) instead.) In summary,
T2 :=
1
T
u 2t :=:
t =1
1
T
(yt x0t T )2
(9.10)
t =1
Theorem 9.1.2. If assumptions 9.1.1 and 9.1.2 both hold, then T2 2 as T .

Proof. We have
T2 =
1
T
1 T
0 2
)
=
(
y
x
t
[ut + x0t ( T )]2

t T
T
t =1
t =1
Expanding out the square, we get

1
T2 =
T
1
u2t + 2( T )0 T
t =1
xt ut + ( T )0
t =1
"
1
T
xt x0t
( T )
t =1
By assumption 9.1.2 and the law of large numbers, the first term on the right-hand
side converges in probability to 2 . Hence it suffices to show that the second and
third term converge in probability to zero as T (recall fact 1.4.1 on page 31).
These results follow from repeated applications of fact 2.5.3 on page 77, combined
with various convergence results we have already established. The details are left
as an exercise.
9.2
Asymptotic Normality
Under our assumptions, we will now show that the term T ( T ) is asymptotically normal. From this information we can develop asymptotic tests and confidence
intervals.
9.2.1
Asymptotic Normality of
Lets begin by establishing asymptotic normality of the OLS estimator:
279
Theorem 9.2.1. Under assumptions 9.1.1 and 9.1.2, the OLS estimator T :=: satisfies
d
1
T ( T ) N (0, 2
xx )
as
Proof. Using the expression (9.3) we obtain

"
# 1
T
T
1
1/2
0
T ( T ) =
T
x
x
t t
ut xt
T t
t =1
=1
(9.11)
Let z be a random variable satisfying z N (0, 2 xx ). Suppose we can show that

T
T 1/2
ut xt z
as
(9.12)
t =1
If (9.12) is valid, then, applying assumption 9.1.1 along with fact 2.5.5, we obtain
"
# 1
T
T
1
d
1
T ( T ) =
xt x0t
T 1/2 ut xt
xx z
T t =1
t =1
13
In view of fact 2.4.6 on page 75 and symmetry of
xx , we have
1
1
1
1 2
1
2 1
xx z N ( 0, xx var[ z ] xx ) = N ( 0, xx xx xx ) = N ( 0, xx )
This completes the proof of theorem 9.2.1, conditional on the assumption that (9.12)
is valid. Lets now check that (9.12) is valid.
By the Cramer-Wold device (fact 2.5.2 on page 77), it suffices to show that for any
a RK we have
"
#
a0 T 1/2
ut xt
a0 z
(9.13)
t =1
Fixing a and letting mt := ut a0 xt , the expression on the left of (9.13) can be rewritten
as follows:
"
#
a0 T 1/2
ut xt = T 1/2
t =1
Since z
N (0, 2
xx ),
ut a0 xt =: T 1/2
t =1
mt
t =1
to establish (9.13) we need to show that

T 1/2
mt N (0, 2 a0 xx a)
(9.14)
t =1
3 Remember
xx
that the transpose of the inverse is the inverse of the transpose, and the transpose of
is just xx , since all variance-covariance matrices are symmetric.
280
From lemma 9.1.1, we already know that {mt } is an identically distributed with
E [m21 ] = 2 a0 xx a, and a martingale difference sequence with respect to the filtration defined by
F t : = { x 1 , . . . , x t , x t +1 , u 1 , . . . , u t }
In view of the martingale difference CLT in theorem 8.2.3, the result (9.14) will hold
whenever
p
1 T
E [m2t | Ft1 ] 2 a0 xx a as T
(9.15)
T t =1
Since xt Ft1 , we have
E [m2t | Ft1 ] = E [u2t (a0 xt )2 | Ft1 ] = (a0 xt )2 E [u2t | Ft1 ] = 2 (a0 xt )2 = 2 a0 xt x0t a
The right-hand side of (9.15) is therefore
1
T
1
E [m2t | Ft1 ] = T
t =1
(2 a0 xt x0t a) = 2 a0
t =1
"
1
T
xt x0t
a 2 a0 xx a
t =1
where the convergence in probability is due to assumption 9.1.1 and (2.7). This
verifies (9.15), and completes the proof of theorem 9.2.1.
Example 9.2.1. Consider again the scalar linear Gaussian AR(1) model xt+1 = $xt +
wt+1 with |$| < 1 and {wt } IID and standard normal. As discussed in 7.5.1, the
OLS estimator of $ is
$ T :=
x0 y
x0 x
y := ( x1 , . . . , x T ) and x := ( x0 , . . . , x T 1 )
Both assumption 9.1.1 and assumption 9.1.2 are satisfied, so T ($ T $) converges

1
2
in distribution to N (0, 2
xx ). In this case, = 1 because the shocks are standard
1
2
normal. Furthermore,
xx reduces to 1/E [ x1 ], where the expectation is under the
stationary distribution. Referring to the discussion on page 243, we see that the
stationary distribution is N (0, 1/(1 $2 )). Hence the inverse of E [ x12 ] is 1 $2 , and
d
T ($ T $) N (0, 1 $2 )
(9.16)
9.2.2
where
Large Sample Tests
In 7.4.2 we considered the problem of testing a hypothesis about an individual

coefficient k . Lets consider this problem again in the large sample setting. The
hypothesis to be tested is
H0 : k = 0k
against
H1 : k 6= 0k
281
In the finite sample theory of 7.4.2, we showed that if the error terms are normally
distributed, then the expression ( k k )/ se( k ) has the t-distribution with N K
degrees of freedom. In the large sample case, we can use the central limit theorem
to show that the same statistic is asymptotically normal. (In a sense, this is not
surprising, because the t-distribution converges to the standard normal distribution
as the degrees of freedom converges to infinity. However, we cannot use this result
directly, as our model assumptions are quite different.)
Theorem 9.2.2. Let T be as defined in (9.10) on page 278. Let assumptions 9.1.1 and 9.1.2
hold, and let
q
T
se( k ) := T2 e0k (X0 X)1 ek

Under the null hypothesis H0 , we have
zkT :=
Tk 0k d
N (0, 1)
se( T )
as
(9.17)
Proof. Recall from theorem 9.2.1 that
d
1
T ( T ) z N (0, 2
xx )
as
where, as usual, is the true parameter vector. It now follows (from which facts?)
that
d
e0k [ T ( T )] e0k z N (e0k E [z], e0k var[z]ek )
d
1
In other words, we have T ( Tk k ) N (0, 2 e0k
xx ek ). Making the obvious
transformation, we obtain
T ( Tk k ) d
q
N (0, 1)
(9.18)
1
2 e0k
e
xx k
p
We have already shown that ( T 1 tT=1 xt x0t )1 xx 1 . Applying (2.7), we then

have
"
# 1
T
p
0
0
1
0 1
0
Tek (X X) ek = ek
xt xt
ek e0k xx 1 ek
T t =1
p
Recall that T2 2 , as shown in theorem 9.1.2. Using this result plus fact 1.4.1 on
page 31, we then have
q
q
p
1/ T2 Te0k (X0 X)1 ek 1/ 2 e0k xx 1 ek
282
In view of (9.18) and fact 1.4.5 on page 34, we then have
T ( Tk k )
d
q
N (0, 1)
T2 Te0k (X0 X)1 ek
Assuming validity of the null hypothesis (so that 0k = k ) and cancelling
have established (9.17).
9.3
T, we
Further Reading
To be written.
9.4
Exercises
Ex. 9.4.1. Verify expression (9.2). Recall here that xt is the t-th row of X.
Ex. 9.4.2. In example 9.1.1 it was claimed that the threshold process studied in that
example satisfies all of the conditions of assumption 9.1.1. Verify that this is the case.
Ex. 9.4.3. Verify the claim that (9.6) holds when assumption 9.1.2 is valid.
Ex. 9.4.4. Let K 1 random vector T be an estimator of . Suppose that this estimator is asymptotically normal, in the sense that
d
T ( T ) N (0, C)
where C is symmetric and positive definite. It is know that for such a C there exists
T be a consistent estimator of Q. Show
a K K matrix Q such that QCQ0 = I. Let Q
that
d
T ( T )k2
T kQ
2 ( K )
(9.19)
Ex. 9.4.5. Consider once more the scalar linear Gaussian AR(1) model xt+1 = $xt +
wt+1 with |$| < 1 and {wt } IID and standard normal. In example 9.2.1 we saw that
the OLS estimator $ T is a consistent and asymptotically normal estimator of $, with
asymptotic variance 1 $2 . Lets now look at another consistent and asymptotically
normal estimate of $ and compare its performance with the OLS estimator in terms
of asymptotic variance (with small variance being better).
283
For this second estimator, we can exploit the fact that the sample variance converges
to the stationary variance of xt , which in this case is 1/(1 $2 ). Once we estimate
this quantity with the sample variance, we can solve for the value of $. The exercise
is to formalize this idea, and show that

1 $2
d
2
(9.20)
T ($ T $) N 0, (1 $ ) 1 +
2$2
where $T is the estimator just described. Comments:
The exercise shows that the OLS estimator has lower asymptotic variance than
$T (cf. (9.16) on page 280). We say that the OLS estimator is relatively more
efficient.
In attempting the exercise, you should first consult exercise 8.6.15 on page 266.
Ex. 9.4.6. To be written. Derive ols estimator of $ in the model of example 8.2.2. (see
zhao.) is it consistent?
9.4.1
Solution to Exercise 9.4.2. The process { xt } studied in example 9.4.2 is identically

distributed as a result of our assumption that x0 (see fact 8.2.1 on page 250).
It remains to check the conditions on xx . In the present case,
xx =
E [ xt2 ]
s (s)ds =
s2 2(s)(qs)ds
where q := $(1 $2 )1/2 , is the standard normal density and is the standard
normal cdf. To verify that xx is positive definite, we need to check that the term
on the right-hand side is strictly positive. This is clearly true, because the function
inside the integral is strictly positive everywhere but zero. To be careful, we should
also check that xx is finite, and this is also true because (qs) 1, and hence
Z
s 2(s)(qs)ds
2
s 2(s)ds = 2
s2 (s)ds = 2
Finally, we need to show that

1
T
xt2 xx = E [xt2 ]
t =1
(9.21)
284
Since the conditions of theorem 8.2.1 (page 251) are satisfied, we can appeal to
theorem 8.2.2 (page 251). This theorem confirms that the convergence in (9.21) is
valid.
Solution to Exercise 9.4.3. Suppose that assumption 9.1.2 (page 275) is valid. We
need to show that
(
2 if s = t
E [ u s u t | x1 , . . . , x t ] =
0
if s < t
On one hand, if s = t, then E [u2t | x1 , . . . , xt ] = E [u2t ] = 2 by independence. On the
other hand, if s < t, then
E [ u s u t | x1 , . . . , x t ] = E [E [ u s u t | x1 , . . . , x t , u s ] | x1 , . . . , x t ]
= E [ u s E [ u t | x1 , . . . , x t , u s ] | x1 , . . . , x t ]
= E [ u s E [ u t ] | x1 , . . . , x t ]
= E [ u s 0 | x1 , . . . , x t ] = 0
Solution to Exercises 9.4.4. By assumption we have

p
T
Q
Q
and
d
T ( T ) z
where z N (0, C). Applying Slutskys theorem, we obtain
d
T T ( T )
Q
Qz
(9.22)
Clearly Qz is normally distributed with mean 0. Moreover,

var[Qz] = Q var[z]Q0 = QCQ0 = I
In other words, Qz is standard normal. As a result, kQzk2 2 (K ). Applying the
continuous mapping theorem to (9.22), we obtain
d
T T ( T )k2
kQ
kQzk2 2 (K )
This is equivalent to (9.19).
285
Incidentally, it should be clear that (9.19) can be used to test the null hypothesis that
= 0 . Under the null hypothesis, we have
d
T ( T 0 )k2
T kQ
2 ( K )
Fixing and letting c be the 1 quantile of the 2 (K ) distribution, we reject the

null if
T ( T 0 )k2 > c
T kQ
This test is asymptotically of size .
Solution to Exercise 9.4.5. Let v := 1/(1 $2 ), the asymptotic variance of xt . We
can solve this for $ to get
r
1
$ = g(v) for g(v) := 1
v
We take as our estimator $T the statistic g(s2T ), where s2T is the sample variance. We
then have
T ($T $) = T { g(s2T ) g(v)}

Applying the delta method (see page 37) combined with the result in exercise 8.6.15
we have
T ($T $) N (0, g0 (v)2 2 )
where
2 : = 2
1 + $2
(1 $2 )3
Using the definition of v and a bit of algebra leads to the expression on the right
hand size of (9.20).
Chapter 10
Further Topics
[roadmap]
10.1
Model Selection
[roadmap]
10.1.1
Ridge Regression
We are going to begin our discussion of model selection by introducing ridge regression. Ridge regression is an important method in its own right, with connections to many areas of statistics and approximation theory. Moreover, it immediately presents us will a class of models we need to choose between, and hence a
model selection problem.
Lets begin by putting ourselves in the classical OLS setting of chapter 7. In particular, we will assume that y = X + u, where is unknown, and u is unobservable,
has unknown distribution, but satisfies E [u | X] = 0 and E [uu0 | X] = 2 I for some
unknown > 0. In traditional OLS theory, we estimate with the OLS estimator
:= (X0 X)1 X0 y = argmin ky Xbk2 = argmin
b
RK
RK
(yn x0n b)2
n =1
In many ways, is a natural choice for estimating . Firstly, it minimizes the empirical risk corresponding to the natural risk function R( f ) = E [(y f (x))2 ] when
286
CHAPTER 10. FURTHER TOPICS
287
the hypothesis space F is the set of linear functions. Second, under our current assumptions, it is unbiased for (theorem 7.2.1 on page 205), and, moreover, it has the
lowest variance among all linear unbiased estimators of (see the Gauss-Markov
theorem on page 207).
These results, and, in particular, the Gauss-Markov theorem are much celebrated
foundation stones of standard OLS theory. But prehaps some of this celebration is
misplaced. Rather than looking at whether an estimator is best linear unbiased, a
better way to evaluate the estimator is to consider its mean squared error, which
tells us directly how much probability mass the estimator puts around the object
its trying to estimate. (This point was illustrated in figure 4.4 on page 120.) In the
vector case, the mean squared error of a estimator b of is defined as
mse(b ) := E [kb k2 ]
It is an exercise (exercise 10.5.1) to show that the mean squared error can also be
expressed as
mse(b ) = E [kb E [b ]k2 ] + kE [b ] k2
(10.1)
This equation is analogous to (4.11) on page 120, and tells us that the mean squared
error is the sum of variance and bias. To minimize mean squared error we
face a trade off between these two terms. In many situations involving trade off,
the optimal choice is not at either extreme, but somewhere in the middle. Many
estimation techniques exhibit this property: Mean squared error is at its minimum
not when bias is zero, but rather when some small amount of bias is admitted.
Applying this idea to the OLS setting, it turns out that we can find a (biased) linear
The estimator is defined as the
estimator that has lower mean squared error that .
solution to the modified least squares problem
(
)
N
min
RK
(yn x0n b)2 + kbk2
(10.2)
n =1
where 0 is called the regularization parameter. In solving (10.2), we are minimizing the empirical risk plus a term that penalizes large values of kbk. The effect
A bit of calculus
is to shrink the solution relative to the unpenalized solution .
shows that the solution to (10.2) is
:= (X0 X + I)1 X0 y
(10.3)
Minimizing the objective in (10.2) is certainly a less obvious approach than simply
minimizing the empirical risk nN=1 (yn x0n b)2 = ky Xbk2 . One indication as to
288
10
6
12
14
10
15
20
Figure 10.1: Effect of Tikhonov regularization, = 1
why it might be a good idea comes from regularization theory. To illustrate regularization, suppose that Ab = c is an overdetermined system, where A is N K
with N > K. Let b be the least squares solution: b = argminb kAb ck2 . Suppose in addition that c cannot be calculated perfectly, due to some form of measurement error. Instead we observe c0 c. In the absence of additional information,
you might guess that the best way to compute an approximation to b is to solve
minb kAb c0 k2 , obtaining the least squares solution to the system Ab = c0 . Surprisingly, it turns out that this is not always the case, especially when the columns
of A are almost linearly dependent. Instead, one often does better by minimizing
kAb c0 k2 + kbk2 for some small but positive . This second approach is called
Tikhonov regularization.
While the theory of Tikhonov regularization is too deep to treat in detail here, we can
illustrate the rather surprising benefits of regularization with a simulation. In our
simulation, A will be chosen fairly arbitrarily, but such that the columns are quite
close to being linearly dependent. To simplify, we first set b := (10, 10, . . . , 10), and
then set c := Ab . By construction, b is then a solution to the system Ab = c, and
also the least squares solution (because it solves minb kAb ck2 ).
289
10
6
12
14
10
15
20
Figure 10.2: Effect of Tikhonov regularization, = 50
When measuring c we will corrupt it with a Gaussian shock. In particular, we draw

c0 N (c, 2 I) where is a small positive number. We then plot the ordinary least
squares solution based on c0 , which minimizes kAb c0 k2 , and the regularized
solution, which minimizes kAb c0 k2 + kbk2 . The former are plotted against their
index in black, while the latter are plotted in blue. The true solution b is plotted in
red. The result is figure 10.1. The figure shows 20 solutions each for the ordinary and
regularized solutions, corresponding to 20 draws of c0 . The regularized solutions
are clearly better on average.
Of course, this result is dependent on a reasonable choice for . If you experiment
with the code (listing 17), you will see that for very small values of , the regularized solutions are almost the same as the unregularized solutions. Conversely, very
large values of pull the regularized solutions too close to the zero vector. Such
a situation is depicted in figure 10.2, where the value of has been increased by a
factor of 50. We can see from the figure that the variance of the regularized solutions
has fallen even further, but they are now substantially biased.
Tikhonov regularization gives some understanding as to why the ridge regression
While is the solution to the least squares
estimator can perform better than .
Listing 17 Effect of Tikhonov regularization
sigma <- 0.5

lambda <- 1
numreps <- 20
# Parameterizes measurement error for c

# Regularization parameter
# Number of times to solve system
# Construct an arbitrary N x K matrix A

N <- 40; K <- 20
A <- matrix ( nrow =N , ncol = K )
A [ ,1] <- 1
for ( i in 2: K ) {
A [ , i ] <- A [ ,i -1] + rnorm (N , sd =0.1)
}
Ap <- t ( A )
bstar <- rep (10 , K )
c <- A % * % bstar
# A transpose
# True solution
# Corresponding c
# Create empty plot

plot ( bstar , type = " n " , ylim = c (6 , 14) , xlab = " " , ylab = " b " )
# Plot the solutions
for ( j in 1: numreps ) {
# Observe c with error
c0 <- c + rnorm (N , sd = sigma )
# Compute the regularized solution
b1 <- solve (( Ap % * % A + lambda * diag ( K ) ) , Ap % * % c0 )
lines ( b1 , col = " blue " )
# Compute the standard least squares solution
b2 <- lm ( c0 ~ 0 + A ) $ coefficients
lines ( b2 , col = " black " , lty =2)
}
lines ( bstar , lwd =2 , col = " red " )
290
291
problem minb ky Xbk2 , the ridge regression estimator is the solution to the regularized problem minb ky Xbk2 + kbk2 . Since y is indeed a noisy observation,
we can expect that the regularized estimator will sometimes perform better.
in the sense that there always
In fact, it turns out that can always outperform ,
exists a > 0 such that mse( ) < mse( ). This was proved by Hoerl and Kennard
(1970), and the details of the argument can be found there. As mentioned above, this
implies that the estimator is biased (see exercise 10.5.2). The reduction in mean
squared error over the least squares estimator occurs because, for some intermediate
value of , the variance of falls by more than enough to offset the extra bias.
It is worth emphasizing two things before we move on. One is that, with the right
choice of , the ridge regression estimator outperforms even though all of the
classical OLS assumptions are completely valid. The other is that the right choice of
is an important and nontrivial problem. This problem falls under the heading of
model selection, which is the topic treated in the next few sections.
10.1.2
Subset Selection and Ridge Regression
One problem frequently faced in specific regression problems is which variables to

include. For example, if we are comparing crime rates across different cites, we can
think of any number of variables that might be relevant (median wage, unemployment, police density, population, etc., etc.). The same is true if we are trying to model
credit default rates for some group of firms or individuals, educational attainment
across schools, adoption of technologies, demand for certain products, and so on.
And a similar problem of variable selection arises in time series models, where we
want to know how many lags of the state variables to include. The general problem
is known as subset selection, since we are trying to choose the right subset of all
candidate regressors.
This problem takes on another dimension (no pun intended) when we start to think
about basis functions. Given a set of covariates x, we have the option to map this into
a larger vector (x) using basis functions, as discussed in 6.2.1. For example, given
a single covariate x, we may consider mapping it into ( x ) = (1, x, x2 , . . . , x d ) and
regressing y on ( x ). This amounts to polynomial regression, where the hypothesis
space is
n
o
F := all functions of the form f ( x ) = b0 x0 + b1 x1 + + bd x d
292
Polynomial regression was discussed extensively in 4.6.2. As we saw in figures 4.18

4.21 (page 148), a good choice of d is crucial. Choosing d is another example of the
subset selection problem because we are trying to decide whether to include the
regressor x j for some given j.
Subset selection can be viewed from the lens of empirical risk minimization. Suppose that we are trying to model a given system with output y and inputs x. We
imagine that x is a large set of K candidate regressors. We will also suppose that
we have already applied any transformations we think might be necessary. For example, x might include not just the original regressors but squares of the regressors,
cross-products, and so on. (In other words, we have already applied the basis functions.) If we want to include all regressors then we can minimize empirical risk (i.e.,
solve the least squares problem) over the hypothesis space of linear functions from
RK to R:
L :=
all functions of the form f (x) = b0 x with b1 , . . . , bK R
Doing so produces the usual OLS estimator.

Now suppose that we wish to exclude some subset of regressors { xi , . . . , x j }. Let
I {1, . . . , K } be the set of indices of the regressors we want to exclude. Regressing
y on the remaining regressors { xk }k/ I is equivalent to minimizing the empirical risk
over the hypothesis space
L I :=
all functions f (x) = b0 x with bk = 0 for all k I
Once again, we are back to the problem of choosing a suitable hypothesis space over
which to minimize empirical risk.
The subset selection problem has been tackled by many researchers. Well-known
approaches include those based on the Akaike Information Criterion (AIC), the
Bayesian Information Criterion (BIC) and Mallows C p statistic. For example, Mallows C p statistic consists of two terms, one increasing in the size of the empirical
risk, and the other increasing in #I, the size of the subset selected. The objective is
to minimize the statistic, which involves trading off poor fit (large empirical risk)
against excess complexity of the hypothesis space (large #I).
One of the problems with subset selection is that there is usually a large number
of possible subsets. With K regressors, there are 2K subsets to step through. To
avoid this problem, one alternative is to use ridge regression. With ridge regression,
the regularization term leads us to choose an estimate with smaller norm. What
293
this means in practice is that the coefficients of less helpful regressors are driven
towards zero. The effect is to almost exclude those regressors. Of course, the
model selection problem is not solved, because we still need to choose the value of
the regularization parameter . However, the problem has been reduced to tuning
a single parameter, rather than searching over 2K subsets.
We can illustrate the idea by reconsidering the regression problem discussed in
4.6.2. Figures 4.184.21 (see page 148) showed the fit we obtained by minimizing empirical risk over larger and larger hypothesis spaces. The hypothesis spaces
were the sets Pd of degree d polynomials for different values of d. For each d we
minimized the empirical risk over Pd , which translates into solving
N
min
b
(yn b0 (xn ))2
where
( x ) = ( x0 , x1 , . . . , x d )
n =1
As discussed above, choosing the right d is a essentially a subset selection problem,

because we are deciding what powers of x to include as regressors. Figure 4.17
(page 147) showed that intermediate values of d did best at minimizing risk.
We can do a very similar thing using ridge regression. First, lets take as our hypothesis space the relatively large space P14 . This space is certainly large enough
to provide a good fit to the data, but with empirical risk minimization the result
is overfitting (see figure 4.21 on page 150). Here, instead of using empirical risk
minimization, we solve the regularized problem
o
N n
min (yn b0 ( xn ))2 + kbk2
b
n =1
for different values of . The data used here is exactly the same data used in the
original figures 4.184.21 from 4.6.2. The solution for each we denote by ,
which is the ridge regression estimator, and the resulting prediction function we
0
denote by f , so that f ( x ) = ( x ).
The function f is plotted in red for increasingly larger values of over figures 10.3
10.6. The black line is the risk minimizing function. In figure 10.3, the value of is
too small to impose any real restriction, and the procedure overfits. In figure 10.4,
the value of is a bit larger, and the fit is good. In figures 10.5 and 10.6, the value of
is too large, and all coefficients are shrunk towards zero.
As in 4.6.2, we can compute the risk of each function f , since we know the underlying model (see (4.31) on page 146). The risk is plotted against in figure 10.7. The
x-axis is on log-scale. On the basis of what weve seen so far, its not surprising that
risk is smallest for small but nonzero values of .
294
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
lambda = 2.78946809286892e10
Figure 10.3: Fitted polynomial, 0
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
lambda = 0.000418942123448384
Figure 10.4: Fitted polynomial, 0.0004
295
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
lambda = 17.973328138195
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
lambda = 22026.4657948067
296
Risk
0.6
0.7
0.5
risk
0.4
0.3
1e10
1e07
1e04
1e01
1e+02
lambda
Figure 10.7: The risk of f plotted against
10.1.3
A Bayesian Perspective
The ideal case with model selection is that we have clear guidance from theory
on which regressors to include, which to exclude, which functional forms to use,
which values of our regularization parameter to choose, and so on. If theory or
prior knowledge provides this information then every effort should be made to exploit it. One technique for injecting prior information into statistical estimation is
via Bayesian analysis. Bayesian methods are currently very popular in econometrics and other fields of statistics (such as machine learning), and perhaps a future
version of these notes will give them more attention. Nevertheless, the brief treatment we present in this section does provide useful intuition on their strengths and
weaknesses. In what follows, we focus on Bayesian linear regression.
To begin, lets recall Bayes formula, which states that for any sets A and B we have
P( A | B ) =
P( B | A )P( A )
P( B )
This formula follows easily from the definition of conditional probability on page 6.
An analogous statement holds true for densities, although the derivation is a bit
297
more involved.
The main idea of Bayesian analysis is to treat parameters as random variables, in the
sense of being unknown quantities for which we hold subjective beliefs regarding
their likely values. These subjective beliefs are called priors. Suppose for example
that we observe input-output pairs (y1 , x1 ), . . . , (y N , x N ). We assume that the pairs
satisfy y = X + u. To simplify the presentation we will assume that X is nonrandom. (Taking X to be random leads to the same conclusions but with a longer
derivation. See, for example, Bishop, 2006, chapter 3). As before, u is random and
unobservable. The new feature provided by the Bayesian perspective is that we take
to be random (and unobservable) as well. While u and are unobservable random
quantities, lets suppose that we have subjective prior beliefs regarding their likely
values, expressed in the form of probability distributions. Here we will take the
priors to be u N (0, I) and N (0, I).
Given our model y = X + u, our prior on u implies that the density of y given is
N (X, I). In generic notation, we can write our distributions as
p(y | ) = N (X, I)
and
p( ) = N (0, I)
(10.4)
Applying Bayes formula to the pair (y, ) , we obtain

p( | y) =
p(y | ) p( )
p(y)
(10.5)
The left-hand side is called the posterior density of given the data y, and represents our new beliefs updated from the prior on the basis of the data y.
Often we wish to summarize the information contained in the posterior, by looking
at the most likely value of given our priors and the information contained in
the data. We can do this by looking either at the mean of the posterior, or at its
maximum value. The maximizer of the posterior is called the maximum a posteriori
probability (MAP) estimate. Taking logs of (10.5) and dropping the term that does
not contain , it can be expressed as
M := argmax {ln p(y | ) + ln p( )}
(10.6)
Inserting the distributions in (10.4), dropping constant terms and multiplying by

1, we obtain (exercise 10.5.3) the expression
(
)
N
2
M = argmin (yn x0n )2 + 2 k k2

(10.7)
n =1
298
This is exactly equivalent to the penalized least squares problem (10.2) on page 287,
where the regularization parameter is equal to (/ )2 . In view of (10.3), the solution is
M := (X0 X + (/ )2 I)1 X0 y
Thus, Bayesian estimation provides a principled derivation of the penalized least
squares method commonly known as ridge regression. Previously, we justified
ridge regression via Tikhonov regularization. Here, Bayesian analysis provides the
same regularization, where regularization arises out of combining prior knowledge
with the data. Moreover, at least in principle, the value (/ )2 is part of our prior
knowledge, and hence there is no model selection problem.
In practice one can of course question the assertion that we have so much prior
knowledge that the regularization parameter := (/ )2 is pinned down. If not,
then we are back at the model selection problem. In the next section we forgo the
assumption that this strong prior knowledge is available, and consider a more automated approach to choosing .
10.1.4
Cross-Validation
The most natural way to think about model selection is to think about minimizing
risk. Recall that, given loss function L and a system producing input-output pairs
(y, x) RK+1 with joint density p, the risk of a function f : RK R is the expected
loss
Z Z
R( f ) := E [ L(y, f (x))] =
L(t, f (s)) p(t, s) dt ds
that occurs when we use f (x) to predict y. Now suppose that we observe N
input-output pairs
D := {(y1 , x1 ), . . . (y N , x N )}
IID
Intuitively, given a selection of models (or estimation procedures), we would like

to find the one that takes this data set D and returns a predictor f such that f has
lower risk than the predictors returned by the other models. Here we have to be a
bit careful in defining risk, because if we simply define the risk as E [ L(y, f(x))] then
we are taking expectation over all randomness, including that in f, which depends
implicitly on the data set D . Thats not a bad idea per-se, and we discuss it further
in 10.1.5. But what we want to do for now is just take the data set as given, and
see how well we can do in terms of predicting new values as evaluated by expected
299
loss. Hence we define the risk of f as the expected loss taking D (and hence f) as
given:
Z Z
R( f | D) := E [ L(y, f (x)) | D] =
L(t, f(s)) p(t, s) dt ds
If we have a collection of models M indexed by m, and fm is the predictor produced
by fitting model m with data D , then we would like to find the model m such that
R( fm | D) R( fm | D)
for all
mM
The obvious problem with this idea is that risk is unobservable. If we knew the joint
density p then we could calculate it, but then again, if we knew p there would be no
need to estimate anything in the first place.
Looking at this problem, you might have the following idea: Although we dont
know p, we do have the data D , which consists of IID draws from p. From the law
of large numbers, we know that expectations can be approximated by averages over
IID draws, so we could approximate R ( f | D) by
1
N
L(yn , f(xn ))
n =1
where the pairs (yn , xn ) are from the training data D . However, if you think about
it for a moment more, you will realize that this is just the empirical risk, and the
empirical risk is a very biased estimator of the risk. This point was discussed extensively in 4.6.2. See, in particular, figure 4.17 on page 147. The point that figure
made was that complex models tend to overfit, producing low empirical risk, but
high risk. In essence, the problem is that we are using the data D twice, for conflicting objectives. First, we are using it to fit the model, producing f. Second, we are
using it to evalute the predictive ability of f on new observations.
So what we really need is fresh data. New data will tell us how f performs out of
sample. If we had J new observations (yvj , xvj ), then we could estimate the risk by
1
J
L(yvj , f(xvj ))
j =1
Of course, this is not really a solution, because we dont have any new data in general. One way that statisticians try to work around this problem is to take D and
split it into two disjoint subsets, called the training set and the validation set. The
training set is used to fit f and the validation set is used to estimate the risk of f. We
then repeat this for all models, and choose the one with lowest estimated risk.
300
Since data is scarce, a more common procedure is cross-validation, which attempts

to use the whole data set for both fitting the model and estimating the risk. To
illustrate the idea, suppose that we partition the data set D into two subsets D1 and
D2 . First, we use D1 as the training set and D2 as the validation set. Next, we use
D2 as the training set, and D1 as the validation set. The estimate of the risk is the
average of the estimates of the risk produced in these two steps.
Of course, we could divide the data into more than two sets. The extreme is to
partition D into N subsets, each with one element (yn , xn ). This procedure is called
leave-one-out cross validation. Letting Dn := D \ {(yn , xn )}, the data set D with
just the n-th data point (yn , xn ) omitted, the algorithm can be expressed as follows:
Algorithm 1: Leave-one-out cross-validation
1
2
3
4
5
for n = 1, . . . , N do
fit fn using data Dn ;
set rn := L(yn , fn (xn )) ;
end
return r := N1 nN=1 rn
At each step inside the loop, we fit the model using all but the n-th data point, and
then try to predict the n-th data point using the fitted model. The prediction quality
is evaluated in terms of loss. Repeating this n times, we then return an estimate of
the risk, using the average loss. On an intuitive level, the procedure is attractive
because we are using the available data quite intensively, but still evaluating based
on out-of-sample prediction.
In terms of model selection, the idea is to run each model through the cross-validation
procedure, and then select the one that produces the lowest value of r, the estimated
risk. Lets illustrate this idea, by considering again the ridge regression procedure
used in 10.1.2. In this problem, the set of models is indexed by , the regularization parameter in the ridge regression. The data set D is the set of points shown in
figures 10.310.6. For each , the fitted function f is
0
f ( x ) = ( x )
where
:= argmin
b
(yn b0 ( xn ))2 + kbk2
n =1
Recall here that ( x ) = ( x0 , x1 , . . . , x d ) with d fixed at 14, so we are fitting a polynomial of degree 14 to the data by minimizing regularized least squares error. The
amount of regularization is increasing in . The resulting functions f were shown
301
for different values of in figures 10.310.6. Intermediate values of produced the

best fit in terms of minimizing risk (see figures 10.4 and 10.7).
In that discussion, we used the fact that we knew the underlying model to evaluate
the risk. Since we could evaluate risk, we were able to determine which values of
produced low risk (figure 10.7). In real estimation problems, risk is unobservable,
and we need to choose on the basis of the data alone (assuming we dont have
prior knowledge, as in the Bayesian casesee 10.1.3). Lets see how a data-based
procedure such as cross-validation performs in terms of selecting a good value of .
In this experiment, for each in the grid
lambda <- exp ( seq ( -22 , 10 , length =10) )
we perform leave-one-out cross-validation as in algorithm 1. The fit at each step
within the loop is via ridge regression, omitting the n-th data point, and the resulting polynomial is used to predict yn from xn . The prediction error is measured by
squared loss. In other words, for each in the grid, we use the following algorithm
to estimate the risk:
Algorithm 2: Leave-one-out cross-validation for ridge regression
1
2
3
4
5
for n = 1, . . . , N do
set ,n := argminb i6=n {(yi b0 ( xi ))2 + kbk2 } ;
0
set r,n := (yn
( xn ))2 ;
,n
end
return r :=
1
N
nN=1 r,n
The value of producing the smallest estimated risk r is around 0.015. This is
in fact very close to the value that minimizes the actual risk (see figure 10.7 on
page 296). The associated function f is plotted in red in figure 10.8, and indeed
the fit is excellent. In this instance, our fully automated procedure is very successful.
10.1.5
The Theory of Model Selection
Generalization error.
Minimizer is regression function.
Variance-bias interpretation?
302
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
lambda = 0.0146660171165271
Figure 10.8: Fitted polynomial, 0.015
Best is prior knowledge: choose the regression function.

Is there one algorithm that outperforms all others over all possible specifications of
the DGP?
10.2
Method of Moments
- IV, method of moments, GMM, simulated method of moments.

Change following to vector case.
GMM is a generalization of the plug in estimator (which is a special case of ERM).
Suppose we want to estimate unknown quantity
:=
h(s) F (ds) = E [h( x )]
The plug in estimator is

:=
1
h(s) FN (ds) =
N
h( xn )
n =1
303
The method of moments is a generalization of this idea. We want to estimate

where
g( ) = E [h( x )]
(10.8)
The method of moments estimator is the solution to
1
g() =
N
h( xn )
(10.9)
n =1
We can express the same thing slightly differently, replacing (10.8) with
E [ g( ) h( x)] = 0
(10.10)
and (10.9) with

1
N
[ g() h(xn )] = 0
(10.11)
n =1
Generalized method of moments extends this idea further, by replacing (10.10) with
the more general expression
E [G(, x)] = 0
(10.12)
and (10.11) with the empirical counterpart
1
N
10.3
G(, xn ) = 0
(10.13)
n =1
Bayesian Methods
To be written
10.4
Further Reading
To be written.
10.5
Exercises
Ex. 10.5.1. Verify the claim in (10.1) that mse(b ) = E [kb E [b ]k2 ] + kE [b ] k2 .
304
Ex. 10.5.2. Derive the expectation of the ridge regression estimator . In particular,
show that is a biased estimator of when > 0.
Ex. 10.5.3. Verify (10.7) on page 297 using (10.4) and (10.6).
Part IV
Appendices
305
Chapter 11
Appendix A: Analysis
11.1
Sets and Functions
In the course we often refer to the real numbers. This set is denoted by R, and we
understand it to contain all of the numbers. R can be visualized as the continuous real line:
It contains both the rational and the irrational numbers.

Whats real about the real numbers? Well, real is in contrast to imaginary,
where the latter refers to the set of imaginary numbers. Actually, the imaginary
numbers are no more imaginary (or less real) than any other kind of numbers, but
we dont need to talk any more about this.
R is an example of a set. A set is a collection of objects viewed as a whole. (In this

case the objects are numbers.) Other examples of sets are the set of all rectangles in
the plane, or the set of all monkeys in Japan.
If A is a set, then the statement x A means that x is contained in (alternatively, is
an element of) A. If B is another set, then A B means that any element of A is also
an element of B, and we say that A is a subset of B. The statement A = B means
that A and B contain the same elements (each element of A is an element of B and
306
CHAPTER 11. APPENDIX A: ANALYSIS
307
Figure 11.1: Sets A and B in S
vice versa). For example, if I is the irrational numbers,1 then I R. Also, 0 R,

R, 3 R, e R, and so on.
Commonly used subsets of R include the intervals. For arbitrary a and b in R, the
open inverval ( a, b) is defined as
( a, b) := { x R : a < x < b}
while the closed inverval [ a, b] is defined as
[ a, b] := { x R : a x b}
We also use half open intervals such as [ a, b) := { x R : a x < b}, half lines such
as (, b) = { x R : x < b}, and so on.
Let S be a set and let A and B be two subsets of S, as illustrated in figure 11.1. The
union of A and B is the set of elements of S that are in A or B or both:
A B := { x S : x A or x B}
Here and below, or is used in the mathematical sense. It means and/or. The
intersection of A and B is the set of all elements of S that are in both A and B:
A B := { x S : x A and x B}
The set A \ B is all points in A that are not points in B:
A \ B := { x S : x A and x
/ B}
The complement of A is the set of elements of S that are not contained in A:
Ac := S \ A :=: { x S : x
/ A}
308
Figure 11.2: Unions, intersection and complements
Here x
/ A means that x is not an element of A. Figure 11.2 illustrate these definitions.
For example, since R consists of the irrationals I and the rationals Q, we have
Q R, I R, Q I = R, Qc = I, etc.
Also,
N := {1, 2, 3, . . .} Q R
The empty set is, unsurprisingly, the set containing no elements. It is denoted by .
If the intersection of A and B equals , then A and B are said to be disjoint.
The next fact lists some well known rules for set theoretic operations.
Fact 11.1.1. Let A and B be subsets of S. The following statements are true:
1. A B = B A and A B = B A.
2. ( A B)c = Bc Ac and ( A B)c = Bc Ac .
1 The
irrationals are those numbers such as and

whole numbers.
2 that cannot be expressed as fractions of
309
3. A \ B = A Bc .
4. ( Ac )c = A.
11.1.1
Functions
Let A and B be sets. A function f from A to B is a rule that associates to each element
a of A one and only one element of B. That element of B is usually called the image
of a under f , and written f ( a). If we write f : A B, this means that f is a function
from A to B.
For an example of a function, think of the hands on an old school clock. Lets say we
know its the morning. Each position of the two hands is associated with one and
only one time in the morning. If we dont know its morning, however, one position
of the hands is associated with two different times, am and pm. The relationship is
no longer functional.
Figure 11.3 is instructive. Top right is not a function because the middle point on
the left-hand side is associated with two different points (images). Bottom right is
not a function because the top point on the left-hand side is not associated with any
image. From the definition, this is not allowed.
11.1.2
Convergence and Continuity
Let { xn }
n=1 be a sequence of real numbers. (For each n = 1, 2, . . . we have a corresponding xn R.) We say that xn converges to 0 if, given any neighborhood of 0,
the sequence points are eventually in that neighborhood. More formally (we wont
use the formal definition, so feel free to skip this), given any e > 0, there exists an
N N such that | xn | < e whenever n N. Symbolically, xn 0.
N
Now let {xn }
n=1 be a sequence of vectors in R . We say that xn converges to x R
if kxn xk 0. Symbolically, xn x. This is the fundamental notion of convergence in R N . Whole branches of mathematics are built on this idea.
Let A R N and B R M . A function f : A B is called continuous at x if

f (xn ) f (x) whenever xn x, and continuous if it is continuous at x for all
x A. Figure 11.4 illustrates. The notion of continuity is also massively important
to mathematical analysis. However, we wont be doing any formal proofs using the
definitionwe just state it for the record.
Figure 11.3: Function and non-functions
Figure 11.4: Continuity
310
311
Figure 11.5: Monotone transforms preserve maximizers
11.2
Optimization
Monotone increasing transformations of functions do not affect maximizers:

Let A be any set, and let f : A R. That is, f is a function such that f ( a) is a
number for each a A. A maximizer of f on A is a point a A such that
f ( a ) f ( a) for all a A
Now let m be a monotone increasing function, in the sense that if x x 0 , then
m( x ) m( x 0 ), and let g be the function defined by g( a) = m( f ( a)). Our claim is
this:
Any maximizer of f on A is also a maximizer of g on A.
Its easy to see why this is the case. Let a A. Since a is a maximizer of f , it must
be the case that f ( a) f ( a ). Since m is monotone increasing, this implies that
m( f ( a)) m( f ( a )). Given that a was chosen arbitrarily, we have now shown that
g( a ) g( a) for all a A
In other words, a is a maximizer of g on A.
Before finishing this topic, lets recall the notions of supremum and infimum. To
illustrate, consider the function f : (0, 1) (0, 1) defined by f ( x ) = x. It should be
clear that f has no maximiser on (0, 1): given any a (0, 1), we can always choose
another point a (0, 1) such that a = f ( a ) > f ( a ) = a . No maximizer exists
and the optimization problem maxx(0,1) f ( x ) has no solution.
312
To get around this kind of problem, we often the notion of supremum instead. If A
is a set, then the supremum s := sup A is the unique number s such that a s for
every a A, and, moreover, there exists a sequence { xn } A such that xn s.
For example, 1 is the supremum of both (0, 1) and [0, 1]. The infimum i := inf A is
the unique number i such that a i for every a A, and, moreover, there exists a
sequence { xn } A such that xn i. For example, 0 is the supremum of both (0, 1)
and [0, 1].
One can show that the supremum and infimum of any bounded set A exist, and any
set A when the values and are admitted as a possible infima and supremum.
Returning to our original example with f ( x ) = x, while maxx(0,1) f ( x ) is not well
defined, supx(0,1) f ( x ) := sup{ f ( x ) : x (0, 1)} = sup(0, 1) = 1.
11.3
Logical Arguments
To be written:
- Role of counterexamples. Many logical statements are of the form if its an A,
then its a B. (Examples.) The statement may be correct or incorrect. To show its
correct, we take an arbitrary A, and prove that its a B. To show its false, we provide
a counterexample. (Give examples of this process.)
- Contrapositive. Simple example with Venn diagram. Explain how its useful with
an example from the text. (fact 2.2.4?)
- Proof by induction. Simple examples. Relate to (7.27) and (8.7). Relate to discussion of statistial learning and induction.
Bibliography
[1] Amemiya, T. (1994): Introduction to Statistics and Econometrics, Harvard UP.
[2] Bishop, C.M. (2006): Pattern Recognition and Machine Learning, Springer.
[3] Casella, G. and R. L. Berger (1990): Statistical Inference, Duxbury Press, CA.
[4] Cheney, W. (2001): Analysis for Applied Mathematics, Springer.
[5] Dasgupta, A. (2008): Asymptotic Theory of Statistics and Probability, Springer.
[6] Davidson, R. (2000): Econometric Theory, Blackwell Publishing.
[7] Davidson, R., and MacKinnon, J. G. (1993): Estimation and Inference in Econometrics, OUP.
[8] Davidson, R., and MacKinnon, J. G. (2009): Econometric Theory and Methods,
OUP.
[9] Devroye, Luc and Gabor Lugosi (2001): Combinatorial Methods in Density
Estimation Springer-Verlag, New York.
[10] Doeblin, W. (1938): Expose de la theorie des chans simples constantes de
Markov a` un nombre fini detats, Rev. Math. Union Interbalkanique, 2, 77105.
[11] Durrett, R. (1996): Probability: Theory and Examples, Duxbury Press.
[12] Evans, G. and S. Honkapohja (2005): An Interview with Thomas Sargent,
Macroeconomic Dynamics, 9, 561583.
[13] Freedman, D. A. (2009): Statistical Models: Theory and Practice, Cambridge UP.
[14] Greene, W. H. (1993): Econometric Analysis, Prentice-Hall, New Jersey.
313
BIBLIOGRAPHY
314
[15] Hall, R. E. (1978): Stochastic implications of the life cycle-permanent income

hypothesis, Journal of Political Economy 86 (6), 97187.
[16] Hamilton, J. D. (1994). Time Series Analysis, Princeton: Princeton university
press.
[17] Hayashi, F. (2000): Econometrics, Princeton UP.
[18] Hoerl, A. E. and R. W. Kennard (1970): Ridge Regression: Biased Estimation
for Nonorthogonal Problems, Technometrics, 12 (1), 5567.
[19] Jones, G. L. (2004): On the Markov chain central limit theorem, Probability
Surveys, 1, 5-1, 299320.
[20] Kennedy, P. (2003): A Guide to Econometrics. MIT press.
[21] Olley, S. and A. Pakes (1996): The Dynamics of Productivity in the Telecommunications Equipment Industry, Econometrica, 64 (6), 1263-97.
[22] Negri, I and Y. Nishiyama (2010): Review on Goodness of Fit Tests for Ergodic
Diffusion processes by Different Sampling Schemes, Economic Notes, 39, 91
106.
[23] Stachurski, J. and V. Martin (2008): Computing the Distributions of Economic
Models via Simulation, Econometrica, 76 (2), 443450.
[24] Stachurski, J. (2009): Economic Dynamics: Theory and Computation, MIT Press.
[25] Van der Vaart, A. W. (2000): Asymptotic statistics, Cambridge university press.
[26] Vapnik, V. N. (2000): The Nature of Statistical Learning Theory, Springer-Verlag,
New York.
[27] Wasserman, L. (2004): All of Statistics, Springer, New York.
[28] Williams, D. (2001): Probabiliity with Martingales, Cambridge Mathematical
Textbooks.
[29] Wooldridge, J. M. (2010): Econometric analysis of cross section and panel data, MIT
press.
Index
N-consistent, 121
Adapted process, 238

Annihilator, 94
Asymptotic variance, 121
Asymptotically normal, 121
Asymptotically unbiased, 121
Basis, 66
Basis functions, 186
Bayes formula, 296
Bernoulli random variable, 10
Bias, 117
Binary response model, 127
Cauchy-Schwartz inequality, 53
cdf, 16
Central limit theorem, 37
Chi-squared distribution, 24
Coefficient of determination, 189
Column space, 68
Column vector, 54
Complement, 307
Conditional density, 26
Conditional expectation, 99
Conditional probability, 6
Confidence interval, 161
Consistent, 121
Consistent test, 178
Convergence in distribution, 33, 76
Convergence in mean square, 31
Convergence in probability, 31, 75
Covariance, 28
Critical value, 169
Cumulative distribution function, 16
Delta method, 39
Density, 18
Determinant, 69
Diagonal matrix, 54
Dimension, 66
Disjoint sets, 308
ecdf, 138
Empirical cdf, 138
Empirical risk, 144
Empirical risk minimization, 144
Empty set, 308
Ergodicity, 243
ERM, see Empirical risk minimization
Estimator, 116
Expectation, 13
Expectation, vector, 73
Explained sum of squares, 185
F-distribution, 25
Filtration, 238
Full rank, 68
Gaussian distribution, 24
Generalization, 110
Global stability, 243, 250
Gradiant vector, 79
Gradient vector, 262
315
INDEX
Hessian, 262
Homoskedastic, 204
Hypothesis space, 144
Idempotent, 71
Identity matrix, 55
Independence, of events, 6
Independence, of r.v.s, 26
Indicator function, 10
Induction, 111
Infimum, 311
Information set, 97, 238
Inner product, 51
Intersection, 307
Inverse matrix, 69
Inverse transform method, 40
Invertible, 69
Irrational numbers, 306
Joint density, 25
Joint distribution, 25
Kullback-Leibler deviation, 152
Law of large numbers, 34
Law of Total Probability, 7
Least squares, 145, 182
Likelihood function, 125
Linear combinations, 60
Linear function, 56
Linear independence, 64
Linear subspace, 62, 99
Log likelihood function, 126
Logit, 128
Loss function, 143
Marginal distribution, 25
Markov process, 236
Matrix, 54
Maximizer, 311
316
Maximum likelihood estimate, 126
Mean squared error, 117
Measurability, 97
Moment, 28
Monotone increasing function, 311
Multicollinearity, 214
Nonnegative definite, 71
Nonparametric class, 129
Norm, 51
Normal distribution, 24
Null hypothesis, 167, 168
Ordinary least squares, 202
Orthogonal projection, 88
Orthogonal projection theorem, 88
Orthogonal vectors, 86
Overdetermined system, 92
Parametric class, 129
Perfect fit, 189
Plug in estimator, 140
Positive definite, 71
Posterior distribution, 297
Power function, 169
Principle of maximum likelihood, 124
Priors, 297
Probit, 128
Projection matrix, 94
Pythagorean law, 86
R squared, centered, 192
R squared, uncentered, 189
Range, 58
Rank, 68
Rational numbers, 306
Real numbers, 306
Rejection region, 168
Risk function, 143
INDEX
Row vector, 54
Sample k-th moment, 114
Sample correlation, 115
Sample covariance, 114
Sample mean, 114
Sample mean, vector case, 116
Sample standard deviation, 114
Sample variance, 114
Sampling distribution, 158
Sampling error, 205
Scalar product, 51
Set, 306
Singular matrix, 69
Size of a test, 169
Slutskys theorem, 34
Span, 61
Spectral norm, 252
Square matrix, 54
Standard deviation, 28
Standard error, 162, 216
Stationary distribution, 245, 250
Statistic, 114
Students t-distribution, 24
Subset, 306
Sum of squared residuals, 185
Sum, vectors, 51
Supremum, 311
Symmetric cdf, 17
Symmetric matrix, 54
Test, 168
Test statistic, 169
Tikhonov regularization, 288
Total sum of squares, 185
Trace, 71
Transition density, 237
Transpose, 70
Triangle inequality, 53
317
Type I error, 169
Type II error, 169
Unbiased, 117
Uniform distribution, 24
Union, 307
Variance, real r.v., 28
Variance-covariance matrix, 74, 116
Vector of fitted values, 184
Vector of residuals, 185

Et

Uploaded by

Copyright:

Available Formats

Et

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Et

Uploaded by

Copyright:

Available Formats

Econometric Theory

Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Span, Dimension and Independence . . . . . . . . . . . . . . . . . . .

Matrices and Equations . . . . . . . . . . . . . . . . . . . . . . . . . . .

Random Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . .

Convergence of Random Matrices . . . . . . . . . . . . . . . . . . . .

Orthogonality and Projection . . . . . . . . . . . . . . . . . . . . . . .

Overdetermined Systems of Equations . . . . . . . . . . . . . . . . . .

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Inductive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Parametric vs Nonparametric Estimation . . . . . . . . . . . . . . . . 129

Empirical Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Empirical Risk Minimization . . . . . . . . . . . . . . . . . . . . . . . 143

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

Making Inference about Theory . . . . . . . . . . . . . . . . . . . . . . 158

Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Hypothesis Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

Use and Misuse of Testing . . . . . . . . . . . . . . . . . . . . . . . . . 175

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

Linear Least Squares

Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

Transformations of the Data . . . . . . . . . . . . . . . . . . . . . . . . 185

Goodness of Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

Variance and Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

The FWL Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

Normal Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

When the Assumptions Fail . . . . . . . . . . . . . . . . . . . . . . . . 222

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

Time Series Models

Some Common Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

Dynamic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

Maximum Likelihood for Markov Processes . . . . . . . . . . . . . . . 256

Models with Latent Variables . . . . . . . . . . . . . . . . . . . . . . . 263

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

Large Sample OLS

Asymptotic Normality . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

10.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286

11.1 Sets and Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

In setting up a model of probability, we usually start with the notion of a sample

P( B) represents the probability that when uncertainty is resolved and

Together, the triple (, F , P) is called a probability space. It describes a set of

P( A) := #A/6, where #A := number of elements in A

P( A B) = #( A B)/6 = (#A + #B)/6 = #A/6 + #B/6 = P( A) + P( B)

P{2, 4, 6} = P[{2} {4} {6}]

Example 1.1.5. Consider a memory chip in a computer, made up of billions of tiny

In this case we have

(i, j, k) where i I, j J and k K

P( A B) = 9/36 = 1/4 = (18/36) (18/36) = P( A)P( B)

P{LM } = P{LM | play}P{play} + P{LM | dont play}P{dont play}

Confession time: As alluded to above, in my presentation of probability spaces Ive

P(i Ai ) := P{ : Ai for some i} = P( Ai )

F := all subsets of and P( A) := 2 N (#A)

Figure 1.1: Simple function x ( ) = s1{ A} + t1{ B}

where C is some subset of . Thus, x is a binary random variable indicating whether