Chapter 5 Limit Theorems

5.2 The Law of Large Numbers (Convergence in Probability)

(2 Examples from last lecture)

Example Let X(n) denote the nth order statistic of a random sample X1, X2, ..., Xn
from a distribution having pdf
1
f (x) =
θ

for 0 < x < θ, 0 < θ < ∞.

Show that X(n) converges in probability to θ. Recall from last lect ure, we found
that for 0 < x < θ,
n
Fn(x) = P [X(n) ≤ x] = P [Xi ≤ x]
Y

i=1
 n Zx 1 x !n
= du =
Y

i=1 0 θ θ

Now, let 0 <  < θ.

n
θ − 
 

P [|X(n) − θ| > ] = P [X(n) < θ − ] =  

θ
which approaches 0 as n approaches infinity.

Example: Let X(n) denote the nth order statistic of a sample

r
of size n from a
uniform distribution on the interval (0, θ). Show that Zn = X(n) converges in
√
probability to θ.

We know from previous example, that X(n) converges in probability to θ. Also, we

√
know that g(x) = x is a continuous function on √ the nonnegative real numbers.
So, the fact that Zn converges in probability to θ follows from your Homework
Problem.
Suppose we didn’t have that general result of applying continuous transformat ion.
How would we solve this problem directly?

In that case, we would need to show that for any  > 0,

√ r
lim P [| θ − X(n)| > ] = 0
n→∞
However, notice that
√ r √ r
P [| θ − X(n)| > ] = P [ θ − X(n) > ]

√ r √ r √ r
= P [( θ − X(n))( θ + X(n)) > ( θ + X(n))]

√
≤ P [θ − X(n) >  θ]
which we know converges to 0 from previous Example.
5.3 Convergence in Distribution and the Central Limit Theorem

We have seen examples of random variables that are created by applying a function
to the observations of a random sample. The distribution of such a statistic often
depends on n, the size of the sample.

For example, let X̄ denote the sample mean of a random sample X1, .X2, ..., Xn
from the distribution N (µ, σ 2).

We have seen that X̄ is distributed N (µ, σ 2/n).

Thus, the distribution of X̄ depends on n. In some cases we might wish to denote

X̄ by X̄n, to emphasize the dependence of the distribution on the size of the sample.
Example 1: Consider a random sample X1, X2, ..., Xn where each Xi has distri-
bution F (x).

Let X(n) = max{X1, X2, ..., Xn}. X(n) is the nth order statistic in a sample of size
n. We have seen that the distribution function for X(n) is

Fn(x) = [F (x)]n

(In general) Let Xn denote a random variable whose distribution function Fn(x)
depends on n for n = 1, 2, 3, .... In many cases, we are interested in knowing if Fn(x)
converges to some fixed distribution function F (x).

Definition Let X1, X2, . . . be a sequence of random variables with cummulative

distributin functions F1, F2, . . ., and let X be a random variable with distribution
function F . We say that Xn converges in distribtuion to X if

lim Fn(x) = F (x)

n→∞

at every point x at which F (x) is continuous.

First, let’s recall the definition of continuity, and the meaning of a limit.

An infinite sequence of real numbers a1, a2, a3, ... has limit a (converges to a), if for
any number  > 0 there is an integer n0 such that for any n > n0

|a − an| < 

Then we can say

lim a
n→∞ n
=a

For example, consider a sequence where an = 1 + n1 . Show that the limit of the
sequence is the number 1.

Let  > 0 and select n0 to be the smallest integer greater than 1/. Then for any
n > n0 ,
1
|1 − an| = <
n
Using this notion, we say that the function F (y) is continuous at y if for any
sequence y1, y2, .. such that

lim y
n→∞ n
=y

we also have

lim F (yn) = F (y).

n→∞
Example 2 Let X(n) denote the nth order statistic of a random sample X1, X2, ..., Xn
from a distribution having pdf
1
f (x) =
θ

for 0 < x < θ, 0 < θ < ∞.

Clearly Fn(x) = 0 for x < 0 and Fn(x) = 1 for θ ≤ x < ∞, and for 0 ≤ x < θ,
n
Fn(x) = P [X(n) ≤ x] = P [Xi ≤ x]
Y

i=1

n Zx 1 x !n
= du =
Y

i=1 0 θ n

We see that

limn→∞ Fn(x) = 0 for −∞ < x < θ and

limn→∞ Fn(x) = 1 for θ ≤ x < ∞
However, notice that F (x) = 0 for −∞ < x < θ and F (x) = 1 for θ ≤ x < ∞ is a
distribution function, and Fn converges to F .

In particular, F is a distribution function of a discrete type random variable that

takes the value θ with probability equal to 1.

A distribution that places all of its mass on a single point is called a degenerate
distribution.

Let’s continue this example, but define a new random variable

Zn = n(θ − X(n)).

Because X(n) < θ with probability equal to 1, we can say that for z < 0, Fn(z) = 0
for all n, where Fn is the distribution function of Zn.

Also, because X(n) > 0 with probability 1, we can say that Fn(z) = 1 for z ≥ nθ.
Now let z take a value in [0, nθ).

Fn(z) = P [Zn ≤ z] = P [n(θ − X(n)) ≤ z]

z z
= P [X(n) > θ − ] = 1 − P [X(n) ≤ θ − ]
n n
 n
θ − z/n  z n
= 1 −   = 1 − (1 − )
θ nθ

To investigate the limit of Fn(z), notice that for any z, z < nθ when n is taken
sufficiently large. Also, we use the fact that for a real number x,
x n x
lim
n→∞
(1 + ) = e .
n

Thus,

limn→∞Fn(z) = 0 for z < 0 and

limn→∞Fn(z) = 1 − e−z/θ for 0 ≤ z < ∞.
Notice that this is the distribution function of a gamma-distribution with α = 1 and
β = θ. More specicially, it is the distribution function of an exponential distribution.
In this case Fn converges in distribution to a nondegenerate distribution.

We have focused on distribution functions rather than probability density functions

for this notion of convergence in distributions. As it turns out, convergence in dis-
tribution may hold when the pdf does not converge to any fixed pdf.

Example 3: Consider a sequence of random variables

X1, X2, X3, ..., for which the pdf of Xn is given by

fn(x) = 1 for x = 2 + n1 and equals 0 elsewhere.

Then we have for −∞ < x < ∞,

lim f (x)
n→∞ n
= 0.

We see that fn converges to the constant function f (x) = 0 which is not a pdf.
However,

Fn(x) = 0 for x < 2 + n1

and Fn(x) = 1 for x ≥ 2 + n1 .

This converges to the distribution function

F (x) = 0 for x < 2

F (x) = 1 for x ≥ 2

at all the continuity points of F (x) (all points other than x=2). F (x) is the cdf of a
random variable that equals 2 with probability 1.
Example 4: Let X1, X2, ..., Xn be a random sample from the distribution N (µ, σ 2).
Show that
n
Zn = Xi
X

i=1

does not have a limiting distribution.

First, note that Zn has distribution N (nµ, nσ 2).

Fn(z) = P [Zn ≤ z]

Zn − nµ z − nµ z − nµ
= P[ √ ≤ √ ] = Φ( √ )
σ n σ n σ n

where Φ is the distribution function of a N (0, 1) distributed random variable.

Suppose the µ < 0. Then for −∞ < z < ∞
z − nµ
lim Φ(
n→∞
√ )=1
σ n
If µ = 0,
z − nµ
lim Φ(
n→∞
√ ) = 1/2.
σ n
If µ > 0,
z − nµ
lim Φ(
n→∞
√ ) = 0.
σ n

In all three cases, Fn converges to a constant value, and does not converge to a
distribution function.
Theorem: Let {Xn} and {Yn} be sequences of random variables such that Xn
converges in distribution to a random variable X, and Yn converges in probability
to a random variable c. Then, it follows that

a. Xn + Yn converges in distribution to X + c

b. YnXn converges in distribution to cX

c. Xn/Yn converges in distribution to X/c

Theorem A Continuity Theorem: Let Fn be a sequence of cdf’s with correspond-
ing mgf’s Mn. Let F be a cdf with the mgf M . If Mn(t) → M (t) for all t in an
open interval containing zero, then Fn(x) → F (x) for all continuity points of F .

We may use the notation: limn→∞M (t; n) = M (t) for Mn(t) → M (t).

In applying this theorem, we make much use of the form

 cn
b ψ(n) 
lim
n→∞
1 +

+ 
n n

where b and c do not depend on n and

lim ψ(n) = 0.
n→∞

An important result in analysis is that under these assumptions,

cn

cn
b ψ(n)  b
 

lim
n→∞
1 +

+  lim 1 +  = ebc
= n→∞
n n n
Example 5: Let Yn have a distribution that is b(n, pn) where pn = µ/n for some
µ > 0. The mgf of Yn is

M (t; n) = E(etYn ) = [(1 − pn) + pnet]n

t n µ(et − 1) n
= [(1 − µ/n) + µe /n] = [1 + ]
n

Using the preceding result we have

t
µ(e −1)
lim
n→∞
M (t; n) = e

for all t. However, this is the moment generating function of a Poisson distribution
with mean µ.

This example indicates that we may use a Poisson distribution to approximate prob-
abilities of events concerning binomial random variables when p is small and n is
very large.
For example suppose that Y has distribution b(50, .04).
50 49
24 1 24
    

P [Y ≤ 1] =   + 50     = 0.400.
25 25 25

However, we can say that Y has approximately the same distribution of a Poisson
random variable W that has mean µ = np = 50(.04) = 2. For example,

P [W ≤ 1] = e−2 + 2e−2 = 0.406.

We can use the computer to compare the cumulative distribution functions of Y and
W.
y<-c(0:7)
Fy<-pbinom(y,50,.04)
Fw<-ppois(y,2)

cbind(y,Fy,Fw)

y Fy Fw
[1,] 0 0.1298858 0.1353353
[2,] 1 0.4004812 0.4060058
[3,] 2 0.6767140 0.6766764
[4,] 3 0.8608692 0.8571235
[5,] 4 0.9510285 0.9473470
[6,] 5 0.9855896 0.9834364
[7,] 6 0.9963899 0.9954662
[8,] 7 0.9992186 0.9989033
Example 6: Let Zn be χ2(n). Then the mgf of Zn is (1 − 2t)−n/2 for t < 1/2. The
mean and variance of Zn are n and 2n, respectively.

Investigate the limiting distribution of the random variable

√
Yn = (Zn − n)/ 2n

The mgf of Yn is

Zn − n 
   

M (t; n) = E exp t  √
2n
√ √
−tn/ 2n tZn / 2n
=e E(e )

√ t
−tn/ 2n
=e (1 − 2 √ )−n/2
2n
 √ √
−tn/ 2n
This holds for for t < n2. If we write e as
√ √
−tn/ 2n t 2/n −n/2
e = (e )

we see that

√ √ √ −n/2
t 2/n 2
M (t; n) = 
e − t √ et 2/n

n

To simplify this, we apply Taylor’s formula and note that

√ r
1 r eλ(n) r
t 2/n 2
e = 1 + t 2/n + (t 2/n) + (t 2/n)3
2 6
r
where λ(n) is some number between 0 and t 2/n.
 √
t 2/n
If we substitute this expression for e in the equation

√ √ √ −n/2
t 2/n 2
M (t; n) = 
e − t √ et 2/n
n

we see that
−n/2
2

t ψ(n) 
1 −
M (t; n) =  + 
n n

where
√ 3 λ(n)
√ 3
2t e 2t 2t4eλ(n)
ψ(n) = √ − √ −
3 n n 3n

Since λ(n) → 0 and eλ(n) → 1 as n → ∞, we see that limn→∞ ψ(n) = 0. This

implies that
2 /2
lim M (t; n) = et
n→∞
Recall that the mgf for a N (µ, σ 2) distribution is

σ 2t2
M (t) = exp[µt + ]
2

If we choose µ = 0 and σ 2 = 1, we see that M (t; n) converges to the mgf of a

standard normal distribution for all real numbers t.

Thus, Yn converges in distribution to a standard normal random variable.

The preceding example is really a special case of the central limit theorem. In
the next lecture, we will state and prove the central limit theorem. However, we’ll
see an empirical illustration of it here.

Example 7: Consider a random sample X1, X2, ..., Xn from a distribution with
pdf
3
f (x) = x2
2
for −1 < x < 1.
From this we find

3 2
Z x 1 x3
F (x) = −1 t dt = +
2 2 2
for −1 < x < 1.
Z 1 3x 2
E[X] = −1 2
x dx = 0

Z 1 3 4 3
V ar(X) = −1 2
x dx =
5

Now consider the distribution of

√
nX̄n
Zn = r
3/5
where
1 Xn
X̄n = Xi
n i=1
for n = 1, 5, 25.
In this case, we investigate these distributions by simulating enormous samples for
each studied value of n. By inspecting the resulting histograms, we can approximate
the distributions.

To simulate observations with distribution function F (x), we find F −1 and apply

this to a uniformly distributed random variable.

1 x3
u = F (x) = +
2 2

for −1 < x < 1. Thus,

x = F −1(u) = (2u − 1)1/3

for 0 < u < 1.

Now use a computer to simulate 10,000 observations of Zn for each value of n, and
plot the histograms.
## Case n=1
m<-10000
n<-1
u<-runif(m*n)
## Since S-plus seems to dislike cube
## roots of negative numbers, I need
## to trick it
x<-abs(2*u-1)^(1/3)
x<-x*((2*u-1)>0)-x*((2*u-1)<0)
z<-x/sqrt(3/5)
postscript("n1hist.ps")
hist(z,nclass=50)
dev.off()
## Case n=5
m<-10000
n<-5
### construct m datasets
u<-runif(m*n)
x<-abs(2*u-1)^(1/3)
x<-x*((2*u-1)>0)-x*((2*u-1)<0)
x<-matrix(x,m,n)

## find sample means and z

z<-apply(x,1,mean)
z<-z/sqrt((3/5)/n)
postscript("n5hist.ps")
hist(z,nclass=50)
dev.off()
## Case n=25
m<-10000
n<-25
### construct m datasets
u<-runif(m*n)
x<-abs(2*u-1)^(1/3)
x<-x*((2*u-1)>0)-x*((2*u-1)<0)
x<-matrix(x,m,n)

## find sample means and z

z<-apply(x,1,mean)
z<-z/sqrt((3/5)/n)
postscript("n25hist.ps")
hist(z,nclass=50)
dev.off()
Figure 1: Histogram of Zn for n = 1

500
400
300
200
100
0

-1.0 -0.5 0.0 0.5 1.0

z
Figure 2: Histogram of Zn for n = 5

500
400
300
200
100
0

-3 -2 -1 0 1 2 3

z
Figure 3: Histogram of Zn for n = 25

800
600
400
200
0

-4 -2 0 2 4