Bayesian Inference

by Hoai Nam Nguyen

September 9, 2017

1
The setting is the same. Given a population that follows a distribution P ,
where P contains 1 or more unknown parameters, we want to construct an
estimator for each of them. In this course, I consider the simple case, where
there is only 1 unknown parameter . To do this, we proceed by collecting
an i.i.d sample X1 , ..., Xn P

Similar to Maximum Likelihood Estimation, we first find the Likelihood

Function L():

L() = fX1 ,...,Xn (x1 , ..., xn |)

In Bayesian inference, we treat the parameter as a random variable. That

is, follows a probability distribution with pdf (). We call () the prior
distribution of

By Bayess formula, we have

fX1 ,...,Xn (x1 , ..., xn |)()

(|x1 , ..., xn ) =
fX1 ,...,Xn (x1 , ..., xn )

fX1 ,...,Xn (x1 , ..., xn |)()

where (|x1 , ..., xn ) is the pdf of given the sample data. This is called the
posterior distribution of

Let me clarify the last step further. The symbol means proportional
to. Since the left-hand side is the distribution of conditional on the sam-
ple data {x1 , ..., xn }, all the xi are assumed to be known and the denominator
fX1 ,...,Xn (x1 , ..., xn ) is, therefore, no more than a constant

In this setting, we are given the population distribution P and the prior
distribution (). We have to find the posterior distribution (|x1 , ..., xn ).
We then use the posterior mean E[|x1 , ..., xn ] to estimate the unknown
parameter . That is,

= E[|x1 , ..., xn ]

NOTE: when calculating (|x1 , ..., xn ), always use proportionality by re-

moving constants because this will simply the calculation a lot

2
Example 1

The population distribution is Bernoulli(p), where p U nif orm(0, 1). Use

Bayesian inference to construct an estimator p

The likelihood function is given by:

n
Y
L(p) = fXi (xi |p)
i=1

n
Y
= pxi (1 p)1xi
i=1

P P
xi
=p (1 p)n xi

The pdf of the prior distribution is (p) = 1, for 0 < p < 1

Therefore, the posterior distribution is given by:

(p|x1 , ..., xn ) fX1 ,...,Xn (x1 , ..., xn |p)(p)

P P
xi
=p (1 p)n xi
, for 0 < p < 1
Recall the pdf of Beta(, ):
(+) 1
fX (x) = ()()
x (1 x)1 , for 0 < x < 1
P
By comparing,
P we can see that the posterior distribution p is Beta( xi +
1, n xi + 1)

We know that the expectation of Beta(, ) is +
. Therefore, the posterior
mean is given by:
P
xi +1
E[p|x1 , ..., xn ] = n+2
P
Xi +1
Thus, p = n+2
is the Bayesian estimator for p

Note that we used proportionality when calculating the posterior distribu-

tion. By comparing with the pdf of Beta(, ), we can easily recover the
missing constant:
(+) (n+2) P
c= ()()
= P
( xi +1)(n xi +1)

3
Example 2

Same as example 1, except that p Beta(a, b), where both a and b are
given constants

The likelihood function stays unchanged:

P P
xi
L(p) = p (1 p)n xi

The pdf of the prior distribution is given by:

(a+b) a1
(p) = (a)(b)
p (1 p)b1 , for 0 < p < 1

Therefore, the pdf of the posterior distribution is given by:

(p|x1 , ..., xn ) fX1 ,...,Xn (x1 , ..., xn |p)(p)

P P
xi
p (1 p)n xi a1
p (1 p)b1

P P
p xi +a1 (1 p)n xi +b1 , for 0 < p < 1
P P
We recognise this is Beta( xi + a, n xi + b)
P
xi +a
The posterior mean is E[p|x1 , ..., xn ] = n+a+b
. The Bayesian estimator for p
is given by:
P
Xi +a
p = n+a+b

Again, you can recover the normalising constant in the pdf of the posterior
distribution:

c= P (n+a+b) P
( xi +a)(n xi +b)

4
Example 3

The population distribution is N (, 2 ), where is unknown and 2 is known.

The parameter follows a prior distribution N (, 2 ), where both and 2
are given constants. Use Bayesian inference to construct an estimator p

The likelihood function is given by:

n
Y
L() = fXi (xi |)
i=1

n  (x )2 
Y 1 i
= exp
i=1 2 2 2 2

n
Y (xi )2 

exp 2
, because 2 is known
i=1
2

Also, the pdf of the prior distribution is given by:

1  ( )2 
() = p exp
2 2 2 2

 ( )2 
exp , because 2 is known
2 2
Then, calculate the pdf of the posterior distribution:
Yn  (x )2   ( )2 
i
(|x1 , ..., xn ) exp exp
i=1
2 2 2 2

n
1 X
   1 
= exp 2 (xi )2 exp 2 (2 2 + 2 )
2 i=1 2

n
 1 X   1 
exp 2 (xi )2 exp 2 (2 2)
2 i=1 2

 2 
by removing exp
2 2

5
 n
1 X 2

2
  1 2

= exp 2 (x 2xi + ) exp 2 ( 2)
2 i=1 i 2

n
1 X
   1 
exp 2 (2xi + 2 ) exp 2 (2 2)
2 i=1 2

n
1 X 2 
by removing exp 2 x
2 i=1 i

n
 X n2   1 2

= exp xi exp ( 2)
2 i=1
2 2 2 2

 n n
h 1  2 1 X  i
= exp + + xi + 2
2 2 2 2 2 i=1

= exp(A2 + B)

 2 B/A 
= exp
1/A

 2 B/A + B 2 /4A2 
exp
1/A

( B/2A)2 i
h
= exp
1/A

Comparing with the pdf of a Normal distribution, we deduce that the pos-
terior distribution of is given by:
 
B 1
|x1 , ...xn N 2A , 2A

Clearly, E[|x1 , ..., xn ] = B/2A. Therefore, = B/2A is the Bayesian esti-

mator for

6
Example 4

Consider the following types of treatment:

Treatment 1: 100% of the patients are cured (3 out of 3)

Treatment 2: 95% of the patients are cured (19 out of 20)

Treatment 3: 90% of the patients are cured (90,000 out of 100,000)

Which one is the best???

Treatment 1 cured 100% of the patients. But the sample was so small that
we should cast doubt on the result. On the other hand, Treatment 3 was
very reassuring, but the percentage was a bit lower

Let p be the probability that a patient is cured. Then, the probability that
a patient is not cured is 1 p

Therefore, the population follows Bernoulli(p), where p is an unknown pa-

rameter
P
xi +1
In example 1, we found that p = n+2
provided an estimate for p
3+1 4
Treatment 1: p = 3+2
= 5
= 0.8
19+1 20
Treatment 2: p = 20+2
= 22
0.909
90000+1 90001
Treatment 3: p = 100000+2
= 100002
0.9

We can see that p for Treatment 2 is the highest. Therefore, we predict that
Treatment 2 is the best one. Treatment 1, despite curing everyone in the
sample, is predicted to be the worst due to its small sample size