LECTURE 4

Conditional Probability and Bayes Theorem

1

The conditional sample space

Physical motivation for the formal mathematical theory

1. Roll a fair die once so the sample space S is given by
S = {1, 2, 3, 4, 5, 6}.
Let

A = 6 appears
B = an even number appears,

so

P (A) =

1
6

P (A) = 12 .
Now what about
P (6 appears given an even number appears) .
The above probability will be written
P (A|B) to be read P (A given B).
How do we compute this probability? In fact we havent defined it but we will compute it
for this case and a few more cases using our intuitive sense of what it ought to be and then
arrive at the definition later.
Now we know an even number occurred so the sample space changes,
n
o
1, 2 , 3, 4 , 5, 6
so there are only 3 possible outcomes given an even number occurred so
1
P (6 given an even number occurred) = .
3
The new sample space is called the conditional sample space.

A very important example

2. Suppose you deal two cards (in the usual way without replacement). What is P () i.e.,
P (two hearts in a row). Well,
P (first heart) =

13
.
52

Now, what about the second heart?

Many of you will come up with

12
51

and

P () =

13
52

12
51

There are TWO theoretical points hidden in the formula. Lets first look at

12
nd
P on 2
= .
| {z }
51
this
isnt
really
correct
What we really computed was the conditional probability

12
P on 2nd deal| on 1st deal = .
51
Why ?
Given we got a heart on the first deal the conditional sample space is the new deck with 51
cards and 12 hearts so we get

12
P on 2nd | on 1st = .
51

(1)

The second theoretical point we used was the formula which we will justify formally later

12
13
st
nd
st
.
P () = P on 1 P on 2 | on 1 =
52
51
Two Basic Motivational Problems
I will use the next two problems to motivate the theory we will develop in the rest of this
lecture. One way to solve these two problems will be contained in the HW Problem which
follows this discussion. Another way is to use the general theory we are going to develop in the
rest of this lecture.

1. What is

P on 1st | on 2nd .

This is the reverse order of the conditional probability considered in Equation 1 where
the answer was intuitively clear but for this reversed order it is not at all clear any more
what the answer it.
2. What is

P on 2nd with no information on what happened on the 1st .

An argument that works but I cannot justify

Here is a proof that will prove that this probability is the same as P on the 1st ,
namely 13/52 = 1/4. First you know a card has been played so at first guess you might say
13/51 but that card could have been a heart with probability 1/4 so we subtract 1/4 from the
51
numerator to get [13 (1/4)]/51 = (4)(51)
= 1/4 as I claimed. Intuitively there are 51 cards
available and 12 and 3/4 hearts left. I know of no way to justify this rigorously - the sum
12 + 34 amounts to adding a probability to a cardinality which I do not know how to justify. I
think the sum is an expected value (to be defined later), the expected number of hearts left .
However at the moment I do not see why the probability should be the ratio of this expected
value divided by the number of cards available. Later in this Lecture and on the Homework
problem that follows there will be rigorous proofs.
The Formal Mathematical Theory of Conditional Probability

B
A intersect B

Figure 1: Conditional Probability Picture

#(S) = n, #(A) = a, #(B) = b, #(A B) = c.

Let S be a finite set with the equally-likely probability measure and A and B and A B be the
events with the cardinalities shown in the picture.
Problem
Compute P (A|B). We are given B occurs so the conditional sample space is B. Once again
we will compute a conditional probability without the formal mathematical definition of what
it is.

Only part of A is allowed since we know B occurred namely A B

P (A|B) =
We can rewrite this as
P (A|B) =

#(A B)
c
= .
#(B)
b

c/n
P (A B)
c
=
=
.
b
b/n
P (B)

so
P (A|B) =

P (A B)
.
P (B)

This intuitive formula for the equally likely probability measure leads to the following.

(*)

The Formal Mathematical Definition

Let A and B be any two events in a sample space S with P (B) 6= 0. The conditional probability
of A given B is written P (A|B) and is defined by
P (A B)
.
P (B)

(*)

P (B A)
P (A B)
=
P (A)
P (A)

(**)

P (A|B) =
So if P (A) 6= 0 then
P (B|A) =
since A B = B A.

We wont prove the next theorem but you could do it and it is useful.
Theorem. Fix B with P (B) 6= 0. P (|B) satisfies the axioms (and theorems) of a probability
measure see Lecture 1.
For example
1. P (A1 A2 |B) = P (A1 |B) + P (A2 |B) P (A1 A2 |B)
2. P (A0 |B) = 1 P (A|B)
P (A|) does not satisfy the axioms and theorems.
The Multiplicative Rule for P (A B)
Rewrite (**) as
P (A B) = P (A)P (B|A)

(#)

(#) is very important, more important than (**). It complements the formula
P (A B) = P (A) + P (B) P (A |B).
Now we know how P interacts with both of the basic binary operations and . We will see
that if A and B are independent then
P (A B) = P (A)P (B).
I remember the above formula # as
P (first second) = P (first)P (second | first).
Hopefully this will help you remember it.
More generally
P (A B C) = P (A)P (B|A)P (C|A B).

Exercise
Write down P (A B C D).
Traditional Example
An urn contains 5 white chips, 4 black chips and 3 red chips. Four chips are drawn sequentially
without replacement. Find P (W RW B).
S = {5W, 4B, 3R}.

Solution

P (W RW B) =

5
12

3
11

4
10

4
.
9

What did we do formally

P (W RW B) = P (W ) P (R|W ) P (W |W R) P (B|W R W ).
Bayes Theorem (pg.762)
Bayes Theorem is a truly remarkable theorem. It tells you how to compute P (A|B) if you
know P (B|A) and a few often things. For example we will get a second way (aside from the
Problem at the end of this Lecture) to compute our favorite probability P ( on 1st | on 2nd )
because we know the reversed probability P ( on 2nd | on 1st ) = 12/51
First we will need a preliminary result.
The Law of Total Probability
Let A1 , A2 , , Ak , be mutually exclusive (Ai Aj = , i 6= j) and exhaustive A1 A2 Ak =
S = the whole space). Then for any event B
P (B) = P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + + P (B|Ak )P (Ak ).

([)

Special Case k = 2 so we have A and A0

P (B) = P (B|A)(P (A) + P (B|A0 )P (A0 ).
Application to our ssecond basic problem
Now we can prove
P ( on 2nd with no other information) =
Put
B = on 2nd
A = on 1st
A0 = 6 on 1st

13
52

([[)

Here we write 6 for nonheart. So

P (6 on 1st ) =

39
52

P ( on 2nd |6 on 1st ) =

13
.
51

Now
P (B) = P (B|A)P (A) + P (B|A0 )P (A)
= P ( on 2nd | on 1st )P ( on 1st )
+P ( on 2nd | 6 on 1st )P (6 on 1st )

12
13
13
39
=
+
51
52
51
52
Add fractions:
=
Factor out 13
=

(12)(13) + (13)(39)
(51)(52)

(13)(12 + 39)
(13)
(13)
=
=
(51)(52)
(52)
(52)

Done!

([) is easy to prove but we wont do it. I did it in class for k = 4.

Now we can state Bayes Theorem.

Bayes Theorem
Let A1 , A2 , , Ak be a collection of k mutually exclusive and exhaustive events. Then for any
event B with P (B) > 0 we have
P (B|Aj )P (Aj )
P (Aj |B) = Pk
i=1 P (B|Ai )P (Aj )
Again we wont give the proof for general k. For most applications k = 2 and in fact we
dont the complicated formula for the denominator on the right-hand side (in fact it is just
P (B) which we can usually easily compute in applications).
Special Case k = 2
Suppose we have two events A and B with P (B > 0). then
P (A|B) =

P (B|A)(P (A)
.
P (B|A)P (A) + P (B|A0 )P (A0 )

There is a sometimes better way to look at this formula. Namely

P (A|B) =

P (A)
P (B|A).
P (B)

The two equations are equivalent because of the Law of Total Probability Equation [[.
I will prove this case in the second form above because it shows what is really going on.
The point is that is commutative so
A B = B A.
We will apply P to both sides of this equation and solve for P (A|B) and thereby prove Bayes
Theorem. Indeed
P (A B) =

P (B A) so

P (A)P (B|A) = P (B)P (A|B) and solving for P (A|B)

P (A)
P (B|A).
P (A|B) =
P (B)
Application to our first basic problem
We return to the computation of
P ( on 1st | on 2nd ).
We will use Bayes Theorem. This is the obvious way to do it since we know the probability
the other way around Put
A = ( on 1st )

and B = ( on 2nd ).

By Bayes Theorem
P (A B)
P ()
=
nd
P (B)
P ( on 2 with no other information)
Now we know from earlier in this lecture that

13
12
P () =
.
52
51
P (A|B) =

and we saw just before the proof of Bayes Theorem (the solution of our first basic problem)
that
13
P ( on 2nd with no other information) = .
52
Hence
13 12
12
P ( on 1st | on 2nd ) = 521351 =
= P ( on 2nd | on 1st )!!
51
52
P ( on 1st | on 2nd ).
This completes the solution of the second basic problem.
Compulsory Reading (for your own health).
In case you or someone you love test positive for a rare (this is the point) disease, read Example
2.30, pg. 73. Misleading (and even bad) statistics is rampant in medicine.
Problems
Problem 1
The point of this problem is to compute P (on1st |on2nd ) and P (on1st ) using an abstract
approach counting pairs.
(i). Let S be the set of unordered pairs of distinct cards. S will be our sample space and the
previous hard to understand problems will be solved by counting subsets of pairs with various
conditions on one or both elements of the pair.
Compute #(S).
(ii). Let A be the subset of S consisting of pairs where both cards in the pair are hearts so
A = {} S.
Compute

#(A)
.
#(S)
(iii) Let B be the subset of all pairs to that it the second card in the pair is a heart and the
first card is anything. Compute #(B) and P (B) solving 2. above (namely the probability that
the second card is a heart with no condition on the first).

(iv) Compute P on 1st | on 2nd by taking the ratio P (A)/P (B), see the defining formula
(*) two pages later, or else by computing within the conditional sample space SB which is a
subspace of the set of all pairs S.
#(A) and P (A =