Conditional Probability, Independence, Bayes’ Theorem

18.05 Spring 2014

Sample Space Confusions

1. Sample space = set of all possible outcomes of an experiment.

2. The size of the set is NOT the sample space.
3. Outcomes can be sequences of numbers.
Examples.
1. Roll 5 dice: Ω = set of all sequences of 5 numbers between 1 and
6, e.g. (1, 2, 1, 3, 1, 5) ∈ Ω.
The size |Ω| = 65 is not a set.
2. Ω = set of all sequences of 10 birthdays,
e.g. (111, 231, 3, 44, 55, 129, 345, 14, 24, 14) ∈ Ω.
|Ω| = 36510
3. n some number, Ω = set of all sequences of n birthdays.
|Ω| = 365n .
Conditional Probability
‘the probability of A given B’.
P(A ∩ B)
P(A|B) = , provided P(B) = 0.
P(B)

B
A
A∩B

Conditional probability: Abstractly and for coin example

Table/Concept Question
(Work with your tablemates, then everyone click in the answer.)
Toss a coin 4 times. Let
A = ‘at least three heads’
B = ‘first toss is tails’.
1. What is P(A|B)?
(a) 1/16 (b) 1/8 (c) 1/4 (d) 1/5

2. What is P(B|A)?
(a) 1/16 (b) 1/8 (c) 1/4 (d) 1/5
answer: 1. (b) 1/8. 2. (d) 1/5.
Counting we find |A| = 5, |B| = 8 and |A ∩ B| = 1. Since all sequences
are equally likely
P(A ∩ B) |A ∩ B| |B ∩ A|
P(A|B) = = = 1/8. P(B|A) = = 1/5.
P(B) |B| |A|
Table Question

“Steve is very shy and withdrawn, invariably

helpful, but with little interest in people, or in the
world of reality. A meek and tidy soul, he has a
need for order and structure and a passion for
detail.”∗
What is the probability that Steve is a librarian?
What is the probability that Steve is a farmer?
Discussion on next slide.
∗ From
Judgment under uncertainty: heuristics and biases by Tversky and
Kahneman.
Discussion of Shy Steve

Discussion: Most people say that it is more likely that Steve is a librarian
than a farmer. Almost all people fail to consider that for every male
librarian in the United States, there are more than fifty male farmers.
When this is explained, most people who chose librarian switch their
solution to farmer.
This illustrates how people often substitute representativeness for
likelihood. The fact that a librarian may be likely to have the above
personality traits does not mean that someone with these traits is likely to
be a librarian.
Multiplication Rule, Law of Total Probability
Multiplication rule: P(A ∩ B) = P(A|B) · P(B).

Law of total probability: If B1 , B2 , B3 partition Ω then

P(A) = P(A ∩ B1 ) + P(A ∩ B2 ) + P(A ∩ B3 )
= P(A|B1 )P(B1 ) + P(A|B2 )P(B2 ) + P(A|B3 )P(B3 )

Ω
B1
A ∩ B1

A ∩ B2 A ∩ B3

B2 B3
Trees
Organize computations
Compute total probability
Compute Bayes’ formula
Example. : Game: 5 red and 2 green balls in an urn. A random ball
is selected and replaced by a ball of the other color; then a second
ball is drawn.
1. What is the probability the second ball is red?
2. What is the probability the first ball was red given the second ball
was red?
5/7 2/7
First draw R1 G1
4/7 3/7 6/7 1/7
Second draw
R2 G2 R2 G2
Solution

5 4 2 6 32
1. The law of total probability gives P(R2 ) = · + · =
7 7 7 7 49

P(R1 ∩ R2 ) 20/49 20
2. Bayes’ rule gives P(R1 |R2 ) = = =
P(R2 ) 32/49 32
Concept Question: Trees 1
x
A1 y A2

B1 z B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

1. The probability x represents

(a) P(A1 )
(b) P(A1 |B2 )
(c) P(B2 |A1 )
(d) P(C1 |B2 ∩ A1 ).
answer: (a) P(A1 ).
Concept Question: Trees 2
x
A1 y A2

B1 z B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

2. The probability y represents

(a) P(B2 )
(b) P(A1 |B2 )
(c) P(B2 |A1 )
(d) P(C1 |B2 ∩ A1 ).
answer: (c) P(B2 |A1 ).
Concept Question: Trees 3
x
A1 y A2

B1 z B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

3. The probability z represents

(a) P(C1 )
(b) P(B2 |C1 )
(c) P(C1 |B2 )
(d) P(C1 |B2 ∩ A1 ).
answer: (d) P(C1 |B2 ∩ A1 ).
Concept Question: Trees 4
x
A1 y A2

B1 z B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

4. The circled node represents the event

(a) C1
(b) B2 ∩ C1
(c) A1 ∩ B2 ∩ C1
(d) C1 |B2 ∩ A1 .
answer: (c) A1 ∩ B2 ∩ C1 .
Let’s Make a Deal with Monty Hall
One door hides a car, two hide goats.
The contestant chooses any door.
Monty always opens a different door with a goat. (He
can do this because he knows where the car is.)
The contestant is then allowed to switch doors if she
wants.
What is the best strategy for winning a car?
(a) Switch (b) Don’t switch (c) It doesn’t matter
Board question: Monty Hall
Organize the Monty Hall problem into a tree and compute
the probability of winning if you always switch.
Hint first break the game into a sequence of actions.
answer: Switch. P(C |switch) = 2/3
It’s easiest to show this with a tree representing the switching strategy:
First the contestant chooses a door, (then Monty shows a goat), then the
contestant switches doors.
Probability Switching Wins the Car
1/3 2/3

Chooses
C G
0 1 1 0
Switches
C G C G

1 2
The (total) probability of C is P(C |switch) = 3 ·0+ 3 · 1 = 23 .
Independence
Events A and B are independent if the probability that
one occurred is not affected by knowledge that the other
occurred.

Independence ⇔ P(A|B) = P(A) (provided P(B) 6= 0)

⇔ P(B|A) = P(B) (provided P(A) 6= 0)

(For any A and B)

⇔ P(A ∩ B) = P(A)P(B)
Table/Concept Question: Independence
(Work with your tablemates, then everyone click in the answer.)

Roll two dice and consider the following events

A = ‘first die is 3’
B = ‘sum is 6’
C = ‘sum is 7’
A is independent of
(a) B and C (b) B alone
(c) C alone (d) Neither B or C .
answer: (c). (Explanation on next slide)
Solution
P(A) = 1/6, P(A|B) = 1/5. Not equal, so not independent.
P(A) = 1/6, P(A|C ) = 1/6. Equal, so independent.

Notice that knowing B, removes 6 as a possibility for the first die and
makes A more probable. So, knowing B occurred changes the probability
of A.

But, knowing C does not change the probabilities for the possible values of
the first roll; they are still 1/6 for each value. In particular, knowing C
occured does not change the probability of A.

Could also have done this problem by showing

P(B|A) 6= P(B) or P(A ∩ B) 6= P(A)P(B).

Bayes’ Theorem

Also called Bayes’ Rule and Bayes’ Formula.

Allows you to find P(A|B) from P(B|A), i.e. to ‘invert’
conditional probabilities.

P(B|A) · P(A)
P(A|B) =
P(B)
Often compute the denominator P(B) using the law of
total probability.
Board Question: Evil Squirrels

Of the one million squirrels on MIT’s campus most are

good-natured. But one hundred of them are pure evil! An
enterprising student in Course 6 develops an “Evil Squirrel
Alarm” which she offers to sell to MIT for a passing
grade. MIT decides to test the reliability of the alarm by
conducting trials.
Evil Squirrels Continued

When presented with an evil squirrel, the alarm goes

off 99% of the time.
When presented with a good-natured squirrel, the
alarm goes off 1% of the time.
(a) If a squirrel sets off the alarm, what is the probability
that it is evil?
(b) Should MIT co-opt the patent rights and employ the
system?
Solution on next slides.
One solution
(This is a base rate fallacy problem)
We are given:
P(nice) = 0.9999, P(evil) = 0.0001 (base rate)

P(alarm | nice) = 0.01, P(alarm | evil) = 0.99

P(alarm | evil)P(evil)
P(evil | alarm) =
P(alarm)

P(alarm | evil)P(evil)
=
P(alarm | evil)P(evil) + P(alarm | nice)P(nice)

(0.99)(0.0001)
=
(0.99)(0.0001) + (0.01)(0.9999)

≈ 0.01
Squirrels continued
Summary:
Probability a random test is correct = 0.99

Probability a positive test is correct ≈ 0.01

These probabilities are not the same!

Alternative method of calculation:

Evil Nice
Alarm 99 9999 10098
No alarm 1 989901 989902
100 999900 1000000
Evil Squirrels Solution

answer: (a) This is the same solution as in the slides above, but in a more
compact notation. Let E be the event that a squirrel is evil. Let A be the
event that the alarm goes off. By Bayes’ Theorem, we have:

P(A | E )P(E )
P(E | A) =
P(A | E )P(E ) + P(A | E c )P(E c )
100
.99 1000000
= 100 999900
.99 1000000 + .01 1000000
≈ .01.

(b) No. The alarm would be more trouble than its worth, since for every
true positive there are about 99 false positives.
Washington Post, hot off the press

Annual physical exam is probably unnecessary if you’re

generally healthy
For patients, the negatives include time away
from work and possibly unnecessary tests.
“Getting a simple urinalysis could lead to a false
positive, which could trigger a cascade of even
more tests, only to discover in the end that you
had nothing wrong with you.” Mehrotra says.

http://www.washingtonpost.com/national/health-science/
annual-physical-exam-is-probably-unnecessary-if-youre-general
2013/02/08/2c1e326a-5f2b-11e2-a389-ee565c81c565_story.html
Table Question: Dice Game
1 The Randomizer holds the 6-sided die in one fist and
the 8-sided die in the other.
2 The Roller selects one of the Randomizer’s fists and
covertly takes the die.
3 The Roller rolls the die in secret and reports the result
to the table.

Given the reported number, what is the probability that

the 6-sided die was chosen? (Find the probability for each
possible reported number.)
answer: If the number rolled is 1-6 then P(six-sided) = 4/7.
If the number rolled is 7 or 8 then P(six-sided) = 0.
Explanation on next page
Dice Solution
This is a Bayes’ formula problem. For concreteness let’s suppose the roll
was a 4. What we want to compute is P(6-sided|roll 4). But, what is
easy to compute is P(roll 4|6-sided).
Bayes’ formula says

P(roll 4|6-sided)P(6-sided)
P(6-sided|roll 4) =
P(4)
(1/6)(1/2)
= = 4/7.
(1/6)(1/2) + (1/8)(1/2)

The denominator is computed using the law of total probability:

1 1 1 1
P(4) = P(4|6-sided)P(6-sided) + P(4|8-sided)P(8-sided) = · + · .
6 2 8 2
Note that any roll of 1,2,. . . 6 would give the same result. A roll of 7 (or
8) would give clearly give probability 0. This is seen in Bayes’ formula
because the term P(roll 7|6-sided) = 0.