1.

Prob Intro

Basics of Probability Modules

1. Intro / Examples
2. Set Theory
3. Experiments and Sample Spaces
4. Denition of Probability
5. Finite Sample Spaces
6. Counting Techniques
7. Applications of Counting Techniques
8. Conditional Probability
9. Bayes Theorem
1
 1.1 Intro / Examples

Mathematical Models for describing observable phe-

nomena:

Deterministic

Probabilistic

2
 1.1 Intro / Examples

Deterministic Models

Ohms Law (I = E/R)

Drop an object from height h0. After t sec, height

is h(t) = h0 16t2.

Deposit $1000 in a continuously compounding check-

ing 3% account. At time t, its worth $1000e.03t.

3
 1.1 Intro / Examples

Probabilistic Models Involve uncertainty

How much snow will fall tomorrow?

Will IBM make a prot this year?

Should I buy a call or put option?

Can I win in blackjack if I use a certain strategy?

4
 1.1 Intro / Examples

Some Cool Examples

1. Birthday Problem Assume all 365 days have

equal probability of being a persons birthday (ignore
Feb 29). Then. . .

If there are 23 people in the room, the odds are better

than 5050 that there will be a match.

If there are 50 people, the probability is about 97%!

5
 1.1 Intro / Examples

2. Monopoly In the long run, the property having

the highest probability of being landed on is Illinois
Ave.

3. Poker Pick 5 cards from a standard deck. Then

Pr(exactly 2 pairs) 0.0475

Pr(full house) 0.00144
Pr(ush) 0.00198

6
 1.1 Intro / Examples

4. Stock Market Monkeys randomly selecting stocks

could have outperformed most market analysts during
the past year.

5. A couple has two kids and at least one is a boy.

Whats the probability that BOTH are boys?

Possibilities: GG, BG, GB, BB. Eliminate GG since we

know that theres at least one boy. Then Pr(BB) =
1/3.
7
 1.1 Intro / Examples

6. Vietnam Lottery

7. Ask Marilyn. You are a contestant at a game show.

Behind one of three doors is a car; behind the other
two are goats. You pick door A. Monty Hall opens
door B and reveals a goat. Monty oers you a chance
to switch to door C. What should you do?

8
 1.1 Intro / Examples

Working Denitions

Probability Methodology that describes the ran-

dom variation in systems. (Well spend about 40% of
our time on this.)

Statistics Uses sample data to draw general con-

clusions about the population from which the sample
was taken. (60% of our time.)

9
 1.2 Set Theory

The Joy of Sets

Denition: A set is a collection of objects. Members

of a set are called elements.

Notation:
A, B, C, . . . for sets; a, b, c, . . . for elements
for membership, e.g., x A

/ for non-membership, e.g., x

/A
U is the universal set (i.e., everything)
is the empty set.
10
 1.2 Set Theory

Examples:

A = {1, 2, . . . , 10}. 2 A, 49
/ A.
B = {basketball, baseball}
C = {x|0 x 1} (| means such that)
D = {x|x2 = 9} = {3} (either is ne)
E = {x|x , x2 = 1} = ( is the real line)

11
 1.2 Set Theory

Denition: If every element of set A is an element of

set B then A is a subset of B, i.e., A B.

Denition: A = B i (if and only if) A B and B A.

Properties:
A; A U ; A A
A B and B C (implies) A C

Remark: Order of element listing is immaterial, e.g,

{a, b, c} = {b, c, a}.
12
 1.2 Set Theory

Denitions: Complement of A with respect to U is

A {x|x U and x
/ A}.

Intersection of A and B is A B {x|x A and x

B}.

Union of A and B is AB {x|x A or x B (or both)}.

(Remember Venn diagrams?)

13
 1.2 Set Theory

Example:

Suppose U = {letters of the alphabet}, A = {vowels},

and B = {a, b, c}. Then

A = {consonants}
A B = {a}
A B = {a, b, c, e, i, o, u}

If A B = , then A and B are disjoint (or mutually

exclusive).
14
 1.2 Set Theory

Denitions:

Minus: A B A B

Symmetric dierence or XOR:

AB (A B) (B A) = (A B) (A B)

The cardinality of A, |A|, is the number of elements

in A. A is nite if |A| < .
15
 1.2 Set Theory

Examples:

A = {3, 4} is nite since |A| = 2.

B = {1, 2, 3, . . .} is countably innite.

C = {x|x [0, 1]} is uncountably innite.

16
 1.2 Set Theory

Laws of Operation:

=A
1. Complement Law: A A = U , A A = , A

2. Commutative: A B = B A, A B = B A

3. DeMorgans: A B = A B, A B = A B

4. Associative: A (B C) = (A B) C,
A (B C) = (A B) C
17
 1.2 Set Theory

5. Distributive: A (B C) = (A B) (A C),
A (B C) = (A B) (A C)

Proofs: Easy. Could use Venn diagrams or many other

ways.

18
 1.3 Experiments

Experiments and Sample Spaces

Consider a random experiment:

E1: Toss a die and observe the outcome.

Denition: A sample space associated with an ex-

periment E is the set of all possible outcomes of E.
Its usually denoted by S or .

19
 1.3 Experiments

Examples:

E1 has sample space S1 = {1, 2, 3, 4, 5, 6}.

Another sample space for E1 is S1 = {odd, even}.

So a sample space doesnt have to be unique!

20
 1.3 Experiments

E2: Toss a coin 3 times and observe the sequence of

Hs and T s.

S2 = {HHH, HHT, HT H, HT T, T HH, T HT, T T H, T T T }.

E3: A new light bulb is tested to see how long it lasts.

S3 = {t|t 0}.

21
 1.3 Experiments

Denition: An event is a set of possible outcomes.

Thus, any subset of S is an event.

Example (for E1): If A1 is the event an even num-

ber occurs, then A1 = {2, 4, 6}, i.e., when the die is
tossed, we get 2 or 4 or 6.

Remark: is an event of S (nothing happens)

S is an event of S (something happens)

22
 1.3 Experiments

Remark: If A is an event, then A is the complementary

(opposite) event.

Example (for E1):

A1 = {2, 4, 6} A1 = {1, 3, 5} (i.e., an odd number

occurs)

Remark: If A and B are events, then A B and A B

are events.
23
 1.3 Experiments

Example (for E2): Let

A2 = exactly one T was observed

= {HHT, HT H, T HH}
B2 = no T s observed = {HHH}
C2 = rst coin is H
= {HHH, HHT, HT H, HT T }

Then

A2 B2 = at most one T observed

= {HHT, HT H, T HH, HHH}
A2 C2 = {HHT, HT H}
24
 1.4 Probability

Probability Basics

Suppose A is some event for a sample space S. Whats

the prob that A will occur, i.e., Pr(A)?

Example: Toss a fair coin. S = {H, T }. Whats the

prob that H will come up?

Pr({H}) = Pr(H) = 1/2.

What does this mean?

25
 1.4 Probability

Frequentist view: If the experiment were repeated n

times, where n is very large, wed expect about 1/2
of the tosses to be Hs.
Total # of Hs out of n tosses
1/2.
n

Example: Toss a fair die. S = {1, 2, 3, 4, 5, 6}, where

each individual outcome has prob 1/6. Then Pr(1, 2) =
1/3.

26
 1.4 Probability

Denition: With each event A S, we associate a

number Pr(A), called the probability of A, satisfy-
ing the following axioms:

(1) 0 Pr(A) 1 (probs are always betw. 0 and 1).

(2) Pr(S) = 1 (prob of some outcome is 1). Example:

Die. Pr(S) = Pr(1, 2, 3, 4, 5, 6) = 1.

27
 1.4 Probability

(3) If A B = , then Pr(A B) = Pr(A) + Pr(B).

Example: Pr(1 or 2) = Pr(1) + Pr(2) = 1/6 + 1/6 =
1/3.

(4) Suppose A1, A2, . . . is a sequence of disjoint events

(i.e., Ai Aj = for i = j). Then


 
Pr Ai = Pr(Ai).
i=1 i=1

28
 1.4 Probability

Example: Toss a coin until the rst H appears.

S = {H, T H, T T H, T T T H, . . .}.

Dene the disjoint events

A1 = {H}, A2 = {T H}, A3 = {T T H}, . . . .

Then


 
1 = Pr(S) = Pr Ai = Pr(Ai).
i=1 i=1

29
 1.4 Probability

More Nifty Properties

Theorem 1: Pr() = 0.

Proof: Since A = , we have that A and are

disjoint. So Axiom (3) implies

Pr(A) = Pr(A ) = Pr(A) + Pr().

Note: Converse is false: Pr(A) = 0 does not imply

A = . Example: Pick a random number betw. 0 and
1.
30
 1.4 Probability

Theorem 2: Pr(A) = 1 Pr(A).

Proof:

1 = Pr(S) (by Axiom (2))

= Pr(A A)
= Pr(A) + Pr(A) (A A = ; Axiom (3)).

Example: The probability that itll rain tomorrow is

one minus the probability that it wont rain.
31
 1.4 Probability

Theorem 3: For any two events A and B,

Pr(A B) = Pr(A) + Pr(B) Pr(A B)

Proof: First observe that B = (A B) (A B) where

A B and A B are disjoint. Thus

Pr(B) = Pr(A B) + Pr(A B) ()

and so

Pr(A B) = Pr(A) + Pr(A B) (A,A B are disjoint)

= Pr(A) + Pr(B) Pr(A B) (by ()).

32
 1.4 Probability

Remark: Can also do an easy Venn diagram proof.

(Subtract Pr(A B) to avoid double-counting.)

Remark: Axiom (3) is a special case of this theorem

in which A B = .

33
 1.4 Probability

Example: Suppose theres. . .

40% chance of colder weather
10% chance of rain and colder weather
80% chance of rain or colder weather.

Find the chance of rain.

Pr(R) = Pr(R C) Pr(C) + Pr(R C)

= 0.8 0.4 + 0.1 = 0.5.

34
 1.4 Probability

Theorem 4: For any three events A, B, and C,

Pr(A B C)
= Pr(A) + Pr(B) + Pr(C)
Pr(A B) Pr(A C) Pr(B C)
+Pr(A B C)

Remark: See any good prob book for a generalization

to Pr(A1 A2 An). Its called the principle of
inclusion-exclusion.
35
 1.4 Probability

Example: 75% of Atlantans jog (J), 20% like ice

cream (I), and 40% enjoy music (M ). Also, 15%
J and I, 30% J and M , 10% I and M , and 5% do
all three. Find the prob that a random resident will
engage in at least one of the three activities.

P (J I M )
= P (J) + P (I) + P (M )
P (J I) P (J M ) P (I M )
+P (J I M )
= .75 + .20 + .40 .15 .30 .10 + .05 = .85.
36
 1.4 Probability

Find the prob of precisely one activity.

P (J I M ) + P (J I M ) + P (J I M )
= .35 + 0 + .05 = .40.

Howd we get those?? Use Venn diagram, starting

from the center and building out.

37
 1.4 Probability

(Bonus) Theorem 5: A B Pr(A) Pr(B).

Proof:

Pr(B) = Pr(A (A B))

= Pr(A) + Pr(A B)
Pr(A).

Remark: A B means that B occurs whenever A

occurs; so the Theorem makes intuitive sense.
38
 1.5 Finite Sample Spaces

Finite Sample Spaces

Suppose S is nite, say S = {a1, a2, . . . , an}.

Let B be an event consisting of r ( n) outcomes,

i.e., B = {aj1 , aj2 , . . . , ajr }, where the jis represent r
r
indices from {1, 2, . . . , n}. Then Pr(B) = i=1 Pr(aji ).

Note: Choosing an object at random means that

each object has the same prob of being chosen.
39
 1.5 Finite Sample Spaces

Example: You have 2 red cards, a blue card, and a

yellow. Pick one card at random.

S = {red, blue, yellow} = {a1, a2, a3}

Pr(a1) = 1/2, Pr(a2) = 1/4, Pr(a3) = 1/4.

Pr(red or yellow) = Pr(a1) + Pr(a3) = 3/4.

40
 1.5 Finite Sample Spaces

Denition: A simple sample space (SSS) is a nite

sample space in which all outcomes are equally likely.

Remark: In the above example, S is not simple since

Pr(a1) = Pr(a2).

Example: Toss 2 fair coins.

S = {HH, HT, T H, T T } is a SSS (all probs are 1/4).

S  = {0, 1, 2} (number of Hs) is not a SSS. Why?

41
 1.5 Finite Sample Spaces

Theorem: For any event A in a SSS S,

|A| # elements in A
Pr(A) = = .
|S| # elements in S

Example: Die. A = {1, 2, 4, 6} (each with prob 1/6).

Pr(A) = 4/6.

42
 1.5 Finite Sample Spaces

Example: Roll a pair of dice. Possible results (each

w.p. 1/36):

1,1 1,2 1,6

2,1 2,2 2,6
..
6,1 6,2 6,6

Sum 2 3 4 5 6 7 8 9 10 11 12
1
Prob 36 2 3 4 5 6 5 4 3 2 1
36 36 36 36 36 36 36 36 36 36

43
 1.6 Counting Techniques

Counting Techniques count the elements in events

from a SSS.

Multiplication Rule

Addition Rule

Permutations

Combinations
44
 1.6 Counting Techniques

Multiplication Rule

Two operations are performed one after the other:

(a) The rst operation can be done in n1 ways.

(b) Regardless of the way in which the rst operation
was performed, the second can be performed in n2
ways.

The # ways to perform the two operations together

is n1n2.
45
 1.6 Counting Techniques

Example: 3 ways to go from City A to B, and 4 ways

to go from B to C. Then the you can go from A to
C (via B) in 12 ways.

Example: Roll 2 dice. How many outcomes? (Assume

(3, 2) = (2, 3).) Answer is 36.

46
 1.6 Counting Techniques

Example: Select 2 cards from a deck without re-

placement and care about order (i.e., (Q, 7) =
(7, Q). How many ways can you do this? Answer:
52 51 = 2652.

Example: Box of 10 sox 2 red and 8 black. Pick

2 w/o repl.

(a) Let A be the event that both are red.

# ways to pick 2 reds 21 1
Pr(A) = = = .
# ways to pick 2 sox 10 9 45
47
 1.6 Counting Techniques

(b) Let B be the event that both are black.

87 28
Pr(B) = = .
10 9 45

(c) Let C be one of each color.

Pr(C) = 1 Pr(C)
= 1 Pr(A B)
= 1 Pr(A) Pr(B) (A and B disjoint)
= 16/45.

48
 1.6 Counting Techniques

Remark: The multiplication rule can be extended to

more than 2 operations.

Example: Flip 3 coins. 222 = 8 possible outcomes.

Example: Toss n dice. 6n possible outcomes.

49
 1.6 Counting Techniques

Addition Rule

Can use method A in nA ways.

Can use method B in nB ways.
If only one method can be used, you have nA + nB
ways of doing so.

Example: Go to Starbucks and have a mun (blue-

berry or oatmeal) or a bagel (sesame, plain, salt), but
not both. 2 + 3 = 5 choices.
50
 1.6 Counting Techniques

Permutations

Denition: An arrangement of n symbols in a denite

order is a permutation of the n symbols.

Example: How many ways to arrange the numbers

1,2,3? Answer: 6 ways 123, 132, 213, 231, 312,
321.

51
 1.6 Counting Techniques

Example: How many ways to arrange 1, 2, . . . , n?

(choose rst)(choose second) (choose nth)

n(n 1)(n 2) 2 1 = n!.

Example: Baseball manager has 9 players on his team.

Find the # of possible batting orders. Answer: 9! =
362880.

52
 1.6 Counting Techniques

Denition: The # of r-tuples we can make from n

dierent symbols (each used at most once) is called
the # of permutations of n things taken r-at-a-
time,
n!
Pn,r ().
(n r)!
Note that 0! = 1 and Pn,n = n!.

Example: How many ways can you take two symbols

from a, b, c, d? Ans: P4,2 = 4!/2! = 12 ab, ac, ad,
ba, bc, bd, ca, cb, cd, da, db, dc.
53
 1.6 Counting Techniques

Proof (of ()):

Pn,r = (choose rst)(second) (rth)

= n(n 1)(n 2) (n r + 1)
n(n 1) (n r + 1)(n r) 2 1
=
(n r) 2 1
n!
= .
(n r)!

54
 1.6 Counting Techniques

Example: How many ways to ll the rst 4 positions

of a batting order?

n = 9 players, r = 4 positions.

P9,4 = 9!/(9 4)! = 3024 ways.

55
 1.6 Counting Techniques

Example: How many of these 3024 ways has Smith

batting rst?

Method 1: First 4 positions: (Smith,?,?,?). This is

equiv to choosing 3 players from the remaining 8.

P8,3 = 8!/(8 3)! = 336 ways.

Method 2: Its clear that each of the 9 players is

equally likely to bat rst. Thus, 3024/9 = 336.
56
 1.6 Counting Techniques

Example: How many license plates of 6 digits can be

made from the numbers 1,2,. . . ,9. . .

(a) with no repetitions? (e.g., 123465) P9,6 = 9!/3! =

60480.

(b) allowing repetitions? (e.g., 123345 or 123465)

9 9 9 = 96 = 531441.

(c) containing repetitions? 53144160480 = 470961.

57
 1.6 Counting Techniques

Combinations

Suppose we only want to count the number of ways

to choose r out of n objects without regard to order,
i.e., count the number of dierent subsets of these n
objects that contain exactly r objects.

Example: How many subsets of {1, 2, 3} contain ex-

actly 2 elements? (Order isnt important.)

3 subsets {1, 2}, {1, 3}, {2, 3}

58
 1.6 Counting Techniques

Denition: The # of subsets with r elements of a set

with n elements is called the number of combinations
of n things taken r-at-a-time.

n

Notation: or Cn,r (read as n choose r). These
r
are also called binomial coecients.

59
 1.6 Counting Techniques

Dierence between permutations and combinations:

Combinations not concerned w/order: (a, b, c) =

(b, a, c).

Permutations concerned w/order: (a, b, c) = (b, a, c).

The number of permutations of n things taken r-at-

a-time is always as least as large as the number of
combinations. In fact,. . .
60
 1.6 Counting Techniques

Remark: Choosing a permutation is the same as rst

choosing a combination and then putting the ele-
ments in order, i.e.,

n! n
r!
=
(n r)! r
So

n

n!
= .
r (n r)!r!

n

n n n n n
= ,
= = 1, = = n.
r nr 0 n 1 n1
61
 1.6 Counting Techniques

Binomial Theorem:

n
 n
(x + y)n = xiy ni
i=0 i
This is where Pascals  comes from!

Corollary: Surprising fact:

n
 n = 2n .
i=0 i
Proof:

n
 n
2n = (1 + 1)n = 1i1ni.
i=0 i

62
 1.6 Counting Techniques

Example: An NBA team has 12 players. How many

ways can the coach choose the starting 5?


12 12!
= = 792.
5 5!7!

Example: Smith is one of the players on the team.

How many of the 792 starting line-ups include him?


11 11!
= = 330.
4 4!7!
(Smith gets one of the ve positions for free; there
are now 4 left to be lled by the remaining 11 players.)
63
 1.6 Counting Techniques

Example: 7 red shoes, 5 blues. Find the number of

arrangements.

R B R R B B R R R B R B

I.e., how many ways to put 7 reds in 12 slots?

12
Answer: .
7

64
 1.7 Counting Applications

Some applications of counting techniques.

Hypergeometric problems

Permutations vs. Combinations

Birthday problem

Poker probabilities

Multinomial coecients
65
 1.7 Counting Applications

Hypergeometric Distribution

You have a objects of type 1 and b objects of type 2.

Select n objects w/o replacement from the a + b.

Pr(k type 1s were picked)

(# ways to choose k 1s)(choose n k 2s)
=

# ways

to choose n out of a + b
a

b
k nk
= (the hypergeometric distrn).

a + b
n
66
 1.7 Counting Applications

Example: 25 sox in a box. 15 red, 10 blue. Pick 7

w/o replacement.


15 10
3 4
Pr(exactly 3 reds are picked) =

25
7

67
 1.7 Counting Applications

Permutations vs. Combinations Its all how you

approach the problem!

Example: 4 red marbles, 2 whites. Put them in a row

in random order. Find. . .

(a) Pr(2 end marbles are W)

(b) Pr(2 end marbles arent both W)
(c) Pr(2 Ws arent side by side)

68
 1.7 Counting Applications

Method 1 (using permutations): Let the sample space

S = {every random ordering of the 6 marbles}.

(a) A: 2 end marbles are W WRRRRW.

|A| 48 1
|A| = 2!4! = 48 Pr(A) = = = .
|S| 720 15

(b) Pr(A) = 1 Pr(A) = 14/15.

69
 1.7 Counting Applications

(c) B: 2 Ws side by side WWRRRR or RWWRRR

or . . . or RRRRWW

|B| = (# ways to select pair of slots for 2 Ws)

(# ways to insert Ws into pair of slots)
(# ways to insert Rs into remaining slots)
= 5 2! 4! = 240.

|B| 240 1
Pr(B) = = = .
|S| 720 3

But The above method took too much time! Heres

an easier way. . .
70
 1.7 Counting Applications

Method 2 (using combinations): Which 2 positions

do the Ws occupy? Now let

S = {possible pairs of slots that the Ws occupy}.

6
Clearly, |S| =
= 15.
2

(a) Since the Ws must occupy the end slots in order

for A to occur, |A| = 1 Pr(A) = |A|/|S| = 1/15.
(b) Pr(A) = 14/15.
(c) |B| = 5 Pr(B) = 5/15 = 1/3.
71
 1.7 Counting Applications

Birthday Problem

n people in a room. Find the prob that at least two

have the same birthday. (Ignore Feb. 29, and assume
that all 365 days have equal prob.)

A: All birthdays are dierent.

S = {(x1, . . . , xn) : xi = 1, 2, . . . , 365} (xi is person is

birthday), and note that |S| = (365)n.
72
 1.7 Counting Applications

|A| = P365,n = (365)(364) (365 n + 1)

(365)(364) (365 n + 1)
Pr(A) =
(365)n
364 363 365 n + 1
= 1
365 365 365
We want


364 363 365 n + 1
Pr(A) = 1 1
365 365 365

73
 1.7 Counting Applications

Notes: When n = 366, Pr(A) = 1.

For Pr(A) to be > 1/2, n must be 23. (surprising)

When n = 50, Pr(A) = 0.97.

74
 1.7 Counting Applications

Poker Problems

Draw 5 cards at random from a standard deck.

52
# of possible hands is |S| = = 2, 598, 960.
5

Terminology:
rank = 2, 3, . . . , Q, K, A,
suit = , , ,

75
 1.7 Counting Applications

(a) 2 pairs e.g., A, A, 3, 3, 10

13
Select 2 ranks (e.g., A, 3). Can do this ways.
2

4
Select 2 suits for rst pair (e.g., , ). ways.
2

4
Select 2 suits for second pair (e.g., , ). ways.
2
Select remaining card to complete the hand. 44 ways.

13 4 4
|2 pairs| = 44 = 123, 552
2 2 2

123, 552
Pr(2 pairs) = 0.0475.
2, 598, 960
76
 1.7 Counting Applications

(b) Full house (1 pair, 3-of-a-kind)

e.g., A, A, 3, 3, 3

Select 2 ordered ranks (e.g., A, 3). P13,2

ways.
4
Select 2 suits for pair (e.g., , ). ways.
2

4
Select 3 suits for 3-of-a-kind (e.g., , , ). ways.
3

4 4
|full house| = 13 12 = 3744
2 3

3744
Pr(full house) = 0.00144.
2, 598, 960
77
 1.7 Counting Applications

(c) Flush (all 5 cards from same suit)

4

Select a suit. ways.
1

13

Select 5 cards from that suit. ways.
5

5148
Pr(ush) = 0.00198.
2, 598, 960

78
 1.7 Counting Applications

(d) Straight (5 ranks in a row)

Select
a starting point for the straight (A, 2, 3, . . . , 10).

10
ways.
1

Select a suit for each card in the straight. 45 ways.

10 45
Pr(straight) = 0.00394.
2, 598, 960

79
 1.7 Counting Applications

(e) Straight ush

Select a starting point for the straight. 10 ways.

Select a suit. 4 ways.
40
Pr(straight ush) = 0.0000154.
2, 598, 960

80
 1.7 Counting Applications

Multinomial Coecients

Example:
n1 blue sox, n2 reds. # of assortments is
n + n2
1 (binomial coecients).
n1

k
Generalization (for k types of objects): n = i=1 ni

# of arrangements is n!/(n1!n2! nk !).

81
 1.7 Counting Applications

Example: How many ways can Mississippi be ar-

ranged?
# perms of 11 letters 11!
= = 34, 650.
(# ms)!(# ps)!(# is)!(# ss)! 1!2!4!4!

82
 1.8 Conditional Probability

Conditional Probability

Denition

Properties

Independence

83
 1.8 Conditional Probability

Conditional Probability

Example: Die. A = {2, 4, 6}, B = {1, 2, 3, 4, 5}. So

Pr(A) = 1/2, Pr(B) = 5/6.

Suppose we know that B occurs. Then the prob of

A given B is
2 |A B|
Pr(A|B) = =
5 |B|
So the prob of A depends on the info that you have!
The info that B occurs allows us to regard B as a
new, restricted sample space. And. . .
84
 1.8 Conditional Probability

|A B| |A B|/|S| Pr(A B)
Pr(A|B) = = = .
|B| |B|/|S| Pr(B)

Denition: If Pr(B) > 0, the conditional prob of A

given B is Pr(A|B) Pr(A B)/Pr(B).

Remarks: If A and B are disjoint, then Pr(A|B) = 0.

(If B occurs, theres no chance that A can also occur.)

What happens if Pr(B) = 0? Dont worry! In this

case, makes no sense to consider Pr(A|B).
85
 1.8 Conditional Probability

Example: Toss 2 dice and take the sum.

A: odd toss = {3, 5, 7, 9, 11}

B: {2, 3}
2 4 2 1
Pr(A) = Pr(3)+ +Pr(11) = + + + = .
36 36 36 2
1 2 1
Pr(B) = + = .
36 36 12
Pr(A B) Pr(3) 2/36
Pr(A|B) = = = = 2/3.
Pr(B) Pr(B) 1/12

86
 1.8 Conditional Probability

Example: 4 white sox, 8 red. Select 2 w/o repl.

A: 1st sock W; B: 2nd W; C: Both W (= A B).

4 3 1
Pr(C) = Pr(AB) = Pr(A)Pr(B|A) = = .
12 11 11

Pr(B) = Pr(A B) + Pr(A B)

= Pr(A)Pr(B|A) + Pr(A)Pr(B|A)
4 3 8 4 1
= + = .
12 11 12 11 3
Could you have gotten this result w/o thinking?
87
 1.8 Conditional Probability

A couple has two kids and at least one is a boy.

Whats the prob that BOTH are boys?

S = {GG, GB, BG, BB}, (BG means boy then girl)

C: Both are boys = {BB}.
D: At least 1 is a boy. = {GB, BG, BB}.
Pr(C D) Pr(C)
Pr(C|D) = = = 1/3.
Pr(D) Pr(D)
(My intuition was 1/2 the wrong answer! The
problem was that we didnt know whether D meant
the rst or second kid.)
88
 1.8 Conditional Probability

Properties analogous to Axioms of probability.

(1) 0 Pr(A|B) 1.

(2) Pr(S|B) = 1.

(3) A1 A2 = Pr(A1 A2|B) = Pr(A1|B) +

Pr(A2|B).

(4) If A1, A2, . . . are all disjoint, then




Pr Ai

B = Pr(Ai|B).
i=1 i=1
89
 1.8 Conditional Probability

Independence Day Any unrelated events are in-

dependent.

A: It rains on Mars tomorrow.

B: Coin lands on H.

Denition: A and B are independent i Pr(A B) =

Pr(A)Pr(B).

Example: If Pr(rains on Mars) = 0.2 and Pr(H) =

0.5, then Pr(rains and H) = 0.1.
90
 1.8 Conditional Probability

Note: If Pr(A) = 0, then A is indep of any other

event.

Remark: Events dont have to be physically unrelated

to be indep.

Example: Die. A = {2, 4, 6}, B = {1, 2, 3, 4}, A B =

{2, 4}, so Pr(A) = 1/2, Pr(B) = 2/3, Pr(AB) = 1/3.

Pr(A)Pr(B) = 1/3 = Pr(A B) A, B indep.

91
 1.8 Conditional Probability

More natural interpretation of independence. . .

Theorem: Suppose Pr(B) > 0. Then A and B are

indep i Pr(A|B) = Pr(A).

Proof: A, B indep Pr(A B) = Pr(A)Pr(B)

Pr(A B)/Pr(B) = Pr(A).

Remark: So if A and B are indep, the prob of A

doesnt depend on whether or not B occurs.
92
 1.8 Conditional Probability

(Bonus) Theorem: A, B indep A, B indep.

Proof: Pr(A) = Pr(A B) + Pr(A B), so that

Pr(A B) = Pr(A) Pr(A B)

= Pr(A) Pr(A)Pr(B) (A, B indep)
= Pr(A)[1 Pr(B)] = Pr(A)Pr(B).

93
 1.8 Conditional Probability

Dont confuse independence with disjointness!

Theorem: If Pr(A) > 0 and Pr(B) > 0, then A and B

cant be indep and disjt at the same time.

Proof: A, B disjt (A B = ) Pr(A B) = 0 <

Pr(A)Pr(B). Thus A, B not indep. Similarly, indep
implies not disjt.

94
 1.8 Conditional Probability

Extension to more than two events.

Denition: A, B, C are indep i

(a) Pr(A B C) = Pr(A)Pr(B)Pr(C)
and
(b) All pairs must be indep:
Pr(A B) = Pr(A)Pr(B)
Pr(A C) = Pr(A)Pr(C)
Pr(B C) = Pr(B)Pr(C)

95
 1.8 Conditional Probability

Note that condition (a) by itself isnt enough.

Example: S = {1, 2, . . . , 8} (each element w.p. 1/8).

A = {1, 2, 3, 4}, B = {1, 5, 6, 7}, C = {1, 2, 3, 8}.

(a) ABC = {1}. Pr(ABC) = Pr(A)Pr(B)Pr(C) =

1/8, so (a) is satised. However, (b) is not. . .

(b) AB = {1}. Pr(AB) = 1/8 = 1/4 = Pr(A)Pr(B).

96
 1.8 Conditional Probability

Also note that (b) by itself isnt enough.

Example: S = {1, 2, 3, 4} (each element w.p. 1/4).

A = {1, 2}, B = {1, 3}, C = {1, 4}. A B = A C =
B C = A B C = {1}.

(b) Pr(A B) = 1/4 = Pr(A)Pr(B). Same deal with

A, C and B, C. So (b) is OK. But (a) isnt. . .

(a) Pr(A B C) = 1/4 = 1/8 = Pr(A)Pr(B)Pr(C).

97
 1.8 Conditional Probability

General Denition: A1, . . . , Ak are indep i

Pr(A1 Ak ) = Pr(A1) Pr(Ak ) and
all subsets of {A1, . . . , Ak } are indep.

Independent Trials: Perform n trials of an experiment

such that the outcome of one trial is indep of the
outcomes of the other trials.

98
 1.8 Conditional Probability

Example: Flip 3 coins indeply.

(a) Pr(1st coin is H) = 1/2. Dont worry about the

other two coins since theyre indep of the 1st.

(b) Pr(1st coin H, 3rd T ) = Pr(1st coin H)Pr(3rd T ) =

1/4.

Remark: For indep trials, you just multiply the indi-

vidual probs.
99
 1.8 Conditional Probability

Example: Flip a coin innitely many times (each ip

is indep of the others).

pn Pr(1st H on nth trial)

= Pr(T T H)
 T 
n1
= Pr(T )Pr(T ) Pr(T ) Pr(H) = 1/2 n.
  
n1


Pr(H eventually) = pn = 1.
n=1

100
 1.9 Bayes Theorem

Bayes Theorem

Partitions

Bayes Theorem

Examples

101
 1.9 Bayes Theorem

Partition of a sample space split the sample

space into disjoint, yet all-encompassing subsets.

Denition: The events A1, A2, . . . , An form a partition

of the sample space S if
(1) A1, A2, . . . , An are disjoint.
n
(2) i=1 Ai = S.
(3) Pr(Ai) > 0 for all i.

102
 1.9 Bayes Theorem

Remark: When an experiment is performed, exactly

one of the Ais occurs.

Example: A and A form a partition.

Example: vowels and consonants form a partition

of the letters.

Example: Suppose A1, A2, . . . , An form a partition of

S, and B is some arbitrary event. Then
n

B = (Ai B).
i=1
103
 1.9 Bayes Theorem

So if A1, A2, . . . , An is a partition,

n 

Pr(B) = Pr (Ai B)
i=1
n

= Pr(Ai B) (A1, A2, . . . , An are disjoint)
i=1
n

= Pr(Ai)Pr(B|Ai) (by defn of condl prob).
i=1

This is the Law of Total Probability.

Example: Pr(B) = Pr(A)Pr(B|A) + Pr(A)Pr(B|A),

which we saw in the last module.
104
 1.9 Bayes Theorem

Bayes Theorem: If A1, A2, . . . , An form a partition of

S and B is any event, then
Pr(Ai B) Pr(Ai)Pr(B|Ai)
Pr(Ai|B) = = n .
Pr(B) j=1 Pr(Aj )Pr(B|Aj )

The Pr(Ai)s are prior probabilities (before B).

The Pr(Ai|B)s are posterior probabilities (after B).

The Pr(Ai|B) add up to 1. Thats why the funny-

looking denominator.
105
 1.9 Bayes Theorem

Example: In a certain city with good police,

Pr(Any defendent brought to trial is guilty) = 0.99.

In any trial,
Pr(Jury acquits if defendent is innocent) = 0.95.
Pr(Jury convicts if defendent is guilty) = 0.95.

Find Pr(Defendent is innocent|Jury acquits).

106
 1.9 Bayes Theorem

Events: I = innocent, G = guilty = I, A =

acquittal. Since the partition is {I, G}, Bayes
Pr(I)Pr(A|I)
Pr(I|A) =
Pr(I)Pr(A|I) + Pr(G)Pr(A|G)

(0.01)(0.95)
=
(0.01)(0.95) + (0.99)(0.05)

= 0.161.

Notice how the posterior probs depend strongly on

the prior probs.
107
 1.9 Bayes Theorem

Example: A store gets 1/2 of its items from Factory

1, 1/4 from Factory 2, and 1/4 from Factory 3.

2% of Factory 1s items are defective.

2% of Factory 2s items are defective.
4% of Factory 3s items are defective.

An item from the store is found to be bad. Find the

prob it comes from Factory 1. [Answer should be
< 1/2 since bad items favor Factory 3.]
108
 1.9 Bayes Theorem

Events: Fi = Factory i, D = defective item. Par-

tition is {F1, F2, F3}.
Pr(F1)Pr(D|F1)
Pr(F1|D) = 3
j=1 Pr(Fj )Pr(D|Fj )

(0.5)(0.02)
=
(0.5)(0.02) + (0.25)(0.02) + (0.25)(0.04)

= 0.4.

It turns out that Pr(F2|D) = 0.2 and Pr(F3|D) = 0.4.

109