DEPARTMENT OF STATISTICS

Chapter 4: Probability
 (Chapter 4.1 Experiments, Sample Spaces, Events)
• Experiments, Sample Spaces, Events
• Set Theory Definitions and Venn Diagrams
 (Chapter 4.2 Introduction to Probability)
• Defining Probability (Frequentist vs. Bayesian Points of View)
• Properties and Rules
 (Chapter 4.3 Conditional Probability and Independence)
• Conditional Probabilities, General Multiplication Rule, and Tree Diagrams
• Law of Partitions, Law of Total Probability, and Baye’s Rule
• Independence and Special Multiplication Rule
• (Additional Forms of Independence for Multiple Events)
1
 DEPARTMENT OF STATISTICS

Language of Probability
Set Theory
A (random) experiment is an activity in which there are at least two possible
outcomes, and the result of the activity cannot be predicted with absolute
certainty.
A trial corresponds to a single execution of a random experiment, representing
one instance of the process or activity.

An outcome () is the result or consequence of conducting a random experiment.

The sample space associated with an experiment is a representation of all the

possible outcomes. A sample space may be an enumeration of all conceivable
outcomes or an interval.
2
 DEPARTMENT OF STATISTICS

Language of Probability
Set Theory
An event is any set of outcomes from an experiment.
Events are indicated by capital Latin letters

A simple event is an event consisting of exactly one outcome.

An event has occurred if the resulting outcome is contained in the event.

The empty event/set is indicated by { } or

3
 DEPARTMENT OF STATISTICS

Sample Space 

Rolling a 20-sided die and viewing the top number is an act of performing
a single trial of a (random) experiment.

Events
Rolling a value of at least 18 is an event we can define it as a set .
The event rolling a perfect 20 defined as is a simple event as it contains only one outcome.

Suppose that a random experiment is performed, and the top number is revealed to be a 20.

This is the outcome of the random experiment, and we say that both the event and the event have
occurred.

4
 DEPARTMENT OF STATISTICS

Set Theory Visualization – Venn Diagrams

Venn Diagrams

We use them to visualize the sets within the sample space.

The outer rectangle denotes the sample space.

S
Each circle denotes a set within the sample space.

B
A

5
 DEPARTMENT OF STATISTICS

The event A complement, denoted A’, consists of all outcomes in the

sample space , not in A.

6
 DEPARTMENT OF STATISTICS

The event A union B denoted , consists of all outcomes in A or B or both.

Additionally, we say a collection of n events is
exhaustive if

The simplest exhaustive events are and

If there are events the union is written as

¿𝒊=𝟏¿𝒏𝑨𝒊 7
 DEPARTMENT OF STATISTICS

The event A intersection B, denoted by , consists of all outcomes in both A and B.

¿𝒊=𝟏¿𝒏𝑨𝒊
If there are events the intersection is written as

𝑨∩ 𝑩=∅ and are mutually exclusive or disjoint events. 8

 DEPARTMENT OF STATISTICS

The subset is a statement about the containment of one set within another denoted as .
The set has all its elements(outcomes) contained in another set labeled .

𝑪 ⊂ 𝑨⊂ 𝑫 𝑨∩ 𝑪 ∩ 𝑫=𝑪
9
 DEPARTMENT OF STATISTICS

Set Theory Operations

The event A complement, denoted A’, consists of all outcomes in the sample space , not in A.

The event A union B, denoted , consists of all outcomes in A or B or both.

The event A intersection B, denoted by , consists of all outcomes in both A and B.

If A and B have no elements in common, they are disjoint or mutually exclusive
events, written .

The subset is a statement about the containment of one set within another denoted as .
The set has all its elements(outcomes) contained in another set labeled .

10
 DEPARTMENT OF STATISTICS

F = set of face cards  F

C = set of clubs  C
S = set of spades  S
H = set of hearts  H
D = set of diamonds  D

Let denote
denotethetheset of of
set redqueens
cards. i.e., {, , , }
Ace Two
How can Three using
Fourthe Five Six andSeven Eight Nine Ten Jack Queen King
Which ofwe
thewrite
sets and is sets and
a subset of? set operations?

Clubs
𝑹=𝑯
𝑸 𝑫
⊂∪𝑭
Spades

Hearts

Diamonds
11
 DEPARTMENT OF STATISTICS

DeMorgan’s Laws
′
[ ¿𝒊 𝑨𝒊 ] =¿ 𝒊 𝑨𝒊 ′
DeMorgan’s First Law states that the complement of the union
equals the intersection of the complements.

12
 DEPARTMENT OF STATISTICS

DeMorgan’s Laws
′
[ ¿𝒊 𝑨𝒊 ] =¿ 𝒊 𝑨𝒊 ′
DeMorgan’s Second Law states that the complement of the
intersection equals the union of the complements.

13
DEPARTMENT OF STATISTICS

Probability (Discrete)

14
 DEPARTMENT OF STATISTICS

Probability as a Function + Axioms

Probability is a mathematical function mapping a set/event to a real number in the interval .
Function: Describes the chances of a set/event happening.
Input a set and evaluate the chances.

𝑷 𝑬() p Output a real value in .

1. For an event , .
Axioms 2. . Note:
3. For outcomes ,

15
 DEPARTMENT OF STATISTICS

But what is a Probability?

Frequentist Interpretation
The probability of any outcome of a chance process is the proportion of times
the outcome would occur in a very long series of repetitions.
Let be an event in some sample space then the probability of is defined as

𝒏𝑬
𝑷 ( 𝑬 ) ≈: lim (approximately defined)
𝒏→ ∞ 𝒏
where is the number of random experiments performed and is the number of times the
event occurs out of the trials. Rare Events Almost Sure Events

𝑷 (𝑬)≈𝟎 𝑷 ( 𝑬 ) ≈𝟏
https://digitalfirst.bfwpub.com/stats_applet/stats_applet_10_prob.html
16
 DEPARTMENT OF STATISTICS

But what is a Probability?

Bayesian Interpretation (Evidential Probability)
The probability of any outcome represents the degree of belief to which one
holds about the outcome of an event. The degree of belief can change in
response to new information.

The Bayesian interpretation starts from some logical prior belief about an
event and uses experiments to update the degree of belief.

The final degree of belief is called the posterior belief (final degree of belief).

17
 DEPARTMENT OF STATISTICS

Probability Rules
The complement rule, for any event

The general addition rule, for any events and

If the events are also mutually exclusive the general addition rule
simplifies to the basic addition rule
𝑷 ( 𝑨∪ 𝑩) =𝑷 ( 𝑨 ) +𝑷 ( 𝑩 )
18
 DEPARTMENT OF STATISTICS

Inclusion Exclusion Principle (3 Events)

𝑷 ( 𝑨∪ 𝑩∪ 𝑪 ) =𝑷 ( 𝑨 )+ 𝑷 ( 𝑩 ) +𝑷 ( 𝑪 ) − 𝑷 ( 𝑨∩ 𝑩 ) − 𝑷 ( 𝑨∩ 𝑪 ) − 𝑷 ( 𝑩∩ 𝑪 ) + 𝑷 ( 𝑨∩ 𝑩∩ 𝑪)

𝑷𝑷( 𝑨 𝑷
( 𝑨) +) +𝑷𝑷
(𝑩((𝑩
) ) )+ 𝑷 ( 𝑪 )
𝑨
Observe we have added the double intersections twice and the triple intersection three times.
If we remove each double intersection once. The triple intersection is removed 3 times.
So, we add back the triple intersection. 19
 DEPARTMENT OF STATISTICS

Dice Example
Suppose we roll 2 dice one time each.
The first die is 6-sided.
The second die is 4-sided.
{outcomes with a sum of 6}
{outcomes with a sum of 10}
{roll doubles}

1. If we consider the possible outcomes what is the sample space?

2. Write out the outcomes for each set.

3. Find the probability of each set.

4. Find ,

5. Find
20
 DEPARTMENT OF STATISTICS

Dice Example
Suppose we roll 2 dice one time each.
The first die is 6-sided.
The second die is 4-sided.
{outcomes with a sum of 6}
{outcomes with a sum of 10}
{roll doubles}

{ }
1. If we consider the possible outcomes what is the sample space? ( 𝟏 ,𝟏 ) , ( 𝟏 , 𝟐 ) , ( 𝟏 ,𝟑 ) , ( 𝟏 , 𝟒 ) ,
Consider the outcomes as pairs i.e., where ( 𝟐 ,𝟏 ) , ( 𝟐 , 𝟐 ) , ( 𝟐 ,𝟑 ) , ( 𝟐 , 𝟒 ) ,
(3,1) denotes that the 6-sided die is 3 and 𝛀 = ( 𝟑 ,𝟏 ) , ( 𝟑 , 𝟐 ) , ( 𝟑 ,𝟑 ) , ( 𝟑 , 𝟒 ) ,
the 4-sided die is 1. ( 𝟒 ,𝟏 ) , ( 𝟒 , 𝟐 ) , ( 𝟒 , 𝟑 ) , ( 𝟒 ,𝟒 ) ,
( 𝟓 ,𝟏 ) , ( 𝟓 , 𝟐 ) , ( 𝟓 ,𝟑 ) , ( 𝟓 , 𝟒 ) ,
( 𝟔 , 𝟏 ) , (𝟔 ,𝟐 ) , ( 𝟔 , 𝟑 ) ,(𝟔 , 𝟒)
Note: Every outcome is equally likely
For

21
 DEPARTMENT OF STATISTICS

Dice Example
Suppose we roll 2 dice one time each.
The first die is 6-sided.
The second die is 4-sided.
{outcomes with a sum of 6}

{ }
{outcomes with a sum of 10}
{roll doubles} ( 𝟏 ,𝟏 ) , ( 𝟏 , 𝟐 ) , ( 𝟏 ,𝟑 ) , ( 𝟏 , 𝟒 ) ,
( 𝟐 ,𝟏 ) , ( 𝟐 , 𝟐 ) , ( 𝟐 ,𝟑 ) , ( 𝟐 , 𝟒 ) ,

2. Write out the outcomes for each set. 𝛀 = ( 𝟑 ,𝟏 ) , ( 𝟑 , 𝟐 ) , ( 𝟑 ,𝟑 ) , ( 𝟑 , 𝟒 ) ,

( 𝟒 ,𝟏 ) , ( 𝟒 , 𝟐 ) , ( 𝟒 , 𝟑 ) , ( 𝟒 ,𝟒 ) ,
( 𝟓 ,𝟏 ) , ( 𝟓 , 𝟐 ) , ( 𝟓 ,𝟑 ) , ( 𝟓 , 𝟒 ) ,
𝑨=\{(2,4),(3,3),(4,2),(5,1)\} ( 𝟔 , 𝟏 ) , (𝟔 ,𝟐 ) , ( 𝟔 , 𝟑 ) ,(𝟔 , 𝟒)
{(6,4)}

{(1,1),(2,2),(3,3),(4,4)}

22
 DEPARTMENT OF STATISTICS

Dice Example
Suppose we roll 2 dice one time each.
The first die is 6-sided.
The second die is 4-sided.
{outcomes with a sum of 6}

{ }
{outcomes with a sum of 10}
{roll doubles} ( 𝟏 ,𝟏 ) , ( 𝟏 , 𝟐 ) , ( 𝟏 ,𝟑 ) , ( 𝟏 , 𝟒 ) ,
( 𝟐 ,𝟏 ) , ( 𝟐 , 𝟐 ) , ( 𝟐 ,𝟑 ) , ( 𝟐 , 𝟒 ) ,

3. Find the probability of each set. Recall 𝑷 ( 𝑬 )= ∑ 𝑷 ( 𝝎𝒊 ) 𝛀 = ( 𝟑 ,𝟏 ) , ( 𝟑 , 𝟐 ) , ( 𝟑 ,𝟑 ) , ( 𝟑 , 𝟒 ) ,

( 𝟒 ,𝟏 ) , ( 𝟒 , 𝟐 ) , ( 𝟒 , 𝟑 ) , ( 𝟒 ,𝟒 ) ,
𝒊 : 𝝎𝒊∈ 𝑬
𝟒 𝟏 ( 𝟓 ,𝟏 ) , ( 𝟓 , 𝟐 ) , ( 𝟓 ,𝟑 ) , ( 𝟓 , 𝟒 ) ,
𝑷 ( 𝑨 )= = ( 𝟔 , 𝟏 ) , (𝟔 ,𝟐 ) , ( 𝟔 , 𝟑 ) ,(𝟔 , 𝟒)
𝟐𝟒 𝟔 The outcomes are equally likely we can
compute a probability as
𝑷 (𝑩)=
𝟏 𝑨=\{(2,4),(3,3),(4,2),(5,1)\}
𝟐𝟒
{(6,4)}
𝟒 𝟏
𝑷 ( 𝑫 )= = {(1,1),(2,2),(3,3),(4,4)}
𝟐𝟒 𝟔
23
 DEPARTMENT OF STATISTICS

Dice Example
Suppose we roll 2 dice one time each.
The first die is 6-sided.
The second die is 4-sided.
{outcomes with a sum of 6}

{ }
{outcomes with a sum of 10}
{roll doubles} ( 𝟏 ,𝟏 ) , ( 𝟏 , 𝟐 ) , ( 𝟏 ,𝟑 ) , ( 𝟏 , 𝟒 ) ,
( 𝟐 ,𝟏 ) , ( 𝟐 , 𝟐 ) , ( 𝟐 ,𝟑 ) , ( 𝟐 , 𝟒 ) ,

4. Find , . 𝛀 = ( 𝟑 ,𝟏 ) , ( 𝟑 , 𝟐 ) , ( 𝟑 ,𝟑 ) , ( 𝟑 , 𝟒 ) ,
( 𝟒 ,𝟏 ) , ( 𝟒 , 𝟐 ) , ( 𝟒 , 𝟑 ) , ( 𝟒 ,𝟒 ) ,
( 𝟓 ,𝟏 ) , ( 𝟓 , 𝟐 ) , ( 𝟓 ,𝟑 ) , ( 𝟓 , 𝟒 ) ,
Because they are ( 𝟔 , 𝟏 ) , (𝟔 ,𝟐 ) , ( 𝟔 , 𝟑 ) ,(𝟔 , 𝟒)
𝟒 𝟏
disjoint events! 𝑷 ( 𝑨 )= =
𝟐𝟒 𝟔
𝟏 𝑨=\{( 2,4),(3,3),( 4,2),(5,1)\}
𝑷 (𝑩)=
𝟏 𝟐𝟒 {(6,4)}
𝑷 ( 𝑨∩ 𝑫 ) =𝑷 ( {𝟑 , 𝟑}) =
𝟐𝟒 𝟒 𝟏 {(1,1),(2,2),(3,3),(4,4)}
𝑷 ( 𝑫 )= =
𝟐𝟒 𝟔
24
 DEPARTMENT OF STATISTICS

Dice Example
Suppose we roll 2 dice one time each.
The first die is 6-sided.
The second die is 4-sided.
{outcomes with a sum of 6}

{ }
{outcomes with a sum of 10}
{roll doubles} ( 𝟏 ,𝟏 ) , ( 𝟏 , 𝟐 ) , ( 𝟏 ,𝟑 ) , ( 𝟏 , 𝟒 ) ,
( 𝟐 ,𝟏 ) , ( 𝟐 , 𝟐 ) , ( 𝟐 ,𝟑 ) , ( 𝟐 , 𝟒 ) ,

5. Find . 𝛀 = ( 𝟑 ,𝟏 ) , ( 𝟑 , 𝟐 ) , ( 𝟑 ,𝟑 ) , ( 𝟑 , 𝟒 ) ,
( 𝟒 ,𝟏 ) , ( 𝟒 , 𝟐 ) , ( 𝟒 , 𝟑 ) , ( 𝟒 ,𝟒 ) ,
is non-doubles. ( 𝟓 ,𝟏 ) , ( 𝟓 , 𝟐 ) , ( 𝟓 ,𝟑 ) , ( 𝟓 , 𝟒 ) ,
𝟒 𝟏 ( 𝟔 , 𝟏 ) , (𝟔 ,𝟐 ) , ( 𝟔 , 𝟑 ) ,(𝟔 , 𝟒)
𝑷 ( 𝑨 )= =
𝟑 𝟏 𝟐𝟒 𝟔
𝑷 ( 𝑫 ′ ∩ 𝑨 )= = 𝑨=\{(2,4),(3,3),(4,2),(5,1)\}
𝟐𝟒 𝟖 𝟏
𝑷 (𝑩)=
𝟐𝟒 {(6,4)}
𝟒 𝟏 {(1,1),(2,2),(3,3),(4,4)}
𝑷 ( 𝑫 )= =
𝟐𝟒 𝟔
25
 DEPARTMENT OF STATISTICS

Conditional Probability (Discrete)

1. Conditional Probabilities
2. Tree Diagrams
3. General Multiplication Rule
4. Law of Partitions
5. Law of Total Probability
6. Bayes’ Rule

26
 DEPARTMENT OF STATISTICS

Conditional Probability
Conditional probability for 2 events
Is the probability that event occurs given the knowledge that event occurred.
𝑷 ( 𝑩 ∩ 𝑨)
𝑷 ( 𝑩) ≠ 𝟎 𝑷 ( 𝑨|𝑩 )=
𝑷 ( 𝑩)

(Set Minus Operation)

27
 DEPARTMENT OF STATISTICS

Conditional Card Example

A card is randomly selected from a 52-card deck.

Clubs

Spades

Condition restricts the sample Hearts

space

Diamonds

What is the probability that the card is a Queen of Hearts?

𝟏
Let and 𝑷 ( 𝑸 ∩ 𝑯 )=
𝟓𝟐
What is the probability that the card is a Queen? 𝟒
𝑷 ( 𝑸 )=
𝟓𝟐
If you are informed that the card is a heart. What is the probability that the card is a Queen?
𝟏
𝑷 ( 𝑸∨ 𝑯 ) =
𝟏𝟑 28
 DEPARTMENT OF STATISTICS

General Multiplication Rule

General Multiplication rule (2 Events)
𝑷 ( 𝑨∩ 𝑩) =𝑷 ( 𝑨 ) 𝑷 ( 𝑩| 𝑨 ) =𝑷 ( 𝑩 ) 𝑷 ( 𝑨∨ 𝑩)

General Multiplication rule (3 events)

𝑷 ( 𝑨∩ 𝑩∩ 𝑪 ) =𝑷 ( 𝑨 ) 𝑷 ( 𝑩| 𝑨 ) 𝑷 ( 𝑪| 𝑨 ∩ 𝑩 )

29
DEPARTMENT OF STATISTICS

Conditional Probability Tree Diagrams

30
 DEPARTMENT OF STATISTICS

Example (Tree Diagram)

Glen and Jia are going to Indianapolis this weekend for one day. They are twice as likely to go on
Sunday as they are on Friday, and they are three times as likely to go on Saturday as they are on
Friday. There is a 45% chance of rain on Friday, a 25% chance of rain on Saturday, and a 30% chance
of rain on Sunday.

Clearly, define the symbols

They went to Indianapolis on Friday
They went to Indianapolis on Saturday
They went to Indianapolis on Sunday
It rained during their trip.

1. Determine the following probabilities:

2. What is the probability that it rains while they are Indianapolis?
3. Given that it did not rain, what is the probability that they went on Friday?

Simplifying assumption  if it rains on the day that they go, it will be during the time they are there.
31
 DEPARTMENT OF STATISTICS

Glen and Jia are going to Indianapolis this weekend for one day. They are twice as likely to go on
Sunday as they are on Friday, and they are three times as likely to go on Saturday as they are on
Friday. There is a 45% chance of rain on Friday, a 25% chance of rain on Saturday, and a 30% chance
of rain on Sunday.

1. Determine the following probabilities:

𝑷 ( 𝑭𝒓𝒊 ∪ 𝑺𝒂𝒕 ∪𝑺𝒖𝒏 ) =𝟏

𝟏
𝑷 ¿ 𝑷 ( 𝑭𝒓𝒊 )=
𝟔

𝑷 ¿

32
 DEPARTMENT OF STATISTICS

Glen and Jia are going to Indianapolis this weekend for one day. They are twice as likely to go on
Sunday as they are on Friday, and they are three times as likely to go on Saturday as they are on
Friday. There is a 45% chance of rain on Friday, a 25% chance of rain on Saturday, and a 30% chance
of rain on Sunday.

2. What is the probability that it rains while they are Indianapolis?

33
 DEPARTMENT OF STATISTICS

Glen and Jia are going to Indianapolis this weekend for one day. They are twice as likely to go on
Sunday as they are on Friday, and they are three times as likely to go on Saturday as they are on
Friday. There is a 45% chance of rain on Friday, a 25% chance of rain on Saturday, and a 30% chance
of rain on Sunday. 2. What is the probability that it rains while they are Indianapolis?
𝟏
𝑷 ( 𝑹 ∩ 𝑺𝒖𝒏 )= 𝑷 ( 𝑹|𝑺𝒖𝒏 ) 𝑷 ( 𝑺𝒖𝒏 ) =
𝟏𝟎

𝟕
𝑷 ( 𝑹′ ∩𝑺𝒖𝒏 ) =𝑷 ( 𝑹′|𝑺𝒖𝒏 ) 𝑷 ( 𝑺𝒖𝒏 )=
𝟑𝟎
𝟏
𝑷 ( 𝑹 ∩ 𝑺𝒂𝒕 ) =𝑷 ( 𝑹|𝑺𝒂𝒕 ) 𝑷 ( 𝑺𝒂𝒕 ) =
𝟖
𝟑
𝑷 ( 𝑹 ′ ∩𝑺𝒂𝒕 )=𝑷 ( 𝑹′ |𝑺𝒂𝒕 ) 𝑷 ( 𝑺𝒂𝒕 )=
𝟖
𝟑
𝑷 ( 𝑹 ∩ 𝑭𝒓𝒊 )= 𝑷 ( 𝑹| 𝑭𝒓𝒊 ) 𝑷 ( 𝑭𝒓𝒊 )=
𝟒𝟎
𝟏𝟏
𝑷 ( 𝑹 ′ ∩ 𝑭𝒓𝒊 ) =𝑷 ( 𝑹 ′ |𝑭𝒓𝒊 ) 𝑷 ( 𝑭𝒓𝒊 )= 34
𝟏𝟐𝟎
 DEPARTMENT OF STATISTICS

Glen and Jia are going to Indianapolis this weekend for one day. They are twice as likely to go on
Sunday as they are on Friday, and they are three times as likely to go on Saturday as they are on
Friday. There is a 45% chance of rain on Friday, a 25% chance of rain on Saturday, and a 30% chance
of rain on Sunday. 2. What is the probability that it rains while they are Indianapolis?

𝟏
𝑷 ( 𝑹 ∩ 𝑺𝒖𝒏 )= 𝑷 ( 𝑹|𝑺𝒖𝒏 ) 𝑷 ( 𝑺𝒖𝒏 ) =
𝟏𝟎

𝟏
𝑷 ( 𝑹 ∩ 𝑺𝒂𝒕 ) =𝑷 ( 𝑹|𝑺𝒂𝒕 ) 𝑷 ( 𝑺𝒂𝒕 ) =
𝟖

𝟑
𝑷 ( 𝑹 ∩ 𝑭𝒓𝒊 )= 𝑷 ( 𝑹| 𝑭𝒓𝒊 ) 𝑷 ( 𝑭𝒓𝒊 )=
𝟒𝟎

35
 DEPARTMENT OF STATISTICS

Glen and Jia are going to Indianapolis this weekend for one day. They are twice as likely to go on
Sunday as they are on Friday, and they are three times as likely to go on Saturday as they are on
Friday. There is a 45% chance of rain on Friday, a 25% chance of rain on Saturday, and a 30% chance
of rain on Sunday.
3. Given that it did not rain, what is the probability that they went on Friday?

𝑷 ( 𝑹 )=𝟎 . 𝟑
𝟏𝟏 𝑷 ( 𝑭𝒓𝒊 ∨ 𝑹 ′ )𝑷=( 𝑭𝒓𝒊𝑷 𝟏𝟏/𝟏𝟐𝟎
∩ 𝑹(′ ) 𝑭𝒓𝒊 ∩𝑹 𝟏𝟏 ) ′

𝑷 ( 𝑹 ′ ∩ 𝑭𝒓𝒊 ) = =¿ = ≈ 𝟎=¿
. 𝟏𝟑𝟏𝟎
𝟏𝟐𝟎 𝟏 − 𝑷 ( 𝑹) 𝑷𝟕/𝟏𝟎 ( 𝑹 ) 𝟖𝟒
′

36
 DEPARTMENT OF STATISTICS

Law of Partitions
for all such that 𝛀=¿𝒊=𝟏¿𝒏𝑨𝒊
A partition of a space is formed by events that are mutually exclusive (disjoint), as well as exhaustive.
The Law of Partitions states that for a collection of non-empty events
that form a partition of and for any

𝑨𝟏 𝑨𝟐 𝑨𝟑
𝑩
𝑨𝟏 ∩ 𝑩 𝑨𝟐 ∩ 𝑩 𝑨𝟑 ∩ 𝑩

Note: The simplest partition is any set and its complement . 37

 DEPARTMENT OF STATISTICS

Law of Total Probability

The Law of Total Probability states that for a collection of non-empty

events that form a partition of and for any

38
 DEPARTMENT OF STATISTICS

Bayes’ Rule
𝑷 ( 𝑩) ≠ 𝟎
Bayes’ Rule states that for a collection of non-empty events that form a
partition of and for any event

General Multiplication Rule

Numerator
Conditional Probability Rule
Law of Total Probability
Denominator

39
 DEPARTMENT OF STATISTICS

𝑷 ( 𝑩| 𝑨 𝒊 ) 𝑷 ( 𝑨𝒊 )
Bayes’ Rule 𝑷 ( 𝑨𝒊 ∨ 𝑩 )= 𝑵 𝑷 ( 𝑩) ≠ 𝟎
∑ 𝑷 ( 𝑩∨ 𝑨 𝒋 ) 𝑷 ( 𝑨 𝒋 )
𝒋=𝟏

𝑷 ( 𝑨𝒊 ) Prior belief or understanding. Prior

Probability

Posterior
𝑷 ( 𝑨𝒊 ∨𝑩 ) The probability we are truly interested in.
Probability

𝑷 ( 𝑩| 𝑨𝒊 ) Information importance. Likelihood

𝑵
𝑷 ( 𝑩 ) = ∑ 𝑷 ( 𝑩∨𝑨 𝒋 ) 𝑷 ( 𝑨 𝒋 ) Information also a rescaling factor (space restriction to )

𝒋=𝟏
40
 DEPARTMENT OF STATISTICS

Simplified Form of Bayes

Two Event Bayes’ Rule states that for two non-empty

events and in a sample space .

41
 DEPARTMENT OF STATISTICS

𝑷 ¿
is defined as a person
Bayes’ Rule has a specific disease

2-Sets
is defined as a
positive test result

𝑷 (𝑫) Example: Propensity of the disease in the population.

Prior
Probability

Posterior
𝑷 ¿ Example: The probability of having the disease
given a positive test . Probability

𝑷 ( +¿ 𝑫 ) Example: Probability of testing positive given a

person has the disease .
Likelihood

𝑷¿ Example: Probability of getting a positive test result

regardless if a person has the disease or not.
42
 DEPARTMENT OF STATISTICS

𝑷 ( 𝑩) ≠ 𝟎 Bayesian Statistics
Bayes’ Rule states that for a collection of non-empty events that form a
partition of and for any

The power of Bayes’ Rule comes from the ability to update the prior using information.

43
 DEPARTMENT OF STATISTICS

Coin Flip Example

Suppose you have a bag of ten coins.
Of these ten coins one of the coins is known to be biased with a probability of heads equal to
0.8 the remaining coins are fair with a probability of heads equal to 0.5.
You reach into the bag and grab a coin, you flip the coin ten times, resulting in the sequence .
What is the probability you got the biased coin given the information?
Let denote that the coin was heads on the flip.

Let denote the bias coin was selected.

44
 DEPARTMENT OF STATISTICS

Coin Flip Example

Let denote that the coin was heads on the flip.

Let denote the bias coin was selected.

{ 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 ,𝑯 , 𝑯 , 𝑯 }
Flip1:

𝑷 ( 𝑯 𝟏|𝑩 ) 𝑷 ( 𝑩) 𝟖
×
𝟏
𝑷 ( 𝑩| 𝑯 𝟏 ) = 𝟏𝟎 𝟏𝟎
=¿
𝑷 ( 𝑯 𝟏|𝑩 ) 𝑷 ( 𝑩 ) + 𝑷 ( 𝑯𝟏 𝑩|
𝟖′
𝟏𝟎 )𝟏
×𝑷 ( +
𝑩
𝟏𝟎 𝟐
𝟏′
)
×
𝟗
𝟏𝟎

𝟖
𝑷 ( 𝑩| 𝑯 𝟏 ) =
𝟓𝟑

45
 DEPARTMENT OF STATISTICS

Coin Flip Example

Let denote that the coin was heads on the flip.

Let denote the bias coin was selected.

{ 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 ,𝑯 , 𝑯 , 𝑯 }
𝑷 ( 𝑯 𝟏 ∩ 𝑯 𝟐∨ 𝑩 ) 𝑷 ( 𝑩)
Flip2: 𝑷 ( 𝑩|{ 𝑯 𝟐 , 𝑯 𝟏 } ) =
𝑷 ( 𝑯 𝟏 ∩ 𝑯 𝟐|𝑩 ) 𝑷 ( 𝑩 ) + 𝑷 ( 𝑯 𝟏 ∩ 𝑯 𝟐| 𝑩 ) 𝑷 ( 𝑩 )
′ ′ 𝟖
𝑷 ( 𝑩| 𝑯 𝟏 ) =
Special Property
𝟓𝟑
(Conditional Independence)
𝑷 ( 𝑯 𝟐∨𝑩 ) 𝑷 ( 𝑯 𝟏 ∨𝑩) 𝑷 ( 𝑩)
¿ ′
𝑷 ( 𝑯 𝟐∨ 𝑩) 𝑷 (𝑯 𝟏∨𝑩) 𝑷 ( 𝑩) + 𝑷 ( 𝑯 𝟐∨𝑩 ′ ) 𝑷 (𝑯 𝟏∨𝑩 ′) 𝑷 ( 𝑩 )

𝑷 ( 𝑯 𝟐∨𝑩 ) 𝑷 ( 𝑩∨𝑯 𝟏) 𝑷 (𝑯 𝟏 )
Check this ¿
𝑷 ( 𝑯 𝟐∨ 𝑩) 𝑷 (𝑩∨ 𝑯 𝟏 ) 𝑷 (𝑯 𝟏)+ 𝑷 ( 𝑯 𝟐∨𝑩′ ) 𝑷 (𝑩′ ∨𝑯 𝟏 ) 𝑷 ( 𝑯 𝟏)
𝟔𝟒
𝑷 ( 𝑩|{ 𝑯 𝟐 , 𝑯 𝟏 } ) = 𝑷 ( 𝑯 𝟐∨𝑩 ) 𝑷 ( 𝑩∨𝑯 𝟏) Update Rule:
𝟐𝟖𝟗 ¿
𝑷 ( 𝑯 𝟐∨ 𝑩) 𝑷 (𝑩∨ 𝑯 𝟏 )+𝑷 ( 𝑯 𝟐∨ 𝑩′ ) ( 𝟏 − 𝑷 (𝑩∨ 𝑯 𝟏) ) Prior has been replaced.
46
 DEPARTMENT OF STATISTICS

Coin Flip Example

Let denote that the coin was heads on the flip.

Let denote the bias coin was selected.

{ 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 , 𝑯 ,𝑯 , 𝑯 , 𝑯 }
𝑷 ( 𝑯 𝟑 ∨𝑩 ) 𝑷 ( 𝑩|{ 𝑯 𝟐 , 𝑯 𝟏 } )
Flip3: 𝑷 ( 𝑩|{ 𝑯 𝟑 , 𝑯 𝟐 , 𝑯 𝟏 } ) = 𝟖
𝑷 ( 𝑯 𝟐∨ 𝑩 ) 𝑷 ( 𝑩|{ 𝑯 𝟐 , 𝑯 𝟏 } ) + 𝑷 ( 𝑯 𝟐∨ 𝑩′ ) (𝟏 − 𝑷 ( 𝑩|{ 𝑯 𝟐 , 𝑯 𝟏 } ) ) 𝑷 ( 𝑩| 𝑯 𝟏 ) =
𝟓𝟑
𝟖 𝟔𝟒
×
𝟔𝟒
𝑷 ( 𝑩|{ 𝑯 𝟐 ,𝑯 𝟏 })=
𝟏𝟎 𝟐𝟖𝟗
¿
𝟖
×
𝟔𝟒
𝟏𝟎 𝟐𝟖𝟗 𝟐
𝟏
+ × 𝟏−
𝟔𝟒
𝟐𝟖𝟗 ( ) 𝟐𝟖𝟗
𝟓𝟏𝟐
𝑷 ( 𝑩|{ 𝑯 𝟑 , 𝑯 𝟐 , 𝑯 𝟏 } ) =
𝟏𝟔𝟑𝟕

47
 DEPARTMENT OF STATISTICS

Coin Flip Example

Repeat the process:

Flip4, Flip5,…Flip9 and Flip10:

𝑷 ( 𝑩|{ 𝑯 𝟏𝟎 , 𝑯 𝟗 , 𝑯 𝟖 , 𝑯 𝟕 , 𝑯 𝟔 , 𝑯 𝟓 , 𝑯 𝟒 , 𝑯 𝟑 , 𝑯 𝟐 , 𝑯 𝟏 } ) ≈ 𝟎 . 𝟗𝟐𝟒𝟑

Simple Example of Bayesian Updating.

(Questions)
1. What if a tails would have occurred at some point during the process? What
would be the result?’

2. This is a simple example. Could it have been done without these updates?

48
DEPARTMENT OF STATISTICS

Independence of Events

49
 DEPARTMENT OF STATISTICS

Independence
We say two events and are independent if the occurrence of one event does not
affect the probability of the other.

𝑷 ( 𝑨|𝑩 )= 𝑷 ( 𝑨 ) 𝑷 ( 𝑩| 𝑨 )= 𝑷 ( 𝑩 )
General Multiplication rule 𝑷 ( 𝑨∩ 𝑩) =𝑷 ( 𝑨 ) 𝑷 ( 𝑩| 𝑨 )
If Independent

Special Multiplication Rule 𝑷 ( 𝑨∩ 𝑩) =𝑷 ( 𝑨 ) 𝑷 (𝑩)

and are independent
if and only if
and are independent
and are independent
and are independent 50
 DEPARTMENT OF STATISTICS

Mutually Exclusive vs. Independence

𝑨∩ 𝑩=∅
and are mutually exclusive or disjoint events if
Note:
and are independent if 𝑷 ( 𝑨|𝑩 )= 𝑷 ( 𝑨 )
Given two non-empty events in a sample space . 𝑷 ( 𝑨∩ 𝑩)
𝑷 ( 𝑨|𝑩 )= =𝟎 ≠ 𝑷 ( 𝑨)
Suppose that and are mutually exclusive then the 𝑷 ( 𝑩)
two events cannot be independent.

Suppose that and are independent then the

Since
two events cannot be mutually exclusive.
and
51
 DEPARTMENT OF STATISTICS

Circuit Example
System 1 System 2

Suppose the following systems will function if current flows from point
A to point B.

If each switch (break in the line) is activated independently (on) with

probability , what is the probability the system functions?
52
 DEPARTMENT OF STATISTICS

Circuit Example (System 1)

Let denote that the line is on (current flows along path).

Let denote that system is functioning.

Demorgan’s Law Independence

𝑷¿ ¿𝟏−𝑷 ([ ¿ 𝒊=𝟏¿𝟒𝒍𝒊 ] ) ¿𝟏−𝑷 (¿ 𝒊=𝟏¿ 𝟒𝒍𝒊′ )

′ 𝟒
¿ 𝟏 − ∏ 𝑷 ( 𝒍𝒊 ′ )
𝒊=𝟏

Complement Rule
𝟒
¿ 𝟏 − ( 𝟎 . 𝟕 ) =𝟎53
.𝟕𝟓𝟗𝟗
 DEPARTMENT OF STATISTICS

Circuit Example (System 2)

Let denote that the line is on (current flows along path).
Let denote that the switch on the line is on.
Let denote that system is functioning.

𝑷 ( 𝒍𝟐 ′ ) =𝟎 . 𝟕 𝑷 ( 𝒍𝟏 ′ ) = 𝑷 ( 𝒔 ∪ 𝒔 ′ ) =𝟏− 𝟎.𝟑
𝟏′
𝟏
𝟐
𝟏
𝟐

𝟑
𝑷 ( 𝒍𝟑 ′ ) =𝟏 − 𝟎 .𝟑
Demorgan’s Law Independence

𝑷¿ ¿𝟏−𝑷 ([ ¿ 𝒊=𝟏¿𝟑𝒍𝒊 ] ) ¿𝟏−𝑷 (¿ 𝒊=𝟏¿𝟑𝒍𝒊 ′ )

′ 𝟑
¿ 𝟏 − ∏ 𝑷 ( 𝒍𝒊 ′ )
𝒊=𝟏

Complement Rule
¿ 𝟏 − 𝟏− 𝟎 .𝟑 × 𝟎 .𝟕 × 𝟏 −𝟎 . 𝟑 ) ≈ 𝟎 .𝟑𝟖𝟎𝟐
( 𝟐
) ( ) ( 𝟑
54
 DEPARTMENT OF STATISTICS

Pair-wise Independence
We say a collection of events are pair-wise independent if all pairs of
events are independent.

Example:
Suppose you have a collection of four events These four events are
pair-wise independent if

55
 DEPARTMENT OF STATISTICS

Mutual Independence
We say a collection of events are mutually independent if the special multiplication rule is
satisfied for all combinations of the events.

Example:
Suppose you have a collection of four events These four events are
mutually independent if

𝑷 ( 𝑨𝟏 ∩ 𝑨𝟐 ∩ 𝑨𝟑 ∩ 𝑨𝟒 ) =𝑷 ( 𝑨𝟏 ) 𝑷 ( 𝑨 𝟐 ) 𝑷 ( 𝑨𝟑) 𝑷 ( 𝑨 𝟒)
56