Chapter 3: Elements of Probability

Li-Pang Chen, PhD

Department of Statistics, National Chengchi University

©Fall 2024

1 / 48
Outlines

1 Introduction

2 Sample Space, Events, and Venn Diagrams

3 Probability

4 Conditional Probability

5 Bayes’ Theorem

2 / 48
3.1 Introduction

Definition
Probability is a numerical measure of the likelihood that an event will
occur.
Example 1: When tossing a die, what is a possibility to have six points?

Example 2: When tossing two dice, what is a possibility that a sum of

two outcomes is equal to 8?

3 / 48
3.1 Introduction

Those two examples contain two important concepts: sample space

and event.
Under sample space and events, we can define (classical) probability.

4 / 48
3.2 Sample Space, Events, and Venn Diagrams

Definition (Experiment)
A process that leads to the occurrence of one and only one of several
possible results.

Definition (Outcome)
A particular result of an experiment.

Definition (Sample space)

It is a set containing all possible outcomes of an experiment. Denote S as
the sample space.

Definition (Event)
A subset of the sample space.

5 / 48
3.2 Sample Space, Events, and Venn Diagrams

Example 1: Roll a die.

Experiment: roll a die.
Outcome: 1,2,3,4,5,6.
Sample space: {1, 2, 3, 4, 5, 6}.
Event:
(a) Observe an even number ⇒ {2, 4, 6}.
(b) Observe a number 3 or less ⇒ {1, 2, 3}.

6 / 48
3.2 Sample Space, Events, and Venn Diagrams

Example 2:
Experiment: the running of a race among 7 horses having post
position 1, 2, · · · , 7.
Outcome: (1, 2, · · · , 7), (2, 1, · · · , 7), · · · , (7, 6, · · · , 2, 1).
Sample space: {all ordering of (1, 2, · · · , 7)}.
Event: Observe the #2 horse is first and the #3 horse is second.

7 / 48
3.2 Sample Space, Events, and Venn Diagrams

In Example 1, we know that given a sample space, there exists at

least one event set.
Suppose that E and F are two events sets from S.
We now introduce some operations of two sets E and F .

8 / 48
3.2 Sample Space, Events, and Venn Diagrams

Definition (Union)
The union of E and F , denoted as E ∪ F , consists of all outcomes that are
either in E or F or both.

Definition (Intersection)
The intersection of E and F , denoted as E ∩ F , consists of all outcomes
that are only in E and F .

9 / 48
3.2 Sample Space, Events, and Venn Diagrams

Example 3:
Continue to Example 1, let E denote the first event and F denote the
second event. Then
E ∪ F = {1, 2, 3, 4, 6}
E ∩ F = {2}.

10 / 48
3.2 Sample Space, Events, and Venn Diagrams

Remark:
Those two definitions can be extended to m events, say E1 , · · · , Em .
That is,
m
S
(a) E1 ∪ · · · ∪ Em ≜ Ei .
i=1
m
T
(b) E1 ∩ · · · ∩ Em ≜ Ei .
i=1
If E ∩ F does not contain any outcome, then we say it’s an empty set,
denoted as ϕ. Moreover, if E ∩ F = ϕ, then we say E and F are
mutually exclusive.

11 / 48
3.2 Sample Space, Events, and Venn Diagrams

Definition (Complement)
For any event E , let E c denote the complement of E , which consists all
outcomes in the sample space S but NOT in E .

Example 4:
Let E denote the first event in Example 1. Then E c = {1, 3, 5} = observe
an odd number.
Remark:
E ∩ Ec =
E ∪ Ec =

12 / 48
3.2 Sample Space, Events, and Venn Diagrams

Definition (Subset)
For any two events E and F , we say E is contained in F , denoted as
E ⊂ F , if all of the outcomes in E are also in F .
?
Remark: How to check two sets are equal, i.e., E = F for some sets E
and F ?

13 / 48
3.2 Sample Space, Events, and Venn Diagrams
Venn diagrams: It is used to illustrate the relationship among sets.

14 / 48
3.2 Sample Space, Events, and Venn Diagrams

By Venn diagrams, we can show the following properties of operations:

(a) Commutative law:

E ∪ F = F ∪ E; E ∩ F = F ∩ E

(b) Associative law:

(E ∪ F ) ∪ G = E ∪ (F ∪ G ); (E ∩ F ) ∩ G = E ∩ (F ∩ G )

(c) Distributive law:

(E ∪ F ) ∩ G = (E ∩ G ) ∪ (F ∩ G ); (E ∩ F ) ∪ G = (E ∪ G ) ∩ (F ∪ G )

(d) DeMorgan’s law:

(E ∪ F )c = E c ∩ F c ; (E ∩ F )c = E c ∪ F c

15 / 48
3.3 Probability

Suppose that S is the sample space and E is an event.

The probability of having an event E , denoted as P(E ), is defined as

Number of favorable outcomes for E

P(E ) = .
Total number of possible outcomes for S
It may be interpreted as the proportion of having an event E among
all possible outcomes in S.

16 / 48
3.3 Probability

Example 5: Toss two dice.

What is the probability that the sum of two dices is equal to 8?

Method 1: List all possible outcomes, and then find outcomes that satisfy
an event.

Method 2: Apply the calculation of “combination”.

17 / 48
3.3 Probability

Method 2:

Recall: For all n subjects, if we would like to select arbitrary k subjects

without ordering, then we have
n!
Ckn =
k!(n − k)!

combinations;
if we want to select k subjects with ordering, then we have
n!
Pkn =
k!
permutations;
if n subjects can be repeatedly selected k times, then we have nk possible
outcomes.
18 / 48
3.3 Probability

Move back to Example 5.

Since six numbers can repeatedly appear 2 times, we have 62 = 36
possible outcomes.
5
⇒ P(E ) = 36 .

19 / 48
3.3 Probability

Extra examples:
(a) A committee of size 5 is to be selected from a group of 6 men and 9
women. If the selection is made randomly, what is the probability that
the committee consists of 3 men and 2 women?
(b) From a set of n items a random sample of size k is to be selected.
What is the probability that a specific item will be among the k
selected?

20 / 48
3.3 Probability

Axioms of probability:
Rigorously, P(E ) is called “probability” if the following three axioms
hold:
(a) 0 ≤ P(E ) ≤ 1.
(b) P(S) = 1.
(c) For any sequence of mutually exclusive events E1 , E2 , · · · , En with
n
S  P n
Ei ∩ Ej = ϕ for all i ̸= j, P Ei = P(Ei ).
i=1 i=1

21 / 48
3.3 Probability
Based on axioms, we have some propositions:
Proposition
Suppose that E is an event. Then we have

P(E c ) = 1 − P(E ),

where E c is the complement of E .

Proof:

22 / 48
3.3 Probability
Proposition
Suppose that E and F are two events in the sample space S. Then

P(F ) = P(E ∩ F ) + P(F ∩ E c ).

Proof:

23 / 48
3.3 Probability
Proposition
Suppose that E and F are two events, and E ∩ F ̸= ϕ. Then

P(E ∪ F ) = P(E ) + P(F ) − P(E ∩ F ).

Proof:

24 / 48
3.3 Probability
Example 6: A total of 28 percent of American males smoke cigarettes, 7
percent smoke cigars, and 5 percent smoke both cigars and cigarettes.
What percentage of males smoke neither cigars nor cigarettes?

25 / 48
3.4 Conditional Probability

Recall: Let E and F be two events. E ∩ F is the intersection of two

events. Then we denote P(E ∩ F ) as the probability of the intersection
E ∩ F . It indicates the possibility of outcomes in events E and F .

Now: Another important issue is the probability of an event E , provided

the happening of event F .

26 / 48
3.4 Conditional Probability

Example: In statistics class, students contain male and female, and they
are from different departments in the university. Then what is the
proportion of male students who come from department of statistics?
In this example, we have two events:
E : gender of students;
F : departments in the university.
The proportion is equivalent to the probability of E given F .
This probability is called conditional probability.

27 / 48
3.4 Conditional Probability

Definition
Conditional probability of E given F , denoted as P(E |F ), can be
interpreted as the probability of E given the condition that event F has
occurred.

28 / 48
3.4 Conditional Probability

Q1: How to calculate P(E |F )?

Q2: What is the relationship between P(E ), P(F ), and P(E |F )?

To answer those questions, one can use Venn diagrams.

29 / 48
3.4 Conditional Probability
From Venn diagrams:

30 / 48
3.4 Conditional Probability

Summary: According to Venn diagram, we conclude that

P(E |F ) =

31 / 48
3.4 Conditional Probability
Example 7: A bin contains 5 defective, 10 partially defective, and 25
acceptable transistors. A transistor is chosen at random from the bin and
put into use. If it does not immediately fail, what is the probability that it
is acceptable?

32 / 48
3.4 Conditional Probability
Example 8: The organization that Jones works for is running a father-son
dinner for those employees having at least one son. If Jones is known to
have two children, what is the conditional probability that they are both
boys given that he is invited to the dinner?

33 / 48
3.4 Conditional Probability
Example 9: Ms. Perez figures that there is a 30 percent chance that her
company will set up a branch office in Phoenix. If it does, she is 60 percent
certain that she will be made manager of this new operation. What is the
probability that Perez will be a Phoenix branch office manager?

34 / 48
3.4 Conditional Probability

Conditional probability also extends a new concept.

Since P(E |F ) says that the probability of E may be affected by F , we
usually have P(E |F ) ̸= P(E ). It is called “E and F are dependent
events”.
On the other hand, if the probability of E is not changed by the
existence of F , i.e., P(E |F ) = P(E ), then we say events E and F are
independent.

35 / 48
3.4 Conditional Probability

Definition (Independence)
Two events E and F are independent, denoted E ⊥ F , if

P(E |F ) = P(E ) or P(F |E ) = P(F ).

36 / 48
3.4 Conditional Probability

Proposition
If E and F are independent, then
(a) P(E ∩ F ) = P(E )P(F ).
(b) E and F c are independent as well.

37 / 48
3.4 Conditional Probability
Example 10: A card is selected at random from an ordinary deck of 52
playing cards. If A is the event that the selected card is an ace and H is
the event that it is a heart. Check the independence of A and H.

38 / 48
3.4 Conditional Probability
Example 11: Two fair dice are thrown. Let E7 denote the event that the
sum of the dice is 7. Let F denote the event that the first die equals 4 and
let T be the event that the second die is 3. Discuss the independence
between E7 and F , T , and FT .

39 / 48
3.4 Conditional Probability

What do we learn from Example 11?

Independence among more than two events.
Definition
Let E , F , and G denote three events. They are said to be independent if

P(EFG ) = P(E )P(F )P(G )

P(FG ) = P(F )P(G )
P(EG ) = P(E )P(G )
P(EF ) = P(E )P(F )

40 / 48
3.4 Conditional Probability

Generally, we can further define the independence of n events.

Specifically, let E1 , E2 , · · · , En be events. They are said to be
independent if for every subset E1 , · · · , Er with r ≤ n, we have

P(E1 E2 · · · Er ) = P(E1 )P(E2 ) · · · P(Er ).

41 / 48
3.4 Conditional Probability
Example 12: A system composed of n separate components is said to be
a parallel system if it functions when as least one of the components
functions. For such a system, if component i, independent of other
components, functions with probability pi , i = 1, · · · , n, what is the
probability the system functions?

42 / 48
3.5 Bayes’ Theorem

Definition (Prior probability)

The initial probability based on the present level of information. That is, if
E is an event, then P(E ) is regarded as the prior probability.

Assume that additional information is available. Specifically, let F

denote the other event. Then we consider two conditional
probabilities P(E |F ) and P(E |F c ) are known.
Now, our interest is to understand P(F |E ).

43 / 48
3.5 Bayes’ Theorem

Before discussing P(F |E ), we first need the following proposition:

Proposition (Law of total probability)

Let E and F be two events. Then

P(E ) = P(E |F )P(F ) + P(E |F c )P(F c ). (1)

44 / 48
3.5 Bayes’ Theorem

We now derive P(F |E ).

P(F |E ) =

Remark: P(F |E ) is also called posterior probability.

45 / 48
3.5 Bayes’ Theorem

Generally, if there are n events F1 , · · · , Fn , then Bayes’ theorem tells

us that for i = 1, · · · , n,

P(Fi |E ) =

46 / 48
3.5 Bayes’ Theorem

Example 13: The Ludlow Wildcats baseball team, a minor league team in
the Cleveland Indians organization, plays 70% of their games at night and
30% during the day. The team wins 50% of their night games and 90% of
their day games. According to today’s newspaper, they won yesterday.
What is the probability the game was played at night?

47 / 48
3.5 Bayes’ Theorem
Example 14: The credit department of Lion’s Department Store in
Anaheim, California, reported that 30% of their sales are cash, 30% are
paid with a credit card, and 40% with a debit card. Twenty percent of the
cash purchases, 90% of the credit card purchases, and 60% of the debit
card purchases are for more than $50. Ms. Tina Stevens just purchased a
new dress that cost $120. What is the probability that she paid cash?

48 / 48