Chapter 1: Fundamental Concepts 1

STAT2001
2023/24 Term 2
Outline
1. Random experiments, Sample Space, Probability
2. Methods of enumeration (Counting)
3. Conditional probability
4. Bayes’s theorem
5. Independent events
Textbook chapters 1.1 - 1.5
1. Random experiments, Sample Space, Probability 2

1.1 A review of set definition and notations

ˆ Set - A collection of objects. One can describe a set by different ways, e.g.
1, by listing the set’s elements, say A = {1, 2, 3, 4, 5}.
2, by describing the set in words, say B is the set of all real numbers between
−1 and 1 inclusively.
3, by means of set notations, say for the above example, B = {x : −1 ≤ x ≤ 1}.
ˆ Subset: A subset is a sub-collection of objects in a set. Say A = {1, 2, 3, 4, 5}
and C = {1, 3}. C is a proper subset of A because C is a subset of A but
C 6= A, or we write as C ⊂ A. If we only know that a set S1 is a subset of a set
S2 but we do not know whether S1 = S2 or not, we write S1 ⊆ S2.
ˆ Universe: The set contains all elements is called Universe. It is usually denoted
by Ω or S.
ˆ Empty set: The set containing no element is called Empty set. It is usually
denoted by ∅.
1. Random experiments, Sample Space, Probability 3

1.2 A review of set operations

ˆ The set of elements of either A or B or possibly both A and B is called the
union of A and B, denoted by A ∪ B;
ˆ The set of elements of both A and B is called the intersection of A and B,
denoted by A ∩ B;
ˆ The complement of a set A is the set of elements of the universe that are not
in A, denoted by A0 or Ac.
 1. Random experiments, Sample Space, Probability 4

A convenient way to illustrate set operations is to use a Venn diagram:

1. Random experiments, Sample Space, Probability 5

ˆ Commutative Laws

A∪B = B∪A
A∩B = B∩A

ˆ Associative Laws

(A ∪ B) ∪ C = A ∪ (B ∪ C)
(A ∩ B) ∩ C = A ∩ (B ∩ C)

ˆ Distributive Laws

A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C)
A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)

ˆ De Morgan’s Laws

(A ∪ B)0 = A0 ∩ B 0
(A ∩ B)0 = A0 ∪ B 0
1. Random experiments, Sample Space, Probability 6

ˆ A1 , A2 , ..., Ak are called mutually exclusive sets if Ai ∩ Aj = ∅ for any

i 6= j;
ˆ A1 , A2 , ..., Ak are called exhaustive sets if A1 ∪ A2 ∪ ... ∪ Ak = Ω;
ˆ A1 , A2 , ..., Ak are called mutually exclusive and exhaustive sets if they
satisfy both conditions above.
 1. Random experiments, Sample Space, Probability 7

1.3 Random experiment and Sample space

In a statistical problem, we consider an experiment for which the outcome cannot
be predetermined with certainty. Such an experiment is called a Random Exper-
iment. The set theory above is useful here to define terminologies:
ˆ Sample space: the collection of all possible outcomes of the random experi-
ment. It corresponds to the Universe we defined for general set theory. Sample
space is usually denoted by S.
ˆ Event: An event is a subset of S.
 1. Random experiments, Sample Space, Probability 8

Examples:
Random experiment 1: throw a fair die and observe the upper face,
ˆ Sample Space : {1, 2, 3, 4, 5, 6};
ˆ Examples of events: ’the result is 3’, ’the result is less than four’, ’the result is
odd’, etc;
Random experiment 2 : The salary of a randomly picked statistics fresh graduate,
ˆ Sample Space : from experience, {x : 0 ≤ x ≤ 20, 000};
ˆ Examples of events: ’the salary is 0’, ’the salary is below 10,000’, etc;
 1. Random experiments, Sample Space, Probability 9

1.4 Probability

What is probability?

Definition: Probability is a real valued function P that assigns to each event

A in the sample space S a number P (A), called the probability of event A, such
that the following Kolmogorov axioms are satisfied:
1. P (A) ≥ 0
2. P (S) = 1
3. If A1, A2, ... are events and Ai ∩ Aj = ∅ for i 6= j, then
ˆ P (A1 ∪ ... ∪ Ak ) = P (A1 ) + ... + P (Ak ), for any positive integer k and
ˆ P (A1 ∪ A2 ∪ ...) = P (A1 ) + P (A2 ) + ..., for a countably infinite number of
events.
 1. Random experiments, Sample Space, Probability 10

Assuming the axioms, one can prove the following properties:

1. P (A) = 1 − P (A0);
2. P (∅) = 0;
3. If A ⊂ B, then P (A) ≤ P (B);
4. For any event A, P (A) ≤ 1;
5. If A and B are any two events, then P (A ∪ B) = P (A) + P (B) − P (A ∩ B)
6. If A, B, C are three events, P (A ∪ B ∪ C) = P (A) + P (B) + P (C) − P (A ∩
B) − P (B ∩ C) − P (A ∩ C) + P (A ∩ B ∩ C)
Remark: The last property can be extended to more than three events. It is called
”Inclusion-Exclusion Formula”. For k events,
P (C1 ∪ C2 ∪ ... ∪ Ck ) = p1 − p2 + p3 − ... + (−1)k+1pk
where pi equals the sum of probabilities of all possible intersections involving i events.
 1. Random experiments, Sample Space, Probability 11

Example:
A survey was taken of a group’s viewing habits of sporting events on TV during
last year. Let A = {watched football}, B = {watched basketball}, C = {watched
baseball}. The results indicate that if a person is randomly selected from the survey
group, then P (A) = 0.43, P (B) = 0.40, P (C) = 0.32, P (A∩B) = 0.29, P (A∩C) =
0.22, P (B ∩ C) = 0.20, and P (A ∩ B ∩ C) = 0.15. What is the probability that
this person watched at least one of these sports?

P (A ∪ B ∪ C) = P (A) + P (B) + P (C) − P (A ∩ B) − P (A ∩ C) − P (B ∩ C)

+ P (A ∩ B ∩ C)
= 0.43 + 0.40 + 0.32 − 0.29 − 0.22 − 0.20 + 0.15
= 0.59
 2. Methods of Enumeration (Counting) 12

When each element of a Sample Space has equal probability of occurrence, we can
assign probability to an event by counting. For example,

A card is drawn randomly from a deck of 52 cards. What is the probability of

drawing a ”King”?

Counting techniques are very helpful to determine the number of outcomes asso-
ciated with events of some random experiments. We first introduce some counting
rules:
1. Multiplication principle: For two experiments E1 and E2 having n1 and n2
outcomes respectively, the composite experiment E1E2 that perform E1 first
and then E2 has n1n2 outcomes.
Example: What is the number of combinations if I draw a card from a deck
and then throw a fair die once?
 2. Methods of Enumeration (Counting) 13

2. Permutation:
If n different objects are to be filled in n positions, the number of ordered ar-
rangement is n! = n(n − 1)...(2)(1)
Each of these arrangements is called a permutation of the n objects.
n!
3. If we only have r (r ≤ n) positions to fill in, there are nPr = (n−r)! possible
ordered arrangements. Each of these arrangements is called a permutation of n
objects taken r at a time
4. Note that ordering is important in the above arrangements. If the ordering of
objects are not important,
 
the number of subsets of size r that can be taken from
n n!
n objects is nCr = 
  =
r!(n−r)!
r
Each of these subsets is called a combination of n objects taken r at a time.
5. Suppose we have n objects of two types, r of one type and n − r of the other
type. The number of distinguishable permutations is nCr .
 2. Methods of Enumeration (Counting) 14

Example: If I have a box of 4 balls. Two are black (B) and 2 are white (W),
the distinguishable arrangements are
BBW W, BW BW, BW W B, W BBW, W BW B, W W BB. We have 4C2 = 6
distinguishable permutations.
6. We can extend the above result. We now have n objects with n1 of them are
of a similar type, n2 are of another type, etc. And we have k types such that
n1 + ... + nk = n. The number of distinguishable permutations of the n objects
is  
n n!
 =
 

n1, ..., nk n1!n2!...nk !

Example: If the students on the boat have three red flags, four yellow flags,
and two blue flags to arrange on a vertical pole, the number of possible signals is
 
9  9!
 = = 1260


3, 4, 2 3!4!2!
 2. Methods of Enumeration (Counting) 15

More examples:
1. Five letters are selected at random one after another with replacement from 26
letters of the English alphabets. Find the number of possible 5-letter codes that
can be formed.
2. How about selected without replacement?
3. What if the ordering is not important for the second case?
 2. Methods of Enumeration (Counting) 16

2.1 From counting to probability

Now we come back to probability. Through the counting techniques, we can some-
times calculate the number of possible outcomes in an event A (denote by N (A)).
If every outcome has the same chance of showing up, we simply have P (A) = N (A)
N (S) .
Example:
I randomly select 5 cards from a deck. What is the probability of getting 2 spades
and 3 hearts?
Ans:
N (S) =52 C5 = 2, 598, 960
N (A) =13 C2 ×13 C3 = 78 × 286 = 22, 308
22,308
P (A) = 2,598,960 = 0.00858
 3. Conditional Probability 17

For any two events A and B, the conditional probability of B given A is defined as:
P (B ∩ A)
P (B|A) = provided that P (A) > 0.
P (A)
Note that Conditional Probability also satisfies the axioms for a probability function
in section 1.4, namely, with P (A) > 0,
1. P (B|A) ≥ 0.
2. P (A|A) = 1.
3. If B1, B2, ... are events and Bi ∩ Bj = ∅ for i 6= j, then
ˆ P (B1 ∪ ... ∪ Bk |A) = P (B1 |A) + ... + P (Bk |A), for any positive integer k
and
ˆ P (B1 ∪ B2 ∪ ...|A) = P (B1 |A) + P (B2 |A) + ..., for a countably infinite
number of events.
 3. Conditional Probability 18

Some useful properties:

1. If A ⊂ B, Pr(B|A) = 1.
2. If A ⊃ B, Pr(B|A) = Pr(B)/ Pr(A).
3. If B ∩ C = φ, Pr(B ∪ C|A) = Pr(B|A) + Pr(C|A)
4. Pr(B c|A) = 1 − Pr(B|A)
5. However, Pr(B|Ac) 6= 1 − Pr(B|A). It is easy to see from the first example in
the following page:
3 4
Pr(A2|A1) = and Pr(A2|Ac1) = . They don’t add up to 1.
51 51
Remark that both B 0 and B c denote the complement of a set B.
 3. Conditional Probability 19

6. Multiplicative rule:
Pr(A ∩ B) = Pr(B|A) Pr(A).

7. Law of Total Probability:

P (B) = P (B ∩ A) + P (B ∩ Ac) = P (B|A)P (A) + P (B|Ac)P (Ac)

Examples:
1. Pick 2 cards (without replacement) from a usual deck of 52 cards. Let A1(respectively,
A2) be the events that the first (resp. second) card is ace.
Without any information about the first card, Pr(A2) =Pr(A1) = 4/52 = 1/13.
(Symmetry)
However, if we know that the first card is an ace,
3 1
Pr(A2|A1) = = 0.0588 < 0.0769 =
51 13
 3. Conditional Probability 20

2. Suppose an urn contains seven black balls and five white balls. We draw two
balls from the urn without replacement. What is the probability that both balls
are black?
Ans: Let A and B denote the events that the first and the second ball drawn is
black respectively.
6 7 42
P (A ∩ B) = P (B|A)P (A) = ( 11 )( 12 ) = 132
3. A test for AIDS is given to 1,000,000 people. Result as follow:
B1: Carry AIDS Virus B2: Not carry AIDS Virus Totals
A1: Test Positive 4,885 73,630 78,515
A2: Test Negative 115 921,370 921,485
Totals 5,000 995,000 1,000,000
If one of these people is randomly selected, what is the probability that (s)he is
not AIDs Virus carrier but has a positive test result?
If we know that (s)he get a positive test result, what is the probability that (s)he
is really a carrier?
Ans: P (B2 ∩ A1) = 73, 630/1, 000, 000 = 0.07363
1 ∩A1 )
P (B1|A1) = P (B P (A1 ) = 4885/78515 = 0.06222
 4. Bayes’s theorem 21

We say that B1, ..., Bm consitute a partition of the sample space S if B1, ..., Bm
are mutually exclusive and exhaustive. In the following, we assume that P (Bk ) > 0
for k = 1, ..., m.
For any event A ⊂ S, P (A) = m
Pm
j=1 P (A ∩ Bj ) = j=1 P (A|Bj )P (Bj ). If
P

k ∩A) P (Bk ∩A)

P (A) > 0, then P (Bk |A) = P (B
P (A) = Pm
P (A|Bj )P (Bj ) .
j=1

Bayes’s Theorem states that:

P (Bk ∩ A) P (A|Bk )P (Bk )

P (Bk |A) = Pm = Pm f or any k = 1, ..., m
j=1 P (A|B j )P (Bj ) j=1 P (A|Bj )P (B j )
Remarks:
1. A very simple rule but very useful. Form the basis of Bayesian Statistics.
2. P (Bk ) can be called prior probability for event Bk , and P (Bk |A) can be called
the posterior probability of Bk .
 4. Bayes’s theorem 22

Examples:
1. At the college entrance exam, each candidate is admitted or rejected according
to whether he has passed or failed the test. Of the candidates who are really
capable, 80% pass the test; and of the incapable, 25% pass the test. Given
that 40% of the candidates are capable, find the proportion of capable college
students.
Ans: Let P denote the event of passing the test, C denote the event of being
capable

P (P |C) = 0.8, P (P |C 0) = 0.25, P (C) = 0.4

P (P |C)P (C) 0.8 × 0.4 32
P (C|P ) = = =
P (P |C)P (C) + P (P |C 0)P (C 0) 0.8 × 0.4 + 0.25 × 0.6 47
 4. Bayes’s theorem 23

2. Urn A contains 5 black balls and 6 white balls, and urn B contains 8 black balls
and 4 white balls. Two balls are transferred from B to A and then a ball is
drawn from A.
(a) What is the probability that this ball is white? Let E be the event that a
white ball is drawn from A, B1 be the event that 2 white balls are selected
from B, B2 be the event that 1 white and 1 black ball are selected from B,
B3 be the event that two black balls are selected from B.
Ans:
P (E) = 3k=1 P (E∩Bi) = P (B1)P (E|B1)+P (B2)P (E|B2)+P (B3)P (E|B3)
P

= 124CC22 × 13
8
+ 8C1214CC2 1 × 13
7
+ 128CC22 × 13
6
= 0.5128

(b) Given that the ball drawn is white, what is the probability that at least one
white ball was transferred to A?
Ans: C 8 2 ×6
P (E|B3 )P (B3 )
P (B30 |E) = 1 − P (B3|E) = 1 − P (E) =1− 12 C2 13
0.5128 = 0.6182
 5. Independent Events 24

Two events, say A and B, are said to be independent if the occurrence of one of
them does not change the probability of occurrence of the other. Mathematically we
may write,
P (A|B) = P (A).
However the above mathematical statement requires P (B) > 0 and the roles of A
and B are not symmetric.

Formal definition of independence is: A and B are independent if and only if

Pr(A ∩ B) = Pr(A) Pr(B)

Remark: If A and B are independent, A and B 0, A0 and B, A0 and B 0 are also

independent. (Proofs can be found in section 1.4 of textbook.)
 5. Independent Events 25

The concept of independence can be extended to more than two events. Consider 3
events, say A, B and C. They are mutually independent if
1. They are pairwise independent; that is,
P (A ∩ B) = P (A)P (B), P (A ∩ C) = P (A)P (C), P (B ∩ C) = P (B)P (C)
2. P (A ∩ B ∩ C) = P (A)P (B)P (C)
Remark: Note that Condition 1 6⇒ Condition 2 and so both conditions are needed.
 5. Independent Events 26

Examples:
1. An urn contains 4 balls numbered 1,2,3,4. One ball is to be drawn at random
from the urn. Let the events A, B, C be defined by A = {1, 2}, B = {1, 3}, C =
{1, 4}.
P (A) = P (B) = P (C) = 1/2
P (A ∩ B) = 1/4 = P (A)P (B)
P (A ∩ C) = 1/4 = P (A)P (C)
P (C ∩ B) = 1/4 = P (C)P (B)
However
P (A ∩ B ∩ C) = 1/4 6= 1/8 = P (A)P (B)P (C)
So the events A, B and C are not mutually independent.
 5. Independent Events 27

2. The following table shows the summary proportions for the credit ratings and
the property ownerships of the credit card owners of a bank last year:
Excellent Good Bad
With Property 0.30 0.05 0.00
Without Property 0.30 0.20 0.15
Assume the probability of the corresponding events for the next year will be
exactly the same as the historic values. Evaluate the following probabilities for
the next year.
(a) P (Excellent), P (Good) and P (Bad).
(b) P (With Property) and P (Without Property).
(c) P (Excellent|With Property), P (Good|With Property) and P (Bad|With Prop-
erty).
(d) Are the event of being rated as Excellent and that of “With Property” inde-
pendent to each other?
(e) Given somebody is rated as Bad, what is the probability that (s)he is a
property owner?