Introduction To Probability Theory: Rong Jin

Introduction to Probability
Theory

Rong Jin
Outline
Basic concepts in probability theory
Bayes rule
Random variable and distributions
Definition of Probability
Experiment: toss a coin twice
Sample space: possible outcomes of an experiment
S = {HH, HT, TH, TT}
Event: a subset of possible outcomes
A={HH}, B={HT, TH}
Probability of an event : an number assigned to an
event Pr(A)
Axiom 1: Pr(A) > 0
Axiom 2: Pr(S) = 1
Axiom 3: For every sequence of disjoint events

Example: Pr(A) = n(A)/N: frequentist statistics

Pr( ) Pr( )
i i
i i
A A =
Joint Probability
For events A and B, joint probability Pr(AB)
stands for the probability that both events
happen.
Example: A={HH}, B={HT, TH}, what is the joint
probability Pr(AB)?

Independence
Two events A and B are independent in case
Pr(AB) = Pr(A)Pr(B)
A set of events {A
i
} is independent in case

Pr( ) Pr( )
i i
i i
A A =
[
Independence
Two events A and B are independent in case
Pr(AB) = Pr(A)Pr(B)
A set of events {A
i
} is independent in case

Example: Drug test
Pr( ) Pr( )
i i
i i
A A =
[
Women Men
Success 200 1800
Failure 1800 200
A = {A patient is a Woman}
B = {Drug fails}
Will event A be independent
from event B ?
Independence
Consider the experiment of tossing a coin twice
Example I:
A = {HT, HH}, B = {HT}
Will event A independent from event B?
Example II:
A = {HT}, B = {TH}
Will event A independent from event B?
Disjoint = Independence

If A is independent from B, B is independent from C, will A
be independent from C?
If A and B are events with Pr(A) > 0, the conditional
probability of B given A is

Conditioning
Pr( )
Pr( | )
Pr( )
AB
B A
A
=

Example: Drug test

Conditioning
Pr( )
Pr( | )
Pr( )
AB
B A
A
=
Women Men
Success 200 1800
Failure 1800 200
A = {Patient is a Women}
B = {Drug fails}
Pr(B|A) = ?
Pr(A|B) = ?

Example: Drug test

Given A is independent from B, what is the relationship
between Pr(A|B) and Pr(A)?
Conditioning
Pr( )
Pr( | )
Pr( )
AB
B A
A
=
Women Men
Success 200 1800
Failure 1800 200
A = {Patient is a Women}
B = {Drug fails}
Pr(B|A) = ?
Pr(A|B) = ?
Which Drug is Better ?
Simpsons Paradox: View I
Drug I Drug II
Success 219 1010
Failure 1801 1190
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 10%
Pr(C|B) ~ 50%
Drug II is better than Drug I
Simpsons Paradox: View II
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Male Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 100%
Pr(C|B) ~ 50%
Female Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 20%
Pr(C|B) ~ 5%
Male Patient
A = {Using Drug I}
B = {Using Drug II}
C = {Drug succeeds}
Pr(C|A) ~ 100%
Pr(C|B) ~ 50%
Drug I is better than Drug II
Conditional Independence
Event A and B are conditionally independent given
C in case
Pr(AB|C)=Pr(A|C)Pr(B|C)
A set of events {A
i
} is conditionally independent
given C in case

Pr( | ) Pr( | )
i i
i i
A C A C =
[
Conditional Independence (contd)
Example: There are three events: A, B, C
Pr(A) = Pr(B) = Pr(C) = 1/5
Pr(A,C) = Pr(B,C) = 1/25, Pr(A,B) = 1/10
Pr(A,B,C) = 1/125
Whether A, B are independent?
Whether A, B are conditionally independent
given C?
A and B are independent = A and B are
conditionally independent
Outline
Important concepts in probability theory
Bayes rule
Random variables and distributions
Given two events A and B and suppose that Pr(A) > 0. Then

Example:

Bayes Rule
Pr(W|R) R R
W 0.7 0.4
W 0.3 0.6
R: It is a rainy day
W: The grass is wet
Pr(R|W) = ?
Pr(R) = 0.8
) Pr(
) Pr( ) | Pr(
) Pr(
) Pr(
) | Pr(
A
B B A
A
AB
A B = =
Bayes Rule
R R
W 0.7 0.4
W 0.3 0.6
R: It rains
W: The grass is wet
R W
Information
Pr(W|R)
Inference
Pr(R|W)
Pr( | ) Pr( )
Pr( | )
Pr( )
E H H
H E
E
=
Bayes Rule
R R
W 0.7 0.4
W 0.3 0.6
R: It rains
W: The grass is wet
Hypothesis H
Evidence E
Information: Pr(E|H)
Inference: Pr(H|E)
Prior Likelihood Posterior
Bayes Rule: More Complicated
Suppose that B
1
, B
2
, B
k
form a partition of S:

Suppose that Pr(B
i
) > 0 and Pr(A) > 0. Then

;
i j i
i
B B B S = C =
1
1
Pr( | ) Pr( )
Pr( | )
Pr( )
Pr( | ) Pr( )
Pr( )
Pr( | ) Pr( )
Pr( ) Pr( | )
i i
i
i i
k
j
j
i i
k
j j
j
A B B
B A
A
A B B
AB
A B B
B A B
=
=
=
=
=

Suppose that B
1
, B
2
, B
k

Suppose that Pr(B
i

;
i j i
i
B B B S = C =
1
1
Pr( | ) Pr( )
Pr( | )
Pr( )
Pr( | ) Pr( )
Pr( )
Pr( | ) Pr( )
Pr( ) Pr( | )
i i
i
i i
k
j
j
i i
k
j j
j
A B B
B A
A
A B B
AB
A B B
B A B
=
=
=
=
=

Suppose that B
1
, B
2
, B
k

Suppose that Pr(B
i

;
i j i
i
B B B S = C =
1
1
Pr( | ) Pr( )
Pr( | )
Pr( )
Pr( | ) Pr( )
Pr( )
Pr( | ) Pr( )
Pr( ) Pr( | )
i i
i
i i
k
j
j
i i
k
j j
j
A B B
B A
A
A B B
AB
A B B
B A B
=
=
=
=
=
A More Complicated Example

R It rains
W The grass is wet
U People bring umbrella
Pr(UW|R)=Pr(U|R)Pr(W|R)
Pr(UW| R)=Pr(U| R)Pr(W| R)
R
W U
Pr(W|R) R R
W 0.7 0.4
W 0.3 0.6
Pr(U|R) R R
U 0.9 0.2
U 0.1 0.8
Pr(U|W) = ?
Pr(R) = 0.8
R It rains
W The grass is wet
R
W U
Pr(W|R) R R
W 0.7 0.4
W 0.3 0.6
Pr(U|R) R R
U 0.9 0.2
U 0.1 0.8
Pr(U|W) = ?
Pr(R) = 0.8
R It rains
W The grass is wet
R
W U
Pr(W|R) R R
W 0.7 0.4
W 0.3 0.6
Pr(U|R) R R
U 0.9 0.2
U 0.1 0.8
Pr(U|W) = ?
Pr(R) = 0.8
Outline
Important concepts in probability theory
Bayes rule
Random variable and probability distribution
Random Variable and Distribution
A random variable X is a numerical outcome of a
random experiment
The distribution of a random variable is the collection
of possible outcomes along with their probabilities:
Discrete case:
Continuous case:
Pr( ) ( ) X x p x
u
= =
Pr( ) ( )
b
a
a X b p x dx
u
s s =
}
Random Variable: Example
Let S be the set of all sequences of three rolls of a
die. Let X be the sum of the number of dots on the
three rolls.
What are the possible values for X?
Pr(X = 5) = ?, Pr(X = 10) = ?

Expectation
A random variable X~Pr(X=x). Then, its expectation is

In an empirical sample, x1, x2,, xN,

Continuous case:

Expectation of sum of random variables

[ ] Pr( )
x
E X x X x = =
1
1
[ ]
N
i
i
E X x
N
=
=

[ ] ( ) E X xp x dx
u
=
}
1 2 1 2
[ ] [ ] [ ] E X X E X E X + = +
Expectation: Example
Let S be the set of all sequence of three rolls of a die.
Let X be the sum of the number of dots on the three
rolls.
What is E(X)?

Let S be the set of all sequence of three rolls of a die.
Let X be the product of the number of dots on the
three rolls.
What is E(X)?
Variance
The variance of a random variable X is the
expectation of (X-E[x])
2
:
2
2 2
2 2
2 2
( ) (( [ ]) )
( [ ] 2 [ ])
( [ ] )
[ ] [ ]
Var X E X E X
E X E X XE X
E X E X
E X E X
=
= +
=
=
Bernoulli Distribution
The outcome of an experiment can either be success
(i.e., 1) and failure (i.e., 0).
Pr(X=1) = p, Pr(X=0) = 1-p, or

E[X] = p, Var(X) = p(1-p)
1
( ) (1 )
x x
p x p p
u

=
Binomial Distribution
n draws of a Bernoulli distribution
X
i
~Bernoulli(p), X=
i=1
n

X
i
, X~Bin(p, n)
Random variable X stands for the number of times
that experiments are successful.

E[X] = np, Var(X) = np(1-p)
(1 ) 1, 2,...,
Pr( ) ( )
0 otherwise
x n x
n
p p x n
X x p x x
u
| |
=
|
= = =

\ .
Plots of Binomial Distribution

Poisson Distribution
Coming from Binomial distribution
Fix the expectation =np
Let the number of trials n
A Binomial distribution will become a Poisson distribution

E[X] = , Var(X) =

>
= = =

otherwise 0
0
!
) ( ) Pr(
x e
x
x p x X
x
Plots of Poisson Distribution

Normal (Gaussian) Distribution
X~N(,o)

E[X]= , Var(X)= o
2
If X
1
~N(
1
,o
1
) and X
2
~N(
2
,o
2
), X= X
1
+ X
2
?
2
2
2
2
2
2
1 ( )
( ) exp
2
2
1 ( )
Pr( ) ( ) exp
2
2
b b
a a
x
p x
x
a X b p x dx dx
u
u
o
to
o
to

=
`

)

s s = =
`

)
} }

Introduction To Probability Theory: Rong Jin

Uploaded by

Copyright:

Available Formats

Introduction To Probability Theory: Rong Jin

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Introduction To Probability Theory: Rong Jin

Uploaded by

Copyright:

Available Formats

Introduction to Probability

Bayes Rule: More Complicated

Bayes Rule: More Complicated

A More Complicated Example

Plots of Binomial Distribution

Plots of Poisson Distribution

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.