Joint Density and Sampling (Lecture 24)

Shilpa Gondhali

16 October 2024

1 / 54
 Change in office hours
Friday: 12.00 PM to 01.00 PM

2 / 54
Example

N! x y
fXY (x, y ) = p p
x!y ! X Y
where x, y are non-negative integers such that x + y = N and
pX , pY be nonzero positive real numbers such that pX + pY = 1.
Check that fXY is a joint density. Calculate ⇢XY if it exists.

Example (Multinomial distribution)

N!
f (X1 = x1 , . . . , Xn = xn ) = p1x1 · · · pnxn
x1 ! · · · xn !
where x1 , . . . , xn are non-negative integers such that
x1 + · · · + xn = N and p1 , . . . , pn be nonzero positive real numbers
such that p1 + · · · + pn = 1.

3 / 54
 Bivariate normal distribution

Fix numbers µ1 , µ2 , 1 , 2 , ⇢ 2 R and 1 , 2 > 0, 1 < ⇢ < 1. A

random variable (X , Y ) is said to have a bivariate normal
distribution if its joint density is given by

exp 2(1A ⇢2 )
fXY (x, y ) := p
2⇡ 1 2 1 ⇢2
where
✓ ◆2 ✓ ◆2 ✓ ◆
x µ1 y µ2 x µ1 y µ2
A := + 2⇢ ·
1 2 1 2

with 1 < x, y < 1. Find marginal densities and verify that

X ⇠ N(µ1 , 12 ) and Y ⇠ N(µ2 , 22 ).

4 / 54
 Expected value
Let (X , Y ) be a 2-dim’l random variable with joint density fXY . Let
H(X , Y ) be a random variable. The expected value of H(X , Y ), denoted
by E [H(X , Y )] is
P P
• E [H(X , Y )] := H(x, y )fXY (x, y ) provided
P P all x all y
|H(x, y )|fXY (x, y ) exists for (X , Y ) discrete,
all x all y
R R
• E [H(X , Y )] := 11 11 H(x, y )fXY (x, y )dy dx provided
R1 R1
1 1
|H(x, y )|fXY (x, y )dy dx exists for (X , Y ) continuous.

7 / 54
 Properties
• E (aX + Y ) = aE (X ) + E (Y )

• Let (X , Y ) be a two-dim’l random variable with joint density

fXY . If X and Y are independent then

E [XY ] = E [X ]E [Y ].

8 / 54
 Textbook Exercise
Problem (20)
An instructor has given a short quiz consisting of two parts. For a
randomly selected student, let X =the number of points earned on
the first part and Y = the number of points earned on the second
part. The joint pmf of X and Y is given below.

X #Y ! 0 5 10 15
0 0.02 0.06 0.02 0.1
5 0.04 0.15 0.20 0.1
10 0.01 0.15 0.14 0.01

a. If the score recorded in the grade book is the total number of

points earned on the two parts, what is the expected recorded
score E(X + Y)?
b. If the maximum of the two scores is recorded, what is the
expected recorded score?

9 / 54
10 / 54
 Covariance
• Just as the mean and variance provided single-number
summaries of the distribution of a single r.v., covariance is a
single-number summary of the joint distribution of two r.v.s.
• Covariance measures a tendency of two r.v.s to go up or down
together, relative to their expected values: positive covariance
between X and Y indicates that when X goes up, Y also tends
to go up, and negative covariance indicates that when X goes
up, Y tends to go down.

Definition
Let X and Y be random variables with means µX and µY
respectively. The covariance between X and Y , denoted by
Cov (X , Y ) or XY is given by

Cov (X , Y ) := E [(X µX )(Y µY )].

Computational formula: Cov (X , Y ) = E [XY ] E [X ]E [Y ].

11 / 54
 Covariance-properties

• Cov(X,X) = Var(X).
• Cov(X, Y ) = Cov(Y,X).
• Cov(X, c) = 0 for any constant c.
• Cov(aX, Y ) = aCov(X, Y ) for any constant a.
• Cov(X + Y,Z) = Cov(X,Z) + Cov(Y,Z).
• Cov(X +Y,Z +W) =
Cov(X,Z)+Cov(X,W)+Cov(Y,Z)+Cov(Y,W).
• Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y ).
• For n r.v.s X1 , · · · , Xn , Var (X1 + · · · + Xn ) =
P
Var (X1 ) + · · · + Var (Xn ) + 2 Cov (Xi , Xj ).
i<j
• Var (X Y ) = Var (X ) + Var (Y ).

12 / 54
13 / 54
Problem (21)
Two fair six-sided dice are rolled (one green and one orange), with
outcomes X and Y respectively for the green and the orange.
Compute the covariance of X + Y and X -Y .

14 / 54
 Remember

• If X and Y are independent then Cov (X , Y ) = 0.

• The converse is false. For example, let X ⇠ N(0, 1), and let
Y = X 2 . Then E (XY ) = E (X 3 ) = 0 because the odd
moments of the standard Normal distribution are equal to 0
by symmetry. Thus X and Y are uncorrelated,
Cov (X , Y ) = E (XY ) E (X )E (Y ) = 0 0 = 0, but they are
certainly not independent: Y is a function of X , so knowing
X gives us perfect information about Y .
• Covariance is a measure of linear association, so RVs can be
dependent in non-linear ways and still have zero covariance, as
this example demonstrates.

15 / 54
Example (22)

fXY (x, y ) := c/x 27  y  x  33.

Find E [X ], E [Y ], E [XY ]. (HW: Check if X and Y are
independent.)

16 / 54
Example (23)
Four temparature gauges are randomly selected from a bin
containing three defective and four non defective gauges. Let X
denote the number of defective gauges selected and Y the number
of non dedective gauges selected. fXY is given by
x\y 0 1 2 3 4
0 0 0 0 0 1/35
1 0 0 0 12/35 0
2 0 0 18/35 0 0
3 0 4/35 0 0 0
Calculate E [X ], E [Y ], E [XY ] and Cov (X , Y ).

18 / 54
 inde. ) cov= 0 but cov= 0 6) inde.

Calculate cov (X , Y ). Are X and Y independent? Justify.

# x\y -1 0 1
0 1/3 0 1/3
1 0 1/3 0

Definition (Pearson coefficient of correlation)

Let X and Y be random variables with means µX and µY and
variances X2 and Y2 , respectively. The correlation ⇢XY , between
X and Y is
Cov (X , Y )
⇢XY = Corr (X , Y ) := p
Var (X ) · Var (Y )

20 / 54
Example (24)

fXY (x, y ) = 1, 1 < x < 1, y > 0, y < |x|.

(i) Is this a joint density?
(ii) If yes then calculate correlation if it exists.
(iii) Calculate marginal densities.
(iv) Are X and Y independent?

21 / 54
22 / 54
Example (HW)

x y
fXY (x, y ) = xye e , x > 0, y > 0.
Check that f is a PDF. Calculate ⇢XY . Are X and Y independent?

Example (HW)

fXY (x, y ) = 1/x, 0<y <x <1

Calculate ⇢XY if it exists.

23 / 54
24 / 54
Fact (
Corr (X , Y ) if ac > 0,
1] Corr (aX + b, cY + d) =
Corr (X , Y ) if ac < 0.
2] The correlation coefficient ⇢XY for any two random variables X
and Y lies between 1 and 1 inclusive. That is,

|⇢XY |  1.

3] Let X and Y be random variables with correlation coefficient

⇢XY . Then, |⇢XY | = 1 if and only if Y = 0 + 1 X for some real
numbers 0 and 1 6= 0.

26 / 54
In fact, we have

27 / 54
 Remember-

• Perfect positive correlation: ⇢ = 1 =) Y = 0 + 1 X

where 1 > 0.
• Perfect negative correlation: ⇢ = 1 =) Y = 0 + 1 X
where 1 < 0.
• ⇢ = 0 then we say that X and Y are uncorrelated (not
unrelated).

28 / 54
 Conditional Probability
• Sometimes, we know an event has happened already and we
want to model what will happen next:
Redi↵’s share price is low and Microsoft will buy it.
Redi↵’s share price is low and Google will buy it.
• Conditional probabilities allow us to understand the reason
about causality.
• In other words, we observe the value of X and will update our
distribution of Y to reflect this information.
• Marginal PMF fY (y ) does not take into account any
information about X and we need to have a PMF that
conditions on the event X = x, where x is the value we
observed for X.
• The above facts naturally lead us to consider conditional
PMFs.
29 / 54
 Conditional Probability-Discrete case
Definition
Let X and Y be two discrete random variables. Then the
conditional probabilities are defined by

fXY (x, y )
fX |Y (x|y ) = P(X = x|Y = y ) = , fY (y ) > 0
fY (y )

and similarly

fXY (x, y )
fX |Y (y |x) = P(Y = y |X = x) = , fX (x) > 0.
fX (x)

30 / 54
 Conditional Probability- Continuous Case
Definition
Let (X , Y ) be a continuous two-dimensional random variable with
joint pdf f Let fX (x) and fY (y ) be the marginal pdfs of X and Y,
respectively.
• The conditional pdf of X for given Y = y is defined by
fX |Y (x|y ) = ffY(x,y )
(y ) , fY (y ) > 0.
• The conditional pdf of Y for given X = x is defined by
fY |X (y |x) = ffX(x,y )
(x) , fX (x) > 0.

32 / 54
Sampling

41 / 54
Assumption: Before the data becomes available, we view each
observation as a random variable and denote the sample by
X1 , X2 , . . . , Xn .
Definition
A statistics is a random variable whose numerical value can be
determined from a random sample. That is, a statistics is a
random variable that is function of the elements of a random
sample X1 , X2 , . . . , Xn .

Example
P
n
• Xi , maxi {Xi }, etc are examples of statistics.
i=1
:
⇠ µ unknown
X1⇠⇠µ
• ⇠ is not a statistics where µ denotes the
population mean.

42 / 54
Example

43 / 54
44 / 54
 Characteristic of a statistical problem

1. Associated with the problem is a large group of objects about

which inferences are to be made. This group of objects is
called population.
2. There is at least one variable whose behavior is to be studied
relative to the population.
3. The population is too large to study in its entirety, or
techniques used in the study are destructive in nature. In
either case, we must draw conclusions about the population
based on observing only a portion or sample of objects drawn
from the population.

45 / 54
Definition (Random sample)
The random variables X1 , X2 , . . . Xn are said to form a random
sample of size n if
(i) The Xi ’s are independent rv’s.
(ii) Every Xi has the same probability distribution.
(That is, Xi ’s are independent and identically distributed (iid).)
Remember:
• Random variables (sample points) are denoted by X1 , X2 , . . . .
• Observed values for X1 , X2 , . . . are denoted by x1 , x2 , . . .
respectively.
• Any statistics, being a random variable, has a probability
distribution. It is refereed to as its sampling distribution.

46 / 54
 Sampling

With Replacement Without Replacement

• If sampling is either with replacement or from an infinite

(conceptual) population, we get random sample.
• In case of sampling without replacement, if sample size n is
much smaller than N, we can proceed as if Xi ’s form a
random sample.
In practice, we need n/N  0.05.

47 / 54
Definition (Sample mean)
Let X1 , X2 , . . . , Xn be a random sample from the distribution of X .
Pn
Xi
The statistic n is called the sample mean and is denoted by X̄ .
i=1

48 / 54
 Median for X

The median of a random variable X is the number M such that

P[X < M]  .50 and P[X  M] .50

If X is continuous then its median is the ‘halfway point’.

Example
1. Calculate median for X ⇠ U({0, 1, 2}).
2. Calculate median for X where X is a Poisson with parameter
k = 0.3.
3. Calculate median for X ⇠ N (3, 27).

49 / 54