The Probability Lifesaver

Steven J. Miller

May 16, 2015

Contents

I General Theory 3
1 Introduction 5
1.1 Birthday Problem 5
1.2 From Shooting Hoops to the Geometric Series 6
1.3 Gambling 7

2 Basic Probability Laws 9

2.1 Paradoxes 9
2.2 Set Theory 10
2.3 Outcome Spaces, Events and The Axioms of Probability 11
2.4 Axioms of Probability 11
2.5 Basic Probability Rules 12
2.6 Probability spaces and σ-algebras 12

3 Counting I: Cards 15
3.1 Factorials and Binomial Coefficients 15
3.2 Poker 16
3.3 Solitaire 17
3.4 Bridge 17

4 Conditional Probability, Independence and Bayes’ Theorem 19

4.1 Conditional Probabilities 19
4.2 The General Multiplication Rule 20
4.3 Independence 20
4.4 Bayes’ Theorem 21
4.5 Partitions and the Law of Total Probability 21
4.6 Bayes’ Theorem Revisited 22
ii • Contents

5 Counting II: Inclusion-Exclusion 23

5.1 Factorial and Binomial Problems 23
5.2 The Method of Inclusion-Exclusion 23
5.3 Derangements 24

6 Counting III: Advanced Combinatorics 25

6.1 Basic Counting 25
6.2 Word Orderings 25
6.3 Partitions 26

II Introduction to Random Variables 27

7 Introduction to Discrete Random Variables 29

7.1 Discrete Random Variables: Definition 29
7.2 Discrete Random Variables: PDFs 29
7.3 Discrete Random Variables: CDFs 30

8 Introduction to Continuous Random Variables 31

8.1 Fundamental Theorem of Calculus 31
8.2 PDFs and CDFs: Definitions 31
8.3 PDFs and CDFs: Examples 32
8.4 Probabilities of Singleton Events 32

9 Tools: Expectation 33
9.1 Calculus Motivation 33
9.2 Expected Values and Moments 33
9.3 Mean and Variance 34
9.4 Joint Distributions 34
9.5 Linearity of Expectation 34
9.6 Properties of the Mean and the Variance 35
9.7 Skewness and Kurtosis 35
9.8 Covariance: TBD 36

10 Convolutions and Changing Variables 37

10.1 Convolutions: Definitions and Properties 37
10.2 Convolutions: Die Example 37
10.3 Convolutions of Several Variables 38
10.4 Change of Variable Formula: Statement 38
10.5 Change of Variables Formula: Proof 39
 Contents • iii

11 Tools: Differentiating Identities 41

11.1 Geometric Series Example 41
11.2 Method of Differentiating Identities 41
11.3 Applications to Binomial Random Variables 42
11.4 Applications to Normal Random Variables 42
11.5 Applications to Exponential Random Variables 42

III Special Distributions 43

12 Discrete Distributions 45
12.1 The Bernoulli Distribution 45
12.2 The Binomial Distribution 45
12.3 The Multinomial Distribution 46
12.4 The Geometric Distribution 46
12.5 The Negative Binomial Distribution 47
12.6 The Poisson Distribution 47
12.7 The Discrete Uniform Distribution 47

13 Continuous Random Variables: Uniform and Exponential 49

13.1 The Uniform Distribution 49
13.2 The Exponential Distribution 49

14 Continuous Random Variables: The Normal Distribution 51

14.1 Determining the Normalization Constant 51
14.2 Mean and Variance 51
14.3 Sums of Normal Random Variables 52
14.4 Generating Random Numbers from Normal Distributions 52
14.5 Examples and the Central Limit Theorem 53

15 The Gamma Function and Related Distributions 55

15.1 Existence of Γ(s) 55
15.2 The Functional Equation of Γ(s) 55
15.3 The Factorial Function and Γ(s) 56
15.4 Special Values of Γ(s) 56
15.5 The Beta Function and the Gamma Function 56
15.6 The Normal Distribution and the Gamma Function 57
15.7 Families of Random Variables 57
15.8 Appendix: Cosecant Identity Proofs 57
iv • Contents

16 The Chi-square Distribution 59

16.1 Origin of the Chi-square Distribution 59
16.2 Chi-square distributions and sums of Normal Random Variables 59

IV Limit Theorems 61

17 Inequalities and Laws of Large Numbers 63

17.1 Inequalities 63
17.2 Markov’s Inequality 63
17.3 Chebyshev’s Inequality 64
17.4 The Boole and Bonferroni Inequalities 64
17.5 Types of Convergence 65
17.6 Weak and Strong Laws of Large Numbers 65

18 Stirling’s Formula 67
18.1 Stirling’s Formula and Probabilities 67
18.2 Stirling’s Formula and Convergence of Series 67
18.3 From Stirling to the Central Limit Theorem 67
18.4 Integral Test and the Poor Man’s Stirling 68
18.5 Elementary Approaches towards Stirling’s Formula 68
18.6 Stationary Phase and Stirling 68
18.7 The Central Limit Theorem and Stirling 69

19 Generating Functions and Convolutions 71

19.1 Motivation 71
19.2 Definition 71
19.3 Uniqueness and Convergence of Generating Functions 72
19.4 Convolutions I: Discrete random variables 72
19.5 Convolutions II: Continuous random variables 73
19.6 Definition and properties of moment generating functions 73
19.7 Applications of Moment Generating Functions 74

20 Proof of The Central Limit Theorem 75

20.1 Key Ideas of the Proof 75
20.2 Statement of the Central Limit Theorem 75
20.3 Means, Variances and Standard Deviations 76
20.4 Standardization 76
20.5 Needed Moment Generating Function Results 77
20.6 Special Case: Sums of Poisson Random Variables 77
20.7 Proof of the CLT for general sums via MGF 77
 • 1

20.8 Using the Central Limit Theorem 78

21 Fourier Analysis and the Central Limit Theorem 79

21.1 Integral transforms 79
21.2 Convolutions and Probability Theory 80
21.3 Proof of the Central Limit Theorem 80
2 •

Greetings again!

The pages below are some quick comments to help you as you read the book. The
goal is to quickly emphasize the main points by asking you some quick questions;
if you are able to answer these you have at least identified the key concepts of the
reading. For each section we highlight the main ideas and then ask some questions;
it’s of course fine to answer these questions the way they were answered in the book!
 Part I

General Theory
Chapter 1

Introduction

1.1 Birthday Problem

Key ideas:
1. Formulate problems carefully: What assumptions are we making? What as-
sumptions should we be making? It is very important to state a problem fully
and carefully.
2. Numerics: If you plot the answer as a function of the parameters, you can often
get a rough sense of the solution. Look at small cases first and build intuition.
Frequently small values / special cases are less intimidating then the general
case.
3. Complementary events: It’s often easier to look at a related problem and then
deduce the answer to the problem you care about. Thus sometimes we calcu-
late the probability our event doesn’t happen, as then the probability it does
happen is just one minus that.
4. Independent events: If events A and B are independent, then the probability
they both happen is the product of the probabilities each happen. It’s sadly a
lot harder when the events are dependent!
5. Probability distributions: We want to describe the probability certain events
happen. Eventually we’ll isolate the definition of a probability density. We
talked about all days being equally likely; this will lead to the uniform distri-
bution.
6. Linearization: One of the biggest ideas in calculus is to replace a complicated
function with a simpler one (often a linear one), which is significantly easier to
analyze. We can use this to get a really good approximation to the true answer.
7. Products: Whenever you see a product you should strongly consider taking a
logarithm. Why? We know how to handle sums and have enormous experience
there; we don’t have similar experience with products. This casts the algebra
in a more familiar light, and can be a great aid in our analysis.
6 • Introduction

8. Generalization: Whenever you see a problem, after you solve it try and think
about other, related problems you could ask, or situations that are similar where
you could apply these techniques. You don’t want to be the trained monkey;
you want to be the person who has the trained monkeys working for you!

Test Questions:

1. State the Geometric Series Formula and the conditions needed to use it.
2. Why do we assume no one is born on February 29th?
3. Give an example where the birthdays are not uniformly distributed over the
year.
4. For the Birthday problem, look at extreme cases: do you think we first reach a
50% chance with just 2 people? With 370 people?
5. For the Birthday problem, do you think we first reach 50% at around 182 peo-
ple? Note that this is the number we need such that each new person, if they’re
entering a room where no two people share a birthday, has about a 50% chance
of sharing a birthday with someone.
6. What do you think would happen to the number of people needed if we now re-
quired there to be a 2/3rds probability of two people sharing a birthday (instead
of 1/2)?
7. Repeat the above question, but now assume that there are D days in a year
and quantify your answer. Explicitly, we showed that the expected number
of
√ people needed to have a 50% chance of two sharing a birthday is about
D · 2 log 2 when there are D days in the year. How do you think this changes
if we now want a 2/3rds chance of a shared birthday?

1.2 From Shooting Hoops to the Geometric Series

Key ideas:

1. Partitioning: We partition the possible outcomes into disjoint possibilities:

Bird wins on his first shot, on his second, on his third.... It’s frequently easier to
break the analysis of a difficult problem into lots of simpler ones and combine.
2. Algebra inputs: We can use the geometric series formula (you should know
what that is, as well as the finite version) to calculate probabilities.
3. Memoryless processes: This is perhaps the most important observation in the
analysis, and allows us to replace an infinite sum with a finite one. The idea
is that at a certain point it is as if we have restarted the game; you might have
seen this idea in the ‘Bring it over’ method for solving some integrals.
4. Solving the right problem: It is very easy to mix up questions and solve the
wrong problem. It’s great to use very detailed variable names to make sure
you stay on the right track. There is a difference between the probability that
Magic wins if Bird shoots first, and if Magic shoots first; to avoid confusion
it’s often good to have extra subscripts on the variables.
 Section 1.3: Gambling • 7

Test Questions:

1. Why do we have an infinite sum for the answer?

2. How are we able to convert the infinite sum to a finite sum?
3. If xb is the probability that Bird shoots first and wins and xm is the probability
that Magic shoots first and wins, must xb + xm = 1? Why or why not?
4. What conditions must be met to use the geometric series formula? What hap-
pens if these conditions are not satisfied?

1.3 Gambling
Key ideas:

1. Expected value: We find the expected value, which is a function of the amount
wagered and the probabilities of each outcome.
2. Solving the right problem: We saw this in the hoops problem, and meet it again
here. There are lots of problems we can study: maximizing return, minimizing
loss, maximizing minimum return. Depending on what we want to do, there
are different optimal strategies.
3. Render unto Caesar: A famous quote, attributed to Jesus in the Gospels, is
“Render unto Caesar the things that are Caesar’s, and unto God the things
that are God’s.” For us, the meaning of this is to remember what belongs
in mathematics, and what belongs to other fields. People’s aversion to risk
and preferences fall outside our realm; these are great topics for psychology
and economics. Our job is to do the analysis and then, based on individual
preferences, people can make their choices.
4. Parameter dependence: When seeing very complicated formulas or plots, it’s a
good idea to take a step back and try to see the big picture. Try to see what will
happen qualitatively if you change the parameters of the problem in a given
way.

Test Questions:

1. Consider the hedging situation where we bet $B on the Giants to win, and
get $x for each dollar bet. What happens to the point of indifference (i.e., the
amount we must bet to be indifferent financially between the Patriots and the
Giants winning) as a function of x? Discuss it qualitatively, and if possible
find a formula for it.
2. Discuss a situation in your life where you can hedge your bets and minimize
your risk.
3. Consider the situation of someone who is sick, let’s call her Pink Bubblegum
(my daughter was sick when I wrote this, and she wanted me to use this fake
name), and goes to the hospital to see the doctor. The doctor does some tests
and they all come back negative; it appears as if she just has a bad cold and
8 • Introduction

doesn’t have a viral infection, even though she has been sick for awhile. The
question is: do we give her antibiotics? The advantage is that if the doctor is
wrong this could knock it out; what is the disadvantage? For health problems
like this are we concerned about minimum or maximum conditions?
Chapter 2

Basic Probability Laws

2.1 Paradoxes
Key ideas:

1. Key terms: Russell’s Paradox.

2. Key takeaway: A collection of items with a given property is not necessarily

a set. A consequence is that we will not be able to assign probabilities to
everything.

3. Be careful: It’s easy to subtly make an assumption and not even be aware of
it! Always think hard about why things are justified. I love the old Statement-
Reason proofs from Geometry, where you divide the paper into two columns,
and for each statement on the left you give a reason on the right.

Test Questions:

1. Give an example of a set that is not an element of itself.

2. Does there exist a set that is an element of itself? Look up the Axiom of
Foundation (also called the Axiom of Regularity).

3. In Russell’s paradox the point is we take the collection of all elements x not
members of themselves, and call that R. Discuss why it is natural to consider
whether or not R is an element of R. What would have happened if we looked
at the property that x is an element of itself?

4. The only set that we really know exists, initially, is the empty set. We denote
it ∅, and one of the axioms of set theory posits its existence. Building on the
empty set, what other sets can we form?

5. Think about connections between the sets ∅, {∅}, {∅, {∅}}, {∅, {∅}, {∅, {∅}}}
and so on, and the numbers 0, 1, 2, and so on. What property does set inclusion
seem to correspond to?
10 • Basic Probability Laws

2.2 Set Theory

Key ideas:

1. Key terms: Element, set, superset, subset, union, intersection, empty set, com-
plement, disjoint, pairwise disjoint, Cartesian product, powerset, finite set, in-
finite set, countable set, uncountable set, 1-1, injective, surjective, onto, bijec-
tion, open sets, closed sets, intervals, interval notation, boundary of sets. Not
surprisingly, there are a lot of terms here as we are quickly covering a vast
subject.

2. Open and closed sets: If you’ve taken a course in analysis (where calculus
is done rigorously) you would have seen these. The idea is that some care is
needed to rigorously investigate certain items. In calculus we use open sets to
talk about notions such as continuity and differentiability at a point, and closed
sets for a discussion of limits.

Test Questions:

1. If A is the set of all pairs of prime numbers (p, q) where p and q are one
digit primes, how large is A? How large is the powerset of A? What is the
intersection of A with the set of primes?

2. Let A be the set of all primes at most 10, and let B be the set of all Fibonacci
numbers at most 10. Find A ∪ B, A ∩ B, and the powersets of these two sets.

3. Give an example of an open set in Rn for each positive integer n, and a closed
set. Give an example of a set that is neither open nor closed.

4. What would be good notation for a rectangle? How would we denote certain
faces being in the set or excluded? Is there good notation that gives us complete
freedom for choosing which of the six faces to include?

5. Find a one-to-one and onto correspondence, or prove one does not exist, be-
tween the set of positive integers and (a) the set of multiples of 2004, (b) the set
of numbers that are equal to the 2004th power of a positive integer, (c) prime
numbers with exactly 2004 digits, (d) real numbers between 1/3 and 5/7, (e)
positive integers that never have 2014 as four consecutive digits, (f) pairs of
positive integers (m, n) with m < n, (g) the same as (f) but now m ≤ n.

6. The following problem is harder and shouldn’t be immediately apparent, but

thinking about it will give you excellent practice in how to attack difficult
problems by doing simpler cases first (by computer code or hand) to detect
a pattern. Let Zn = {1, 2, . . . , n}. How many subsets of Zn are there such
that we never choose two adjacent elements of Zn ? Thus {1, 4, 6, 12, 15} and
{1, 4, 6, 12, 15, 20} would both work for Z20 , but {1, 4, 5, 12, 15} would not.
What if we additionally require n to be in the set: how many subsets are there
now?
 Section 2.3: Outcome Spaces, Events and The Axioms of Probability • 11

2.3 Outcome Spaces, Events and The Axioms of Prob-

ability
Key ideas:
1. Key terms: Event, outcome space, sample space, multiplication rule, indepen-
dent.
2. Techniques: Probability tree diagrams to calculate the probability of several
independent events happening.
Test Questions:
1. What are the events and the outcome space for the following: (a) rolling a fair
die 3 times; (b) rolling a biased die that lands on heads 75% of the time 3 times;
(c) rolling dice three times where the first die has a 50% chance of landing on
heads, and for the remaining rolls if the previous one is a head you use a die
that lands on heads 75% of the time, while if the previous roll was a tail you
use a die that lands on a tail 75% of the time.
2. What are the events and the outcome space for dealing a person 5 cards from a
standard deck?
3. Calculate the probability tree for tossing a coin three times, where all tosses
are independent and each toss lands on a head with probability .7. What would
be a good number of coins to start with?
4. Redo the previous problem but for four tosses, then five, then six. What would
be a good number of coins to start with in each case? Look at the ratio of the
different outcomes in the last row (compare all of them to the number of all
heads). Do you recognize these numbers?

2.4 Axioms of Probability

Key ideas:
1. Key terms: σ-algebra, probability space (Ω, Σ, Prob), counting model.
2. Key axioms: Kolmogorov’s three Axioms of Probability.
3. Symmetry: We can frequently use the method of symmetry to attack problems.
We saw an application here with two outcomes that were equally likely. In gen-
eral, using symmetry can greatly reduce the amount of computations needed.
Test Questions:
1. What are the possible values for the probability of an event? In particular, what
do we know about the probability of the empty set, or of the entire outcome
space?
2. Is the probability of a union always the sum of the probabilities? Is the proba-
bility of an intersection the difference of the probabilities? Does it matter if the
unions or intersections are finite, countably infinite, or uncountably infinite?
12 • Basic Probability Laws

3. Describe the counting model where the outcome space is the results of four
independent tosses of fair coins.

4. Describe the counting model where the outcome space are all hands of five
cards drawn from a standard 52 card deck, assuming each card is equally likely
to be chosen and is chosen independently of the other cards.

2.5 Basic Probability Rules

Key ideas:

1. Key terms: Prob, Venn diagram.

2. Useful probability rules: Law of Total Probability, probability of a union, prob-

ability of a subset.

3. Useful methods: The Method of Complementary Probabilities, The Method of

Inclusion-Exclusion, say it aloud technique.

4. Useful proof techniques: Adding zero, multiplying by one.

Test Questions:

1. Draw the Venn diagram for rolling a fair die twice. Let event A be the sum of
the rolls is even, event B be the first roll is an even number, and event C be the
second roll is an odd number.

2. Find a formula for the probability that A and B both happen; i.e., Prob(A∩B).
What about Prob(A ∩ B ∩ C)?

3. We used the Method of Inclusion-Exclusion to derive a formula for the prob-

ability of a union of finitely many events; can you derive a formula for the
probability of a finite intersection?

4. Use the Method of Complementary Probability to find the probability we have

at least one king if we randomly are dealt seven cards (assuming all hands are
equally likely). If you see the phrase ‘at least’ you should consider using this
method.

5. For nice applications of adding zero / multiplying by one (which hopefully

you’ve seen before), prove the product rule of differentiation and the chain
rule of differentiation.

2.6 Probability spaces and σ-algebras

This is an advanced section; while it is often omitted in a first course, it is worthwhile
to at least skim the material to get a sense of what is needed to set the subject on firm
foundations.
Key ideas:

1. Key terms: σ-algebra, probability space.

 Section 2.6: Probability spaces and σ-algebras • 13

2. Key results: Properties of a σ-algebra (probabilities of countable unions and

intersections), Kolmogorov’s Axioms of Probability.

Test Questions:

1. Give an example of a σ-algebra. For example, find a σ-algebra for n indepen-

dent tosses of a fair coin, or for choosing a number uniformly in [0, 1].
2. Let S be the set of all subintervals I of [0, 1] such that there exist 0 ≤ a ≤ b ≤
1 such that I equals [a, b], [a, b), (a, b] or (a, b). Is S a σ-algebra?

3. Knowing probabilities of closed intervals, is it possible to get probabilities of

open intervals? Of singleton points? If yes describe how; if not describe why
not.
4. State Kolmogorov’s Axioms of Probability.

5. Can we assign probabilities to all subsets of an outcome space?

14 • Basic Probability Laws
Chapter 3

Counting I: Cards

3.1 Factorials and Binomial Coefficients

Key ideas:
1. Key terms: Factorial function, standard deck of cards (the four suits are spades
♠, hearts ♥, diamonds ♦ and clubs ♣, each has cards numbered 2 through
10 plus a jack, a queen, a king and an ace), binomial coefficients, Gamma
function, permutations.
2. Key techniques: Tree diagram to enumerate possibilities, multiplying by one to
gain a better perspective on an algebraic expression, ordered versus unordered
sets (as well as adding and removing order to aid in the analysis), proof by
story.
Test Questions:
1. Define the factorial function. Find some simple bounds for how rapidly it
grows (upper and lower bounds).
2. Give a recursive formula for the factorial function.
3. Define the binomial coefficients. Find some simple bounds for how rapidly it
grows (upper and lower bounds).
for the binomial coefficient n+1


4. Give a recursive formula k in terms of the
binomial coefficients n` with ` ≤ n; how many `’s do you need?

5. How many ways are there to choose 341 objects from 1793 objects? Is this
problem well phrased or is additional information needed (if so provide that
information and then answer the question).
6. How many ways are there to choose three cards from a standard deck such that
no kings are chosen?
7. How many ways are there to choose three cards from a standard deck such that
all the kings are chosen?
16 • Counting I: Cards

3.2 Poker
Key ideas:

1. Key terms: Know the different poker hands (nothing, pair, two pair, three pair,
four of a kind, full house, straight, flush, straight flush).

2. Key ideas: Be very careful in assembling hands. If you
want
to look at how

many five card hands have at least two kings, it is not 42 50
3 . This is a com-
mon mistake: people say we first choose two of the four kings, and then must
choose 3 of the remaining 50 cards. The problem is this leads to double count-
ing, one of the greatest problems in the subject. For example, think about how
many times the hand with all four kings is counted in the above (a correct
enumeration would count it just once).

3. Key techniques: Sometimes it’s easier to put in some order and then remove
it. For example, if we’re trying to count how many flushes there are we first
assume the flush is in hearts, and then multiply by the number of ways to
choose a suit.

4. Algebra aid: I: I often find it helps to be very explicit, and put in lots of extra
detail. In combinatorial problems
instead of writing 13 (if we are choosing one
of the 13 numbers) I’ll write 13 1 .

5. Algebra aid: II: In a lot of combinatorial problems the sum of the bottoms of
the binomial coefficients add up to the number of objects we need to choose.
For example, if we want a hand of five cards with two
kings
and three non-
kings, the number of ways to choose such a hand is 42 48 3 ; note that 48 is
52 − 4 and 2 + 3 = 5.

Test Questions:

1. How many five card hands have exactly two kings?

2. How many five card hands have at least two kings? Is it easier to solve this di-
rectly, or to count how many five card hands there are and subtract the number
of hands with exactly zero and the number of hands with exactly one king?

3. Imagine now that we add two jokers to the deck. What are the probabilities of
each type of hand? Assume if we have a joker we always choose its value to
make the highest ranked hand possible. Before the possible hands were ranked
by their likelihood of occurring; has that order changed now that there is a wild
card?

4. Building on our previous problem with two jokers, note that a new hand is
possible: five of a kind. How should its value be ranked relative to the other
hands?

5. Imagine now we have hands of seven cards, but no wild cards. One new possi-
bility is to have a straight of six numbers, or of seven. We could also have two
different three of a kind. What other new hands emerge, and how should we
order the worth of these relative to each other and our old possibilities?
 Section 3.3: Solitaire • 17

6. A Kangaroo straight is where we skip a number in our counting; thus 4, 6, 8,

10, Q would be a Kangaroo straight. If we allow wrap-arounds (so 10, Q, A,
3, 5 would be considered a Kangaroo straight) is it more likely, equally likely
or less likely to have a Kangaroo straight than a regular straight? What if we
don’t allow wrap-arounds?

7. Imagine our first two cards are two spades. What is the probability we get a
flush if we pick up three more cards? What if instead our first four cards were
spades? What if our first four cards are spades and we see 10 other cards turned
up on the table, with three of those ten cards spades?

8. Imagine our first two cards are the ace of spades and the six of diamonds. What
are the probabilities for the various hands if we pick three more cards? What
if instead we started with two aces?

3.3 Solitaire
Key ideas:

1. Key terms: Different games of Solitaire (Klondike, Aces Up, FreeCell), un-
playable game (no valid moves!).

2. Key techniques: Multiplying by one (was useful in expressing ratios of facto-

rials as ratios of binomial coefficients), Pigeonhole Principle.

Test Questions:

1. Consider the analysis of Aces Up; what if we use a superdeck with s suits, each
suit having c cards. What is the probability now that we cannot win because
the last round yields all cards in different suits? What do you think happens as
s or c vary? Think about some extreme cases.

2. Create and analyze your own variant of Solitaire. For example, imagine we
modify Klondike so that instead of placing red cards on black cards (and vice-
versa) now we can only place hearts on spades, diamonds on hearts, clubs on
diamonds, and spades on clubs. What can you say about unplayable hands
now? If you want, you may change the number of cards in the suits, or the
number of columns in the game.

3.4 Bridge
Key ideas:

1. Key terms: Trump, tic-tac-toe, perfect deals, semi-perfect deals, trump splits.

2. Key ideas: Using symmetry to simplify the analysis, bounding the probability
of an unlikely event by first looking at a significantly more likely (but still rare)
event (I call this the Method of the Simpler Example).

3. Warnings: Be careful not to double count, and not to miss configurations!

18 • Counting I: Cards

Test Questions:

1. How many tic-tac-toe games are there on a 2 × 2 board? Answer this both with
and without taking into account symmetry (i.e., whether or not all opening
moves in a corner are equivalent, whether or not mirror symmetry is the same,
....).

2. How many tic-tac-toe games are there if no one is allowed to move in the
center?
3. How many tic-tac-toe games are there (up to symmetry) if the first two moves
must be center then corner?
4. Imagine now we play superbridge, where there are s suites of c cards; let’s
assume that s is even and that there are s/2 pairs of two. What is the probability
at least one person is dealt all the cards in one suit? What is the probability
exactly one person is dealt all the cards in one suit (if you cannot calculate this
exactly, find a good approximation). What is the probability everyone is dealt
a one-suited hand?

5. In many card games point values are assigned to certain cards to assist in mea-
suring the strength of a hand; typically an ace is worth 4 points, a king 3, a
queen 2 and a jack 1 (all other cards are valued at 0). How many points do you
expect a typical hand to have? What if now we have c cards in a suit and the
top k cards (with k ≤ c) are assigned the values k, k − 1, . . . , 1?

6. What is the probability someone is dealt at least n cards in a suit? Solve this
for n = 7, 8, 9, 10, 11 or 12 (we already discussed 13). Why do you think I
excluded n = 6? What is different about calculating the probability of having
at least 6 cards in a suit from calculating the probability of at least 7 cards in a
suit?

7. Redo the previous problem for n = 6 (i.e., what is the probability someone is
dealt a suit with at least six cards?).
Chapter 4

Conditional Probability, Independence and

Bayes’ Theorem

4.1 Conditional Probabilities

Key ideas:

1. Key terms: Conditional probability (definition and notation Pr(A|B)).

2. Key techniques: Expected Counts Approach, Multiplying by 1 (we use that
again to rewrite some expressions to highlight what is happening), Venn dia-
gram.
3. Key formulas: Conditional probability formula (Prob(A|B) = Prob(A ∩
B)/Prob(B)).

Test Questions:

1. Why, when calculating the conditional probability of A given B, do we assume

that the probability of B is greater than zero?
2. Give an example where the conditional probability of A given B is greater
than the probability of A. Also give an example where the two probabilities
are equal, and where the conditional probability is less.
3. What happens to the conditional probability of A given B when the probability
B happens is 1?
4. Try to think of a formula for the conditional probability that A happens given
that B and C happen.
5. Try to think of a formula for the conditional probability that A happens given
that B happens, subject to the probability that B happens given that C happens.
6. What is the sum of the conditional probability that A happens given B happens
plus the conditional probability that A happens given that B does not happen?
20 • Conditional Probability, Independence and Bayes’ Theorem

7. What is the conditional probability that the sum of two independent rolls of
fair die is even, given that the first roll is a five?
8. What is the conditional probability that the sum of two independent rolls of
fair die is seven, given that the first roll is a five?
9. What is the conditional probability that the sum of two independent rolls of
fair die is three, given that the first roll is a five?

4.2 The General Multiplication Rule

Key ideas:
1. Key terms: conditional probability, General multiplication formula (Pr(A ∩
B) = Pr(A|B)Pr(B)).
2. Key techniques: expressing a complicated probability in terms of simpler ones.
Test Questions:
1. Imagine a batter has a 30% chance of getting a hit, a 10% chance of walking,
and a 60% chance of getting out. If the batter safely reaches base, what is the
probability she got a hit? What is the probability she was walked?
2. Extrapolating (incorrectly!) certain trends at Williams College, imagine in
2116 that the probability someone majors in mathematics or economics is 55%.
If 30% of the students major in math and 28% major in economics, what’s the
probability that a math major also majors in economics?
3. Returning to the previous problem, imagine 30% of the students at Williams
College major in math and 28% major in economics. What is the minimum
possible percentage of students who double major in math and economics?
What is the maximum possible percentage?

4.3 Independence
Key ideas:
1. Key terms: Independence, mutually independent.
2. Key formulas: Independence formula.
Test Questions:
1. Write down explicitly what it means for four events to be independent.
2. Give an example of four events such that any three of them are independent
but all four are not independent.
3. Let event A be the result of rolling a fair die. If A is an even number, flip a fair
coin twice and let B be 1 if the second toss is a head and 0 if it is a tail; if A
is an odd number, flip a fair once once and let B be 1 if we get a head and 0 if
we get a tail. Are A and B independent events?
4. Find a formula for Pr(A ∪ B ∪ C ∪ D) if the four events are independent.
 Section 4.4: Bayes’ Theorem • 21

4.4 Bayes’ Theorem

Key ideas:

1. Key terms: Partitions, partitions in two.

2. Key formulas: Bayes’ Theorem.

3. Key techniques: Expected counts approach.

Test Questions:

1. Why must we assume that event B has a probability greater than 0 in Bayes’
Theorem?

2. Ephraim has two bags containing purple and yellow marbles. In the first bag,
40% of the marbles are purple, while in the second bag the percentage is 60%.
Suppose we randomly choose one of the bags, with each bag equally likely to
be chosen. From that bag we draw at random one marble. If it is purple what
is the probability we chose the first bag? What is the probability we chose the
second bag? If instead it was yellow what would those two probabilities equal?
Are there any relations between these probabilities?

3. Revisit the previous problem, but now after choosing a bag at random imagine
we draw three marbles with replacement (this means that we pick a marble,
record its color, and then place it back in the bag). If we draw more purple
marbles than yellow marbles what is the probability we chose the first bag?
The second bag? Is there any relation between these two probabilities?

4. Expanding on the previous problem, imagine that instead of picking just three
marbles with replacement we pick m marbles with replacement. If we observe
more purple marbles than yellow, as m → ∞ what is the probability we chose
from the first bag? The second bag?

5. A couple in Williamstown with two little kids own two cars, a practical station
wagon and a less practical sports car. When the family goes out they take the
station wagon 90% of the time, and the other 10% of the time they ride in the
sports car. If they are in the station wagon they don’t make good time on the
highways, and they are able to drive to see their families in Boston in under
three hours only 50% of the time, while if they take the sports car they are
able to get to Boston in under three hours an amazing 95% of the time. Given
that they traveled to Boston in 2.5 hours, what is the probability they drove the
station wagon? The sports car?

4.5 Partitions and the Law of Total Probability

Key ideas:

1. Key terms: Disjoint sets, partition.

2. Key formulas: Law of Total Probability.

22 • Conditional Probability, Independence and Bayes’ Theorem

Test Questions:

1. In the Law of Total Probability, why must every set in the partition have posi-
tive probability?
2. A teacher is going to give a surprise quiz; the later in the week the quiz hap-
pens, the higher the probability Bart has of passing as he has more time to
study (and sadly Bart often puts work off to the last minute). There is a 10%
chance of the quiz on Monday, a 20% chance on Tuesday, a 30% chance on
Wednesday and a 40% on Thursday. If the quiz is on Monday Bart has a
40% chance of passing, if it is on Tuesday he has a 60%, if on Wednesday the
chance of passing increases to 80%, and if it is on Thursday then there is a 90%
chance that Bart will pass. What is the probability that Bart passes? What is
the probability that Bart passes if we know the quiz is not on a Monday?
3. Bart has a sister, Lisa, who always prepares. She too has a 10% chance of a
quiz on Monday, a 20% chance on Tuesday, a 30% chance on Wednesday and
a 40% on Thursday, but her probability of passing is always 99%. What is
the probability that Lisa passes? What is the probability that Lisa passes if we
know the quiz is not on a Friday?
4. Partition the set of positive integers at most 20 into two sets of equal size.
How many different ways are there to do this? Does the answer change if you
consider {1, 2, . . . , 10} ∪ {11, 12, . . . , 20} different than {11, 12, . . . , 20} ∪
{1, 2, . . . , 10}?

5. Partition the set of positive integers at most 21 into three sets of equal size.
How many ways are there to do this?

4.6 Bayes’ Theorem Revisited

Key ideas:

1. Key terms: Partitions, conditional probability.

2. Key formulas: Bayes’ Theorem for the probability of A given B with a parti-
tion of A.

Test Questions:

1. State Bayes’ Theorem when we partition the sample space.

2. Let {Ai } be a partition of A, and assume that whenever any Ai happens, B
does not happen. Are we allowed to ask about Prob (A|B), the probability A
happens given B? Why or why not.
3. Assume Teph Williams either gets a single, a double, walks, or strikes out. He
gets a single 30% of the time, a double 10% of the time, and a walk 8% of the
time. Can you figure out what is the probability Teph got a single given that
he’s on first? If you can, what is that probability?
Chapter 5

Counting II: Inclusion-Exclusion

5.1 Factorial and Binomial Problems

Key ideas:

1. Key terms: Know the difference between ‘how many’ and ‘what’s the proba-
bility’, circular orderings, relative orderings.
2. Key ideas: Method of Inclusion-Exclusion / Inclusion-Exclusion Principle.

Test Questions:

1. How many ways are there to order 8 people along a circle, assuming that all
that matters is relative ordering?
2. Consider an equilateral triangle table, with one seat at each vertex and two
other seats on each side. How many ways are there to sit 9 people at this table,
if all that matters is the relative ordering?

5.2 The Method of Inclusion-Exclusion

Key ideas:

1. Key terms: number of elements in a set, binomial coefficients.

2. Key ideas: Method of Inclusion-Exclusion / Inclusion-Exclusion Principle, the
At Least to Exactly Method.
3. Key formula: The Inclusion-Exclusion Principle, the Binomial Theorem.

Test Questions:

1. The first seven Fibonacci numbers are {1, 2, 3, 5, 8, 13, 21}, the first seven
prime numbers are {2, 3, 5, 7, 11, 13, 17}, and the first seven integers of the
form nk where n, k ≥ 2 are {4, 8, 9, 16, 25, 27, 32}. If we choose an integer
24 • Counting II: Inclusion-Exclusion

from 1 to 32 (and all integers are equally likely to be chosen), what is the prob-
ability we choose something in one of these three sets? In at least two of these
sets? In all three?

2. Consider four rolls of a fair die, and let Ai be the event that the ith roll is a six.
What is the probability we roll at least one six? Solve this using the Method of
Inclusion-Exclusion, and check your answer by doing the computation another
way.
3. Assume each card is equally likely to be in a hand of five cards. Does this
imply that there is a positive probability of getting all four aces in your hand?
Why or why not.

5.3 Derangements
Key ideas:

1. Key terms: Derangement.

2. Secondary terms (for applications): Graph, vertex, edge, complete graph,
compound edge, self-loop, bipartite

3. Key techniques: Method of Inclusion-Exclusion.

Test Questions:

1. As n increases, does the probability that a random arrangement of {1, 2, . . . , n}

is a derangement increase or decrease? Why? What if n was always even?
What if n were always odd?
2. As n tends to infinity, approximately how much more likely is it that a random
arrangement of {1, 2, . . . , n} begins with a 1 followed by something other than
2, versus beginning with a 1 followed by a 2? How much more likely is it that
at least the first number is in the first position than the first number and at least
one more are in the original positions?
3. What are the series expansion for e, e−1 and ex ? Do you think the series
expansion for e converges faster, slower or at the same rate as e−1 ? In other
words, if we take the first N terms of each series, which do you think is closer
to the true value?
Chapter 6

Counting III: Advanced Combinatorics

6.1 Basic Counting

Key ideas:

1. Key terms: Sampling with replacement, sampling without replacement.

2. Key techniques: Enumerating all cases.

Test Questions:

1. How many six digit license plate numbers are there where each digit is a num-
ber from 0 to 9 and no number is used twice?

2. How many lottery numbers are possible when each lottery number is an in-
creasing sequence of six numbers chosen from 1 to 36 (inclusive) where no
two numbers may be adjacent to each other?

3. How many lottery numbers are possible when each lottery number is an in-
creasing sequence of six numbers chosen from 1 to 36 (inclusive) and exactly
two numbers are adjacent to each other?

4. Consider a standard deck of cards. Choose a card uniformly at random. If it is

a king we stop; if it is not a king we choose again and then stop. What is the
probability we chose a king?

6.2 Word Orderings

Key ideas:

1. Key terms: Distinguishable words (or arrangements or reorderings), relative

location of letters, multinomial coefficients.

2. Key ideas: Adding and then removing structure to aid in the analysis.
26 • Counting III: Advanced Combinatorics

3. Key formulas: Multinomial formula.

4. Key techniques: Proof by grouping.

Test Questions:

1. Can every multinomial coefficient n1 ,n2N,...,nk be written as a product of bi-

nomial coefficients m


`i , where N ≥ m1 ≥ m2 · · · ≥ mL ? (If we
don’t put a
i

restriction on the binomial coefficients the problem is trivial, as m

1 = m.)
3N


2. Consider multinomial coefficients k,`,m ; which multinomial coefficient (or
coefficients) is largest?
N
3. Expand (w + x + y + z)N by viewing it as ((w + x) + (y + z)) .
4. How many distinguishable words can you make from OHIOSTATE? What if
now you write the word in a circle?

6.3 Partitions
Key ideas:

1. Key terms: Cookie problem, stars and bars problem, partitioning a set, parti-
tions.
2. Key ideas: Adding partitions.

3. Key techniques: Method of grouping, story method, counting two different

ways, laziness principle.

Test Questions:

1. How many ways are there to divide 8 cookies among 12 people? What if
everyone must receive at least one cookie?
2. How many ways are there to divide 8 cookies among 12 people, given that
everyone must receive an even number of cookies? What if everyone must
receive an odd number of cookies?
3. How many lottery combinations are there where we use each number from 1
to 36 at most twice, and we must choose 6 numbers?
 Part II

Introduction to Random
Variables
Chapter 7

Introduction to Discrete Random Variables

7.1 Discrete Random Variables: Definition

Key ideas:

1. Key terms: Discrete random variable.

2. Key ideas: Random variables take on real numbers (so adding numbers, not
heads and tails).

Test Questions:

1. Define a discrete random variable. Give an example.

2. Why do we not want our random variable to take values from {Heads, Tails}?

7.2 Discrete Random Variables: PDFs

Key ideas:

1. Key terms: Probability density function (pdf), probability mass function, bino-
mial distribution.
2. Key ideas: Using binomial coefficients to compute probabilities when order
doesn’t matter.

Test Questions:

1. Define probability density function. Give an example.

R∞
P density function pX is defined as −∞ xpX (x)dx
2. The mean of a probability
if it is continuous, and n xn pX (xn ) if it is discrete. Must the mean always
exist?
3. An n-sided die is numbered 1 through n; what is the probability mass function
of the sum of two independent rolls?
30 • Introduction to Discrete Random Variables

4. Consider independent tosses of a fair coin. Imagine we keep tossing it until we

get two heads. What is the probability mass function of the number of tosses
needed?

7.3 Discrete Random Variables: CDFs

Key ideas:

1. Key terms: Cumulative distribution functions (CDFs).

2. Key techniques: Cumulative Distribution Function Method.

3. Key formulas: Limiting behavior for a cdf.

Test Questions:

1. Define the CDF of a random variable.

2. If the CDF of a random variable is zero at x = 0 what does that imply about
the probability density function?
3. Consider a fair coin that is tossed repeatedly (and independently) until we get
a head. Find the CDF of how long we must wait.

4.
Chapter 8

Introduction to Continuous Random

Variables

8.1 Fundamental Theorem of Calculus

Key ideas:

1. Key terms: Indefinite integral, anti-derivative, piecewise continuous function,

definite integral

2. Key ideas: Area under a curve given by an integral and corresponds to a prob-
ability.

3. Key formula: Fundamental Theorem of Calculus.

Test Questions:

1. State the Fundamental Theorem of Calculus; remember to include the word

‘area’!

2. Find the anti-derivatives of x2 ex , x cos(x) and ln(x) (the last is a bit tricky –
try to guess something and then correct your guess).

3. Find the area under the cubic y = x3 from x = 1 to x = 2.

8.2 PDFs and CDFs: Definitions

Key ideas:

1. Key terms: Continuous random variable, probability density function (pdf),

cumulative distribution function (cdf)

2. Key ideas: Calculus allows us to compute probabilities by looking at areas

under pdfs.
32 • Introduction to Continuous Random Variables

Test Questions:

1. Define the pdf and cdf of a continuous random variable.

2. If pX (x) = x for 1 ≤ x ≤ 2, is pX a density of a random variable? Why or
why not. If it isn’t, is there a simple modification of it that is a pdf?
3. Let pX (x) = 2x for 0 ≤ x ≤ 1 and 0 otherwise. Show that pX is a pdf and
calculate the probability that X is at most 1/2.

8.3 PDFs and CDFs: Examples

Key ideas:

1. Key terms: CDF, PDF.

2. Key ideas: Normalizing potential densities, derivative of CDF is the PDF.
3. Key formulas: Probabilities for four types of intervals.

Test Questions:

1. Is fX (x) = 1/(1 + x2 ) a pdf? If not what simple modification of this is?

2. Consider your modification of fX from the previous problem. What is the
probability that X is positive? Hint: you can solve this problem without solving
the previous!

8.4 Probabilities of Singleton Events

Key ideas:

1. Key terms: Probabilities of singleton events.

2. Key ideas: Probabilities of singleton events from a continuous random variable
are zero, unlike the situation of a discrete random variable (where they are non-
zero).

Test Questions:

1. Let X be chosen uniformly on [0, 1], so its density is fX (x) = 1 for 0 ≤ x ≤ 1

and 0 otherwise. What is the probability the first digit of X is an even number?
2. Consider the density from the previous problem. What is the probability every
digit of our chosen number is 2?
Chapter 9

Tools: Expectation

9.1 Calculus Motivation

Key ideas:

1. Key terms: Taylor series, Taylor coefficients, Maclaurin series.

2. Key ideas: Replace complicated function with Taylor series (locally close and
easier to work with).

Test Questions:

1. Find the Taylor series to degree 3 for sin x, cos x, ex and ex sin x.

2. If f (x) is an even function (so f (x) = f (−x)) does this imply anything about
the Taylor coefficients?

9.2 Expected Values and Moments

Key ideas:

1. Key terms: Expected values, moments, centered moments.

2. Key ideas: Use calculus (often integration by parts) to compute moments of

continuous random variables, moments give information on the distribution.

Test Questions:

1. For each positive integer n there is a constant cn such that cn /(1 + x2n ) is a
pdf. Which moments exist, and what can you say about the odd moments?

2. Which has a larger mean (i.e., first moment): a uniform random variable on
[−29, 29] or a uniform random variable on [1, 2]?
34 • Tools: Expectation

9.3 Mean and Variance

Key ideas:

1. Key terms: Mean, variance, standard deviation.

2. Key ideas: Variance versus the standard deviation.

R∞
3. Warning: Need −∞ |x|fX (x)dx to exist and be finite for the mean to exist.

Test Questions:

1. Find the mean and variance of a uniform random variable on [−a, a].

2. Find a random variable where the variance is larger than the standard deviation,
and another where the standard deviation is larger than the mean.

9.4 Joint Distributions

Key ideas:

1. Key terms: Joint probability density function, marginal density.

2. Key ideas: Finding marginal densities by integrating out other variables.

3. Key formula: Two random variables are independent if the joint density factors
as a product.

Test Questions:

1. Find a constant c such that f (x, y) = cxy for 0 ≤ x, y ≤ 1 and 0 elsewhere is

a probability density. Are X and Y independent? What are the marginals?

2. Find a constant c such that f (x, y) = cy cos(πxy/2) for 0 ≤ x, y ≤ 1 and 0

elsewhere is a probability density. Are X and Y independent? What are the
marginals? Why do you think there is an asymmetry with a y outside the sine
function and not an xy?

9.5 Linearity of Expectation

Key ideas:

1. Key terms: Expectation, variance.

2. Key ideas: Linearity of Expectation (the expected value of a sum is the sum of
the expected values).

3. Key formulas: Expected value of a sum, variance (in terms of expected values
of X and X 2 ).

Test Questions:
 Section 9.6: Properties of the Mean and the Variance • 35

1. Give two formulas for the variance, and talk about the advantages and disad-
vantages of each expression.

2. If we roll a fair die 6000 times, with all rolls independent, what is the expected
value and what is the variance of the sum of the rolls? What if instead we rolled
a 36 sided die 1000 times; now what is the mean and what is the variance? Do
you expect the two means to be the same? The two variances? Why or why
not.

9.6 Properties of the Mean and the Variance

Key ideas:

1. Key terms: Mean, variance, investment portfolios, risk, covariance

2. Key ideas: Linearity of expectation.

3. Key formulas: Expected value of a product of independent random variables,

mean and variance of sums of random variables, covariance formula.

Test Questions:

1. What is the formula for the covariance of two random variables?

2. If the covariance is zero must two random variables be independent? If not

give an example.

3. Consider a fair die and a fair coin. We roll the die first and let X be the result.
If X is an even number we roll the die again and call the outcome Y ; if we get
an odd number we toss the coin and let Y be 1 if we get a head and 0 if we get
a tail. Are X and Y independent? What is the covariance?

9.7 Skewness and Kurtosis

Key ideas:

1. Key terms: Skewness, kurtosis

2. Key ideas: Higher moments determine shape.

Test Questions:

1. Give an example of a distribution with zero skewness, or prove none exist.

What about positive skewness? Negative skewness?

2. Give an example of a distribution with zero kurtosis, or prove none exist. What
about positive kurtosis? Negative kurtosis?

3.

4.
36 • Tools: Expectation

9.8 Covariance: TBD

Key ideas:

1. Key terms:
2. Key ideas:

3. :
4. :

Test Questions:

1.
2.
3.
4.
Chapter 10

Convolutions and Changing Variables

10.1 Convolutions: Definitions and Properties

Key ideas:
1. Key terms: Convolution.
2. Key ideas: Commutativity of addition leads to commutativity of convolution.
3. Key formulas: Expressions for convolutions of discrete and continuous random
variables.
Test Questions:
1. Give a formula for the convolution of f and g, and f and g and h.
2. If f and g are just two functions that have finite integrals, is their convolution
still commutative? In other words, do we need f and g to be positive and
integrate to 1? What are the weakest conditions needed?
3. Find the convolution of f (x) = x2 on [−1, 1] and 0 elsewhere with g(x) =
e−x on [0, ∞) and 0 elsewhere.
4. Find the convolution of two independent rolls of a fair die.

10.2 Convolutions: Die Example

Key ideas:
1. Key terms: Convolution.
2. Key formula: Formula for the convolution of two discrete random variables.
Test Questions:
1. What is the shape of the sum of the rolls of two fair die?
2. Imagine you have a fair n sided die. Without doing the math, what do you
think the probability mass function will be for the sum of two rolls?
38 • Convolutions and Changing Variables

10.3 Convolutions of Several Variables

Key ideas:

1. Key terms: Convolution.

2. Key ideas: Grouping.

3. Key formulas: Convolution for a sum of random variables.

Test Questions:

1. What is the formula for the convolution of a sum of n random variables?

2. Imagine we have four die, each having n sides. If the four rolls are indepen-
dent, what is the probability mass function? Can you guess the answer from
looking at the analysis of regular die?

10.4 Change of Variable Formula: Statement

Key ideas:

1. Key terms: Change of variables, inverse function.

2. Key ideas: Using inverse functions, the need for the derivative to be positive
everywhere or negative everywhere.

3. Key formulas: Change of Variable Formula, derivatives of inverse functions.

Test Questions:

1. State the Change of Variables formula. Give the big examples from Calc III
(polar, cylindrical and spherical coordinates).

2. Why do we need to derivative to either always be positive or always be nega-

tive?

3. Go through the calculation to find the density of Y = 2X if X has the uniform

distribution on [−1, 1]. What about Z = 2X + 5? You might be able to just
write down the answer, but it’s good to do the calculation formally to make
sure you’re fine with the method.

4. Let X have density Ce−x/2 for x ≥ 0 and 0 otherwise. Find C so that this is a
probability density, and find the density of Yr = X r for any positive r. What
if r is negative? What if r is zero?
 Section 10.5: Change of Variables Formula: Proof • 39

10.5 Change of Variables Formula: Proof

Key ideas:

1. Key terms: Chain rule, cdf.

2. Key ideas: Using the cdf and the chain rule.

3. Key techniques: Method of the Cumulative Distribution Function.

Test Questions:

1. State the Chain Rule. Where is it used in the proof?

2. If X is uniform on [0, a] what is the pdf of Y = X r for a fixed > 0?

40 • Convolutions and Changing Variables
Chapter 11

Tools: Differentiating Identities

11.1 Geometric Series Example

Key ideas:

1. Key terms: Geometric series.

2. Key ideas: Differentiating identities.

Test Questions:

1. State the geometric series formula.

P∞
2. Find a formula for n=0 nk /2n for k ∈ {2, 3, 4}.
P∞ P∞ P∞
3. Find a formula for n=0 n(n−1)/2n , n=0 n(n−1)(n−2)/2n and n=0 n(n−
1)(n − 2)(n − 3)/2n .
4. Which is easier: the second problem (involving sums of nk ) or the previous
problem (involving more elaborate polynomials)? Why?

11.2 Method of Differentiating Identities

Key ideas:

1. Key terms: Differentiating identities.

2. Key formulas: Method of Differentiating Identities.

Test Questions:

1. Why must we be careful in the argument deriving the Method of Differentiat-

ing Identities formula?
2. Give an example where the derivative of a sum is not the sum of the derivatives.
Do you think you need to have infinitely many terms?
42 • Tools: Differentiating Identities

11.3 Applications to Binomial Random Variables

Key ideas:

1. Key terms: Binomial random variables, mean, variance.

2. Key ideas: Choosing special values of an extra parameter (q = 1/2 here).
3. Key formulas: Mean and variance for binomial random variables.

Test Questions:

1. What is the mean and variance of X if X ∼ Bin(n, p) (a binomial random

variable with parameters n and p)?
2. What is the k th moment Rof X ∼ Bin(n, p), where k is a positive integer?
∞
Recall the k th moment is −∞ xk fX (x)dx.

11.4 Applications to Normal Random Variables

Key ideas:

1. Key terms: Normal random variable, N (µ, σ 2 ), density of a Gaussian.

2. Key ideas: Using the Method of Differentiating Identities, moving key param-
eters between sides of equalities to facilitate differentiation.

Test Questions:

1. What are the mean and variance of X ∼ N (µ, σ 2 )?

th
2. If X ∼ N (0, 1),
R ∞find kthe k moment where k is a positive integer. Recall the
th
k moment is −∞ x fX (x)dx.

11.5 Applications to Exponential Random Variables

Key ideas:

1. Key terms: Exponential random variable, Exp(λ)

2. Key ideas: Method of differentiating identities, moving parameter between
sides of equalities, probability densities integrate to 1, different definitions are
possible for random variables depending on your normalization convention.
3. Key formulas: Mean and variance of exponential random variables.

Test Questions:

1. What are the mean and variance of X ∼ Exp(λ)?

th
2. If X ∼ Exp(λ),
R ∞find kthe k moment where k is a positive integer. Recall the
th
k moment is −∞ x fX (x)dx.
 Part III

Special Distributions
Chapter 12

Discrete Distributions

12.1 The Bernoulli Distribution

Key ideas:

1. Key terms: Bernoulli distribution, binary random variable, success, failure.

2. Key ideas: Random variables are real valued.

3. Key formulas: Mean and variance of a Bernoulli random variable.

Test Questions:

1. Why are random variables real valued?

2. What kind of random variable is the product of two binary random variables?

3. If X is a binary random variable with probability of success pX and Y is a

binary random variable with probability of success pY , can you say anything
about the random variable XY ? Why or why not.

12.2 The Binomial Distribution

Key ideas:

1. Key terms: Binomial distribution, Binomial theorem.

2. Key techniques: Differentiating identities, proof by grouping, linearity of ex-

pectation, computing something two different ways.

3. Key formulas: Binomial theorem, mean and variance of binomial distribution.

Test Questions:

1. What is the mean and variance of X ∼ Bin(n, p)?

46 • Discrete Distributions

2. If X1 ∼ Bin(n1 , p1 ) and X2 ∼ Bin(n2 , p2 ) are two independent binomial

random variables, for what choices of n1 , n2 , p1 , p2 do we have a nice formula
for the density of X1 + X2 ? Find the density in those cases.
3. Let X ∼ Bin(1701, 1/2). What is the probability X is larger than its mean?

12.3 The Multinomial Distribution

Key ideas:

1. Key terms: Multinomial distribution, multinomial coefficient

2. Key techniques: Method of grouping.
3. Key formulas: Multinomial coefficients.

Test Questions:

1. Let n = 6. Consider
all binomial coefficients and all trinomial coefficients

of level n (thus nk and k1 ,k2 ,n−k
n


1 −k2
). What is the largest coefficient, and
why?
2. Write the trinomial coefficients as a product of binomial coefficients.
3. Find a formula for the multinomial coefficients with four summands as a prod-
uct involving a trinomial coefficient, and as a product of two binomial coeffi-
cients.
4. How many words can you make from MISSISSIPPI, if each letter must be used
exactly once and we cannot distinguish copies of the same letter? What is the
probability one of these words has all four of its S’s consecutive?

12.4 The Geometric Distribution

Key ideas:

1. Key terms: Geometric random variable.

2. Key idea: Memoryless process.
3. Key techniques: Differentiating identities.

Test Questions:

1. What is the mean and variance of a geometric random variable?

2. What isRthe k th moment of a geometric random variable? Recall the k th mo-
∞
ment is −∞ xk fX (x)dx.
3. Let X be a geometric random variable with probability p. What is the most
likely value of n? In other words, what n (as a function of p) has the greatest
probability of happening?
 Section 12.5: The Negative Binomial Distribution • 47

12.5 The Negative Binomial Distribution

Key ideas:

1. Key terms: Negative binomial distribution.

2. Key techniques: Proof by story, differentiating identities.

3. Key formulas: Mean and variance of negative binomial distribution.

Test Questions:

1. What is the mean and variance of a negative binomial random variable with
parameters r and p.

2. If X1 is a negative binomial random variable with parameters r1 and p1 , and

X2 is a negative binomial random variable with parameters r2 and p2 , for what
choices of r1 , r2 , p1 , p2 can we write down a simple formula for the probability
density of X1 + X2 ? Write down the density in those cases.

3. Let X be a negative binomial random variable with parameters p and r. What

is the most likely value of n? In other words, what n (as a function of r and p)
has the greatest probability of happening?

12.6 The Poisson Distribution

Key ideas:

1. Key terms: Poisson random variable.

2. Key techniques: Method of differentiating identities.

Test Questions:

1. What is the mean and variance of a Poisson random variable with parameter
λ?

2. What is the k th moment

R ∞ of a Poisson random variable with parameter λ? Recall
the k th moment is −∞ xk fX (x)dx.

3. Let X be a Poisson random variable with parameter λ. What is the most likely
value of n? In other words, what n (as a function of λ) has the greatest proba-
bility of happening?

12.7 The Discrete Uniform Distribution

Key ideas:

1. Key terms: Discrete uniform distribution.

2. Key formulas: Formulas for sums of powers of consecutive integers.

48 • Discrete Distributions

Test Questions:

1. What is the mean and variance of the discrete uniform random variable on
{10, 11, . . . , 20}?
2. If X is the discrete uniform random variable on {1, 2, 4, 8} and Y is the dis-
crete uniform random variable on {1, 3, 9} what can you say about XY ?
Chapter 13

Continuous Random Variables: Uniform

and Exponential

13.1 The Uniform Distribution

Key ideas:

1. Key terms: Uniform distribution.

2. Key formulas: Mean and variance of uniform random variables.
3. Key ideas: Using calculus to find means and variances, convolutions.
4. Key techniques: Using the chain rule to generate random variables from a
uniform random variable.

Test Questions:

1. What is the mean and variance of X ∼ Unif(a, b)?

2. What is the mean and variance of the sum of four independent random vari-
ables on [0, 1]?
3. What is the density of the sum of four independent uniform random variables
on [0, 1]?

13.2 The Exponential Distribution

Key ideas:

1. Key terms: Exponential random variable, Gamma function, Erlang distribu-

tion.
2. Key formulas: Mean and variance of exponential random variables, integral
formula for the Gamma function.
50 • Continuous Random Variables: Uniform and Exponential

3. Key ideas: Integration by parts.

4. Key techniques: Cumulative distribution method for generating random vari-
ables (also known as the inverse transform sampling or the inverse transforma-
tion method).

Test Questions:

1. Use the method of differentiating identities to find the mean and variance of an
exponential random variable.

2. Let X and Y be two independent exponential random variables with parameter

λ. Can you say anything nice about the density of XY ? If so what is it?
3. Show Γ(n + 1) = n! if n is a non-negative integer.
4. Some authors write an exponential random variable differently, with parameter
1/λ (so the argument of the exponential is xλ and not x/λ. In this convention
find the normalization constant for the density, the mean and the variance.
Chapter 14

Continuous Random Variables: The Normal

Distribution

14.1 Determining the Normalization Constant

Key ideas:

1. Key terms: N (µ, σ 2 ) and N (0, 1), Gaussian random variable, bell curve (de-
fined in the introduction to this chapter).
2. Key techniques: Theory of normalization constants, change of variable to polar
coordinates (polar trick).

Test Questions:

1. If X ∼ N (0, 1) find the mean and variance of Y = 10X + 4.

2. What is the probability that the square of a standard normal random variable is
positive?

14.2 Mean and Variance

Key ideas:

1. Key terms: Density of a Gaussian.

2. Key ideas: Changing variables (in particular bounds of integration), L’Hopital’s
rule.
3. Key formulas: Mean and variance of a normal.

Test Questions:

1. What is the k th moment of aR normal random variable with parameters µ and

∞
σ? Recall the k th moment is −∞ xk fX (x)dx.
52 • Continuous Random Variables: The Normal Distribution

2. Let X ∼ N (µ, σ 2 ) with µ > 0. What happens to the probability that |X −µ| ≤
µ/2 as σ → 0?

3. Let Y = |X|. What are the mean and variance of Y ?

14.3 Sums of Normal Random Variables

Key ideas:

1. Key terms: Density of normal random variables, convolution.

2. Key ideas: Attack special case and change variables, separate variables, group-
ing parentheses, method of divine inspiration.

3. Key formulas: Density for sums of independent normal random variables (the
sum is normal).

Test Questions:

1. Is there a way to group sums of normal random variables to use fewer sums
than other ways, or are all groupings using the same number of sums?

2. Using the method of grouping and the result for the sum of two standard nor-
mals, find the sum of eight independent standard normals.

3. Do the methods of this section carry over to sums of absolute values of inde-
pendent normal random variables?

14.4 Generating Random Numbers from Normal Dis-

tributions
Key ideas:

1. Key terms: Cumulative distribution function (cdf), error function (Erf).

2. Key ideas: Approximation of the cdf, exploiting symmetries.

3. Key formulas: Series expansion for Erf.

Test Questions:

1. Why, when finding probabilities from the √standard normal, do we have to di-
vide the argument of the error function by 2?

2. Discuss the series expansion for the error function. Does it converge faster
or slower than the series expansion of a normal random variable? Does it
converge faster or slower for positive or negative inputs?
 Section 14.5: Examples and the Central Limit Theorem • 53

14.5 Examples and the Central Limit Theorem

Key ideas:

1. Key terms: Central Limit Theorem, standardization.

2. Key ideas: Standardization.

Test Questions:

1. What is the probability a normal random variable is within two standard devi-
ations of its mean? Does the answer depend on the value of the mean and the
value of the standard deviation?

2. What is the probability a normal random variable with mean 4 and standard
deviation 2 is positive?
54 • Continuous Random Variables: The Normal Distribution
Chapter 15

The Gamma Function and Related

Distributions

15.1 Existence of Γ(s)

Key ideas:

1. Key terms: Definition of Γ(s) (from the introduction to the chapter).

2. Key ideas: Borrowing decay.

Test Questions:
R1 R∞
1. Discuss when 0 xr dx and 1 xs dx converge and diverge.
R1
2. Does 0 x log xdx converge or diverge?

15.2 The Functional Equation of Γ(s)

Key ideas:

1. Key terms: Analytic continuation, pole.

2. Key ideas: Analytic (or meromorphic) continuation.
3. Key techniques: Integration by parts.
4. Key formulas: Geometric series formula, functional equation of Γ(s).

Test Questions:

1. Write down the functional equation of Γ(s).

2. In the proof of the functional equation of Γ(s), what must we assume about s
and why?
56 • The Gamma Function and Related Distributions

3. Why is it sufficient to understand Γ(s) for the real part of s between 0 and 1 in
order to know its behavior for all real s?
4. For what s is Γ(s) undefined?

15.3 The Factorial Function and Γ(s)

Key ideas:
1. Key terms: Gamma function, factorial function.
2. Key techniques: Recursion, induction.
3. Key formulas: Γ(n + 1).
Test Questions:
n


1. Express k as a ratio of Gamma functions.
n+1 n n




2. Can you use Gamma function identities to prove k+1 = k + k+1 ?
3. Write the double factorial of 2n in terms of Gamma functions.

15.4 Special Values of Γ(s)

Key ideas:
1. Key terms: Γ(1/2).
2. Key ideas: Exploiting symmetry.
3. Key formulas: Cosecant identity.
Test Questions:
1. Find a formula for Γ(1/2 + t)Γ(1/2 − t).
√
2. Find a formula for Γ(1/2 + it)Γ(1/2 − it), where i = −1. How should we
interpret the left hand side? It represents a very important quantity.

15.5 The Beta Function and the Gamma Function

Key ideas:
1. Key terms: Beta distribution, B(a, b).
2. Key ideas: Using fundamental relation of Beta distribution to find Γ(1/2).
3. Key formulas: Density of beta random variable, fundamental relation of the
Beta distribution.
Test Questions:
1. What is Γ(1/2)?
2. What are the mean and variance of X ∼ B(a, b)?
 Section 15.6: The Normal Distribution and the Gamma Function • 57

15.6 The Normal Distribution and the Gamma Func-

tion
Key ideas:
1. Key terms: Double factorial.
2. Key ideas: Adding zero, finding moments of the normal distribution through
the Gamma function.
Test Questions:
1. What are the moments of the standard normal?
2. What are the moments of the absolute value of the standard normal?

15.7 Families of Random Variables

Key ideas:
1. Key terms: Gamma distribution, Weibull distribution, shape and scale param-
eters, family of distributions.
2. Key ideas: Distributions with free parameters.
3. Key formulas: Densities of Gamma and Weibull distributions.
Test Questions:
1. What are the mean and variance of a Gamma distribution with parameters k
and σ?
2. What are the mean and variance of a Weibull distribution with parameters k
and σ?

15.8 Appendix: Cosecant Identity Proofs

Key ideas:
1. Key terms: Gamma function, cosecant identity.
2. Key ideas: Careful when using dummy variables about using the same name
twice.
3. Key techniques: Good change of variables, polar coordinate trick.
4. Key formulas: Cosecant identity, Gregory-Leibniz formula for π.
Test Questions:
1. Why does dxdy, the area element in Cartesian coordinates, become rdrdθ in
polar coordinates?
2. Find the integral over a circle of radius 5, centered at the origin, of (x2 + y 2 )a
when possible (i.e., for which a does the integral converge)?
3. Can you use the cosecant formula to find Γ(1/4)? Why or why not.
58 • The Gamma Function and Related Distributions
Chapter 16

The Chi-square Distribution

16.1 Origin of the Chi-square Distribution

Key ideas:

1. Key terms: Chi-square distribution and density (from the introduction).

2. Key techniques: Cumulative distribution method.
3. Key formulas: Relation between chi-square random variables and normal ran-
dom variables.

Test Questions:

1. What is the density of a chi-square random variable?

2. What is the probability a chi-square random variable with parameter µ is less
than −µ?

16.2 Chi-square distributions and sums of Normal Ran-

dom Variables
Key ideas:

1. Key terms: Chi-square distribution, Jacobian, spherical coordinates.

2. Key ideas: Sums of squares of chi-square random variables are normal.
3. Key formulas: Change of variables formula, density for sum of squares of chi-
square random variables.
4. Key techniques: Direct integration, change of variables formula, theory of nor-
malization constants, convolutions.

Test Questions:
60 • The Chi-square Distribution

1. What is the probability that the sum of the squares of n independent chi-square
random variables, each with parameter 1, exceeds n?
2. If X ∼ χ2 (1) and Y = aX + b, what is the density function of Y and Y 2 ?

3. What is are the volume and surface areas of the n-sphere? What is their ratio as
n → ∞? What is the ratio of the volume to the volume of the unit hypercube
as n → ∞?
 Part IV

Limit Theorems
Chapter 17

Inequalities and Laws of Large Numbers

17.1 Inequalities
Key ideas:

1. Key ideas: Looking at specific random variables to get results in special cases,
and then seeing what general results can include them all.

Test Questions:

1. Calculate, for c > 2, the variance for the random variable X where
(
1
if |x| ∈ [2, c]
fX (x) = 2(c−2)
0 otherwise.

If we are told the variance is at most σ 2 , what is the largest c we can take, and
what is the resulting bound for Prob(X ≥ 2)?
2. Calculate, for p < 1, the variance for the random variable X where
 
2p
Prob(X = 2) = p, Prob X = − = 1 − p.
1−p

If we are told the variance is at most σ 2 , what is the largest p we can take (i.e.,
what is the resulting bound for Prob(X ≥ 2))?

17.2 Markov’s Inequality

Key ideas:

1. Key terms: Non-negative random variable.

2. Key ideas: Finding the appropriate metric to study (such as ratios of a and
E[X]), if x ≥ a then x/a ≥ 1 (useful in finding probabilities), recognizing the
definition of the mean.
64 • Inequalities and Laws of Large Numbers

3. Key techniques: Intermediate Value Theorem, non-negativity.

4. Key formulas: Markov’s inequality.

Test Questions:

1. Where did we use non-negativity in the proof of Markov’s inequality?

2. What fails in the proof of Markov’s inequality if we have an infinite mean?
3. Can we use Markov’s inequality for X ∼ N (0, 1)? Why or why not.
4. What is the probability X ∼ Exp(λ) is three times its mean? What bound
does Markov’s inequality give?

17.3 Chebyshev’s Inequality

Key ideas:

1. Key terms: Chebyshev’s inequality

2. Key ideas: Finding the appropriate metric to study (such as measuring how far
we are from the mean in multiples of the standard deviation), if |x − µ| ≥ kσ
then |x − µ|/(kσ) ≥ 1, recognizing the definition of the variance.
3. Key techniques: Deriving Chebyshev’s inequality from a special case of Markov’s
inequality.
4. Key formulas: Chebyshev’s Inequality.

Test Questions:

1. State Chebyshev’s inequality.

2. Bound the probability an exponential random variable is more than half of its
mean away from its mean.
3. Bound the probability that a normal random variable with mean 1 and variance
5 is more than 15 units from its mean.

17.4 The Boole and Bonferroni Inequalities

Key ideas:

1. Key terms: Boole and Bonferroni inequalities

2. Key ideas: Inclusion-exclusion principle.

Test Questions:

1. State the Boole and Bonferroni inequalities.

2. Assume all the sets Ai have the same probability. For what n is Boole’s in-
equality non-trivial?
 Section 17.5: Types of Convergence • 65

17.5 Types of Convergence

Key ideas:

1. Key terms: Convergence in distribution, weak convergence, convergence in

probability, almost sure convergence, almost everywhere, with probability 1,
converges surely, floor function.
2. Key ideas: Using CDF to calculate probabilities of events.

Test Questions:

1. Give an example of each time of convergence.

2. Some of the types of convergence imply others; when and where do you think
this happens? For example, if we have almost sure convergence do we have
convergence in distribution? Explore some examples of convergence and guess
when one type of convergence implies another type of convergence.

3.
4.

17.6 Weak and Strong Laws of Large Numbers

Key ideas:

1. Key terms: Weak and Strong Laws of Large Numbers

2. Key ideas: Applying Chebyshev’s inequality.

Test Questions:

1. State the Weak and Strong Laws of Large Numbers.

2. If you knew the fourth moment were finite, could you get a better rate of con-
vergence in the Weak Law of Large Numbers?

3. Can you prove the Weak Law of Large Numbers if the random variables do
not have finite variance? What if you know they are non-negative? Is that
assumption enough? Can you prove the result for general random variables?
4. What if the random variables are allowed to be dependent (but with finite vari-
ance): must the Weak Law of Large Numbers still hold?
66 • Inequalities and Laws of Large Numbers
Chapter 18

Stirling’s Formula

18.1 Stirling’s Formula and Probabilities

Key ideas:

1. Key terms: Stirling’s formula, binomial coefficients.

2. Key ideas: Series expansions.

Test Questions:

1. State Stirling’s formula.

2. For n very large, use Stirling’s formula to estimate log(n!).

18.2 Stirling’s Formula and Convergence of Series

Key ideas:

1. Key terms: Geometric series, double factorial.

2. Key ideas: Convergence of a series.

Test Questions:

1. Estimate (2n)!!; in particular, compare it to n!.

2. Generalize the above problem to triple factorials (or higher!).

18.3 From Stirling to the Central Limit Theorem

Key ideas:

1. Key terms: Central Limit Theorem.

68 • Stirling’s Formula

2. Key ideas: Taking logarithms of products.

Test Questions:

1. Why is the limit of a product not always the product of the limits?
2. Can you generalize the argument to p 6= 1/2?

18.4 Integral Test and the Poor Man’s Stirling

Key ideas:

1. Key terms: Geometric mean, Euler-Maclaurin formula

2. Key ideas: Taking a logarithm of a product, using the integral test.

Test Questions:
PN
1. Use the integral test to approximate n=1 1/nc for c > 0.
2. Generalize the arguments here to estimate the double factorial (2n)!!.

18.5 Elementary Approaches towards Stirling’s For-

mula
Key ideas:

1. Key terms: Dyadic decompositions.

2. Key ideas: Dyadic decompositions, multiplying by 1, telescoping sums.

Test Questions:

1. What if we used a triadic decomposition (breaking into three parts)? Do you

think this would do better or worse? What if we broke into a prime number of
pieces and iterated?

18.6 Stationary Phase and Stirling

Key ideas:

1. Key terms: Gamma function.

2. Key ideas: Method of Stationary Phase.

Test Questions:
2
1. If we replace e−x with e−x in the definition of the Gamma function, calling
the new expression G(s), can we use stationary phase to estimate the resulting
integral, and if so what bound would we obtain?
 Section 18.7: The Central Limit Theorem and Stirling • 69

18.7 The Central Limit Theorem and Stirling

Key ideas:

1. Key terms: Poisson distribution.

2. Key ideas: Using the Central Limit Theorem for theoretical investigations and
not just estimating probabilities of events.

Test Questions:

1. Can you prove Stirling’s formula by looking at other random variables, for
example sums of Bernoulli random variables?
70 • Stirling’s Formula
Chapter 19

Generating Functions and Convolutions

19.1 Motivation
Key ideas:

1. Key terms: Poisson random variable, probability density function.

2. Key ideas: Multiplying by 1.

Test Questions:

1. What is the probability density function for a Poisson random variable?

2. Poisson random variables are stable: if you sum two independent Poisson ran-
dom variables you get a Poisson random variable. What other distributions are
stable?

19.2 Definition
Key ideas:

1. Key terms: Generating function, Fibonacci numbers, eigenvector, eigenvalue.

2. Key ideas: Geometric series formula, partial fractions.

3. Key formulas: Binet’s formula.

Test Questions:
 √ n  √ n
1. State Binet’s formula. More generally, consider an = √1α 1+β α − √1
α
1− α
β .
For what α, β is the following an integer for all integer n?

2. What would be the analogue for a telescoping sum for multiplication?

72 • Generating Functions and Convolutions

3. Come up with an interpretation for what it means for an infinite product to

Q∞ 2
converge. Does n=2 n2n−1 converge, and if so to what?

4. Do A and AT have the same eigenvalues? The same eigenvectors? Why or

why not.

19.3 Uniqueness and Convergence of Generating Func-

tions
Key ideas:

1. Key terms: Generating function.

2. Key ideas: Using generating functions to show sequences are equal, recovering
terms by differentiating.

Test Questions:

1. State the generating function for the sequence {an }; what must we assume
about the growth rate of an to ensure it converges for |s| < δ?

2. Hypothesize what a generating function of a two-sided sequence {an }∞

n=−∞
might be.

3. Hypothesize what a generating function of a doubly indexed sequence {am,n }∞

m,n=0
might be.

19.4 Convolutions I: Discrete random variables

Key ideas:

1. Key terms: Convolutions, probability generating function.

2. Key ideas: Binomial theorem, proof by grouping, convolutions give the density
of the sum of independent random variables.

Test Questions:

1. Define the convolution of two independent random variables. More generally,

what is the convolution of three, four, or n independent random variables?

2. The convolution gives the density of the sum of two independent random vari-
ables; find the analogue for the product or the quotient. Are any conditions
needed on the random variables for these to make sense?
 Section 19.5: Convolutions II: Continuous random variables • 73

19.5 Convolutions II: Continuous random variables

Key ideas:

1. Key terms: Probability generating function, convolution of functions.

2. Key ideas: Density of sums of random variables given by convolutions, con-

volutions are commutative.

Test Questions:

1. Is the difference of two independent random variables given by a convolution,

or a modified convolution? If so, is it commutative?

2. Generalize from sums of random variables to products of random variables: is

the resulting convolution commutative?

19.6 Definition and properties of moment generating

functions
Key ideas:

1. Key terms: Moment, centered moment, moment generating function.

2. Key ideas: If Y = αX + β, the moment generating functions of X and Y are

simply related.
2
3. Key example: The function f (x) = e−1/x for x 6= 0 (and 0 for x = 0) is
infinitely differentiable but does not agree with its Taylor series on any interval.

Test Questions:

1. Find another function which is infinitely differentiable but agrees with its Tay-
lor series only at a point.

2. If f (x) and g(x) have converging Taylor series for |x| < δ, does f (x)g(x)
have a converging Taylor series? If yes, what is it and for what x does it
converge?

3. If W = αX 2 + βX + γ, with α 6= 0, is there a nice relation between the mo-

ment generating function of X and W ? Hint: Consider the standard normal
and a chi-square with one degree of freedom.

4. Find the moments of the standard exponential and the standard normal in terms
of values of the Gamma function.
74 • Generating Functions and Convolutions

19.7 Applications of Moment Generating Functions

Key ideas:

1. Key terms: Mean, variance.

2. Key ideas: Computing moments from the moment generating function.

Test Questions:

1. Find the moment generating function of the standard normal and the standard
exponential random variables, and use that to calculate the first four moments.
2. The first two moments E[X] and E[X 2 ] always satisfy E[X 2 ] − E[X]2 ≥ 0.
Are there similar relations involving the first three or the first four moments?
Try to find something by looking at specific distributions to gather data.
Chapter 20

Proof of The Central Limit Theorem

20.1 Key Ideas of the Proof

Key ideas:

1. Key terms: Test function.

2. Key ideas: Equal integrals against a large class of test functions implies equal-
ity of functions.

Test Questions:

1. Assume f and g are continuous functions on [−1, 1] and

Z 1 Z 1
x2n f (x)dx = x2n f (x)dx
−1 −1

for all non-negative integers n. Must f = g? Why or why not? If f and g are
not equal, are there any common properties they must share? In other words,
what can you say about f − g?

20.2 Statement of the Central Limit Theorem

Key ideas:

1. Key terms: Normal distribution, Central Limit Theorem, Berry - Esseen theo-
rem.

2. Key ideas: Standardizing sums of random variables.

Test Questions:

1. State the Central Limit Theorem – remember to list the conditions on the ran-
dom variables.
76 • Proof of The Central Limit Theorem

2. Why is it important that the random variables be independent in the Central

Limit Theorem?

3. Let U1 , . . . , Un be i.i.d.r.v. random variables whose odd moments vanish,

whose variance is 1 and whose fourth moment is 1/2, let X1 , . . . , Xn be i.i.d.r.v.
random variables whose odd moments vanish, whose variance is 1 and whose
fourth moment is 1, and let Y1 , . . . , Yn be i.i.d.r.v. random variables whose odd
moments vanish, whose variance is 1 and whose fourth moment is 2, and let
Z1 , . . . , Zn be i.i.d.r.v. random variables whose odd moments vanish, whose
variance is 1 and whose fourth moment is 4. Give examples of such random
variables, or prove they cannot exist.

4. Consider the sums from the previous problem when they exist (thus U = U1 +
· · · + Un , . . . , Z = Z1 + · · · + Zn ). Which sums do you think converge to
being normally distributed fastest?

20.3 Means, Variances and Standard Deviations

Key ideas:

1. Key terms: Mean, variance, standard deviation.

2. Key ideas: Variance measures how spread out a distribution is about its mean.

Test Questions:

1. Define the mean and the standard deviation of a distribution.

2. We know E[X 2 ] > E[X]2 ; is there a similar relation involving the second and
fourth moments?

3. If 0 ≤ X ≤ 100, which distributions have the smallest variance, and which

have the largest?

20.4 Standardization
Key ideas:

1. Key terms: Standardization of a random variable.

2. Key ideas: Method of the cumulative distribution function.

Test Questions:

1. Let X1 , . . . , Xn be independent random variables where Xk ∼ Exp(k). Stan-

dardize X = X1 + · · · + Xn .

2. If Y = X m for m > 0, find a relation between the densities fY and fX .

 Section 20.5: Needed Moment Generating Function Results • 77

20.5 Needed Moment Generating Function Results

Key ideas:

1. Key terms: Moment generating functions.

2. Key ideas: Complex analysis provides important results on moment generating

functions, densities and cumulative distribution functions, adding zero.

Test Questions:

1. What is the moment generating function of X ∼ N (µ, σ 2 )?

2. State the two key results from Complex Analysis.

20.6 Special Case: Sums of Poisson Random Variables

Key ideas:

1. Key terms: Poisson random variables, Big-Oh notation.

2. Key ideas: Taylor expanding.

Test Questions:

1. Taylor expand log(1 + x2 ) for x small.

2. Taylor expand log(1 + ex ) for x small.

3. Find two functions f (x) and g(x) such that each is Big-Oh of the other.

20.7 Proof of the CLT for general sums via MGF

Key ideas:

1. Key terms: Moment generating function.

2. Key ideas: Taylor expansion.

Test Questions:

1. Give an example of a function whose moment generating function does not

converge.

2. Go through the proof for a specific probability distribution, say a uniform on

[−1, 1].
78 • Proof of The Central Limit Theorem

20.8 Using the Central Limit Theorem

Key ideas:

1. Key terms: Z-transform.

2. Key ideas: Change of base formula.

Test Questions:

1. If X ∼ N (2, 9), what is Z? If we observe x = 20 what is the corresponding z

value?
2. Let Z ∼ N (0, 1) and let Sp be the smallest interval such that Prob(Z ∈ Sp ) =
p for a fixed p ∈ (0, 1). Prove Sp is symmetric about Z’s mean.
Chapter 21

Fourier Analysis and the Central Limit

Theorem

21.1 Integral transforms

Key ideas:

1. Key terms: Integral transform, kernel, Laplace transform, Fourier transform,

characteristic function, Schwartz space, complex conjugate, compact support,
step function.

2. Key ideas: Often it is easier to transform to another space, analyze the problem
there, and then transform back.

Test Questions:

1. Define the Fourier and Laplace transform. Show for suitable functions that the
two transforms are related.

2. Consider a new transform K given by

Z ∞
2
(K(f ))(t) = e−(tx) f (x)dx.
−∞

For what probability densities f does this exist? If K(f ) = K(g) must f = g?
1 1
3. Is the Cauchy density π 1+x2 in the Schwartz space? Why or why not.

4. If the Laplace transforms of two continuous functions are equal, are the func-
tions equal? Why or why not.

5. Find formulas for cos(nx) and sin(nx) by using eix = cos(x) + i sin(x) and
eix eiy = ei(x+y) .
80 • Fourier Analysis and the Central Limit Theorem

21.2 Convolutions and Probability Theory

Key ideas:

1. Key terms: Convolution, square-integrable, Cauchy-Schwarz inequality.

2. Key ideas: Adding zero, grouping, calculating probabilities in small intervals
and taking limits.

Test Questions:

1. When is the Cauchy-Schwarz inequality an equality?

2. See how close the Cauchy-Schwarz inequality is to an equality on [0, 1] for

f (x) = xn and g(x) = xm , and then for f (x) = enx and g(x) = emx .
3. What is the Fourier transform of the Fourier transform, assuming both exist?
4. If F denotes the Fourier transform, what happens if we apply this operator four
times to a function f ?

21.3 Proof of the Central Limit Theorem

Key ideas:

1. Key terms: Statement of the Central Limit Theorem, big-Oh notation

2. Key ideas: Taylor expansions

Test Questions:

1. Try going through the proof of the Central Limit Theorem but without assum-
ing the mean is zero. What happens? Does the argument still work at the cost
of more algebra?
2. Instead of using characteristic functions (coming from the Fourier transform)
what if we used moment generating functions? If we assume these exist would
a similar argument hold?

3. Does the moment generating function of a Cauchy random variable exist?

What of the Fourier transform?
4. What if instead of sums of independent random variables we had products –
does a Central Limit Theorem hold in this case, and if yes what is the density?
One approach is to use logarithms, but one can also use other transforms and
attack it directly.