Mathematics of Probability

Probability is all about the possibility of various outcomes. The set of all possible outcomes is
called the sample space. The sample space for a coin flip is {heads, tails}. The sample space
for the temperature of water is all values between the freezing and boiling point. Only one
outcome in the sample space is possible at a time, and the sample space must contain all possible
values. The sample space is often depicted as Ω (capital omega) and a specific outcome as ω
(lowercase omega). We represent the probability of an event ω as P(ω).

P(A) means "Probability Of Event A"

The two basic axioms of probability are:

Axioms of Probability

In plain English, the probability of any event has to be between 0 (impossible) and 1 (certain),
and the sum of the probabilities of all events should be 1. This follows from the fact that the
sample space must contain all possible outcomes. Therefore, we are certain (probability 1) that
one of the possible outcomes will occur.

A random variable x, is a variable which randomly takes on values from a sample space. We
often indicate a specific value x can take on with italics. For example, if x represented the
outcome of a coin flip, we might discuss a specific outcome as x = heads. Random variables can
either be discrete like the coin, or continuous (can take on an uncountably infinite amount of
possible values).

The empirical probability of an event is given by number of times the event occurs divided by
the total number of incidents observed. If for n trials and we observe successes, the probability
of success is s/n. In the above example. Any sequence of coin tosses may have more or less than
exactly 50% heads.

Theoretical probability on the other hand is given by the number of ways the particular event
can occur divided by the total number of possible outcomes. So a head can occur once and
possible outcomes are two (head, tail). The true (theoretical) probability of a head is 1/2.
Joint Probability
Probability of events A and B denoted by P(A and B) or P(A ∩ B)is the probability that events
A and B both occur. P(A ∩ B) = P(A). P(B) . This only applies if A and B are independent,
which means that if A occurred, that doesn’t change the probability of B, and vice versa.

Conditional Probability
Independent Events
Events can be "Independent", meaning each event is not affected by any other events.
Example: Tossing a coin.

Dependent Events
But events can also be "dependent" ... which means they can be affected by previous
events ...
Example: Marbles in a Bag
2 blue and 3 red marbles are in a bag.
What are the chances of getting a blue marble?
The chance is 2 in 5
But after taking one out the chances change!

So the next time:

if we got a red marble before, then the chance of a blue marble next is 2 in 4

if we got a blue marble before, then the chance of a blue marble next is 1 in 4

This is because we are removing marbles from the bag.

So the next event depends on what happened in the previous event, and is called dependent.

Replacement*

Note: if we replace the marbles in the bag each time, then the chances do not change and the events
are independent:

• With Replacement: the events are Independent (the chances don't change)
• Without Replacement: the events are Dependent (the chances change)

Ques: "What are the chances of drawing 2 blue marbles?"

Answer: it is a 2/5 chance followed by a 1/4 chance:
Did you see how we multiplied the chances? And got 1/10 as a result.
The chances of drawing 2 blue marbles is 1/10
Notation

We love notation in mathematics! It means we can then use the power of algebra to play
around with the ideas. So here is the notation for probability:
 P(A) means "Probability Of Event A"

In our marbles example Event A is "get a Blue Marble first" with a probability of 2/5:
P(A) = 2/5
And Event B is "get a Blue Marble second" ... but for that we have 2 choices:
• `If we got a Blue Marble first the chance is now 1/4
• If we got a Red Marble first the chance is now 2/4

So we have to say which one we want, and use the symbol "|" to mean "given":
P(B|A) means "Event B given Event A"
In other words, event A has already happened, now what is the chance of event B?

P(B|A) is also called the "Conditional Probability" of B given A.

And in our case:

P(B|A) = 1/4

So the probability of getting 2 blue marbles is:

"Probability of event A and event

B equals the probability
of event A times the probability of event B given event A"

Using Algebra we can also "change the subject" of the formula, like this:
Start with: P(A and B) = P(A) x P(B|A)
Swap sides: P(A) x P(B|A) = P(A and B)
Divide by P(A): P(B|A) = P(A and B) / P(A)

Let us consider A and B are not independent, because if A occurred, the probability of
B is higher. When A and B are not independent, it is often useful to compute the conditional
probability, P (A|B), which is the probability of A given that B occurred: P(A|B) = P(A ∩ B)/
P(B).
 Similarly, P(B|A) = P(A ∩ B)/ P(A) . We can write the joint probability of as A and B
as P(A ∩ B)= P(A)xP(B|A), which means : “The chance of both things happening is the chance
that the first one happens, and then the second one given the first happened.”

Example 1: A jar contains black and white marbles. Two marbles are chosen without
replacement. The probability of selecting a black marble and then a white marble is 0.34, and
the probability of selecting a black marble on the first draw is 0.47. What is the probability of
selecting a white marble on the second draw, given that the first marble drawn was black?

Solution:
P(Black and White) 0.34
P(White|Black) = = = 0.72 = 72%
P(Black) 0.47
Example 2: The probability that it is Friday and that a student is absent is 0.03. Since there
are 5 school days in a week, the probability that it is Friday is 0.2. What is the probability that
a student is absent given that today is Friday?
Solution:
P(Friday and Absent) 0.03
P(Absent|Friday) = = = 0.15 = 15%
P(Friday) 0.2

Bayes Theorem
Have you ever seen the popular TV show ‘Sherlock’ (or any crime thriller show)? Think
about it – our beliefs about the culprit change throughout the episode. We process new
evidence and refine our hypothesis at each step. This is Bayes’ Theorem in real life!

Now, let’s understand this mathematically. This will be pretty simple now that our basics are
clear.
Consider that A and B are any two events from a sample space S where P(B) ≠ 0. Using our
understanding of conditional probability, we have:

P(A|B) = P(A ∩ B) / P(B)

Similarly, P(B|A) = P(A ∩ B) / P(A)

It follows that P(A ∩ B) = P(A|B) * P(B) = P(B|A) * P(A)

Thus, P(A|B) = P(B|A)*P(A) / P(B)

This is the Bayes’ Theorem.

Here, P(A) and P(B) are probabilities of observing A and B independently of each other.
That’s why we can say that they are marginal probabilities. P(B|A) and P(A|B) are
conditional probabilities.

P(A) is called Prior probability and P(B) is called Evidence.

P(B) = P(B|A)*P(A) + P(B|~A)*P(~A)

P(B|A) is called Likelihood and P(A|B) is called Posterior probability.

Equivalently, Bayes Theorem can be written as:

posterior = likelihood * prior / evidence

The formula of Bayes’ Theorem involves the posterior probability P(H | E) as the product of
the probability of hypothesis P(E | H), multiplied by the probability of the hypothesis P(H) and
divided by the probability of the evidence P(E).

Let us now understand each term of the Bayes’ Theorem formula in detail –

• P(H | E) – This is referred to as the posterior probability. Posteriori basically means

deriving theory out of given evidence. It denotes the conditional probability of H
(hypothesis), given the evidence E.
• P(E | H) – This component of our Bayes’ Theorem denotes the likelihood. It is the
conditional probability of the occurrence of the evidence, given the hypothesis. It
calculates the probability of the evidence, considering that the assumed hypothesis holds
true.
• P(H) – This is referred to as the prior probability. It denotes the original probability of
the hypothesis H being true before the implementation of Bayes’ Theorem. That is, this
probability is without the involvement of the data or the evidence.
 • P(E) – This is the probability of the occurrence of evidence regardless of the
hypothesis.

An Illustration of Bayes’ Theorem

Let’s solve a problem using Bayes’ Theorem. This will help you understand and visualize
where you can apply it. We’ll take an example which I’m sure almost all of us have seen in
school.
There are 3 boxes labeled A, B, and C:

• Box A contains 2 red and 3 black balls

• Box B contains 3 red and 1 black ball
• And box C contains 1 red ball and 4 black balls

The three boxes are identical and have an equal probability of getting picked. Consider that a
red ball is chosen. Then what is the probability that this red ball was picked out of box A?

Let E denote the event that a red ball is chosen and A, B, and C denote that the respective box
is picked. We are required to calculate the conditional probability P(A|E).

We have prior probabilities P(A) = P(B) = P (C) = 1 / 3, since all boxes h

ave equal

probability of getting picked.

P(E|A) = Number of red balls in box A / Total number of balls in box A = 2

/ 5

Similarly, P(E|B) = 3 / 4 and P(E|C) = 1 / 5

Then evidence P(E) = P(E|A)*P(A) + P(E|B)*P(B) + P(E|C)*P(C)

= (2/5) * (1/3) + (3/4) * (1/3) + (1/5) * (1/3) = 0.45

Therefore, P(A|E) = P(E|A) * P(A) / P(E) = (2/5) * (1/3) / 0.45 = 0.296

Applications of Bayes’ Theorem

We will discuss the three main applications of Bayes’ Theorem:

• Naive Bayes’ Classifiers

• Discriminant Functions and Decision Surfaces
• Bayesian Parameter Estimation