Module 5 - Bayee's Theorem

Module 5 - Lesson

Some Basics

Marginal probability is the probability of an event, irrespective of other random variables. If

the random variable is independent, then it is the probability of the event directly, otherwise,
if the variable is dependent upon other variables, then the marginal probability is the
probability of the event summed over all outcomes for the dependent variables, called the
sum rule.

Marginal Probability: The probability of an event irrespective of the outcomes of other random
variables, e.g. P(A).

The joint probability is the probability of two (or more) simultaneous events, often described
in terms of events A and B from two dependent random variables, e.g. X and Y. The joint
probability is often summarized as just the outcomes, e.g. A and B.

Joint Probability: Probability of two (or more) simultaneous events, e.g. P(A and B) or P(A, B).

The conditional probability is the probability of one event given the occurrence of another
event, often described in terms of events A and B from two dependent random variables e.g.
X and Y.

Conditional Probability: Probability of one (or more) event given the occurrence of another event,
e.g. P(A given B) or P(A | B).

The joint probability can be calculated using the conditional probability; for example:

P(A, B) = P(A | B) * P(B)

This is called the product rule. Importantly, the joint probability is symmetrical, meaning that:

P(A, B) = P(B, A)

The conditional probability can be calculated using the joint probability; for example:

P(A | B) = P(A, B) / P(B)

The conditional probability is not symmetrical; for example:

The conditional probability can be calculated using the other conditional probability; for
example:
P(A|B) = P(B|A) * P(A) / P(B)

The reverse is also true; for example:

P(B|A) = P(A|B) * P(B) / P(A)

This alternate approach of calculating the conditional probability is useful either when the
joint probability is challenging to calculate (which is most of the time), or when the reverse
conditional probability is available or easy to calculate.

This alternate calculation of the conditional probability is referred to as Bayes Rule or Bayes
theorem.

It is based on the concept that probabilities could be revised based on the new information
that is made available. Baye'e theorem is used to validate the claims (or to calculate
probabilities) with the best available data. The initial belief will get improved with the new
available data that leads to a better belief.

For example, to decide whether Mr. X will develop cancer or not, if we know the values for the
population, we simply equate this value. In case if we get some additional information as to
what is the probability of someone getting cancer provided he is a smoker. In case Mr.X is a
smoker, then we can revise the probability of Mr.X getting cancer. Hence, with the newly
available data, the probabilities will get revised and better.

The basic formula is P(B|E) = P(B) X P(E|B) / P(E),

where P stands for probability, B for belief and E for evidence. P(B) is the probability that B is
true, and P(E) is the probability that E is true. P(B|E) means the probability of B if E is true, and
P(E|B) is the probability of E if B is true.

Bayee’s Theorem:

Let E1, E2, ..... , En be the set of n mutually exclusive and exhaustive events whose union is
the random space S of an experiment. If A is any arbitrary event of the sample space of the
above experiment with P(A) not equal to 0, then the probability of the event Ei, when the event
A has actually occurred is given by P(Ei/A) where

P(Ei/A) = P(Ei)* P(A/Ei) / ∑ P(Ei)* P(A/Ei)

P(Ei) = Probability associated with the happening of the event Ei, i=1,2,3,...n. We call this as
prior probabilities- probabilities before revision i.e., before the new information is taken into
account.

P(A/Ei) = Conditional probability of event A given that Ei has already occurred

P(Ei/A) = Posterior probability . The information A has already occurred allows us to re-
assess the probability P(Ei).

Example 1

There are 2 types of boxes. If a card is picked from a Type I box, Queen turns up 55% of time
and from Type II box, queen turns up 65% of time. A card is picked and turns it to be queen.
What is the probability that it is from a type I box?

There are two events, type

Card being picked from Type I box P(Type I) and card being picked from Type II box P(Type
II)

P(Type I)=0.5 and P(Type II)=0.5 (since we have only two types of boxes)

Another event we have is picking an queen

P(Queen/Type I)=0.55

P(Queen/Type II) = 0.65

To find: P(Type I/Queen)

By Baye's theorem,

P(Type I/Queen) = P(Type I)P(Queen /Type I)/ P(Type I)P(Queen /Type I)+ P(Type II)P(Queen
/Type II)

= 0.5 * 0.55 / 0.5*0.55+0.5*0.65

= 0.275/0.275+0.325

=0.275/0.6

=0.4583
Note:

1. Similarly, if a card picks and turns up queen, what is the probability that it is from type II
box?

By Baye's theorem,

P(Type II/Queen) = P(Type II)P(Queen /Type II)/ P(Type I)P(Queen /Type I)+ P(Type
II)P(Queen /Type II)

= 0.5 * 0.65 / 0.5*0.55+0.5*0.65

= 0.325/0.275+0.325

=0.325/0.6

=0.5417

Before picking the card we say that it is 0.5 from Type I box & 0.5 from Type II box.

After picking we have revised it to 0.4583 and 0.5417 using posterior probabilities. Hence, we
have revised the decisions.

The denominator denotes the probability of queen, P(queen)

Example 2

Sasha, the daughter of an Indian Billionaire, is getting married tomorrow, at a beachfront in

Italy. In recent years, it has rained only 5 days each year. Unfortunately, the weatherman has
predicted rain for tomorrow. When it actually rains, the weatherman correctly forecasts rain
90% of the time. When it doesn't rain, he incorrectly forecasts rain 10% of the time. What is
the probability that it will rain on the day of Sasha's wedding?

Solution: The sample space is defined by two mutually-exclusive events - it rains or it does
not rain. Additionally, a third event occurs when the weatherman John predicts rain. Notation
for these events appears below.

Event A1. It rains on Sasha's wedding.

Event A2. It does not rain on Sasha's wedding.

Event B. The weatherman John predicts rain.

From the given information, taking into consideration there are 365 days in a year,

P( A1 ) = 5/365 =0.0136985 [It rains 5 days out of the year.]

P( A2 ) = 360/365 = 0.9863014 [It does not rain 360 days out of the year.]

P( B | A1 ) = 0.9 [When it rains, the weatherman predicts rain 90% of the time.]

P( B | A2 ) = 0.1 [When it does not rain, the weatherman predicts rain 10% of the time.]

We want to know P( A1 | B ), the probability it will rain on the day of Marie's wedding, given a
forecast for rain by the weatherman. The answer can be determined from Bayes' theorem, as
shown below.

Note the somewhat unintuitive result. Even when the weatherman predicts rain, it rains only
about 11% of the time. Despite the weatherman's gloomy prediction, there is a good chance
that Sasha will not get rained on at her wedding.

Baye's theorem application can be evidently seen in medical tests.

3. Doctor has decided to prescribe medicine to 200 heart patients as follows: 50 get drug A,
50 get drug B and 100 get both. The 200 patients were chosen so that each had an 80%
chance of having a heart attack if given neither drug. Drug A reduces the probability of heart
attack by 35%, drug B reduces the probability of heart attack by 20% and when the two drugs
taken together work independently.

If a randomly selected patient has a heart attack, what is the probability that the patient was
given both the drugs?

If a randomly selected patient has a heart attack, what is the probability that the patient was
given drug A?

Solution

P(A) be the probability of giving drug A

P(B) be the probability of giving drug B

P(C) be the probability of giving drug A and B

P(A) = 0.25
P(B)=0.25

P(C) = 0.5

P(H/A)= probability of having heart attack when given drug A

P(H/B)= probability of having heart attack when given drug B

P(H/C)= probability of having heart attack when given both drugs A and B

P(H/A)=0.8*0.65 = 0.520

You may wonder, how is this calculated?

Note, the patients had an 80% chance of having a heart attack if given neither drug, hence 0.8
is taken above.

The drug A reduces the probability of heart attack by 35%., i.e., 0.35 and hence the probability
of having heart attack is 1-0.35 = 0.65

Similarly, P(H/B)= 0.8 *0.8 = 0.640

P(H/C)= 0.8 *0.65 *0.8 = 0.416

Note, the two drugs work independently, hence, 0.65 *0.8 is taken in the above step.

By Bayee’s theorem

P(C/H) = P(C) P(H/C) / P(A)P(H/A)+P(B)P(H/B)+P(C)P(H/C)

= 0.5 *0.416 / 0.25*0.520+0.25*0.640+0.5*0.416

= 0.208/0.13+0.16+0.208

= 0.208/0.498

= 0.4177

Hence,

If a randomly selected patient has a heart attack, the probability that the patient was given
both the drugs is 0.4177

P(A/H) = P(A) P(H/A) / P(A)P(H/A)+P(B)P(H/B)+P(C)P(H/C)

= 0.25 *0.520 / 0.25*0.520+0.25*0.640+0.5*0.416

= 0.13/0.13+0.16+0.208

= 0.13/0.498

= 0.2610

Hence,

If a randomly selected patient has a heart attack, the probability that the patient was given
drug A is 0.2610
 1. An urn contains five balls. Two balls are drawn and found to be white. The
probability that all the balls are white is

2. One card is drawn at random from a pack of 52 cards. What is the probability that the
card drawn is a face card (Jack, Queen and King only)?
Explanation:
Clearly, there are 52 cards, out of which there are 12 face cards.
P (getting a face card) = 12/52=3/13.
3. Balls numbered 1 through 20 are placed in a bag. Three balls are drawn out
of the bag without replacement. What is the probability that all the balls have
odd numbers on them?
4. A box contains 4 red and 6 blue balls. two balls
are picked at random without replacement.
What's the probability of drawing balls not of
the same color?
There is a 4/10 probability of picking a red ball, after which there is a 6/9 probability of
picking a blue ball. To get the probability of picking a red then blue, multiply these.
P1=24/90

Now we do the same calculation for drawing blue first, where the respective
probabilities are 6/10 and 4/9. To get the probability of picking blue then red, we
multiply again, and the answer is the same.
P2=24/90

To get the total probability, we add P1 and P2.

P=P1+P2=48/90
5.
6. You can simplify or convert to a decimal as needed.