10/15/2023

Probability Theory and

Stochastic Processes
Lecture (2)
Prof. Mohammed Elmusrati
Email: [email protected]
University of Vaasa - Finland

Conditional Probability and Bayes’ Theorem

› Suppose that A has occurred, i.e., P (A) ≠0, the conditional

probability that B occurs on the condition that A has
occurred is given by

PA  B
P  B A 
P  A
› One simple way to prove the above formula by using Venn
diagram as shown in the next slide:

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 2

1
 10/15/2023

Conditional Probability

A AB B

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 3

Conditional Probability and Bayes’

Theorem
› If A B   , then events A and B are mutually exclusive, i.e.,
if A occurs then B could not have occurred and P B A  0
 
› If A  B , then the knowledge that event A has occurred
implies that event B has occurred and hence,
P  B A  1
› If A and B are statistically independent, then
P  A  B   P  A P  B   P  B A  P  B 

If P  B A   1 , is it necessary that P  A B   1 ? Explain by simple example!

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 4

2
 10/15/2023

Example
› Assume 9 cards numbered with {1,2,3,---,9}. One card has
been drawn randomly.
› Find
– The probability that this card is numbered 9
– The probability that it is greater than 8 given it is odd.
– The probability that it is odd given that it is greater than 8.
– Solution
– Let X={9} P(X)=1/9
– Let Y={1,3,5,7,9}  P(X\Y)=P(XY)/P(Y)=P(X)/P(Y)=1/5
– P(Y\X)=P(XY)/P(X)=P(X)/P(X)=1

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 5

Conditional Probability and Bayes’

Theorem

Bayes’
Theorem

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 6

3
 10/15/2023

Conditional Probability and Bayes’

Theorem
› In general, the events Bi, i=1,…,n are independent if and only
if for all collections of k distinct integers (i1,i2,…,ik) chosen
from the set (1,2,..,n) we have

 2 k
      
P Bi Bi Bi  P Bi P Bi P Bi ,2  k  n
1 1 2 k

› Total Probability:
The collection of sets {Bi}, i=1,…,n forms a partition of the
sample space S if Bi  B j   for i≠j and
B1UB2…UBn=S. For any event A  S we can write

       
n n
P A   P A  Bi   P A | Bi P Bi
i1 i1
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 7

Conditional Probability and Bayes’

Theorem
A
S
B6 B3
B1

B5 B4
B2

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 8

4
 10/15/2023

Conditional Probability and Bayes’

Theorem
› The Bayes’ theorem can be written as

P  B A 
   
P A Bi P Bi

 P A B  P B 
i n

i i
i1

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 9

Marginal Probability
› Suppose that the sample space S can be partitioned into two
different families of disjoint sets {Ai} and {Bj}, i.e.,
S  i1
m
Ai   nj1 B j
› It is clear that for any Bk, we represent it as the total intersection
with all events A’s such as Bk  i1 m
 Ai  Bk 
› Next slide shows graphical explanation for that.
› Then the marginal probability of even Bk is computed as
m
P  Bk    P  Ai  Bk 
i1
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 10

10

5
 10/15/2023

Marginal Probability
A5 A4 A3 A2 A1
S
B1

B2

B3

B2  A3
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 11

11

Example
› Suppose cards numbered one through ten are placed
in a hat, and then one of the cards is drown. A) What
is the probability that it is ten?
› If you are told that the number on the drawn card is
at least five, then, B) what is the conditional
probability that it is ten?
› S={1,2,3,4,5,6,7,8,9,10}, E={10}, P(E)=1/10
› F={5,6,7,8,9,10}, P(E\F)=P(EF)/P(F)=P(E)/P(F)
=1/10÷6/10=1/6

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 12

12

6
 10/15/2023

Example
› Suppose that three sensors transmit digital signals in a form of
packets to a receiver. The sensors are working on TDM such that
the first sensor sends its packet first then the second then the third
and so on. Assume that the TDM organizer fails so that every
sensor will send its packet in random and independent time.
Assume that the received packets are not overlapped so that the
receiver can separate between them.
› A) What is the probability that the receiver will receive the packets
in the correct order?
› B) What is the probability that the receiver will receive at least one
packet in its correct order?
› C) What is the probability that none of the packets will be arrived in
the correct order?

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 13

13

Example
› Define Ei, i=1,2,3, is the event that the ith packet arrive in its correct
order. It is clear that P(Ei)=1/3, i=1,2,3, and from conditional
probability
› P(EJ \ Ei)=P(Ei EJ)/P(Ei);
› P(Ei EJ)= P(EJ \ Ei)P(Ei)=1/2*1/3=1/6
› To solve A we should compute P(E1E2E3)
› P(E1E2E3)= P(E1E2)* P(E3\E1E2)=1/6 P(E3\E1E2)
› Then P(E1E2E3)=1/6
› B) P(E1or E2orE3)=P(E1)+ P(E2)+ P(E3)- P(E1E2)- P(E1E3)- P(E3E2)+
P(E1E2E3)=1-1/2+1/6=2/3
› C) The probability that none of the packets arrive in correct order is
1-2/3=1/3

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 14

14

7
 10/15/2023

Example

› The same solutions can A1 A2 A3

be obtained by looking
to the sample space of A1 A3 A2
the problem as shown in A2 A1 A3
the following table
where Ai refer to the A3 A1 A2
packet arrived from
A2 A3 A1
sensor i.
A3 A2 A1

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 15

15

Example
› Suppose we toss two fair dice. Let E denote the event
that the sum of the dice is six and F denote the event that
the first die equals four. Are those two events
independent? Prove!

– P(EF)=P({4,2})=1/36
– While
– P(E)*P(F)=5/36*1/6=5/216
– It is clear that E and F are not independent

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 16

16

8
 10/15/2023

Example
› It has been estimated in certain country that 10% of the population are
very obese (high levels of overweight).
› It has been estimated that 70% of those obese suffering from high blood
pressure. However, only 10% of non-obese people have this decease.
› One person has been randomly selected from the country, find:
– The probability that she has high pressure (regardless her weight)
– If we found that she has high pressure, what is the probability that she has
overweight?
– Solution:
– Let’s define the event O={the person is obese}, H={has H. pressure}, Oc=
complement{O}= Non-obese

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 17

17

Example
› In the previous example, write octave code to simulate the example
and compare the results of the theoretical and simulations

Simulation Results:

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 18

18

9
 10/15/2023

Example
› The same simulation problem
could be performed without using
the command find. For example
we can perform it with for loops
as shown next.
› The execution time for this code
was about 600 times more than
the first code.
› This is inefficient way to write
your codes. Always avoid loops
as much as you can.
› If you have a complex simulation
requires 1 day running time of
your computer, it will take 2 years
to finish with inefficient for loops.

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 19

19

Example
› A Quality control test is 95% effective in detecting damaged mobile
phone. However, the test also yields a “false positive” result for 1% of
the good mobile phones. It is known that 0.5% of the total mobile
phones have some kind of damages.
› For a test of certain mobile phone, the QC test was positive, What is
the probability that the mobile phone is really damaged
› Define K is the event that the tested mobile phone is damaged, and
C is the event of positive test is obtained.
P  K  C  P C K  P  K  P C K  P  K 
P K C    
P C  P C  P C K  P  K   P C K P  K   
0.95  0.005
 P K C    0.323!!
0.95  0.005  0.01  0.995
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 20

20

10
 10/15/2023

Exercise
› In example (8), to improve the damaged phone detection
process, should we increase the quality control test (to be
more than 95%) or reducing the false positive results (to
be less than 1%) ?
› To increase the probability of detection of really damaged
mobiles to 95% how should we reduce the false positive
probability?

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 21

21

Example
› Two temperature sensors send warning signal when T>30 oC. One sensor is
defected so that it sends always wrong warning. The other sensor working in
good way so that it sends always correct warning. The probability of the
temperature to be >30 oC is 0.1. If we receive warning, what is the probability that
it comes from the good sensor?
› Let’s define the following events A={good sensor sends}
B={defected sensor sends}, C={warning signal received}, and we want to find
P(A\C). It can be easily determined as

P C A  P  A  P C A  P  A 
P A C   
P C  P C A  P  A   P C B  P  B 
0.1  0 .5
 P A C    0. 091
0 . 1  0 .5  1  0 . 5
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 22

22

11
 10/15/2023

Random Variables
› The real valued functions defined on the sample space are known
as random variables. It can be discrete as well as continuous.
› An experiment consists of tossing two fair coins. Letting Y denotes
the number of heads appearing. Y is RV with possible values 0,1,2
with respective probabilities
› P(Y=0)=1/4, P(Y=1)=1/2, and P(Y=2)=1/4

0.5
P(Y)

0.25

0 1 2 Y
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 23

23

Example
› Suppose that we toss a coin having a probability p of coming up
heads. Letting N denotes the number of flips required to obtain
head, then assuming that the outcome of successive flips are
independent, N is a RV taking on one of the values 1,2,3,…, with
respective probabilities
– P(N=1)=p
– P(N=2)= p(1- p)
– P(N=3)= p(1- p)2 .....
– P(N=n)= p(1- p)n-1 P(N)

1 2 3 N
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 24

24

12
 10/15/2023

Continuous Random Variable

› Discrete random variables are discussed in the previous two slides.
› The random variables can be continuous as well.
› For example the temperature can be considered as a continuous RV. The
continuous random variables are represented by the probability density
function which will be discussed in the next slides.
› The probability density function of the temperature in Vaasa during
February may be described as

PDF
P T0  T  T0  T   f T  T T
0
T

-50 -40 -30 -20 T-10

0
0 +10 T
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 25

25

The Probability Density Function

› The amplitude of a received signal after a random channel is also random
variable but with uncountable possible values.
› We define a nonnegative function f(x) for all real x∈(-∞, ∞), having the
property that for any set B of real numbers

PX  B    f x dx
B

› Where f(x) is called probability density function (pdf)

› Some properties of the pdf
f x   0 for all values of x


 f x dx  1


MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 26

26

13
 10/15/2023

The Cumulative Distribution Function

› The Cumulative Distribution Function (CDF) is defined
such as
a
F  a   P  X  a   f X  x  dx


› The CDF is always a non-decreasing function, i.e. for any

a and b where b≥a, F(b) ≥F(a).

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 27

27

Discrete Random Variables

› The probability mass function p(a) is defined as
– p(a)=P(X=a)
› Properties 
p ( x i )  0, and  p x   1
i  
i

› The cumulative distribution function (CDF) can be

expressed as
F a   P X  a   p x  i
all xi  a

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 28

28

14
 10/15/2023

Example
› Suppose X has a probability  0, a 1
mass function given by 1
– p(1)=1/2, p(2)=1/3,p(3)=1/6  1 a  2
– Then the CDF is given by 
F a    2
5 2a3
6
1 a3
F(x) 
1
0.83
0.5

1 2 3 x
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 29

29

The Bernoulli Random Variable

› Suppose that a trail, or an experiment, whose
outcome can be classified as either a
“success” or as a “failure” is performed. Let
X=1 if the output is success and 0 if it is
failure, then the probability mass function of X
is given by
p 1  P X  1  p
p 0   P X  0  1  p 1-p
p

0 1
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 30

30

15
 10/15/2023

The Binomial Random Variable

› Suppose that the Bernoulli experiment has been performed n
independent times.
› If Y represents the number of successes that occurs in the n
trails, then Y is called binomial RV with parameters (n,p).
› In other words let RV X=1 with probability p or X=0 with
probability (1-p) as shown in the figure of previous slide.

Y  X k , X k , k  1,2,...n, are statistically independent

n

k 1

and identically distributed RV

 n  n
P Y  i   P  i     p i 1  p  , i  0,1, 2,..., n, where   
ni n!
 i  i  i ! n  i !
MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 31

31

Example
› Four unfair coins are flipped with probability of head
is 0.7. What is the probability that two heads are
obtained. What is the probability that at least one
tail is obtained.

4!
p ( 2)  0 .7 2 0 .3 2  0 .2646
2!2!

› The probability that at least one tail is obtained is

1-(0.7)^4=0.76

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 32

32

16
 10/15/2023

Example
› In a survey performed by service company regarding its
service like: Good or Bad. The survey has been
performed over large number of customers and they
found 67% of customers thought that the service is
good.
› In a randomly selected group of 7 people, find:
– The probability that non of them think the service is good
– At least one thing the service is good
– Exactly 4 believe the service is bad

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 33

33

Solution of Example
› Non of them believes that the services is good
means that all 7 will say it is bad. This leads to
n
p i     p i 1  p 
ni

 
i

› At least one thing the service is good: 1-

p(i=0)=0.9996
› Exactly 4 believe the service is bad

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA 34

34

17
 10/15/2023

Multinomial Distribution
› Multinomial distribution is a general extension of the
binomial distribution. In Binomial distribution each
event Xi has only two values 0 or 1. In Multinomial, it
can have K different values (events), and each value
has probability pk. Assume that there is n different
trails performed, the number of selecting each value k
is xk where x1+x2+..+xK=n, with probability:

MOHAMMED S. ELMUSRATI – UNIVERSITY OF VAASA

35

Example
› In the same survey example in slide 71, assume it
contains more than two outcome, for example
1. Excellent
2. Very good
3. Good
4. Not bad
5. bad
› In this situation we cannot use Binomial distribution.
However, Multinomial is the right distribution. Assume,
p1=0.2, p2=0.2, p3=0.4, p4=0.1, p5=0.1. In sample of 12
persons, what is the probability that x1=3, x2=3, x3=4,
x4=1, x5=1?
PROF. MOHAMMED ELMUSRATI

36

18
 10/15/2023

Example
› We can directly substitute in the PMF to obtain

› This leads to the probability of 0.0091.

PROF. MOHAMMED ELMUSRATI

37

19