EE325 - Probability and Random Processes

1 Set Theory 1.2 Functions

1.1 Introduction • Mapping: A mapping or a map f of A into B is a re-

lation such that ∀a ∈ A, ∃ a unique b s.t. (a,b) ∈ f.
• Def-Set: A collection of well defined objects. We then write f : A → B and b = f(a)
Examples: A = set of people, B = set of players in a Example: f : N → N ; f(n) = n2
match, C = {1, 2, 3, 4}, D = {x ∈ R|1 ≤ x ≤ 2}
• Def-Function:A function f: A → B is a rule that maps
• Russell’s Paradox: Let S is a set of sets which do not every element of A to exactly one element in B.
contain themselves as elements. This is not well de-
fined. If S in in the set, it shouldn’t be in it, and if it • ∀x ∈ A, ∃ unique y ∈ B such that, y=f(x).
is, it should be. Hence a contradiction.
Example, set of all men that shave themselves, so does • y is the image of x, and x is the pre-image of y.
a single barber belong to S?? A is the domain and B is the co-domain.
Range R is the subset of co-domain such that,
• To prove A ⊆ B : Take a ∈ A → x ∈ B {y ∈ B : ∃x ∈ A for which y=f(x)}
To prove A=B: A⊆ , B ⊆ A.
1.2.1 Types of Functions
• De-Morgan’s Laws:

A∪B =A∩B • One-to-one or Injective: f : A → B is said to be in-

jective if every element in the range R has a unique
A∩B =A∪B pre-image. Example f (x) = ex

• A-B = {x—x ∈ A and x ∈

/ B} • Onto or Surjective: f : A → B is surjective if R=B.
Every element in B has a pre-image in A. Example,
A4B = (A − B) ∪ (B − A) f : R → [0, 1], f (x) = sin(x) is onto but not one-to-one
= (A ∪ B) − (A ∩ B)
• Bijective: f : a → B is bijective if it is both injective
and surjective. Example, f : R → R, f (x) = x3 .
• AB = set of maps of B into A (B→ A)
A2 ≡ A{1,2} because
X n is the set of all possible n-tuples (ordered). 1.3 Cardinatility and Countability
Each ordered n-tuple is a map between the in-
dex of the elements in the n-tuple and the el- • Cardinality: No .of elements in the set.
ements in that n-tuple.
• Cantor introduced the concept of comparing cardinal-
• Power set: The power set of A contains all the subsets ity of infinite sets based on what kind of functions are
of A. possible to construct between the two sets.
Elements in a power set is 2|A|
• Two sets A and B are said to be equicardinal if there
• Relation: A relation of A into B is a subset of carte- exists a bijective function from A to B.
sian product AXB. Written as aRb like ‘a≤b’. |B| ≥ |A| if ∃ a one-to-one function from A to B.
|B| > |A| if ∃ a one-to-one function from A to B, but
• Def- Equivalence Relation: A relation R of A into A NOT a bijective function.
is an equivalence relation if:
1.3.1 Countable Sets
– Reflexive: (a,a) ∈ R, ∀a ∈ A
– Symmetric: ∀ (a, b) ∈ A, if (a,b) ∈ R then (b,a) • A set A is said to be countably infinite if it is equicar-
∈R dinal with the set of natural numbers N.
– Transitive: ∀ a,b,c ∈ A, if (a,b),(b,c) ∈ R, then • Example 1: consider the set of all integers, Z. To show
(a,c) ∈ R that it is countably infinite, we need to construct a bi-
jective function from N.
Examples, ”similar” is equivalence, ”less than” in not
Such a bijective function could be
equivalence as it does not satisfy the reflexive and
symmetric property. (
n/2 , n → even
f (n) =
• Equivalence Class: Any equivalence relation parti- −(n − 1)/2 , n → odd
tions the set into a disjoint union of subsets, which
are known as equivalence classes, such that two ele- Another trait of countably infinite set is that its ele-
ments are related iff they are in the same equivalence ments can be put in a numbered list, or can be pro-
class. duced using a for, or a nested for loop.
Example ”Mod 5” gives 5 equivalence classes.
Given any partition of A, i.e. Pi ⊆ A; i ∈ I, s.t. ∪ Pi = • Example 2: Prove that set of even natural numbers is
A, one can define an equivalence relation R: (a,b) ∈ countably infinite.
R iff a,b ∈ Pi for some i. Proof: The bijective function can be f (x) = 2x
 • Example 3: Set of rational numbers (0,1] • If the sample space is finite or countably infitite, it is
Proof: Create a infinite 2D grid. where rows indicate called a discrete sample spaces
values of a and columns value of b. (a,b) indicates
• Events: Subsets of the sample space where probability
rational a/b. Since all elements are less than 1, a¡b.
is assigned in a consistent way.
Traverse different points on the grid, and hence it is
equicardinal to natural numbers. • In any experiment, if A,B ⊆ S are events, then A ∪ B
and A ∩ B should also be events.
• All countably infinite sets are equicardinal!
• Hence, for mathematical consistency, we expect that
• A set is said to be countable id it is finite or countably countable union and intersections of events should also
infinite. be events, i.e.
∞
S
• Theorem: Union of countable sets is an event if A1 , A2 .. are events, and
i=1
Let I be a countable index set. S
I={1,2,3,...}. Let i be ∞
T
countable sets for i ∈ I. Then i is also countable. is an event if A1 , A2 .. are events
i∈I i=1
Proved in Tut1.
• If A is an event, then Ac should also be an event.
• Using this theorem, we can clearly prove that the set • The whole sample space S and empty set ∅ must be
of all rational is also countably infinite. events.
1.3.2 Uncountable Sets • For discrete sample spaces, we usually take P(S), the
• Def- A set is uncountable if its cardinality is strictly power set of S, to be the set of events.
greater than N. Example, [0,1], R. • If A ∩ B = ∅, then A and B are disjoint events.
• Lemma: The set of all infinite length binary strings • Properties of events/subsets: Commutativity, associa-
{0, 1}∞ is uncountable. tivity and distributivity
Proof: Cantor’s diagonalization argument
Assume {0, 1}∞ is countable, then it should be pos- • De Morgan’s Laws:
sible to enumerate as a numbered list. Let the ∞ ∞
Ai ) c = Aci
T S
first element be (a11 , a12 , a13 ...), and ith element be (
i=1 i=1
(ai1 , ai2 , ...). ∞ ∞
Ai )c = Aci
S T
Consider the following string, (a11 , a22 ...). Let this ex- (
i=1 i=1
ist in the list at position t. But according to the string
considered, att should be equal to att .This is clearly
a contradiction. Hence, {0, 1}∞ is NOT a countable 2.2 Probability Measure
set.
• Def- Probability measure: A probability measure P
• Lemma: [0,1], R, R/Q are uncountable. on (Ω, F ) is a function P: F→ [0, 1] , where F⊆P(Ω)
Proof 1:[0,1] → Prove that ∃ bijection from is the event space and Ω the sample space, satisfying:
{0, 1}∞ → [0, 1]. Associate any no. in [0,1] with
P∞ 1. P(∅)=0, P(Ω)=1
its binary expansion g(a)= k=1 2−k .ak . It is possi-
ble that 2 binary expansions may point to the same 2. If A1 , A2 , ... is a collection of disjoint events, i.e.
no. for Dyadic Rationals,i.e. of the form 2ab . Example Ai ∩ Aj = ∅ ∀ i 6= j, i.e. countably additive, then
{1,0,0,0,...} and {0,1,1,1,...} both point to half?????. ∞
[ ∞
X
P( Ai ) = P (Ai )
Proof 2: Define f:[0,1] → R like f (x) = tan(πx−π/2) i=1 i=1

• The triplet (Ω,F,P) is called a Probability Space

Proof 3: R = Q ∪ R/Q and since Q is countable,
the set of irrationals will be uncountable. • Lemma 1:
• Cantor’s Theorem: For any set A, the power set 1. P(Ac ) = 1 - P(A)[P ]
of A has a strictly greater cardinality than A itself. 2. If A ⊆ B. then P(A) ≤ P(B)[P ]
(Proved in the tutorial)
3. P(A ∪ B) = P(A) + P(B)- P(A ∩ B)[P ]
n n
(−1)j−1 Ai )[P ]
S P P T
4. P( Ai ) = P(
2 Probability Theory i=1 j=1 S⊆[1:n],|S|=j i∈S

• Lemma 2:
2.1 Sample space and events [P ]
1. Let A1 ⊆ A2 ⊆ A3 .... be a increasing sequence
• Sample Space (S or Ω): The set all possible outcomes of events and
of a random experiment. ∞
Example, for a coin toss, S = {H,T}. For a rolling
[
A = lim Ai = Ai
die, it is S={1,2,3,4,5,6} i→∞
i=1

• The sample space can be finite, countably infinite or then,

uncountably infinite. P (A) = lim P (Ai )
i→∞
 [P ]
2. Let B1 ⊇ B2 ⊇ B3 ... be a decreasing sequence
of events, and
∞
\
B = lim Bi = Bi
i→∞
i=1

then,
P (B) = lim P (Bi )
i→∞

• Many experiments give equiprobable outcomes, much

of the probabilistic calculations are combinatorial in
such problems.

• For a discrete sample space Ω with F=P(Ω), a prob-

ability measure is simply P
defined by a function
P : Ω → [0, 1], such that P(x)=1
• For such a space, any event is a countable union of
singleton events.
∞
S n
P
• Union bound : P ( Ei ) ≤ P (Ei )
i=1 i=1

∞
S n
P P
• P( Ei ) ≥ P (Ei ) − P (Ei Ej )
i=1 i=1 {i,j}⊆[1:n]
adding subtracting more terms leads to changing size
of LHS and RHS, so we get a series of bounds.

2.3 Conditional Probability

• P(A|B) means given B occurred, how likely is A to
occur = P(A ∩ B)/P (B), P(B)>0.

• For any given event B s.t. P(B)>0, PB (.)=P(.|B) is

a valid probability measure.[P ]
• Properties:
1. Multiplication Rule: P(A ∩ B)=P(B).(A|B) =
P(A).P(B|A)
In general,
n
T n
Q
P( Ai ) = P (A1 ). P (Ai |A1 ∩ A2 ∩ ...Ai−1 )
i=1 i=2
This applies for any countable set.
2. Law
P of total probability: Let A be any event,
Bi is countable collection of events that par-
tition Ω, P(Bi ) > 0). Then,
X
P (A) = P (Bi ) = P (A|Bi )
i∈I

i.e. weighted average of conditional probabili-

ties.[P ]
3. Bayes Rule: Similar setting as that of LoTP.

P (Bi |A) = P (A ∩ Bi )/P (A)

= P (Bi ).P (A|Bi )/ΣP (Bi ).P (A|Bi )
 Appendix: Proofs and Calculations

T

1. Proof: P(Ac ) = 1 - P(A)
S S
PB Ai = P Ai B /P(B)
i∈I i∈I
A ∪ Ac = Ω, A ∩ Ac = ∅
S P
=P {Ai ∩ B}/P(B) = P (Ai ∩ B)/P (B)
So, P(A ∪ Ac )=P(A)+P(Ac )=1 P i∈I P i∈I
P(Ac )=1-P(A) = P (Ai |B) = PB (Ai )
i∈B i∈B
2. Proof: If A ⊆ B. then P(A) ≤ P(B) Hence, all the axioms are satisfied. And it indeed is a
valid probability measure.
P(B) = P(A) + P(B/A)≥ P(A)
8. Proof: Law of Total probaility
3. Proof: P(A ∪ B) = P(A) + P(B) - P(A ∩ B)
T A ∩ Bi ’s P
Let Bi ’s partition Ω, then partition A and
P (A ∪ B) = P (A) + P (B/A) are disjoint. i.e., A= A ∩ Bi = P (A ∩ Bi ) =
i∈I i∈I
= P (A) + P (B) − P (B ∩ A) P
P (Bi ).P (A|Bi )
i∈I
n n
j−1
S P P T
4. Proof: P( Ai ) = (−1) P( Ai )
i=1 j=1 S⊆[1:n],|S|=j i∈S

For any x∈ Ω, suppose x is in t set among

A1 , A2 , ...An . Then the multiplicity of P(x) in RHS
is:
=t C1 −t C2 +t C3 ....(−1)(t−1)t Ct
=t C1 −t C2 +t C3 ....(−1)(t−1)t Ct − 1 + 1
= 1 - (1 - t C1 −t C2 +t C3 ....(−1)(t−1)t Ct )
=1-(1-1)t = 1
n
S
So, for each x ∈ Ai , P(x) has been added exactly
i=1
n
S
once on the RHS. Hence RHS= P( Ai )=LHS
i=1

5. Proof: Let A1 ⊆ A2 ⊆ A3 .... be a increasing se-

quence of events and
∞
[
A = lim Ai = Ai
i→∞
i=1

then,
P (A) = lim P (Ai )
i→∞

A = A1 ∪(A2 /A1 )∪(A3 /A2 )... is a union of the disjoint

∞
P
events. Hence, P(A)= P(A1 ) + P (Ai+1 /Ai )
i=1

∞
X
 
= lim P (A1 ) + [P (Ai+1 − P (Ai )] = lim P (An )
n→∞ n→∞
i=1

6. Proof: Let B1 ⊇ B2 ⊇ B3 ... be a decreasing se-

quence of events, and
∞
\
B = lim Bi = Bi
i→∞
i=1

then,
P (B) = lim P (Bi )
i→∞

follows by taking complement and using the previous

lemma.
7. Proof: For any given event B s.t. P(B)>0,
PB (.)=P(.|B) is a valid probability measure.
PB (∅) = 0, PB (Ω) = 1
Suppose {Ai , i ∈ I} is a countable collection of dis-
joint events.