ECE 302: Lecture 4.
1 Probability Density Functions
Prof Stanley Chan
School of Electrical and Computer Engineering
Purdue University
c Stanley Chan 2020. All Rights Reserved.
1 / 20
How to define probability for continuous events?
c Stanley Chan 2020. All Rights Reserved.
2 / 20
Outline
Today’s lecture: Understand probability density functions (PDFs).
Intuition
Definition and properties
Connecting with PMF
c Stanley Chan 2020. All Rights Reserved.
3 / 20
Intuition
How would you define P[{x ∈ A}]?
Measure the size of a set:
“size” of A
P[{x ∈ A}] = . (1)
“size” of Ω
c Stanley Chan 2020. All Rights Reserved.
4 / 20
Example
Suppose that the sample space is the interval Ω = [0, 5] and the event is
A = [2, 3]. To measure the “size” of A, we can integrate A to determine
the length. That is,
R R3
“size” of A A dx dx 1
P[{x ∈ [2, 3]}] = =R = R25 = . (2)
“size” of Ω dx 5
Ω 0 dx
More formally,
R R
A dx dx
P[{x ∈ A}] = R = A
Ω dx |Ω|
Z
1
= dx. (3)
A |Ω|
| {z }
equally important over Ω
c Stanley Chan 2020. All Rights Reserved.
5 / 20
Relax equiprobable
What happens if we want to relax the “equiprobable” assumption?
Replace the constant function 1/|Ω| with fX (x). This will give us
Z
P[{x ∈ A}] = fX (x) dx. (4)
A | {z }
replace 1/|Ω|
If you compare it with a PMF, we note that when X is discrete, then
X
P[{x ∈ A}] = pX (x).
x∈A
c Stanley Chan 2020. All Rights Reserved.
6 / 20
Probability density function
Figure: [Left] A probability mass function (PMF) tells us the relative frequency of
a state when computing the probability. In this example, the “size” of A is
pX (x2 ) + pX (x3 ). [Right] A probability density function (PDF) is the infinitesimal
version of the PMF. Thus, the “size” of A is the integration over the PDF.
c Stanley Chan 2020. All Rights Reserved.
7 / 20
Outline
Today’s lecture: Understand probability density functions (PDFs).
Intuition
Definition and properties
Connecting with PMF
c Stanley Chan 2020. All Rights Reserved.
8 / 20
Definition
Definition
A probability density function fX of a random variable X is a mapping
fX : Ω → R, with the property that
Non-negativity: fX (x) ≥ 0 for all x ∈ Ω
R
Unity: Ω fX (x)dx = 1
R
Measure of a set: P[{x ∈ A}] = A fX (x)dx
c Stanley Chan 2020. All Rights Reserved.
9 / 20
Definition
Definition
Let X be a continuous random variable. The probability density function
(PDF) of X is a function fX : Ω → R, when integrated over an interval
[a, b], yields the probability of obtaining a ≤ X ≤ b:
Z b
P[a ≤ X ≤ b] = fX (x)dx. (5)
a
c Stanley Chan 2020. All Rights Reserved.
10 / 20
Examples
Example 1. Let fX (x) = 3x 2 with Ω = [0, 1]. Let A = [0, 0.5]. Then, the
probability P[{X ∈ A}] is
Z 0.5
1
P[0 ≤ X ≤ 0.5] = 3x 2 dx = .
0 8
Example 2. Let fX (x) = 1/|Ω| with Ω = [0, 5]. Let A = [3, 5]. Then, the
probability P[{X ∈ A}] is
Z 5 Z 5
1 1 2
P[3 ≤ X ≤ 5] = dx = dx = .
3 |Ω| 3 5 5
c Stanley Chan 2020. All Rights Reserved.
11 / 20
Can fX (x) ≥ 1?
Yes.
fX (x) is not the probability of having X = x
fX (x) is the probability per unit length
Z x+δ
P[x ≤ X ≤ x + δ] = fX (x)dx ≈ fX (x) · δ. (6)
x
c Stanley Chan 2020. All Rights Reserved.
12 / 20
Examples
Example. Consider a random variable X with PDF fX (x) = 2√1 x for any
0 < x ≤ 1, and is 0 otherwise. We can show that fX (x) → ∞ as x → 0.
However, fX (x) remains a valid PDF because
∞ 1 1
√
Z Z
1
fX (x)dx = √ dx = x = 1.
−∞ 0 2 x 0
Remark. Since isolated points have zero measure in the continuous space,
the probability of an open interval (a, b) is exactly the same as the
probability of a closed interval:
P[[a, b]] = P[(a, b)] = P[(a, b]] = P[[a, b)].
c Stanley Chan 2020. All Rights Reserved.
13 / 20
Outline
Today’s lecture: Understand probability density functions (PDFs).
Intuition
Definition and properties
Connecting with PMF
c Stanley Chan 2020. All Rights Reserved.
14 / 20
How to write a PMF in terms of a PDF?
Use delta function:
X
fX (x) = pX (xk )δ(x − xk ). (7)
xk ∈Ω
c Stanley Chan 2020. All Rights Reserved.
15 / 20
Examples
Example 1. If X is a Bernoulli random variable with PMF pX (1) = p and
pX (0) = 1 − p, then the corresponding PDF can be written as
fX (x) = pδ(x − 1) + (1 − p)δ(x − 0).
Example 2. If X is a binomial random variable with PMF
pX (k) = kn p k (1 − p)n−k , then the corresponding PDF can be written as
n n
X X n k
fX (x) = pX (k)δ(x − k) = p (1 − p)n−k δ(x − k).
k
k=0 k=0
c Stanley Chan 2020. All Rights Reserved.
16 / 20
Example
Example 3. Let X be a discrete random variable with PMF
1
pX (k) = , k = 1, 2, . . . ,
2k
The continuous representation of the PMF can be written as
∞ ∞
X X 1
fX (x) = pX (k)δ(x − k) = δ(x − k).
2k
k=1 k=1
Find the probability P[1 ≤ X ≤ 2].
c Stanley Chan 2020. All Rights Reserved.
17 / 20
Example
P[1 ≤ X ≤ 2]
Z 2 Z ∞
2X
1
= fX (x)dx = δ(x − k)dx
1 1 k=1 2k
Z 2
1 1
= δ(x − 1) + δ(x − 2) + . . . dx
1 2 4
Z 2
1 2 1 2
Z Z
1
= δ(x − 1)dx + δ(x − 2)dx + δ(x − 3)dx + |{z}
...
2 1 4 1 8 1
| {z } | {z } | {z } =0
=1 =1 =0
1 1 3
= + = .
2 4 4
c Stanley Chan 2020. All Rights Reserved.
18 / 20
Summary
Intuition
Probability is a measure of the size of a set
Use length/area/volume to measure the size of a continuous set
fX (x) is the weight when calculating the size
Definition and properties
Probability per unit length
fX (x) ≥ 1 is okay
Connecting with PMF
PMF is a train of delta function c Stanley Chan 2020. All Rights Reserved.
19 / 20
Questions?
c Stanley Chan 2020. All Rights Reserved.
20 / 20
