Statistics

Moment Generating Functions

王泰裕教授
成功大學工業與資訊管理學系

1/71
 Contents
◼ Review of Continuous Distribution
Functions

2/33
 Continuous Distributions

The Uniform distribution from a to b

1
𝑓 𝑥 = ൝ 𝑎 ≤ 𝑥 ≤ 𝑏
𝑏−𝑎
0 otherwise
0.4
f ( x)
0.3

0.2


1 
0.1 
b−a 

0
0 5 10 x 15
a b
The Normal distribution
(mean m, standard deviation s)

1 2
𝑓 𝑥 = 𝑒 − 𝑥2−𝜇2
𝜎
2𝜋 𝜎
The Exponential distribution
𝑓 𝑥 = ቊ𝜆 𝑒 −𝜆𝑥 𝑥 ≥ 0
0 𝑥 < 0

0.2

0.1

0
-2 0 2 4 6 8 10
 The Gamma distribution
Let the continuous random variable X have
density function:

𝛼
𝑓 𝑥 = ቐ 𝜆 𝛼 1 𝜆𝑥 𝑥 ≥ 0
𝑥 − 𝑒−
Γ 𝛼
0 𝑥 < 0

Then X is said to have a Gamma distribution with parameters a and l.

 Moment Generating function of
a Random Variable X
◼ Moment-generating functions are just another
way of describing distributions, but they do
require getting used as they lack the intuitive
appeal of pdfs.
◼ Definition:
The moment-generating function (mgf) of the
(distribution of the) random variable X is the function
mX of a real parameter t defined by mX(t) = E[etY],for
all t ∈ R for which the expectation E[etY] is well
defined.
Moment Generating function of
a Random Variable X

෍ 𝑒 𝑡𝑥 𝑝 𝑥 𝑖𝑓 𝑋 𝑖𝑠 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒
𝑥
𝑚𝑋 𝑡 = 𝐸 𝑒 𝑡𝑋 = ∞

න 𝑒 𝑡𝑥 𝑓 𝑥 𝑑𝑥 𝑖𝑓 𝑋 𝑖𝑠 𝑐𝑜𝑛𝑡𝑖𝑛𝑢𝑜𝑢𝑠
−∞
 Moment Generating function of
a Random Variable X
◼ Moment Generating Function
◼ The moment generating function is a function often used to
characterize the distribution of a random variable.
◼ The moment generating function has great practical
relevance because:
◼ it can be used to easily derive moments; its derivatives at
zero are equal to the moments of the random variable;
◼ a probability distribution is uniquely determined by its mgf.
 Moment Generating function of
a Random Variable X
Examples
1. The Binomial distribution (parameters p, n)
𝑚𝑋 𝑡 = 𝐸 𝑒 𝑡𝑋 = σ𝑥 𝑒 𝑡𝑥 𝑝(𝑥)
𝑛 𝑡𝑥 𝑛 𝑥
σ
= 𝑥=0 𝑒 𝑝 (1 − 𝑝)𝑛−𝑥
𝑥
𝑛 𝑡𝑥 𝑥
= 𝑥=0
σ 𝑛
𝑒 𝑝 (1 − 𝑝)𝑛−𝑥
𝑥
𝑛 𝑛 𝑥 𝑛−𝑥
= σ𝑛𝑥=0 (𝑒 𝑡 𝑝)𝑥 (1 − 𝑝)𝑛−𝑥 = σ𝑛𝑥=0 𝑎 𝑏
𝑥 𝑥
𝑛
= (𝑎 + 𝑏)
where 𝑎 = 𝑒 𝑡 𝑝, 𝑏 = (1 − 𝑝)
Moment Generating function of
a Random Variable X
2. The Poisson distribution (parameter l)
𝑥
𝜆
𝑝 𝑥 = 𝑒 −𝜆 𝑥 = 0,1,2 …
𝑥!
The moment generating function of X , mX(t) is:
𝑡𝑋 𝑡𝑥 ∞ 𝑡𝑥 −𝜆 𝜆𝑥
𝑚𝑋 𝑡 = 𝐸 𝑒 = σ𝑥 𝑒 𝑝 𝑥 = σ𝑥=0 𝑒 𝑒
𝑥!
∞
(𝑒 𝑡 𝜆) 𝑥
= 𝑒 −𝜆 ෍
𝑥!
𝑥=0
𝜆𝑒 𝑡 𝑡 −1)
= 𝑒 −𝜆 𝑒 = 𝑒 𝜆(𝑒
∞
𝑢𝑥
Using 𝑒𝑢 =෍
𝑥!
𝑥=0
 Moment Generating function of
a Random Variable X
3. The Exponential distribution (parameter m= 1/l)
−𝜆𝑥
𝑓 𝑥 =ቊ 𝜆𝑒 𝑥≥0
0 𝑥<0
The moment generating function of X , mX(t) is:
∞

𝑚𝑋 𝑡 = 𝐸 𝑒 𝑡𝑋 = න 𝑒 𝑡𝑥 𝜆𝑒 −𝜆𝑥 𝑑𝑥
0
∞
𝑒 𝑡−𝜆 𝑥 ∞
𝑡−𝜆 𝑥
= න 𝜆𝑒 𝑑𝑥 = 𝜆
𝑡−𝜆 0
0 ൝
𝜆
𝑡 < 𝜆
=
𝜆 𝜆−𝑡
undefined 𝑡 ≥ 𝜆
= 𝑖𝑓 𝑡 < 𝜆
 Moment Generating function of
a Random Variable X
4. The Standard Normal distribution (m = 0, s = 1)
1 1
− 𝑥2
𝑓 𝑥 = 𝑒 2
2𝜋

The moment generating function of X , mX(t) is:

∞
1 1
−2𝑥 2
𝑚𝑋 𝑡 = 𝐸 𝑒 𝑡𝑋 = න 𝑒 𝑡𝑥 𝑒 𝑑𝑥
2𝜋
−∞
Because
1 1
−2𝑥 2
1 1
−2(𝑥−𝑡)2 2𝑡 2
𝑒 𝑡𝑥 𝑒 = 𝑒 𝑒
2𝜋
Moment Generating function of
a Random Variable X
∞
1 1
−2𝑥 2
𝑚𝑋 𝑡 = 𝐸 𝑒 𝑡𝑋 = න 𝑒 𝑡𝑥 𝑒 𝑑𝑥
2𝜋
−∞
∞
1 2 1 1
−2(𝑥−𝑡)2
1 2
= 𝑒 2𝑡 න 𝑒 𝑑𝑥 = 𝑒 2𝑡
2𝜋
−∞

=1
 Moment Generating function of
a Random Variable X
5. Uniform distribution (interval (0, 1))
1
𝑓 𝑥 =
𝑏−𝑎
The moment generating function of X , mX(t) is:
∞ 1
1 𝑡
𝑚𝑋 𝑡 = 𝐸 𝑒 𝑡𝑋 = න 𝑒 𝑡𝑥 𝑓 𝑥 𝑑𝑥 = න 𝑒 𝑡𝑥 1𝑑𝑥 = (𝑒 − 1)
𝑡
−∞ 0
 Properties of Moment Generating
Functions (W=aX+b)
◼ Suppose that the random variable X has the mgf mX(t).
Then the mgf of random variable W=aX+b , where a
and b are constants, is given by
𝑚𝑊 𝑡 = 𝑒 𝑡𝑏 𝑚𝑋(at)
 Properties of Moment Generating
Functions (X=X1+…+Xn)
◼ Let X1, X2, …Xn be independent random variables
with mgf of mX1(t), mX2(t),…, mXn(t). Then the mgf of
their sum X=X1+…+Xn is given by

𝑚𝑋 𝑡 = 𝑚𝑋1 𝑡 ∗ 𝑚𝑋2 𝑡 ∗ ⋯ ∗ 𝑚𝑋𝑛 𝑡

 Properties of Moment Generating
Functions (X=X1+…+Xn)
◼ Example:
◼ Let X ∼ B(n, p). We know that X counts the number of
successes in n independent Bernoulli trials, so we can
represent (in distribution) as X=X1+…+Xn where each Xi is
a B(p)-random variable.
◼ If the mgf, mXi (t) , of each Xi is q + pet, the mgf of mX (t)
is
𝑚𝑋 𝑡 = 𝑚𝑋1 𝑡 ∗ 𝑚𝑋2 𝑡 ∗ ⋯ ∗ 𝑚𝑋𝑛 𝑡 = 𝑞 + 𝑝𝑒 𝑡 𝑛
 Properties of Moment Generating
Functions
◼ If a random variable X possesses a mgf 𝑚𝑋(𝑡) , then
the kth moment of X, denoted by 𝜇𝑘 , exists and is
finite for any 𝑘 ∈ 𝑁. Furthermore
𝑘
𝑘
𝑑 𝑚𝑋(𝑡)
𝜇𝑘 = 𝐸 𝑋 = 𝑘
|𝑡=0
𝑑𝑡
where is the kth derivative of 𝑚𝑋(𝑡) with respect to t,
evaluated at the point t=0
 Properties of Moment Generating
Functions
m (t ) = E ( e ) , hence m ( 0) = E (e ) = E (1) = 1
X
tX
X
0 X

1. mX(0) = 1
Note: the moment generating functions of the following
distributions satisfy the property mX(0) = 1
i) Binomial Dist'n mX (t ) = ( e p +1 − p )
n
t

l ( et −1)
ii) Poisson Dist'n mX (t ) = e
 l 
iii) Exponential Dist'n mX (t ) =  
 tl
2 −t 

iv) Std Normal Dist'n mX ( t ) = e 2 a

 l 
v) Gamma Dist'n mX (t ) =  
 l − t 
 Properties of
Moment Generating Functions
m2 m3 mk
2. mX (t ) = 1 + m1t + t +
2
t +
3
+ t +
k

2! 3! k!
We use the expansion of the exponential function:
2 3 k
u u u
eu = 1 + u + + + + +
2! 3! k!
mX (t ) = E ( etX )
 t2 2 t3 3 tk k 
= E 1 + tX + X + X + + X + 
 2!
2
3!
3
k!
k

= 1 + tE ( X ) + E ( X ) + E ( X ) + + E ( X k ) +
t 2 t 3 t
2! 3! k!
t2 t3 tk
= 1 + t m1 + m2 + m3 + + mk +
2! 3! k!
 Properties of
Moment Generating Functions
Now
k
d
3. mX (k )
( 0 ) = k mX ( t ) = mk
dt t =0

m2 m3 mk
mX (t ) = 1 + m1t + t +
2
t +
3
+ tk +
2! 3! k!
m2 m3 mk
mX (t ) = m1 + 2t + 3t +2
+ kt k −1
+
2! 3! k!
m3 mk
= m1 + m2t + t2 + + t k −1 +
2! ( k − 1)!
Properties of
Moment Generating Functions

and mX ( 0 ) = m1

m4 mk
mX ( t ) = m2 + m3t + t+ + t k −2 +
2! ( k − 2 )!
and mX ( 0 ) = m2

continuing we find mX ( 0) = mk
(k )
 Properties of
Moment Generating Functions
Above property is very useful in determining the moments
of a random variable X.
Examples

(
Binomial Dist'n mX (t ) = e p +1 − p )
n
t
i)

( ) ( )
n−1
mX (t ) = n e p +1 − p
t
pet

( ) ( )
n−1
mX ( 0) = n e p +1− p
0
pe0 = np = m1 = m
 Properties of Moment Generating
Functions

(
mX ( t ) = np ( n − 1) et p + 1 − p ) ( ) ( ) et 
n −2 n −1
e p e + e p +1− p
t t t
 

( ) ( ) ( )
n−2
= npe e p + 1 − p
t t
( n − 1) et p + et p + 1 − p 
 
= npe ( e p + 1 − p )
n−2
t t
net p + 1 − p 

= np  np + 1 − p  = np  np + q  = n 2 p 2 + npq = m2
 Properties of
Moment Generating Functions
(
l et −1 )
ii) Poisson Dist'n mX (t ) = e
(
l et −1 ) ( )
l et −1 + t
mX ( t ) = e l e  = l e
 t

(
l et −1 + t )  t  2 l ( e −1) + 2 t l ( e −1) + t
mX ( t ) = l e
t t

l e + 1 = l e + le

(
2 l e −1 + 2 t ) l e + 2 + l e ( )  t 
l et −1 + t
mX ( t ) = l e
t

l e + 1
t

(
2 l e −1 + 2 t
t
) ( )
l et −1 + t
=l e l e + 3 + l e
t

(
3 l e −1 +3t
t
) ( )
2 l e −1 + 2t
t
( )
l et −1 +t
=l e + 3l e + le
 Properties of
Moment Generating Functions
To find the moments we set t = 0.
( )
l e0 −1 +0
m1 = mX ( 0) = le
 =l
(
2 l e −1 +0 ) ( )
l e0 −1 +0
m2 = mX ( 0) = l e
0

+ le = l +l
2

m3 = mX ( 0 ) = l 3e0 + 3l 2e0t + l e0 = l 3 + 3l 2 + l

 Properties of Moment Generating
Functions
 l 
iii) Exponential Dist'n mX (t ) =  
 l − t 
d (l − t )
−1
d  l 
mX ( t ) =  =l
dt  l − t  dt

= l ( −1)(l − t ) ( −1) = l (l − t )
−2 −2
Properties of Moment Generating
Functions

mX (t ) = l ( −2)(l − t ) ( −1) = 2l (l − t )

−3 −3

mX (t ) = 2l ( −3)(l − t ) ( −1) = 2(3) l (l − t )

−4 −4

mX (t ) = 2 (3) l ( −4)(l − t )
( 4)
( −1) = (4!) l (l − t )
−5 −5

mX (t ) = ( k !) l ( l − t )
(k ) − k −1
 Properties of Moment Generating
Functions
Thus
1
m1 = m = mX ( 0) = l ( l ) =
−2

2
m2 = mX ( 0) = 2l ( l ) =
−3

l2

k!
mk = mX ( 0) = ( k !) l ( l )
(k ) − k −1
=
lk
 Properties of
Moment Generating Functions
The moments for the exponential distribution can be calculated in an
alternative way. This is note by expanding mX(t) in powers of t and
equating the coefficients of tk to the coefficients in:
m2 m3 mk
mX (t ) = 1 + m1t + t +
2
t +
3
+ t + k

2! 3! k!
l 1 1
mX ( t ) = = = = 1 + u + u 2 + u3 +
l − t 1− t 1− u
l
t t2 t3
= 1+ + + +
l l 2
l 3
 Properties of Moment Generating
Functions
Equating the coefficients of tk we get:

mk 1 k!
= or mk =
k! l k
lk
 The moments for the standard
normal distribution

mX ( t ) = e
t2
2

We use the expansion of eu.

 k 2 3
u u u uk
eu =  = 1+ u + + + + +
k =0 k ! 2! 2 3!2 k!
( ) ( )
2 3
( )
k
t t t2

mX ( t ) = e = 1 + ( )
t2 2 2 2
2 t2
2 + + + + +
2! 3! k!
1 4 1 6 1 2k
= 1+ 2 t + 2 t + 3 t +
1 2
+ k t +
2 2! 2 3! 2 k!
 The moments for the standard
normal distribution

We now equate the coefficients tk in:

m2 mk m2 k
mX ( t ) = 1 + m1t + t +
2
+ t +
k
+ t 2k
+
2! k! ( 2k ) !
 Properties of Moment Generating
Functions
For even 2k:

m2 k 1
= k
( 2k ) ! 2 k !
If k is odd: mk = 0.
( 2k )!
or m2 k = k
2 k!
2! 4!
Thus m1 = 0, m2 = = 1, m3 = 0, m4 = 2 =3
2 2 ( 2!)
 The log of Moment Generating
Functions
Let lX (t) = ln mX(t) = the log of the moment
generating function

Then l X ( 0 ) = ln mX ( 0 ) = ln1 = 0

1 mX ( t )
l X ( t ) = mX ( t ) =
mX ( t ) mX ( t )

mX ( 0)
lX ( 0) = = m1 = m
mX ( 0)
 The log of Moment Generating
Functions

mX ( t ) mX ( t ) − mX ( t )

2

l X ( t ) =
mX ( t )
2

mX ( 0) mX ( 0) − mX ( 0)

2

lX ( 0) = = m2 −  m1  = s 2
2

mX ( 0)
2
The log of Moment Generating
Functions
Thus lX (t) = ln mX(t) is very useful for calculating
the mean and variance of a random variable
1. l X ( 0 ) = m

2. l X ( 0 ) = s 2
 The log of Moment Generating
Functions
1. The Binomial distribution (parameters p, n)

(
mX (t ) = e p +1− p ) = (e p + q )
n n
t t

(
lX (t ) = ln mX (t ) = n ln et p + q )
Examples
1
m = l X ( 0 ) = n
1
l X ( t ) = n t et p p = np
e p+q p+q
 The log of Moment Generating
Functions

l X ( t ) = n
t
( t
)
e p e p +q −e p e p t
( ) t

(e p + q)
2
t

p ( p + q) − p ( p)
s = l X ( 0 ) = n
2
= npq
( p + q)
2
 The log of Moment Generating
Functions
2. The Poisson distribution (parameter l)
( )
l et −1
mX (t ) = e
lX (t ) = ln mX (t ) = l et −1 ( )
l X ( t ) = l et m = l X ( 0 ) = l

l X ( t ) = l e t s 2 = l X ( 0 ) = l
 The log of Moment Generating
Functions

3. The Exponential distribution (parameter l)

 l
 t l
mX ( t ) =  l − t

undefined t  l

l X ( t ) = ln mX ( t ) = ln l − ln ( l − t ) if t  l
The log of Moment Generating
Functions
1
lX (t ) = = (l − t )
−1

l −t

1
lX (t ) = −1( l − t ) ( −1) =
−2

(l − t )
2

1 1
Thus m = lX ( 0) = and s = lX ( 0) =
2

l l2
 The log of Moment Generating
Functions

4. The Standard Normal distribution (m = 0, s = 1)

mX ( t ) = e
t2
2

lX (t ) = ln mX (t ) = t2
2

l X ( t ) = t , l X ( t ) = 1

Thus m = l X ( 0 ) = 0 and s 2 = l X ( 0 ) = 1

 Summary
Summary of Discrete Distributions
Moment
generating
Name probability function p(x) Mean Variance function MX(t)
Discrete 1 N+1 N2-1 et etN-1
Uniform p(x) = N x=1,2,...,N 2 N et-1
12
Bernoulli p x=1 p pq q + pet
p(x) = q x=0

Binomial N Np Npq (q + pet)N
p(x) =  x  pxqN-x
Geometric p(x) =pqx-1 x=1,2,... 1 q pet
p p2 1-qet
Negative  x-1  k kq  pet  k
Binomial p(x) =  k-1  pkqx-k p p2  
1-qet
x=k,k+1,...
Poisson lx -l l l el(e -1)
t
p(x) = x! e x=1,2,...
Hypergeometric  A  N-A  A A  AN-n not useful
   n N n N 1-NN-1
 x  n-x       
p(x) =
N
 
n
 Summary
Summary of Continuous Distributions
probability Moment generating
Name density function f(x) Mean Variance function MX(t)
Continuous  1 a+b (b-a)2 ebt-eat
Uniform  a xb 2
f ( x) =  b − a 12 [b-a]t
0 otherwise
Exponentia le − lx x = 0 1 1  l 
l f ( x) =  l l2 l − t  for t < l
0 x0
Gamma  la a a a
 x a −1e −lx x = 0  l 
f(x) = f(x) =  G ( a ) l l2 l − t  for t < l
0 x0
2 (1/2) − -(1/2)x    1 
/2
x e x? 0 1-2t for t < 1/2
f(x) =  (/2)  
 d.f. 0 x<0
Normal 1 2 2 m s2 etm+(1/2)t
2s2
f(x) = e-(x-m) /2s
2 s
Weibull   − -x  (

)
+1   not
f(x) = 
x e x? 0 
 (
 +2
 )-[( )] 
+1
 avail.
0 x<0
 Expectation of functions of
Random Variables
X is discrete

E  g ( X ) =  g ( x ) p ( x ) =  g ( xi ) p ( xi )
x i

X is continuous

E  g ( X ) =  g ( x ) f ( x ) dx
−
 Moments of Random Variables
The kth moment of X
mk = E ( X k
)
  xk p ( x ) if X is discrete
 x
= 
  x k f ( x ) dx if X is continuous
- 
 Moments of Random Variables
The 1th moment of X

  x1 p ( X )
X
m1 = E ( X ) =  
1
=m
  f (X )
1
X
 −
 Moments of Random Variables
The kth central moment of X

m = E (X − m) 

0 k
k
 
  ( x − m )k p ( x ) if X is discrete
 x
= 
  ( x − m ) f ( x ) dx if X is continuous
k

 - 
where m = m1 = E(X) = the first moment of X .
 Rules for expectation
Rules:
1. E c  = c where c is a constant

2. E  aX + b = aE  X  + b where a, b are constants

3. var ( X ) = m = E ( X − m ) 

0 2
2
 

( ) −  E ( X ) = m2 − m12
2
=E X 2

4. var ( aX + b ) = a 2 var ( X )