Continuous-Time
Markov Chains
Professor Izhak Rubin
Electrical Engineering Department
UCLA
[email protected]
2014-2015 by Izhak Rubin
Continuous-Time Markov Chains:
Definition
Definition: The stochastic process X X t , t 0 whose
states assume values in a countable state space S,
where X t denotes the state of the process at time t 0, is
said to be a continuous-time (or continuous parameter)
Markov Chain (CTMC) if it satisfies the Markov Property (MP):
Markov Property:
P X t s y X u , u t P X t s y X t , y S , t , s 0 .
Prof. Izhak Rubin
Transition Probability Function
Its Transition Probability Function (TPF) is defined as
Ps ,t x, y
P X t y X s x , x,y S , t s 0
Consider only time-homogeneous Markov chains, for which:
Ps ,t x, y P0,t s x, y Pt s x, y , x,y S , t s 0.
The Transition Probability Function (TPF) satisfies the
following properties:
1. Pt x, y 0, x, y S , t 0
2.
P x, y 1, t 0, x S
yS
3. Pt s x, y Pt x, z Ps z , y , x, y S , s,t 0.
zS
(Chapman Kolmogorov Equation: CKE)
Prof. Izhak Rubin
Chapman-Kolmogorov Equation
and Poisson Counting Process
Proof of Chapman Kolmogorov Equation:
Pt s x, y P X t s y X 0 x
P X t s y, X t z X 0 x
zS
P X t s y X t z, X 0 x P X t z X 0 x
zS
P X t s y X t z P X t z X 0 x Pt x, z Ps z , y
zS
zS
s,t 0, x,y S.
Example: Let N N t , t 0 be the Poisson Counting Process with intensity ,
over the state space of non-negative integers S = {0,1,2,...}.
Then, we readily show that:
1. N satisfies the Markov Property
2. Pt x, y e
t I y x 0 ,
y x !
yx
t 0, x,y S.
Prof. Izhak Rubin
Probabilistic Structure
We consider a CTMC X with a standard TPF for which
lim Pt i, j ij , i, j S
t 0
We define Vt to designate the residual time period occupied by the state
assumed by the process at time t.
Theorem : P Vt x X t i e qi x , x 0, i S , t 0, for some qi 0
If qi , we set e qi x 0.
A state i is said to be absorbing if qi 0.
A state i is said to be stable if 0 qi .
A state i is said to be instantaneous if qi .
A Markov chain with a finite state space contains no instantaneous states.
Prof. Izhak Rubin
Probabilistic Structure: Embedded DT Processes
Normally (when the embedded point process is honest), a realization of a
Markov chain X, also known as a stochastic jump process, can be described
in terms of two discrete-time embedded processes:
1. Embedded Point Process representing the times at which jumps occur,
A An , n 0 , where A n = time at which the n-th jump occurs
2. Embedded State Sequence representing the successive states into which
the process jumps; , Y Yn , n 0 , Yn X A = state following the n-th jump
n
Xt
Y3
Y1
Y0
A0 A1
Y4
Y2
A2
A3
A4
Prof. Izhak Rubin
t
6
Probabilistic Structure: Embedded
Processes
Y is a DT Markov Chain; its TPF is denoted as R = {R(i, j ), i, j S }:
P Yn 1 j | Y0 , Y1 ,..., Yn i P Yn 1 j | Yn i R i, j
where R i, j 0, R i, i 0,
R i, j 1.
jS
Example :
For Poisson counting process N N t , t 0 with intensity ,
P An 1 An t | Yn i e t , t 0, qi 0, each i S.
For the embedded state sequence Y , the TPF is given as
R i, j j ,i 1 , i,j S; where j ,i 1 is the Kronecker Delta function, so that
j ,i 1 =1 if j = i+1 and = 0, otherwise.
Prof. Izhak Rubin
Transient State Behavior
Let X be a regular continuous time Markov chain.
Given {qi }, {R(i. j )} and the TPF {Pt (i, j )}.
For each t 0, i, j S , Pt (i, j ) is differentiable and its derivative is continuous.
At t 0, the derivative is given by:
i j
qi ,
d
qij lim Pt i, j
t 0 dt
qi R i, j , i j
Infinitesimal Generator of the Markov Chain: Q qij , i, j S .
Also, for i,j S:
qij
, i j , qi qii 0
R i, j qi
0, i j, q 0
i
Prof. Izhak Rubin
Transition
Intensities
We define:
qij
d
1
Pt i, j lim [ Ph i, j P0 i, j ]; P0 i, j ij
t 0 dt
h 0 h
lim
so that
Ph (i, j ) qij h o(h); for i j; where lim [o(h)/h] = 0
h 0
1 Ph (i, i ) qi h o(h); where lim [o(h)/h] = 0
h 0
Noting that qi qii
and that
q
jS
ij
0.
Thus :
1
1
For i j: qij lim [ Ph i, j ] lim [ Avg.# transitions i j over (0, h)]
h 0 h
h 0 h
transition rate from state i to state j;
1
1
qi lim [1 Ph i, i ] lim [ Avg.# transitions out of state i over (0, h)]
h0 h
h0 h
transition rate out of state i.
Prof. Izhak Rubin
Transient State Behavior
Kolmogorov's Backward Equations
d
Pt i, j qik Pt k , j , i, j S
dt
kS
Kolmogorov's Forward Equations
d
Pt i, j Pt i, k qkj , i, j S ,
dt
kS
Proof of KFE:
Pt h i, j Pt i, k Ph k , j
k
1
1
Pt h i, j Pt i, j Pt i, k Ph k , j kj
h
h
k
1
d
lim Pt h i, j Pt i, j Pt i, j Pt i, k qkj
h 0 h
dt
k
Prof. Izhak Rubin
10
Steady State Behavior
Consider a CTMC X X t , t 0 with an irreducible recurrent
Markov chain Y . The steady state distribution for X is:
P j lim P X t j ,
t
jS
Using KFE,
d
lim Pt i, j 0
t dt
P k qkj 0, j S
kS
Since qii q j , we obtain the following set of balance equations:
P j q j P k qkj ,
jS
(2.1)
k j
P j 1.
(2.2)
jS
Prof. Izhak Rubin
11
CT Birth-and-Death Process
A continuous time process X X t , t 0 , S = {0,1,2,...}, is defined to be a
Birth and Death process if it is a CTMC with the following generator:
j i 1, i 0
i ,
,
j i 1, i 1
i
qij
i i , j i, i 0
0,
o.w.
where qi i i , 0 0; i 0, i 0; i 0, i 1.
The TPF for the embedded DTMC Y is thus given by:
i
, j i 1
qij i i
R i, j
qi i
, j i 1
i i
for qi 0 and i 1; R 0,1 1.
Prof. Izhak Rubin
12
Birth-and-Death Process: Steady State
The limiting (steady state) probabilities P P j , j S , for j 0, j 0;
j 0 for j 0, are calculated as follows :
For j 0 :
For j 1:
P 0 0 P 1 1.
P j j j P j 1 j 1 P j 1 j 1.
P 0 0 P 1 1 0
P j j P j 1 j 1 P j 1 j 1 P j j 0
0
P
1
P 0
j 1
i
P j P 0
i 0 i 1
j 1
Let a j
i 0
i
for j 1, and set a0 1 ; then :
i 1
P j P 0 a j , j 0 .
Prof. Izhak Rubin
13
CT Birth-and-Death Process: Steady
State Distribution
Case 1:
a
j 0
P 0 1
; then, the steady-state distribution exists and is given by
a
j 0
, P i ai
a
j 0
, i 0.
Case 2: a j ; then, steady-state distribution does not exist, and we have:
j 0
lim Pi X t j 0,
t
j 0.
Prof. Izhak Rubin
14
Finite-State Birth and Death
Process
A continuous time process X X t , t 0 , S = {0,1,2,...,N}, N < is defined
to be a finite-state Birth and Death process if it is a CTMC with the following generator:
j i 1, N>i 0
i ,
,
j i 1, N i 1
qij i
i i , j i, N i 0
0,
o.w.
where qi i i , 0 0, N 0, and i 0, i 0, otherwise.
The TPF for the embedded DTMC Y is thus given by:
i
, j i 1
qij i i
R i, j
qi i
, j i 1
i i
for qi 0 and i 1, j N ; R 0,1 1, R N , N 1 1.
Prof. Izhak Rubin
15
Finite-State Birth and Death Process:
Steady State Distribution
X = CTMC birth-and-death with a finite state space: S {0,1,..., N }
The derivation of the steady state probabilities proceeds as performed for the
infinite state space case.
N
Since
a
j 0
always exists; the steady-state distribution always exists
(assuming positive birth and rate transition intensities, except on the boundaries)
and is given by
P i P 0 ai ai
noting that
P 0 1
a
j 0
, i0
a
j 0
.
Prof. Izhak Rubin
16
