Incrtain decesion
GUETTICHE Mourad
2023/2024
1
Marchov chains
- A Markov chain describes a system whose state
changes over time.
- The changes are not completely predictable, but
rather are governed by probability distributions.
- The future of the system depends only to its
present state, and not the path by which the
system got to this latter.
2
I. Marchov chains
Definitions 1
A time-homogeneous Markov chain is a discrete-
time stochastic process (Xn,n ≥ 0) with values in a
finite or countable set S (the state space) such that:
P(Xn+1=j/Xn=i,Xn-1=in-1,Xn-2=in-2,…,x0=i0)=P(Xn+1=j/Xn=i)=Pij.
for every n ≥ 0 and j, i, in−1 , . . . , i1 , io ∈ S.
3
I. Marchov chains
The transition matrix of the chain is the matrix
P = (pij) i,j∈S defined as pij = P(Xn+1= j|Xn= i).
It satisfies the following properties:
0 ≤ p ij ≤ 1 ∀i, j ∈ S and Σj∈Sp ij =Σj∈SP(Xn+1 = j|Xn
= i) = 1 ∀i ∈ S
Note however that for a given j ∈ S, Σi∈Sp ij can be
anything.
4
I. Markov chains
The transition graph of the chain is the oriented
graph where vertices are states and an arrow
from i to j exists if and only if pij > 0, taking value
pij when it exists.
5
0.6 0.2 0.2
0.6
0.05 0.8 0.15
R
0.3 0.2 0.5
0.2 0.05 0.3 0.2
0.2
R
F
S
0,8 0.15 0.5
II. Distribution
The distribution of the Markov chain at time n ≥ 0
is given by:
πi(n) = P(Xn= i) i ∈ S
and its initial distribution is given by
πi(0) = P(X0= i) i ∈ S
For every n ≥ 0, we have
Σi∈Sπi(n)=1
7
II Distribution
The main question that will retain our attention
for the first part of the course:
When does π(n) (the distribution at time n) converge at n
→ ∞ to some limiting distribution π?
8
III. m-step transition
Definition 2. we define m-step transition
probabilities, for m ≥ 1 and i, j ∈ S,
pij(m)= P(Xn+m = j|Xn= i)= P(Xm= j|X0= i)
9
III. m-step transition
How to compute these probabilities? Using the
Chapman-Kolmogorov equations.
For m = 2, these lead.
pij2 =Σk∈S pik pkj = (P · P)ij = (P 2) ij
Indeed, we check that :
pij (2)= P(X2= j|X0=i) =Σk∈SP(X2=j, X1 = k|X0 = i)
=Σk∈SP(X2=j/X1 = k,X0 = i)*p(X1=k/X0=i)
=Σk∈Spkj*pik
10
III. m-step transition
For higher values of m and 0 ≤l ≤ m, Chapman-
Kolmogorov equations read:
pij(m) =Σk∈S pik(l)pkj(m-l) = (P (l)· P(m-l))ij = (Pm) ij
pij(m) =P[Xm=j/x0=i]=Σk∈S P[Xm=j, Xl=k/x0=i]=
Σk∈S P[Xm=j/ Xl=k,x0=i]*P[Xl=k/ x0=i]
=Σk∈S pik(l)pkj(m-l).
11
III. m-step transition
Example 1
Consider a Markov chain given in previous section.
Determine P[X3=F / X0=S] and P[X5=R / X2=F].
0.6 0.2 0.2 0.43 0.32 0.25 0.349 0.392 0.259
0.05 0.8 0.15 0.115 0.68 0.205 0.1645 0.608 0.2275
0.3 0.2 0.5 0.34 0.32 0.34 0.322 0.392 0.268
P P232=0.32 P313=0.259
12
IV Classification of states
Definitions 3.
Two states i, j ∈ S communicate (”i ←→ j”) if ∃n, m
≥ 0 such that pij(n) > 0 and p ji(m)> 0.
A chain is said to be irreducible if all states
communicate (a single class).
A state i is said to be absorbing if pii = 1.
13
IV Classification of states
Definitions 4.
Periodicity. For a state i ∈ S, define di = gcd({n ≥ 1 :
piin > 0}). If di = 1, we say that state i is aperiodic. Else if
di > 1, we say that state i is periodic with period di.
- In a given class, all states have the same period di = d.
- If there is at least on self-loop in the class (∃i ∈ S such
as pii > 0), then all states in the class are aperiodic.
14
1
P11(())=0
P11(1)=0 4 2
P11(2)>0
P11(3)=0
P11(4)>0
d1=d=2 so its periodic 3
IV Classification of states
Definition 5
A state i ∈ S is recurrent if fii = P(∃n ≥ 1 such that Xn= i |
X0=i)=1
(i.e., the probability that the chain returns to state i in
finite time is equal to 1).
- A state i ∈ S is transient if fii < 1.
So a state is recurrent if and only if it is not transient.
Note in particular that it is not necessary that fii=0 for
state i to be transient.
16
1
3 2
states 1, 2 and 3 are transient and 4 is recurrent.
IV Classification of states
- In a given class, either all states are recurrent, or all
states are transient.
- In a finite chain, a class is recurrent iff there is no arrow
leading out of it. (So a finite irreducible chain is always
recurrent.)
18
V Stationary distribution
As a reminder, the distribution of the Markov
chain (Xn , n ≥ 0) at time n is given by:
πj(n)= P(Xn = j), j ∈ S:
Let us compute for j ∈ S:
πj(n+1)= P(Xn+1 = j) =Σi ∈ SP(Xn+1 = j, Xn = i)
=Σi ∈ SP(Xn+1 = j/ Xn = i) P( Xn = i)=Σi ∈ Sπi(n)pij
19
V Stationary distribution
In vector notation (considering π(n) , π(n+1) as row vectors),
this reads:π(n+1) = π(n) P
which further implies that π(n) = π(0) Pn . This motivates
also the following definition:
Let (Xn , n ≥ 0) be a Markov chain with transition matrix P
A probability distribution π = (πi , i ∈ S) is said to be a
stationary distribution of the chain X if:
πj=Σi ∈ Sπipij ∀j ∈ S i.e. π = π P.
20
VI Ergodic Markov chain
Definition 6
A Markov chain is said to be ergodic if it is irreducible,
aperiodic and recurrent.
Theorem (Ergodic theorem). Let X be an ergodic Markov
chain. Then it admits a unique limiting and stationary
distribution π, i.e., ∀π (0) , limn→∞ π (n) = π and π = πP.
21
VII Stationary distribution
If π is stationary then πPn = πP P n−1 = πPn−1 = ... = π.
• In particular, if the initial distribution π (0) = π, then ∀n
≥ 0, π (n) = π 0) P n = πPn = π.
Remarks
we can see than π is a (left-)eigenvector of P .
22
Stationary distribution
The stationary distribution of the Markov chain
previously mentioned is
π=[¼, ½, ¼].
23
