MARKOV CHAIN – SESSION 7
Sumanta Basu
Operations Management Group
Indian Institute of Management Calcutta
CONCEPTS DISCUSSED IN SESSION 6
¢ State
¢ Random Variable
¢ Stochastic Process
¢ Markovian Property
¢ Markov Chain
¢ Transition Probability Matrix (Stationary)
¢ State Transition Diagram
¢ Discrete time period assumption
¢ Finite state assumption
¢ Homogeneous and Stationary Markov Chain
BLOOD BANK EXAMPLE
Visionaries of Charity run a blood bank. The blood bank can store up to 3 units of blood. The
blood has a shelf life of 2 weeks, that is, blood once received at the blood bank remains
usable for the next two weeks. Any blood which remains in the blood bank after two weeks is
discarded. The blood bank faces a weekly (Monday to Friday) random demand for blood that
follows Poisson distribution with mean 2 units. While fulfilling the demand the blood bank
follows First In First Out policy. That is, the blood with shortest remaining shelf life is used
first to fulfill the demand.
The remaining life of a blood unit is checked only in the weekend (Saturday or Sunday) and
a blood unit is discarded if there is no remaining life. The manager checks the inventory of
blood after discarding the blood units with no remaining shelf life and if the inventory is
either 0 or 1 unit, the blood bank organizes blood donation camps. However, in order to
simplify the problem, consider that the bank manages to get just enough blood to bring the
inventory level to 3 units and this blood reaches the bank before the start of the next week
(by Monday morning). However, if in the weekend the bank has 2 or 3 units of blood in stock
after discarding expired blood units, it does not place any order that week.
In this process the bank incurs four types of cost. It incurs inventory carrying cost which
includes the cost of storing the blood under specified conditions, as well as other handling
costs. It also incurs the ordering cost which essentially covers the cost of organizing donation
camps. The third cost that the bank incurs is the shortage cost, i.e., the cost of not being able
to meet the demand. The blood bank also incurs a spoilage cost, i.e. social cost of an unusable
blood unit and the cost of disposal.
EXPLANATION OF STATE TRANSITION
Blood unit with remaining life 2 weeks
Blood unit with remaining life 1 week
Mon Fri
Week 1 Week 2
Mon Fri
EXPLANATION OF STATE TRANSITION
Mon Fri
Week 1 Week 2
Mon Demand = 0 Fri Demand = 1
(2, 0)
(0, 3) (0, 3) (3, 0)
Mon Fri
Week 1 Week 2
Mon Demand = 2 Fri Demand = 1
(0, 2)
(0, 3) (0, 1) (1, 2)
EXPLANATION OF STATE TRANSITION
Mon Fri
Week 1 Week 2
Mon Demand = 2 Fri Demand = 1
(0, 2)
(3, 0) (1, 0) (0, 3)
Mon Fri
Week 1 Week 2
Mon Demand = 1 Fri Demand = 1
(0, 2)
(3, 0) (2, 0) (0, 3)
STATE DEFINITION
¢State is defined as number of blood units
segregated by remaining life period at the end of
the week (Friday evening)
For example, state (i, j) is defined as ‘i‘ number of
blood units with remaining life period 1 and ‘j’
number of blood units with remaining life period
2 on Friday evening
¢ List of all possible states:
(0,3) – (0,2) – (0,1) – (0,0) – (1,2) – (1,1) – (1,0) –
(2,0) – (2,1) – (3,0)
TRANSITION PROBABILITY MATRIX
(0,3) (0,2) (0,1) (0,0) (1,2) (1,1) (1,0) (2,0) (2,1) (3,0)
(0,3)
(0,2)
(0,1)
(0,0)
(1,2)
(1,1)
(1,0)
(2,0)
(2,1)
(3,0)
TRANSITION PROBABILITY MATRIX
(0,3) (0,2) (0,1) (0,0) (1,2) (1,1) (1,0) (2,0) (2,1) (3,0)
(0,3) -- -- -- D>3 -- -- D=2 D=1 -- D=0
(0,2)
(0,1)
(0,0)
(1,2)
(1,1)
(1,0)
(2,0)
(2,1)
(3,0)
TRANSITION PROBABILITY MATRIX
(0,3) (0,2) (0,1) (0,0) (1,2) (1,1) (1,0) (2,0) (2,1) (3,0)
(0,3) -- -- -- D>3 -- -- D=2 D=1 -- D=0
(0,2) -- -- -- D>2 -- -- D=1 D=0 -- --
(0,1)
(0,0)
(1,2)
(1,1)
(1,0)
(2,0)
(2,1)
(3,0)
TRANSITION PROBABILITY MATRIX
(0,3) (0,2) (0,1) (0,0) (1,2) (1,1) (1,0) (2,0) (2,1) (3,0)
(0,3) -- -- -- D>3 -- -- D=2 D=1 -- D=0
(0,2) -- -- -- D>2 -- -- D=1 D=0 -- --
(0,1) -- D=1 D=2 D>3 D=0 -- -- -- -- --
(0,0)
(1,2)
(1,1)
(1,0)
(2,0)
(2,1)
(3,0)
TRANSITION PROBABILITY MATRIX
(0,3) (0,2) (0,1) (0,0) (1,2) (1,1) (1,0) (2,0) (2,1) (3,0)
(0,3) -- -- -- D>3 -- -- D=2 D=1 -- D=0
(0,2) -- -- -- D>2 -- -- D=1 D=0 -- --
(0,1) -- D=1 D=2 D>3 D=0 -- -- -- -- --
(0,0) D=0 D=1 D=2 D>3 -- -- -- -- -- --
(1,2) -- -- -- D>2 -- -- D=1 D=0 -- --
(1,1) -- D=1 D=2 D>3 D=0 -- -- -- -- --
(1,0) D=0 D=1 D=2 D>3 -- -- -- -- -- --
(2,0) D=0 D=1 D=2 D>3 -- -- -- -- -- --
(2,1) -- D=1 D=2 D>3 D=0 -- -- -- -- --
(3,0) D=0 D=1 D=2 D>3 -- -- -- -- -- --
MULTIPLE STEP TRANSITIONS
¢ Instructor for Risk Management course decides
that there will be surprise quizzes in each week.
There can be two possible outcomes in a quiz:
“good” or “bad”. If a student did “good” in the
previous quiz, chance of doing “good” in the next
quiz is 0.4. But if the student did “bad” in
previous quiz, chance of doing “good” in the next
quiz is 0.7.
What will be the probability of the student doing good
in the third week if (s)he did “good” in the first week.
QUIZ PROBLEM: MULTI-STEP TRANSITION
1
B
QUIZ PROBLEM: MULTI-STEP TRANSITION
G 2
1
B
2
QUIZ PROBLEM: MULTI-STEP TRANSITION
G 3
G 2 B
3
1
B
G
3
2
3
QUIZ PROBLEM: MULTI-STEP TRANSITION
G 3 B
G 2 B G
3 B
1
B G
G
3
2 B
B
G
3
B
QUIZ PROBLEM: MULTI-STEP TRANSITION
G
0.4
G 3 B
0.6
0.4
G 2 B G
0.6 0.7
3 B
1 0.3
B G
G
3 0.4
2 0.7 B
0.6
B
G
0.3
0.7
3
B
0.3
QUIZ PROBLEM: MULTI-STEP TRANSITION
0.16
G
0.4
G 3 B
0.6 0.24
0.4
G 2 B G 0.42
0.6 0.7
3 B
1 0.3 0.18
B G 0.28
G
3 0.4
2 0.7 B
0.6
B 0.42
G 0.21
0.3
0.7
3
B
0.3 0.09
QUIZ PROBLEM: MULTI-STEP TRANSITION
0.16
G
0.4
G 3 B
0.6 0.24
0.4
G 2 B G 0.42
0.6 0.7
3 B
1 0.3 0.18
B G 0.28
G
3 0.4
2 0.7 B
0.6
B 0.42
t+2 G
0.3 0.21
0 1 0.7
3
t 0 ?? B
0.3 0.09
1
QUIZ PROBLEM: MULTI-STEP TRANSITION
0.16
G
0.4
G 3 B
0.6 0.24
0.4
G 2 B G 0.42
0.6 0.7
3 B
1 0.3 0.18
B G 0.28
G
3 0.4
2 0.7 B
0.6
B 0.42
t+2 G
0.3 0.21
0 1 0.7
3
t 0 0.51 ?? B
0.3 0.09
1
QUIZ PROBLEM: MULTI-STEP TRANSITION
0.16
G
0.4
G 3 B
0.6 0.24
0.4
G 2 B G 0.42
0.6 0.7
3 B
1 0.3 0.18
B G 0.28
G
3 0.4
2 0.7 B
0.6
B 0.42
t+2 G
0.3 0.21
0 1 0.7
3
t 0 0.51 0.49 B
0.3 0.09
1 ??
QUIZ PROBLEM: MULTI-STEP TRANSITION
0.16
G
0.4
G 3 B
0.6 0.24
0.4
G 2 B G 0.42
0.6 0.7
3 B
1 0.3 0.18
B G 0.28
G
3 0.4
2 0.7 B
0.6
B 0.42
t+2 G
0.3 0.21
0 1 0.7
3
t 0 0.51 0.49 B
0.3 0.09
1 0.42 ??
QUIZ PROBLEM: MULTI-STEP TRANSITION
0.16
G
0.4
G 3 B
0.6 0.24
0.4
G 2 B G 0.42
0.6 0.7
3 B
1 0.3 0.18
B G 0.28
G
3 0.4
2 0.7 B
0.6
B 0.42
t+2 G
0.3 0.21
0 1 0.7
3
t 0 0.51 0.49 B
0.3 0.09
1 0.42 0.58
EXAMPLE: QUIZ PROBLEM (MULTI-STEP)
¢ Transition probability matrix for the problem:
Week 1
0 1
Week 0
0 0.3 0.7
1 0.6 0.4
é0.3 0.7ù é0.3 0.7ù é 0.51 0.49 ù
P(2) =ê ú ê0.6 0.4ú =ê ú
ë 0.6 0.4 û ë û ë 0.42 0.58 û
¢ Formula for n-step transition probability matrix:
P(n) = P.P(n-1) = P.Pn-1 = Pn [Compute the nth power of
the one-step transition matrix P].
What will be the probability of the student doing good in the third
week if (s)he did “good” in the first week.
EXAMPLE: PENSIONER PROBLEM
¢ A pensioner receives Rs. 2000 at the beginning of
each month. The amount of money he intends to
spend is independent of the amount he has and is
equal to amount ‘1000.i’ with probability pi , i =
1,2,3,4. The pensioner does not carry more than
Rs. 3000 from one month to the next and give the
excess amount to this son (if any). Pensioner has
a capital of Rs. 3000 at the end of month 1 after
giving excess money to his son.
What is the probability that his capital is Rs. 1000 or less at the end
of the fifth month considering equal probability for each level of
expenditure intent?
What is the probability that his capital is ever Rs. 1000 or less till
the end of the fifth month?
EXAMPLE: PENSIONER PROBLEM
TPM for Question 1:
What is the probability that his capital is Rs. 1000 or less at the end
of the fifth month considering equal probability for each level of
expenditure intent?
State is defined as the amount of money (in thousands) that the pensioner
has at the end of month (after giving excess money to his son)
0 1 2 3
0
1
2
3
EXAMPLE: PENSIONER PROBLEM
TPM for Question 1:
0 1 2 3
0 P2 + P3 +P4 P1 0 0
1 P3 + P4 P2 P1 0
2 P4 P3 P2 P1
3 0 P4 P3 P1 + P2
TPM for Question 1 with equal probability values:
0 1 2 3
0 0.75 0.25 0 0
1 0.5 0.25 0.25 0
2 0.25 0.25 0.25 0.25
3 0 0.25 0.25 0.5
EXAMPLE: PENSIONER PROBLEM
What is the probability that his capital is ever Rs. 1000 or less till
the end of the fifth month?
Example of the word ‘ever’:
Month 1 Month 2 Month 3 Month 4 Month 5
3 1 2 3 2
3 2 0 1 2
3 3 2 1 1
3 3 3 2 0
EXAMPLE: PENSIONER PROBLEM
TPM for Question 2:
0+1 2 3
0+1 1 0 0
2 P3 + P4 P2 P1
3 P4 P3 P1 + P2
TPM for Question 2 (with equal probability value):
0+1 2 3
0+1 1 0 0
2 0.5 0.25 0.25
3 0.25 0.25 0.5
EXAMPLE: INVENTORY PROBLEM (MULTI-
STEP)
¢ E-zone has the following inventory problem. It
stocks a particular model camera that can be
ordered weekly. Let D1, D2, … represent the
demand for this camera (the number of units of
camera that would be sold if the stock is not
already depleted) during the first week, second
week respectively. If in any particular week the
inventory gets exhausted, store manager orders
three cameras for stock replenishment. Demand
for camera is poisson distributed with mean 1.
What will be the multi-step transition probability
matrix?
INVENTORY EXAMPLE (MULTI-STEP)
¢ The two step TPM is as follows:
0.080 0.184 0.368 0.368 0.080 0.184 0.368 0.368
0.632 0.368 0 0 0.632 0.368 0 0
0.264 0.368 0.368 0 0.264 0.368 0.368 0
0.080 0.184 0.368 0.368 0.080 0.184 0.368 0.368
0.249 0.286 0.300 0.165 • Given that there is one
0.283 0.252 0.233 0.233 camera left in stock at the
= end of a week, what is the
0.351 0.319 0.233 0.097 probability that there will
0.249 0.286 0.300 0.165 be no camera in stock 2
weeks later?
INVENTORY EXAMPLE (MULTI-STEP)
¢ The four step TPM is as follows:
0.249 0.286 0.300 0.165 0.249 0.286 0.300 0.165
0.283 0.252 0.233 0.233 0.283 0.252 0.233 0.233
0.351 0.319 0.233 0.097 0.351 0.319 0.233 0.097
0.249 0.286 0.300 0.165 0.249 0.286 0.300 0.165
• Given that there are two
0.289 0.286 0.261 0.164 cameras left in stock at
0.282 0.285 0.268 0.166 the end of a week, what is
= the probability that there
0.284 0.283 0.263 0.171
will be three cameras in
0.289 0.286 0.261 0.164 stock after 4 weeks?
INVENTORY EXAMPLE (MULTI-STEP)
¢ The four step transition matrix can be obtained
as follows:
P(8) = P(4) . P(4)
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
LIMITING PROBABILITIES
¢ Limiting probability is a probability value that
the process will be in a particular state after a
large number of transitions, and this value is
independent of the initial state.
