18.
445 Problem Set 2
Exercise 6 In class we showed that, in the nearest neighbor random walk on Z,
{ Xn }n≥1 , the time T0 of first return to 0 has the following probability distribution:
n − 1 n/2 n/2
2
P[ T0 = n] = p q .
n − 1 n/2
Prove, by a direct computation, that
q
T0
E[ s ] = 1 − 1 − 4s2 pq.
We first remark that the value given in the problem statement for P[ T0 = n] has a caveat:
since we have to take an even number of steps to get back to 0, and it is the time of
2 n−1 n/2 n/2
first return, we have specifically that P[ T0 = n] = n− 1 ( n/2 ) p q for n = 2k for k =
1, 2, 3, . . ., and P[ T0 = n] = 0 otherwise. Then, we can write
∞
T0
E[ s ] = ∑ s · P[ T0 = 2k]
2k
k =1
To proceed, we make the following claim:
2k−1
Claim. 1
2k−1 ( k ) = 2(−4)k−1 (1/2
k ).
Proof. We write
(1/2)(−1/2)(−3/2) · · · ((2k − 3)/2) (−1)k−1 1 3 2k − 3
1/2
2 =2 = · ···
k k! k! 2 2 2
(−1)k−1 1
= −
(1 · 3 · · · (2k − 3)) = (1 · 3 · · · (2k − 3)).
k! · 2 k 1 k! · (−2)k−1
Then,
(−4)k−1 2k −1
k −1 1/2
2(−4) = ( 1 · 3 · · · ( 2k − 3 )) = (1 · 3 · · · (2k − 3))
k k! · (−2)k−1 k!
2k −1 2 · 4 · 6 · · · (2k − 2)
= (1 · 3 · · · (2k − 3)) ·
k! 2 · 4 · 6 · · · (2k − 2)
2k −1 2 · 4 · 6 · · · (2k − 2) 1 · 2 · 3 · 4 · · · (2k − 2)
= (1 · 3 · · · (2k − 3)) · k−1 =
k! 2 · 1 · 2 · 3 · · · ( k − 1) k!(k − 1)!
1
1 · 2 · 3 · 4 · · · (2k − 2) 2k − 1 (2k − 1)! 2k − 1
1 1
= · = · = ,
k!(k − 1)! 2k − 1 2k − 1 k!(k − 1)! 2k − 1 k
as desired. Thus, the claim is proven.
Now, using our claim, we write
2k − 1 k k
2 k−1 1/2 k k k 1/2
P[ T0 = 2k] = p q = 2 · 2(−4) p q = −(−4) pk qk .
2k − 1 k k k
Then,
∞ ∞
1/2 k k
T0
E[ s ] = ∑ s · P[ T0 = 2k ] =
2k
∑ 2k
−s · (−4) k
k
p q
k =1 k =1
∞
1/2
=− ∑ k
2 k
(−4s pq) .
k =1
By the Binomial Theorem, ∑∞ α k
k=0 ( k ) x = (1 + x ) for | x | < 1, so we know that
α
∞ q
1/2
∑ k
2 k
(−4s pq) = 1 − 4s2 pq,
k =0
and
∞ ∞
1/2 1/2
∑ k
(−4s pq) = 1 + ∑
2 k
k
2
(−4s pq) k
k =0 k =1
so
∞ q
1/2
∑ k
2 k
(−4s pq) = 1 − 4s2 pq − 1
k =1
. Thus,
∞ q
1/2
q
T0
E[ s ] = − ∑ k
2 k
(−4s pq) = − 1 − 4s pq − 1 = 1 − 1 − 4s2 pq .
2
k =1
2
Exercise 7 The Smiths receive the paper every morning and place it on a pile after
reading it. Each afternoon, with probability 1/3, someone takes all the papers in the pile
and puts them in the recycling bin. Also, if there are 5 papers in the pile, Mr. Smith (with
probability 1) takes the papers to the bin. Consider the number of papers Xn in the pile
in the evening of day n. Is it reasonable to model this by a Markov chain? If so, what are
the state space and the transition matrix?
Yes , it is reasonable to model this by a Markov chain. Specifically we use the state space
{0, 1, 2, 3, 4} , since there cannot be 5 papers at the end of day n because if there were
5 papers in the morning, Mr. Smith would have taken all of them to the bin during the
afternoon. Let Xk be the state at the end of day n, and let Xk 6= 4. Then, there are two
possibilities for Xk+1 ; specifically, there is a 31 chance that Xk+1 = 0, and there is a 23 chance
that Xk+1 = Xk + 1. The probability of Xk+1 being anything else is 0. In the unique case of
Xk = 4, we note that Xk+1 must be 0 (with probability 1) since if a fifth paper was added,
Mr. Smith would automatically throw them all out. Using this, we can then construct the
probability transition matrix as follows:
1 2
3 3 0 0 0
1 2
3 0 3 0 0
1
P=
3 0 0 23 0 .
1 2
3 0 0 0 3
1 0 0 0 0
Exercise 8 Consider a Markov chain with state space {0, 1} and transition matrix
1/3 2/3
P= .
3/4 1/4
Assuming that the chain starts in state 0 at time n = 0, what is the probability that it is
in state 1 at time n = 2?
We have that
P[ X2 = 1| X0 = 0] = P[ X1 = 0, X2 = 1| X0 = 0] + P[ X1 = 1, X2 = 1| X0 = 0]
which by the Markov property is precisely
P [ X1 = 0 | X0 = 0 ] · P [ X2 = 1 | X1 = 0 ] + P [ X1 = 1 | X0 = 0 ] · P [ X2 = 1 | X1 = 1 ]
1 2 2 1 2 2 7
= P00 · P01 + P01 · P11 = · + · = + = .
3 3 3 4 9 12 18
3
Exercise 9 (K&T 1.5 p.99) A Markov chain X0 , X1 , X2 , . . . has the transition probability
matrix (for the states {0, 1, 2}):
0.3 0.2 0.5
P = 0.5 0.1 0.4
0.5 0.2 0.3
and initial distribution: p0 = 0.5, p1 = 0.5, p2 = 0. Determine the probabilities:
(1) P[ X0 = 1, X1 = 1, X2 = 0],
(2) P[ X0 = 1, X1 = 1, X3 = 0].
Using the Markov property, we have that
P[ X0 = 1, X1 = 1, X2 = 0] = P[ X2 = 0| X1 = 1] · P[ X1 = 1| X0 = 1] · P[ X0 = 1]
= P10 · P11 · p0 = 0.5 · 0.1 · 0.5 = 0.025 .
Using the Markov property, we have that
P[ X0 = 1, X1 = 1, X3 = 0]
= P[ X0 = 1, X1 = 1, X2 = 0, X3 = 0]
+P[ X0 = 1, X1 = 1, X2 = 1, X3 = 0]
+P[ X0 = 1, X1 = 1, X2 = 2, X3 = 0]
= P [ X3 = 0 | X2 = 0 ] · P [ X2 = 0 | X1 = 1 ] · P [ X1 = 1 | X0 = 1 ] · P [ X0 = 1 ]
+ P [ X3 = 0 | X2 = 1 ] · P [ X2 = 1 | X1 = 1 ] · P [ X1 = 1 | X0 = 1 ] · P [ X0 = 1 ]
+ P [ X3 = 0 | X2 = 2 ] · P [ X2 = 2 | X1 = 1 ] · P [ X1 = 1 | X0 = 1 ] · P [ X0 = 1 ]
= P00 · P10 · P11 · p0 + P10 · P11 · P11 · p0 + P20 · P12 · P11 · p0
= p0 · P11 · ( P00 · P10 + P10 · P11 + P20 · P12 ) = 0.5 · 0.1 · (0.3 · 0.5 + 0.5 · 0.1 + 0.5 · 0.4)
= 0.5 · 0.1 · (.15 + .05 + .2) = 0.5 · 0.1 · 0.4 = 0.02 .
4
Exercise 10 (K&T 1.4 p.100) The random variables ξ 1 , ξ 2 , . . . are independent identi-
cally distributed, with common probability distribution
P[ξ = 0] = 0.1, P[ξ = 1] = 0.3, P[ξ = 2] = 0.2, P[ξ = 3] = 0.4.
Set X0 = 0 and Xn = max (ξ 1 , . . . , ξ n ) be the largest ξ observed to date. Determine the
transition probability matrix for the Markov chain { Xn }.
We first note that Xk+1 ≥ Xk for all k, since if ξ k+1 ≤ Xk , Xk+1 = Xk , and if ξ k+1 > Xk ,
then Xk+1 = ξ k+1 > Xk . Thus, Pab = 0 if a > b, so
P10 = P20 = P21 = P30 = P31 = P32 = 0.
In the cases where a > b, we have simply that
P [ X k +1 = a | X k = b ] = P [ ξ = a ] ,
so
P01 = 0.3 P02 = P12 = 0.2 P03 = P13 = P23 = 0.4.
Finally, we note that
a
P [ X k +1 = a | X k = a ] = ∑ P[ ξ = j ]
j =0
so
P00 = P[ξ = 0] = 0.1
P11 = P[ξ = 0] + P[ξ = 1] = 0.1 + 0.3 = 0.4
P22 = P[ξ = 0] + P[ξ = 1] + P[ξ = 2] = 0.1 + 0.3 + 0.2 = 0.6
P33 = P[ξ = 0] + P[ξ = 1] + P[ξ = 2] + P[ξ = 3] = 0.1 + 0.3 + 0.2 + 0.4 = 1
Thus, we can construct the transition probability matrix
0.1 0.3 0.2 0.4
0.0 0.4 0.2 0.4
P=
0.0
.
0.0 0.6 0.4
0.0 0.0 0.0 1.0
