Modeling and

Analysis of
Stochastic
Systems
Second Edition
CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Series Editors
Bradley P. Carlin, University of Minnesota, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada

Analysis of Failure and Survival Data Epidemiology — Study Design and

P. J. Smith Data Analysis, Second Edition
The Analysis of Time Series — M. Woodward
An Introduction, Sixth Edition Essential Statistics, Fourth Edition
C. Chatfield D.A.G. Rees
Applied Bayesian Forecasting and Time Series Extending the Linear Model with R — Generalized
Analysis Linear, Mixed Effects and Nonparametric Regression
A. Pole, M. West and J. Harrison Models
Applied Nonparametric Statistical Methods, J.J. Faraway
Fourth Edition A First Course in Linear Model Theory
P. Sprent and N.C. Smeeton N. Ravishanker and D.K. Dey
Applied Statistics — Handbook of GENSTAT Generalized Additive Models:
Analysis An Introduction with R
E.J. Snell and H. Simpson S. Wood
Applied Statistics — Principles and Examples Interpreting Data — A First Course
D.R. Cox and E.J. Snell in Statistics
A.J.B. Anderson
Applied Stochastic Modelling, Second Edition
An Introduction to Generalized
B.J.T. Morgan Linear Models, Third Edition
Bayesian Data Analysis, Second Edition A.J. Dobson and A.G. Barnett
A. Gelman, J.B. Carlin, H.S. Stern Introduction to Multivariate Analysis
and D.B. Rubin C. Chatfield and A.J. Collins
Bayesian Methods for Data Analysis, Introduction to Optimization Methods and Their
Third Edition Applications in Statistics
B.P. Carlin and T.A. Louis B.S. Everitt
Beyond ANOVA — Basics of Applied Statistics Introduction to Probability with R
R.G. Miller, Jr. K. Baclawski
Computer-Aided Multivariate Analysis, Introduction to Randomized Controlled Clinical
Fourth Edition Trials, Second Edition
A.A. Afifi and V.A. Clark J.N.S. Matthews
A Course in Categorical Data Analysis Introduction to Statistical Inference and Its
T. Leonard Applications with R
M.W. Trosset
A Course in Large Sample Theory
T.S. Ferguson Introduction to Statistical Methods for
Clinical Trials
Data Driven Statistical Methods
T.D. Cook and D.L. DeMets
P. Sprent
Large Sample Methods in Statistics
Decision Analysis — A Bayesian Approach
P.K. Sen and J. da Motta Singer
J.Q. Smith
Linear Models with R
Elementary Applications of Probability Theory,
J.J. Faraway
Second Edition
H.C. Tuckwell Logistic Regression Models
J.M. Hilbe
Elements of Simulation
B.J.T. Morgan
Markov Chain Monte Carlo — Sampling Methodologies with Applications
Stochastic Simulation for Bayesian Inference, P.S.R.S. Rao
Second Edition Statistical Analysis of Reliability Data
D. Gamerman and H.F. Lopes M.J. Crowder, A.C. Kimber,
Mathematical Statistics T.J. Sweeting, and R.L. Smith
K. Knight Statistical Methods for Spatial Data Analysis
Modeling and Analysis of Stochastic Systems, O. Schabenberger and C.A. Gotway
Second Edition Statistical Methods for SPC and TQM
V.G. Kulkarni D. Bissell
Modelling Binary Data, Second Edition Statistical Methods in Agriculture and Experimental
D. Collett Biology, Second Edition
Modelling Survival Data in Medical Research, R. Mead, R.N. Curnow, and A.M. Hasted
Second Edition Statistical Process Control — Theory and Practice,
D. Collett Third Edition
Multivariate Analysis of Variance and Repeated G.B. Wetherill and D.W. Brown
Measures — A Practical Approach for Behavioural Statistical Theory, Fourth Edition
Scientists B.W. Lindgren
D.J. Hand and C.C. Taylor
Statistics for Accountants
Multivariate Statistics — A Practical Approach S. Letchford
B. Flury and H. Riedwyl
Statistics for Epidemiology
Pólya Urn Models N.P. Jewell
H. Mahmoud
Statistics for Technology — A Course in Applied
Practical Data Analysis for Designed Experiments Statistics, Third Edition
B.S. Yandell C. Chatfield
Practical Longitudinal Data Analysis Statistics in Engineering — A Practical Approach
D.J. Hand and M. Crowder A.V. Metcalfe
Practical Statistics for Medical Research Statistics in Research and Development,
D.G. Altman Second Edition
A Primer on Linear Models R. Caulcutt
J.F. Monahan Stochastic Processes: An Introduction,
Probability — Methods and Measurement Second Edition
A. O’Hagan P.W. Jones and P. Smith
Problem Solving — A Statistician’s Guide, Survival Analysis Using S — Analysis of
Second Edition Time-to-Event Data
C. Chatfield M. Tableman and J.S. Kim
Randomization, Bootstrap and Monte Carlo The Theory of Linear Models
Methods in Biology, Third Edition B. Jørgensen
B.F.J. Manly
Time Series Analysis
Readings in Decision Analysis H. Madsen
S. French
Texts in Statistical Science

Modeling and
Analysis of
Stochastic
Systems
Second Edition

Vidyadhar G. Kulkarni
Department of Statistics and Operations Research
University of North Carolina at Chapel Hill
U.S.A.
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2009 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Version Date: 20130920

International Standard Book Number-13: 978-1-4398-0877-1 (eBook - PDF)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti-
lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy-
ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the
publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
 To

my wife

Radhika
 Jack and Harry were lost over a vast farmland while on their balloon ride.
When they spotted a bicyclist on trail going through the farmland below, they
lowered their balloon and yelled, “Good day, sir! Could you tell us where we
are?”
The bicyclist looked up and said, “Sure! You are in a balloon!”
Jack turned to Harry and said, “This guy must be a mathematician!”
“What makes you think so?” asked Harry.
“Well, his answer is correct, but totally useless!”
The author sincerely hopes that a student mastering this book will be able to
use stochastic models to obtain correct as well as useful answers.
 Contents

Preface xix

1 Introduction 1

1.1 What in the World is a Stochastic Process? 1

1.2 How to Characterize a Stochastic Process 4
1.3 What Do We Do with a Stochastic Process? 6
1.3.1 Characterization 7
1.3.2 Transient Behavior 7
1.3.3 Limiting Distribution 7
1.3.4 First Passage Times 8
1.3.5 Costs and Rewards 8

2 Discrete-Time Markov Chains: Transient Behavior 9

2.1 Definition and Characterization 9

2.2 Examples 13
2.3 DTMCs in Other Fields 22
2.3.1 Genomics 22
2.3.2 Genetics 23
2.3.3 Genealogy 25
2.3.4 Finance 27
2.3.5 Manpower Planning 28
2.3.6 Telecommunications 30
2.4 Marginal Distributions 31
2.5 Occupancy Times 36

xi
xii CONTENTS

2.6 Computation of Matrix Powers 38

2.6.1 Method of Diagonalization 38
2.6.2 Method of Generating Functions 41
2.7 Modeling Exercises 42
2.8 Computational Exercises 49
2.9 Conceptual Exercises 53

3 Discrete-Time Markov Chains: First Passage Times 55

3.1 Definitions 55
3.2 Cumulative Distribution Function of T 56
3.3 Absorption Probabilities 60
3.4 Expectation of T 69
3.5 Generating Function and Higher Moments of T 74
3.6 Computational Exercises 76
3.7 Conceptual Exercises 81

4 Discrete-Time Markov Chains: Limiting Behavior 85

4.1 Exploring the Limiting Behavior by Examples 85

4.2 Irreducibility and Periodicity 89
4.3 Recurrence and Transience 93
4.4 Determining Recurrence and Transience: Infinite DTMCs 99
4.4.1 Foster’s Criterion 103
4.5 Limiting Behavior of Irreducible DTMCs 105
4.5.1 The Transient Case 105
4.5.2 The Discrete Renewal Theorem 106
4.5.3 The Recurrent Case 109
4.5.4 The Null Recurrent Case 110
4.5.5 The Positive Recurrent Aperiodic Case 111
4.5.6 The Positive Recurrent Periodic Case 113
4.5.7 Necessary and Sufficient Condition for Positive Recurrence 115
CONTENTS xiii

4.5.8 Examples 116

4.6 Examples: Limiting Behavior of Infinite State-Space Irreducible
DTMCs 118
4.7 Limiting Behavior of Reducible DTMCs 122
4.8 DTMCs with Costs and Rewards 126
4.8.1 Discounted Costs 126
4.8.2 Average Costs 128
4.9 Reversibility 129
4.10 Computational Exercises 132
4.11 Conceptual Exercises 139

5 Poisson Processes 145

5.1 Exponential Distributions 145

5.1.1 Memoryless Property 146
5.1.2 Hazard Rate 147
5.1.3 Probability of First Failure 148
5.1.4 Minimum of Exponentials 149
5.1.5 Strong Memoryless Property 150
5.1.6 Sum of iid Exponentials 152
5.1.7 Sum of Distinct Exponentials 153
5.1.8 Random Sums of iid Exponentials 154
5.2 Poisson Process: Definitions 155
5.3 Event Times in a Poisson Process 162
5.4 Superposition and Splitting of Poisson Processes 166
5.4.1 Superposition 166
5.4.2 Splitting 169
5.5 Non-Homogeneous Poisson Process 173
5.5.1 Event Times in an NPP 177
5.6 Compound Poisson Process 177
5.7 Computational Exercises 180
xiv CONTENTS

5.8 Conceptual Exercises 185

6 Continuous-Time Markov Chains 189

6.1 Definitions and Sample Path Properties 189

6.2 Examples 194
6.3 Transient Behavior: Marginal Distribution 203
6.4 Transient Behavior: Occupancy Times 207
6.5 Computation of P (t): Finite State-Space 209
6.5.1 Exponential of a Matrix 209
6.5.2 Laplace Transforms 212
6.5.3 Uniformization 214
6.6 Computation of P (t): Infinite State-Space 217
6.7 First-Passage Times 221
6.7.1 Cumulative Distribution of T 222
6.7.2 Absorption Probabilities 223
6.7.3 Moments and LST of T 224
6.8 Exploring the Limiting Behavior by Examples 227
6.9 Classification of States 229
6.9.1 Irreducibility 229
6.9.2 Transience and Recurrence 229
6.10 Limiting Behavior of Irreducible CTMCs 233
6.10.1 The Transient Case 233
6.10.2 The Continuous Renewal Theorem 234
6.10.3 The Null Recurrent Case 237
6.10.4 The Positive Recurrent Case 237
6.11 Limiting Behavior of Reducible CTMCs 246
6.12 CTMCs with Costs and Rewards 249
6.12.1 Discounted Costs 249
6.12.2 Average Costs 251
6.13 Phase Type Distributions 252
CONTENTS xv

6.14 Reversibility 254

6.15 Modeling Exercises 259
6.16 Computational Exercises 265
6.17 Conceptual Exercises 272

7 Queueing Models 275

7.1 Introduction 275

7.2 Properties of General Queueing Systems 279
7.2.1 Relationship between πj∗ and π̂j 279

7.2.2 Relationship between πj∗ and πj 281

7.2.3 Relationship between π̂j and pj 282

7.2.4 Little’s Law 290
7.3 Birth and Death Queues 293
7.3.1 M/M/1 Queue 293
7.3.2 M/M/1/K Queue 294
7.3.3 M/M/s Queue 295
7.3.4 M/M/∞ Queue 296
7.3.5 Queues with Finite Populations 297
7.3.6 M/M/1 Queue with Balking and Reneging 297
7.4 Open Queueing Networks 298
7.4.1 State-Dependent Service 306
7.4.2 State-Dependent Arrivals and Service 306
7.5 Closed Queueing Networks 308
7.6 Single Server Queues 312
7.6.1 M/G/1 Queue 312
7.6.2 G/M/1 Queue 317
7.7 Retrial Queue 320
7.8 Infinite Server Queue 324
xvi CONTENTS

7.9 Modeling Exercises 325

7.10 Computational Exercises 327

8 Renewal Processes 339

8.1 Introduction 339

8.2 Properties of N (t) 343
8.3 The Renewal Function 349
8.4 Renewal-Type Equation 355
8.5 Key Renewal Theorem 359
8.6 Recurrence Times 363
8.7 Delayed Renewal Processes 367
8.8 Alternating Renewal Processes 373
8.9 Semi-Markov Processes 378
8.10 Renewal Processes with Costs/Rewards 384
8.11 Regenerative Processes 390
8.11.1 RGPs with Costs/Rewards 393
8.11.2 Little’s Law 396
8.12 Computational Exercises 397
8.13 Conceptual Exercises 406

9 Markov Regenerative Processes 409

9.1 Definitions and Examples 409

9.2 Markov Renewal Process and Markov Renewal Function 413
9.3 Key Renewal Theorem for MRPs 417
9.4 Extended Key Renewal Theorem 420
9.5 Semi-Markov Processes: Further Results 424
9.6 Markov Regenerative Processes 428
9.7 Applications to Queues 433
9.7.1 The Birth and Death Queues 433
9.7.2 The M/G/1 Queue 435
CONTENTS xvii

9.7.3 The G/M/1 Queue 437

9.7.4 The M/G/1/1 Retrial Queue 438
9.8 Modeling Exercises 440
9.9 Computational Exercises 442
9.10 Conceptual Exercises 443

10 Diffusion Processes 445

10.1 Brownian Motion 445

10.2 Sample Path Properties of BM 448
10.3 Kolmogorov Equations for Standard Brownian Motion 450
10.4 First Passage Times 452
10.5 Reflected SBM 458
10.6 Reflected BM and Limiting Distributions 461
10.7 BM and Martingales 463
10.8 Cost/Reward Models 466
10.9 Stochastic Integration 470
10.10 Stochastic Differential Equations 475
10.11 Applications to Finance 481
10.12 Computational Exercises 484
10.13 Conceptual Exercises 486

Epilogue 489

Appendix A Probability of Events 491

Appendix B Univariate Random Variables 493

Appendix C Multivariate Random Variables 497

Appendix D Generating Functions 503

Appendix E Laplace-Stieltjes Transforms 505

xviii CONTENTS
Appendix F Laplace Transforms 507

Appendix G Modes of Convergence 509

Appendix H Results from Analysis 511

Appendix I Difference and Differential Equations 513

Answers to Selected Problems 515

References 533

Index 539
 Preface

Probabilistic methodology has now become a routine part of graduate education in

operations research, statistics, computer science, economics, business, public policy,
bioinformatics, engineering, etc. The following three aspects of the methodology are
most vital for the students in these disciplines:

1. Modeling a “real-life” situation with stochastic or random elements

2. Analysis of the resulting stochastic model
3. Implementation of the results of the analysis

Of course, if the results of Step 2 show that the model does not “fit” the real-life situa-
tion, then one needs to modify the model and repeat Steps 1 and 2 until a satisfactory
solution emerges. Then one proceeds to Step 3. As the title of the book suggests, we
emphasize the first two steps. The selection, the organization, and the treatment of
topics in this book are dictated by the emphasis on modeling and analysis.
Based on my teaching experience of over 25 years, I have come to the conclusion
that it is better (from the students’ points of view) to introduce Markov chains be-
fore renewal theory. This enables the students to start building interesting stochastic
models right away in diverse areas such as manufacturing, supply chains, genet-
ics, communications, biology, queueing, and inventory systems, etc. This gives them
a feel for the modeling aspect of the subject early in the course. Furthermore, the
analysis of Markov chain models uses tools from matrix algebra. The students feel
comfortable with these tools since they can use the matrix-oriented packages, such
as Matlab, to do numerical experimentation. Nothing gives them better confidence
in the subject than seeing the analysis produce actual numbers that quantify their in-
tuition. We have also developed a collection of Matlab-based programs that can be
downloaded from:

1. www.unc.edu/∼vkulkarn/Maxim/maxim.zip
2. www.unc.edu/∼vkulkarn/Maxim/maximgui.zip

The instructions for using them are included in the readme files in these two zip files.
After students have developed familiarity with Markov chains, they are ready for
renewal theory. They can now appreciate it because they now have a lot of renewal,
renewal-reward, or regenerative processes models. Also, they are more ready to use
the tools of Laplace transforms.

xix
xx PREFACE
I am aware that this sequence is contrary to the more prevalent approach that starts
with renewal theory. Although it is intellectually appealing to start with renewal the-
ory, I found that it confuses and frustrates students, and it does not give them a feel for
the modeling aspect of the subject early on. In this new edition, I have also changed
the sequence of topics within Markov chains; I now cover the first passage times be-
fore the limiting behavior. This seems more natural since the concepts of transience
and recurrence depend upon the first passage times.
The emphasis on the analysis of the stochastic models requires careful develop-
ment of the major useful classes of stochastic processes: discrete and continuous time
Markov chains, renewal processes, regenerative processes, and Markov regenerative
processes. In the new edition, I have included a chapter on diffusion processes. In or-
der to keep the length of the book under control, some topics from the earlier edition
have been deleted: discussion of numerical methods, stochastic ordering, and some
details from the Markov renewal theory. We follow a common plan of study for each
class: characterization, transient analysis, first passage times, limiting behavior, and
cost/reward models. The main aim of the theory is to enable the students to “solve”
or “analyze” the stochastic models, to give them general tools to do this, rather than
show special tricks that work in specific problems.
The third aspect, the implementation, involves actually using the results of Steps
1 and 2 to manage the “real-life” situation that we are interested in managing. This
requires the knowledge of statistics (for estimating the parameters of the model) and
organizational science (how to persuade the members of an organization to follow
the new solution, and how to set up an organizational structure to facilitate it), and
hence is beyond the scope of this book, although, admittedly, it is a very important
part of the process.
The book is designed for a two-course sequence in stochastic models. The first
six chapters can form the first course, and the last four chapters, the second course.
The book assumes that the students have had a course in probability theory (measure
theoretic probability is not needed), advanced calculus (familiarity with differential
and difference equations, transforms, etc.), and matrix algebra, and a general level
of mathematical maturity. The appendix contains a brief review of relevant topics. In
the second edition, I have removed the appendix devoted to stochastic ordering, since
the corresponding material is deleted from the chapters on discrete and continuous
time Markov chains. I have added two appendices: one collects relevant results from
analysis, and the other from differential and difference equations. I find that these
results are used often in the text, and hence it is useful to have them readily accessible.
The book uses a large number of examples to illustrate the concepts as well as
computational tools and typical applications. Each chapter also has a large number
of exercises collected at the end. The best way to learn the material of this course
is by doing the exercises. Where applicable, the exercises have been separated into
three classes: modeling, computational, and conceptual. Modeling exercises do not
involve analysis, but may involve computations to derive the parameters of the prob-
lem. A computational exercise may ask for a numerical or algebraic answer. Some
PREFACE xxi
computational exercises may involve model building as well as analysis. A concep-
tual exercise generally involves proving some theorem, or fine tuning the understand-
ing of some concepts introduced in the chapter, or it may introduce new concepts.
Computational exercises are not necessarily easy, and conceptual exercises are not
necessarily hard. I have deleted many exercises from the earlier edition, especially
those that I found I never assigned in my classes. Many new exercises have been
added. I found it useful to assign a model building exercise and then the correspond-
ing analysis exercise. The students should be encouraged to use computers to obtain
the solutions numerically.
It is my belief that a student, after mastering the material in this book, will be well
equipped to build and analyze useful stochastic models of situations that he or she
will face in his or her area of interest. It is my fond hope that the students will see
a stochastic model lurking in every corner of their world as a result of studying this
book.

Vidyadhar Kulkarni
Department of Statistics and Operations Research
University of North Carolina
Chapel Hill, NC
 CHAPTER 1

Introduction

The discipline of operations research was borne out of the need to solve military
problems during World War II. In one story, the air force was using the bullet holes
on the airplanes used in combat duty to decide where to put extra armor plating. They
thought they were approaching the problem in a scientific way until someone pointed
out that they were collecting the bullet hole data from the planes that returned safely
from their sorties.

1.1 What in the World is a Stochastic Process?

Consider a system that evolves randomly in time, for example, the stock market
index, the inventory in a warehouse, the queue of customers at a service station,
water-level in a reservoir, the state of the machines in a factory, etc.
Suppose we observe this system at discrete time points n = 0, 1, 2, · · ·, say, every
hour, every day, every week, etc. Let Xn be the state of the system at time n. For
example, Xn can be the Dow-Jones index at the end of the n-th working day; the
number of unsold cars on a dealer’s lot at the beginning of day n; the intensity of the
n-th earthquake (measured on the Richter scale) to hit the Continental United states
in this century; or the number of robberies in a city on day n, to name a few. We say
that {Xn , n ≥ 0} is a discrete-time stochastic process describing the system.
If the system is observed continuously in time, with X(t) being its state at time
t, then it is described by a continuous-time stochastic process {X(t), t ≥ 0}. For
example, X(t) may represent the number of failed machines in a machine-shop at
time t, the position of a hurricane at time t, or the amount of money in a bank account
at time t, etc.
More formally, a stochastic process is a collection of random variables
{X(τ ), τ ∈ T }, indexed by the parameter τ taking values in the parameter set T .
The random variable takes values in the set S, called the state-space of the stochastic
process. In many applications the parameter t represents time. Throughout this book
we shall encounter two cases:

1
2 INTRODUCTION
1. T = {0, 1, 2, · · ·}. In this case we write {Xn , n ≥ 0} instead of {X(τ ), τ ∈ T }.
2. T = [0, ∞). In this case we write {X(t), t ≥ 0} instead of {X(τ ), τ ∈ T }.

Also, we shall almost always encounter S ⊆ {0, 1, 2, · · ·} or S ⊆ (−∞, ∞). We

shall refer to the former case as the discrete state-space case, and the latter case as
the continuous state-space case.
Let {X(τ ), τ ∈ T } be a stochastic process with state-space S, and let x : T → S
be a function. One can think of {x(τ ), τ ∈ T } as a possible evolution (trajectory) of
{X(τ ), τ ∈ T }. The functions x are called the sample paths of the stochastic process.
Figure 1.1 shows typical sample paths of stochastic processes. Since the stochastic
process follows one of the sample paths in a random fashion, it is sometimes called
a random function. In general, the set of all possible sample paths, called the sample
space of the stochastic process, is uncountable. This can be true even in the case of a
discrete time stochastic process with finite state-space. One of the aims of the study
of the stochastic processes is to understand the behavior of the random sample paths
that the system follows, with the ultimate aim of prediction and control of the future
of the system.
X(t)

t
(a) Continuous-time, discrete state space

X(t)

t
(b) Continuous-time, continuous state space

Xn

× × × ×

× × ×

× × ×

× × ×

n
(c) Discrete-time, discrete state space

Figure 1.1 Typical sample paths of stochastic processes.

Stochastic processes are used in epidemiology, biology, demography, polymer

science, physics, telecommunication networks, economics, finance, marketing, and
WHAT IN THE WORLD IS A STOCHASTIC PROCESS? 3
social networks, to name a few areas. A vast literature exists in each of these ar-
eas. Our applications will generally come from queueing theory, inventory systems,
supply chains, manufacturing, computer and communication networks, reliability,
warranty management, mathematical finance, and statistics. We illustrate a few such
applications in the example below. Although most of the applications seem to in-
volve continuous time stochastic process, they can easily be converted into discrete
time stochastic processes by simply assuming that the system in question is observed
at a discrete set of points, such as each hour, or each day, etc.

Example 1.1 Examples of Stochastic Processes in Real Life

Queues. Let X(t) be the number of customers waiting for service in a service
facility. {X(t), t ≥ 0} is a continuous-time stochastic process with state-space
S = {0, 1, 2, · · ·}.
Inventories. X(t) = the number of automobiles in the parking lot of a dealership
available for sale at time t. Y (t) = the number of automobiles on order (the cus-
tomers have paid a deposit for them and are now waiting for the delivery) at the
dealership at time t. Both {X(t), t ≥ 0} and {Y (t), t ≥ 0} are continuous-time
stochastic processes with state-space S = {0, 1, 2, · · ·}.
Supply Chains. Consider a supply chain of computer printers with three levels: the
manufacturer (level 1), the regional warehouse (level 2), and the retail store (level
3). The printers are stored at all three levels. Let Xi (t) = the number of printers at
level i(= 1, 2, 3). Then {X(t) = (X1 (t), X2 (t), X3 (t)), t ≥ 0} is a vector-valued
continuous time stochastic process with state space S = {(x1 , x2 , x3 ) : x1 , x2 , x3 =
0, 1, 2, · · ·}.
Manufacturing. Consider a machine shop with N machines. Each machine can be
either working or under repair. Let Xi (t) = 1, if the i-th machine is working at time
t, and 0 otherwise. Then {X(t) = (X1 (t), X2 (t), · · · , XN (t)), t ≥ 0} is a continu-
ous time stochastic process with state-space S = {0, 1}N .
Communication Networks. A telecommunication network is essentially a network
of buffers (say N ) connected by communication links. Let Xi (t) be the number of
packets in the i-th buffer (i = 1, 2, · · · , N ). Then the state of the system at time t is
given by X(t) = (X1 (t), X2 (t), · · · , XN (t)).
Reliability. A complex system consists of many (say N ) parts, each of which can
be up or down. Let Xi (t) = 1, if the i-th component is working at time t, and
0 otherwise. The state of the system at time t is given by the vector X(t) =
(X1 (t), X2 (t), · · · , XN (t)). {X(t), t ≥ 0} is a continuous time stochastic process
with state-space S = {0, 1}N . Finally, the functionality of the system is described by
a structure function Φ : S → {0, 1}, with the following interpretation: The system is
up at time t if Φ(X(t)) = 1, and down otherwise.
Warranty Management. Consider a company that makes and sells hard drives for
personal computers. Each hard drive is sold under a one year warranty, which stipu-
lates that the company will replace free of charge any hard drive that fails while under
warranty by a new one, which is under warranty for the remaining time period. Such a
warranty is called a free-replacement non-renewable warranty. In order to keep track
of warranty liabilities the company typically keeps track of X(t), the number hard
4 INTRODUCTION
drives currently under warranty. In addition, it keeps track of the remaining warranty
period for each of those hard drives. This produces a rather complicated stochastic
process.
Mathematical Finance. Let X(t) be the stock price of IBM stock at time t. Then
{X(t), t ≥ 0} is a continuous-time stochastic process with state space [0, ∞). A Eu-
ropean call option with strike price K, and expiry date T , gives the owner the option
(but not the obligation) of buying the IBM stock at price K at time T , regardless of
the actual price of the stock at that time. Clearly, the holder of the option stands to
make max(0, X(t) − K) amount of money at time T . Since there is no downside
risk (no loss) in this option, how much money should the owner of this option be
willing to pay to buy this option at time 0? Clearly that will depend on how the stock
process {X(t), t ≥ 0} evolves, what the value of the stock is at time 0, and what
other investment opportunities are available.
Quality Control. A sequential quality control plan is used to continuously monitor
the quality of a product. One such plan is described by two non-negative integers k
and r, and operates in two phases as follows: in phase one we inspect each item and
mark it as defective or non-defective. If we encounter k non-defective items in a row,
we switch to phase two, in which we inspect every r-th item. We continue in phase
two until we hit a defective item, at which time we switch back to phase 1. Let Xn
be the phase the plan is in after inspecting the n-th item. Then {Xn , n ≥ 0} is a
discrete time stochastic process on state-space {1, 2}.
Insurance. Let X(t) be the number of active policies at time t that are covered by a
life insurance company. Each active policy pays premiums at a fixed rate r per unit
time. When a policy expires, the company pays out a random pay-off. Let R(t) be
the total funds at the company at time t. Then {X(t), t ≥ 0} is a continuous-time
stochastic process with state-space S = {0, 1, 2, · · ·}. It goes up by one every time
a new policy is sold, and goes down by a one every time an active policy expires.
{R(t), t ≥ 0} is a continuous-time stochastic process with state-space S = [0, ∞).
It increases at rate rX(t) at time t and jumps down by a random amount whenever
a policy expires. If this process goes negative, we say that the company has gone
bankrupt!
DNA Analysis. DNA (deoxyribose nucleic acid) is a long molecule that consists of
a string of four basic molecules called bases, represented by A, T, G, C. Let Xn be
the n-th base in the DNA. Then {Xn , n ≥ 0} is a discrete time stochastic process
with state-space S = {A, T, G, C}.

1.2 How to Characterize a Stochastic Process

In this book we develop the tools for the analysis of the stochastic processes and study
several special classes of stochastic processes in great detail. To do this we need
a mathematically precise method to describe a stochastic process unambiguously.
Since a stochastic process is a collection of random variables, it makes sense to start
by reviewing how one “describes” a single random variable.
From elementary probability theory (see Appendices A and B) we see that a single
random variable X is completely described by its cumulative distribution function
HOW TO CHARACTERIZE A STOCHASTIC PROCESS 5
(cdf)
F (x) = P(X ≤ x), −∞ < x < ∞. (1.1)
A multivariate random variable (X1 , X2 , · · · , Xn ) is completely described by its
joint cdf
F (x1 , x2 , · · · , xn ) = P(X1 ≤ x1 , X2 ≤ x2 , · · · , Xn ≤ xn ), (1.2)
for all −∞ < xi < ∞ and i = 1, 2, · · · , n. Thus if the parameter set T is finite, the
stochastic process {X(τ ), τ ∈ T } is a multi-variate random variable, and hence is
completely described by the joint cdf. But what about the case when T is not finite?
Consider the case T = {0, 1, 2, · · ·} first. One could naively look for a direct ex-
tension of the finite dimensional joint cdf to an infinite dimensional case as follows:
F (x0 , x1 , · · ·) = P(X0 ≤ x0 , X1 ≤ x1 , · · ·), −∞ < xi < ∞, i = 0, 1, · · · .
(1.3)
However, the probability on the right hand side of the above equation is likely to be
zero or one in most of the cases. Thus such a function is not likely to give much
information regarding the stochastic process {Xn , n ≥ 0} . Hence we need to look
for an alternative. Suppose we are given a family of finite dimensional joint cdfs
{Fn , n ≥ 0} such that
Fn (x0 , x1 , · · · , xn ) = P(X0 ≤ x0 , X1 ≤ x1 , · · · , Xn ≤ xn ), (1.4)
for all −∞ < xi < ∞, and i = 0, 1, · · · , n. Such a family is called consistent if it
satisfies
Fn (x0 , x1 , · · · , xn ) = Fn+1 (x0 , x1 , · · · , xn , ∞), (1.5)
for all −∞ < xi < ∞, and i = 0, 1, · · · , n, n ≥ 0. A discrete-time stochastic
process is completely described by a consistent family of finite dimensional joint
cdfs, that is, any probabilistic question about {Xn , n ≥ 0} can be answered in terms
of {Fn , n ≥ 0}. Technically speaking, what one means by “completely describe” a
stochastic process is to construct a probability space (Ω, F , P) on which the process
is defined. The more curious reader is referred to more advanced texts on stochastic
processes to answer further questions.
Next we turn to the case of T = [0, ∞). Unfortunately the matter of completely
describing a continuous time stochastic process {X(t), t ≥ 0} is not so simple, since
this case deals with an uncountable number of random variables. The situation can
be simplified if we can make certain assumptions about the continuity of the sample
paths of the process. We shall not deal with the details here, but shall give the main
result:
Suppose the sample paths of {X(t), t ≥ 0} are, with probability 1, right continu-
ous with left limits, i.e.,
lim X(s) = X(t), (1.6)
s↓t

and lims↑t X(s) exists for each t. Furthermore, suppose the sample paths have a
finite number of discontinuities in a finite interval of time. Then {X(t), t ≥ 0} is
6 INTRODUCTION
completely described by a consistent family of finite dimensional joint cdfs
Ft1 ,t2 ,···,tn (x1 , x2 , · · · , xn ) = P (X(t1 ) ≤ x1 , X(t2 ) ≤ x2 , · · · , X(tn ) ≤ xn ),
(1.7)
for all −∞ < xi < ∞, i = 1, · · · , n, n ≥ 1 and all 0 ≤ t1 < t2 < · · · < tn .

Example 1.2 Independent and Identically Distributed Random Variables.

Let {Xn , n ≥ 1} be a sequence of independent and identically distributed (iid) ran-
dom variables with common distribution F (·). This stochastic process is completely
described by F (·) as we can create the following consistent family of joint cdfs:
n
Y
Fn (x1 , x2 , · · · , xn ) = F (xi ), (1.8)
i=1

for all −∞ < xi < ∞, i = 1, 2, · · · , n, and n ≥ 1.

In a sense a sequence of iid random variables is the simplest kind of stochastic pro-
cess, but it does not have any interesting structure. However, one can construct more
complex and interesting stochastic processes from it as shown in the next example.

Example 1.3 Random Walk.

Let {Xn , n ≥ 1} be as in Example 1.2. Define
S0 = 0, Sn = X1 + · · · + Xn , n ≥ 1. (1.9)
The stochastic process {Sn , n ≥ 0} is called a random walk. It is also completely
characterized by F (·), since the joint distribution of (S0 , S1 , · · · , Sn ) is completely
determined by that of (X1 , X2 , · · · , Xn ), which, from Example 1.2, is determined
by F (·).

1.3 What Do We Do with a Stochastic Process?

Now that we know what a stochastic process is and how to describe them precisely,
the next question is: what do we with it?
In Section 1.1 we have seen several situations where stochastic processes appear.
For each situation we have a specific goal for studying it. The study of stochastic
processes will be useful if it somehow helps us achieve that goal. In this book we shall
develop a set of tools to help us achieve those goals. This will lead us to the study
of special classes of stochastic processes: Discrete-time Markov chains (Chapters
2, 3, 4), Poisson processes (Chapter 5), continuous-time Markov chains (Chapter
6), renewal and regenerative processes (Chapter 8), Markov-regenerative processes
(Chapter 9) and Brownian motion (Chapter 10).
For each of these classes, we follow a more or less standard format of study, as
described below.
WHAT DO WE DO WITH A STOCHASTIC PROCESS? 7
1.3.1 Characterization

The first step is to define the class of stochastic processes under consideration. Then
we look for ways of uniquely characterizing the stochastic process. In many cases
we find that the consistent family of finite dimensional joint probability distributions
can be described by a rather compact set of parameters. Identifying these parameters
is part of this step.

1.3.2 Transient Behavior

The second step in the study of the stochastic process is to study its transient behav-
ior. We concentrate on two aspects of transient behavior. First, we study the marginal
distribution, that is, the distribution of Xn or X(t) for a fixed n or t. We develop
methods of computing this distribution. In many cases, the distribution is too hard
to compute, in which case we satisfy ourselves with the moments or transforms of
the random variable. (See Appendices B through F.) Mostly we shall find that the
computation of transient behavior is quite difficult. It may involve computing matrix
powers, or solving sets of simultaneous differential equations, or inverting trans-
forms. Very few processes have closed form expressions for transient distributions,
for example, the Poisson process. Second, we study the occupancy times, that is, the
expected total time the process spends in different states up to time n or t.

1.3.3 Limiting Distribution

Since computing the transient distribution is intractable in most cases, we next turn
our attention to the limiting behavior, viz., studying the convergence of Xn or X(t)
as n or t tends to infinity. Now, there are many different modes of convergence of a
sequence of random variables: convergence in distribution, convergence in moments,
convergence in probability, and convergence with probability one (or almost sure
convergence). These are described in Appendix G. Different stochastic processes ex-
hibit different types of convergence. For example, we generally study convergence
in distribution in Markov chains. For renewal processes, we obtain almost sure con-
vergence results.
The first question is if the convergence occurs at all, is the limit unique when it
does occur. This is generally a theoretical exploration, and proceeds in a theorem-
proof fashion. The second part is how to compute the limit if the convergence does
occur and the limit is unique. Here we may need mathematical tools like matrix alge-
bra, systems of difference and differential equations, Laplace and Laplace-Stieltjes
transforms, generating functions, and, of course, numerical methods. The study of
the limiting distributions forms a major part of the study of the stochastic processes.
8 INTRODUCTION
1.3.4 First Passage Times

Let {Xn , n ≥ 0} be a stochastic process with state-space S. Let B ⊂ S be a given

subset of states. The first passage time to B is a random variable defined as
T = min{n ≥ 0 : Xn ∈ B}. (1.10)
Thus the stochastic “passes into” B for the first time at time T . We can define similar
first passage time in a continuous time stochastic process as follows:
T = min{t ≥ 0 : X(t) ∈ B}. (1.11)
We study the random variable T : the probability that it is finite, its distribution, mo-
ments, transforms, etc. The first passage times occur naturally when we use stochastic
processes to model a real life system. For example it may represent time until failure,
or time until bankruptcy, or time until a production process goes out of control and
requires external intervention, etc.

1.3.5 Costs and Rewards

Stochastic processes in this book are intended to be used to model systems evolv-
ing in time. These systems typically incur costs or earn rewards depending on their
evolution. In practice, the system designer has a specific set of operating policies in
mind. Under each policy the system evolution is modeled by a specific stochastic
process which generates specific costs or rewards. Thus the analysis of these costs
and rewards (we shall describe the details in later chapters) is critical in evaluating
comparative worth of the policies. Thus we develop methods of computing different
cost and reward criteria for the stochastic processes.
If the reader keeps in mind these five main aspects that we study for each class
of stochastic processes, the organization of the rest of the book will be relatively
transparent. The main aim of the book is always to describe general methods of
studying the above aspects, and not to go into special methods, or “tricks,” that work
(elegantly) for specialized problems, but fail in general. For this reason, some of
our analysis may seem a bit long winded for those who are already familiar with
the tricks. However, it is the general philosophy of this book that the knowledge of
general methods is superior to that of the tricks. Finally, the general methods can be
adopted for implementation on computers, while the tricks cannot. This is important,
since computers are a great tool for solving practical problems.
 CHAPTER 2

Discrete-Time Markov Chains:

Transient Behavior

“Come, let us hasten to a higher plane

Where dyads tread the fairy fields of Venn,
Their indices bedecked from one to n
commingled in an endless Markov Chain!”
– Stanislaw Lem, ‘Cyberiad’

2.1 Definition and Characterization

Consider a system that is modeled by a discrete-time stochastic process {Xn , n ≥

0} with a countable state-space S, say {0, 1, 2, · · ·}. Consider a fixed value of n
that we shall call “the present time” or just the “present.” Then Xn is called the
present (state) of the system, {X0 , X1 , · · · , Xn−1 } is called the past of the system,
and {Xn+1 , Xn+2 , · · ·} is called the future of the system. If Xn = i and Xn+1 = j,
we say that the system has jumped (or made a transition) from state i to state j from
time n to n + 1.
In this chapter we shall restrict attention to the systems having the following prop-
erty: if the present state of the system is known, the future of the system is independent
of the past. This is called the Markov property and this seemingly innocuous prop-
erty has surprisingly far-reaching consequences. For a system having the Markov
property, the past affects the future only through the present; or, stated in yet another
way, the present state of the system contains all the relevant information needed
to predict the future of the system in a probabilistic sense. The stochastic process
{Xn , n ≥ 0} with state-space S used to describe a system with Markov property is
called a discrete-time Markov chain, or, DTMC for short. We give the formal defini-
tion below.

9
10 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
Definition 2.1 Discrete Time Markov Chain. A stochastic process {Xn , n ≥
0} with countable state-space S is called a DTMC if
(i) for all n ≥ 0, Xn ∈ S,
(ii) for all n ≥ 0, and i, j ∈ S,
P(Xn+1 = j|Xn = i, Xn−1 , Xn−2 , · · · , X0 ) = P(Xn+1 = j|Xn = i). (2.1)

Equation 2.1 is a formal way of stating the Markov property for discrete-time
stochastic processes with countable state spaces. We next define an important sub-
class of DTMCs called the time-homogeneous DTMCs.

Definition 2.2 Time Homogeneous DTMC. A DTMC {Xn , n ≥ 0} with count-

able state-space S is said to be time-homogeneous if
P(Xn+1 = j|Xn = i) = pi,j for all n ≥ 0, i, j ∈ S. (2.2)

Thus if a time homogeneous DTMC is in state i at time n, it jumps to state j at time

n + 1 with probability pi,j , for all values of n. From now on we assume that the
DTMC is time-homogeneous, unless otherwise specified. Let
P = [pi,j ]
denote the matrix of conditional probabilities pi,j . The matrix P is called the one-
step transition probability matrix or just the transition probability matrix. When S is
finite, say S = {1, 2, · · · , m}, one can represent P as a matrix as follows:
 
p11 p12 ··· p1,m−1 p1m
 p21 p22 ··· p2,m−1 p2m 
. . . ..
 
P =
 .. .. . .. .. .

(2.3)
 . 
 pm−1,1 pm−1,2 · · · pm−1,,m−1 pm−1,m 
pm1 pm2 ··· pm,m−1 pmm
Next we define an important property of a matrix.

Definition 2.3 Stochastic Matrix. A matrix P = [pij ] is called stochastic if

(i) p ij ≥ 0 for all i, j ∈ S,
P (2.4)
(ii) j∈S pij = 1 for all i ∈ S.

In short, the elements on each row of a stochastic matrix are non-negative and add
up to one. The relevance of this definition is seen from the next theorem.

Theorem 2.1 Transition Probability Matrix. The one-step transition probability

matrix of a DTMC is stochastic.
DEFINITION AND CHARACTERIZATION 11
Proof. Statement (i) is obvious since pij is a conditional probability. Statement (ii)
can be seen as follows:
X X
pij = P(Xn+1 = j|Xn = i),
j∈S j∈S

= P(Xn+1 ∈ S|Xn = i) = 1,
since, according to the definition of DTMC, Xn+1 ∈ S with probability 1.
Next, following the general road map laid out in Chapter 1, we turn our attention
to the question of characterization of a DTMC. Clearly, any stochastic matrix can
be thought of as a transition matrix of a DTMC. This generates a natural question:
is a DTMC completely characterized by its transition probability matrix? In other
words, are the finite dimensional distributions of a DTMC completely specified by
its transition probability matrix? The answer is no, since we cannot derive the distri-
bution of X0 from the transition probability matrix, since its elements are conditional
probabilities. So suppose we specify the distribution of X0 externally. Let
ai = P(X0 = i) i ∈ S, (2.5)
and
a = [ai ]i∈S (2.6)
be a row vector representing the probability mass function (pmf) of X0 . We say that
a is the initial distribution of the DTMC.
Next we ask: is a DTMC completely described by its transition probability matrix
and its initial distribution? The following theorem answers this question in the affir-
mative. The reader is urged to read the proof, since it clarifies the role played by the
Markov property and the time-homogeneity of the DTMC.

Theorem 2.2 Characterization of a DTMC.

A DTMC {Xn , n ≥ 0} is completely described by its initial distribution a and the
transition probability matrix P .

Proof. We shall prove the theorem by showing how we can compute the finite di-
mensional joint probability mass function P(X0 = i0 , X1 = i1 , · · · , Xn = in ) in
terms of a and P . Using Equation 2.5 we get
ai0 = P(X0 = i0 ), i0 ∈ S.
Next we have
P(X0 = i0 , X1 = i1 ) = P(X1 = i1 |X0 = i0 )P(X0 = i0 )
= ai0 pi0 ,i1
by using the definition of P . Now, as an induction hypothesis, assume that
P(X0 = i0 , X1 = i1 , · · · , Xk = ik ) = ai0 pi0 ,i1 pi1 ,i2 · · · pik−1 ,ik (2.7)
12 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
for k = 1, 2, · · · , n − 1. We shall show that it is true for k = n. We have
P(X0 = i0 , X1 = i1 , · · · , Xn = in )
= P(Xn = in |Xn−1 = in−1 , · · · , X1 = i1 , X0 = i0 ) ·
P(X0 = i0 , X1 = i1 , · · · , Xn−1 = in−1 )
= P(Xn = in |Xn−1 = in−1 )P(X0 = i0 , X1 = i1 , · · · , Xn−1 = in−1 )
(by Markov Property)
= pin−1 ,in P(X0 = i0 , X1 = i1 , · · · , Xn−1 = in−1 )
(by time homogeneity)
= pin−1 ,in ai0 pi0 ,i1 pi1 ,i2 · · · pin−2 ,in−1
(by induction hypothesis),
which can be rearranged to show that the induction hypothesis holds for k = n.
Hence the result follows.
The transition probability matrix of a DTMC can be represented graphically by its
transition diagram, which is a directed graph with as many nodes as there are states
in the state-space, and a directed arc from node i to node j if pij > 0. In particular, if
pii > 0, there is a self loop from node i to itself. The dynamic behavior of a DTMC
is best visualized via its transition diagram as follows: imagine a particle that moves
from node to node by choosing the outgoing arcs from the current node with the
corresponding probabilities. In many cases it is easier to describe the DTMC by its
transition diagram, rather than displaying the transition matrix.

Example 2.1 Transition Diagram. Consider a DTMC {Xn , n ≥ 0} on state-space

{1, 2, 3} with the following transition probability matrix:
 
.1 .2 .7
P =  .6 0 .4  .
.4 0 .6
The transition diagram corresponding to this DTMC is shown in Figure 2.1.

Example 2.2 Joint Distributions. Let {Xn , n ≥ 0} be a DTMC on state-space

{1, 2, 3, 4} and transition probability matrix given below:
 
0.1 0.2 0.3 0.4
 0.2 0.2 0.3 0.3 
P =  0.5 0.0 0.5 0.0  .


0.6 0.2 0.1 0.1

The initial distribution is
a = [0.25 0.25 0.25 0.25].
1. Compute P(X3 = 4, X2 = 1, X1 = 3, X0 = 1). From Equation 2.7, we get
P(X3 = 4, X2 = 1, X1 = 3, X0 = 1)
= a1 p13 p31 p14 = 0.25 × 0.3 × 0.5 × 0.4 = 0.015.
EXAMPLES 13

2
0.2

0.6

0.1 1 0.4
0.7

0.4
3 0.6

Figure 2.1 Transition diagram of a DTMC.

2. Compute P(X3 = 4, X2 = 1, X1 = 3). We have

P(X3 = 4, X2 = 1, X1 = 3)
4
X
= P(X3 = 4, X2 = 1, X1 = 3|X0 = i)P(X0 = i)
i=1
X4
= ai pi3 p31 p14 = 0.06.
i=1

2.2 Examples

Now we describe several examples of time homogeneous DTMCs. In many cases

we omit the specification of the initial distribution, since it can be arbitrary. Also,
we specify the transition probability matrix, or the transition probabilities, or the
transition diagram, depending on what is more convenient. As we saw before, they
carry the same information, and are thus equivalent.

Example 2.3 Two-State DTMC. One of the simplest DTMCs is one with two
states, labeled 1 and 2. Thus S = {1, 2}. Such a DTMC has a transition matrix
as follows:  
α 1−α
,
1−β β
where 0 ≤ α, β ≤ 1. The transition diagram is shown in Figure 2.2. Thus if the
DTMC is in state 1, it jumps to state 2 with probability 1 − α; if it is in state 2, it
jumps to state 1 with probability 1 − β, independent of everything else.

Next we give several examples of two-state DTMCs.

Example 2.4 Two-State Weather Model. Consider a simple weather model in

which we classify the day’s weather as either “sunny” or “rainy.” On the basis of
14 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
1–α

α 1 2 β

1–β

Figure 2.2 Transition diagram of a two-state DTMC.

previous data we have determined that if it is sunny today, there is an 80% chance
that it will be sunny tomorrow regardless of the past weather; whereas, if it is rainy
today, there is a 30% chance that it will be rainy tomorrow, regardless of the past.
Let Xn be the weather on day n. We shall label sunny as state 1, and rainy as state 2.
Then {Xn , n ≥ 0} is a DTMC on S = {1, 2} with transition probability matrix
 
.8 .2
.
.7 .3
Clearly, the DTMC has a higher tendency to move to state 1, thus implying that this
is a model of the weather at a sunny place!

Example 2.5 Clinical Trials. Suppose two drugs are available to treat a particular
disease, and we need to determine which of the two drugs is more effective. This is
generally accomplished by conducting clinical trials of the two drugs on actual pa-
tients. Here we describe a clinical trial setup that is useful if the response of a patient
to the administered drug is sufficiently quick, and can be classified as “effective” or
“ineffective.” Suppose drug i is effective with probability pi , (i = 1, 2). In practice
the values of p1 and p2 are unknown, and the aim is to determine if p1 ≥ p2 or
p2 ≥ p1 . Ethical reasons compel us to use the better drug on more patients. This is
achieved by using the play the winner rule as follows.
The first patient is given either drug 1 or 2 at random. If the nth patient is given
drug i (i = 1, 2) and it is observed to be effective for that patient, then the same drug
is given to the (n + 1)-st patient; if it is observed to be ineffective then the (n + 1)-
st patient is given the other drug. Thus we stick with a drug as long as its results
are good; when we get a bad result, we switch to the other drug – hence the name
“play the winner.” Let Xn be the drug (1 or 2) administered to the n-th patient. If the
successive patients are chosen from a completely randomized pool, then we see that
P(Xn+1 = 1|Xn = 1, Xn−1 , · · · , X1 )
= P(drug 1 is effective on the n-th patient) = p1 .
We can similarly derive P(Xn+1 = j|Xn = i; history) for all other (i, j) combina-
tions, thus showing that {Xn , n ≥ 0} is a DTMC. Its transition probability matrix is
given by  
p1 1 − p1
.
1 − p2 p2
EXAMPLES 15
If p1 > p2 , the DTMC has a higher tendency to move to state 1, thus drug 1 (the better
drug) is used more often. Thus the ethical purpose served by the play the winner
rule.

Example 2.6 Two-State Machine. Consider a machine that can be either up or

down. If it is up on day n, it is up on day n + 1 with probability pu , independent
of the past. If it is down on day n, it is down on day n + 1 with probability pd ,
also independent of the past. Let Xn be the state of the machine (0 if is down, and
1 if it is up) on day n. We see that the Markov property is given as an assumption.
Thus {Xn , n ≥ 0} is a DTMC on state-space S = {0, 1} with transition probability
matrix given by  
pd 1 − pd
.
1 − pu pu
Note that the above transition probability matrix implies that the up and down times
are geometrically distributed.

Example 2.7 Two-Machine Workshop. Suppose a work shop has two identical
machines as described in Example 2.6. The two machines behave independently of
each other. Let Xn be the number of working machines on day n. Is {Xn , n ≥ 0} a
DTMC?
The state-space is S = {0, 1, 2}. Next we verify the Markov property given by
Equation 2.1. For example, we have
P(Xn+1 = 0|Xn = 0, Xn−1 , · · · , X0 )
= P(Xn+1 = 0|Both machines are down on day n, Xn−1 , · · · , X0 )
= P(Both machines are down on day n + 1| Both machines are down on day n)
= pd pd .
Similarly we can verify that P(Xn+1 = j|Xn = i, Xn−1 , · · · , X0 ) depends only on
i and j for all i, j ∈ S. Thus {Xn , n ≥ 0} is a DTMC on state-space S = {0, 1, 2}
with transition probability matrix given by
p2d (1 − pu )2
 
pd (1 − pu )
P =  pd (1 − pu ) pd pu + (1 − pd )(1 − pu ) pu (1 − pd )  .
(1 − pu )2 2pu (1 − pu ) p2u
This example can be extended to r ≥ 2 machines in a similar fashion.

Example 2.8 Independent and Identically Distributed Random Variables. Let

{Xn , n ≥ 0} be a sequence of iid random variables with common probability mass
function (pmf)
αj = P(Xn = j), j ≥ 0.
Then {Xn , n ≥ 0} is a DTMC on S = {0, 1, 2, · · ·} with transition probabilities
given by
pi,j = P(Xn+1 = j|Xn = i, Xn−1 , · · · , X0 ) = P(Xn+1 = j) = αj ,
16 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
for all i ∈ S. Thus the transition probability matrix P of this DTMC has identical
rows, each row being the row vector α = [α0 , α1 , · · ·].

Example 2.9 Random Walk. Let {Zn , n ≥ 1} be a sequence of iid random vari-
ables with common pmf
αk = P(Xn = k), k = 0, ±1, ±2, · · · .
Define
n
X
X0 = 0, Xn = Zn , n ≥ 1.
k=1
The {Xn , n ≥ 0} has state-space S = {0, ±1, ±2, · · ·}. To verify that it is a DTMC,
we see that, for all i, j ∈ S,
P(Xn+1 = j|Xn = i, Xn−1 , · · · , X0 )
n+1
X n
X
= P( Zk = j| Zk = i, Xn−1 , · · · , X0 )
k=1 k=1
= P(Zn+1 = j − i) = αj−i .
Thus {Xn , n ≥ 0} is a DTMC with transition probabilities
pi,j = αj−i , i, j ∈ S.
This random walk is called space-homogeneous, or state-independent, since, Zn ,
the size of the n-th step does not depend on the position of the random walk at
time n.

Example 2.10 State-Dependent Random Walk.

Consider a particle that moves randomly on a doubly infinite one dimensional lat-
tice where the lattice points are labeled · · · , −2, −1, 0, 1, 2, · · · . The particle moves
by taking steps of size 0 or 1 or -1 as follows: if it is on site i at time n, then at time
n + 1 it moves to site i + 1 with probability pi , or to site i − 1 with probability qi , or
stays at site i with probability ri = 1 − pi − qi , independent of its motion up to time
n. Let Xn be the position of the particle (the label of the site occupied by the particle)
at time n. Thus {Xn , n ≥ 0} is a DTMC on state-space S = {0, ±1, ±2, · · ·} with
transition probabilities given by
pi,i+1 = pi , pi,i−1 = qi , pi,i = ri , i ∈ S.
This random walk is not space homogeneous, since the step size distribution depends
upon where the particle is. When ri = 0, pi = p, qi = q for all i ∈ S, we call it a
simple random walk. Further, if p = q = 1/2, we call it a simple symmetric random
walk.
If q0 = 0 and P(X0 ≥ 0) = 1, the random walk {Xn , n ≥ 0} has state-space
S = {0, 1, 2, · · ·}. If q0 = r0 = 0 (and hence p0 = 1), the random walk is said to
have a reflecting barrier at 0, since whenever the particle hits site zero, it “bounces
back” to site 1. If q0 = p0 = 0 (and hence r0 = 1), the random walk is said to have
EXAMPLES 17
an absorbing barrier at 0, since in this case when the particle hits site zero, it stays
there forever. If, for some integer N ≥ 1, q0 = pN = 0, and P(0 ≤ X0 ≤ N ) = 1,
the random walk has state-space S = {0, 1, 2, · · · , N − 1, N }. The barrier at N can
be reflecting or absorbing according to whether rn is zero or one. Simple random
walks with appropriate barriers have many applications. We describe a few in the
next three examples.

Example 2.11 Gambler’s Ruin. Consider two gamblers, A and B, who have a
combined fortune of N dollars. They bet one dollar each on the toss of a coin. If the
coin turns up heads, A wins a dollar from B, and if the coin turns up tails, B wins
a dollar from A. Suppose the successive coin tosses are independent, and the coin
turns up heads with probability p and tails with probability q = 1 − p. The game
ends when either A or B is broke (or ruined).
Let Xn denote the fortune of gambler A after the n-th toss. We shall assume the
coin tossing continues after the game ends, but no money changes hands. With this
convention we can analyze the stochastic process {Xn , n ≥ 0}. Obviously, if Xn =
0 (A is ruined) or Xn = N (B is ruined), Xn+1 = Xn . If 0 < Xn < N , we have

Xn + 1 with probability p
Xn+1 =
Xn − 1 with probability q.
This shows that {Xn , n ≥ 0} is a DTMC on state-space S = {0, 1, 2, · · · , N }. Its
P P P P

1 0 1 2 N–1 N 1

q q q q

Figure 2.3 Transition diagram of the gambler’s ruin DTMC.

transition diagram is shown in Figure 2.3. Thus {Xn , n ≥ 0} is a simple random

walk on {0, 1, 2, · · · , N } with absorbing barriers at 0 and N .
A somewhat more colorful situation giving rise to the same Markov chain involves
a suitably intoxicated person taking a step towards home (at site 0) with probability
q or towards a bar (placed at site N ) with probability p, in a random manner. As soon
as he reaches the home or the bar he stays there forever. The random walk executed
by such a person is called the “drunkard’s walk.”

Example 2.12 Discrete-Time Queue: Bernoulli Arrivals and Departures. Con-

sider a service system where arrivals and departures occur at times n = 0, 1, 2, · · ·.
At each time n, a customer is removed (a departure occurs) from the head of the line
(if the queue is not empty) with probability q, then a customer is added (an arrival
occurs) at the end of the line with probability p (0 ≤ p, q ≤ 1). All the arrivals
and departures are independent of everything else. Thus the number of arrivals and
(potential) departures at each time are Bernoulli random variables. Let Xn be the
number of customers in the system at time n, after the departures and arrivals at n
18 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
are accounted for. Here we study the queue-length process {Xn , n ≥ 0}. Its state-
space is S = {0, 1, 2, · · ·}. Now, for i > 0, we have
P(Xn+1 = i + 1|Xn = i, Xn−1 , · · · , X0 )
= P(one arrival and no departure at time n + 1|Xn = i)
= p(1 − q).
Carrying out similar calculations, one can show that {Xn , n ≥ 0} is a simple random
walk on {0, 1, 2, · · ·} with the following parameters:
p0 = p q0 = 0, pi = p(1 − q), qi = q(1 − p), i ≥ 1.
We also have ri = 1 − pi − qi . Note that the barrier at 0 is neither absorbing, nor
reflecting.

Example 2.13 Urn Model. Consider two urns labeled A and B, containing a total
of N white balls and N red balls among them. An experiment consists of picking one
ball at random from each urn and interchanging them. This experiment is repeated
in an independent fashion. Let Xn be the number of white balls in urn A after n
repetitions of the experiment. Assume that initially urn A contains all the white balls,
and urn B contains all the red balls. Thus X0 = N . Note that Xn tells us precisely
the contents of the two urns after n experiments. That {Xn , n ≥ 0} is a DTMC
on state-space S = {0, 1, · · · , N } can be seen from the following calculation. For
0<i<N
P(Xn+1 = i + 1|Xn = i, Xn−1 , · · · , X0 )
= P(A red ball from A and a white ball from B is picked on the n-th experiment)
N −i N −i
= · = pi,i+1 .
N N
The other transition probabilities can be computed similarly to see that {Xn , n ≥
0} is a random walk on S = {0, 1, · · · , N } with the following parameters:
r0 = 0, p0 = 1,
 2      2
i i N −i N −i
qi = , ri = 2 · , pi = , 0 < i < N,
N N N N
rN = 0, qN = 1.
Thus the random walk has reflecting barriers at 0 and N . This urn model was used
initially by Ehrenfest to model diffusion of molecules across a permeable membrane.
One can think of the white balls and red balls as the molecules of two different gases
and the switching mechanism as the model for the diffusion across the membrane. It
also appears as Moran model in genetics.

Example 2.14 Brand Switching. A customer chooses among three brands of beer,
say A, B, and C, every week when he buys a six-pack. Let Xn be the brand he
purchases in week n. From his buying record so far, it has been determined that
EXAMPLES 19
{Xn , n ≥ 0} is a DTMC with state-space S = {A, B, C} and transition probability
matrix given below:
0.1 0.2 0.7
 

P =  0.2 0.4 0.4  . (2.8)

 

0.1 0.3 0.6

Thus if he purchases brand A beer in week n, he will purchase brand C beer in
the next week with probability .7, regardless of the purchasing history and the value
of n. Such brand switching models are used quite often in practice to analyze and
influence consumer behavior. A major effort is involved in collecting and analyzing
data to estimate the transition probabilities. This involves observing the buying habits
of a large number of customers.

Example 2.15 Success Runs. Consider a game where a coin is tossed repeatedly
in an independent fashion. Whenever the coin turns up heads, which happens with
probability p, the player wins a dollar. Whenever the coin turns up tails, which hap-
pens with probability q = 1 − p, the player loses all his winnings so far. Let Xn
denote the players fortune after the n-th toss. We have
(
0 with probability q
Xn+1 =
Xn + 1 with probability p.
This shows that {Xn , n ≥ 0} is a DTMC on state-space S = {0, 1, 2, · · ·} with
transition probabilities
pi,0 = q, pi,i+1 = p, i ∈ S.
A slightly more general version of this DTMC can be considered with transition

p0 p1 p2
0 1 2 3
q1
q2
q3

Figure 2.4 Transition diagram for the success-runs DTMC.

probabilities
pi,0 = qi , pi,i+1 = pi , i ∈ S.
Such a DTMC is called a success runs Markov chain. Its transition diagram is shown
in Figure 2.4.

Example 2.16 Production-Inventory System: Batch Production. Consider a pro-

duction inventory system in discrete time where production occurs in integer valued
random batches and demands occur one at a time at times n = 0, 1, 2, · · ·. The inven-
tory of unsold items is maintained in a warehouse. Let Xn be the number of items
in the warehouse at time n. If the warehouse is empty when a demand occurs, the
20 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
demand is lost. Let Yn be the size of the batch produced at time n (after the demand
is satisfied at time n). This implies that

Yn if Xn = 0
Xn+1 = (2.9)
Xn − 1 + Yn if Xn > 0.
This can also be written as
Xn+1 = max{Xn − 1 + Yn , Yn }.
Now suppose {Yn , n ≥ 0} is a sequence of iid random variables, with common pmf
given by
αk = P(Yn = k), k = 0, 1, 2, · · · .
We shall show that under this assumption {Xn , n ≥ 0} is a DTMC on state-space
S = {0, 1, 2, · · ·}. First, we have
P(Xn+1 = j|Xn = 0, Xn−1 , · · · , X0 )
= P(Yn = j|Xn = 0, Xn−1 , · · · , X0 ) = αj
since Yn is independent of X0 , X1 , · · · , Xn . Similarly, for i > 0, we have
P(Xn+1 = j|Xn = i, Xn−1 , · · · , X0 )
= P(Xn − 1 + Yn = j|Xn = i, Xn−1 , · · · , X0 )
= P(Yn = j − i + 1|Xn = i, Xn−1 , · · · , X0 ) = αj−i+1
assuming j ≥ i − 1. Otherwise the above transition probability is zero. Thus
{Xn , n ≥ 0} is a DTMC with transition probability matrix given by
 
α0 α1 α2 α3 · · ·
 α0 α1 α2 α3 · · · 
 
 0 α0 α1 α2 · · · 
P = 0 . (2.10)
 
 0 α0 α1 · · ·  
 0 0 0 α 0 · · · 
.. .. .. ..
 
..
. . . . .
Note that {Xn , n ≥ 0} can jump up by any integral amount in one step, but can
decrease by at most one. Matrices of the form above are known as the upper Hessen-
berg matrices. Markov chains with this type of transition probability matrix arise in
many applications, especially in queueing theory.

Example 2.17 Production-Inventory System: Batch Demands. Now consider

a production-inventory system where the demands occur in integer valued random
batches and the production occurs one at a time at times n = 0, 1, 2, · · ·. The inven-
tory of unsold items is maintained in a warehouse. Let Xn be the number of items
in the warehouse at time n. We assume that the production occurs before demand, so
that Xn + 1 items are available to satisfy the demand at time n. Let Yn be the size of
the batch demand at time n. Any part of the demand that cannot be satisfied is lost.
EXAMPLES 21
This implies that

0 if Yn ≥ Xn + 1
Xn+1 =
Xn + 1 − Yn if Yn < Xn + 1.
This can also be written as
Xn+1 = max{Xn + 1 − Yn , 0}. (2.11)
Now suppose {Yn , n ≥ 0} is a sequence of iid random variables, with common pmf
given by
αk = P(Yn = k), k = 0, 1, 2, · · · .
We shall show that under this assumption {Xn , n ≥ 0} is a DTMC on state-space
S = {0, 1, 2, · · ·}. We have, for 0 < j ≤ i + 1
P(Xn+1 = j|Xn = i, Xn−1 , · · · , X0 )
= P(max{Xn + 1 − Yn , 0} = j|Xn = i, Xn−1 , · · · , X0 )
= P(Xn + 1 − Yn = j|Xn = i, Xn−1 , · · · , X0 )
= P(Yn = i − j + 1|Xn = i, Xn−1 , · · · , X0 ) = αi−j+1
since Yn is independent of X0 , X1 , · · · , Xn . For j = 0, we have
P(Xn+1 = 0|Xn = i, Xn−1 , · · · , X0 )
= P(max{Xn + 1 − Yn , 0} = j|Xn = i, Xn−1 , · · · , X0 )
= P(Yn ≥ i + 1|Xn = i, Xn−1 , · · · , X0 )
X∞
= αk = βi (say).
k=i+1

All other transition probabilities are zero. Thus {Xn , n ≥ 0} is a DTMC with tran-
sition probability matrix given by
 
β0 α0 0 0 0 ···
 β1 α1 α0 0 0 ··· 
 
 β2 α2 α1 α0 0 · · · 
P = . (2.12)
 β3 α3 α2 α1 α0 · · · 
.. .. .. .. ..
 
..
. . . . . .
Note that {Xn , n ≥ 0} can increase by at most one. Matrices of the form above are
known as the lower Hessenberg matrices. Markov chains with this type of transition
probability matrix also arise in many applications, again in queueing theory.

The last two examples illustrate a general class of DTMCs {Xn , n ≥ 0} that are
generated by the following recursion
Xn+1 = f (Xn , Yn ), n ≥ 0,
where {Yn , n ≥ 0} is a sequence of iid random variables. The reader is urged to
construct more DTMCs of this structure for further understanding of the DTMCs.
22 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
2.3 DTMCs in Other Fields

In this section we present several examples where the DTMCs have been used to
model real life situations.

2.3.1 Genomics

In 1953 Francis Crick and James Watson, building on the experimental research of
Rosalind Franklin, first proposed the double helix model of DNA (deoxyribonu-
cleic acid), the key molecule that contains the genetic instructions for the making
of a living organism. It can be thought of as a long sequence of four basic nu-
cleotides (or bases): adenine, cytosine, guanine and thymine, abbreviated as A, C, G,
and T , respectively. The human DNA consists of roughly 3 billion of these bases.
The sequence of these bases is very important. A typical sequence may read as:
CT T CT CAAAT AACT GT GCCT C · · ·.
Let Xn be the n-th base in the sequence. It is clear that {Xn , n ≥ 1} is a discrete-
time stochastic process with state-space S = {A, C, G, T }. Clearly, whether it is a
DTMC or not needs to be established by statistical analysis. We will not get into the
mechanics of doing so in this book. By studying a section of the DNA molecule one
might conclude that the {Xn , n ≥ 1} is a DTMC with transition probability matrix
given below (rows and column are ordered as A, C, G, T )
 
0.180 0.274 0.426 0.120
 0.170 0.368 0.274 0.188 
P (1) =  0.161 0.339 0.375 0.135  .


0.079 0.355 0.384 0.182

However, by studying another section of the molecule one may conclude that
{Xn , n ≥ 1} is a DTMC with transition probability matrix given below:
 
0.300 0.205 0.285 0.210
 0.322 0.298 0.078 0.302 
 0.248 0.246 0.298 0.208  .
P (2) =  

0.177 0.239 0.292 0.292

Clearly the sequence behaves differently in the two sections. In the first section, with
transition probability matrix P (1), the base G is much more likely to follow A, than
in the second section with transition probability matrix P (2).
The researchers have come up with two ways to model this situation. We describe
them briefly below.
Hidden Markov Models. We assume that DNA molecule is segmented into non-
overlapping sections of random lengths, some of which are described by the matrix
P (1) (called type 1 segments) and the others by the matrix P (2) (called type 2 seg-
ments). We define Yn to be the section (1 or 2) the n-th base falls in. This cannot
be observed. We assume that {Yn , n ≥ 1} is itself a DTMC on state-space {1, 2},
DTMCS IN OTHER FIELDS 23
with the following transition probability matrix (estimated by the statistical analysis
of the entire sequence):  
.1 .9
Q= .
.3 .7
The above transition probability matrix says that the sequence is more likely to switch
from section type 1 to 2, than from 2 to 1. Now we create a bivariate stochastic
process {(Xn , Yn ), n ≥ 1}. It is clear that this is a DTMC. For example,
P(Xn+1 = b, Yn+1 = j|Xn = a, Yn = i) = Qi,j P (i)a,b ,
for i, j ∈ {1, 2} and a, b ∈ {A, C, G, T }. Such a DTMC model is called a hidden
Markov model, because we can observe Xn , but not Yn . The second component is
only a modeling tool to describe the behavior of the DNA molecule.
Higher-Order Markov Models. We illustrate with an example of a second order
DTMCs. A stochastic process {Xn , n ≥ 0} is called a second order DTMC on
state-space S if, for n ≥ 1,
P(Xn+1 = k|Xn = j, Xn−1 = i, Xn−2 , · · · , X0 )
= P(Xn+1 = k|Xn = j, Xn−1 = i) = p(i,j);k ,
for all i, j, k ∈ S. Note that we do not need new theory to study such processes.
We simply define Zn = (Xn , Xn−1 ). Then {Zn , n ≥ 1} is a DTMC on S × S.
Such higher order Markov models have been found useful in DNA analysis since
it has been empirically observed that, for example, the frequency with which A is
observed to follow CG, is different than the frequency with which it follows AG.
This should not be the case if the simple 4-state DTMC model is valid. For our
DNA sequence, the second order Markov model produces a DTMC with a state-
space having 16 elements {AA, AC, · · · , T G, T T }, and needs a 16 by 16 transition
probability matrix to describe it. The matrix has only 16*4 non-zero entries since the
Zn process can jump from a state to only four other states. The higher order Markov
models have more parameters to estimate (the k-th order Markov model has a state-
space of cardinality 4k ), but generally produce a better fit, and a better description of
the sequence.
One can try to use non-homogeneous DTMCs to model a DNA molecule, but this
tends to have too many parameters (one transition probability matrix for each n),
and cannot be estimated from a single sequence. The reader is referred to Biological
Sequence Analysis by Durbin, Eddy, Krogh, and Mitchison (2001) for more details.

2.3.2 Genetics

The study of genetics originated with Gregor Mendel’s pioneering experiments on

the interbreeding of pea plants. Since then the subject of genetics, based on solid ex-
perimental work, has given rise to sophisticated stochastic models. We shall illustrate
here the simplest models from mathematical genetics. First we introduce some basic
background information about population genetics.
24 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
It is well known that the physical characteristics like skin color, eye color, height,
etc., are passed from parents to the children though genes. Genes can be classified
as dominant (d) or recessive (r). Each cell in the parent contains two genes for each
characteristic. A human DNA contains 20,000 to 30,000 distinct gene pairs. During
conception, a cell from each parent contributes one of its genes from each pair to
create a complete new cell of the offspring.
For example, consider a particular characteristic such as presence or absence of
skin pigmentation. An individual who carries two dominant genes for this character-
istic is denoted by dd and is called dominant, an individual carrying two recessive
genes is called recessive and is denoted by rr, and an individual carrying one reces-
sive and one dominant gene is called hybrid, denoted by dr. The physical charac-
teristic of a dd individual is indistinguishable from that of a dr individual, i.e., they
have different genotypes, but the same phenotype.
The Genotype Evolution Model. Consider a population of two individuals. We rep-
resent the state of the population by specifying the genotype of each individual. Thus
a state (dd, dd) means both individuals are dominant. There are six possible states
of the population: S = {(dd, dd), (dd, dr), (dd, rr), (dr, dr), (dr, rr), (rr, rr)}. For
example, suppose the initial state of the population is (dd, dr). When these two in-
dividuals produce an offspring, there are four possibilities—depending upon which
gene from each parent the offspring inherits—namely, dd, dr, dd, dr. Thus the off-
spring will be dominant with probability .5, and hybrid with probability .5. Suppose
the initial two individuals produce two independent offsprings. Thus the next gen-
eration again consists of two individuals. This process is repeated indefinitely. Let
Xn be the state of the population in the n-th generation. Then it can be seen that
{Xn , n ≥ 0} is a DTMC on state—space S with transition probability matrix given
below (where the rows and columns are indexed in the same sequence as in S):
 
1 0 0 0 0 0
 1/4 1/2 0 1/4 0 0 
 
 0 0 0 1 0 0 
P =  1/16 1/4 1/8 1/4 1/4 1/16  .
 (2.13)
 
 0 0 0 1/4 1/2 1/4 
0 0 0 0 0 1
Thus once the population state reaches (dd, dd) or (rr, rr), it stays that way forever.
This model can be extended to populations of size more than two, however, the state-
space gets increasingly large. The next two models attempt to alleviate this difficulty.
The Wright-Fisher Model. Consider a population of a fixed number, N , of genes.
If i of these are dominant and N − i are recessive, we say that the state of the popu-
lation is i. The composition of the N genes in the next generation is determined by
randomly sampling N genes from this population with replacement in an indepen-
dent fashion. Let Xn be the state of the population in the n-th generation.
Since a dominant gene is picked with probability Xn /N and a recessive gene is
picked with probability 1 − Xn /N , it can be seen that {Xn , n ≥ 0} is a DTMC on
DTMCS IN OTHER FIELDS 25
state-space S = {0, 1, 2, · · · , N } with transition probabilities:
   j  N −j
N i i
pi,j = 1− , i, j ∈ S.
j N N
In other words,
Xn+1 ∼ Bin(N, Xn /N ).
Once all the genes in the population are dominant (state N ), or recessive (state 0)
they stay that way forever.
The Moran Model. This model also considers a population with a fixed number, N ,
of genes. If i of these are dominant and N −i are recessive, we say that the state of the
population is i. Unlike in the Fisher-Wright model, the composition of genes in the
population changes by sampling one gene from the exiting population, and replacing
a randomly chosen gene with it. Let Xn be the state of the population after the n-th
step. Thus the randomly deleted gene is dominant with probability Xn /N and the
randomly added gene is independent of the deleted gene, and is dominant with prob-
ability Xn /N . Thus {Xn , n ≥ 0} is a simple random walk on S = {0, 1, 2, · · · , N }
with parameters:
p0 = 0, r0 = 1,
 
i i
pi = qi = · 1− , ri = 1 − pi − qi , 1 ≤ i ≤ N − 1
N N
qN = 0 rN = 1.
States 0 and N are absorbing.
The above two models assume that there are no mutations, migrations, or selec-
tions involved in the process. It is possible to modify the model to account for these
factors. For more details see Statistical Processes of Evolutionary Theory by P. A. P.
Moran.

2.3.3 Genealogy

Genealogy is the study of family trees. In 1874 Francis Galton and Henry Watson
wrote a paper on extinction probabilities of family names, a question that was origi-
nally posed by Galton about the likelihood of famous family names disappearing due
to lack of male heirs. It is observed that ethnic populations such as the Koreans and
Chinese have been using family names that are passed on via male children for sev-
eral thousand years, and hence are left with very few family names. While in other
societies where people assume new family names more easily, or where the tradition
of family names is more recent, have many family names.
In its simplest form, we consider a patriarchal society where the family name
is carried on by the male heirs only. Consider the Smith family tree as shown in
Figure 2.5.
The initiator is Steve Smith, who we shall say constitutes the zeroth generation.
Steve Smith has two sons: Peter and Eric Smith, who constitute the first generation.
26 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR

Steve

Peter Eric

Robert Edward

Bruce

Figure 2.5 The Smith family tree.

Peter does not have any male offspring, whereas Eric has two: Robert and Edward
Smith. Thus the second generation has two males. The third generation has only one
male: Bruce Smith, who dies without a male heir, and hence the family name Smith
initiated by Steve dies out. We say that the family name of Steve Smith became
extinct in the third generation.
To model this situation let Xn be the number of individuals in the n-th generation,
starting with X0 = 1. We index the individuals of the n-th generation by integers
1, 2, · · · , Xn . Let Yr,n be the number of male offspring to the r-th individual of the
n-th generation. Then we have
Xn
X
Xn+1 = Yr,n . (2.14)
r=1

Now suppose {Yr,n } are iid random variables. Then {Xn , n ≥ 0} is a DTMC on
state-space S = {0, 1, 2, · · ·} with transition probabilities
pi,j = P(Xn+1 = j|Xn = i, Xn−1 , · · · , X0 )
Xn
!
X
= P Yr,n = j|Xn = i, Xn−1 , · · · , X0
r=1
i
!
X
= P Yr,n = j ,
r=1

where the last probability can be (in theory) computed as a function of i and j. The
{Xn , n ≥ 0} process is called a branching process.
DTMCS IN OTHER FIELDS 27
Typical questions of interest are: What is the probability that the family name
eventually becomes extinct? How many generations does it take before it becomes
extinct, given that it does become extinct? How many total males are produced in the
family? What is the size of the n-th generation?
Although we have introduced the branching process as a model of propagation of
family names, the same models arises in other areas, such as nuclear physics, spread
of diseases, or rumors or chain-letters or internet jokes in very large populations. We
give an example of a nuclear reaction: A neutron (zeroth generation) is introduced
from outside in a fissionable material. The neutron may pass through the material
without hitting any nucleus. If it does hit a nucleus, it will cause a fission resulting in
a random number of new neutrons (first generation). These new neutrons themselves
behave like the original neutron, and each will produce its own random number of
new neutrons through a possible collision. Thus Xn , the number of neutrons after
the n-th generation, can be modeled as a branching process. In nuclear reactors the
evolution of this branching process is controlled by inserting moderator rods in the
fissionable material to absorb some of the neutrons.

2.3.4 Finance

DTMCs have become an important tool in mathematical finance. This is a very rich
and broad subject, hence we concentrate on relatively narrow and basic model of
stock fluctuations. Let Xn be the value of a stock at time n (this could be a day,
or a minute). We assume that {Xn , n ≥ 0} is a stochastic process with state-space
(0, ∞), not necessarily discrete, although stock values are reported in integer cents.
Define the return in period n as
Xn − Xn−1
Rn = , n ≥ 1.
Xn−1
Thus R3 = .1 implies that the stock value increased by 10% from period two to
three, R2 = −.05 is equivalent to saying that the stock value decreased by 5% from
period one to two. From this definition it follows that
n
Y
Xn = X0 (1 + Ri ), n ≥ 1. (2.15)
i=1

Thus specifying the stochastic process of returns {Rn , n ≥ 1} along with X0 is

equivalent to specifying the stochastic process {Xn , n ≥ 0} . The simplest model
for the return process is to assume that it is a sequence of iid random variables taking
values in (−1, ∞). Note that this ensures that Xn > 0 for all n ≥ 1 if X0 > 0. Also,
since
Xn+1 = Xn (1 + Rn+1 ), n ≥ 0,
it follows that {Xn , n ≥ 0} is a DTMC with state-space (0, ∞), which may not be
discrete.
A further simplification occurs if we assume that {Rn , n ≥ 1} is a sequence of iid
28 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
random variables with common pmf

P(Rn = u) = p, P(Rn = −d) = q = 1 − p,

where 0 ≤ p ≤ 1, u ≥ 0, and 0 ≤ d < 1 are fixed real numbers. Thus if u = .1

and d = .05, the stock goes up by 10% with probability p and goes down by 5%
with probability q = 1 − p in each period. This is a very useful model of discrete
time stock movements and is used frequently in financial literature. To see why it
is called a binomial model, let {Yn , n ≥ 1} be a sequence of iid Ber(p) (Bernoulli
with parameter p) random variables. Then Zn = Y1 + · · · + Yn is a binomial random
variable with parameters n and p. With this notation we can write Equation 2.15 as

Xn = X0 (1 + u)Zn (1 − d)n−Zn .

Note that for each value of X0 , Xn can take n discrete values. This model of stock
fluctuations is called the binomial model.
Since stock values can go up or down, investors take a risk when they invest in
stocks, rather than putting their money in a risk-less money market account that gives
a fixed positive rate of return. Hence the financial industry has created several finan-
cial instruments that mitigate or bound such risks. A simple instrument of this type is
the European call option. It gives the holder of the option a right, but not an obliga-
tion, to buy the stock on a specified day, say T , at a specified price, say K. Clearly, if
XT , the value of the stock on day T , is less than K, the holder will not exercise the
option, and the option will expire with no profit and no loss to the holder. If XT > K,
the holder will exercise the option and buy the stock at K, and immediately sell it at
XT and make a tidy profit of XT − K. In general the holder gets max{0, XT − K}
at time T , which is never negative. Thus there is no risk of loss in this financial in-
strument. Clearly, the buyer of this option must be willing to pay a price to buy this
risk-less instrument. How much should she pay? What is the fair value of such an
option? There are many other such options. For example, a put option gives a right
to sell. American versions of these options can be exercised anytime until T , and not
just at time T , as is the case in the European options. Valuation of these options is a
highly technical area, and DTMCs play a major role in the discrete time versions of
these problems. See Options, Futures, and Other Derivatives by J. C. Hull for more
details at an elementary level.

2.3.5 Manpower Planning

Large organizations, such as government, military, or multinational corporations,

employ a large number of people who are generally categorized into several grades
and are promoted from one grade to another depending upon performance, seniority,
and need. The employees also leave the organization from any grade, and new em-
ployees are hired into different grades. Thus the state the organization, described by
the employee histories, changes dynamically over time, and it is of interest to model
this evolution by a stochastic process.
DTMCS IN OTHER FIELDS 29
Here we present a simple stochastic model of an organization consisting of M
employees distributed in N grades. We assume that the number of employees is kept
constant by the hiring of a new employee whenever an existing employee leaves the
organization. We assume that all employees behave independently and identically.
We assume that the grade changes and hiring and departures occur at times n =
1, 2, · · · . If an employee is in grade i at time n, he or she moves to grade j at time
n + 1 with probability rij , or leaves the organization with probability ri . Clearly, we
must have
XN
ri + rij = 1, for all i = 1, 2, · · · , N.
j=1
When an employee leaves the organization, he or she is replaced instantaneously by a
new employee who starts in grade j with probability aj . Note that this is an extremely
simplified version of reality: we have ignored the issues of seniority, constraints on
the number of employees in a given grade, changing number of employees, etc.
Let Xn (r) be the grade of the r-th employee (1 ≤ r ≤ M ) at time n. Think of r
as an employee identification number, and when the r-th employee leaves the system
and is replaced by a new employee, that new employee gets the identification number
r. From the above description we see that {Xn (r), n ≥ 0} is a DTMC on state-space
S = {1, 2, · · · , N } with transition probabilities given by
pij = P(Xn+1 (r) = j|Xn (r) = i)
= P(r-th employee moves from grade i to j)
+P(r-th employee leaves and is replaced by a new one in grade j)
= rij + ri aj .
Now let Xn = [Xn (1), Xn (2), · · · , Xn (M )]. Since Xn (r), 1 ≤ r ≤ M are inde-
pendent we see that {Xn , n ≥ 0} is a DTMC on state-space S M , with the following
transition probabilities:
P(Xn+1 = [j1 , j2 , · · · , jM ]|Xn = [i1 , i2 , · · · , iM ])
M
Y
= P(Xn+1 (r) = jr |Xn (r) = ir )
r=1
M
Y
= pir ,jr .
r=1

An alternate method of modeling the system is to define Yn (r) to be the num-

ber of employees in grade r at time n. We leave to the reader to show that Yn =
[Yn (1), Yn (2), · · · , Yn (N )] is a DTMC and compute its transition probabilities.
Typically we are interested in the following: what is the composition of the work-
force at time n? What is the expected number of employees in grade r at time n?
What is the expected amount of time an employee spends in the organization before
leaving? How can we control the distribution of workers in the various grades? We
30 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
shall see how these questions can be answered with the help of the tools we shall
develop in this and the following two chapters.

2.3.6 Telecommunications

Wireless communication via cell phones has become commonly available. In such
a system a wireless device such as a cell phone communicates with a cell tower in a
bi-directional fashion, that is, it receives data from the cell tower and sends data to it.
The rate at which data can be transmitted changes randomly with time due to many
factors, for example, changing position of the user, weather, topology of the terrain,
to name a few. The cell tower knows at all times how many users are registered with
it, and what data-rate is available to each user. We consider a technology in which
the time is slotted into short intervals, say a millisecond long, and the cell tower can
communicate with exactly one user during each time slot.
Let Rn (u) be the data rate (in kilo-bits per second) available to user u in the n-th
slot. It is commonly assumed {Rn (u), n ≥ 0} is a DTMC with a finite state-space,
for example, {38.4, 76.8, 102.6, 153.6, 204.8, 307.2, 614.4, 921.6, 1228.8, 1843.2,
2457.6}, and that the data-rates available to different users are independent. Now
let Xn (u) be the amount of data (in kilobits) waiting for transmission at user u at the
beginning of the n-th time slot, and An (u) be the new data that arrives for the user
in the n-th slot. Thus if user u is served during the n-th time slot, Xn (u) + An (u)
amount of data is available for transmission, out of which Rn (u) is actually trans-
mitted. Using v(n) to denote the user that is served in the n-th slot, we see that the
following recursion holds:

max{Xn (u) + An (u) − Rn (u), 0} if u = v(n)
Xn+1 (u) =
Xn (u) + An (u) if u 6= v(n).
Now suppose there are a fixed number N of users in the reception area of the cell
tower. Let Rn = [Rn (1), Rn (2), · · · , Rn (N )], Xn = [Xn (1), Xn (2), · · · , Xn (N )],
and An = [An (1), An (2), · · · , An (N )]. Suppose the cell tower knows the state of
the system (Rn , Xn , An ) at the beginning of the n-th time slot. It decides which
user to serve next based solely on this information. If we assume that {An (u), n ≥
0} are independent (for different u’s) sequences of iid random variables, it follows
that {(Rn , Xn ), n ≥ 0} is a DTMC with a rather large state-space and complicated
transition probabilities.
The cell tower has to decide which user to serve in each time slot so that the data is
transferred at the highest possible rate (maximize throughput) and at the same time
no user is starved for too long (ensure fairness). This is the main scheduling problem
in wireless communications. For example, the cell tower may decide to serve that
user in the n-th time slot who has the highest data rate available to it. This may
maximize the throughput, but may be unfair for those users who are stuck with low
data rate environment. On the other hand, the cell tower may decide to serves the user
u with the highest backlog Xn (u) + An (u). This may also be unfair, since the user
with largest data requirement will be served most of the time. One rule that attempts
MARGINAL DISTRIBUTIONS 31
to strike a balance between these two conflicting objectives serves the user u that has
the largest value of min(Rn (u), Xn (u) + An (u)) · (Xn (u) + An (u)).
DTMCs have been used in a variety of problems arising in telecommunications.
The above model is but one example. DTMCs have been used in the analysis of radio
communication networks using protocols like ALOHA, performance of the ethernet
protocol and the TCP protocol in the Internet, to name a few other applications. The
reader is referred to the classic book Data Networks by Bertsekas and Gallager for a
simple introduction to this area.

2.4 Marginal Distributions

The examples in Sections 2.2 and 2.3 show that many real-world systems can be
modeled by DTMCs and provide the motivation to study them. Section 2.1 tells us
how to characterize a DTMC. Now we follow the road map laid out in Chapter 1 and
study the transient behavior of the DTMCs. In this section we first study the marginal
distributions of DTMCs.
Let {Xn , n ≥ 0} be a DTMC on state-space S = {0, 1, 2, · · ·} with transition
probability matrix P and initial distribution a. In this section we shall study the
distribution of Xn . Let the pmf of Xn be denoted by
(n)
aj = P(Xn = j), j ∈ S, n ≥ 0. (2.16)
(0)
Clearly aj = aj is the initial distribution. By using the law of total probability we
get
X
P(Xn = j) = P(Xn = j|X0 = i)P(X0 = i)
i∈S
X
= P(Xn = j|X0 = i)ai
i∈S
(n)
X
= ai pij , (2.17)
i∈S

where
(n)
pij = P(Xn = j|X0 = i), i, j ∈ S, n ≥ 0 (2.18)
is called the n-step transition probability, since it is the probability of going from
state i to state j in n transitions. We have
(0)
pij = P(X0 = j|X0 = i) = δij , i, j ∈ S, (2.19)
where δij is one if i = j and zero otherwise, and
(1)
pij = P(X1 = j|X0 = i) = pij , i, j ∈ S. (2.20)
(n)
If we can compute the n-step transition probabilities pij , we can compute the
marginal distribution of Xn . Intuitively, the event of going from state i to state j
involves going from state i to some intermediate state r at time k ≤ n, followed by a
32 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
trajectory from state r to state j in the remaining n − k steps. This intuition is used
(n)
to derive a method of computing pij in the next theorem. The proof of the theorem
is enlightening in itself since it shows the critical role played by the assumptions of
Markov property and time homogeneity.

Theorem 2.3 Chapman-Kolmogorov Equations. The n-step transition probabil-

ities satisfy the following equations:
(n)
X (k) (n−k)
pij = pir prj , i, j ∈ S, (2.21)
r∈S

where k is a fixed integer such that 0 ≤ k ≤ n.

Proof: Fix an integer k such that 0 ≤ k ≤ n. Then

(n)
pij = P(Xn = j|X0 = i)
X
= P(Xn = j, Xk = r|X0 = i)
r∈S
X
= P(Xn = j|Xk = r, X0 = i)P(Xk = r|X0 = i)
r∈S
X
= P(Xn = j|Xk = r)P(Xk = r|X0 = i) (Markov property)
r∈S
X
= P(Xn−k = j|X0 = r)P(Xk = r|X0 = i) (time homogeneity)
r∈S
(n−k) (k)
X
= prj pir
r∈S

which can be rearranged to get Equation 2.21. This proves the theorem.

Equations 2.21 are called the Chapman-Kolmogorov equations and can be written
more succinctly in matrix notation. Let
(n)
P (n) = [pij ].
It is called the n-step transition probability matrix. The Chapman-Kolmogorov equa-
tions can be written in matrix form as
P (n) = P (k) P (n−k) , 0 ≤ k ≤ n. (2.22)
The next theorem gives an important implication of the Chapman-Kolmogorov equa-
tions.

Theorem 2.4 The n-Step Transition Probability Matrix.

P (n) = P n , (2.23)
where P n is the n-th power of P .
MARGINAL DISTRIBUTIONS 33
Proof: We prove the result by induction on n. Let I be the identity matrix of the
same size as P . Equations 2.19 and 2.20 imply that
P (0) = I = P 0
and
P (1) = P = P 1 .
Thus the result is valid for n = 0 and 1. Suppose it is valid for some k ≥ 1. From
Equation 2.22 we get
P (k+1) = P (k) P (1)
= P k P (From the induction hypothesis)
= P k+1 .
The theorem then follows by induction.

Example 2.18 Two-State DTMC. Let P be the transition probability matrix of the
two-state DTMC of Example 2.3 on page 13. If α + β = 2, we must have α = β = 1
and hence P = I. In that case P n = I for all n ≥ 0. If α + β < 2, it can be shown
by induction that, for n ≥ 0,
(α + β − 1)n 1 − α α − 1
   
1 1−β 1−α
Pn = + .
2−α−β 1−β 1−α 2−α−β β−1 1−β
In general such a closed form expression for P n is not available for DTMCs with
larger state spaces.

Example 2.19 Simple Random Walk. Consider a simple random walk on all inte-
gers with the following transition probabilities
pi,i+1 = p, pi,i−1 = q = 1 − p, −∞ < i < ∞,
where 0 < p < 1. Compute
(n)
p00 = P(Xn = 0|X0 = 0), n ≥ 0.
Starting from state 0, the random walk can return to state 0 only in even number of
steps. Hence we must have
(n)
p00 = 0, for all odd n.
Now let n = 2k be an even integer. To return to state 0 in 2k steps starting from state
0, the random walk must take a total of k steps to the right, and k steps to the left, in
any order. There are (2k)!/(k!k!) distinct sequences of length 2k made up of k right
and k left steps, and the probability of each sequence is pk q k . Hence we get
(2k) (2k)! k k
p00 = p q , k = 0, 1, 2, · · · . (2.24)
k!k!
In a similar manner one can show that
 
(n) n a b
pij = p q , if n + j − i is even (2.25)
b
34 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
where a = (n + j − i)/2 and b = (n + i − j)/2. If n + j − i is odd the above
probability is zero.

It is not always possible to get a closed form expression for the n-step transition
probabilities, and one must do so numerically. We will study this in more detail in
Section 2.6.
(n)
Now let a(n) = [aj ] be the probability mass function (pmf) of Xn . The next
Theorem gives a simple expression for a(n) .

Theorem 2.5 Probability Mass Function of Xn .

a(n) = aP n , n ≥ 0. (2.26)

Proof: Equation 2.17 can be written in matrix form as

a(n) = a(0) P (n) .
The result follows from the observation that a(0) = a, and Equation 2.23.
It follows that the i-th row of P n gives the pmf of Xn conditioned on X0 =
i. The above theorem gives a numerically straight forward method of computing
the marginal distribution of Xn . We illustrate these concepts with several examples
below.

Example 2.20 A Numerical Example. Let {Xn , n ≥ 0} be the DTMC of Exam-

ple 2.2 on page 12. Compute the pmf of X4 .
From Theorem 2.5 we get the pmf of X4 as
 4
0.1 0.2 0.3 0.4
0.2 0.2 0.3 0.3 
a(4) aP 4 = [0.25 0.25 0.25 0.25] · 

=  0.5 0.0

0.5 0.0 
0.6 0.2 0.1 0.1
 
0.3616 0.1344 0.3192 0.1848
 0.3519 0.1348 0.3222 0.1911 
= [0.25 0.25 0.25 0.25] · 
 0.3330 0.1320

0.3340 0.2010 
0.3177 0.1404 0.3258 0.2161
= [0.34105 0.13540 0.32530 0.19825].

Example 2.21 Urn Model Continued. Let {Xn , n ≥ 0} be the stochastic process
of the urn model of Example 2.13 on page 18 with N = 10. Compute E(Xn ) for
n = 0, 5, 10, 15, and 20, starting with X0 = 10. Using the transition matrix P from
Example 2.13 we get
10
X
en = E(Xn |X0 = 10) = iP(Xn = i|X0 = 10) = aP n b,
i=1
MARGINAL DISTRIBUTIONS 35
where a = (0, 0, · · · , 0, 1) and b = (0, 1, · · · , 9, 10)′ . Numerical computations yield:
e0 = 10, e5 = 6.6384, e10 = 5.5369, e15 = 5.1759, and e20 = 5.0576. We can
see numerically that as n → ∞, en converges to 5.

Example 2.22 The Branching Process. Consider the branching process {Xn , n ≥
0} introduced in Section 2.3.3. Here we compute µn = E(Xn ) and σn2 = Var(Xn )
as a function of n. Since X0 = 1, clearly
µ0 = 1, σ02 = 0. (2.27)
2
Let µ and σ be the mean and variance of the number of offspring to a single individ-
ual. Since X1 is the number of offspring to the single individual in generation zero,
we have
µ1 = µ, σ12 = σ 2 . (2.28)
Furthermore, one can show that
Xn−1
!
X
E(Xn ) = E Yr,n−1 = µE(Xn−1 ),
r=1
X
!
Xn−1

Var(Xn ) = Var Yr,n−1 = σ 2 E(Xn−1 ) + µ2 Var(Xn−1 ).

r=1
Hence we have the following recursive equations:
µn = µµn−1 ,
σn2 = σ 2 µn−1 + µ2 σn−1
2
.
Using the initial condition from Equation 2.27, we can solve these equations to get
µn = µn
nσ 2 if µ = 1

σn2 = n .
σ 2 µn−1 µµ−1
−1
if µ 6= 1.
Thus, if µ > 1, µn → ∞, as n → ∞, that is, the branching process grows without
bounds. If µ < 1, µn → 0, as n → ∞, that is, the branching process becomes extinct.
If µ = 1, we see that µn = 1 for all n. We shall later show a rather counterintuitive
result that the branching process becomes extinct with probability 1 in this case.
In the context of nuclear reaction, if µ > 1, the reaction grows without bounds
and we get an explosion or a reactor meltdown. If µ < 1, the reaction is not self
sustaining, and dies out naturally. In a nuclear power station, in the initial firing
stage, one maintains µ > 1 to get the reactor to a “hot” stage. Once the reactor is
hot, moderator rods are inserted to absorb some neutrons, essentially reducing µ,
and the reaction is controlled. The moderator rods are moved in and out constantly
in response to the reactor temperature. If µ < 1, the reaction starts dying out, the
reactor starts cooling, and the moderator rods are pulled out. This raises µ to above
1, the reaction starts exploding, the reactor starts heating up, and the moderator rods
are pushed back in. This control keeps the reactor producing heat (which is converted
to electricity) at a constant rate.
36 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
In the context of propagation of family names, royal (or wealthy) males in histor-
ical times have been known to marry repeatedly until they produce a male heir. This
is an obvious attempt to maintain µ ≥ 1.

Example 2.23 Manpower Planning. Consider the manpower planning model of

Section 2.3.5. Suppose the organization classifies the employees into four grades
{1, 2, 3, 4}. Every year an employee in grades 1, 2, and 3 gets promoted from the
current grade to the next grade with probability .2, or leaves with probability .2, or
stays in the same grade. An employee in the fourth grade leaves with probability .2 or
stays in the same grade. A departing employee is immediately replaced by a new one
starting in grade 1. Suppose the organization has 100 employees distributed evenly in
the four grades. What the expected number of employees in each grade in the eighth
year?
Suppose we pick an employee at random and track his evolution. Let Xn be
the grade of this employee in year n. Thus the initial grade of this employee is
i with probability .25 for i = 1, 2, 3, 4. Following the analysis in Section 2.3.5
it follows that {Xn , n ≥ 0} is a DTMC on {1, 2, 3, 4} with initial distribution
[0.25, 0.25, 0.25, 0.25] and the following transition probability matrix:
 
0.8 0.2 0 0
 0.2 0.6 0.2 0 
 0.2 0 0.6 0.2  .
 

0.2 0 0 0.8
Then the pmf of the grade of this employee in year 8 is given by
[0.25 0.25 0.25 0.25] ∗ P 8 = [0.4958, 0.2388, 0.1140, 0.1514].
We can interpret this to mean that 49.58% of the employees are in grade 1
in year 8, etc. Thus the expected number of employees in the four grades are
[49.58, 23.88, 11.40, 15.14]. One can numerically see that after several years (24
in this example), the expected number of employees in the four grades stabilize at
[50, 25, 12.5, 12.5].

2.5 Occupancy Times

Let {Xn , n ≥ 0} be a DTMC on state-space S with transition probability matrix P .

In this section we compute the expected time spent by the DTMC in various states.
(n)
Let Vj be the number of visits to state j by the DTMC over {0, 1, 2, · · · , n}. Note
that we count the visit at time 0, that is, Vj (0) = 1 if X0 = j, and zero otherwise.
Define
(n) (n)
Mij = E(Vj |X0 = i), i, j ∈ S, n ≥ 0. (2.29)
(n)
Mij is called the occupancy time of state j up to time n starting from state i. Define
the occupancy times matrix as
(n)
M (n) = [Mij ].
The next theorem shows how to compute the occupancy times matrix M (n) .
OCCUPANCY TIMES 37
Theorem 2.6 Occupancy Times Matrix. We have
n
X
M (n) = P r , n ≥ 0, (2.30)
r=0

where P 0 = I, the identity matrix.

Proof: Fix a j ∈ S. Let Zr = 1 if Xr = j, and zero otherwise. Then

n
(n)
X
Vj = Zr .
r=0
Hence we get
(n) (n)
Mij = E(Vj |X0 = i)
n
!
X
= E Zr |X0 = i
r=0
n
X
= E(Zr |X0 = i)
r=0
Xn
= P(Xr = j|X0 = i)
r=0
n n n
(r)
X X X
(r)
= pij = [P ]ij = [P r ]ij ,
r=0 r=0 r=0

where the last equality follows from Theorem 2.4. Writing the above equation in ma-
trix form yields Equation 2.30. This proves the theorem.

We illustrate with two examples.

Example 2.24 Two-State DTMC. Consider the two-state DTMC of Example 2.3.
The n-step transition probability matrix of the DTMC was given in Example 2.18 on
page 33. Using that, and a bit of algebra, we see that the occupancy matrix for the
two-state DTMC is given by
1 − (α + β − 1)(n+1) 1 − α α − 1
   
(n) n+1 1−β 1−α
M = + .
2−α−β 1−β 1−α (2 − α − β)2 β−1 1−β
Thus, if the DTMC starts in state 1, the expected number of times it visits state 2 up
(n)
to time n is given by M12 .

It is rarely possible to compute the occupancy times matrix analytically. In most

cases we have to do the calculations numerically, as shown in the next example.

Example 2.25 Brand Switching Model Continued. Consider the brand switching
model of Example 2.14 on page 18. Compute the expected number of each brand
sold in the first ten weeks.
38 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
Let P be the transition matrix given by Equation 2.8. Since we are interested in
purchases over the weeks 0 through 9, we need to compute M (9) . Using Theorem 2.6
we get  
X 9 2.1423 2.7412 5.1165
M (9) = P n =  1.2631 3.9500 4.7869  .
n=1 1.1532 2.8511 5.9957
Thus if the customer chooses brand A in the initial week, his expected number of
purchases on brand A over the weeks 0 through 9 is 2.1423, of brand B is 2.7412,
and of brand C is 5.1165.

2.6 Computation of Matrix Powers

If P is a finite matrix with numerical entries, computing P n is relatively straight

forward, especially with the help of matrix oriented languages like Matlab or Math-
ematica. Here we describe two other methods of computing P n .

2.6.1 Method of Diagonalization

We start with some preliminaries from matrix algebra. The reader is referred to
texts by Fuller (1962) and Gantmacher (1960) for proofs and other details. An m×m
square matrix A is called diagonalizable if there exist an invertible matrix X and a
diagonal matrix
D = diag[λ1 , λ2 , · · · , λm ],
such that
D = XAX −1 .
Elements of X and D maybe complex even if A is real. It is known that the λ’s are
the eigenvalues of A, the j-th column xj of X is the right eigenvector of λj , and the
j-th row yj of X −1 is the left eigenvector of λj . That is, the λ’s are the m roots of
det(λI − A) = 0,
where det is short for determinant, xj satisfies
Axj = λj xj , (2.31)
and yj satisfies
yj A = λj yj .
If all the eigenvalues are distinct, then A is diagonalizable. This is a sufficient condi-
tion, but not necessary. With this notation we get the following theorem:

Theorem 2.7 Powers of a Square Matrix. We have

m
X
An = XDn X −1 = λnj xj yj . (2.32)
j=1
COMPUTATION OF MATRIX POWERS 39
Proof: Since A is diagonalizable, we have
A = XDX −1 . (2.33)
Hence
An = [XDX −1 ][XDX −1 ] · · · [XDX −1 ] = XDn X −1 .
Using
Dn = diag[λn1 , λn2 , · · · , λnm ]
we get the second equality in Equation 2.32.
Computation of eigenvalues and eigenvectors is done easily using a matrix ori-
ented language like Matlab. The following theorem gives important information
about the eigenvalues of a stochastic matrix P .

Theorem 2.8 Eigenvalues of P . Let P be an m × m transition probability matrix,

with m eigenvalues λi , 1 ≤ i ≤ m. Then

1. At least one of the eigenvalues is one.

2. |λi | ≤ 1 for all 1 ≤ i ≤ m.

Proof: The first result follows from the fact that P is stochastic and hence the column
vector e with all coordinates equal to one satisfies
P e = e,
thus proving that 1 is an eigenvalue of P .
To derive the second results, define the norm of an m-vector x as
kxk = max{xi : 1 ≤ i ≤ m},
and the norm of P as
kP k = sup kP xk.
x:kxk=1
Then one can show that
m
X
kP k = max{ pij } = 1,
i
j=1

since P is stochastic. Furthermore,

kP xk ≤ kP kkxk = kxk.
Now, using Equation 2.31, we get
|λi |kxi k = kλi xi k = kP xi k ≤ kxi k.
Since 0 < kxi k < ∞, the above inequality implies
|λi | ≤ 1
for all 1 ≤ i ≤ m. This proves the Theorem.
40 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
We illustrate with two examples.

Example 2.26 Three-state DTMC. Consider a three state DTMC with the follow-
ing transition probability matrix:
 
0 1 0
P =  q 0 p ,
0 1 0
where 0 < p < 1 and q = 1 − p. Simple matrix multiplications show that
 
q 0 p
P 2n =  0 1 0  , n ≥ 1, (2.34)
q 0 p
 
0 1 0
P 2n+1 =  q 0 p  , n ≥ 0. (2.35)
0 1 0
Derive these formulas using the method of diagonalization.
Simple calculations show that P has three eigenvalues 1, 0, and -1, consistent with
Theorem 2.8. Thus P is diagonalizable with
 
1 0 0
D =  0 0 0 ,
 
0 0 −1
 
1 p 1
X =  1 0 −1  ,
 
1 −q 1
and  
q 1 p
1
X −1 =  2 0 −2  .

2
q 1 −p
Thus
 
q(1 + (−1)n ) 1 − (−1)n p(1 + (−1)n )
1
P n = XDn X −1 =  q(1 − (−1)n ) 1 + (−1)n p(1 − (−1)n )  , n ≥ 1.

2
q(1 + (−1)n ) 1 − (−1)n p(1 + (−1)n )
The above equation reduces to Equation 2.34 when n is even, and Equation 2.35
when n is odd. Thus the powers of P show an oscillatory behavior as a function
of n.

Example 2.27 Genotype Evolution. Consider the six-state DTMC of the Geno-
type Evolution Model described on page 24, with the transition matrix P given by
Equation 2.13 on page 24. Compute P n .
COMPUTATION OF MATRIX POWERS 41
A tedious calculation are λ1 =
√shows that the six eigenvalues in decreasing order √
1, λ2 = 1, λ3 = (1 + 5)/4 = 0.8090, λ4 = 0.5, λ5 = 0.25, λ6 = (1 − 5)/4 =
−0.3090. The matrix of right eigenvectors is given by
 
4 0 0 0 0 0
 3 1 0 0 0 0 
 
 2 2 λ23 −1 −1 λ26 
X=  .
 2 2 1 0 4 1 
 1 3 λ3 0 1 λ6 
0 4 λ23 1 −1 λ26
Note that the eigenvalues are not distinct, since the eigenvalue 1 is repeated twice.
However, the matrix X is invertible. Hence P is diagonalizable, and the representa-
tion in Equation 2.33 holds with
D = diag[1, 1, 0.809, 0.5, 0.25, −0.3090].
Thus we get P = XDn X −1 .
n

2.6.2 Method of Generating Functions

We assume that the reader is familiar with generating functions, see Appendix D for
relevant details. Let P be an m × m matrix of transition probabilities and define
∞
X
P (z) = znP n, (2.36)
n=0

where z is a complex number. The above series converges absolutely, element by

element, if |z| < 1. From Theorem 2.8 it follows that the eigenvalues of zP are all
strictly less than one in absolute value if |z| < 1, and hence I − zP is invertible.
Now,
X∞
P (z) = I + z n P n = I + zP (z)P.
n=1
Hence
P (z) = (I − zP )−1 .
Thus the method of generating function works as follows:

1. Compute A(z) = [aij (z)] = (I − zP )−1 .

2. The function aij (z) is a rational functions of z, that is, it is a ratio of two polyno-
mials in z. Expand it in power series of z to get
∞
(n)
X
aij (z) = aij z n .
n=0

n (n)
3. P = [aij ] is the required power of P .

Generally, this method is numerically inferior to the method of diagonalization.

42 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
Example 2.28 Manpower Planning. Compute [P n ]1,1 for the transition probabil-
ity matrix P on page 36 of Example 2.23 using the generating function method.
Direct calculation, or the use of a symbolic calculation tool on a computer, yields
∞
X 5 − 4z
[P (z)]1,1 = [P n ]1,1 z n = [(I − zP )−1 ]1,1 = .
n=0
3z 2 − 8z + 5
The denominator has two roots:
√ √
4+ 31 4 − 31
z1 = , z2 = .
3 3
Using partial fractions we get
1 5 − 4z1 5 − 4z2
[P (z)]1,1 = ( − ).
z1 − z2 z − z1 z − z2
Expanding in power series in z, we get
∞ ∞ ∞
X 3 5 − 4z2 X 5 − 4z1 X
[P n ]1,1 z n = √ · ( (z/z2 )n − (z/z1 )n ).
n=0
2 31 z 2 n=0
z 1 n=0

Equating the coefficients of z n we get

3 5 − 4z2 5 − 4z1
[P n ]1,1 = √ · ( n+1 − n+1 ).
2 31 z2 z1
One can compute the expressions for other elements of P n in a similar fashion.

2.7 Modeling Exercises

2.1 We have an infinite supply of light bulbs, and Zi is the lifetime of the i-th light
bulb. {Zi , i ≥ 1} is a sequence of iid discrete random variables with common pmf
P(Zi = k) = pk , k = 1, 2, 3, · · · ,
P∞
with k=1 pk = 1. At time zero, the first light bulb is turned on. It fails at time Z1 ,
when it is replaced by the second light bulb, which fails at time Z1 + Z2 , and so on.
Let Xn be the age of the light bulb that is on at time n. Note that Xn = 0, if a new
light bulb was installed at time n. Show that {Xn , n ≥ 0} is a DTMC and compute
its transition probability matrix.

2.2 In the above exercise, let Yn be the remaining life of the bulb that is in place at
time n. For example, Y0 = Z1 . Show that {Yn , n ≥ 0} is a DTMC and compute its
transition probability matrix.

2.3 An urn contains w white balls and b black balls initially. At each stage a ball is
picked from the urn at random and is replaced by k balls of similar color (k ≥ 1). Let
Xn be the number of black balls in the urn after n stages. Is {Xn , n ≥ 0} a DTMC?
If yes, give its transition probability matrix.
MODELING EXERCISES 43
2.4 Consider a completely connected network of N nodes. At time 0 a cat resides
on node N and a mouse on node 1. During one time unit, the cat chooses a random
node uniformly from the remaining N − 1 nodes and moves to it. The mouse moves
in a similar way, independently of the cat. If the cat and the mouse occupy the same
node, the cat promptly eats the mouse. Model this as a Markov chain.

2.5 Consider the following modification of the two-state weather model of Exam-
ple 2.4: given the weather condition on day n − 1 and n, the weather condition on
day n + 1 is independent of the weather on earlier days. Historical data suggests that
if it rained yesterday and today, it will rain tomorrow with probability 0.6; if it was
sunny yesterday and today, it will rain tomorrow with probability 0.2, if it was sunny
yesterday but rained today, it will rain tomorrow with probability .5, and if it rained
yesterday but is sunny today, it will rain tomorrow with probability .25. Model this
as a four state DTMC.

2.6 A machine consists of K components in series, i.e., all the components must be
in working condition for the machine to be functional. When the machine is func-
tional at the beginning of the nth day, each component has a probability p of failing
at the beginning of the next day, independent of other components. (More than one
component can fail at the same time.) When the machine fails, a single repair person
repairs the failed components one by one. It takes exactly one day to repair one failed
component. When all the failed components are repaired the machine is functional
again, and behaves as before. When the machine is down, the working components
do not fail. Let Xn be the number of failed components at the beginning of the nth
day, after all the failure and repair events at that time are accounted for. Show that
{Xn , n ≥ 0} is a DTMC. Display the transition matrix or the transition diagram.

2.7 Two coins are tossed simultaneously and repeatedly in an independent fashion.
Coin i (i = 1, 2) shows heads with probability pi . Let Yn (i) be the number of heads
observed during the first n tosses of the i-th coin. Let Xn = Yn (1) − Yn (2). Show
that {Xn , n ≥ 0} is a DTMC. Compute its transition probabilities.

2.8 Consider the following weather forecasting model: if today is sunny (rainy) and
it is the k-th day of the current sunny (rainy) spell, then it will be sunny (rainy)
tomorrow with probability pk (qk ) regardless of what happened before the current
sunny (rainy) spell started (k ≥ 1). Model this as a DTMC. What is the state-space?
What are the transition probabilities?

2.9 Let Yn ∈ {1, 2, 3, 4, 5, 6} be the outcome of the n-th toss of a fair six-sided die.
Let Sn = Y1 + · · · + Yn , and Xn = Sn (mod 7), the remainder when Sn is divided
by 7. Assume that the successive tosses are independent. Show that {Xn , n ≥ 1} is
a DTMC, and display its transition probability matrix.

2.10 This is a generalization of the above problem. Let {Yn , n ≥ 1} be a sequence

of iid random variables with common pmf P(Yn = k) = αk , k = 1, 2, 3, · · ·. Let r
be a given positive integer and define Sn = Y1 + · · · + Yn , and Xn = Sn (mod r).
Show that {Xn , n ≥ 1} is a DTMC, and display its transition probability matrix.
44 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
2.11 A boy and a girl move into a two-bar town on the same day. Each night, the
boy visits one or the other of the two bars, starting in bar 1, according to a DTMC
with transition probability matrix
 
a 1−a
.
1−b b
Likewise the girl visits one or the other of the same two bars according to a DTMC
with transition probability matrix
 
c 1−c
,
1−d d
but starting in bar 2. Here, 0 < a, b, c, d < 1. Assume that the Markov chains are
independent. Naturally, the story ends when the boy meets the girl, i.e., when they
both go to the same bar. Model the situation by a DTMC that will help answer when
and where the story ends.

2.12 Suppose that {Yn , n ≥ 0} is a DTMC on S = {0, 1, · · · , 10} with transition

probabilities as follows:
p0,0 = .98, p0,1 = .02,
pi,i−1 = .03, pi,i = .95, pi,i+1 = .02, 1 ≤ i ≤ 9,
p10,9 = .03, p10,10 = .97.
Let Xn be the price (in dollars) of a gallon of unleaded gas on day n. We model the
price fluctuation by assuming that there is an increasing function f : S → (0, ∞)
such that Xn = f (Yn ). For example, f (i) = 4 + .05i implies that the gas price
fluctuates between $4 and $4.50, with increments of $.05. A student uses exactly one
gallon of gas per day. In order to control gas expenditure, the student purchases gas
only when the gas tank is empty (don’t ask how he does it!). If he needs gas on day
n, he purchases 11 − Yn gallons of gas. Let Zm be the price per gallon of gas paid
by the student on his m-th purchase. Show that {Zm , m ≥ 0} is a DTMC. Compute
its transition probability matrix.

2.13 Consider the following extension of Example 2.5. Suppose we follow the play
the winner rule with k drugs (k ≥ 2) as follows. The initial player is given drug 1. If
the drug is effective with the current patient, we give it to the next patient. If the result
is negative, we switch to drug 2. We continue this way until we reach drug k. When
we observe a failure of drug k, we switch back to drug 1 and continue. Suppose the
successive patients are independent, and that drug i is effective with probability pi .
Let Xn be i if the n-th patient is given drug i. Show that {Xn , n ≥ 1} is a DTMC.
Derive its transition probability matrix.

2.14 A machine with two components is subject to a series of shocks that occur
deterministically one per day. When the machine is working a shock can cause failure
of component 1 alone with probability α1 , or of component 2 alone with probability
α2 or of both components with probability α12 or no failures with probability α0 .
(Obviously α0 + α1 + α2 + α12 = 1.) When a failure occurs, the machine is shut
MODELING EXERCISES 45
down and no more failures occur until the machine is repaired. The repair time (in
days) of component i (i = 1, 2) is a geometric random variable with parameter ri ,
0 < ri < 1. Assume that there is a single repair person and all repair times are
independent. (Thus if both components fail they are repaired sequentially, and the
machine is turned on once both components are fixed.) Give the state-space and the
transition probability matrix of an appropriate DTMC that can be used to model the
state-evolution of the machine.

2.15 Suppose three players - 1,2,3 - play an infinite tournament as follows: Initially
player 1 plays against player 2. The winner of the n-th game plays against the player
who was not involved in the n-th game. Suppose bij is the probability that in a game
between players i and j, player i will win. Obviously, bij + bji = 1. Suppose the
outcomes of the successive games are independent. Let Xn be the pair that played
the n-th game. Show that {Xn , n ≥ 0} is a DTMC. Display its transition probability
matrix or the transition diagram.

2.16 Mr. Al Anon drinks one six-pack of beer every evening! Let Yn be the price
of the six-pack on day n. Assume that the price is either L or H > L, and
that {Yn , n ≥ 0} is a DTMC on state-space {H, L} with transition probability
matrix  
α 1−α
.
1−β β
Mr. Al Anon visits the beer store each day in the afternoon. If the price is high
and he has no beer at home, he buys one six pack, which he consumes in the
evening. If the price is high and he has at least one six pack at home, he does
not buy any beer. If the price is low, he buys enough six packs so that he will
have a total of five six packs in the house when he reaches home. Model this sys-
tem by a DTMC. Describe its state-space and compute the transition probability
matrix.

2.17 Let {Yn , n ≥ 1} be a sequence of iid random variables with common pmf
P(Yn = k) = αk , k = 0, 1, 2, 3, ..., M.
Define X0 = 0 and
Xn = max{Y1 , Y2 , ..., Yn }, n = 1, 2, 3....
Show that {Xn , n ≥ 0} is a DTMC. Display the transition probability matrix.

2.18 Consider a machine that alternates between two states: up and down. The suc-
cessive up and down times are independent of each other. The successive up times
are iid positive integer valued random variables with common pmf
P(up time = i) = ui , i = 1, 2, 3, . . . ,
and the successive down times are iid positive integer valued random variables with
common pmf
P(down time = i) = di , i = 1, 2, 3, . . . .
46 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
Assume that
∞
X ∞
X
ui = 1, di = 1.
i=1 i=1
Model this system by a DTMC. Describe its state-space and the transition probability
matrix.

2.19 Ms. Friendly keeps in touch with her friends via email. Every day she checks
her email inbox at 8:00am. She processes each message in the inbox at 8:00am inde-
pendently in the following fashion: she answers it with probability p > 0 and deletes
it, or she leaves it in the inbox, to be visited again the next day. Let Yn be the number
of messages that arrive during 24 hours on day n. Assume that {Yn , n ≥ 0} is a
sequence of iid random variables with common pmf
αk = P(Yn = k), k = 0, 1, 2, · · · .
Let Xn be the number of messages in the inbox at 8:00am on day n. Assume that no
new messages arrive while she is deleting the messages. Show that {Xn , n ≥ 0} is a
DTMC.

2.20 A buffer of size B bytes is used to store and play a streaming audio file. Sup-
pose the time is slotted so that one byte is played (and hence removed from the buffer)
at the end of each time slot. Let An be the number of bytes streaming into the buffer
from the internet during the nth time slot. Suppose that {An , n ≥ 1} is a sequence
of iid random variables with common pmf
αk = P(An = i), k = 0, 1, 2.
Let Xn be the number of bytes in the buffer at the end of the nth time slot, after the
input during that slot followed by the output during that slot. If the buffer is empty no
sound is played, and if the buffer becomes full, some of the incoming bytes may be
lost if there is no space for them. Both create a loss of quality. Model {Xn , n ≥ 0}
as a DTMC. What is its state-space and transition probability matrix?

2.21 A shuttle bus with finite capacity B stops at bus stops numbered 0, 1, 2, · · · on
an infinite route. Let Yn be the number of riders waiting to ride the bus at stop n.
Assume that {Yn , n ≥ 0} is a sequence of iid random variables with common pmf
αk = P(Yn = k), k = 0, 1, 2, · · · .
Every passenger who is on the bus alights at a given bust stop with probability p. The
passengers behave independently of each other. After the passengers alight, as many
of the the waiting passengers board the bus as there is room on the bus. Show that
{Xn , n ≥ 0} is a DTMC. Compute the transition probabilities.

2.22 A production facility produces one item per hour. Each item is defective with
probability p, the quality of successive items being independent. Consider the fol-
lowing quality control policy parameterized by two positive integers k and r. In the
beginning the policy calls for 100% inspection, the expensive mode of operation.
MODELING EXERCISES 47
As soon as k consecutive non-defective items are encountered, it switches to econ-
omy mode and calls for inspecting each item with probability 1/r. It reverts back
to the expensive mode as soon as an inspected item is found to be defective. The
process alternates this way forever. Model the inspection policy as a DTMC. Define
the state-space, and show the transition probability matrix or the transition diagram.

2.23 Let Dn be the demand for an item at a store on day n. Suppose {Dn , n ≥ 0}
is a sequence of iid random variables with common pmf
αk = P(Dn = k), k = 0, 1, 2, · · · .
Suppose the store follows the following inventory management policy, called the
(s, S) policy: If the inventory at the end of the n-th day (after satisfying the demands
for that day) is s or more, (here s ≥ 0 is a fixed integer) the store manager does
nothing. If it is less than s, the manager orders enough to bring the inventory at the
beginning of the next day up to S. Here S ≥ s is another fixed integer. Assume the
delivery to the store is instantaneous. Let Xn be the number of items in the inventory
in the store at the beginning of the n-th day, before satisfying that day’s demand, but
after the inventory is replenished. Show that {Xn , n ≥ 0} is a DTMC, and compute
its transition probabilities.

2.24 A machine requires a particular component in functioning order in order to

operate. This component is provided by two vendors. The maintenance policy is
as follows: whenever the component fails, it is instantaneously replaced by a new
component from vendor i with probability vi . (v1 > 0, v2 > 0, v1 + v2 = 1.)
The lifetimes of successive components from vendor i are iid random variables with
common distribution
αik = P(Life Time = k), k = 1, 2, 3, ...
with
∞
X
αik = 1, (i = 1, 2).
k=1
Let Xn be the age of the component in use on day n. (If there is a replacement at
time n, Xn = 0.) Let Yn = i if the component in place at time n (after replacement,
if any) is from vendor i. Show that {(Xn , Yn ), n ≥ 0} is a DTMC. Display its
transition diagram.

2.25 Consider the following simple model of the software development process: the
software is tested at times n = 0, 1, 2 · · · . If the software has k bugs, the test at time
n will reveal a bug with probability βk independent of the history. (β0 = 0.) If a bug
is revealed, the software is updated so that the bug is fixed. However, in the process
of fixing the bug, additional i bugs are introduced with probability αi (i = 0, 1, 2)
independent of the history. Let Xn be the number of bugs in the software just before
it is tested at time n. Show that {Xn , n ≥ 0} is a DTMC and display its transition
diagram.
48 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
2.26 Consider a closed society of N individuals. At time 0 one of these N indi-
viduals hears a rumor (from outside the society). At time 1 he tells it to one of the
remaining N − 1 individuals, chosen uniformly at random. At each time n, every
individual who has heard the rumor (and has not stopped spreading it) picks a person
from the remaining N − 1 individuals at random and spreads the rumor. If he picks
a person who has already heard the rumor, he stops spreading the rumor any more;
else, he continues. If k ≥ 2 persons tell the rumor to the same person who has not
heard the rumor before, all k + 1 will continue to spread the rumor in the next period.
Model this rumor spreading phenomenon as a DTMC.

2.27 Consider the following genetic experiment. Initially we cross an rr individual

with a dr individual, thus producing the next generation individual. We generate the
(n + 1)-st generation individual by crossing the n-th generation individual with a
dr individual. Let Xn be the genotype of the n-th generation individual. What is the
transition probability matrix of the DTMC {Xn , n ≥ 0}?

2.28 Redo the above problem by assuming that we begin with an rr individual but
always cross with a dd individual.

2.29 An electronic chain letter scheme works as follows. The initiator emails K per-
sons exhorting each to email it to K of their own friends. It mentions that complying
with the request will bring mighty good fortunes, while ignoring the request would
bring dire supernatural consequences. Suppose a recipient complies with the request
with probability α and ignores it with probability 1−α, independently of other recip-
ients. Assume the population is large enough so that this process continues forever.
Show that we can model this situation by a branching process with X0 = 20.

2.30 Slotted ALOHA is a protocol used to transmit packets by radio communica-

tions. Under this protocol time is slotted and each user can transmit its message at
the beginning of the n-th slot (n = 0, 1, 2, · · ·). Each message takes exactly one time
slot to transmit. When a new user arrives to the system, it transmits its message at
the beginning of the next time slot. If only one user transmits during a time slot, the
message is received by its intended recipient successfully. If no one transmits during
a slot, that slot is wasted. If two or more user transmit during a slot, a collision occurs
and the messages are garbled, and have to be retransmitted. If a user has experienced
a collision while transmitting its message, it is said to be backlogged. A backlogged
user attempts retransmission at the beginning of the next time slot with probability
p, independent of its history. All users behave independently of each other. Once a
user’s message is transmitted successfully, the user leaves the system. Let Yn be the
number of new user that arrive during the n-th time slot. Assume that {Yn , n ≥ 0} is
a sequence of iid random variables with common pmf
αk = P(Yn = k), k = 0, 1, 2, · · · .
Let Xn be the number of backlogged users at time n (before any transmissions have
occurred). Show that {Xn , n ≥ 0} is a DTMC and compute its transmission proba-
bilities.
COMPUTATIONAL EXERCISES 49
2.31 Under TDM (time division multiplexing) protocol, a user is allowed to trans-
mit packets one at a time at times n = 0, 1, 2, · · · . If the user has no packets to
transmit at time n, he must wait until time n + 1 for the next opportunity to transmit,
even though new packets may arrive between time n and n + 1. Let Xn be the num-
ber of packets before transmission is completed at time n and before any arrivals, say
Yn , between time n and n + 1. Assume that {Yn , n ≥ 0} is a sequence of iid random
variables with common pmf
αk = P(Yn = k), k = 0, 1, 2, · · · .
Show that {Xn , n ≥ 0} is a DTMC and compute its transition probabilities.

2.32 A manufacturing setup consists of two distinct machines, each producing one
component per hour. Each component is tested instantly and is identified as defective
or non-defective. Let αi be the probability that a component produced by machine
i is non-defective, i = 1, 2. The defective components are discarded and the non-
defective components are stored in two separate bins, one for each machine. When
a component is present in each bin, the two are instantly assembled together and
shipped out. Bin i can hold at most Bi components, i = 1, 2. (Here B1 and B2 are
fixed positive integers.) When a bin is full the corresponding machine is turned off.
It is turned on again when the bin has space for at least one component. Assume that
successive components are independent. Model this system by a DTMC.

2.33 Consider the following variation of Modeling Exercise 2.1: We replace the
light bulb upon failure, or upon reaching age K, where K > 0 is a fixed integer.
Assume that replacement occurs before failure if there is a tie. Let Xn be as in Mod-
eling Exercise 2.1. Show that {Xn , n ≥ 0} is a DTMC and compute its transition
probability matrix.

2.8 Computational Exercises

2.1 Redo Example 2.21 for the following initial conditions

(a). X0 = 8, (b). X0 = 5, (c). X0 = 3.

2.2 Let {Xn , n ≥ 0} be a DTMC with state-space {1, 2, 3, 4, 5} and the following
transition probability matrix:
 
0.1 0.0 0.2 0.3 0.4
 0.0 0.6 0.0 0.4 0.0 
 
 0.2 0.0 0.0 0.4 0.4  .
 
 0.0 0.4 0.0 0.5 0.1 
0.6 0.0 0.3 0.1 0.0
Suppose the initial distribution is a = [0.5, 0, 0, 0, 0.5]. Compute the following:
(a) The pmf of X2 ,
(b) P(X2 = 2, X4 = 5),
50 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
(c) P(X7 = 3|X3 = 4),
(d) P(X1 ∈ {1, 2, 3}, X2 ∈ {4, 5}).

2.3 Prove the result in Example 2.18 on page 33 by using (a) induction, and (b) the
method of diagonalization of Theorem 2.7 on page 38.

2.4 Compute the expected fraction of the patients who get drug 1 among the first
n patients in the clinical trial of Example 2.5 on page 14. Hint: Use the results of
Example 2.24 on page 37.

2.5 Suppose a market consists of k independent customers who switch between the
three brands A, B, and C according to the DTMC of Example 2.14. Suppose in week
0 brand A is chosen with probability 0.3 and brand B is chosen with probability 0.3.
Compute the probability distribution of the number of customers who choose brand
B in week 3.

2.6 Consider the two machine workshop of Example 2.7 on page 15. Suppose each
machine produces a revenue of $r per day when it is up, and no revenues when it
is down. Compute the total expected revenue over the first n days assuming both
machines are up initially. Hint: Use independence and the results of Example 2.24
on page 37.

2.7 Consider the binomial model of stock fluctuation as described in Section 2.3.4
on page 27. Suppose X0 = 1. Compute E(Xn ) and Var(Xn) for n ≥ 0.

2.8 Let {Xn , n ≥ 0} be a DTMC on state-space {1, 2, 3, 4} and transition proba-

bility matrix given below:
 
0.4 0.3 0.2 0.1
 0.5 0.0 0.0 0.5 
 0.5 0.0 0.0 0.5  .
P = 

0.1 0.2 0.3 0.4

Suppose X0 = 1 with probability 1. Compute
(a) P(X2 = 4),
(b) P(X1 = 2, X2 = 4, X3 = 1),
(c) P(X7 = 4|X5 = 2),
(d) E(X3 ).

2.9 Consider the Modeling Exercise 2.23 with the following parameters: s =
10, S = 20, α0 = 0.1, α1 = 0.2, α2 = 0.3, and α3 = 0.4. Suppose
X0 = 20 with probability 1. Compute E(Xn ) for n = 1, 2, · · · , 10.

2.10 Consider the discrete time queue of Example 2.12 on page 17. Compute
P(X2 = 0|X0 = 0).

2.11 Derive the n-step transition probabilities in Equation 2.25 on page 33.
COMPUTATIONAL EXERCISES 51
2.12 Consider the weather forecasting model of Modeling Exercise 2.5. What is the
probability distribution of the length of the rainy spell predicted by this model? Do
the same for the length of the sunny spell.

2.13 Consider the Modeling Exercise 2.21 with a bus of capacity 20, and Yn ∼
P (10), Poisson with mean 10, and p = .4. Compute E(Xn |X0 = 0) for n =
0, 1, · · · , 20.

2.14 Let P be a m × m transition probability given below

 
α1 α2 α3 · · · αm
 αm α1 α2 · · · αm−1 
 
P =  αm−1 αm α1 · · · αm−2 .
 
 .. .. .. .. .. 
 . . . . . 
α2 α3 α4 ··· α1
√
Such a matrix is called the circulant matrix. Let ι = −1, and
ι2π
ek = exp( k), 1 ≤ k ≤ m
m
be the m-th roots of unity. Define
m
X
λk = αi ei−1
k , 1 ≤ k ≤ m,
i=1

ι2π
xjk = exp( kj), 1 ≤ j, k ≤ m,
m
1 ι2π
yjk = exp(− kj), 1 ≤ j, k ≤ m.
m m
Show that λk is the k-th eigenvalue of P , with right eigenvector [x1k x2k · · · xmk ]′
and the left eigenvector [yk1 yk2 · · · ykm ]. Hence, using D = diag(λ1 λ2 · · · λm ),
X = [xjk ] and Y = [ykj ], show that
P = XDY.
Thus the powers of a circulant transition probability matrix can be written down
analytically.

2.15 Let {Xn , n ≥ 0} be a success runs DTMC on {0, 1, 2, · · ·} with

pi,0 = q = 1 − pi,i+1 , i ≥ 0.
Show that pn00 = q for all n ≥ 1.

2.16 Four points are arranged in a circle in a clockwise order. A particle moves on
these m points by taking a clockwise step with probability p and a counterclockwise
step with probability q = 1 − p, at time n = 0, 1, 2, · · · . Let Xn be the position
of the particle at time n. Thus {Xn , n ≥ 0} is a DTMC on {1, 2, 3, 4}. Display its
transition probability matrix P . Compute P n by using the diagonalization method of
Section 2.6. Hint: Use the results of Computational Exercise 2.14 above.
52 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
2.17 Consider the urn model of Example 2.13. Show that the transition probability
matrix has the following eigenvalues: λk = k(k + 1)/N 2 − 1/N, 0 ≤ k ≤ N .
Show that the transition probability matrix is diagonalizable. In general, finding the
eigenvalues is the hard part. Finding the corresponding eigenvectors is the easy part.
(n)
2.18 Compute pij for the success runs Markov chain of Computational Exer-
cise 2.15.

2.19 The transition probability matrix of a three-state DTMC is as given below:

 
0.3 0.4 0.3
P =  0.4 0.5 0.1  .
0.6 0.2 0.2
(n)
Find p11 using the method of generating functions.

2.20 Consider a DTMC on S = {0, 1, 2, · · ·} with the following transition proba-

bilities:  1
i+1 for 0 ≤ j ≤ i
pij =
0 otherwise.
Show that
∞  z Qi k+1
X (n) if 0 ≤ j < i
pij (z) = pij z n = i+1
i+1
k=j k+1−z
i+1−z if i = j.
n=0

2.21 Use the method of diagonalization to compute the probability distribution of

the genotype of the n-th individual in the model described in Modeling Exercise 2.28.

2.22 Use the method of diagonalization to compute the probability distribution of

the genotype of the n-th individual in the model described in Modeling Exercise 2.27.

2.23 Compute the mean of variance of the number of individuals in the n-th gen-
eration in the branching process of Section 2.3.3 on page 25, assuming the initial
generation consists of i individuals. Hence compute the mean and variance of the
number of letters in the n-th generation in Modeling Exercise 2.29. Hint: Imagine i
independent branching processes, each initiated by a single individual, and use the
results of Example 2.22.

2.24 Use the method of diagonalization to compute the probability distribution of

the gambler A’s fortune after n games in the gambler’s ruin model of Example 2.11,
with N = 3.

2.25 Consider the Wright-Fisher model described on page 24. Let i ∈

{0, 1, · · · , N } be a given integer and suppose X0 = i with probability 1. Compute
E(Xn ) and Var(Xn ) for n ≥ 0.

2.26 Consider the Moran model described on page 25. Let i ∈ {0, 1, · · · , N } be a
given integer and suppose X0 = i with probability 1. Compute E(Xn ) and Var(Xn )
for n ≥ 0.
CONCEPTUAL EXERCISES 53
2.9 Conceptual Exercises

2.1 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC on S = {0, 1, 2, · · ·}

with transition probability matrix P . Show that
P(Xn+2 ∈ B, Xn+1 ∈ A|Xn = i, Xn−1 , · · · , X0 ) = P(X2 ∈ B, X1 ∈ A|X0 = i),
where A and B are subsets of S.

2.2 Suppose {Xn , n ≥ 0} and {Yn , n ≥ 0} are two independent DTMCs with
state-space S = {0, 1, 2, · · ·}. Prove or give a counterexample to the following state-
ments:
(a) {Xn + Yn , n ≥ 0} a DTMC.
(b) {(Xn , Yn ), n ≥ 0} is a DTMC.

2.3 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC on S = {0, 1, 2, · · ·}.

Prove or give a counterexample to the following statements:
(a) P(Xn+1 = j|Xn ∈ A, Xn−1 , · · · , X0 ) = P(Xn+1 = j|Xn ∈ A), where A ⊂ S
has more than one element.
(b) P(Xn = j0 |Xn+1 = j1 , Xn+2 = j2 , · · · , Xn+k = jk ) = P(Xn = j0 |Xn+1 =
j1 ), where j0 , j1 , · · · , jk ∈ S and n ≥ 0.
(c) P(Xn = j0 , Xn+1 = j1 , Xn+2 = j2 , · · · , Xn+k = jk ) = P(X0 = j0 , X1 =
j1 , X2 = j2 , · · · , Xk = jk ).

2.4 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC on S = {0, 1, 2, · · ·}.

Prove or give a counterexample to the following statements:
(a) Let bj = P(Xk = j), for j ∈ S, and a given k > 0. Then {Xn , n ≥ 0} is
completely described by [bj , j ∈ S] and the transition probability matrix.
(b) Let f : S → S be any function. {f (Xn ), n ≥ 0} is a DTMC.

2.5 Suppose {Xn , n ≥ 0} and {Yn , n ≥ 0} are two independent DTMCs with
state-space S = {0, 1, 2, · · ·}. Let {Zn , n ≥ 0} be a sequence iid Ber(p) random
variables. Define 
Xn if Zn = 0
Wn =
Yn if Zn = 1.
Is {Wn , n ≥ 0} a DTMC (not necessarily time homogeneous)?

2.6 Let {Yn , n ≥ 0} be a sequence of iid random variables with common pmf
αk = P(Yn = k), k = 0, 1, 2, · · · .
We say that Yn is a record if Yn > Yr , 0 ≤ r ≤ n − 1. Let X0 = Y0 , and Xn be the
value of the n-th record, n ≥ 1. Show that {Xn , n ≥ 0} is a DTMC and compute its
transition probability matrix.

2.7 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC on S = {0, 1, 2, · · ·} and

transition probability matrix P . Define
N = min{n > 0 : Xn 6= X0 }.
54 DISCRETE-TIME MARKOV CHAINS: TRANSIENT BEHAVIOR
Thus N is the first time the DTMC leaves the initial state. Compute
P(N = k|X0 = i), k ≥ 1.

2.8 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC on S = {0, 1, 2, · · ·} and

transition probability matrix P . Let A be a strict non-empty subset of S. Assume
X0 ∈ A with probability 1. Define N0 = 0 and
Nr+1 = min{n > Nr : Xn ∈ A}, r ≥ 0.
Thus Nr is the time of the r-th visit by the DTMC to the set A. Define
Yr = XNr , r ≥ 0.
Is {Yr , r ≥ 0} a DTMC? Prove or give a counterexample.

2.9 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC with the following prop-
erty: there is a j ∈ S such that pij = p for all i ∈ S. Show that P(Xn = j) = p for
all n ≥ 1, no matter what the initial distribution is.

2.10 Suppose {Xn , n ≥ 0} is a time non-homogeneous DTMC with the following

transition probabilities:

aij if n is even
P(Xn+1 = j|Xn = i) =
bij if n is odd,
where A = [aij ] and B = [bij ] are two given stochastic matrices. Construct a time
homogeneous DTMC {Yn , n ≥ 0} that is equivalent to {Xn , n ≥ 0}, i.e., a sample
path of {Xn , n ≥ 0} uniquely determines that of {Yn , n ≥ 0} and vice-versa.

2.11 Let {Xn , n ≥ 0} be a simple random walk of Example 2.19. Show that
{|Xn |, n ≥ 0} is a DTMC. Compute its transition probability matrix.

2.12 Suppose {Xn , n ≥ 0} is a time homogeneous DTMC on S = {0, 1, 2, · · ·}

and transition probability matrix P . Let f : S → {1, 2, · · · , M } be a given on-to
function, that is, f −1 (i) is non-empty for all 1 ≤ i ≤ M . Give the necessary and
sufficient condition under which {f (Xn ), n ≥ 0} is a DTMC.
 CHAPTER 3

Discrete-Time Markov Chains: First

Passage Times

A frequent flyer business traveler is concerned about the risk of encountering a ter-
rorist bomb on one of his flights. He consults his statistician friend to get an estimate
of the risk. After studying the data the statistician estimates that the probability of
finding a bomb on a random flight is one in a thousand. Alarmed by such a large risk,
the businessman asks his friend if there is any way to reduce the risk. The statistician
offers, “Carry a bomb with you on the plane, since the probability of two bombs on
the same flight is one in a million.”

3.1 Definitions

Let {Xn , n ≥ 0} be a DTMC on S = {0, 1, 2, · · ·}, with transition probability

matrix P , and initial distribution a. Let
T = min{n ≥ 0 : Xn = 0}. (3.1)
The random variable T is called the first passage time into state 0, since T is the first
time the DTMC “passes into” state 0. Although we shall specifically concentrate on
this first passage time, the same techniques can be used to study the first passage time
into any set A ⊂ S. (See Conceptual Exercises 3.1 and 3.2.) In this chapter we shall
study the following aspects of the first passage time T :

(1) Complementary cumulative distribution function of T : v(n) = P(T > n), n ≥

0,
(2) Probability of eventually visiting state 0: u = P(T < ∞),
(3) Moments of T : m(k) = E(T k ), k ≥ 1,
(4) Generating function of T : φ(z) = E(z T ).

There are two reasons for studying the first passage times. First, they appear natu-
rally in applications when we are interested in the time until a given event occurs in a
stochastic system modeled by a DTMC. Second, we shall see in the next chapter that

55
56 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
the quantities u (probability of eventually visiting a state) and m(1) (the expected
time to visit a state) play an important part in the study of the limiting behavior of
the DTMC. Thus the study of the first passage time has practical as well as theoretical
motivation.
Next we introduce the conditional quantities for i ∈ S:
vi (n) = P(T > n|X0 = i),
ui = P(T < ∞|X0 = i),
mi (k) = E(T k |X0 = i),
φi (z) = E(z T |X0 = i).

Using the initial distribution we see that

X
v(n) = ai vi (n),
i∈S
X
u = ai u i ,
i∈S
X
m(k) = ai mi (k),
i∈S
X
φ(z) = ai φi (z).
i∈S

Thus we can compute the unconditional quantities from the conditional ones. We
develop a method called the first-step analysis to compute the conditional quantities.
The method involves computing the conditional quantities by further conditioning
on the value of X1 (i.e., the first step), and then using time-homogeneity and Markov
property to derive a set of linear equations for the conditional quantities. These equa-
tions can then be solved numerically or algebraically depending on the problem at
hand.
We shall see later that sometimes we need to study an alternate first passage time
as defined below:
T̃ = min{n > 0 : Xn = 0}. (3.2)

We leave it to the reader to establish the relationship between between T and T̂ .

(See Conceptual Exercises 3.3 and 3.4.)

3.2 Cumulative Distribution Function of T

The following theorem illustrates how the first-step analysis produces recursive
method of computing the cumulative distribution of T . We first introduce the fol-
lowing matrix notation:
v(n) = [v1 (n), v2 (n), · · ·]′ , n ≥ 0
CUMULATIVE DISTRIBUTION FUNCTION OF T 57
B = [pij : i, j ≥ 1]. (3.3)
Thus B is a submatrix of P obtained by deleting the row and column corresponding
to the state 0.

Theorem 3.1
v(n) = B n e, n ≥ 0, (3.4)
where e is column vector of all ones.

Proof: We prove the result by using the first-step analysis. For n ≥ 1 and i ≥ 1 we
have
vi (n) = P(T > n|X0 = i)
X∞
= P(T > n|X1 = j, X0 = i)P(X1 = j|X0 = i)
j=0
X∞
= pij P(T > n|X1 = j, X0 = i)
j=0
∞
X
= pi0 P(T > n|X1 = 0, X0 = i) + pij P(T > n|X1 = j, X0 = i)
j=1
∞
X
= pij P(T > n|X1 = j)
j=1
X∞
= pij P(T > n − 1|X0 = j)
j=1
X∞
= pij vj (n − 1).
j=1

Here we have used the fact that X1 = 0 implies that T = 1 and hence P(T >
n|X1 = 0, X0 = i) = 0, and the Markov property and time homogeneity implies
that the probability of T > n given X1 = j, X0 = i is the same as the probability
that T > n − 1 given X0 = j. Writing the final equation in matrix form yields
v(n) = Bv(n − 1), n ≥ 1. (3.5)
Solving this equation recursively yields
v(n) = B n v(0).
Finally, X0 = i ≥ 1 implies that T ≥ 1. Hence vi (0) = 1. Thus
v(0) = e. (3.6)
This yields Equation 3.4. Note that it is valid for n = 0 as well, since B 0 = I, the
identity matrix, by definition.
58 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
Since we have studied the computation of matrix powers in Section 2.6, we have
an easy way of computing the complementary cdf of T . We illustrate with several
examples.

Example 3.1 Two State DTMC. Consider the two state DTMC of Example 2.3 on
page 13. The state space is {1, 2}. Let T be the first passage time to state 1. One can
use Theorem 3.1 with B = [β], or direct probabilistic reasoning, to see that
v2 (n) = P(T > n|X0 = 2) = β n , n ≥ 0.
Hence we get
P(T = n|X0 = 2) = v2 (n − 1) − v2 (n) = β n−1 (1 − β), n ≥ 1,
which shows that T is a geometric random variable with parameter 1 − β.

Example 3.2 Genotype Evolution. Consider the genotype evolution model on

page 24 involving a DTMC with state-space {1, 2, 3, 4, 5, 6} and transition proba-
bility matrix given below:
1 0 0 0 0 0
 
 1/4 1/2 0 1/4 0 0 
 
 0 0 0 1 0 0 
 
P =  1/16 1/4 1/8 1/4 1/4 1/16  .
 (3.7)
 
 0 0 0 1/4 1/2 1/4 
 

0 0 0 0 0 1

Let
T = min{n ≥ 0 : Xn = 1}
be the first passage time to state 1. Let B be the submatrix of P obtained by deleting
rows and columns corresponding to state 1. Then, using v(n) = [v2 (n), · · · , v6 (n)]′ ,
Theorem 3.1 yields
v(n) = B n e, n ≥ 0.
Direct numerical calculation yields, for example,
v(5) = B 5 e = [0.4133, 0.7373, 0.6921, 0.8977, 1]′ ,
and
lim v(n) = [0.25, 0.50, 0.50, 0.75, 1]′ .
n→∞
Thus T is a defective random variable, and the probability that the DTMC will never
visit state 1 starting from state 2 is .25, which is the same as saying that the proba-
bility of eventually visiting state 1 starting from state 2 is .75.

Example 3.3 Success Runs. Consider the success runs DTMC of Example 2.15 on
page 19 with transition probabilities
pi,0 = qi , pi,i+1 = pi , i = 0, 1, 2, · · · .
CUMULATIVE DISTRIBUTION FUNCTION OF T 59
Let T be the first passage time to state 0. Compute the complementary cdf of T
starting from state 1.
In this case it is easier to use the special structure of the DTMC. We have v1 (0) = 1
and, for n ≥ 1,
v1 (n) = P(T > n|X0 = 1)
= P(X1 = 2, X2 = 3, · · · , Xn = n + 1|X0 = 1)
= p1 p2 · · · pn ,
since the only way T > n starting from X0 = 1 is if the DTMC increases by one for
each of the next n steps. Since the pi ’s can take any value in [0, 1], it follows that we
can construct a success runs DTMC such that T has any pre-specified distribution on
{1, 2, 3, · · ·}.

Example 3.4 Coin Tosses. Suppose a coin is tossed repeatedly and independently.
The probability of head is p and the probability of tail is q = 1 − p on any given toss.
Compute the distribution of the number of tosses needed to get two heads in row.

Let {Xn , n ≥ 0} be a success runs DTMC with transition probabilities

pi,0 = q, pi,i+1 = p, i = 0, 1, 2, · · · .
One can think of Xn as the length of the current run of heads after the nth toss. Let
T = min{n ≥ 0 : Xn = 2}.
Then starting with X0 = 0, T is the number of tosses needed to obtain two heads in
a row. Let
v(n) = [v0 (n), v1 (n)], n ≥ 0,
and  
q p
B= .
q 0
Then we can see that
v(n) = B n e, n ≥ 0.
Using methods of eigenvalues of Section 2.6 we get
1 − λ2 n 1 − λ1 n
v0 (n) = λ1 − λ , n ≥ 0,
λ1 − λ2 λ1 − λ2 2
where
1 p 1 p
λ1 = (q + q 2 + 4pq), λ2 = (q − q 2 + 4pq).
2 2
We can obtain the required distribution as follows:
P(n tosses are needed to get two heads in a row)
= P(T = n|X0 = 0)
= v0 (n − 1) − v0 (n)
(1 − λ1 )(1 − λ2 ) n−1
= [λ1 − λn−1
2 ], n ≥ 1.
λ1 − λ2
60 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES

3.3 Absorption Probabilities

Since vi (n) is a monotone bounded sequence we know that

vi = lim vi (n)
n→∞

exists. The quantity

ui = 1 − vi = P(T < ∞|X0 = i)
is the probability that the DTMC eventually visits state 0 starting from state i, also
called the absorption probability into state 0 starting from state i when p00 = 1. In
this section we develop methods of computing ui or vi . The main result is given by
the next theorem.

Theorem 3.2 The vector v = limn→∞ v(n) is given by the largest solution to
v = Bv (3.8)
such that v ≤ e, where B is as defined in Equation 3.3, and e is a vector of all ones.

Proof: Equation 3.8 follows by letting n → ∞ on both sides of Equation 3.5 since
we know the limit exists. To show that it is the largest solution bounded above by
e, suppose w ≤ e is another solution to Equation 3.8. Then, from Equation 3.6, we
have
v(0) = e ≥ w.
As an induction hypothesis assume
v(k) ≥ w
for k = 0, 1, 2, · · · , n. Using the fact that w = Bw, and that B is a non-negative
matrix, we can use Equation 3.5 to get
v(n + 1) = Bv(n) ≥ Bw = w.
Thus
v(n) ≥ w
for all n ≥ 0. Thus letting n → ∞
v = lim v(n) ≥ w.
n→∞

This proves the theorem.

Notice that v = 0 is always a solution to Equation 3.8. Thus the largest solu-
tion is bound to be non-negative. What is more surprising is that a vector v that is
componentwise maximum exists at all! We can also derive Equation 3.8 by the first-
step analysis. However, that analysis will not show that the required solution is the
largest one bounded above by one. How does one find the largest solution in prac-
tice? The general method is to identify all the solutions, and then pick the one that
ABSORPTION PROBABILITIES 61
is the maximum. In practice, this is not as impractical as it sounds. We explain by
several examples below.

Example 3.5 Genotype Evolution. Suppose that the population in the genotype
evolution model of Example 3.2 initially consists of one dominant and one hybrid
individual. What is the probability that eventually the population will contain only
dominant individuals?
Let T be the first passage time as defined in Example 3.2. We are asked to compute
u2 = P(T < ∞|X0 = 2) = 1 − v2 . From Theorem 3.2 we see that
v = Bv
where v = [v2 , v3 , · · · , v6 ]′ and
 
1/2 0 1/4 0 0
 0 0 1 0 0 
 
B =  1/4 1/8 1/4 1/4 1/16 

. (3.9)
 0 0 1/4 1/2 1/4 
0 0 0 0 1
The above equation implies that v6 = v6 , thus there are an infinite number of solu-
tions to v = Bv. Since we are looking for the largest solution bounded above by 1,
we must choose v6 = 1. Notice that we could have concluded this by observing that
X0 = 6 implies that the DTMC can never visit state 1 and hence T = ∞. Thus we
must have u6 = 0 or v6 = 1. Once v6 is chosen to be 1, we see that v = Bv has a
unique solution given by
v = [0.25, 0.50, 0.50, 0.75, 1]′ .
Thus the required answer is given by u2 = 1 − v2 = 0.75.

Example 3.6 Gambler’s Ruin. Consider the Gambler’s ruin model of Exam-
ple 2.11 on page 17. The DTMC in that example is a simple random walk on
{0, 1, 2, · · · , N } with transition probabilities given by
pi,i+1 = p for 1 ≤ i ≤ N − 1,
pi,i−1 = q = 1 − p for 1 ≤ i ≤ N − 1,
p00 = pN N = 1.
Compute the probability that the DTMC eventually visits state 0 starting from state
i.
Let T = min{n ≥ 0 : Xn = 0}. We can write the Equation 3.8 as follows
vi = pvi+1 + qvi−1 , 1 ≤ i ≤ N − 1. (3.10)
The above equation can also be derived from first step analysis, or argued as follows:
If the DTMC starts in state i (1 ≤ i ≤ N − 1), it will be in state i + 1 with probability
p, and the probability of never visiting state 0 from then on is vi+1 ; and it will visit
state i − 1 with probability q and the probability of never visiting state 0 from then
on will be vi−1 . Hence we get Equation 3.10. To complete the argument we must
62 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
decide appropriate values for v0 and vN . Clearly, we must have v0 = 0, since once
the DTMC visits state 0, the probability of never visiting state 0 will be 0. We can
also argue that vN = 1, since once the DTMC visits state N it can never visit state
0. We also see that Equation 3.8 implies that vN = vN , and hence we set vN = 1 in
order to get the largest solution bounded above by 1.
Equation 3.10 is a difference equation with constant coefficients, and hence using
the standard methods of solving such equations (See Appendix I) we get
(
1−(q/p)i
1−(q/p)N
if q 6= p
vi = i
(3.11)
N if q = p.

The required probability is ui = 1 − vi .

Example 3.7 Success Runs. Consider the success runs DTMC of Example 3.3.
Compute the probability that the DTMC never visits state 0 starting form state i.
Using the derivation in Example 3.3, we get
vi (n) = pi pi+1 · · · pi+n−1 , i ≥ 1, n ≥ 1.
Hence
vi = lim pi pi+1 · · · pi+n−1 , i ≥ 1.
n→∞
The above limit is zero if pn = 0 for some n ≥ i. To avoid this triviality, assume that
pi > 0 for all i ≥ 1. Now it is known that
∞
X
lim p1 p2 · · · pn = 0 ⇔ qn = ∞,
n→∞
n=1

and
∞
X
lim p1 p2 · · · pn > 0 ⇔ qn < ∞.
n→∞
n=1
Hence we see that
∞
X
vi = 0, i.e., ui = 1 for all i ⇔ qn = ∞,
n=1

and
∞
X
vi > 0, i.e., ui < 1 for all i ⇔ qn < ∞.
n=1
The reader should try to obtain the same results by using Theorem 3.2.

Example 3.8 Simple Space Homogeneous Random Walk.

Consider a simple random walk on {0, 1, 2, · · ·} with an absorbing barrier at zero.
That is, we have
pi,i+1 = p for i ≥ 1,
pi,i−1 = q = 1 − p for i ≥ 1,
p00 = 1.
ABSORPTION PROBABILITIES 63
Let T be the first passage time into state 0. Equation 3.8 for this system, written in
scalar form, is as follows:
vi = qvi−1 + pvi+1 , i ≥ 1, (3.12)
with v0 = 0. This a difference equation with constant coefficients. The general solu-
tion is given by
 i
q
vi = α + β , i ≥ 0.
p
Using the condition v0 = 0 we get β = −α, and thus
 i !
q
vi = α 1 − , i ≥ 0. (3.13)
p
Now, if q ≥ p, the largest solution bounded above by one is obtained by setting
α = 0; if q < p, such a solution is obtained by setting α = 1. Thus we get
(
0 if q ≥ p
vi =  i
q
1− p if q < p.

If q ≥ p, the random walk has a drift towards zero (since it goes towards zero with
higher probability than away from it), and it makes intuitive sense that the random
walk will hit zero with probability 1, and hence vi = 0. On the other hand, if q < p,
there is a drift away from zero, and there is a positive probability that the random
walk will never visit zero. Note that without Theorem 3.2 we would not be able to
choose any particular value of α in Equation 3.13.

Example 3.9 General Simple Random Walk. Now consider the space-
nonhomogeneous simple random walk with the following transition probabilities:
pi,i+1 = pi for i ≥ 1,
pi,i−1 = qi = 1 − pi for i ≥ 1,
p00 = 1.
Let T be the first passage time into state 0. Equation 3.8 for this system, written in
scalar form, is as follows:
vi = qi vi−1 + pi vi+1 , i ≥ 1, (3.14)
with a single boundary condition v0 = 0. Now let
xi = vi − vi−1 , i ≥ 1.
Using pi + qi = 1, we can write Equation 3.14 as
(pi + qi )vi = qi vi−1 + pi vi+1
which yields
qi xi = pi xi+1 ,
64 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
or, assuming pi > 0 for all i ≥ 1,
qi
xi+1 = xi , i ≥ 1.
pi
Solving recursively, we get
xi+1 = αi x1 , i ≥ 0, (3.15)
where α0 = 1 and
q1 q2 · · · qi
αi = , i ≥ 1. (3.16)
p1 p2 · · · pi
Summing Equation 3.15 for i = 0 to j, and rearranging, we get
j
!
X
vj+1 = αi v1 .
i=0

Now, Theorem 3.2 says that v1 must be the largest value such that 0 ≤ vj ≤ 1 for all
j ≥ 1. Hence we must choose
P∞
P∞1
(
α
if i=0 αi < ∞,
v1 = i=0 i
P∞
0 if i=0 αi = ∞.

Thus the final solution is

 Pi−1
 Pj=0 αj P∞
∞ if j=0 αj < ∞,
vi = α
j=0 j P∞ (3.17)
0 if αj = ∞.

j=0

We can compute ui = 1 − vi from the above. PNotice that visiting state 0 is certain
from any state i (that is, ui = 1) if the sum αj diverges. The reader is urged to
verify that this consistent of the results of Example 3.8.

Example 3.10 Branching Processes. Consider the branching {Xn , n ≥ 0} de-

scribed in Section 2.3.3. Let αk be the probability that an individual produces k
offsprings, and let µ be the expected number of offsprings produced by an individ-
ual. In Example 2.22 on page 35 we saw that E(Xn ) = µn . Here we shall compute
the probability that the process eventually becomes extinct, i.e., it hits state 0.
Let T be the first passage time into the state 0, and let ui = P(T < ∞|X0 = i)
and u = u1 . We first argue that ui = ui , i ≥ 0. Note that a branching process start-
ing with i individuals can be thought of as i independent branching processes, each
starting with one individual. Thus a branching process starting with i individuals will
become extinct if and only if each of these i separate processes become extinct, the
probability of which is ui . Hence ui = ui . Note that this relationship is valid even
for i = 0. With this we apply the first step analysis:
u = P(T < ∞|X0 = 1)
X∞
= P(T < ∞|X0 = 1, X1 = j)P(X1 = j|X0 = 1)
j=0
ABSORPTION PROBABILITIES 65
∞
X
= αj P(T < ∞|X1 = j)
j=0
X∞
= αj P(T < ∞|X0 = j)
j=0
X∞
= αj uj .
j=0

Now let
∞
X
ψ(u) = αj uj (3.18)
j=0
be the generating function of the offspring size. The absorption probability u satisfies
u = ψ(u). (3.19)
′′
Since ψ (u) ≥ 0 for all u ≥ 0, ψ is a convex function of u ∈ [0, 1]. Also, ψ(0) = α0
and ψ(1) = 1. Clearly, if α0 = 0, each individual produces at least one offspring,
and hence the branching process never goes extinct, i.e., u = 0. Hence assume that
α0 > 0. Figures 3.1 and 3.2 show two possible cases that can arise.
The situation in Figure 3.1 arises if
∞
X
′
ψ (1) = iαi = µ > 1,
i=0

while that in Figure 3.2 arises if µ ≤ 1. In the first case there is a unique value of

ψ(u)

a0

u
0 1

Figure 3.1 Case1: µ > 1.

u ∈ (0, 1) satisfying ψ(u) = u, in the second case the only value of u ∈ [0, 1] that
satisfies u = ψ(u) is u = 1. Thus we conclude that extinction is certain if µ ≤ 1,
66 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES

ψ(u)

a0

u
0 1

Figure 3.2 Case1: µ ≤ 1.

while there is a positive probability of the branching process growing without bounds
if µ > 1. This is intuitive, except for the critical case of µ = 1. In this case our pre-
vious analysis shows that E(Xn ) = 1 for all n ≥ 0, however the branching process
becomes extinct. This seemingly inconsistent result is a manifestation of the fact that
convergence in distribution or with probability one does not imply convergence of
the means.

We will encounter Equation 3.19 multiple times during the study of DTMCs. Hence it
is important to know how to solve it. Except in very special cases (See Computational
Exercise 3.34) we need to resort to numerical methods to solve this equation. We
describe one such method here.
Let ψ be as defined in Equation 3.18, and define
ρ0 = 0, ρn+1 = ψ(ρn ), n ≥ 0.
We shall show that if µ > 1, ρn → u, the desired solution in (0,1) to Equation 3.19.
We have
ρ1 = a0 > 0 = ρ0 .
Now, since ψ is an increasing function, it is clear that
ρn+1 ≥ ρn , n ≥ 0.
Also, ρn ≤ 1 for all n ≥ 0. Hence {ρn , n ≥ 0} is a bounded monotone increasing
sequence. Hence it has a limit u, and this u satisfies Equation 3.19. The sequence
{ρn } is geometrically illustrated in Figure 3.3.
Similarly, if µ > 1, one can use the bisection method to find a ρ∗0 < 1 such that
ψ(ρ∗0 ) < ρ∗0 . Then recursively define
ρ∗n+1 = ψ(ρ∗n ), n ≥ 0.
ABSORPTION PROBABILITIES 67

ψ(u)

a0

u
ρ0 ρ1 ρ2 ρ

Figure 3.3 The iteration ρn+1 = ψ(ρn ).

One can show that

ρ∗n+1 ≤ ρ∗n , n ≥ 0
so that the sequence {ρ∗n } monotonically decreases to u. Using these two monotonic
sequences we see that if, for a given ǫ > 0, we stop the iterations when |ρ∗n −ρn | < ǫ,
then we are guaranteed that 0 ≤ u − ρn < ǫ. Thus we have simple numerical method
of solving Equation 3.19 to any degree of accuracy.

Example 3.11 Production-Inventory System: Batch Production. Consider the

DTMC of Example 2.16 on page 19. Compute ui , the probability that the DTMC
eventually visits state 0 starting from state i.
The DTMC has state-space{0, 1, 2, · · ·} and transition probability matrix as given
in Equation 2.10. First step analysis yields
∞
X
ui = αj ui+j−1 , i ≥ 1, (3.20)
j=0

with the boundary condition u0 = 1. Since the above equation is a difference equa-
tion with constant coefficients, we try a geometric solution ui = ui , i ≥ 0. Substi-
tuting in the above equation and canceling ui−1 from both sides we get
∞
X
u= αj uj , i ≥ 1.
j=0

Using the notation ψ(u) of Equation

P 3.18 we see that this is the same as Equa-
tion 3.19. Using the notation µ = iαi and the results of Example 3.10, we see that
u = 1 if and only if µ ≤ 1, and u < 1 if and only if µ > 1.
68 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
Example 3.12 Production-Inventory System: Batch Demands. Consider the
DTMC of Example 2.17 on page 20. Compute v1 , the probability that the DTMC
never visits state 0 starting from state i.
The DTMC has state-space{0, 1, 2, · · ·} and transition probability matrix as given
in Equation 2.12. First step analysis yields
i
X
vi = αj vi−j+1 , i ≥ 1. (3.21)
j=0

It is intuitively clear that vi is an increasing function of i. It can be formally proved

by induction, see Conceptual Exercise 3.12. There is no closed form expression for
vi in terms of the α’s. Hence we compute the generating function of the vi ’s defined
as
X∞
φ(z) = z i vi
i=1
in terms of the generating function of the α’s as defined below
∞
X
ψ(z) = z i αi .
i=0
i
Multiplying Equation 3.21 by z on both sides and summing up from i = 1 to ∞ and
rearranging, we get
∞
X
φ(z) = z i vi
i=1
∞
X i
X
= zi αj vi−j+1
i=1 j=0
ψ(z)
= −a0 v1 + φ(z).
z
This yields
a0 z
φ(z) = v1 .
ψ(z) − z
Since {vi , i ≥ 1} is a monotone bounded sequence, it has a limit. Using the proper-
ties of the generating functions (see Appendix D) we get
lim vi = lim (1 − z)φ(z)
i→∞ z→1
(1 − z)a0 z
= lim v1 .
z→1 ψ(z) − z
Using L’Hopital’s rule we get
(1 − z)a0 z a0
lim v1 = v1 .
z→1 ψ(z) − z 1 − ψ ′ (1)
EXPECTATION OF T 69
Since we want to find the largest solution to Equation 3.21 that is bounded above by
1, the above limit must be 1. Hence, using
∞
X
ψ ′ (1) = kαk = µ,
k=0

we see that we must choose

 1−µ
a0 if µ < 1, a0 > 0, a0 + a1 < 1,
v1 =
0 if µ ≥ 1.
The conditions µ < 1, a0 > 0, a0 + a1 < 1 are necessary and sufficient to ensure
that 0 < 1−µ
a0 < 1.

3.4 Expectation of T

Let {Xn , n ≥ 0} and T be as defined in Section 3.1. In this section we assume that
ui = P(T < ∞|X0 = i) = 1 for all i ≥ 1 and compute
mi = E(T |X0 = i), i ≥ 1.
Notice that ui < 1 would imply that mi = ∞. Let
m = [m1 , m2 , m3 , · · ·]′ ,
and B be as defined in Equation 3.3. The main result is given by the following The-
orem.

Theorem 3.3 Suppose ui = 1 for all i ≥ 1. Then m is given by the smallest non-
negative solution to
m = e + Bm, (3.22)
where e is a column vector of all ones.

Proof: We shall follow the first-step analysis:

mi = E(T |X0 = i)
X∞
= E(T |X0 = i, X1 = j)P(X1 = j|X0 = i)
j=0
∞
X
= pij E(T |X0 = i, X1 = j).
j=0

Now, using
E(T |X0 = i, X1 = 0) = 1,
and
E(T |X0 = i, X1 = j) = 1 + E(T |X0 = j) = 1 + mj , j ≥ 1
70 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
and simplifying, we get
∞
X
mi = 1 + pij mj , i ≥ 1. (3.23)
j=1

This yields Equation 3.22 in matrix form. The proof that it is the smallest non-
negative solution follows along the same lines as the proof of Theorem 3.2.

Notice that Equation 3.23 can be intuitively understood as follows: If the DTMC
starts in state i ≥ 1, it has to take at least one step before hitting state 0. If after the
first step it is in state 0, it does not need anymore steps to reach state zero. On the
other hand, if it is in state j ≥ 1, it will need an additional mj steps on average to
reach state zero. Weighing all these possibilities we get Equation 3.23. We illustrate
with several examples below.

Example 3.13 Genotype Evolution. Consider the six-state DTMC in the Geno-
type Evolution model of Example 3.2 with transition probability matrix given in
Equation 3.7. Suppose initially the population consists of one dominant and one hy-
brid individual. Compute the expected time until the population becomes entirely
dominant or entirely recessive.
Let
T = min{n ≥ 0 : Xn = 1 or 6}.
We are asked to compute m2 = E(T |X0 = 2). Since we know that eventually the
entire population will be either dominant or recessive, ui = 1 for all i. Now let
m = [m2 , m3 , m3 , m4 ]′ , and B be the 4 by 4 submatrix of P obtained by deleting
rows and columns corresponding to states 1 and 6. Then, from Theorem 3.3 we get
m = e + Bm.
Solving numerically, we get
5 2 2 5
m = [4 , 6 , 5 , 4 ]′ .
6 3 3 6
Thus, on the average, the population genotype gets fixed in 4 65 steps if it starts with
one dominant and one hybrid individual.

The next example shows that some questions can be answered by using the first
passage times even if the problem does not explicitly involve a DTMC!

Example 3.14 Coin Tossing. Suppose a coin tossed repeatedly in an independent

fashion until k consecutive heads are obtained. Compute the expected number of
tosses needed if probability of obtaining a head on any toss is p.
Consider the DTMC {Xn , n ≥ 0} defined in Example 3.4. Since Xn can be
thought of as the legth of the current run of heads, it is clear that the answer is given
by m0 = E(T |X0 = 0), where
T = min{n ≥ 0 : Xn = k}.
EXPECTATION OF T 71
Using the first-step analysis we get
mi = 1 + qm0 + pmi+1 , 0 ≤ i ≤ k − 1
with mk = 0. Solving recursively we get
 
1
mi = + m0 (1 − pk−i ), 0 ≤ i ≤ k − 1.
q
Setting i = 0 we get an equation for m0 , which can be solved to get
 
1 1
m0 = − 1 .
q pk
The reader should verify that this produces the correct result if we set k = 1.

Example 3.15 General Simple Random Walk. Compute the expected time to hit
state 0 starting from state i ≥ 1 in the general simple random walk of Example 3.9.
Let αi be as given Equation 3.16. We shall assume that
∞
X
αr = ∞,
r=1

so that, from Example 3.9, ui = 1 for all i. The first-step analysis yields
mi = 1 + qi mi−1 + pi mi+1 , i ≥ 1 (3.24)
with boundary condition m0 = 0. Now let
xi = mi − mi−1 , i ≥ 1.
Then Equation 3.24 can be written as
qi xi = 1 + pi xi+1 , i ≥ 1.
Solving recursively, we get
xi+1 = −αi bi + αi m1 , i ≥ 0 (3.25)
where
i
X 1
bi = ,
j=1
pj αj
with b0 = 0. Summing Equation 3.25 we get
mi+1 = xi+1 + xi + · · · + x1 + x0
Xi Xi
= − αj bj + m1 αj , i ≥ 0.
j=1 j=0

Since m1 is the smallest non-negative solution we must have

Pi i Pi
j=1 αj bj 1 r=j αr
X
m1 ≥ P i = Pi , i ≥ 1. (3.26)
j=0 αj
p α
j=1 j j r=0 αr
72 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
It can be shown that the right hand
P side of the above equation is an increasing
P∞ function
of i, and its limit as i → ∞ is ∞ j=1 1/pj αj , since we have assumed r=1 αr = ∞.
Hence the smallest m1 to satisfy Equation 3.26 is given by
∞
X 1
m1 = .
j=1
pj αj

Note that m1 may be finite or infinite. Using the above expression for m1 we get
 
i−1 ∞
X X 1 
mi = αk  .
pj αj
k=0 j=k+1

Example 3.16 Production-Inventory System: Batch Production. Consider the

DTMC of Example 2.16 on page 19. Compute mi , the expected time when the
DTMC reaches state 0 starting from state i.
Let µ be the mean production batch size as defined in Example 3.11. Assume that
µ ≤ 1, so that, from the results of Example 3.11, the DTMC reaches state 0 with
probability 1 starting from any state. The first step analysis yields
∞
X
mi = 1 + αj mi+j−1 , i ≥ 1,
j=0

with the boundary condition m0 = 0. It can be shown that the solution to this set
of equations is given by mi = im for some positive m. Substituting in the above
equation we get
∞
X
im = 1 + αj (i + j − 1)m, i ≥ 1.
j=0
Simplifying, we get
(1 − µ)m = 1.
Since mi is the smallest non-negative solution, we must have
 1
1−µ if µ < 1
m=
∞ if µ ≥ 1.
Hence  i
1−µ if µ < 1
mi =
∞ if µ ≥ 1.

Example 3.17 Production-Inventory System: Batch Demands. Consider the

DTMC of Example 2.17 on page 20. Compute mi , the expected time when the
DTMC reaches state 0 starting from state i.
Let µ be the mean demand batch size as defined in Example 3.12. Assume that
µ ≥ 1, so that, from the results of Example 3.12, the DTMC reaches state 0 with
EXPECTATION OF T 73
probability 1 starting from any state. First step analysis yields
i
X
mi = 1 + αj mi−j+1 , i ≥ 1. (3.27)
j=0

It is intuitively clear that mi is an increasing function of i. There is no closed form

expression for mi in terms of the α’s. Hence we compute the generating function of
the mi ’s defined as
X ∞
φ(z) = z i mi
i=1
in terms of the generating function of the α’s as defined below
∞
X
ψ(z) = z i αi .
i=0

Multiplying Equation 3.27 by z i on both sides and summing up from i = 1 to ∞ and

rearranging, we get
∞
X
φ(z) = z i mi
i=1
 
∞
X i
X
= z i 1 + αj mi−j+1 
i=1 j=0

z ψ(z)
= − a0 m 1 + φ(z).
1−z z
This yields
a0 m1 z(1 − z) − z 2
φ(z) = .
(1 − z)(ψ(z) − z)
One can show that φ(z) < ∞ for |z| < 1. Hence if there is a z with |z| < 1 for
which the denominator on the above equation becomes zero, the numerator must
P we see that ψ(z) − z has a
also become zero. From the results of Example 3.10,
solution z = α with 0 < α < 1 if and only if µ = kαk > 1. Thus, in this case we
must have
a0 m1 α(1 − α) = α2
or
α
m1 = .
a0 (1 − α)
Substituting in φ(z) we see that
α
1−α z(1 − z) − z 2
φ(z) = ,
(1 − z)(ψ(z) − z)
if a0 > 0 and µ > 1. Clearly, if a0 = 0 or if µ ≤ 1 we get m1 = ∞.
74 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
3.5 Generating Function and Higher Moments of T

Let {Xn , n ≥ 0} and T be as defined in Section 3.1. We begin this section with the
study of the generating function of T defined as
∞
X
φi (z) = E(z T |X0 = i) = z n P(T = n|X0 = i), i ≥ 1.
n=0

The above generating function is well defined for all complex z with |z| ≤ 1. Let B
be as defined in Equation 3.3 and let
b = [p10 , p20 , · · ·]′ .
The next theorem gives the main result concerning
φ(z) = [φ1 (z), φ2 (z), φ3 (z), · · ·]′ .

Theorem 3.4 The vector φ(z) is the smallest solution (for z ∈ [0, 1]) to
φ(z) = zb + zBφ(z). (3.28)

Proof: Using the first step analysis, we get, for i ≥ 1,

∞
X
φi (z) = E(z T |X0 = i) = pij E(z T |X0 = i, X1 = j).
j=0

Now, X0 = i, X1 = 0 ⇒ T = 1. Hence
E(z T |X0 = i, X1 = 0) = 1.
Also, for j ≥ 1, X0 = i, X1 = j implies that T has the same distribution as 1 + T
starting from X0 = j. Hence
E(z T |X0 = i, X1 = j) = E(z 1+T |X0 = j) = zφj (z).
Using these two observations we get
∞
X
φi (z) = zpi0 + z pij φj (z).
j=1

The above equation in matrix form yields Equation 3.28. The proof that it is the
smallest solution (for z ∈ [0, 1]) follows along the same lines as that of the proof of
Theorem 3.2.
Note that
∞
X
φi (1) = P(T = n|X0 = i) = P(T < ∞|X0 = i) = ui = 1 − vi . (3.29)
n=0

Using u = φ(1) in Equation 3.28 we get

u = b + Bu, (3.30)
GENERATING FUNCTION AND HIGHER MOMENTS OF T 75
or using v = e − u in the above equation, we get
v = Bv
which is Theorem 3.2. We have
∞
X
φ′ (1) = nP(T = n|X0 = i).
n=0

Thus, if ui = 1, φ′ (1) = E(T |X0 = i) = mi . Now, taking the derivatives of both

sides of Equation 3.28 we get
φ′ (z) = b + Bφ(z) + zBφ′ (z).
Setting z = 1 and recognizing φ(1) = u the above equation reduces to
φ′ (1) = b + Bu + Bφ′ (1).
Using Equation 3.30 this yields
φ′ (1) = u + Bφ′ (1).
Now assume u = e, that is absorption in state 0 is certain from all states. Then, using
φ′ (1) = m, we get
m = e + Bm,
which is Equation 3.22. We can similarly compute the higher moments by taking
higher derivatives. In particular, if we assume that u = e, and denoting the k-th
derivative of φ(z) by φ(k) (z), we see that
φ(k) = E(T (k) |X0 = i) = mi (k), (3.31)
(k)
where T = T (T − 1) · · · (T − k + 1) is the k-th factorial moment of T . Note that
T (1) = T and m(1) = m. The next theorem gives the equations satisfied by
m(k) = [m1 (k), m2 (k), · · ·]′ .

Theorem 3.5 Suppose u = e. Then m(k) is the smallest positive solution to

m(k) = kBm(k − 1) + Bm(k), k ≥ 2.

Proof: Follows by successively differentiating Equation 3.28 and using Equa-

tion 3.31. The proof of the minimality follows along the same lines as in the proof
Theorem 3.2.

Example 3.18 Genotype Evolution. Consider the six-state DTMC in the Geno-
type Evolution model of Example 3.2 with transition probability matrix given in
Equation 3.7. Suppose initially the population consists of one dominant and one hy-
brid individual. Compute the variance of time until the population becomes entirely
dominant or entirely recessive.
Let
T = min{n ≥ 0 : Xn = 1 or 6}.
 76 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
We will first compute mi (2) = E(T (T − 1)|X0 = i), i = 2, 3, 4, 5. Since we know
that eventually the entire population will be either dominant or recessive, ui = 1
for all i. Now let m(2) = [m2 (2), m3 (2), m3 (2), m4 (2)]′ , and B be the 4 by 4
submatrix of P obtained by deleting rows and columns corresponding to states 1 and
6. From Example 3.13, we have
5 2 2 5
m = m(1) = [4 , 6 , 5 , 4 ]′ .
6 3 3 6
Theorem 3.5 implies that
m(2) = 2Bm(1) + Bm(2).
Numerical calculations yield
8 4 1 8
m(2) = [39 , 60 , 49 , 39 ]′ .
9 9 9 9
Hence the required answer is
2
Var(T |X0 = 2) = m2 (2) + m2 (1) − m2 (1)2 = 22 .
3

3.6 Computational Exercises

3.1 Let {Xn , n ≥ 0} be a DTMC on state space {0, 1, 2, 3} with the following
transition probability matrix:
 
0.2 0.1 0.0 0.7
 0.1 0.3 0.6 0.0 
 .
 0.0 0.4 0.2 0.4 
0.7 0.0 0.1 0.2
Let T = min{n ≥ 0 : Xn = 0}. Compute

(a) P(T ≥ 3|X0 = 1),

(b) E(T |X0 = 1),
(c) Var(T |X0 = 1).

3.2 Do the above problem with the following transition probability matrix:
 
0.0 1.0 0.0 0.0
 0.2 0.0 0.8 0.0 
 0.0 0.8 0.0 0.2  .
 

0.0 0.0 1.0 0.0

3.3 Consider a 4 by 4 chutes and ladders game as shown in Figure 3.4. A player
starts on square one. He tosses a six sided fair die and moves 1 square if the die
shows 1 or 2, 2 squares if it shows 3 or 4, and 3 squares if it shows 5 or 6. Toward
the end of the play, when the player is near square 16, he has to land on square 16
COMPUTATIONAL EXERCISES 77

15
16 14 13

9 11 12
10

5
8 7 6

1 2 3 4

Figure 3.4 The 4 by 4 chutes and ladders.

exactly in order to finish the game. If he overshoots 16, he has to toss again. (In a
two person game he loses a turn.) Compute the expected number of tosses needed to
finish the game. (You may need to use a computer.)

3.4 If two players are playing the game compute the probability distribution of the
time until the game ends (that is, when one of the two players lands on 16.) As-
sume that there is no interaction among the two players, except that they take turns.
Compute the mean time until the game terminates.

3.5 Compute the expected number of times the ladder from 3 to 5 is used by a player
during one game.

3.6 Develop a computer program that analyzes a general chutes and ladders game.
The program accepts the following input: n, size of the board (n by n); k, the number
of players; the placements of the chutes and the ladders, and the distribution of the
number of squares moved in one turn. The program produces the following output:

1. Distribution of the number of tosses needed for one player to finish the game.
2. Distribution of the time to complete the game played by k players,
3. Expected length of the game played by k players.

3.7 Consider a maze of nine cells as shown in Figure 3.5. At time 0 a rat is placed
in cell one, and food in cell 9. The rat stays in the current cell for one unit of time
and then chooses one of the doors in the cell at random and moves to an adjacent
cell. Its successive moves are independent and completely uninfluenced by the food.
Compute the expected time required for the rat to reach the food.
78 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES

1 2 3

4 5 6

7 8 9

Figure 3.5 The nine cell maze.

3.8 Suppose the food in the above problem is replaced by a cat that moves like the
rat, but in an independent fashion. Of course, when the cat and the rat occupy the
same cell, the cat promptly eats the rat. Compute the expected time when the cat gets
its meal.

3.9 Let {Xn , n ≥ 0} be a DTMC on state-space {0, 1, 2, · · ·} with the following

transition probabilities
 1
i+2 0 ≤ j ≤ i + 1, i ≥ 0
pij =
0 otherwise.
Let T = min{n ≥ 0 : Xn = 0}. Compute E(T |X0 = 1).

3.10 Consider the Moran model on page 25 with N genes in the population, i of
which are initially dominant (1 ≤ i ≤ N ). Compute the expected time until all the
genes are dominant or all the genes are recessive.

3.11 Suppose a coin is tossed repeatedly and independently, the probability of ob-
serving a head on any toss being p. Compute the probability that a string of r con-
secutive heads is observed before a string of m consecutive tails.

3.12 In the Computational Exercise 3.10 compute the probability that eventually all
the genes become dominant.

3.13 Let {Zn , n ≥ 1} be a sequence of iid random variables with common pmf
pZ (1) = pZ (2) = pZ (3) = pZ (4) = .25. Let X0 = 0 and Xn = Z1 +Z2 +· · ·+Zn ,
n ≥ 1. Define T = min{n > 0 : Xn is divisible by 7}. Note the strict inequality in
n > 0. Compute E(T |X0 = 0).
COMPUTATIONAL EXERCISES 79
3.14 Consider the binomial model of the stock fluctuation as described on page 28.
Suppose X0 = 1, and an individual owns one stock at time 0. Suppose the individual
has decided to sell the stock as soon as it reaches a value of 2 or more. Compute
the expected time when such a sale would occur, assuming that p = .5, u = .2 and
d = .1. Hint: It might be easier to deal with log of the stock price.

3.15 In the coin tossing experiment of Computational Exercise 3.11 compute the
expected number of tosses needed to observe the sequence HHT T for the first time.

3.16 Solve Computational Exercise 3.14 if the individual decides to sell the stock
when it goes above 2 or below .7.

3.17 Let {Xn , n ≥ 0} be a simple random walk on {0, 1, · · · , N } with absorbing

barriers at 0 and N . That is p00 = pN N = 1, and pi,i+1 = p, pi,i−1 = q for
1 ≤ i ≤ N − 1. Let T = min{n ≥ 0 : Xn = 0 or N }. Show that
i N 1 − (q/p)i
E(T |X0 = i) = − ·
q − p q − p 1 − (q/p)N
if q 6= p. What is the corresponding formula if p = q?

3.18 Suppose a DNA sequence is appropriately modeled by a 4-state DTMC with

transition probability matrix P (1) given on page 22. Suppose the first base is A.
Compute the expected length of the sequence until we see the triplet ACT .

3.19 Let {Xn , n ≥ 0} be a simple random walk on {0, 1, 2, · · ·} with p00 = 1 and
pi,i+1 = p, pi,i−1 = q for i ≥ 1. Let T = min{n ≥ 0 : Xn = 0}. Show that, for
|z| ≤ 1,
E(z T |X0 = i) = φ(z)i
where p
φ(z) = (1 − 1 − 4pqz 2 )/2pz.
Give a probabilistic interpretation of φ(z)i in terms of the generating functions of the
convolutions of iid random variables.

3.20 Compute the expected number of bases between two consecutive appearances
of the triplet CAG in the DNA sequence in Computational Exercise 3.18.

3.21 Let {Xn , n ≥ 0} be the DTMC of Example 2.16. Let T = min{n ≥ 0 :

Xn = 0}. Show that
E(z T |X0 = i) = φ(z)i , i ≥ 0
where φ(z) is given by the smallest solution to
∞
X
φ(z) = z ai φ(z)i .
i=0

3.22 In the Computational Exercise 3.18 compute the probability that we see the
triplet ACT before the triple GCT .
80 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
3.23 In the Gambler’s Ruin model of Example 2.11 on page 17 compute the ex-
pected number of bets won by the player A until the game terminates, assuming that
the player A starts the game with i dollars, 1 ≤ i ≤ N − 1.

3.24 Consider the discrete time queue with Bernoulli arrivals and departures as de-
scribed in Example 2.12 on page 17. Suppose the queue has i customers in it initially.
Compute the probability that the server will eventually become idle.

3.25 Consider the clinical trials of Example 2.5 on page 14 using play the winner
rule. Suppose the trial stops as soon as either drug produces r successes, with the
drug producing the r successes first being declared the superior drug. Compute the
probability that the better of the two drugs gets declared as the best.

3.26 Derive a recursive method to compute the expected number of patients needed
to successfully conclude the clinical trial of the Computational Exercise 3.25.

3.27 Compute numerically the extinction probability of a branching process

{Xn , n ≥ 0} with X0 = 1 if each individual produces 20 offspring with proba-
bility 0.8 and 0 offspring with probability 0.2.

3.28 Consider a branching process {Xn , n ≥ 0} with X0 = 1 and E(X1 |X0 =

1) = µ ≤ 1. Let N be the total number of individuals that ever live in this colony
until the colony becomes extinct (which it does with probability 1, seePExample 3.10
∞
on page 64). N is called the total progeny and is given by N = n=0 Xn . Let
X1 N
ψ(z) = E(z |X0 = 1) and φ(z) = E(z |X0 = 1). Show that
φ(z) = zψ(φ(z)).
Hence show that
1
E(N |X0 = 1) = .
1−µ
Thus when µ = 1, the process becomes extinct with probability one, but the expected
total progeny is infinite!

3.29 Consider the DTMC of Modeling Exercise 2.17 with

1
αk = , 0 ≤ k ≤ M.
M +1
Compute the expected time until the DTMC reaches the absorbing state M .

3.30 A clinical trial is designed to identify the better of two experimental treatments.
The trial consists of several stages. At each stage two new patients are selected ran-
domly from a common pool of patients and one is given treatment 1, and the other is
given treatment 2. The stage is complete when the result of each treatment is known
as a success or a failure. At the end of the nth stage we record Xn = the number of
successes under treatment 1 minus those on treatment 2 observed on all the stages
so far. The trial stops as soon as Xn reaches +k or −k, where k is a given positive
CONCEPTUAL EXERCISES 81
integer. If the trial stops with Xn = k, treatment 1 is declared to be better than 2, else
treatment 2 is declared to be better than 1. Now suppose the probability of success
of the ith treatment is pi , i = 1, 2. Suppose p1 > p2 . Assume that the results of
the successive stages are independent. Compute the probability that the clinical trial
reaches correct decision, as a function of p1 , p2 , and k.

3.31 In the Computational Problem 3.30 compute the expected number of patients
subjected to treatments during the entire clinical trial.

3.32 Consider the manufacturing set up described in Modeling Exercise 2.32. Sup-
pose initially both bins are empty. Compute the expected time (in hours) until one of
the machines is turned off.

3.33 Consider the two bar town of Modeling Exercise 2.11. Assume that the two
transition probability matrices are identical. Compute the expected time when boy
and girl meet.

3.34 Solve Equation 3.19 for the special case when αi = 0 for i ≥ 4.

3.7 Conceptual Exercises

3.1 Let {Xn , n ≥ 0} be a DTMC on state-space S = {0, 1, 2, · · ·} and transition

probability matrix P . Let A ⊂ S and define
T (A) = min{n ≥ 0 : Xn ∈ A}.
Show that
vi (A) = P(T (A) = ∞|X0 = i)
are given by the largest solution bounded above by 1 to the following equations:
X
vi (A) = pij vj (A), i ∈
/ A.
j∈
/A

Use first step analysis and follow the proof of Theorem 3.2.

3.2 For the Conceptual Exercise 3.1 derive an analog of Theorem 3.3 for
mi (A) = E(T (A)|X0 = i).

3.3 Let {Xn , n ≥ 0} be a DTMC on state-space S = {0, 1, 2, · · ·} and transition

probability matrix P . Let
T̃ = min{n > 0 : Xn = 0}.
Show how to compute
ṽi = P(T̃ = ∞|X0 = i), i ≥ 0,
in terms of vi , i ≥ 1, as defined in Section 3.3.
82 DISCRETE-TIME MARKOV CHAINS: FIRST PASSAGE TIMES
3.4 For the Conceptual Exercise 3.3 show how to compute
m̃i = E(T̃ |X0 = i), i ≥ 0,
in terms of mi , i ≥ 1, as defined in Section 3.4.

3.5 Let T be as in Equation 3.1, and define

n
X
mi,n = kP(T = k|X0 = i)
k=0

and ui,n = P(T ≤ n|X0 = i). Using the first step analysis show that
∞
X
ui,n+1 = pi,0 + pij uj,n , i > 0, n ≥ 0,
j=1

and
∞
X
mi,n+1 = ui,n+1 + pij mj,n , i > 0, n ≥ 0.
j=1

3.6 Let {Xn , n ≥ 0} be a DTMC on state-space S = {0, 1, 2, ..., N } with transition

probability matrix P . Let A be a subset of the state-space, and T (A) be the time it
takes the DTMC to visit all the states in A at least once. Thus when A = {k}, T (A)
is the usual first passage time to the state k.

1. Develop simultaneous equations for

µj (k) = E(T ({k})|X0 = j), j, k ∈ S.
2. Using the quantities µ0 (N ) and µN (0) as known, develop simultaneous equations
for
µi (0, N ) = E(T ({0, N })|X0 = i), i ∈ S.

3.7 Let A be a fixed subset of the state-space S. Derive a set of equations to compute
the probability that the DTMC visits every state in A before visiting state 0.

3.8 Let {Xn , n ≥ 0} be a DTMC with state-space {0, 1, · · ·}, and transition proba-
bility matrix P . Let wij be the expected number of visits to state j starting from state
i before hitting state 0, counting the visit at time 0 if i = j. Using first-step analysis
show that
X∞
wij = δij + pik wkj , i ≥ 1
k=1
where δij = 1 if i = j, and zero otherwise.

3.9 Let {Xn , n ≥ 0} be a DTMC with state-space {0, 1, · · ·}, and transition proba-
bility matrix P . Let T = min{n ≥ 0 : Xn = 0}. A state j is called a gateway state
to state 0 if XT −1 = j. Derive equations to compute the probability wij that state j
is a gateway state to state 0 if the DTMC starts in state i.
CONCEPTUAL EXERCISES 83
3.10 Let {Xn , n ≥ 0} be a DTMC with state-space {0, 1, · · ·}, and transition proba-
bility matrix P . Let (i0 , i1 , · · · , ik ) be a sequence of states in the state-space. Define
T = min{n ≥ 0 : Xn = i0 , Xn+1 = i1 , · · · , Xn+k = ik }. Derive a method to
compute E(T |X0 = i).

3.11 Let {Xn , n ≥ 0} be a DTMC with state-space {0, 1, · · ·}, and transition prob-
ability matrix P . Let T = min{n ≥ 0 : Xn = 0}. Let M be the largest state visited
by the DTMC until hits state 0. Derive a method to compute the distribution of M .

3.12 Show by induction that the sequence {vi , i ≥ 0} satisfying Equation 3.21 is
an increasing sequence.
 CHAPTER 4

Discrete-Time Markov Chains:

Limiting Behavior

“God does not play dice with the universe”

– Albert Einstein

“God not only plays dice, he also sometimes throws the dice where they cannot be
seen”
– Stephen Hawking

4.1 Exploring the Limiting Behavior by Examples

Let {Xn , n ≥ 0} be a DTMC on S = {0, 1, 2, · · ·} with transition probability

matrix P . In Chapter 2 we studied two main aspects of the transient behavior of
the DTMC: the n-step transition probability matrix P (n) and the occupancy matrix
M (n) . Theorem 2.4 showed that
P (n) = P n , n ≥ 0,
and Theorem 2.6 showed that
n
X
M (n) = P r , n ≥ 0. (4.1)
r=0

In this chapter we study the limiting behavior of P (n) as n → ∞. Since the row
sums of M (n) are n + 1, we study the limiting behavior of M (n) /(n + 1) as n → ∞.
Note that [M (n) ]ij /(n + 1) can be interpreted as the fraction of the time spent by the
DTMC in state j starting from state i during {0, 1, · · · , n}. Hence studying this limit
makes practical sense. We begin by some examples illustrating the types of limiting
behavior that can arise.

Example 4.1 Two-State Example. Let

 
α 1−α
P =
1−β β

85
86 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
be the transition probability matrix of the two-state DTMC of Example 2.3 on
page 13 with α + β < 2. From Example 2.18 on page 33 we get
(α + β − 1)n 1 − α α − 1
   
n 1 1−β 1−α
P = + , n ≥ 0.
2−α−β 1−β 1−α 2−α−β β−1 1−β
Hence we get  
n 1 1−β 1−α
lim P = .
n→∞ 2−α−β 1−β 1−α
Thus the limit of P (n) exists and its row sums are one. It has an interesting feature
that both the rows of the limiting matrix are the same. This implies that the limiting
distribution of Xn does not depend upon the initial distribution of the DTMC. We
shall see that a large class of DTMCs share this feature. From Example 2.24 on
page 37 we get, for n ≥ 0,
1 − (α + β − 1)(n+1) 1 − α α − 1
   
(n) n+1 1−β 1−α
M = + .
2−α−β 1−β 1−α (2 − α − β)2 β−1 1−β
Hence, we get
M (n)
 
1 1−β 1−α
lim = .
n→∞ n + 1 2−α−β 1−β 1−α
Thus, curiously, the limit of M (n) /(n + 1) in this example is the same as that of
P (n) . This feature is also shared by a large class of DTMCs. We will identify this
class in this chapter.

Example 4.2 Three-State DTMC. Let

 
0 1 0
P = q 0 p ,
0 1 0
where 0 < p < 1 and q = 1 − p. We saw in Example 2.26 on page 40 that
 
q 0 p
P (2n) =  0 1 0  , n ≥ 1, (4.2)
q 0 p
and  
0 1 0
P (2n+1) = q 0 p  , n ≥ 0. (4.3)
0 1 0
Since the sequence {P (n) , n ≥ 1} shows an oscillatory behavior, it is clear that the
limit of P (n) does not exist. Now, direct calculations show that
 
1 + nq n np
M (2n) =  nq 1 + n np  , n ≥ 0, (4.4)
nq n np
EXPLORING THE LIMITING BEHAVIOR BY EXAMPLES 87
and  
1 + nq 1+n np
M (2n+1) =  (n + 1)q 1+n (n + 1)p  , n ≥ 0. (4.5)
nq 1+n 1 + np
Hence  
q/2 1/2 p/2
M (n)
lim =  q/2 1/2 p/2  . (4.6)
n→∞ n + 1
q/2 1/2 p/2
Thus, even if M (n) shows periodic behavior, M (n) /(n + 1) has a limit, and all its
rows are identical and add up to one!

Example 4.3 Genotype Evolution Model. Consider the six-state DTMC of the
genotype evolution model with the transition probability matrix as given in Equa-
tion 2.13 on page 24. Direct numerical calculations show that
 
1 0 0 0 0 0
 3/4 0 0 0 0 1/4 
(n)
 
(n) M  1/2 0 0 0 0 1/2 
lim P = lim =  . (4.7)
n→∞ n→∞ n + 1  1/2 0 0 0 0 1/2 

 1/4 0 0 0 0 3/4 
0 0 0 0 0 1
Thus this example also shows that the limit of P (n) exists and is the same as the
limit of M (n) /(n + 1), and the row sums of the limiting matrix are one. However,
in this example all the rows of the limiting matrix are not identical. Thus the limiting
distribution of Xn exists, but depends upon the initial distribution of the DTMC. We
will identify the class of DTMCs with this feature later on in this chapter.

Example 4.4 Simple Random Walk. A simple random walk on all integers has the
following transition probabilities
pi,i+1 = p, pi,i−1 = q = 1 − p, −∞ < i < ∞,
where 0 < p < 1. We have seen in Example 2.19 on page 33 that
(2n)(2n)! n n
p00 = p q , n = 0, 1, 2, · · · . (4.8)
n!n!
We show that the above quantity converges to zero as n → ∞. We need to use the
following asymptotic expression called Stirling’s formula:
√
n! ∼ 2πnn+1/2 e−n
where ∼ indicates that the ratio of the two sides goes to one as n → ∞. We have
(2n) (2n)! n n
p00 = p q
n!n!
√
2πe−2n (2n)2n+1/2 n n
∼ √ p q
( 2πe−n nn+1/2 )2
88 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
(4pq)n
= √
πn
1
≤ √
πn
where the last inequality follows because 4pq = 4p(1 − p) ≤ 1 for 0 ≤ p ≤ 1. Thus
(2n)
p00 approaches zero asymptotically. We also know that p2n+1
00 is always zero. Thus
we see that
(n)
lim p00 = 0.
n→∞
Similar calculation shows that
lim [P (n) ]ij = 0
n→∞

for all i and j. It is even more tedious, but possible, to show that
[M (n) ]ij
lim = 0.
n→∞ n + 1

Thus in this example the two limits again coincide, but the row sums of the limiting
matrix are not one, but zero! Since P (n) is a stochastic matrix, we have
∞
(n)
X
pij = 1, n ≥ 0.
j=−∞

Hence we must have

∞
(n)
X
lim pij = 1.
n→∞
j=−∞
However, the above calculations show that
∞
(n)
X
lim p = 0.
n→∞ ij
j=−∞

This implies that the interchange of limit and the sum in this infinite state-space
DTMC is not allowed. We shall identify the class of DTMCs with this feature later
in this chapter.

Thus we have identified four cases:

Case 1: Limit of P (n) exists, has identical rows, and each row sums to one.

Case 2: Limit of P (n) exists, does not have identical rows, each row sums to one.

Case 3: Limit of P (n) exists, but the rows may not sum to one.

Case 4: Limit of P (n) does not exist.

 IRREDUCIBILITY AND PERIODICITY 89
We have also observed that limit of M (n) /(n + 1) always exists, and it equals the
limit of P (n) when it exists. We develop the necessary theory to help us classify the
DTMCs so we can understand their limiting behavior better.

4.2 Irreducibility and Periodicity

In this section we introduce the concepts of irreducibility and periodicity. These

concepts will play an important role in the study of the limiting behavior of a DTMC.

Definition 4.1 Accessibility. A state j is said to be accessible from a state i if there

(n)
is an n ≥ 0 such that pij > 0.

Note that j is accessible from i if and only if there is a directed path in the diagraph
(n)
representation of the DTMC. This is so, since pij > 0 implies that there is a se-
quence of states i = i0 , i1 , · · · , in = j such that pik ,ik+1 > 0 for k = 0, 1, · · · , n − 1.
This in turn is equivalent to the existence of a directed path i = i0 , i1 , · · · , in = j in
the diagraph.
(0)
We write i → j if state j is accessible from i. Since pii = 1, it trivially follows
that i → i.

Definition 4.2 Communication. States i and j are said to communicate if i → j

and j → i.

If i and j communicate, we write i ↔ j. The following theorem states some

important properties of the relation “communication.”

Theorem 4.1 Properties of Communication.

(i) i ↔ i, (reflexivity).
(ii) i ↔ j ⇔ j ↔ i, (symmetry).
(iii) i ↔ j, j ↔ k ⇒ i ↔ k, (transitivity).

Proof: (i) and (ii) are obvious from the definition. To prove (iii) note that i → j and
j → k imply that there are integers n ≥ 0 and m ≥ 0 such that
(n) (m)
pij > 0, pjk > 0.
Hence
(n+m) (n) (m)
X
pik = pir prk (Theorem 2.3)
r∈S
(n) (m)
≥ pij pjk > 0.
Hence i → k. Similarly k → j and j → i imply that k → i. Thus
i ↔ j, j ↔ k ⇒ i → j, j → k and k → j, j → i ⇒ i → k, k → i ⇒ i ↔ k.
 90 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
It is possible to write a computer program that can check the accessibility of one
state from another in O(q) time where q is the number of non-zero entries in P .
The same algorithm can be used to check if two states communicate. See Conceptual
Exercise 4.3.
From the above theorem it is clear that “communication” is a reflexive, symmetric,
and transitive binary relation. (See Conceptual Exercise 4.1 for more examples of
binary relations.) Hence we can use it to partition the state-space S into subsets
known as communicating classes.

Definition 4.3 Communicating Class. A set C ⊂ S is said to be a communicating

class if

(i) i ∈ C, j ∈ C ⇒ i ↔ j,
(ii) i ∈ C, i ↔ j ⇒ j ∈ C.

Property (i) assures that any two states in a communicating class communicate
with each other (hence the name). Property (ii) forces C to be a maximal set, i.e.,
no strict superset of C can be a communicating class. Note that it is possible to
have a state j outside C that is accessible from a state i inside C, but in this case, i
cannot be accessible from j. Similarly, it is possible to have a state i inside C that is
accessible from a state j outside C, but in this case, j cannot be accessible from i.
This motivates the following definition.

Definition 4.4 Closed Communicating Class. A communicating class is said to be

closed i ∈ C and j ∈
/ C implies that j is not accessible from i.

Note that once a DTMC visits a state in a closed communicating class C, it cannot
leave it, i.e,
Xn ∈ C ⇒ Xm ∈ C for all m ≥ n.
Two distinct communicating classes must be disjoint, see Conceptual Exercise 4.2.
Thus we can uniquely partition the state-space S as follows:
S = C1 ∪ C2 ∪ · · · ∪ Ck ∪ T, (4.9)
where C1 , C2 , · · · Ck are k disjoint closed communicating classes, and T is the union
of all the other communicating classes. We do not distinguish between the communi-
cating classes if they are not closed, and simply lump them together in T . Although
it is possible to have an infinite number of closed communicating classes in a DTMC
with a discrete state-space, in practice we shall always encounter DTMCs with a
finite number of closed communicating classes.
It is possible to develop an algorithm to derive the above partition in time O(q),
see Conceptual Exercise 4.4. Observe that the classification depends only on which
elements of P are positive and which are zero. It does not depend upon the actual
values of the positive elements.
IRREDUCIBILITY AND PERIODICITY 91
Definition 4.5 Irreducibility. A DTMC with state-space S is said to be irreducible
if S is a closed communicating class, else it is called reducible.

It is clear that all states in an irreducible DTMC communicate with each other. We
shall illustrate the above concepts by means of several examples below. In each case
we show the diagraph representation of the DTMC. In our experience the diagraph
representation is the best visual tool to identify the partition of Equation 4.9.

Example 4.5 Two-State DTMC. Consider the two state DTMC of Example 2.3
with 0 < α, β < 1. The diagraph is shown in Figure 4.1. In this DTMC 1 ↔ 2, and

1–α

α 1 2 β

1–β

Figure 4.1 The two-state DTMC: irreducible case.

hence {1, 2} is a closed communicating class, and the DTMC is irreducible. Next
suppose α = 1 and 0 < β < 1. The diagraph representation is shown in Figure 4.2.
Here we have 1 ↔ 1, 2 ↔ 2, 2 → 1, but 2 is not accessible from 1. Thus C1 = {1}

1 1 2 1–β
β

Figure 4.2 The two-state DTMC: reducible case.

is a closed communicating class, and T = {2} is a communicating class that is not

closed. Thus this DTMC is reducible. Finally, suppose α = β = 1. In this case
C1 = {1} and C2 = {2} are two closed communicating classes and T = φ. The
DTMC is reducible.

Example 4.6 Genotype Evolution. Consider the six state DTMC arising in the
genotype evolution model of Example 4.3. Since p11 = p66 = 1, it is clear that C1 =
{1} and C2 = {6} are two closed communicating classes. Also, T = {2, 3, 4, 5} is
a communicating class that is not closed, since states 1 and 6 are accessible from all
the four states in T . The DTMC is reducible.

Example 4.7 Simple Random Walk. Consider the simple random walk on
{0, 1, · · · , N } with p00 = pN N = 1, pi,i+1 = p, pi,i−1 = q, 1 ≤ i ≤ N − 1.
Assume p > 0, q > 0, p + q = 1. The diagraph representation is shown in Fig-
ure 4.3. In this DTMC C1 = {1} and C2 = {N } are two closed communicating
classes. Also, T = {1, 2, · · · , N − 1} is a communicating class that is not closed,
92 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
1 1
p p p p p p p

0 1 i–1 i i+1 N–1 N

q q q q q q q

Figure 4.3 Transition diagram of a random walk with absorbing barriers.

since states 1 and N are accessible from all the states in T . The DTMC is reducible.
What happens if p or q is 1?

Next we introduce the concept of periodicity.

Definition 4.6 Periodicity. A state i is said to be periodic with period d if d is the

(n)
greatest common divisor (gcd) of the integers in the set {n ≥ 1 : pii > 0}. A state
i is said to be aperiodic if it has period 1.

Note that a state i is aperiodic if pii > 0. The term periodic is self explanatory: the
DTMC, starting in a state i with period d, can return to state i only at times that
are integer multiples of d. An alternate definition of periodicity involves the strictly
positive first passage time defined in Conceptual Exercise 3.3, reiterated here with a
slight variation:
T̃i = min{n > 0 : Xn = i}, i ∈ S. (4.10)

Definition 4.7 Periodicity. A state i is said to be periodic with period d if d is the

gcd of the integers in the set {n ≥ 1 : P(T̃i = n|X0 = i) > 0}.

We leave it to the reader (see Conceptual Exercise 4.5) to show that the two defi-
nitions are equivalent. If the DTMC never returns to state i once it leaves state i, the
sets in the two definitions above are empty. In this case we define the period to be ∞.
It will be seen later that for such states the concept of periodicity is irrelevant. The
next theorem gives a very useful result about periodicity.

Theorem 4.2 Periodicity. If i ↔ j, then i and j have the same period.

Proof: Let d be the period of state i. Now, i ↔ j implies that there are two integers
n and m such that
(n) (n)
pij > 0, and pji > 0.
We have
(n+r+m) (n) (r) (m) (n) (r) (m)
X
pii = pik pkj pji ≥ pij pjj pji .
k,j
For r = 0, we get
(n+m) (n) (m)
pii ≥ pij pji > 0.
Hence n + m must be an integer multiple of d. Now suppose r is not divisible by
(n+r+m)
d. Then n + m + r is not divisible by d. Hence pii = 0 since i has period d.
(n) (r) (m) (n) (m)
Thus pij pjj pji = 0. However, pij and pji are not zero. Hence we must have
RECURRENCE AND TRANSIENCE 93
(r) (r)
pjj = 0 if r is not divisible by d. Thus the gcd of {r ≥ 1 : pjj> 0} must be
an integer multiple of d. Suppose it is kd for some integer k ≥ 1. However, by the
same argument, if the period of j is kd, that of i must be an integer multiple of kd.
However, we have assumed that the period of i is d. Hence we must have k = 1. This
proves the theorem.
The above theorem implies that all states in a communicating class have the same
period. We say that periodicity is a class property. This enables us to talk about the
period of a communicating class. A communicating class or an irreducible DTMC
is said to be periodic with period d if any (and hence every) state in it has period
d. If the period is 1, it is called aperiodic. It should be noted that periodicity, like
communication, depends only upon which elements of P are positive, and which are
zero; and not on the actual magnitudes of the positive elements.

Example 4.8 Two-State DTMC. Consider the two state DTMC of Example 4.1.
In each case considered there pii > 0 for i = 1, 2, hence the both the states are
aperiodic. Now consider the case with α = β = 0. The transition probability matrix
in this case is  
0 1
P .
1 0
Thus we see that if X0 = 1 we have
X2n = 1, X2n−1 = 2, n ≥ 1.
Hence the DTMC returns to state 1 only at even times, and hence it has period 2.
Since this is an irreducible DTMC, all states must have period 2. Thus state 2 has
period 2.

Example 4.9 Simple Random Walk. Consider the simple random walk of Exam-
ple 4.4. Since p00 = pN N = 1, states 0 and N are aperiodic. Now suppose the
DTMC starts in state 1. Clearly, DTMC cannot return to state 1 at odd times, and can
return at every even time. Hence state 1 has period 2. Since states 1, 2, · · · , N − 1
communicate with each other, all of them must have period 2.

It is straight forward to develop an O(N 3 ) algorithm to find the periodicity of the

states in a DTMC with N states. By using more sophisticated data structures one
develops an more efficient algorithm to do the same in O(q) steps, where q is the
number of non-zero entries in its transition probability matrix.

4.3 Recurrence and Transience

In this section we introduce the concepts of recurrence and transience of states. They
play an important role in the study of the limiting behavior of the DTMCs. Let T̃i be
as defined in Equation 4.10. Define
ũi = P(T̃i < ∞|X0 = i) (4.11)
 94 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
and
m̃i = E(T̃i |X0 = i). (4.12)
When ũi < 1, m̃i = ∞. However, as the following example suggests, m̃i can be
infinite even if ũi = 1.

Example 4.10 Success Runs. Consider the success runs Markov chain on
{0, 1, 2, · · ·} with
i+1 1
pi,i+1 = , pi,0 = , i ≥ 0.
i+2 i+2
Now, direct calculations as in Example 3.3 on page 58 show that
P(T̃0 = n|X0 = 0) = P(Xi = i, i = 1, 2, · · · , n − 1, Xn = 0|X0 = 0)
1
= , n ≥ 1.
(n + 1)(n + 2)
Hence,
ũ0 = P(T̃0 < ∞|X0 = 0)
X∞
= P(T̃0 = n|X0 = 0)
n=1
∞
X 1
= = 1.
n=1
(n + 1)(n + 2)
The last equality above is seen by writing
1 1 1
= −
(n + 1)(n + 2) n+1 n+2
and using telescoping sums. However, we get
∞
X
1 + m̃0 = (n + 1)P(T̃0 = n|X0 = 0)
n=1
∞
X n+1
=
n=1
(n + 1)(n + 2)
∞
X 1
= = ∞.
n=1
n + 2
Here, the last inequality follows because the last sum is a harmonic series, which is
known to diverge.

With this discussion we make the following definitions:

Definition 4.8 Recurrence and Transience. A state i is said to be

(i) recurrent if ũi = 1,

(ii) transient if ũi < 1.
 RECURRENCE AND TRANSIENCE 95
Definition 4.9 Positive and Null Recurrence. A recurrent state i is said to be
(i) positive recurrent if m̃i < ∞,
(ii) null recurrent if m̃i = ∞.

Now, in Section 2.5 we studied Vi (n), the number of visits to state i by a DTMC
over the finite time period {0, 1, 2, · · · , n}. Here we study Vi , the number of visits
by the DTMC to state i over the infinite time period {0, 1, 2, · · ·}. The next theorem
gives the main result.

Theorem 4.3 Number of Visits. If state i is recurrent

P(Vi = ∞|X0 = i) = 1.
If state i is transient
P(Vi = k|X0 = i) = ũk−1
i (1 − ũi ), k ≥ 1.

Proof: Follows from Markov property and time homogeneity. See Conceptual Exer-
cise 4.6.
The next theorem yields the necessary and sufficient condition for recurrence and
transience.

Theorem 4.4 Criterion for Recurrence and Transience.

(i) State i is recurrent if and only if
∞
(n)
X
pii = ∞.
n=0
(ii) State i is transient if and only if
∞
(n)
X
pii < ∞.
n=0

Proof: Let Vi (n) be the number of visits to state i over 0 through n. Let Vi be the
number of visits to state i over 0 to ∞. We see that ũi = 1 implies that the DTMC,
starting from state i, returns to it with probability 1, and hence, due to the Markov
property and time homogeneity, returns infinitely often. Hence, in this case,
E(Vi |X0 = i) = ∞.
On the other hand, if ũi < 1, the DTMC, starting from state i, returns to state i
exactly k times with probability ũk−1
i (1 − ũi ). Hence
1
E(Vi |X0 = i) = < ∞.
1 − ũi
Now, from the results in Section 2.5 we have
n
(m)
X
E(Vi (n)|X0 = i) = Mii (n) = pii .
m=0
 96 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Hence
∞
(n)
X
E(Vi |X0 = i) = pii .
n=0

The theorem follows from this.

Next we give a necessary and sufficient condition for the positive and null recur-
(n)
rence. Notice that limn→∞ Mii /(n + 1) can be thought of as the long run number
of visits to state i per unit time. Since the expected time between two consecutive
visits to state i is m̃i , it is intuitively clear that the long run number of visits to state
i per unit time should be 1/m̃i . Thus if the limit is positive, that would imply a finite
m̃i , and if the limit is 0, that would imply an infinite m̃i . This provides the intuition
behind the next theorem.

Theorem 4.5 Criterion for Null and Positive Recurrence.

(i) A recurrent state i is positive recurrent if and only if

n
1 X (m)
lim pii > 0.
n→∞ n + 1
m=0

(ii) A recurrent state i is null recurrent if and only if

n
1 X (m)
lim pii = 0.
n→∞ n + 1
m=0

Proof: This theorem is a special case of a general theorem called elementary renewal
theorem, which will be proved in Chapter 8. Hence we do not include a formal proof
here.
Unlike periodicity, transience and recurrence are dependent upon the actual mag-
nitudes of the transition probabilities pij . Like periodicity, they are class properties.
This is shown in the next two theorems.

Theorem 4.6 Recurrence and Transience as Class Properties.

(i) i ↔ j, i is recurrent ⇒ j is recurrent.

(ii) i ↔ j, i is transient ⇒ j is transient.

(n)
Proof: (i) Suppose i ↔ j. Then there are integers n and m such that pij > 0 and
(m)
pji > 0. Now,
∞ ∞
(r) (r+n+m)
X X
pjj ≥ pjj
r=0 r=0
∞ X
(m) (r) (n)
X
≥ pjk pkk pkj
r=0 k∈S
 RECURRENCE AND TRANSIENCE 97
∞
(m) (r) (n)
X
≥ pji pii pij
r=0
∞
(m) (n) (r)
X
= pji pij pii = ∞
r=0

since i is recurrent. Hence, from Theorem 4.4, j is recurrent.

(ii) Suppose i is transient and i ↔ j, but j is recurrent. By (i) above, j is recurrent,
i ↔ j, implies that i is recurrent, a contradiction. Hence j must be transient. This
proves the theorem.

Theorem 4.7 Null and Positive Recurrence as Class Properties.

(i) i ↔ j, i is positive recurrent ⇒ j is positive recurrent.
(ii) i ↔ j, i is null recurrent ⇒ j is null transient.

Proof: (i) Let n and m be as in the proof of Theorem 4.6. We have

(r+n+m)
X (m) (r) (n) (m) (r) (n)
pjj ≥ pjk pkk pkj ≥ pji pii pij .
k∈S

Now,
t t
1 X (r) 1 X (r+n+m)
lim pjj = lim pjj
t→∞ t + 1 t→∞ t + 1
r=0 r=0
n
!
(m) 1 X (r) (n)
≥ pji lim pii pij > 0.
n→∞ n + 1
r=0

The last inequality follows because the positive recurrence of i implies that the last
(n) (m)
limit is positive (from Theorem 4.5), and we also have pij > 0 and pji > 0. Hence
state j is positive recurrent.
(ii) Follows along the same lines as the proof of part (ii) in Theorem 4.6.
Theorems 4.6 and 4.7 greatly simplify the task of identifying the transient and
recurrent states. If state i is recurrent then all states that belong to the same commu-
nicating class as i must be recurrent. The same conclusion holds for the transient or
positive or null recurrent states. Hence we can make the following definitions.

Definition 4.10 Recurrent Class. A communicating class is called recurrent (tran-

sient, positive recurrent, null recurrent) if all the states in it are recurrent (transient,
positive recurrent, null recurrent).

Definition 4.11 Recurrent DTMC. An irreducible DTMC is called recurrent (tran-

sient, positive recurrent, null recurrent) if all its states are recurrent (transient, posi-
tive recurrent, null recurrent).

The task of determining the transience and recurrence of a finite communicating

class is particularly easy, as seen from the next theorem.
98 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Theorem 4.8 Finite Closed Classes. All states in a finite closed communicating
class are positive recurrent.

Proof: Let C be a finite closed communicating class. Then for all i ∈ C, we have
X (k)
1 = P(Xk ∈ C|X0 = i) = pij , k ≥ 0.
j∈C

Hence
n
1 X X (k)
pij = 1.
n+1
k=0 j∈C
Taking limits, we get
n n
1 X X (k) X 1 X (k)
lim pij = lim pij = 1,
n→∞ n + 1 n→∞ n + 1
k=0 j∈C j∈C k=0

where the interchange of the limit and the sum is allowed since C is finite. Thus there
must be at least one j for which
n
1 X (k)
lim pij > 0. (4.13)
n→∞ n + 1
k=0

Now suppose the states in C are not positive recurrent. Then they must be all null
recurrent or all transient. In either case we must have
n
1 X (k)
lim pjj = 0, j ∈ C. (4.14)
n→∞ n + 1
k=0

This follows from Theorem 4.7(ii) for the null recurrent case, and Theorem 4.6(ii)
(r)
for the transient case. Since C is finite, there is an r = r(i, j) such that pji > 0.
Now,
(k+r) (r) (k)
pjj ≥ pji pij . (4.15)
Combining Equations 4.14 and 4.15, we get
n n
1 X (k) 1 X (k+r)
0 = lim pjj = lim pjj
n→∞ n + 1 n→∞ n + 1
k=0 k=0
n
(r) 1 X (k)
≥ pji lim pij .
n→∞ n+1
k=0

Hence we must have

n
1 X (k)
lim pij = 0,
n→∞ n + 1
k=0
for all i, j ∈ C. But this contradicts Equation 4.13. Thus the states in C cannot be all
null recurrent or transient. Hence they must be positive recurrent.

Theorem 4.9 Non-closed Classes. All states in a non-closed communicating class

are transient.
DETERMINING RECURRENCE AND TRANSIENCE: INFINITE DTMCS 99
Proof: Let C be a non-closed communicating class. The there exists an i ∈ C and
aj ∈ / C such that i is not accessible from j and pij > 0. Thus, if the DTMC visits
visits state j starting from state i, it will never return to state i. Hence
1 − ũi = P(T̃i = ∞|X0 = i) ≥ pij > 0.
Thus ũi < 1, and hence state i is transient. Since C is a communicating class, all the
states in it must be transient.
Since all communicating classes in a finite state DTMC must be finite, and they
must be all closed or not, Theorems 4.8 and 4.9 are sufficient to determine the tran-
sience and recurrence of any state in such a DTMC. We leave it to the reader to
show that there are no null recurrent states in a finite state DTMC (See Conceptual
Exercise 4.9) and that not all states in a finite state DTMC can be transient (See Con-
ceptual Exercise 4.11).

We illustrate with two examples.

Example 4.11 Genotype Evolution. Consider the six-state DTMC of Example 4.3.
We saw in Example 4.6 that it has two closed communicating classes C1 = {1} and
C2 = {6}, and a non-closed class T = {2, 3, 4, 5}. Thus states 1 and 6 are positive
recurrent and states 2,3,4,5 are transient.

Example 4.12 Simple Random Walk. Consider the simple random walk of Ex-
ample 4.4. We saw in Example 4.9 that it has two closed communicating classes
C1 = {1} and C2 = {N }, and a non-closed class T = {2, 3, · · · , N − 1}. Thus
states 1 and N are positive recurrent and state 2, 3, · · · , N − 1 are transient.

Determining the transience or recurrence of states in a communicating class with

an infinite number of states is a much harder problem. We discuss several methods
of doing so in the next section.

4.4 Determining Recurrence and Transience: Infinite DTMCs

Without loss of generality we consider an irreducible DTMC on state-space S =

{0, 1, 2, · · ·} with transition probability matrix P . The states in such a DTMC may
be all positive recurrent, or all null recurrent, or all transient. There is no easy way of
determining this. We shall illustrate with several examples.

Example 4.13 General Success Runs. Consider the general success runs DTMC of
Example 3.3 on page 58. This DTMC has state-space S = {0, 1, 2, · · ·} and transition
probabilities
pi,0 = qi , pi,i+1 = pi , i ∈ S.
Assume that pi > 0, qi > 0 for all i ≥ 0. Then the DTMC is irreducible. Thus
if we determine the transience or recurrence of the state 0, that will automatically
 100 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
determine the transience and recurrence of all the states in S. From the first step
analysis we get
ũ0 = 1 − p0 v1 ,
where v1 is the probability that the DTMC never visits state 0 starting from state 1.
From the analysis in Example 3.7 on page 62 we have
∞
X
v1 = 0 ⇔ qn = ∞,
n=1

and
∞
X
v1 > 0 ⇔ qn < ∞.
n=1
It follows that the general success runs DTMC is

(i) recurrent if and only if ∞

P
i=1 qi = ∞, and
P∞
(ii) transient if and only if i=1 qi < ∞.

Now suppose the DTMC is recurrent. Then we have

m̃0 = 1 + p0 m1 ,
where m1 is the expected first passage time T to state 0 starting from state 1. Using
the results of Example 3.3 we get
∞
X
m1 = nP(T = n|X0 = 1)
n=1
X∞
= P(T ≥ n|X0 = 1)
n=1
∞ n−1
X Y
= 1+ pi .
n=2 i=1

Thus the recurrent DTMC is positive recurrent if the last sum converges, otherwise
it is null recurrent.

Example 4.14 General Simple Random Walk. Consider the random walk on
{0, 1, 2, · · ·} with the following transition probabilities:
pi,i+1 = pi for i ≥ 0,
pi,i−1 = qi = 1 − pi for i ≥ 1.

We assume that pi > 0, i ≥ 0 and qi > 0, i ≥ 1 so that the DTMC is irreducible.

We have
ũ0 = 1 − p0 v1 ,
DETERMINING RECURRENCE AND TRANSIENCE: INFINITE DTMCS 101
where v1 is the probability that the DTMC never visits state 0 starting from state 1.
Using the results from Example 3.9 we get
v1 > 0 if P∞
P
i=0 αi < ∞,
∞
v1 = 0 if i=0 αi = ∞,

where α0 = 1
q1 q2 · · · qi
αi = , i ≥ 1.
p1 p2 · · · pi
P∞
Thus state 0 (and hence the entire
P∞ DTMC) is recurrent if and only if i=0 αi = ∞,
and transient if and only if i=0 α i < ∞. Next, assuming the DTMC is recurrent,
we can use the first step analysis to obtain
m̃0 = 1 + p0 m1 ,
where m1 is the expected first passage time into state 0 starting from state 1. From
Example 3.15 we have
∞
X 1
m1 = .
p α
j=1 j j
Now let ρ0 = 1 and
p0 p1 · · · pi−1
ρi = , i ≥ 1.
q1 q2 · · · qi
Thus
∞
X
m̃0 = ρi .
i=0
P
Thus state 0 (and hence the whole DTMC) is positive recurrent if the series ρi
converges, and null recurrent if it diverges. Combining all these results, we get the
following complete classification: the state 0 (and hence the entire DTMC) is
P∞
(i) positive recurrent if and only if ρi < ∞,
P∞ i=0 P∞
(ii) null recurrent if and only if i=0 ρi = ∞, and i=0 αi = ∞,
P∞
(iii) transient if and only if i=0 αi < ∞.

Example 4.15 Production-Inventory System: Batch Production. Consider the

DTMC of Example 2.16 on page 19. It has state-space {0, 1, 2, · · ·} and transition
probability matrix as given in Equation 2.10. We assume that α0 > 0 and α0 +α1 < 1
so that the DTMC is irreducible. We study the recurrence properties of state 0. From
the first step analysis we get
X∞
ũ0 = αj uj ,
i=0
where uj is the probability that the DTMC eventually visits state 0 starting from state
j. Using the results from Example 3.11 we see thatP uj = 1 for all j ≥ 0 (and hence
ũ0 = 1) if the mean production batch size µ = ∞ k=0 kαk ≤ 1, otherwise uj < 1
for all j ≥ 1 (and hence ũ0 < 1). This implies that state 0 (and hence the DTMC) is
 102 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
recurrent if µ ≤ 1 and transient if µ > 1. Next, assuming the DTMC is recurrent, the
first step analysis yields
∞
X
m̃0 = 1 + αj mj ,
j=0
where mj is the expected time it takes the DTMC to reach state zero starting from
state j. Using the results of Example 3.16, we see that m̃0 < ∞ if µ < 1. Hence the
DTMC is

(i) positive recurrent if and only if µ < 1,

(ii) null recurrent if and only if µ = 1,
(iii) transient if and only if µ > 1.

This completes the classification of the DTMC. This makes intuitive sense since this
DTMC models the situation where exactly one item is removed from the inventory
every time period, while µ items are added to the system per period on the average.

Example 4.16 Production-Inventory System: Batch Demands. Consider the

DTMC of Example 2.17 on page 20. It has state-space {0, 1, 2, · · ·} and transition
probability matrix as given in Equation 2.12. We assume that α0 > 0 and α0 +α1 < 1
so that the DTMC is irreducible. We study the recurrence properties of state 0. From
the first step analysis we get
ũ0 = β0 + α0 u1 ,
where u1 is the probability that the DTMC eventually visits state 0 starting from state
1. Using the results from Example 3.12 P we see that u1 = 1 (and hence ũ0 = 1) if
the mean production batch size µ = ∞ k=0 kαk ≥ 1, otherwise u1 < 1 (and hence
ũ0 < 1). This implies that state 0 (and hence the DTMC) is recurrent if µ ≥ 1
transient if µ < 1. Next, assuming the DTMC is recurrent, the first step analysis
yields
m̃0 = 1 + α0 m1 ,
where m1 is the expected time it takes the DTMC to reach state zero starting from
state 1. Using the results of Example 3.17, we see that
1
m̃0 = ,
1−α
where α is the unique solution in (0, 1) if µ > 1. We can also conclude that
m̃0 = ∞,
if µ = 1. Hence the DTMC is

(i) positive recurrent if and only if µ > 1,

(ii) null recurrent if and only if µ = 1,
(iii) transient if and only if µ < 1.

This completes the classification of the DTMC.

DETERMINING RECURRENCE AND TRANSIENCE: INFINITE DTMCS 103
4.4.1 Foster’s Criterion

In this section we shall derive a sufficient condition for the positive recurrence of an
irreducible DTMC {Xn , n ≥ 0} with state-space S and transition probability matrix
P . Let ν : S → [0, ∞) and, for i ∈ S, define
d(i) = E(ν(Xn+1 ) − ν(Xn )|Xn = i)
X
= pij ν(j) − ν(i).
j∈S

The function ν is called a potential function, and the quantity d(i) is called the gen-
eralized drift in state i. It is called drift when ν(i) = i for all i ∈ S. The main result,
called Foster’s criterion, is given below.

Theorem 4.10 Foster’s Criterion. Let {Xn , n ≥ 0} be an irreducible DTMC on

a countable state-space S. If there exists a potential function ν : S → [0, ∞), an
ǫ > 0, and a finite set H ⊂ S such that
|ν(i)| < ∞ for i ∈ H (4.16)
ν(i) < −ǫ for i ∈
/H (4.17)
the DTMC is positive recurrent.

Before we prove the above result, we give the intuition behind it. Suppose the hypoth-
esis of the above theorem holds. Since ν is a non-negative function, E(ν(Xn )) ≥ 0
for all n ≥ 0. However, whenever Xn ∈ / H, we have E(ν(Xn+1 )) < E(ν(Xn )) − ǫ.
Hence the DTMC cannot stay outside of H for too long, else E(ν(Xn )) will become
negative. Hence the DTMC must enter the finite set H often enough. The theorem
says that the visits to H are sufficiently frequent to make the states in H (and hence,
due to irreducibility, the whole chain) positive recurrent. A formal proof follows.
Proof: Define X
w(k) = pkj ν(j), k ∈ H.
j∈S
Then, from Equation 4.16, |w(k) − ν(k)| < ∞ for all k ∈ H. Hence w(k) < ∞ for
all k ∈ H. Now let
(0)
yi = ν(i),
and
(n)
X (n)
yi = pij ν(j).
j∈S
Then
(r+1)
X
yi (r + 1) = pij ν(j)
j∈S
(r)
XX
= pik pkj ν(j)
j∈S k∈S
(r) (r)
X X X X
= pik pkj ν(j) + pik pkj ν(j)
k∈H j∈S k/
∈H j∈S
 104 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
X (r) X (r)
≤ pik w(k) + pik (ν(k) − ǫ)
k∈H k/
∈H
(r) (r)
X X
≤ pik (w(k) + ǫ) + pik ν(k) − ǫ (since ν(k) ≥ 0 for all k)
k∈H k∈S
(r) (r)
X
≤ pik (w(k) + ǫ) + yi − ǫ.
k∈H

Adding the above inequalities for r = 0, 1, 2, · · · , n yields

n
(r+1) (r) (0)
XX
yi ≤ pik (w(k) + ǫ) + yi − (n + 1)ǫ.
k∈H r=0
(n+1) (0)
However, yi ≥ 0 and yi is finite for each i. Hence, dividing by n + 1 and
letting n → ∞ yields
n
!
X 1 X (r)
lim pik (w(k) + ǫ) ≥ ǫ.
n→∞ n + 1
k∈H r=0

Since 0 ≤ w(k) < ∞ for all k ∈ H, the above inequality implies that there exists a
k ∈ H for which
n
1 X (r)
lim pik > 0. (4.18)
n→∞ n + 1
r=0
Using the fact that k → i, and arguments similar to those in Theorem 4.7, we can
show that
n
1 X (r)
lim pkk > 0. (4.19)
n→∞ n + 1
r=0
(See conceptual Exercise 4.12.) Hence state k is recurrent, as a consequence of part
(i) of Theorem 4.5. Since the DTMC is assumed irreducible, all the states are positive
recurrent.
Foster’s criterion is especially easy to apply since checking Equations 4.16 and
4.17 is a simple task. In many applications S = {0, 1, 2, · · ·} and usually ν(i) = i
suffices to derive useful sufficient conditions for positive recurrence. In this case we
have the following result, called Pakes’ lemma, whose proof we leave to the reader,
see Conceptual Exercise 4.13.

Theorem 4.11 Pakes’ Lemma. Let {Xn , n ≥ 0} be an irreducible DTMC on

S = {0, 1, 2, · · ·} and let
d(i) = E(Xn+1 − Xn |Xn = i), i ∈ S.
The DTMC is positive recurrent if

(i) d(i) < ∞ for all i ∈ S, and

(ii) lim supi∈S d(i) < 0.
 LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 105
Pakes’ lemma is useful in many DTMCs arising out of storage or queueing context.
We show its usefulness in the following example.

Example 4.17 Production-Inventory System: Batch Production. Consider the

DTMC of Example 4.15. Assume that µ, the mean production batch size is finite.
Then,
d(0) = E(Xn+1 − Xn |Xn = 0) = µ,
and
d(i) = E(Xn+1 − Xn |Xn = i) = µ − 1, i ≥ 1.
Hence, by Pakes’ lemma, the DTMC is positive recurrent if µ < 1. Note that Fos-
ter’s criterion or Pakes’ lemma does not say whether the DTMC is null recurrent or
transient if µ ≥ 1.

Theorem 4.11 is so intuitive that it is tempting to assume that we will get a suf-
ficient condition for transience if we reverse the inequality in condition (ii) of that
Theorem. Unfortunately, this turns out not to be true. However, we do get the follow-
ing more restricted result, which we state without proof:

Theorem 4.12 Let {Xn , n ≥ 0} be an irreducible DTMC on S = {0, 1, 2, · · ·}.

The DTMC is transient if there exists a k > 0 such that

(i) −k < d(i) < ∞ for all i ∈ S, and

(ii) lim supi∈S d(i) > 0.

We end this section with the remark that the recurrence and transience properties
of an infinite state DTMC are dependent on the actual magnitudes on the elements
of P , and not just whether they are zero or one. Using the concepts developed in this
section we shall study the limiting behavior of DTMCs in the next section.

4.5 Limiting Behavior of Irreducible DTMCs

In this section we derive the main results regarding the limiting distribution of an
irreducible DTMC {Xn , n ≥ 0} on state-space S = {0, 1, 2 · · ·} and transition
probability matrix P . We treat the four cases separately: the DTMC is transient, null
recurrent, aperiodic positive recurrent, and periodic positive recurrent.

4.5.1 The Transient Case

We begin with the main result in the following theorem.

Theorem 4.13 Transient DTMC. Let {Xn , n ≥ 0} be an irreducible transient

DTMC. Then
(n)
lim pij = 0, i, j ∈ S. (4.20)
n→∞
106 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Proof: Since the DTMC is transient, we have from Theorem 4.4,
∞
(n)
X
pjj < ∞, j ∈ S.
n=0
(n)
Since pjj ≥ 0, this implies that
(n)
lim p = 0, j ∈ S.
n→∞ jj

Now, consider a state i ∈ S, i 6= j. Since the DTMC is irreducible, j → i, and hence

(m)
there exists an m > 0 such that pij > 0. Hence, for n ≥ m,
(n)
X (m) (n−m)
pjj = pjr prj (Theorem 2.3)
r∈S
(m) (n−m)
≥ pji pij .
(n) (m)
Since pjj converges to zero, and pji > 0, the above inequality implies Equa-
tion 4.20.
By using dominated convergence theorem, we see that the above theorem implies
that
lim P(Xn = j) = 0, j ∈ S,
n→∞
or more generally, for any finite set A ⊂ S,
lim P(Xn ∈ A) = 0.
n→∞

Even more importantly, we can show that

∞
(n)
X
pij < ∞, (4.21)
n=0

which implies that

∞
X
P(Xn ∈ A) < ∞,
n=0
which implies that
P(Xn ∈ A infinitely often) = 0. (4.22)
See Conceptual Exercise 4.14. Thus a transient DTMC will eventually permanently
exit any finite set with probability 1.

4.5.2 The Discrete Renewal Theorem

We begin the study of the convergence results for recurrent DTMC with the discrete
renewal theorem. This is the discrete analog of a general theorem called the Key
Renewal Theorem, which is presented in Chapter 8. We first give the main result.
 LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 107
Theorem 4.14 Discrete Renewal Theorem. Let {un , n ≥ 1} be a sequence of real
numbers with
X∞
un ≥ 0, un = 1,
n=1
P∞
and let µ = n=1 nun . Let d, called the period, be the largest integer such that
∞
X
und = 1.
n=1

Let {νn , n ≥ 0} be another given sequence with

∞
X
|νn | < ∞. (4.23)
n=1

Suppose the sequence {gn , n ≥ 0} satisfies

n
X
gn = νn + um gn−m , n ≥ 0. (4.24)
m=1

(i) If d = 1, the sequence {gn , n ≥ 0} converges and

∞
1X
lim gn = νn . (4.25)
n→∞ µ n=0

(ii) If d > 1, the sequence {gn , n ≥ 0} has d convergent subsequences {gnd+k , n ≥

0} (0 ≤ k ≤ d − 1) and
∞
dX
lim gnd+k = νnd+k . (4.26)
n→∞ µ n=0

If µ = ∞, the limits are to be interpreted as 0.

Proof: (i). The hard part is to prove that the limit exists. We refer the reader to
Karlin and Taylor (1975) or Kohlas (1982) for the details. Here we assume that the
limit exists and show that it is given as stated in Equation 4.25.
First, it is possible to show by induction that
n
X
|gn | ≤ |νk |, n ≥ 0.
k=0

From Equation 4.23, we conclude that {gn , n ≥ 0} is a bounded sequence. Summing

Equation 4.24 from n = 0 to k we get
k
X k
X k X
X n
gn = νn + um gn−m , n ≥ 0.
n=0 n=0 n=0 m=1
108 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Rearranging the terms and using fn = ∞
P
k=n+1 uk , we get
k
X k
X
gk−r fr = νr .
r=0 r=0

Assuming limk→∞ gk = g exists, we use the dominated convergence theorem to get

∞
X k
X
νr = lim νr
k→∞
r=0 r=0
k
X
= lim gk−r fr
k→∞
r=0
∞
X
= g fr
r=0
X∞
= g rur
r=1
= gµ,

which gives the desired result.

(ii). From the definition of d it follows that ur = 0 if r is not an integer multiple

of d, and {u′r = urd , r ≥ 1} has period 1. We also have
∞
X ∞
X
µ′ = ru′r = rurd = µ/d.
r=0 r=0

Now, fix a 0 ≤ k < d and define

νn′ = νnd+k , gn′ = gnd+k , n ≥ 0.
Then, Equation 4.24 reduces to
n
X
gn′ = νn′ + u′r gn−r
′
.
r=0

Since {u′r , r ≥ 1} has period one, we can use Equation 4.25 to get
∞
1 X ′
lim gn′ = ν .
n→∞ µ′ n=0 n

This yields Equation 4.26 as desired.

The case d = 1 is called the aperiodic case, and the case d > 1 is called the
periodic case. Equation 4.24 is called the discrete renewal equation.
 LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 109
4.5.3 The Recurrent Case
(n)
The next Theorem shows that {pjj , n ≥ 0} satisfy a discrete renewal equation.

(n)
Theorem 4.15 Discrete Renewal Equation for pjj . Fix a j ∈ S. Let

T̃j = min{n > 0 : Xn = j}, j ∈ S,

(n)
gn = pjj , n ≥ 0,
ν0 = 1, νn = 0, n ≥ 1,
un = P(T̃j = n|X0 = j), n ≥ 1.
Then Equation 4.24 is satisfied.

Proof: We have
(0)
g0 = pjj = 1 = ν0 .
For n ≥ 1, conditioning on T̃j ,
(n)
gn = pjj = P(Xn = j|X0 = j)
n
X
= P(Xn = j|X0 = j, T̃j = m)P(T̃j = m)
m=1
Xn
= P(Xn = j|X0 = j, Xr 6= j, 1 ≤ r ≤ m − 1, Xm = j)ũj (m)
m=1
Xn
= um P(Xn = j|Xm = j)
m=1
Xn
= um gn−m ,
m=1

where we have used time homogeneity to get the last equality and Markov property
to get the one before that. This proves the theorem.
Using the above theorem we get the next important result.

Theorem 4.16 Limiting Behavior of Recurrent States. Let T̃j be as defined in

Equation 4.10 and
m̃j = E(T̃j |X0 = j).

(i) If state j is aperiodic

(n)
lim p = 1/m̃j .
n→∞ jj
(ii) If state j is periodic with period d > 1
(nd)
lim p = d/m̃j .
n→∞ jj

If m̃j = ∞ the limits are to be interpreted as 0.

110 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
(n)
Proof: We see from Theorem 4.16 that {pjj , n ≥ 0} satisfies the discrete renewal
equation. Since j is recurrent, we have
∞
X
P(T̃j = n|X0 = j) = 1,
n=1

and
∞
X
|νn | = ν0 = 1.
n=0
We also have
∞
X
µ= νn = E(T̃j |X0 = j) = m̃j .
n=1
Now suppose state j is aperiodic. Hence d = 1, and we can apply Theorem 4.14 part
(i). We get
∞
(n) 1X
lim pjj = νn = 1/m̃j .
n→∞ µ n=0
This yields part (i). Part (ii) follows similarly from part (ii) of Theorem 4.14.

4.5.4 The Null Recurrent Case

In this subsection we consider the limiting behavior of an irreducible null recurrent

DTMC. Such a DTMC necessarily has infinite state-space. The main result is given
by

Theorem 4.17 The Null Recurrent DTMC. For an irreducible null recurrent
DTMC
(n)
lim p = 0.
n→∞ ij

Proof: Since the DTMC is null recurrent, we know that

m̃j = ∞, j ∈ S.
Hence from part (ii) of Theorem 4.16 we see that
(nd)
lim p = 0.
n→∞ jj

Following the proof of Theorem 4.13 and using the assumption of irreducibility we
can show that
(n)
lim pij = 0, i, j ∈ S.
n→∞

This proves the theorem.

By using dominated convergence theorem we see that the above theorem implies
that
lim P(Xn = j) = 0, j ∈ S,
n→∞
LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 111
or more generally, for any finite set A ⊂ S,
lim P(Xn ∈ A) = 0.
n→∞

Unlike in the transient case, Theorem 4.4 implies that

∞
(n)
X
pij = ∞.
n=0

Since the DTMC is recurrent each state is visited infinitely often. Hence, in contrast
to the transient case,
P(Xn ∈ A infinitely often) = 1.
Thus a null DTMC will visit every finite set infinitely often over the infinite horizon,
even though the limiting probability that the DTMC is in the set A is zero. This non-
intuitive behavior is the result of the fact that although each state is visited infinitely
often, the expected time between two consecutive visits to the state is infinite.

4.5.5 The Positive Recurrent Aperiodic Case

In this subsection we assume that {Xn , n ≥ 0} is an irreducible positive recurrent

aperiodic DTMC. Such DTMCs are also called ergodic. Now, for a positive recurrent
DTMC, m̃j < ∞ for all j ∈ S. Hence, from part (i) of Theorem 4.16 we get
(n)
lim p = 1/m̃j > 0, j ∈ S.
n→∞ jj
(n)
The next theorem yields the limiting behavior of pij as n → ∞.

Theorem 4.18 The Positive Recurrent Aperiodic DTMC. For an irreducible pos-
itive recurrent DTMC

(n)
lim p = πj > 0, i, j ∈ S, (4.27)
n→∞ ij
where {πj , j ∈ S} are given by the unique solution to
X
πj = πi pij , j ∈ S, (4.28)
i∈S
X
πj = 1. (4.29)
j∈S

Proof: We have already seen that Equation 4.27 holds when i = j with πj =
1/m̃j > 0. Hence assume i 6= j. Following the proof of Theorem 4.15 we get
n
(n) (n−m)
X
pij = um pjj , n ≥ 0,
m=1

where
um = P(T̃j = m|X0 = i).
112 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Since i ↔ j, it follows that
∞
X
um = P(Xn = j for some n ≥ 0 |X0 = i) = 1.
m=1

Now let 0 < ǫ < 1 be given. Thus it is possible to pick an N such that
∞
X
um ≤ ǫ/2,
m=N +1

and
(n)
|pjj − πj | ≤ ǫ/2, for all n ≥ N.
Then, for n ≥ 2N , we get
n
(n) (n−m)
X
|pij − πj | = | um pjj − πj |
m=1
n−N n ∞
(n−m) (n−m)
X X X
= | um (pjj − πj ) + um (pjj − πj ) − um πj |
m=1 m=n−N +1 m=n+1
n−N n ∞
(n−m) (n−m)
X X X
≤ um |pjj − πj | + um pjj + um πj
m=1 m=n−N +1 m=n−N +1
n−N
X n
X ∞
X
≤ um ǫ/2 + um + um
m=1 m=n−N +1 m=n+1
≤ ǫ/2 + ǫ/2 ≤ ǫ.
(n)
This proves Equation 4.27. Next we derive Equations 4.28 and 4.29. Now let aj =
P(Xn = j). Then Equation 4.27 implies
(n)
lim a = πj , j ∈ S.
n→∞ j

Now, the Chapman-Kolmogorov equations 2.21 yield

(n+m)
X (m) (n)
aj = ai pij , n, m ≥ 0.
i∈S

Let m → ∞ on both sides. The interchange of the limit and the sum on the right
hand side is justified due to bounded convergence theorem. Hence we get
(n)
X
πj = πi pij .
i∈S

Equation 4.28 results from the above by setting n = 1. Now let n → ∞. Again,
bounded convergence theorem can be used to interchange the sum and the limit on
the right hand side to get !
X
πj = πi πj ,
i∈S
LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 113
P
but πj > 0. Hence we must have πi = 1, yielding Equation 4.29.
Now suppose {πi′ , i ∈ S} is another solution to Equations 4.28 and 4.29. Using the
same steps as before we get
(n)
X
πj′ = πi′ pij , n ≥ 0.
i∈S

Letting n → ∞ we get !
X
πj′ = πi′ πj = πj .
i∈S

Thus the Equations 4.28 and 4.29 have a unique solution.

Equation 4.28 is called the balance equation, since it balances the probability of
entering a state with the probability of exiting a state. Equation 4.29 is called the
normalizing equation, for obvious reasons. The solution {πj , j ∈ S} satisfying bal-
ance and the normalizing equations is called the limiting distribution, since it is the
limit of the distribution of Xn as n → ∞. It is also called the steady state distribu-
tion. It should be noted that it is the state-distribution that is steady, not the state itself.

Now suppose the DTMC starts with initial distribution

P(X0 = j) = πj , j ∈ S.
Then it can be shown that (see Conceptual Exercise 4.15)
P(Xn = j) = πj , j ∈ S, for all n ≥ 1.
Thus the distribution of Xn is independent of n if the DTMC starts with the initial
distribution {πj , j ∈ S}. Hence {πj , j ∈ S} is also called the stationary distribu-
tion of the DTMC.

As a consequence of Equation 4.27 (See Abel’s theorem in Marsden (1974)) we

have
n
1 X (r)
lim pij = πj .
n→∞ n + 1
r=0
Thus in a positive recurrent aperiodic DTMC, the limiting fraction of the time spent
in state j (called the limiting occupancy distribution) is also given by πj . Thus for
a DTMC the limiting distribution, the stationary distribution, and the limiting occu-
pancy distribution all coincide.

4.5.6 The Positive Recurrent Periodic Case

Let {Xn , n ≥ 0} be an irreducible positive recurrent DTMC with period d > 1.

From part (ii) of Theorem 4.16 we get
114 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR

(nd)
lim p = d/m̃j = dπj > 0, j ∈ S. (4.30)
n→∞ jj
Now let
αij (r) = P(T̃j (mod d) = r|X0 = i), 0 ≤ r ≤ d − 1
X∞
= P(T̃j = nd + r|X0 = i).
n=0
(n)
The next theorem shows that pij does not have a limit, but does have d convergent
subsequences.

Theorem 4.19 Let {Xn , n ≥ 0} be an irreducible positive recurrent DTMC with

period d > 1. Then
(nd+r)
lim pjj = dπj αij (r) (4.31)
n→∞
for i, j ∈ S and 0 ≤ r ≤ d − 1.

Proof: Follows along the same lines as that of Theorem 4.18 by writing
n
(nd+r) (n−k)d
X
pij = P(T̃j = kd + r|X0 = i)pjj
k=0

and using Equation 4.30 and the definition of αij (r).

The next theorem gives the result about the limiting occupancy distribution.

Theorem 4.20 Limiting Occupancy Distribution.

(i)
Mij (n + 1)
lim = πj , i, j ∈ S. (4.32)
n→∞ n+1
(ii) {πj , j ∈ S} are given by the unique solution to
X
πj = πi pij , j ∈ S, (4.33)
i∈S
X
πj = 1. (4.34)
j∈S

Proof: (i) As an easy consequence of Equation 4.31 we get

n
1 X (kd+r)
lim pij = dπj αij (r), i, j ∈ S.
n→∞ n + 1
k=0

(This just says that the Cesaro limit agrees with the usual limit when the latter exists.
See Marsden (1974).) Using this we get, for 0 ≤ m ≤ d − 1,
nd+m d−1 n−1
1 X (k) 1X d X (kd+r)
lim pij = lim pij
n→∞ nd + m d r=0 n→∞ nd + m
k=0 k=0
LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 115
m
1 X (nd+r)
+ lim pij
n→∞ nd + m r=0
d−1
1 X
= dπj αij (r) = πj .
d r=0
The Equation 4.32 follows from this.

(ii) We have
X Mij(n) X n
1 X (k)
pjm = pij pjm
n+1 n+1
j∈S j∈S k=0
n
1 X (k+1)
= pim
n+1
k=0
n+1
n + 2 1 X (k) 1 (0)
= pim − p
n+1n+2 n + 1 im
k=0
(n+1)
n+2 Mij 1 (0)
= − p .
n+1 n+2 n + 1 im
Letting n → ∞ on both sides of the above equation we get
X
πj pjm = πm ,
j∈S

which is Equation 4.33. By repeating the above step k times we get

(k)
X
πj pjm = πm , k ≥ 0.
j∈S

By summing the above equation over k = 0 to n and dividing by n + 1 we get

(n)
X Mjm
πj = πm .
n+1
j∈S

Now let n → ∞. Using Equation 4.32 we get

 
X
 πj  πm = πm .
j∈S

Since πm > 0 this implies Equation 4.34. Uniqueness can be proved in a manner
similar to the proof of Theorem 4.18.

4.5.7 Necessary and Sufficient Condition for Positive Recurrence

The next theorem provides a necessary and sufficient condition for positive recur-
rence. It also provides a sort of converse to Theorems 4.18 and 4.20.
 116 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Theorem 4.21 Positive Recurrence. Let {Xn , n ≥ 0} be an irreducible DTMC. It
is positive recurrent if and only if there is a non-negative solution to
X
πj = πi pij , j ∈ S, (4.35)
i∈S
X
πj = 1. (4.36)
j∈S

If there is a solution to the above equations, it is unique.

Proof: Theorems 4.18 and 4.20 give the “if” part. Here we prove the “only if” part.
Suppose {Xn , n ≥ 0} is an irreducible DTMC that is either null recurrent or tran-
sient. Suppose there is a non-negative solution to Equation 4.35. Then following the
same argument as before, we have
(n)
X
πj = πi pij , n ≥ 1.
i∈S

Letting n → ∞ and using Theorems 4.13 and 4.17 we get

(n) (n)
X X
πj = lim πi pij = πi lim pij = 0.
n→∞ n→∞
i∈S i∈S

Here the interchange of the limits and the sum is justified by dominated convergence
theorem. Thus the solution cannot satisfy Equation 4.36. This proves the Theorem.
Uniqueness follows from the uniqueness of the limiting distribution of the positive
recurrent DTMCs.

The above theorem is very useful. For irreducible DTMCs it allows us to directly
solve Equations 4.35 and 4.36 without first checking for positive recurrence. If we
can solve these equations, the DTMC is positive recurrent. Note that we do not insist
on aperiodicity in this result. However, the interpretation of the solution {πj , j ∈ S}
depends upon whether the DTMC is aperiodic or periodic. In an aperiodic DTMC
there are three possible interpretations:

(i) It is the limiting distribution of the DTMC,

(ii) It is the stationary distribution of the DTMC,
(iii) It is the limiting occupancy distribution for the DTMC.

When the DTMC is periodic, the second and the third interpretations continue to
hold, where as the first one fails, since a periodic DTMC does not have a limiting
distribution.

4.5.8 Examples

We end this section with two examples.

LIMITING BEHAVIOR OF IRREDUCIBLE DTMCS 117
Example 4.18 Two-State DTMC. Consider the two-sate DTMC of Example 4.1
on state-space {1, 2}. Assume that 0 < α + β < 2, so that the DTMC is positive
recurrent and aperiodic. The Equation 4.35 yields
π1 = απ1 + (1 − β)π2 , (4.37)
π2 = (1 − α)π1 + βπ2 . (4.38)
Note that these two equations are identical. Using the normalizing equation
π1 + π2 = 1 (4.39)
we get the following unique solution:
1−β 1−α
π1 = , π2 = .
2−α−β 2−α−β
We can verify Theorem 4.18 directly from the results of Example 4.1.

Example 4.19 Three-State DTMC. Consider the three state DTMC of Exam-
ple 4.2. The DTMC is periodic with period 2. Equations 4.35 and 4.36 yield:
π1 = qπ2
π2 = π1 + π3
π3 = pπ2
π1 + π2 + π3 = 1.
These have a unique solution given by
π1 = q/2, π2 = 1/2, π3 = p/2.
The above represents the stationary distribution, and the limiting occupancy distri-
bution of the DTMC. Since the DTMC is periodic, it has no limiting distribution.
These results are consistent with the results of Example 4.2. Using the matrix nota-
tion α(r) = [αij (r)], we see that
   
1 0 1 0 1 0
α(0) =  0 1 0  , α(1) =  1 0 1  .
1 0 1 0 1 0
Using this we can verify that Theorem 4.20 produces the results that are consistent
with Equations 4.2 and 4.3.

Example 4.20 Genotype Evolution. Consider the six state DTMC of Example 4.3.
This a reducible DTMC, and one can see that Equations 4.35 and 4.36 have an infinite
number of solutions. We shall deal with the reducible DTMCs in Section 4.7.

It should be clear by now that the study of the limiting behavior of irreducible
positive recurrent DTMCs involves solving the balance equations and the normal-
izing equation. If the state-space is finite and the transition probabilities are given
118 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
numerically, one can use standard numerical procedures to solve them. When the
DTMC has infinite state-space or the transition probabilities are given algebraically,
numerical methods cannot be used. In such cases one obtains the solution by analyt-
ical methods. The analytical methods work only if the transition probabilities have a
special structure. We shall describe various examples of analytical examples in the
next section.

4.6 Examples: Limiting Behavior of Infinite State-Space Irreducible DTMCs

In this section we consider several examples of infinite-state DTMCs with special

structures, and study their limiting behavior. The methods of solving the balance
and normalizing equations can generally be classified into two groups: the recursive
methods, and the generating function methods. In the recursive methods one uses
the balance equations to express each πi in terms of π0 , and then computes π0 using
the normalizing equation. In the generating function methods one obtains the gen-
erating function of the πi ’s, which in theory can be used to compute πi ’s. We have
encountered these two methods in Chapter 3.

Example 4.21 Success Runs. Consider the success runs DTMC of Example 2.15
on page 19 with transition probabilities
pi,0 = qi , pi,i+1 = pi , i = 0, 1, 2, · · · .
We assume that pi > 0 and qi > 0 for all i ≥ 0, making the DTMC positive recurrent
and aperiodic. The balance equations yield
πi+1 = pi πi , i ≥ 0.
Solving the above equation recursively yields
πi = ρi π0 , i ≥ 0,
where
ρ0 = 1, ρi = p0 p1 · · · pi−1 , i ≥ 1.
The normalizing equation yields
∞ ∞
!
X X
1= πi = π0 ρi .
i=0 i=0
P
Thus, if ρi converges, we have
∞
!−1
X
π0 = ρi .
i=0

Thus, from Theorem

P 4.20, we see that the success runs DTMC is positive recurrent
if and only if ρi converges. Combining with the results of Example 4.13 we get
the complete classification as follows: The success runs DTMC is
EXAMPLES: LIMITING BEHAVIOR OF INFINITE STATE-SPACE DTMCS 119
P
(i) positive recurrent if ρi < ∞,
P P
(ii) null recurrent if ρi = ∞ and qi = ∞,
P
(iii) transient qi < ∞.
When the DTMC is positive recurrent, the limiting distribution is given by
ρi
πi = P∞ , i ≥ 0.
j=0 ρj

This is also the stationary distribution and the limiting occupancy distribution of the
DTMC.

Special case. Suppose pi = p, for all i ≥ 0, and 0 < p < 1. Then

ρi = pi , i ≥ 0,
P
and hence ρi = 1/(1 − p) < ∞. Hence the DTMC is positive recurrent and its
limiting distribution is given by
πi = pi (1 − p), i ≥ 0.
Thus, in the limit, Xn is a modified geometric distribution with parameter 1 − p.

Example 4.22 General Simple Random Walk. Now consider the general simple
random walk with the following transition probabilities:
pi,i+1 = pi for i ≥ 0,
pi,i−1 = qi = 1 − pi for i ≥ 1,
p00 = 1 − p0 .
Assume that 0 < pi < 1 for all i ≥ 0, so that the DTMC is recurrent and aperiodic.
The balance equations for this DTMC are:
π0 = (1 − p0 )π0 + q1 π1 ,
πj = pj−1 πj−1 + qj+1 πj+1 , j ≥ 1.
It is relatively straightforward to prove by induction that the solution is given by (see
Conceptual Exercise 4.16).
πi = ρi π0 , i ≥ 0, (4.40)
where
p0 p1 · · · pi−1
ρ0 = 1, ρi = , i ≥ 1. (4.41)
q1 q2 · · · qi
The normalizing equation yields
∞ ∞
!
X X
1= πi = π0 ρi .
i=0 i=0
P
Thus, if ρi converges, we have
∞
!−1
X
π0 = ρi .
i=0
120 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Thus, from Theorem 4.20, P we see that the general simple random walk is positive
recurrent if and only if ρi converges. This is consistent with the results of Ex-
ample 4.14. When the DTMC is positive recurrent, the limiting distribution is given
by
ρi
πi = P∞ , i ≥ 0. (4.42)
j=0 ρj
This is also the stationary distribution and the limiting occupancy distribution of the
DTMC. Note that the DTMC is periodic with period 2 if p0 = 1. In this case the
expressions for πi remain valid, but now {πi , i ≥ 0} is not a limiting distribution.

Special Case 1: Suppose pN = 0, and pN,N = 1 − qN , for a given N ≥ 0. In this

case we can restrict our attention to the irreducible DTMC over {0, 1, 2, · · · , N }. In
this case ρi = 0 for i > N and the above results reduce to
ρi
πi = PN , 0 ≤ i ≤ N.
j=0 ρj

Special case 2. Suppose pi = p for all i ≥ 0, and 0 < p < 1. In this case the
DTMC is positive recurrent and aperiodic, and we have
ρi = ρi , i ≥ 0,
P
where ρ = p/q. Hence ρi converges if p < q and diverges if p ≥ q. Combining
this with results from Example 4.14, we see that the space homogeneous simple
random walk is
(i) positive recurrent if p < q,
(ii) null recurrent if p = q,
(iii) transient p > q.
In case p < q, the limiting distribution is given by
πi = ρi (1 − ρ), i ≥ 0.
Thus, in the limit, Xn is a modified geometric random variable with parameter
1 − ρ.

Example 4.23 Production-Inventory System: Batch Production. Consider the

DTMC of Example 2.16 on page 19. It has state-space S = {0, 1, 2, · · ·} and transi-
tion probabilities:
p0,j = αj , j ≥ 0,
pij = αj−i+1 , j ≥ i − 1 ≥ 0,
P
where αj ≥ 0 and αj = 1. We assume that α0 > 0, α0 + α1 < 1 so that the
DTMC is irreducible and aperiodic. The balance equations for this DTMC are given
by
X i
πi = αj πi−j+1 + αi π0 , i ≥ 0. (4.43)
j=0
EXAMPLES: LIMITING BEHAVIOR OF INFINITE STATE-SPACE DTMCS 121
These equations look quite similar to Equation 3.21 on page 68. Hence we follow a
similar generating function approach to solve them. Thus we shall obtain
∞
X
φ(z) = z i πi
i=0

in terms of the known generating function

∞
X
ψ(z) = z i αi .
i=0

Following the steps as in Example 3.12 we get

ψ(z)(1 − z)
φ(z) = π0 .
ψ(z) − z
The normalizing equation yields
∞
X
1= πi = lim φ(z)
z→1
i=0

which can be used to compute π0 as follows: We have

ψ(z)(1 − z)
1 = π0 lim
ψ(z) − z
z→1
−1
= π0 ′ , (by L’Hopital’s rule)
ψ (1) − 1
where
∞
d X
ψ ′ (1) = ψ(z)|z=1 = kαk = µ.
dz
k=0
Thus, if µ < 1, we have π0 = 1 − µ > 0. Hence from Theorem 4.20, we see that the
DTMC is positive recurrent if and only if µ < 1. This is consistent with the results
of Example 4.15. When the DTMC is positive recurrent, the generating function of
its limiting distribution is given by
ψ(z)(1 − z)
φ(z) = (1 − µ) . (4.44)
ψ(z) − z
We also get
π0 = 1 − µ
as the long run probability that the system is empty. A numerical way of computing
the limiting distribution is described in Computational Exercise 4.6.

Example 4.24 Production-Inventory System: Batch Demands. Consider the

DTMC of Example 4.16. It has state-space S = {0, 1, 2, · · ·} and transition prob-
abilities:
pi,j = αi−j+1 , 0 < j ≤ i + 1,
122 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
∞
X
pi0 = βi = αk , i ≥ 0,
k=i+1
P
where αj ≥ 0 and αj = 1. We assume that α0 > 0, α0 + α1 < 1 so that the
DTMC is irreducible. The balance equations for this DTMC are given by
∞
X
πi = αj πi+j−1 , i ≥ 1.
j=0

These equations look the same as Equation 3.20 on page 67. Using the results of
Example 3.11, we see that the solution to the balance equations is given by
πi = cρi , i ≥ 0,
where c > 0 is a constant, and ρ satisfies
∞
X
ρ = ψ(ρ) = αk ρk . (4.45)
k=0

From Example 3.10, we know that there is a solution to the above equation in (0,1)
if an only if µ > 1. This is then the necessary and sufficient condition for positive
recurrence. This is consistent with the results of Example 4.16. When the DTMC is
positive recurrent (i.e., there is a unique ρ < 1 satisfying the above equation), the
normalizing equation yields
∞ ∞
X X c
1= πi = c ρi = .
i=0 i=0
1−ρ
This yields the limiting distribution as

πi = (1 − ρ)ρi , i ≥ 0. (4.46)
Thus, in the limit, Xn is a modified geometric random variable with parameter
1 − ρ.

4.7 Limiting Behavior of Reducible DTMCs

In this section we consider a reducible DTMC {Xn , n ≥ 0} with state-space S.

Assume that there are k closed communicating classes Ci , 1 ≤ i ≤ r, and T is
the set of states that do not belong to any closed communicating class. Then, as in
Equation 4.9, the state-space is partitioned as follows :
S = C1 ∪ C2 ∪ · · · ∪ Ck ∪ T.
Now, if i ∈ Cr and j ∈ Cs , with r 6= s, then i is not accessible from j and vice
versa. Hence pij = pji = 0. Now, relabel the states in S by integers such that i ∈ Cr
and j ∈ Cs with r < s implies that i < j, and i ∈ Cr and j ∈ T implies that i < j.
LIMITING BEHAVIOR OF REDUCIBLE DTMCS 123
With this relabeling, the matrix P has the following canonical block structure:
 
P (1) 0 ··· 0 0
 0 P (2) · · · 0 0 
 .. .. . . .. ..  .
 
P = . . . . .  (4.47)
 
 0 0 · · · P (k) 0 
D Q

Here P (i) is a |Ci | × |Ci | stochastic matrix (1 ≤ i ≤ k), Q is a |T | × |T | sub-

stochastic matrix (i.e., all row sums of Q being less than or equal to one, with at least
one being strictly less than one), and D is a |T | × |S − T | matrix. Elementary matrix
algebra shows that the nth power of P has the following structure:
P (1)n 0
 
··· 0 0
 0 P (2)n · · · 0 0 
 .. . . ..
 
n
P = . .. . . . .. .

 . 
 0 n
0 · · · P (k) 0 
Dn Qn
Since P (r) (1 ≤ r ≤ k) is a transition probability matrix of an irreducible DTMC
with state space Cr , we already know how P (r)n behaves as n → ∞. Similarly,
since all states in T are transient, we know that Qn → 0 as n → ∞. Thus the study
of the limiting behavior of P n reduces to the study of the limiting behavior of Dn as
n → ∞. This is what we proceed to do.

Let T (r) be the first passage time to visit the set Cr , i.e.,
T (r) = min{n ≥ 0 : Xn ∈ Cr }, 1 ≤ r ≤ k.
Let
ui (r) = P(T (r) < ∞|X0 = i), 1 ≤ r ≤ k, i ∈ T. (4.48)
The next theorem gives a method of computing the above probabilities.

Theorem 4.22 Absorption Probabilities. The quantities {ui (r), i ∈ T, 1 ≤ r ≤

k} are given by the smallest solution to
X X
ui (r) = pij + pij uj (r). (4.49)
j∈Cr j∈T

Proof: This proof is similar to the proof of Theorem 3.2 on page 60. See also Con-
ceptual Exercise 3.1.
Using the quantities {ui (r), i ∈ T, 1 ≤ r ≤ k} we can describe the limiting
distribution of Dn as n → ∞. This is done in the theorem below.

Theorem 4.23 Limit of Dn . Let {ui (r), i ∈ T, 1 ≤ r ≤ k} be as defined in

Equation 4.48. Let i ∈ T and j ∈ Cr .
 124 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
(i) If Cr is transient or null recurrent,
lim Dn (i, j) = 0. (4.50)
n→∞

(ii) If Cr is positive recurrent aperiodic,

lim Dn (i, j) = ui (r)πj ,
n→∞

where {πj , j ∈ Cr } is the unique solution to

X
πj = πm pmj ,
m∈Cr
X
πm = 1.
m∈Cr

(iii) If Cr is positive recurrent and periodic, Dn (i, j) does not have a limit. However,
n
1 X
lim Dm (i, j) = ui (r)πj ,
n→∞ n + 1
m=0

where {πj , j ∈ Cr } is as in part (ii) above.

Proof: (i) If ui (r) = 0, then Dn (i, j) = 0 for all n ≥ 1, and hence Equation 4.50
follows. If ui (r) > 0, then it is possible to go from i to any state j ∈ Cr , i.e., there is
(m)
an m ≥ 1 such that pij = Dm (i, j) > 0. Since state j is null recurrent or transient,
(n)
pjj → 0 as n → ∞. The proof follows along the same lines as that of Theorem 4.13.
(ii) Let i ∈ T and j ∈ Cr be fixed. Following Theorem 4.18 let
T̃j = min{n > 0 : Xn = j},
and
um = P(T̃j = m|X0 = i).
Then, since Cr is a closed recurrent class T̃j < ∞ if and only if T (r) < ∞. Hence
∞
X
um = αi (r).
m=1

Thus, given an ǫ > 0, it is possible to find an N > 0 such that

∞
X
um − αi (r) ≤ ǫ/2
m=N +1

and
(m)
|pjj − πj | ≤ ǫ/2
for all m ≥ N . The rest of the proof is similar to that of Theorem 4.18.
(iii) Similar to the proof of Theorem 4.20.
We illustrate with the help of several examples.
LIMITING BEHAVIOR OF REDUCIBLE DTMCS 125
Example 4.25 Genotype Evolution Model. Consider the six-state DTMC of the
genotype evolution model with the transition probability matrix as given in Equa-
tion 2.13 on page 24. From Example 4.6 we see that this is a reducible DTMC
with two closed communicating classes C1 = {1}, C2 = {6}, and the open class
T = {2, 3, 4, 5}. We also have
P (1) = [1], P (2) = [1],
 
1/2 0 1/4 0
 0 0 1 0 
Q=
 ,
1/4 1/8 1/4 1/4 
0 0 1/4 1/2
and  
1/4 0
 0 0 
 1/16 1/16  .
D= 

0 1/4
Remember that the rows of D are indexed {2, 3, 4, 5} and the columns are indexed
{1, 6}.
We have
P (1)n → [1], P (2)n → [1],
 
0 0 0 0
n
 0 0 0 0 
Q →  .
0 0 0 0 
0 0 0 0
Equations for {ui (1), i ∈ T } are given by
u2 (1) = .25 + .5u2 (1) + .25u4 (1)
u3 (1) = u4 (1)
u4 (1) = .25u2 (1) + .125u3 (1) + .25u4 (1) + .25u5(1)
u5 (1) = .25u4 (1) + .5u5 (1).
The solution is given by
[u2 (1), u3 (1), u4 (1), u5 (1)]′ = [.75, .5, .5, .25]′ .
Similar calculations yield
[u2 (2), u3 (2), u4 (2), u5 (2)]′ = [.25, .5, .5, .75]′ .
Hence we get  
3/4 1/4
 1/2 1/2 
Dn → 
 1/2
,
1/2 
1/4 3/4
126 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
and  
1 0 0 0 0 0

 3/4 0 0 0 0 1/4 

1/2 0 0 0 0 1/2
lim P (n)
 
= .
n→∞ 
 1/2 0 0 0 0 1/2 

 1/4 0 0 0 0 3/4 
0 0 0 0 0 1
This matches the result in Example 4.6.

4.8 DTMCs with Costs and Rewards

Let Xn be the state of a system at time n. Suppose {Xn , n ≥ 0} is a DTMC with

state-space S and transition probability matrix P . Furthermore, the system incurs an
expected cost of c(i) at time n if Xn = i. For other cost models, see Conceptual
Exercises 4.17 and 4.18. Rewards can be thought of as negative costs. We consider
costs incurred over infinite horizon {n ≥ 0}. For the analysis of costs over finite
horizon, see Conceptual Exercises 4.19 and 4.20.

4.8.1 Discounted Costs

Suppose the costs are discounted at rate α, where 0 ≤ α < 1 is a fixed discount
factor. Thus if the system incurs a cost of c at time n, its present value at time 0 is
αn c, i.e., it is equivalent to incurring a cost of αn c at time zero. Let C be the total
discounted cost over the infinite horizon, i.e.,
∞
X
C= αn c(Xn ).
n=0

Let φ(i) be the expected total discounted cost (ETDC) incurred over the infinite
horizon starting with X0 = i. That is,
φ(i) = E(C|X0 = i).
The next theorem gives the main result regarding the ETDC. We introduce the fol-
lowing column vectors
c = [c(i)]i∈S , φ = [φ(i)]i∈S .

Theorem 4.24 ETDC. Suppose 0 ≤ α < 1. Then φ is given by

φ = (I − αP )−1 c. (4.51)

Proof: Let C1 be the total discounted cost incurred over {1, 2, · · ·} discounted
back to time 0, i.e.,
X∞
C1 = αn c(Xn ).
n=1
DTMCS WITH COSTS AND REWARDS 127
From time-homogeneity it is clear that
E(C1 |X1 = j) = αφ(j), j ∈ S.
Using the first step analysis we get
X
E(C1 |X0 = i) = pij E(C1 |X0 = i, X1 = j)
j∈S
X
= α pij φ(j).
j∈S

Hence,
φ(i) = E(C|X0 = i)
= E(c(X0 ) + C1 |X0 = i)
X
= c(i) + α pij φ(j).
j∈S

In matrix form the above equation becomes

φ = c + αP φ. (4.52)
The matrix I − αP is invertible for 0 ≤ α < 1. Hence we get Equation 4.51.
Note that there is no assumption of transience or recurrence or periodicity or irre-
ducibility behind the above theorem. Equation 4.51 is valid for any transition proba-
bility matrix P .

Example 4.26 Two-State Machine. Consider the two-state machine of Exam-

ple 2.6 on page 15. It was modeled by a DTMC {Xn , n ≥ 0} with state-space
{0, 1} (0 being down, and 1 being up), and transition probability matrix
 
pd 1 − pd
P = ,
1 − pu pu
where 0 ≤ pu , pd ≤ 1. Now suppose the machine produces a revenue of $r per day
when it is up, and it costs $d in repair costs per day when the machine is down. Sup-
pose a new machine in working order costs $m. Is it profitable to purchase it if the
discount factor is 0 ≤ α < 1?

Let c(i) be the expected cost of visiting state i. We have

c = [c(0) c(1)]′ = [d − r]′ .
Then, using Theorem 4.24 we get
φ = [φ(0) φ(1)]′ = (I − αP )−1 c.
Direct calculations yield
 
1 d(1 − αpu ) − rα(1 − pd )
φ= .
(1 − αpu )(1 − αpd ) − α2 (1 − pu )(1 − pd ) d(1 − pu ) − r(1 − αpd )
128 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Thus it is profitable to buy a new machine if the expected total discounted net revenue
from a new machine over the infinite horizon is greater than the initial purchase price
of m, i.e., if
r(1 − αpd ) − d(1 − pu )
m≤ .
(1 − αpu )(1 − αpd ) − α2 (1 − pu )(1 − pd )
Note that the above inequality reduces to m ≤ r if α = 0, as expected. How much
should you be willing to pay for a machine in down state?

4.8.2 Average Costs

The discounted costs have the disadvantage that they depend upon the discount
factor and the initial state, thus making decision making more complicated. These
issues are addressed by considering the long run cost per period, called the aver-
agePcost. The expected total cost up to time N , starting from state i, is given by
E( N n=0 c(Xn )|X0 = i). Dividing it by N + 1 gives the cost per period. Hence the
long run expected cost per period is given by:
N
!
1 X
g(i) = lim E c(Xn )|X0 = i ,
N →∞ N + 1
n=0

assuming that the above limit exists. To keep the analysis simple, we will assume that
the DTMC is irreducible and positive recurrent with limiting occupancy distribution
given by {πj , j ∈ S}.
P Intuitively, it makes sense that the long run cost per period
should be given by πj c(j), independent of the initial state i. This intuition is
formally proved in the next theorem:

Theorem 4.25 Average Cost. Suppose {Xn , n ≥ 0} is an irreducible positive

recurrent DTMC with limiting occupancy distribution {πj , j ∈ S}. Suppose
X
πj |c(j)| < ∞.
j∈S

Then X
g(i) = g = πj c(j).
j∈S

(N )
Proof: Let Mij be the expected number of visits to state j over {0, 1, 2, · · · , N }
starting from state i. See Section 2.5. Then, we see that
1 X (N )
g(i) = lim Mij c(j)
N →∞ N + 1
j∈S

X Mij(N )
= lim c(j)
N →∞ N +1
j∈S
REVERSIBILITY 129
(N )
X Mij
= lim c(j)
N →∞ N +1
j∈S
X
= πj c(j).
j∈S

Here the last interchange of sum and the limit is allowed because the DTMC is posi-
tive recurrent. The last equality follows from Theorem 4.20.
We illustrate with an example.

Example 4.27 Brand Switching. Consider the model of brand switching as de-
scribed in Example 2.14, where a customer chooses among three brands of beer, say
A, B, and C, every week when he buys a six-pack. Let Xn be the brand he purchases
in week n. We assume that {Xn , n ≥ 0} is a DTMC with state-space S = {A, B, C}
and transition probability matrix given below:
 
0.1 0.2 0.7
P =  0.2 0.4 0.4  . (4.53)
0.1 0.3 0.6
Now suppose a six pack costs $6.00 for brand A, $5.00 for brand B, and $4 for brand
C. What is the weekly expenditure on beer by the customer in the long run?
We have
c(A) = 6, c(B) = 5, c(C) = 4.
Also solving the balance and normalizing equations we get
πA = 0.132, π2 = 0.319, πC = 0.549.
Hence the long run cost per week is
g = 6πA + 5πB + 4πC = 4.583.
Thus the customer spends $4.58 per week on beer.

It is possible to use the results in Section 4.7 to extend this analysis to reducible
DTMCs. However, the long run cost rate may depend upon the initial state in that
case.

4.9 Reversibility

In this section we study a special class of DMTCs called the reversible DTMCs.
Intuitively, if we watch a movie of a reversible DTMC we will not be able to tell
whether the time is running forward or backward. Thus, the probability of traversing
a cycle of r + 1 states i0 → i1 → i2 → · · · → ir−1 → ir → i0 is the same as
traversing it in reverse order i0 → ir → ir−1 → · · · → i2 → i1 → i0 . We make this
more precise in the definition below.
130 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
We begin with the definition:

Definition 4.12 Reversibility. A DTMC with state-space S and transition proba-

bility matrix P , is called reversible if for every r ≥ 1, and i0 , i1 , · · · , ir ,
pi0 ,i1 pi1 ,i2 · · · pir−1 ,ir pir ,i0 = pi0 ,ir pir ,ir−1 pir−1 ,ir−2 · · · pi1 ,i0 . (4.54)

The next theorem gives a simple necessary and sufficient condition for reversibil-
ity.

Theorem 4.26 An irreducible, positive recurrent DTMC with state-space S, transi-

tion probability matrix P and stationary distribution {πi , i ∈ S} is reversible if and
only if
πi pij = πj pji , i, j ∈ S. (4.55)

Proof: Suppose the DTMC is irreducible and positive recurrent, and Equation 4.55
holds. Consider a cycle of states i0 , i1 , · · · , ir , i0 . Then we get
πi0 pi0 ,i1 pi1 ,i2 · · · pir−1 ,ir pir ,i0 = pi1 ,i0 πi1 pi1 ,i2 · · · pir−1 ,ir pir ,i0
..
.
= pi1 ,i0 pi2 ,i1 · · · pir ,ir−1 pi0 ,ir πi0 .
Since the DTMC is positive recurrent, πi0 > 0. Hence the above equation implies
Equation 4.54.
To prove necessity, suppose Equation 4.54 holds. Summing over paths of length r
from i1 to i0 we get
(r) (r)
pi0 ,i1 pi1 ,i0 = pi0 ,i1 pi1 ,i0 .
Now sum over r = 0 to n, and divide by n + 1 to get
n n
! !
1 X (r) 1 X (r)
pi0 ,i1 p = p pi1 ,i0 ,
n + 1 r=0 i1 ,i0 n + 1 r=0 i0 ,i1
or
(n) (n)
Mi1 ,i0 Mi0 ,i1
pi0 ,i1
= pi ,i .
n+1 n+1 1 0
Now let n → ∞. Since the DTMC is irreducible and positive recurrent, we can use
Theorem 4.20 to get
pi0 ,i1 πi0 = pi1 ,i0 πi1 ,
which is Equation 4.55.
The Equations 4.55 are called the local balance or detailed balance equations, as
opposed to Equations 4.35, which are called global balance equations. Intuitively, the
local balance equations say that, in steady state, the expected number of transitions
from state i to j per period is the same as the expected number of transitions per
period from j to i. This is in contrast to stationary DTMCs that are not reversible:
REVERSIBILITY 131
for such DTMCs the global balance equations imply that the expected number of
transitions out of a state over any time period is the same as the expected number
of transitions into that state over the same time period. It can be shown that the
local balance equations imply global balance equations, but not the other way. See
Conceptual Exercise 4.21.

Example 4.28 Symmetric DTMCs. Consider an irreducible DTMC on a finite

state-space {1, 2, · · · , N } with symmetric transition probability matrix P . Show that
it is reversible.

Since P is symmetric, it must be doubly stochastic. See Conceptual Exercise 4.22.

Since the DTMC is assumed to be irreducible, it has a unique stationary distribution
given by
πj = 1/N, j ∈ S.
Thus
πi pij = (1/N )pij = (1/N )pji = πj pji , i, j ∈ S.
Here the second equality is a consequence of symmetry. Thus the DTMC satisfies
local balance equations, and hence is reversible.

Example 4.29 General Simple Random Walk. Consider a positive recurrent gen-
eral simple random walk as described in Example 4.22. Show that it is a reversible
DTMC.
From Example 4.22 we see that the stationary distribution is given by
πi = ρi π0 , i ≥ 0,
where
p0 p1 · · · pi−1
ρ0 = 1, ρi = , i ≥ 1.
q1 q2 · · · qi
Hence we have
p0 p1 · · · pi−1 p0 p1 · · · pi−1 pi
πi pi,i+1 = π0 pi = π0 qi+1 = πi+1 pi+1,i .
q1 q2 · · · qi q1 q2 · · · qi+1
Since the only transitions in a simple random walk are from i to i + 1, and i to i − 1,
we see that Equations 4.55 are satisfied. Hence the DTMC is reversible.

Reversible DTMCs are nice because it is particularly easy to compute their station-
ary distribution. A large class of reversible DTMCs are the tree DTMCs, of which
the general simple random walk is a special case. See Conceptual Exercise 4.23 for
details.
We end this section with an interesting result about the eigenvalues of a reversible
DTMC with finite state space.

Theorem 4.27 Eigenvalues and Reversibility. Let {Xn , n ≥ 0} be an irreducible,

positive recurrent, reversible DTMC with finite state-space S = {1, 2, · · · , N } and
transition probability matrix P . Then all the eigenvalues of P are real.
 132 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Proof: Since {Xn , n ≥ 0} is irreducible, positive recurrent , and reversible, we have
πi pij = πj pji ,
where {πi , i ∈ S} is the stationary distribution. Now let Π = diag[π1 , π2 , · · · , πN ].
Then the above equation can be written in matrix form as
ΠP = P T Π,
where P T is the transpose of P . This can be rewritten as
Π1/2 P π −1/2 = Π−1/2 P T Π1/2 . (4.56)
Now let λ be an eigenvalue of Π1/2 P Π−1/2 and x be the corresponding eigenvector,
i.e.,
Π1/2 P Π−1/2 x = λx.
The above equation can be written as
P Π−1/2 x = λΠ−1/2 x.
Thus λ is an eigenvalue of P . Similarly, any eigenvalue of P is an eigenvalue of
Π1/2 P Π−1/2 . However, Equation 4.56 implies that Π1/2 P Π−1/2 is symmetric, and
hence all its eigenvalues are real. Thus all eigenvalues of P must be real.

4.10 Computational Exercises

4.1 Numerically study the limiting behavior of the marginal distribution and the
occupancy distribution in the brand switching example of Example 2.14 on page 18,
by studying P n and M (n) /(n + 1) for n = 1 to 20.
4.2 Study the limiting behavior of the two-state DTMC of Example 2.3 if
(i) α + β = 0,
(ii) α + β = 2.
4.3 Numerically study the limiting behavior of the random walk on {0, 1, · · · , N }
with
p0,1 = p0,0 = pN,N = pN,N −1 = 1/2, pi,i+1 = pi,i−1 = 1/2, 1 ≤ i ≤ N − 1,
by studying P n and M (n) /(n + 1) for n = 1 to 20, and N = 1 to 5. What is your
guess for the limiting values a general N ?
4.4 Establish the conditions of transience, null recurrence, and positive recurrence
for the space-nonhomogeneous simple random walk on {0, 1, 2, · · ·} with the follow-
ing transition probabilities:
pi,i+1 = pi for i ≥ 0,
pi,i−1 = qi for i ≥ 1,
pi,i = ri , for i ≥ 0.
Assume that 0 < pi < 1 − ri for all i ≥ 0.
COMPUTATIONAL EXERCISES 133
4.5 Establish the conditions of transience, null recurrence, and positive recurrence
for the simple random walk on {0, 1, 2, · · ·} with reflecting boundary at zero, i.e.,
pi,i+1 = p for i ≥ 1,
pi,i−1 = q for i ≥ 1,
p0,1 = 1.
Assume that 0 < p < p + q < 1.

4.6 Consider the DTMC of example 4.15. Suppose it is positive recurrent. Show
that the limiting distribution {πj , j ≥ 0} can be recursively calculated by
π0 = 1 − µ,
1 − α0
π1 = π0 ,
α0
1 − α0 − α1
π2 = (π0 + π1 ),
α0
j
!
1 − ji=0 αi X
P
πj+1 = πi
α0 i=0
j j
X X αk
+ πi , j ≥ 2.
i=2
α0
k=j−i+2

This is a stable recursion since it involves adding positive terms.

4.7 Study the limiting behavior of the random walk in Computational Exercise 4.5,
assuming it is positive recurrent.

4.8 Study the limiting behavior of the random walk in Computational Exercise 4.4,
assuming it is positive recurrent.

4.9 Classify the states of the brand switching model of Example 4.27. That is, iden-
tify the communicating classes, and state whether they are transient, null recurrent,
or positive recurrent, and whether they are periodic or aperiodic.

4.10 Study the limiting behavior of P n as n → ∞ for the following transition

probability matrices by using the method of diagonalization to compute P n and then
letting n → ∞.
   
.5 .5 0 0 0 1 0 0
 .5 0 .5 0   .5 0 .5 0 
(a) 
 0 .5 0 .5  , (b)  0 .5 0 .5  .
  

0 0 .5 .5 0 0 1 0

4.11 Compute the limits of M (n) /(n + 1) as n → ∞ for the P matrices in Compu-
tational Exercise 4.10, using the method of diagonalization to compute the P n .
 134 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
4.12 Do a complete classification of the states of the DTMC in

(i) Example 2.13 on page 18,

(ii) The Wright-Fisher model on page 24,
(iii) The Moran model on page 25.

4.13 Do a complete classification of the states of the DTMCs with the following
transition probability matrices:
   
.5 .5 0 0 .2 .8 0 0
 0 .5 .5 0   .6 .4 0 0 
(a)  0 0 .5
 , (b)  .
.5   .2 .3 .5 0 
0 0 0 1 0 0 0 1

4.14 Consider a DTMC on all integers with following transition probabilities

pi,i+1 = αi , pi,0 = 1 − αi , i > 0,
pi,i−1 = βi , pi,0 = 1 − βi , i < 0,
po1 = p, p0,−1 = 1 − p.
Find the condition for transience, null recurrence, and positive recurrence.

4.15 Classify the following success runs DTMCs as transient, null recurrent, or pos-
itive recurrent

(i) pi = 1/(1 + i), i ≥ 0,

(ii) q0 = 0, qi = 1/(1 + i), i ≥ 1,

1/(1 + i) if i even,
(iii) pi =
i/(1 + i) if i odd.

4.16 State whether the following simple random walks are transient, null recurrent,
or positive recurrent.

(i) p01 = 1, pi,i+1 = p, pi,i−1 = q = 1 − p, i ≥ 1,

(ii) p01 = 1, pi,i+1 = 1/(1 + i), pi,i−1 = i/(1 + i), i ≥ 1,
(iii) p01 = 1, pi,i+1 = i/(1 + i), pi,i−1 = 1/(1 + i), i ≥ 1.

4.17 Consider a DTMC on non-negative integers with following transition proba-

bilities
p0,i = pi , i ≥ 0,
pi,i−1 = 1, i ≥ 1.
P
Assume that pi = 1. Establish the condition for transience, null recurrence, and
positive recurrence.

4.18 Use Pakes’ lemma to derive a sufficient condition for the positive recurrence
of the DTMC in Example 2.17 on page 20.
COMPUTATIONAL EXERCISES 135
4.19 Show that the discrete time queue of Example 2.12 is positive recurrent if
p < q, null recurrent if p = q, and transient if p > q.

4.20 Compute the limiting distribution of the DTMC of Computational Exer-

cise 2.9.

4.21 Compute the limiting distributions and the limiting occupancy distributions of
the DTMCs in Computational Exercise 4.10.

4.22 Is the DTMC in Computational Exercise 3.9 positive recurrent? If so, compute
its limiting distribution.

4.23 Numerically compute the limiting distribution of the number of passengers

on the bus in the Modeling Exercise 2.21. Assume that the bus capacity is 20, the
number passengers waiting for the bust at any stop is a Poisson random variable with
mean 10, and that a passenger on the bus alights at any stop with probability .4.

4.24 Consider the following modification of the transition probabilities of the

DTMC of Example 2.16:
p0j = βj , j ≥ 0
pij = αj−i+1 , j ≥ i − 1 ≥ 0,
P∞ P∞ P∞ P∞
with αj , βj ≥ 0, j=0 αj = j=0 βj = 1, µ = j=0 jαj , ν = j=0 jβj .
Assume the chain is irreducible. Show that this DTMC is positive recurrent if and
only if µ < 1. Let {πj , j ∈ S} be the limiting distribution, assuming it is positive
recurrent. Show that
∞
X A(z) − zB(z)
φ(z) = z j πj = π0 ,
j=0
A(z) − z

where
∞
X ∞
X
A(z) = z j αj , B(z) = z j βj ,
j=0 j=0
and
1−µ
π0 = .
1−µ+ν
4.25 Compute the long run fraction of patients who receive drug i if we follow the
play the winner policy of Modeling Exercise 2.13.

4.26 Let S be a set of all permutations of the integers 1, 2, · · · , N . Let X0 =

(1, 2, · · · , N ) be the initial permutation. Construct Xn+1 from Xn by interchang-
ing the I-th and J-th component of Xn , where I and J are independent and uni-
formly distributed over {1, 2, · · · , N }. (If I = J, then Xn+1 = Xn .) Show that
{Xn , n ≥ 0} is a DTMC and compute its limiting distribution. (Conceptual Exer-
cise 4.22 may be useful here.) This gives one method of generating a permutation
uniformly from the set of all permutations.
136 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
4.27 Compute the limiting distribution of the DTMC in Example 2.17 in the fol-
lowing cases

(i) αi = αi (1 − α), i ≥ 0, for a fixed constant α > 1/2,

1
(ii) αi = m+1 , 0 ≤ i ≤ m, for a constant m > 2.

You may need to solve Equation 4.45 numerically.

4.28 Let Xn be the sum of the first n outcomes of tossing a six-sided fair die re-
peatedly and independently. Compute
lim P(Xn is dividible by 7).
n→∞

4.29 (This is a generalization of Computational Exercise 4.28.) Let {Yn , n ≥ 1}

be a sequence of iid random variables with common pmf P(Yn = k) = αk , k =
1, 2, 3, · · ·. Let Xn = Y1 + · · · + Yn . Compute
lim P(Xn is dividible by r),
n→∞

where r is fixed positive integer.

4.30 Consider the discrete time queue of Example 2.12 on page 17. Assume that the
queue is positive recurrent and compute its limiting distribution.

4.31 Compute the limiting distribution of the number of white balls in urn A in the
urn model described in Example 2.13 on page 18.

4.32 Consider the Modeling Exercise 2.18 on page 45. Assume that u, the expected
up time and d, the expected down time are finite.

(i) Is this DTMC irreducible? Positive recurrent? Aperiodic?

(ii) Show that the long run probability that the machine is up is given by u/(u + d).

4.33 Consider the Modeling Exercise 2.1 on page 42. Assume that τ , the expected
lifetime of each light bulb, is finite. Show that the DTMC is positive recurrent and
compute its limiting distribution.

4.34 Consider the Modeling Exercise 2.19 on page 46. Let µ = E(Yn ), and
v =Var(Yn ). Suppose X0 = 0 with probability 1. Compute µn = E(Xn ) as a func-
tion of n.

4.35 Show that the DTMC in Modeling Exercise 2.19 on page 46 is positive recur-
rent if µ < ∞. Let ψ(z) be the g.f. of Yn and φn (z) be the g.f. of Xn . Compute φ(z),
the generating function of the limiting distribution of Xn .

4.36 Consider the Modeling Exercise 2.22 on page 46.

(i) Is this DTMC irreducible? Aperiodic?

COMPUTATIONAL EXERCISES 137
(ii) Compute the long run average fraction of items inspected.
(iii) Suppose items found to be defective are instantaneously repaired before being
shipped. Compute the long run fraction of defective items in the shipped items.

4.37 Consider the Modeling Exercise 2.20 on page 46. Compute the long run frac-
tion of the time the audio quality is impaired.

4.38 Consider the Modeling Exercise 2.32 on page 49. Compute

(i) the long run fraction of the time both machine are working,
(ii) the average number of assemblies shipped per period,
(iii) the fraction of the time the i-th machine is off.

4.39 Consider the DTMC in the Modeling Exercise 2.24 on page 47. Let τi be the
mean lifetime of the machine from vendor i, i = 1, 2. Assume τi < ∞. Show that the
DTMC is positive recurrent and compute the long run fraction of the time machine
from vendor 1 is in use.

4.40 Compute the limiting distribution of the DTMC in the Modeling Exercise 2.27
on page 48.

4.41 Compute the limiting distribution of the DTMC in the Modeling Exercise 2.28
on page 48.

4.42 Compute the limiting distribution of the DTMC in the Modeling Exercise 2.6
on page 43.

4.43 Compute limn→∞ P n for the following probability transition matrices:

 
  .3 .3 .4 0 0 0
.3 .7 0 0  .5 0 .5 0 0 0 
 
 .4 .6 0 0   .9 0 .1 0 0 0 
(a) 
  , (b) 
 .
.2 .3 .5 0   0 0 0 1 0 0 

.5 0 .4 .1  0 0 .4 .2 .2 .2 
0 .1 0 .4 .1 .4

4.44 Let P be the transition probability matrix of the gambler’s ruin DTMC in the
Example 2.11 on page 17. Compute limn→∞ P n .

4.45 Let {Xn , n ≥ 0} be the DTMC of Modeling Exercise 2.33. Suppose it costs
C1 dollars to replace a failed light bulb, and C2 dollars to replace a working light
bulb. Compute the long run replacement cost of this replacement policy.

4.46 A machine produces two items per day. Each item is non-defective with prob-
ability p, the quality of the successive items being independent. Defective items are
thrown away immediately, and the non-defective items are stored to satisfy the de-
mand of one item per day. Any demand that cannot be satisfied immediately is lost.
138 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Let Xn be the number of items in storage at the beginning of each day (before the
production and demand for that day is taken into account). Show that {Xn , n ≥ 0} is
a DTMC. When is it positive recurrent? Compute its limiting distribution when it
exists.

4.47 Suppose it costs $c to store an item for one day and $d for every demand that
is lost. Compute the long run cost per day of operating the facility in Computational
Exercise 4.46 assuming it is stable.

4.48 Consider the production facility of Computational Exercise 4.46. Suppose the
following operating procedure is followed. The machine is turned off as soon the
inventory (the items in storage) reaches K, a fixed positive integer. It then remains
off until the inventory reduces to k, another fixed positive integer less than K, at
which point it is turned on again. Model this by an appropriate DTMC and compute

(i) the steady-state probability that the machine is off,

(ii) the steady-state probability that there are i items in storage, 0 ≤ i ≤ K.

4.49 Consider the production facility of Computational Exercise 4.48. Suppose

shutting the machine down costs $A and turning it on costs $B. Compute the long
run cost per day of operating this system.

4.50 Consider the DTMC developed in Modeling Exercise 2.14.

(i) Is the DTMC irreducible and aperiodic?

(ii) What is the long run probability that the machine is up and running?
(iii) Suppose repairing the ith component costs ci dollars per day (i = 1, 2). The
daily output of a working machine generates a revenue of R dollars. What is the
long run net revenue per day of this machine? Find the minimum value of R for
which it is profitable to operate this machine.

4.51 Consider the beer-guzzling Mr. Al Anon of Modeling Exercise 2.16 on

page 45. Assume the following data is given:
α = .6, β = .2, H = $4.00, L = $3.00.
Compute the long run daily beer expense of Mr. Al Anon.

4.52 Price of an item fluctuates between two levels: high ($H) and low ($L). Let
Yn be the price of the item at the beginning of day n. Suppose {Yn , n ≥ 0} is a
DTMC with state space {0, 1}, where state 0 is low, and state 1 is high. The transition
probability matrix is  
1−α α
.
β 1−β
A production facility consumes one such item at the beginning of each day and has
a storage capacity of K. The production manager uses the following procurement
CONCEPTUAL EXERCISES 139
policy: If the price of the item at the beginning of day n is low, then procure enough
items instantly to bring the number of items in stock to K. If the price level is high,
and there is at least one item in stock, do nothing; else procure one item instantly. Let
Xn be the number of items in stock at the beginning of day n. Suppose the inventory
costs are charged at the rate of $h per item in stock at the beginning of a day per
day. (Assume the following sequence of events: at the beginning of the nth day, we
first consume an item, then observe the new price level Yn , then procure the quantity
determined by the above rule, then observe the new stock level Xn .)

1. Compute the long run procurement plus holding cost per day of following this
policy.
2. Suppose the price is known to be $25 on 20% of the days, and $15 for 80% of
the days on the average. The price cycle lasts on the average 25 days. The holding
cost is $.50 per item per day. Plot or tabulate the long run average cost per day as
a function of K. What is value of K that minimizes the cost rate?

4.53 Consider the DTMC model of a buffer of size B as described in Modeling

Exercise 2.20.

(i) At the beginning, when the play program is initiated, the system waits until the
buffer has a fixed number b of bytes, and then starts playing. Let T be the first
time either the buffer becomes empty, or some bytes are lost. A song of K bytes
plays flawlessly if T > K. Show how to compute the probability that the song
of K bytes plays flawlessly.
(ii) Consider the following parameters:
B = 100, α0 = .2, α1 = .5, K = 512.
What value of b should be used to maximize the probability that this song is
played flawlessly? What is this maximum probability?

4.54 Consider the DTMC developed in Modeling Exercise 2.12.

(i) Compute its steady state distribution.

(ii) Compute the long run gasoline cost per day to the student. Compare this with
the long run gasoline cost per day if the student filled up the tank whenever it
became empty, regardless of the price of gasoline.

4.11 Conceptual Exercises

4.1 Let S be a finite set. A relation on S is a function R : S × S → {0, 1}. If x and

y in S satisfy the relation R, define R(x, y) = 1, and 0 otherwise. A relation R is
said to be

(i) reflexive if R(x, x) = 1 for all x ∈ S,

(ii) symmetric if R(x, y) = R(y, x) for all x, y ∈ S,
140 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
(iii) transitive if R(x, y) = 1, R(y, z) = 1 ⇒ R(x, z) = 1, for all x, y, z ∈ S.

Investigate the properties of the following relations:

(i) S = set of all humans, R(x, y) = 1 if and only if x and y are blood relatives,
(ii) S = set of all humans, R(x, y) = 1 if and only if x is a brother of y,
(iii) S = set of all students at a university, R(x, y) = 1 if x and y are attending at
east one class together,
(iv) S = set of all polynomials, R(x, y) = 1 if the degree of the product of x and y
is 10.

4.2 Let C1 and C2 be two communicating classes of a DTMC. Show that either
C1 = C2 or C1 ∩ C2 = φ.

4.3 Let {Xn , n ≥ 0} be a DTMC with state-space {1, 2, · · · N } and transition

probability matrix P with q non-zero entries. Develop an algorithm to check whether
or not state i is accessible from state j in O(q) steps.

4.4 For the DTMC in Conceptual Exercise 4.3 develop an algorithm to identify the
partition in Equation 4.9 in O(q) steps.

4.5 Show that the two definitions of periodicity given in Definitions 4.6 and 4.7 are
equivalent.

4.6 Complete the proof of Theorem 4.3.

4.7 Show that in an irreducible DTMC with N states, it is possible to go from any
state to any other state in N steps or less.

4.8 Show that the period of an irreducible DTMC with N states is N or less.

4.9 Show that there are no null recurrent states in a finite state DTMC.

4.10 Show by example that it is possible for an irreducible DTMC with N states to
have any period d ∈ {1, 2, · · · , N }.

4.11 Show that not all states in a finite state DTMC can be transient.

4.12 Deduce Equation 4.19 from Equation 4.18 by following an argument similar
to the one in the proof of Theorem 4.7.

4.13 Prove Theorem 4.11 (Pakes’ lemma) using Theorem 4.10 (Foster’s criterion).

4.14 Show that a transient DTMC eventually permanently exits any finite set with
probability 1. (Use Borel-Cantelli lemma of Appendix A.)
CONCEPTUAL EXERCISES 141
4.15 Let {πj , j ∈ S} be a solution to the balance and normalizing equations. Now
suppose the DTMC starts with initial distribution
P(X0 = j) = πj , j ∈ S.
Show that
P(Xn = j) = πj , j ∈ S, for all n ≥ 1.

4.16 Establish Equation 4.40 by induction.

4.17 Let {Xn , n ≥ 0} be a DTMC with state-space S and transition probability

matrix P . Suppose the system incurs an expected cost of c(i, j) at time n if Xn = i
and Xn+1 = j. Let
X
c(i) = c(i, j)pij , c = [c(i)]i∈S .
j∈S

Let
∞
X
φ(i) = E( αn c(Xn , Xn+1 |X0 = i), φ = [φ(i)]i∈S .
n=0
Show that φ satisfies Equation 4.52.

4.18 Consider the cost model of Conceptual Exercise 4.17, and assume that the
DTMC is irreducible and positive recurrent with limiting occupancy distribution
{πi , i ∈ S}. Show that g, the long run expected cost per unit time, is given by
XX
g= πi c(i, j)pi,j .
i∈S j∈S

4.19 Consider the cost set up of Section 4.8. Let φ(N, i) be the expected discounted
cost incurred over time {0, 1, 2, · · · , N } starting from state i, i.e.,
N
!
X
n
φ(N, i) = E α c(Xn )|X0 = i .
n=0

Show that φ(N, i) can be computed recursively as follows:

φ(0, i) = c(i),
X
φ(N, i) = c(i) + α pij φ(N − 1, j), N ≥ 1.
j∈S

4.20 Consider the cost set up of Section 4.8. Let g(N, i) be the expected cost per
period incurred over time {0, 1, 2, · · · , N } starting from state i, i.e.,
N
!
1 X
g(N, i) = E c(Xn )|X0 = i .
N +1 n=0
142 DISCRETE-TIME MARKOV CHAINS: LIMITING BEHAVIOR
Show that g(N, i) can be computed recursively as follows:
g(0, i) = c(i),
 
1  X 
g(N, i) = c(i) + N pij g(N − 1, j) , N ≥ 1.
N +1 
j∈S

4.21 Show that if {πj , j ∈ S} satisfy the local balance Equations 4.55, they satisfy
the global balance Equations 4.35. Show by counter example that the reverse is not
true.

4.22 A transition probability matrix is called doubly stochastic if

X
pij = 1, j ∈ S.
i∈S

Let {Xn , n ≥ 0} be an irreducible DTMC on a finite state-space {1, 2, · · · , N }

with a doubly stochastic transition probability matrix P . Show that its stationary
distribution is given by
πj = 1/N, j ∈ S.

4.23 A DTMC is said to be tree if between any two distinct states i and j there is
exactly one sequence of distinct states i1 , i2 , · · · ir such that
pi,i1 pi1 ,i2 · · · pir ,j > 0.
Show that a positive recurrent tree DTMC is reversible.

4.24 Let P be a diagonalizable N × N transition probability matrix. Using the

representation in Equation 2.32 study the limiting behavior of P n and M (n) /(n + 1)
in terms of the eigenvectors. Consider the following cases

(i) λ = 1 is an eigenvalue of multiplicity one, and it is the only eigenvalue with

|λ| = 1.
(ii) λ = 1 has multiplicity k, and these are the only eigenvalues with |λ| = 1.
(iii) There are eigenvalues with |λ| = 1 other than λ = 1.

4.25 Let {Xn , n ≥ 0} be an irreducible and positive DTMC on state space S. Let
P be its transition probability matrix and π be its stationary distribution. Suppose the
Markov chain earns a reward of ri everytime it visits state i. Let Ri (n), n ≥ 1, be
the total expected reward earned by the DTMC over time periods {0, 1, ..., n − 1}
starting from state i. Let R(n) = [Ri (n), i ∈ S] be the column vector of these
expected rewards.

(i) Derive a set of recursive equations satisfied by Ri (n), i ∈ S. Write them in

matrix form using P and r = [ri , i ∈ S].
 CONCEPTUAL EXERCISES 143
(ii) It is known that (you do not need to prove this) there exists a constant g and a
column vector h = [hi , i ∈ S] such that
R(n) = h + nge + o(n), (4.57)
where e is a column vector of ones, and o(n) is a vector with the following prop-
erty:
lim o(n) = 0.
n→∞
Compute the value of g. Hint: Substitute equation (4.57) in the result obtained in
part (i) above. Pre-multiply by π.
(iii) Derive the equations that uniquely determine h.

4.26 Suppose j is recurrent, and i → j. Show that

∞
(n)
X
pij = ∞.
n=0

4.27 Suppose i → j. Show that

1
Pn (r)
(i) limn→∞ n+1 r=0 pij > 0 if j is positive recurrent,
1 Pn (r)
(ii) limn→∞ n+1 r=0 pij = 0 if j is null recurrent.

4.28 Let {Xn , n ≥ 0} be an irreducible positive recurrent aperiodic DTMC with

state-space S and limiting distribution {πj , j ∈ S}. Show by counter example that
P(Xn = j) = πj , for all j ∈ S, for a given n > 0 does not imply that P(Xm =
j) = πj for all j ∈ S and m ≥ 0.

4.29 Let {Xn , n ≥ 0} be an irreducible positive recurrent DTMC with state-space

S and stationary distribution {πj , j ∈ S}. Show that the long run fraction of transi-
tions that take the DTMC from state i to j is given by πi pij .

4.30 Let P be a transition probability matrix of an irreducible positive recurrent

DTMC. Show that a DTMC with transition probability matrix (P + P T )/2 is re-
versible.

4.31 Let {Xn , n ≥ 0} be an irreducible recurrent DTMC with state-space

{0, 1, · · ·}, and transition probability matrix P . Let T = min{n ≥ 0 : Xn = 0}.
Now suppose the DTMC earns a reward ri every time it visits state i. Let R be the
total reward earned from time {0, 1, ...T − 1}. Using first step analysis derive a set
of linear equations satisfied by
g(i) = E(R|X0 = i)
for i = 0, 1, 2, · · ·. (Note that g(0) = 0.)
 CHAPTER 5

Poisson Processes

“Basic research is what I am doing when I don’t know what I am doing.”

— Wernher von Braun

5.1 Exponential Distributions

In this chapter we study an important special class of continuous-time stochastic

processes called Poisson Processes. These processes are defined in terms of random
variables with exponential distributions, called “exponential random variables” for
short. We shall see in this and the next few chapters that exponential random variables
are used to build a large number of stochastic models. It is critical for the student to
develop an ability to deal with the exponential distributions freely, without having to
think too much. With this in mind we have collected several relevant properties of
the exponential distributions in this section. We begin with the definition.

Definition 5.1 Exponential Distribution. A non-negative random variable X is

said to be an exponential random variable with parameter λ, denoted as X ∼
exp(λ), if 
0 if x < 0
FX (x) = P(X ≤ x) = , (5.1)
1 − e−λx if x ≥ 0
where λ > 0 is a fixed constant.

The cumulative distribution function (cdf) FX is plotted in Figure 5.1. The proba-
bility density function (pdf) fX of the exp(λ) random variable is given by

d 0 if x < 0
fX (x) = FX (x) =
dx λe−λx if x ≥ 0.
The density function is plotted in Figure 5.2. The Laplace Stieltjest transform (LST)
of X ∼ exp(λ) is given by
Z ∞
−sX λ
F̃X (s) = E(e )= e−sx fX (x)dx = , Re(s) > −λ, (5.2)
0 λ + s
where the Re(s) denotes the real part of the complex number s. Taking the deriva-
tives of F̃X (s) we can compute the rth moments of X for all positive integer values

145
146 POISSON PROCESSES
F(x)

Figure 5.1 The cdf of an exponential random variable.

f(x)

Figure 5.2 The pdf of an exponential random variable.

of r as follows:
dr r!
E(X r ) = (−1)r r
F̃X (s)|s=0 = r .
ds λ
In particular we have
1 1
E(X) = , Var(X) = 2 .
λ λ
Thus the coefficient of variation of X, Var(X)/(E(X)2), is 1. We now study many
special and interesting properties of the exponential random variable.

5.1.1 Memoryless Property

We begin with the definition of the memoryless property.

Definition 5.2 A non-negative random variable X said to have the memoryless

property if
P(X > s + t|X > s) = P(X > t), s, t ≥ 0. (5.3)

Thus if X represents the lifetime of a component (say a computer hard drive), the
memoryless property says that the probability that an s year old hard drive will last
EXPONENTIAL DISTRIBUTIONS 147
another t years is the same as the probability that a new hard drive will last t years. It
is as if the hard drive has no memory that it has already been functioning for s years!
The next theorem gives an important characterization of an exponential random vari-
able.

Theorem 5.1 Memoryless Property. A continuous non-negative random variable

has memoryless property if and only if it is an exp(λ) random variable for some
λ > 0.

Proof: We first show the “if” part. So, suppose X ∼ exp(λ) for some λ > 0. Then,
for s, t ≥ 0,
P(X > s + t, X > s)
P(X > s + t|X > s) =
P(X > s)
P(X > s + t) e−λ(s+t)
= =
P(X > s) e−λs
= e−λt = P(X > t).
Hence, by definition, X has memoryless property. Next we show the “only if” part.
So, let X be a non-negative random variable with complementary cdf
F c (x) = P(X > x), x ≥ 0.
Then, from Equation 5.3, we must have
F c (s + t) = F c (s)F c (t), s, t ≥ 0.
This implies
F c (2) = F c (1)F c (1) = (F c (1))2 ,
and
F c (1/2) = (F c (1))1/2 .
In general, for all positive rational a we get
F c (a) = (F c (1))a .
The only continuous function that will satisfy the above equation for all rational
numbers is
c
F c (x) = (F c (1))x = elnF (1)x = e−λx x ≥ 0,
where λ = −ln(F c (1)). Since F c (x) is a probability we must have λ > 0, and
hence FX (x) = 1 − F c (x) satisfies Equation 5.1. Thus X is an exp(λ) random
variable.

5.1.2 Hazard Rate

Let X be a non-negative continuous random variable with pdf f and complementary

cdf F c (x) = P(X > x). Hazard (or failure) rate of X is defined as
f (x)
r(x) = , x ≥ 0.
F c (x)
148 POISSON PROCESSES
Note that, for small h,
f (x)h + o(h)
r(x)h + o(h) = = P(X ∈ (x, x + h)|X > x).
F c (x)
Thus, in the limit as h → 0, r(x)h can be interpreted as the conditional probability
that a machine with lifetime X fails in the interval (x, x + h) given that it has not
failed until x. Hence the name “failure rate.” In insurance literature it is also known
as mortality rate. Note that r(x) is not a probability density function, since its integral
from 0 to ∞ need not be 1.
The next theorem characterizes the exponential distribution via its hazard rate.

Theorem 5.2 A continuous non-negative random variable has constant hazard rate
r(x) = λ > 0, x ≥ 0, if and only if it is an exp(λ) random variable.

Proof: We first show the “if” part. So, suppose X ∼ exp(λ) for some λ > 0. Then
its hazard rate is given by
f (x)
r(x) =
F c (x)
λe−λx
= = λ.
e−λx
To show the “only if” part, note that the hazard rate completely determines the com-
plementary cdf by the following formula (see Conceptual Exercise 5.1)
 Z x 
F c (x) = exp − r(u)du , x ≥ 0. (5.4)
0

Hence, if the random variable X has hazard rate r(u) = λ for all u ≥ 0, we must
have
F c (x) = e−λx , x ≥ 0.
Thus X ∼ exp(λ).

5.1.3 Probability of First Failure

Let Xi ∼ exp(λi ), i = 1, 2, be two independent random variables representing the

lifetimes of two machines. Then the probability of machine 1 failing before machine
2 is given by
Z ∞
P(X1 < X2 ) = P(X1 < X2 |X1 = x)λ1 e−λ1 x dx
Z0 ∞
= P(X2 > x)λ1 e−λ1 x dx
0
Z ∞
= e−λ2 x λ1 e−λ1 x dx
0
λ1
= .
λ1 + λ2
EXPONENTIAL DISTRIBUTIONS 149
Thus the probability of first failure is proportional to the failure rate of the expo-
nential random variables. Note that the assumption of independence is critical. The
above result can be easily extended to more than two exponential random variables,
see Conceptual Exercise 5.3.

Example 5.1 A running track is 1 km long. Two runners start on it at the same time.
The speed of runner i is Xi , i = 1, 2. Suppose Xi ∼ exp(λi ), i = 1, 2, and X1 and
X2 are independent. The mean speed of runner 1 is 20 km per hour and that of runner
2 is 22 km per hour. What is the probability that runner 1 wins the race?
The required probability is given by
 
1 1 λ2
P < = P(X2 < X1 ) =
X1 X2 λ1 + λ2
1/22 20
= = = 0.476.
1/22 + 1/20 20 + 22

5.1.4 Minimum of Exponentials

Let {Xi , 1 ≤ i ≤ k} be k non-negative random variables. Define

Z = min{X1 , X2 , · · · , Xk } (5.5)
and
N = i if Xi = Z. (5.6)

Note that N is unambiguously defined if the random variables {Xi , 1 ≤ i ≤ k}

are continuous, since in that case exactly one among the k random variables will
equal Z. Thus, if Xi represents the time when an event of type i occurs (1 ≤ i ≤ k),
then Z represents the time when the first of these k events occurs, and N represents
the type of the event that occurs first. The next theorem gives the joint distribution of
N and Z.

Theorem 5.3 Let Xi ∼ exp(λi ), 1 ≤ i ≤ k, be independent and let N and Z be as

defined by Equations 5.6 and 5.5. Then
λi −λz
P(N = i, Z > z) = e , z ≥ 0, 1 ≤ i ≤ k,
λ
where
k
X
λ= λi .
i=1

Proof: For a fixed i ∈ {1, 2, · · · , k} and z ≥ 0, we have

P(N = i, Z > z) = P(Xj > Xi > z, j 6= i, 1 ≤ j ≤ k)
Z ∞
= P(Xj > Xi > z, j 6= i, 1 ≤ j ≤ k|Xi = x)λi e−λi x dx
0
150 POISSON PROCESSES
Z ∞
= P(Xj > x, j 6= i, 1 ≤ j ≤ k)λi e−λi x dx
z
Z ∞ k
Y
= e−λj x λi e−λi x dx
z j=1,j6=i
Z ∞
−λx
= λi e dx
z
λi −λz
= e
λ
as desired.
An important implication of the above theorem is that N and Z are independent
random variables. Also, the marginal distribution of Z is exponential with parameter
λ. Thus the minimum of k independent exponential random variables is an exponen-
tial random variable whose parameter is the sum of the parameters of the original
random variables. These properties will play a critical role in later development of
continuous time Markov chains in Chapter 6.

5.1.5 Strong Memoryless Property

In this section we generalize the memoryless property of Equation 5.3 from a fixed
t to a random t. We call this the strong memoryless property. The precise result is
given in the next Theorem.

Theorem 5.4 Let X ∼ exp(λ) and T be another non-negative random variable

that is independent of X. Then
P(X > s + T |X > T ) = e−λs , s ≥ 0.

Proof: We have
Z ∞
P(X > s + T, X > T ) = P(X > s + T, X > T |T = t)dFT (t)
Z0 ∞
= P(X > s + t, X > t)dFT (t)
Z0 ∞
= e−λ(s+t) dFT (t)
0
Z ∞
−λs
= e e−λt dFT (t)
0
−λs
= e P(X > T ).
Then
P(X > s + T, X > T )
P(X > s + T |X > T ) = = e−λs ,
P(X > T )
as desired.
EXPONENTIAL DISTRIBUTIONS 151
Another way of interpreting the strong memoryless property is that, given X > T ,
X − T is an exp(λ) random variable. Indeed it is possible to prove a multivariate
extension of this property as follows. Let Xi ∼ exp(λi ), 1 ≤ i ≤ n, be independent
and let T be a non-negative random variable that is independent of them. Then
n
Y
P(Xi > si + T, 1 ≤ i ≤ n|Xi > T, 1 ≤ i ≤ n) = e−λi si , si ≥ 0. (5.7)
i=1

A simple application of this property is given in the following example.

Example 5.2 Parallel System. Consider a system of n components in parallel, i.e.,

the system functions as long as at least one of the n component functions. Let Xi be
the lifetime of component i and assume that Xi ∼ exp(λ) are iid random variables.
Then the lifetime of the system is given by
Z = max{X1 , X2 , · · · , Xn }.
Compute E(Z).
One can solve this problem by first computing the cdf of Z as follows:

n
P(Z ≤ z) = P(Xi ≤ z, 1 ≤ i ≤ n) = 1 − e−λz ,
and then computing its expected value. Here we show an alternate method using the
properties of the exponential distributions. Let Zk be the time of the k-th failure,
1 ≤ k ≤ n. Thus
Z1 = min{X1 , X2 , · · · , Xn } ∼ exp(nλ),
Zn = max{X1 , X2 , · · · , Xn }.
Now, at time Z1 , one component fails, and the remaining n − 1 components survive.
Due to Equation 5.7, the remaining lifetimes of the surviving components are iid
exp(λ) random variables. Hence the time until the next failure, namely Z2 − Z1 ,
equals the minimum of the n − 1 iid exp(λ) random variables. Hence
Z2 − Z1 ∼ exp((n − 1)λ).
Proceeding in this way we see that
Zk+1 − Zk ∼ exp((n − k)λ), 1 ≤ k ≤ n − 1.
Thus, writing Zn = n−1
P
k=0 (Zk+1 − Zk ), we get
n−1 n−1 n
X X 1 X 1
E(Z) = E(Zn ) = E(Zk+1 − Zk ) = = .
(n − k)λ kλ
k=0 k=0 k=1

The above equation is an example of the law of diminishing returns. A system

of one component (with exp(λ) lifetime) has an expected lifetime of 1/λ. A sys-
tem of two independent exp(λ) components in parallel has an expected lifetime of
1/λ+1/(2λ) = 1.5/λ. Thus doubling the number of components increases the mean
152 POISSON PROCESSES
lifetime by only 50%. One reason behind this diminishing return is that all compo-
nents are subject to failure, even though only one is needed for the system to function.
We say that the system is operating with “warm standbys.” Another way is to operate
the system with only one component and (n − 1) spares. When the working item
fails, replace it with a spare item. Since the spares do not fail while they are not in
use, they are said to be “cold standbys.” In this case the expected lifetime of the sys-
tem is n/λ. Obviously cold standby strategy is more efficient system than the warm
standby strategy. However, there are cases (in life-critical systems) where we cannot
use the cold standby strategy since the process of replacing the failed component by
a new one will interrupt the system operation.

Example 5.3 System A has two components in parallel, with iid exp(λ) lifetimes.
System B has a single component with exp(µ) lifetime, independent of system A.
What is the probability that system A fails before system B?
Let Zi be time of ith failure in system A. System A fails at time Z2 . From Exam-
ple 5.2, we have Z1 ∼ exp(2λ) and Z2 − Z1 ∼ exp(λ). Let X ∼ exp(µ) be the
lifetime of the component in system B. Then
P(System A fails before System B) =
= P(Z1 < X)P(Z2 − Z1 < X − Z1 |X > Z1 )
= P(exp(2λ) < exp(µ))P(exp(λ) < exp(µ))
2λ λ
= · .
2λ + µ λ + µ

5.1.6 Sum of iid Exponentials

Let {Xi , 1 ≤ i ≤ n} be iid random variables, representing the lifetimes of n compo-

nents. Suppose we start by putting component 1 in use, and when it fails, replace it
with component 2, and so on until all components fail. The replacements are instan-
taneous. The lifetime of the system is thus given by
Z = X1 + X2 + · · · + Xn .
The next theorem gives the pdf of Z if the lifetimes are iid exp(λ) random variables.

Theorem 5.5 Suppose {Xi , 1 ≤ i ≤ n} are iid exp(λ) random variables. Then Z
is an Erlang (or Gamma) (n, λ) random variable (denoted as Erl(n, λ)), with density
0 if z < 0

fZ (z) = −λz (λz)
n−1
λe n! if z ≥ 0,
and cdf 
0 if z < 0
FZ (z) = (λz)r
1 − e−λz n−1
P
r=0 r! if z ≥ 0.
EXPONENTIAL DISTRIBUTIONS 153
Proof: We compute the LST of Z as follows:
E(e−sZ ) = E(e−s(X1 +X+2+···+Xn ) )
= E(e−sX1 )E(e−sX2 ) · · · E(e−sXn )
 n
λ
= .
s+λ
The result follows by taking the inverse LST from the table in Appendix F.

Example 5.4 Suppose the times between consecutive births at a maternity ward
in a hospital are iid exponential random variables with mean 1 day. What is the
probability that the 10th birth in a calendar year takes place after Jan 15?
Note that it does not matter when the last birth in the previous year took place,
since, due to strong memoryless property, the time until the first birth into the new
year is exponentially distributed. Thus Z, the time of the tenth birth is a sum of 10
iid exp(1) random variables. Therefore the required probability is given by
9
X 15r
P(Z > 15) = e−15 = 0.1185.
r=0
r!

We shall see later that the stochastic process of births is a Poisson process.

5.1.7 Sum of Distinct Exponentials

In the previous subsection we computed the distribution of the sum of independent

and identically distributed exponential random variables. In the next theorem we
study the sum of independent but distinct exponential random variables.

Theorem 5.6 Let Xi ∼ exp(λi ), 1 ≤ i ≤ n, be independent random variables.

Assume that all the λi ’s are distinct. Then the pdf of
Z = X1 + X2 + · · · + Xn
is given by
n
X
fZ (z) = αi λi e−λi z ,
i=1
where
n
Y λj
αi = , 1 ≤ i ≤ n.
λj − λi
j=1,j6=i

Proof: The LST of Z is given by

E(e−sZ ) = E(e−s(X1 +X+2+···+Xn ) )
= E(e−sX1 )E(e−sX2 ) · · · E(e−sXn )
n
Y λi
= .
i=1
s + λi
154 POISSON PROCESSES
The result follows by using partial fraction expansion (see Appendix D) to compute
the inverse LST.

5.1.8 Random Sums of iid Exponentials

In this section we study the distribution of a random sum of iid exp(λ) random
variables. The main result is given in the next theorem.

Theorem 5.7 Let {Xi , i ≥ 1} be a sequence of iid exp(λ) random variables and
N be a G(p) random variable (i.e., geometric with parameter p), independent of the
X’s. Then the random sum
XN
Z= Xi
i=1
is an exp(λp) random variable.

Proof: We have  n
λ
E(e−sZ |N = n) = .
s+λ
Hence we get
∞
X
E(e−sZ ) = E(e−sZ |N = n)P(N = n)
n=1
∞  n
X λ
= (1 − p)n−1 p
n=1
s + λ
∞ 
λp X λ(1 − p) n

=
s + λ n=0 s+λ
λp 1
= ·
s + λ 1 − λ(1 − p)/(s + λ)
λp
= .
s + λp
Hence, from Equation 5.2, Z must be an exp(λp) random variable.

Example 5.5 A machine is subject to a series of randomly occurring shocks. The

times between two consecutive shocks are iid exponential random variables with
common mean of 10 hours. Each shock results in breaking the machine (if it is not
already broken) with probability .3. Compute the distribution of the lifetime of the
machine.
Suppose the N th shock breaks the machine. Then N is G(.3) random variable.
That is
P(N = k) = (.7)k−1 (.3), k ≥ 1.
POISSON PROCESS: DEFINITIONS 155
Let Xi be the time between the (i − 1) and ith shock. We know that {Xi , i ≥ 1} is a
sequence of iid exp(.1) random variables. The lifetime of the machine is given by
N
X
Z= Xi .
i=1

Hence from Theorem 5.7, we see that Z ∼ exp(.03). Thus the lifetime is exponen-
tially distributed with mean 1/.03 = 33.33 hours.

5.2 Poisson Process: Definitions

A Poisson process is frequently used as a model for counting events occurring one at
a time, such as the number of births in a hospital, the number of arrivals at a service
system, the number of calls made, the number of accidents on a given section of a
road, etc. In this section we give three equivalent definitions of a Poisson process and
study its basic properties.
Let {Xn , n ≥ 1} be a sequence of non-negative random variables representing
inter-event times. Define
S0 = 0, Sn = X1 + X2 + · · · + Xn , n ≥ 1.
Then Sn is the time of occurrence of the nth event, n ≥ 1. Now, for t ≥ 0, define
N (t) = max{n ≥ 0 : Sn ≤ t}.
Thus N (t) is the number of events that take place over the time interval (0, t], and
{N (t), t ≥ 0} is called a counting process generated by {Xn , n ≥ 1}. Poisson
process is a special case of a counting process as defined below.

Definition 5.3 Poisson Process. The counting process {N (t), t ≥ 0} generated by

{Xn , n ≥ 1} is called a Poisson process with parameter (or rate) λ if {Xn , n ≥ 1}
is a sequence of iid exp(λ) random variables.

Thus the number of births in the maternity ward of Example 5.4 is a Poisson
process. We denote a Poisson process with parameter (or rate) λ by the shorthand
notation PP(λ). A typical sample path of a PP(λ) is shown in Figure 5.3. Note that
N (0) = 0 and the process jumps up by one at t = Sn , n ≥ 1. Thus it has piece-
wise constant, right-continuous sample paths. The next theorem gives the transient
distribution of a Poisson process.

Theorem 5.8 Transient Distribution. Let {N (t), t ≥ 0} be a PP(λ). Then

(λt)k
P(N (t) = k) = e−λt , k = 0, 1, 2, · · · .
k!
Proof: We have
N (t) ≥ k ⇔ k or more events in (0, t]
156 POISSON PROCESSES
N(t)
4

t
S0 S1 S2 S3 S4

Figure 5.3 A typical sample path of a Poisson process.

⇔ kth event takes place at or before t

⇔ Sk ≤ t.
Hence
P(N (t) ≥ k) = P(Sk ≤ t).
From Theorem 5.5, Sk ∼ Erl(k, λ) random variable. Hence,
P(N (t) = k) = P(N (t) ≥ k) − P(N (t) ≥ k + 1)
= P(Sk ≤ t) − P(Sk+1 ≤ t)
k−1 k
" # " #
r r
−λt (λt) −λt (λt)
X X
= 1− e − 1− e
r=0
r! r=0
r!
(λt)k
= e−λt
k!
as desired.
Theorem 5.8 says that, for a fixed t, N (t) is a Poisson random variable with param-
eter λt, denoted as P(λt). This provides one justification for calling {N (t), t ≥ 0} a
Poisson process.

Example 5.6 Arrivals at a Post Office. Suppose customers arrive at a post office
according to a PP with rate 10 per hour.

(i) Compute the distribution of the number of customers who use the post office
during an 8-hour day.
Let N (t) be the number of arrivals over (0, t]. We see that the arrival process is
PP(λ) with λ = 10 per hour. Hence
N (8) ∼ P(λ · 8) = P(80).
Thus
(80)k
P(N (8) = k) = e−80 , k = 0, 1, 2, · · · .
k!
(ii) Compute the expected number of customers who use the post office during an
8-hour day. Since N (8) ∼ P(80), the desired answer is given by E(N (8)) = 80.
POISSON PROCESS: DEFINITIONS 157
Next we compute the finite dimensional joint probability distributions of a Poisson
process. The above theorem does not help us there. We develop a crucial property of
a Poisson process that will help us do this. We start with a definition.

Definition 5.4 Shifted Poisson Process. Let {N (t), t ≥ 0} be a PP(λ), and define,
for a fixed s ≥ 0,
Ns (t) = N (t + s) − N (s), t ≥ 0.
The process {Ns (t), t ≥ 0} is called a shifted Poisson process.

Theorem 5.9 Shifted Poisson Process. A shifted Poisson process {Ns (t), t ≥ 0}
is a PP(λ), and is independent of {N (u), 0 ≤ u ≤ s}.

Proof: It is clear from the definition of Ns (t) that it equals the number of events in
(s, s + t]. From Figure 5.4 we see that the first event after s occurs at time SN (s)+1

N(t)
Ns(t)

exp(λ)
t

t
S0 S1 SN(s) s SN(s)+1

Figure 5.4 A shifted Poisson process.

and the last event at or before time s occurs at SN (s) . We have

P(SN (s)+1 − s > y|N (s) = k, SN (s) = x, N (u) : 0 ≤ u ≤ s)
= P(Sk + Xk+1 − s > y|N (s) = k, Sk = x, N (u) : 0 ≤ u ≤ s)
= P(Xk+1 > s + y − x|Xk+1 > s − x)
= e−λy
due to memoryless property of Xk+1 . Thus the first event in the counting process
{Ns (t), t ≥ 0} takes place after an exp(λ) amount of time. The rest of the inter-
event times are iid exp(λ). Hence the result follows.
Next we introduce the concept of a stochastic process with stationary and inde-
pendent increments.

Definition 5.5 Stationary and Independent Increments. Let {X(t), t ≥ 0} be a

continuous-time real-valued stochastic process. For a given s, t ≥ 0, X(s+t)−X(s)
is called the increment over the interval (s, s + t]. {X(t), t ≥ 0} is said to have
stationary and independent increments if
158 POISSON PROCESSES
(i) the distribution of the increment over an interval (s, s + t] is independent of s,
(ii) the increments over non-overlapping intervals are independent.

The next theorem states an important property of a Poisson process.

Theorem 5.10 Stationary and Independent Increments. A Poisson process has

stationary and independent increments.

Proof: Let {N (t), t ≥ 0} be a PP(λ). From Theorems 5.8 and 5.9 it follows that
(λt)k
P(N (t + s) − N (s) = k) = e−λt , k = 0, 1, 2, · · · (5.8)
k!
which is independent of s. Thus {N (t), t ≥ 0} has stationary increments.
Now suppose 0 ≤ t1 ≤ t2 ≤ t3 ≤ t4 are fixed. Then N (t2 ) − N (t1 ) and
N (t4 ) − N (t3 ) are increments over non-overlapping intervals (t1 , t2 ] and (t3 , t4 ].
From Theorem 5.9 Nt3 (t4 − t3 ) = N (t4 ) − N (t3 ) is independent of {N (u), 0 ≤
u ≤ t3 }, and hence independent of N (t2 ) − N (t1 ). This proves the independence of
increments over non-overlapping intervals. This proves the theorem.
The above theorem helps us compute the finite dimensional joint probability dis-
tributions of a Poisson process, as shown in the next theorem.

Theorem 5.11 Finite Dimensional Distributions. Let {N (t), t ≥ 0} be a PP(λ)

and 0 ≤ t1 ≤ · · · ≤ tn be given real numbers and 0 ≤ k1 ≤ · · · ≤ kn be given
integers. Then
P(N (t1 ) = k1 , N (t2 ) = k2 , · · · , N (tn ) = kn )
(λt1 )k1 (λ(t2 − t1 ))k2 −k1 (λ(tn − tn−1 ))kn −kn −1
= e−λtn · ··· .
k1 ! (k2 − k1 )! (kn − kn−1 )!

Proof:
P(N (t1 ) = k1 , N (t2 ) = k2 , · · · , N (tn ) = kn )
= P(N (t1 ) = k1 , N (t2 ) − N (t1 )
= k2 − k1 , · · · , N (tn ) − N (tn−1 ) = kn − kn−1 )
= P(N (t1 ) = k1 )P(N (t2 ) − N (t1 ) = k2 − k1 ) · · · P(N (tn ) − N (tn−1 )
= kn − kn−1 )
(by independent increments property)
(λt1 )k1 −λ(t2 −t1 ) (λ(t2 − t1 ))k2 −k1
= e−λt1 ·e
k1 ! (k2 − k1 )!
(λ(tn − tn−1 ))kn −kn−1
· · · e−λ(tn −tn−1 ) ,
(kn − kn−1 )!
where the last equation follows from Equation 5.8. Further simplification yields the
desired result.
POISSON PROCESS: DEFINITIONS 159
The independent increments property is very useful in computing probabilistic
quantities associated with a Poisson process as shown in the next two examples.

Example 5.7 Auto-Covariance Function. Let {N (t), t ≥ 0} be a PP(λ). Compute

its auto-covariance function, namely, Cov(N (s), N (s + t)), for t ≥ 0, s ≥ 0.
We have
E(N (s)N (s + t)) = E(N (s)(N (s + t) − N (s) + N (s)))
= E(N (s)(N (s + t) − N (s)) + E(N (s)2 )
= E(N (s))E(N (s + t) − N (s)) + E(N (s)2 )
(by independent increments property)
= λs · λt + (λs)2 + λs.
Hence,
Cov(N (s), N (s + t)) = E(N (s)N (s + t)) − E(N (s))E(N (s + t))
= λs · λt + (λs)2 + λs − λs(λ(s + t))
= λs.
The fact that the auto-covariance function is independent of t is a result of the sta-
tionarity of increments. In general we can write
Cov(N (s), N (t)) = λ min(s, t).

Example 5.8 Consider the post office of Example 5.6. What is the probability that
one customer arrives between 1:00pm and 1:06pm, and two customers arrive between
1:03pm and 1:12pm?
Using time homogeneity and hours as units of time, we write the required proba-
bility as P(N (0.1) = 1; N (0.2) − N (0.05) = 2). Using independence of increments
over non-overlapping intervals (0, 0.05], (0.05, 0.1], and (0.1, 0.2] we get
P(N (0.1) = 1; N (0.2) − N (0.05) = 2)
1
X
= P(N (0.05) = k, N (0.1) − N (0.05) = 1 − k, N (0.2) − N (0.1) = 1 + k)
k=0
X1
= P(N (0.05) = k)P(N (0.1) − N (0.05) = 1 − k)P(N (0.2) − N (0.1) = 1 + k)
k=0
1
X (0.5)k −.5 (0.5)1−k −.1 (1)(1+k)
= e−.5 e e .
k! (1 − k)! (1 + k)!
k=0

Numerical calculations yield the desired probability as 0.1015.

The next theorem states that the properties stated in Theorems 5.8 and 5.10 in fact
characterize a Poisson process.
160 POISSON PROCESSES
Theorem 5.12 Alternate Characterization 1. A stochastic process {N (t), t ≥ 0}
is a PP(λ) if and only if

(i) it has stationary and independent increments,

(ii) N (t) ∼ P(λt), for all t ≥ 0.

Proof: The “only if” part is contained in Theorems 5.8 and 5.10. Here we prove
the “if ” part. From (ii) it is clear that N (0) = 0 with probability 1. Also, since
N (s + t) − N (s) is a P (λt) random variable, it is clear that almost all sample paths
of the process are piecewise constant with jumps of size 1. Let
S1 = inf{t ≥ 0 : N (t) = 1}.
Now, by an argument similar to that in Theorem 5.8,
P(S1 > t) = P(N (t) = 0) = e−λt .
Hence S1 ∼ exp(λ). Similarly, we can define
Sk = inf{t ≥ 0 : N (t) = k},
and, using stationary and independent increments, show that {Xk = Sk −Sk−1 , k ≥
1} (with S0 = 0) is a sequence of iid exp(λ) random variables. Hence {N (t), t ≥ 0}
is a PP(λ). This completes the proof.
Since the conditions given in the above theorem are necessary and sufficient, they
can be taken as an alternate definition of a Poisson process. Finally, we give yet
another characterization of a Poisson process. We need the following definition.

Definition 5.6 o(h) Functions. A function f : R → R is said to be an o(h) function

(written f (h) = o(h) and read “f is little o of h”) if
f (h)
lim = 0.
h→0 h

We illustrate the above definition with an example.

Example 5.9

(i) Let f (x) = ∞ i

P
i=0 ai x be an absolutely convergent series with radius of con-
vergence R > 0. f is an o(h) function if and only if a0 = a1 = 0. This follows
since
∞ ∞
1X X
lim ai hi = lim ai hi = 0.
h→0 h h→0
i=2 i=2
(ii) The function f (x) = eλx − 1 − λx is o(h).
(iii) If f and g are o(h), then so is f + g.

We use this definition to give the second alternate characterization of a Poisson pro-
cess in the next theorem.
POISSON PROCESS: DEFINITIONS 161
Theorem 5.13 Alternate Characterization 2. A counting process {N (t), t ≥ 0}
is a PP(λ) if and only if
(i) it has stationary and independent increments,
(ii)
P(N (0) = 0) = 1,
P(N (h) = 0) = 1 − λh + o(h),
P(N (h) = 1) = λh + o(h),
P(N (h) = j) = o(h), j ≥ 2.

Proof: Suppose {N (t), t ≥ 0} is a PP(λ). Thus it satisfies conditions (i) and (ii)
of Theorem 5.12. Condition (i) above is the same as condition (i) of Theorem 5.12.
We shall show that condition (ii) of Theorem 5.12 implies condition (ii) above. We
have N (0) ∼ P(0) = 0 with probability 1. Furthermore,
1 1 −λh

lim (P(N (h) = 0) − 1 + λh) = lim e − 1 + λh = 0.
h→0 h h→0 h

Hence P(N (h) = 0) − 1 + λh is an o(h) function. Thus we can write

P(N (h) = 0) = 1 − λh + o(h).
The other statements in (ii) can be proved similarly. This proves that conditions (i)
and (ii) of Theorem 5.12 imply the conditions (i) and (ii) of this theorem.
Next we assume conditions (i) and (ii) above, and derive condition (ii) of Theo-
rem 5.12.. Now, let
pk (t) = P(N (t) = k), k ≥ 0.
Then, for k ≥ 1, we get
pk (t + h) = P(N (t + h) = k)
k
X
= P(N (t + h) = k|N (t) = j)P(N (t) = j)
j=0
k
X
= P(N (t + h) − N (t) = k − j|N (t) = j)pj (t)
j=0
k
X
= P(N (t + h) − N (t) = k − j)pj (t)
j=0
(by independence of increments)
k
X
= P(N (h) − N (0) = k − j)pj (t)
j=0
(by stationarity of increments)
k
X
= P(N (h) = k − j)pj (t) ( since N (0) = 0)
j=0
162 POISSON PROCESSES
k
X
= P(N (h) = 0)pk (t) + P(N (h) = 1)pk−1 (t) + P(N (h) = j)pk−j (t)
j=2
k
X
= (1 − λh + o(h))pk (t) + (λh + o(h))pk−1 (t) + o(h)pk−j (t)
j=2
k
X
= (1 − λh)pk (t) + λhpk−1 (t) + o(h)pk−j (t).
j=0

Rearranging and dividing by h, we get

k
1 o(h) X
(pk (t + h) − pk (t)) = −λpk (t) + λpk−1 (t) + pk−j (t).
h h j=0

Letting h → 0, we get
dpk (t)
p′k (t) = = −λpk (t) + λpk−1 (t). (5.9)
dt
Proceeding in a similar fashion for the case k = 0 we get
p′0 (t) = −λp0 (t). (5.10)
Using the initial condition p0 (0) = 1, the above equation admits the following solu-
tion:
p0 (t) = e−λt , t ≥ 0.
Using the initial condition pk (0) = 0 for k ≥ 1, we can solve Equation 5.9 recur-
sively to get
(λt)k
pk (t) = e−λt , t ≥ 0, (5.11)
k!
which implies that N (t) ∼ P (λt). Thus conditions (i) and (ii) of Theorem 5.13
imply conditions (i) and (ii) of Theorem 5.12. This proves the result.
Since the conditions of the above theorem are necessary and sufficient, they can
be taken as yet another definition of a Poisson process.

5.3 Event Times in a Poisson Process

In this section we study the joint distribution of the event times S1 , S2 , · · · , Sn , given
that N (t) = n. We begin with some preliminaries about order statistics of uniformly
distributed random variables. (See Appendix C for more details.)

Let U1 , U2 , · · · , Un be n iid random variables that are uniformly distributed over

[0, t], where t > 0 is a fixed number. Let Ũ1 , Ũ2 , · · · , Ũn be the order statistics of
U1 , U2 , · · · , Un , i.e., a permutation of U1 , U2 , · · · , Un such that
Ũ1 ≤ Ũ2 ≤ · · · ≤ Ũn .
EVENT TIMES IN A POISSON PROCESS 163
Thus
Ũ1 = min{U1 , U2 , · · · , Un },
Ũn = max{U1 , U2 , · · · , Un }.
Let f (t1 , t2 , · · · , tn ) be the joint density of Ũ1 , Ũ2 , · · · , Ũn . Then one can show that
(see Appendix C)
 n!
tn if 0 ≤ t1 ≤ t2 ≤ · · · ≤ tn ≤ t.
f (t1 , t2 , · · · , tn ) = (5.12)
0 otherwise.
From the above equation it is possible to derive the marginal density fk (u) of Ũk
(1 ≤ k ≤ n) as
 
k n  u k−1  u n−k
fk (u) = 1− , 0 ≤ u ≤ t, (5.13)
t k t t
and the expected value of Ũk as
kt
E(Ũk ) =
. (5.14)
n+1
The main result about the joint distribution of S1 , S2 , · · · , Sn is given in the next
theorem.

Theorem 5.14 Campbell’s Theorem. Let Sn be the nth event time in a PP(λ)
{N (t), t ≥ 0}. Given N (t) = n,
(S1 , S2 , · · · , Sn ) ∼ (Ũ1 , Ũ2 , · · · , Ũn ).

Proof: Let 0 = t0 ≤ t1 ≤ t2 ≤ · · · ≤ tn ≤ t. We have

P(Si ∈ (ti + dti ); 1 ≤ i ≤ n|N (t) = n)
P(Si ∈ (ti + dti ); 1 ≤ i ≤ n, ; N (t) = n)
=
P(N (t) = n)
P(Si ∈ (ti + dti ); 1 ≤ i ≤ n, ; Sn+1 > t)
=
P(N (t) = n)
P(Xi ∈ (ti − ti−1 + dti ); 1 ≤ i ≤ n, ; Xn+1 > t − tn )
=
P(N (t) = n)
Qn −λ(ti −ti−1 )

−λ(t−t )
i=1 λe dt i e
n
= (λt) n
e−λt n!
e−λt λn
= n · dt1 dt2 · · · dtn .
e−λt (λt)
n!

Hence the conditional joint density of (S1 , S2 , · · · , Sn ), given N (t) = n, is given

by  n!
tn if 0 ≤ t1 ≤ t2 ≤ · · · ≤ tn ≤ t
f (t1 , t2 , · · · , tn ) =
0 otherwise,
164 POISSON PROCESSES
which is the joint density of (Ũ1 , Ũ2 , · · · , Ũn ). This proves the theorem.
Note the curious fact that the joint distribution is independent of λ! This theorem
can be interpreted as follows: Suppose N (t) = n is given. Let (U1 , U2 , · · · , Un ) be n
iid random variables uniformly distributed over [0, t]. The smallest among them can
be thought of as S1 , the second smallest as S2 , and so on, with the largest as Sn . We
give several applications of the above theorem.

Example 5.10
(i) Let {N (t), t ≥ 0} be a PP(λ), and let Sn be the time of the nth event. Compute
P(S1 > s|N (t) = n). Assume n ≥ 1 is a given integer.
Theorem 5.14 implies that, given N (t) = n, S1 ∼ min{U1 , U2 , · · · , Un }. From
Equation 5.13 we get
n u n−1
f1 (u) = 1− , 0 ≤ u ≤ t.
t t
Hence
Z t  s n
P(S1 > s|N (t) = n) = f1 (u)du = 1 − , 0 ≤ s ≤ t.
s t
(ii) Compute E(Sk |N (t) = n).
From Theorem 5.14 and Equation 5.14, for 1 ≤ k ≤ n, we get
kt
E(Sk |N (t) = n) = E(Ũk ) = .
n+1
For k > n, using memoryless property of the exponentials, we get
k−n
E(Sk |N (t) = n) = t + .
λ
Example 5.11 Suppose passengers arrive at a bus depot according to a PP(λ). Buses
leave the depot every T time units. Assume that the bus capacity is sufficiently large
so that when a bus leaves there are no more passengers left at the depot. What is the
average waiting time of the passengers?
Let {N (t), t ≥ 0} be a PP(λ). Suppose a bus has just left the depot at time 0,
so the bus depot is empty at time 0. Consider the time interval (0, T ]. Number of
passengers waiting for the bus at time T is N (T ).
Now suppose N (T ) = n > 0. Let S1 , S2 , · · · , Sn be the arrival times of these n
passengers. The waiting time of the ith passenger is Wi = T − Si . Hence the average
waiting time is
n n n
1X 1X 1X
W = Wi = (T − Si ) = T − Si .
n i=1 n i=1 n i=1
Now, using Theorem 5.14, we get
n
X n
X n
X
Si ∼ Ũi = Ui ,
i=1 i=1 i=1
EVENT TIMES IN A POISSON PROCESS 165
where Ui ’s are iid uniformly distributed uniformly over [0, T ]. Hence
n
1X
E(W |N (T ) = n) = T − E( Ui ) = T − T /2 = T /2.
n i=1
Hence
T
E(W̄ ) = .
2
Thus the average waiting time is T /2, which makes intuitive sense. This is one more
manifestation of the fact that the events in a Poisson process occur in a uniform,
time-independent fashion.

Example 5.12 Customers arrive at a service station according to a PP(λ). Each

customer pays a service charge of $1 when he enters the system. Suppose the service
station manager discounts the revenues at a continuous discount factor α > 0 so that
a dollar earned at time t has a present value of e−αt at time 0. Compute the expected
present value of all the service charges earned until time t.
Let N (t) be the number of arrivals over (0, t]. We are given that {N (t), t ≥ 0} is
a PP(λ). Let C(t) be the total discounted service charges accumulated over (0, t]. It
is given by
N (t)
X
C(t) = e−αSi ,
i=1
where Si is the arrival time of the ith customer. We have
n
!
X
−αSi
E(C(t)|N (t) = n) = E e | N (t) = n
i=1
n
!
X
−αŨi
= E e
i=1
( from Theorem 5.14)
n
!
X
−αUi
= E e
i=1
= nE(e−αU ),
where U is uniformly distributed over [0, t]. We have
1 t −αu 1 − e−αt
Z
E(e−αU ) = e du = .
t 0 αt
Hence
1 − e−αt
E(C(t)|N (t)) = N (t) .
αt
This yields
1 − e−αt λ
E(C(t)) = E(N (t)) = (1 − e−αt ).
αt α
166 POISSON PROCESSES
The expected present value of all future revenue is
λ
lim E(C(t)) = .
t→∞ α
Note that this is a finite quantity although the total revenue is infinite.

5.4 Superposition and Splitting of Poisson Processes

The superposition of two counting processes is a counting process that counts events
in both the counting processes. Splitting of a counting process is the operation of
generating two counting processes by classifying the events in the original counting
process as belonging to one or the other counting process. We first study the super-
position and then the splitting of Poisson processes.

5.4.1 Superposition

Superposition occurs naturally when two Poisson processes merge to generate a com-
bined process. For example, a telephone exchange may get domestic calls and inter-
national calls, each forming a Poisson process. Thus the process that counts both
calls is the superposition of the two processes. Figure 5.5 illustrates such a superpo-
sition. In this section we study the superposition of two or more independent Poisson
processes.
Let {Ni (t), t ≥ 0}, (i = 1, 2, · · · r), be independent Poisson processes. Define
N (t) = N1 (t) + N2 (t) + · · · + Nr (t), t ≥ 0.
The process {N (t), t ≥ 0} is called the superposition of the r processes {Ni (t), t ≥
0}, (i = 1, 2, · · · r). The next theorem describes the superposed process.

Theorem 5.15 Superposition of Poisson Processes. Let {Ni (t), t ≥ 0}, (i =

1, 2, · · · r), be independent Poisson processes. Let λi be the parameter of the ith
process. Define
N (t) = N1 (t) + N2 (t) + · · · + Nr (t), t ≥ 0.
{N (t), t ≥ 0} is a Poisson process with parameter λ, where
λ = λ1 + λ2 + · · · + λr .

Proof: Since the r independent processes {Ni (t), t ≥ 0}, (i = 1, 2, · · · , r), have
stationary and independent increments, it follows that {N (t), t ≥ 0} inherits this
property. Thus in order to show that {N (t), t ≥ 0} is a PP(λ) it suffices to show that
N (t) is a P (λt) random variable. (See Theorem 5.12.) For a fixed t, we know that
{Ni (t) ∼ P (λi t), 1 ≤ i ≤ r} are independent Poisson random variables. Hence
their sum, N (t), is a Poisson random variable with parameter λt = (λ1 + λ2 + · · · +
λr )t. This proves the Theorem.
SUPERPOSITION AND SPLITTING OF POISSON PROCESSES 167
N1(t)

N2(t)

N(t) = N1(t) + N2(t)

Figure 5.5 Superposition of two Poisson processes.

We illustrate the above result by an example below.

Example 5.13 Jobs submitted for execution on a central computer are divided into
four priority classes, indexed 1,2,3, and 4. The inter-arrival times for the jobs of class
i are exponential random variables with mean mi minutes, with m1 = 10, m2 = 15,
m3 = 30, and m4 = 60. Assume all arrival streams behave independently of each
other. Let N (t) be the total number of jobs of all classes that arrive during (0, t].
Characterize the stochastic process {N (t), t ≥ 0}.
Let Ni (t) be the number of jobs of call i that arrive during (0, t]. Due to the
iid exponential inter-arrival times, we know that {Ni (t), t ≥ 0} is a PP(λi ) where
λi = 1/mi , and the four arrival processes are independent. Now, we have
N (t) = N1 (t) + N2 (t) + N3 (t) + N4 (t).
Hence, from Theorem 5.15, it follows that {N (t), t ≥ 0} is a PP(λ) where, using the
time units of hours,
λ = λ1 + λ2 + λ3 + λ4
= 60/m1 + 60/m2 + 60/m3 + 60/m4
= 6 + 4 + 2 + 1 = 13 per hour.

We now study how events from the individual processes {Ni (t), t ≥ 0}, (i =
1, 2, · · · , r), are interleaved in the superposed process {N (t), t ≥ 0}. Let Zn = i
168 POISSON PROCESSES
if the nth event in the superposed process belongs to the ith process. Thus, for
the sample path shown in Figure 5.5, we have Z1 = 1, Z2 = 2, Z3 = 2, Z4 =
1, Z5 = 2, Z6 = 1, · · ·. The next theorem gives an interesting property of the se-
quence {Zn , n ≥ 1}.

Theorem 5.16 {Zn , n ≥ 1} is a sequence of iid random variables with

λi
P(Zn = i) = , 1 ≤ i ≤ r.
λ

Proof: Let Si be the time of occurrence of the first event in {Ni (t), t ≥ 0}. Then
Si ∼ exp(λi ). Also, since the r processes are independent, {Si , 1 ≤ i ≤ r} are
independent. Hence
λi
P(Zn = i) = P(Si = min{Sj , 1 ≤ j ≤ r}) = ,
λ
where we have used the marginal distribution of N in Theorem 5.3. Now suppose
the first event in the {N (t), t ≥ 0} process takes place at time s. Then, from Theo-
rem 5.9 it follows that the shifted processes {Ni (t + s), t ≥ 0}, (i = 1, 2, · · · r), are
independent Poisson processes with parameters λi , respectively. Hence Z2 , the type
of the next event in {N (t), t ≥ 0}, has the same distribution as Z1 , and is indepen-
dent of it. Proceeding in the fashion we see that Zn has the same distribution as Z1
and is independent of {Zk , 1 ≤ k ≤ n − 1}. This proves the theorem.
This is yet another indication that the events in a PP take place uniformly in time.
That is why the probability of an event being from the ith process is proportional to
the rate of the ith process, and is independent of when the event occurs.

Example 5.14 Customers arriving at a bank can be classified into three categories.
Customers of category 1 deposit money, those of category 2 withdraw money, and
those of category 3 do both. The deposit transaction takes 3 minutes, the withdrawals
take 4 minutes, and the combined transaction takes 6 minutes. Customers of category
i arrive according to PP(λi ) with λ1 = 20, λ2 = 15, and λ3 = 10 per hour. What
is the average transaction time of a typical customer in the bank? Are the successive
transaction times iid?
Let Z be the category of a typical customer. From Theorem 5.16 we get
20 15 10
P(Z = 1) = , P(Z = 2) = , P(Z = 3) = .
45 45 45
Hence the average transaction time of a typical customer is
20 15 10
3· +4· +6· = 4 minutes.
45 45 45
The successive transaction times are iid since the categories of the successive cus-
tomers are iid random variables, from Theorem 5.16.
SUPERPOSITION AND SPLITTING OF POISSON PROCESSES 169
5.4.2 Splitting

Splitting is the opposite of superposition: we start with a single Poisson process and
“split” it to create two or more counting processes. For example, the original process
may count the number of arrivals at a store, while the split processes might count
the male and female arrivals separately. The nature of these counting processes will
depend on the rule used to split the original process. We use a special rule called the
Bernoulli splitting, described below.
Let {N (t), t ≥ 0} be a Poisson process. Suppose each event is classified as a type
i event (1 ≤ i ≤ r) with probability pi , where {pi , 1 ≤ i ≤ r} are given numbers
such that
Xr
pi > 0, (1 ≤ i ≤ r), pi = 1.
i=1
The successive classifications are independent. This classification mechanism is
called the Bernoulli splitting mechanism. Now let Ni (t) be the number of events
during (0, t] that get classified as type i events. We say that the original process
{N (t), t ≥ 0} is “split” into r processes {Ni (t), t ≥ 0}, (1 ≤ i ≤ r). Clearly,
N (t) = N1 (t) + N2 (t) + · · · + Nr (t), t ≥ 0,
so {N (t), t ≥ 0} is a superposition of {Ni (t), t ≥ 0}, (1 ≤ i ≤ r).
The next theorem gives the probabilistic structure of the split processes.

Theorem 5.17 Bernoulli Splitting. Let {N (t), t ≥ 0} be a PP(λ) and let

{Ni (t), t ≥ 0}, (1 ≤ i ≤ r), be generated by the Bernoulli splitting mechanism
using splitting probabilities [p1 , p2 , · · · , pr ]. Then {Ni (t), t ≥ 0} is a PP(λpi ),
(1 ≤ i ≤ r), and the r processes are independent.

Proof: We shall first show that {Ni (t), t ≥ 0} is a PP(λpi ) for a given i ∈
{1, 2, · · · ,r}. There are many ways of showing this. We shall show it by using Theo-
rem 5.7. Let {Xn , n ≥ 1} be the iid exp(λ) inter-event times in {N (t), t ≥ 0}, and
{Yn , n ≥ 1} be the inter-event times in the {Ni (t), t ≥ 0} process. Let {Rn , n ≥ 1}
be a sequence of iid be geometric random variables with parameter pi , i.e.,
P(Rn = i) = (1 − pi )k−1 pi , k ≥ 1.
Let T0 = 0 and Tn = R1 +R2 +· · ·+Rn , n ≥ 1. The Bernoulli splitting mechanism
implies that
Tn
X
Yn = Xi , n ≥ 1.
i=Tn−1 +1

Thus {Yn , n ≥ 1} is a sequence of iid random variables. Now Y1 is a geometric sum

of iid exponential random variables. Hence, from Theorem 5.7, Y1 is an exp(λpi )
random variable. Thus the inter-event times in {Ni (t), t ≥ 0} are iid exp(λpi ) ran-
dom variables. Hence {Ni (t), t ≥ 0} is a PP(λpi ).
170 POISSON PROCESSES
Next we show independence of the r processes. We treat the case r = 2, the
general case follows similarly. To show that {N1 (t), t ≥ 0} and {N2 (t), t ≥ 0} are
independent, it suffices to show that any increment in {N1 (t), t ≥ 0} is independent
of any increment in {N2 (t), t ≥ 0}. This reduces to showing
P(N1 (t) = i, N2 (t) = j) = P(N1 (t) = i)P(N2 (t) = j),
for all i, j ≥ 0. We have
P(N1 (t) = i, N2 (t) = j)
= P(N1 (t) = i, N2 (t) = j|N (t) = i + j)P(N (t) = i + j)
i + j i j −λt (λt)i+j
 
= p1 p2 e
i (i + j)!
i
(λp1 t) −λp2 t (λp2 t)j
= e−λp1 t e
i! j!
= P(N1 (t) = i)P(N2 (t) = j).
This proves the theorem.
It should be noted that the independence of the r split processes is rather surpris-
ing, since they all arise out of the same original process. This theorem makes the
Bernoulli splitting mechanism very attractive in applications. We illustrate with two
examples.

Example 5.15 Geiger Counter. A geiger counter is a device to count the radioac-
tive particles emitted by a source. Suppose the particles arrive at the counter accord-
ing to a PP(λ) with λ = 1000 per second. The counter fails to count a particle with
probability .1, independent of everything else. Suppose the counter registers four par-
ticles in .01 seconds. What is the probability that at least six particles have actually
arrived at the counter during this time period?
Let N (t) be the number of particles that arrive at the counter during (0, t], N1 (t)
the number of particles that are registered by the counter during (0, t], and N2 (t) the
number of particles that go unregistered by the counter during (0, t]. Then {N (t), t ≥
0} is a PP(1000), and Theorem 5.17 implies that {N1 (t), t ≥ 0} is a PP(900), and
it is independent of {N2 (t), t ≥ 0}, which is a PP(100). We are asked to compute
P(N (.01) ≥ 6|N1 (.01) = 4). We have
P(N (.01) ≥ 6|N1 (.01) = 4) = P(N2 (.01) ≥ 2|N1 (.01) = 4)
= P(N2 (.01) ≥ 2) = 0.264.
Here we have used independence to get the second equality, and used N2 (.01) ∼
P(1) to compute the numerical answer.

Example 5.16 Turnpike Traffic. Consider the toll highway from Orlando to Mi-
ami, with n interchanges where the cars can enter and exit, interchange 1 is at
the start in Orlando, and n is at the end in Miami. Suppose the cars going to Mi-
ami enter at interchange i according to a PP(λi ) and travel at the same speed,
SUPERPOSITION AND SPLITTING OF POISSON PROCESSES 171
(1 ≤ i ≤ n − 1). A car entering at interchange i will exit at interchange j with
probability pij , 1 ≤ i < j ≤ n, independent of each other. Let Ni (t) be the number
of cars that cross a traffic counter between interchange i and i + 1 during (0, t]. Show
that {Ni (t), t ≥ 0} is a PP and compute its rate.
Let Nki (t) be the number of cars that enter at interchange k ≤ i and cross the
traffic counter between interchange i and i + 1 during (0, t]. Assuming Bernoulli
splitting, we see that
X
{Nki (t), t ≥ 0} ∼ PP(λk pkj ).
j>i

Now,
i
X
Ni (t) = Nki (t).
k=1

Theorem 5.17 implies that {Nki (t), t ≥ 0}, 1 ≤ k ≤ i, are independent Poisson
processes. Hence, from Theorem 5.15, we have
Xi X
{Ni (t), t ≥ 0} ∼ PP( λk pkj ).
k=1 j>i

Are {Ni (t), t ≥ 0}, 1 ≤ i ≤ n − 1, independent?

Now we consider the non-homogeneous Bernoulli splitting. Let p : [0, ∞) →

[0, 1] be a given function. Let {N (t), t ≥ 0} be a PP(λ). Under the non-homogeneous
Bernoulli splitting mechanism, an event occurring at time s is registered with prob-
ability p(s), independent of everything else. Let R(t) be the number of registered
events during (0, ∞]. The next theorem shows that R(t) is a Poisson random vari-
able.

Theorem 5.18 Let p be an integrable function over [0, t]. Then R(t) is a Poisson
random variable with parameter
Z t
λ p(s)ds.
0

Proof: Let N (t) = m be given, and let Si (1 ≤ i ≤ m) be the occurrence time of

the ith event in {N (t), t ≥ 0}. Then we can write
m
X
R(t) = I(Si ),
i=1

where, given (S1 , S2 , · · · Sm ), {I(Si ), 1 ≤ i ≤ m} are independent random vari-

ables with distribution

1 with probability p(Si )
I(Si ) =
0 with probability 1 − p(Si ).
172 POISSON PROCESSES
Using Theorem 5.14 and the notation there, we get
m
X
R(t) = I(Si )
i=1
Xm
∼ I(Ũi )
i=1
Xm
= I(Ui ).
i=1

However, {I(Ui ), 1 ≤ i ≤ m} are iid Ber(α) random variables with

1 t
Z
α= p(s)ds.
t 0
Hence  
m k
P(R(t) = k|N (t) = m) = α (1 − α)m−k .
k
Hence
∞
X
P(R(t) = k) = P(R(t) = k|N (t) = m)P(N (t) = m)
m=0
∞ 
(λt)m

X m k
= α (1 − α)m−k e−λt
k m!
m=k
∞
αk X (λt)m
= e−λt (1 − α)m−k
k! (m − k)!
m=k
∞
(λαt)k X (λ(1 − α)t)m−k
= e−λt
k! (m − k)!
m=k
(λαt)k λ(1−α)t
= e−λt e
k!
(λαt)k
= e−λαt .
k!
Thus R(t) is Poisson random variable with parameter
Z t
λαt = λ p(s)ds.
0

This proves the Theorem.

Note that although R(t) is a Poisson random variable for each t, and {R(t), t ≥
0} independent increments, but these increments are not stationary, and hence
{R(t), t ≥ 0} is not a Poisson process. We shall identify such a process as an non-
homogeneous Poisson process, and study it in the next section. An interesting appli-
cation of a non-homogeneous Bernoulli splitting is given in the next example.
NON-HOMOGENEOUS POISSON PROCESS 173
Example 5.17 Infinite Server Queue. Users arrive at a public library according to
a PP(λ). User i stays in the library for Ti amount of time and then departs. Assume
that {Ti , i ≥ 1} are iid non-negative random variables with common cdf G·) and
mean τ . Suppose the library opens at time zero, with zero patrons in it. Compute the
distribution of R(t), the number of users in the library at time t.

Let N (t) be the number of users who enter the library during (0, t]. If a user i
enters the library at time s ≤ t, then he or she will be in the library at time t with
probability
p(s) = P(Ti > t − s) = 1 − G(t − s).
We can imagine that the user entering at time s is registered with probability p(s).
Thus R(t), the number of users who are registered during (0, t], is the same as the
number of users in the library at time t. Since p(·) is a monotone bounded function,
it is integrable, and we can apply Theorem 5.18 to see that R(t) is a Poisson random
variable with parameter
Z t Z t
λ p(s)ds = λ (1 − G(t − s))ds
0 0
Z t
= λ (1 − G(s))ds.
0
Thus the expected number of users in the library at time t is
Z t
E(R(t)) = λ (1 − G(s))ds.
0

Thus, as t → ∞, R(t) converges in distribution to a Poisson random variable with

parameter Z ∞
λ (1 − G(s))ds = λτ.
0
We shall see later that what we analyzed here is called the infinite server queue.

5.5 Non-Homogeneous Poisson Process

In this section we study a generalization of the Poisson process by relaxing the re-
quirement that the increments be stationary.
We begin with some preliminaries. Let λ : [0, ∞) → [0, ∞) be a given function.
Assume that it is integrable over finite intervals and define
Z t
Λ(t) = λ(s)ds.
0

Definition 5.7 A counting process {N (t), t ≥ 0} is said to be non-homogeneous

Poisson process with rate function λ(·) if
174 POISSON PROCESSES
(i) {N (t), t ≥ 0} has independent increments,
(ii) N (t) ∼ P (Λ(t)), for t ≥ 0.

We denote a non-homogeneous Poisson process with rate function λ(·) as

NPP(λ(·)). Note that when λ(t) = λ for all t ≥ 0, the N P P (λ(·)) reduces to
the standard PP(λ). The following are an immediate consequence of the definition:
If {N (t), t ≥ 0} is an N P P (λ(·)),

(i) N (0) = 0,
(ii) N (s + t) − N (s) ∼ P (Λ(s + t) − Λ(s)).

(See Conceptual Exercise 5.7.) Using the independence of increments one can derive
the finite dimensional joint probability distributions of an NPP as in Theorem 5.11.
We state the result in the next theorem and omit the proof.

Theorem 5.19 Finite Dimensional Distributions. Let {N (t), t ≥ 0} be a

N P P (λ(·)) and 0 ≤ t1 ≤ · · · ≤ tn be given real numbers and 0 ≤ k1 ≤ · · · ≤ kn
be given integers. Then
P(N (t1 ) = k1 , N (t2 ) = k2 , · · · , N (tn ) = kn )
(Λ(t1 ))k1 (Λ(t2 ) − Λ(t1 ))k2 −k1 (Λ(tn ) − Λ(tn−1 )))kn −kn−1
= e−Λ(tn ) · ··· .
k1 ! (k2 − k1 )! (kn − kn−1 )!

We illustrate with an example.

Example 5.18 A fast food restaurant is open from 6am to midnight. During this
time period customers arrive according to an N P P (λ(·)), where the rate function
λ(·) (in customers/hr) is as shown in Figure 5.6. The corresponding Λ(·) is shown in
Figure 5.7.

λ(t)

30
20
10
t
6 am 8 10 12 2 4 6 8 10 12
Noon Midnight

Figure 5.6 Arrival rate function for the fast food restaurant.

(i) Compute the mean and variance of the number arrivals in one day (6am to mid-
night).
Let N (t) be the number of arrivals in the first t hours after 6am. The number of
arrivals in one day is given by N (18) which is a P(Λ(18)) random variable by
definition. Hence the mean and the variance is given by Λ(18) = 275.
NON-HOMOGENEOUS POISSON PROCESS 175
Л(t)
280
260
240
220
200
180
160
140
120
100
80
60
40
20
t
6 am 8 10 12 2 4 6 8 10 12
Noon Midnight

Figure 5.7 Cumulative arrivals function for the fast food restaurant.

(ii) Compute the mean and variance of the number arrivals from 6pm to 10pm.
The number of arrivals from 6pm to 10pm is given by N (16) − N (12) which
is a P(Λ(16) − Λ(12)) random variable by definition. Hence the mean and the
variance is given by Λ(16) − Λ(12) = 255 − 180 = 75.
(iii) What is the probability that exactly one person arrives from 8am to 8:06am?
The number of arrivals from 8am to 8:06am is given by N (2.1) − N (2) which is
a P(Λ(2.1) − Λ(2)) = P(1.95) random variable by definition. Hence the required
probability is given by e−1.95 (1.95)1 /1! = 0.2774.

The next theorem gives necessary and sufficient condition for a counting process
to be an NPP.

Theorem 5.20 A counting process {N (t), t ≥ 0} is a N P P (λ(·)) if and only if

(i) {N (t), t ≥ 0} has independent increments,
(ii) N (0) = 0 and
P(N (t + h) − N (t) = 0) = 1 − λ(t)h + o(h),
P(N (t + h) − N (t) = 1) = λ(t)h + o(h),
P(N (t + h) − N (t) = j) = o(h), j ≥ 2.

Proof: The “if” part is straightforward and we leave the details to the reader, see
Conceptual Exercise 5.8. To establish the “only if” part it suffices to show that the
conditions (i) and (ii) in this theorem imply that N (t) ∼ P (Λ(t)). To do this we
define
pk (t) = P(N (t) = k), k ≥ 0.
We follow the proof of Theorem 5.13. For k ≥ 1, we get
pk (t + h) = P(N (t + h) = k)
176 POISSON PROCESSES
k
X
= P(N (t + h) = k|N (t) = j)P(N (t) = j)
j=0
k
X
= P(N (t + h) − N (t) = k − j|N (t) = j)pj (t)
j=0
k
X
= P(N (t + h) − N (t) = k − j)pj (t)
j=0
(by independence of increments)
= P(N (t + h) − N (t) = 0)pk (t) + P(N (t + h) − N (t) = 1)pk−1 (t)
k
X
+ P(N (t + h) − N (t) = j)pk−j (t)
j=2
k
X
= (1 − λ(t)h + o(h))pk (t) + (λ(t)h + o(h))pk−1 (t) + o(h)pk−j (t)
j=2
k
X
= (1 − λ(t)h)pk (t) + λ(t)hpk−1 (t) + o(h)pk−j (t).
j=0

Rearranging and dividing by h, we get

k
1 o(h) X
(pk (t + h) − pk (t)) = −λ(t)pk (t) + λ(t)pk−1 (t) + pk−j (t).
h h j=0

Letting h → 0, we get
dpk (t)
p′k (t) = = −λ(t)pk (t) + λ(t)pk−1 (t). (5.15)
dt
Proceeding in a similar fashion for the case k = 0 we get
p′0 (t) = −λ(t)p0 (t).
Using the initial condition p0 (0) = 1, the above equation admits the following solu-
tion:
p0 (t) = e−Λ(t) , t ≥ 0.
Using the initial condition pk (0) = 0 for k ≥ 1, we can solve Equation 5.15 recur-
sively to get
Λ(t)k
pk (t) = e−Λ(t) , t ≥ 0,
k!
which implies that N (t) ∼ P (Λt). This proves the theorem.
Since the conditions in the above theorem are necessary and sufficient, we can use
them as an alternate definition of an NPP.
COMPOUND POISSON PROCESS 177
5.5.1 Event Times in an NPP

Suppose the nth event in an NPP occurs at time Sn (n ≥ 1). Define Xn = Sn −Sn−1
(S0 = 0) be the nth inter-event time. In general the inter-event times are neither
independent, nor identically distributed. We can compute the marginal distribution
of Xn as given in the next theorem.

Theorem 5.21
e−Λ(t)

if n = 0,
P(Xn+1 > t) = R ∞ −Λ(t+s) Λ(t+s)n
0 e n! ds if n ≥ 1.

Proof: Left as Conceptual Exercise 5.14.

Next we generalize Theorem 5.14 to non-homogeneous Poisson processes. Let t ≥
0 be fixed and {Ui , 1 ≤ i ≤ n} be iid random variables with common distribution
Λ(s)
P(Ui ≤ s) = , 0 ≤ s ≤ t.
Λ(t)
Let Ũ1 ≤ Ũ2 ≤ · · · ≤ Ũn be the order statistics of {Ui , 1 ≤ i ≤ n}.

Theorem 5.22 Campbell’s Theorem for NPPs. Let Sn be the nth event time in an
N P P (λ(·)) {N (t), t ≥ 0}. Given N (t) = n,
(S1 , S2 , · · · , Sn ) ∼ (Ũ1 , Ũ2 , · · · , Ũn ).

Proof: Left as Conceptual Exercise 5.11.

5.6 Compound Poisson Process

In this section we shall study another generalization of a Poisson process, this time
relaxing the assumption that the events occur one at a time. Such a generalization is
useful in practice where multiple events can occur at the same time. For example,
customers arrive in batches to a restaurant, multiple cars are involved in an accident,
demands occur in batches, etc. To model such situations we introduce the compound
Poisson process CPP as defined below.

Definition 5.8 Let {N (t), t ≥ 0} be a PP(λ) and {Zn , n ≥ 1} be a sequence of iid

random variables that is independent of the PP. Let
N (t)
X
Z(t) = Zn , t ≥ 0. (5.16)
n=1

Then {Z(t), t ≥ 0} is call a compound Poisson process.

178 POISSON PROCESSES
Note Z(t) = 0 if N (t) = 0. Also, if Zn = 1 for all n ≥ 1, Equation 5.16 reduces
to Z(t) = N (t). Thus CPP is a generalization of a PP. We have not restricted Zn ’s
to be positive, or integer valued. It is convenient to think of Zn as the cost incurred
(or reward earned) when the nth event occurs in the PP. Then Z(t) is the cumulative
cost incurred (or reward earned) over (0, t]. Since Zn ’s are allowed to be positive as
well as negative, the sample paths of {Z(t), t ≥ 0} may jump up or down. A typical
sample path is shown in Figure 5.8.

Z(t)

Z4

Z1 Z2
Z3
t
S0 S1 S2 S3 S4

Figure 5.8 A typical sample path of a compound Poisson process.

Since {Zn , n ≥ 1} is a sequence of iid random variables, and the PP has inde-
pendent increments, it follows that {Z(t), t ≥ 0} also has independent increments.
Thus if we know the marginal distribution of Z(t), then all finite dimensional dis-
tributions of {Z(t), t ≥ 0} are known. Specifically, consider the case where Zn is a
non-negative integer valued random variable. Then the state-space of {Z(t), t ≥ 0}
is {0, 1, 2, · · ·}. Let
pk (t) = P(Z(t) = k), k = 0, 1, 2, · · · .
The next theorem gives the finite dimensional distribution.

Theorem 5.23 Let {Z(t), t ≥ 0} be a CPP with state-space {0, 1, 2, · · ·}. Let 0 ≤
t1 ≤ · · · ≤ tn be given real numbers and 0 ≤ k1 ≤ · · · ≤ kn be given integers. Then
P(Z(t1 ) = k1 , Z(t2 ) = k2 , · · · , Z(tn ) = kn )
= pk1 (t1 )pk2 −k1 (t2 − t1 ) · · · pkn −kn−1 (tn − tn−1 ).

Proof: See Conceptual Exercise 5.18.

Thus the entire CPP is described by the marginal distribution of Z(t) for all t ≥
0. In general, computing the distribution of Z(t) is hard, but the Laplace Stieltjes
Transform (LST) of Z(t) can be computed easily, as shown in the next theorem.

Theorem 5.24 Let {Z(t), t ≥ 0} be a CPP as defined by Equation 5.16. Let

A(s) = E(e−sZn ), n ≥ 1.
COMPOUND POISSON PROCESS 179
Then
φ(s) = E(e−sZ(t) ) = e−λt(1−A(s)) .

Proof: We have
E(e−sZ(t) |N (t) = n) = E(e−(sZ1 +Z2 +···+Zn ) ) = A(s)n ,
since {Zn , n ≥ 1} are iid with common LST A(s). Thus
φ(s) = E(e−sZ(t) ) = E(E(e−sZ(t) |N (t))) = E((A(s))N (t) ),
which is the generating function of N (t) evaluated at A(s). Since N (t) ∼ P(λt), we
know that
E(z N (t) ) = e−λt(1−z) .
Hence
φ(s) = e−λt(1−A(s)) .
This proves the theorem.
In general it is difficult to invert φ(s) analytically to obtain the distribution of Z(t).
However, the moments of Z(t) are easy to obtain. They are given in the following
theorem.

Theorem 5.25 Let {Z(t), t ≥ 0} be a CPP as defined by Equation 5.16. Let τ and
s2 be the mean and the second moment of Zn , n ≥ 1. Then
E(Z(t)) = λτ t,
Var(Z(t)) = λs2 t.

Proof: See Conceptual Exercise 5.19.

We end this section with an example.

Example 5.19 Suppose customers arrive at a restaurant in batches that are iid with
the following common distribution
pk = P(Batch Size = k), 1 ≤ k ≤ 6,
where
[p1 , p2 , p3 , p4 , p5 , p6 ] = [.1, .25, .1, .25, .15, .15].
The batches themselves arrive according to a PP(λ). Compute the mean and variance
of Z(t), the number of arrivals during (0, t].
{Z(t), t ≥ 0} is a CPP with batch arrival rate λ and the mean and second moment
of batch sizes given by
X 6
τ= kpk = 3.55,
k=1
6
X
s2 = k 2 pk = 15.15.
k=1
180 POISSON PROCESSES
Hence, using Theorem 5.25, we get
E(Z(t)) = 3.55λt, Var(Z(t)) = 15.15λt.

5.7 Computational Exercises

5.1 Consider a network with three arcs as shown in Figure 5.9. Let Xi ∼ exp(λi )
be the legth of arc i, (1 ≤ i ≤ 3). Suppose arc lengths are independent. Compute the
distribution of the length of the shortest path from node a to node c.

X1 X2
a b c

X3

Figure 5.9 The three arc network.

5.2 Compute the probability that a − b − c is the shortest path from node a to node
c in the network in Figure 5.9.

5.3 Compute the distribution of the length of the longest path from node a to node
c in the network in Figure 5.9.

5.4 Compute the probability that a − b − c is the longest path from node a to node
c in the network in Figure 5.9.

5.5 Consider a system consisting of n components in parallel, that is, the system
fails when all components fail. The lifetimes of the components are iid exp(λ) ran-
dom variables. Compute the cdf of the time when the system fails, assuming that all
components are functioning at time 0.

5.6 A spaceship is controlled by three independent computers on board. The ship

can function as long as at least two of these computers are functioning. Suppose the
lifetimes of the computers are iid exp(λ) random variables. Assume that at time zero
all computers are functional. What is the probability that the ship functions over the
duration (0, t]?

5.7 The lifetime of a machine is an exp(λ) random variable. A maintenance crew

inspects this machine every T time units, starting at time 0, where T > 0 is a fixed
number. Find the expected time when the machine is discovered to have failed. What
is the expected duration of time when the machine is down before it is discovered to
be down?
COMPUTATIONAL EXERCISES 181
5.8 There are n organisms in a colony at time 0. The lifetimes of these organisms
are iid exp(λ) random variables. What is the probability that exactly k organisms are
living at time t? What is the expected time by which the colony experiences its kth
death? (0 ≤ k ≤ n).

5.9 A machine has two parts: A and B. There are two spares available for A and one
for B. The machine needs both parts to function. As soon as a part fails, it is replaced
instantaneously by its spare. The lifetimes of all parts are independent. The part A
and its spares have iid exp(λ) lifetimes and B and its spare have iid exp(µ) lifetimes.
Spares fail only while in service. What is the expected lifetime of the machine?

5.10 A rope consists of n strands. When a strand is carrying a load of x tons, its
failure rate is λx. At time zero all strands are working and equally share a combined
load of L tons. When a strand fails, the remaining strands share the load equally. This
process continues until all strands break, at which point the rope breaks. Compute the
distribution of the lifetime of the rope.

5.11 Two jobs are waiting to be processed by a single machine. The time required to
complete the ith job is an exp(λi ) random variable (i = 1, 2). The processing times
are independent of each other, and of the sequence in which the jobs are processed. To
keep the ith job in the machine shop costs Ci dollars per unit time. In what sequence
should the jobs be processed so that the expected total cost is minimized?

5.12 Suppose customers arrive at a single-server queue according to a PP(λ), require

iid exp(µ) service times, and are served in a first come first served fashion. At time
0 there is one customer in the system, and he is in service. What is the probability
that there will be no customer in the system after the second customer completes his
service?

5.13 Suppose there are n ≥ 3 sticks whose lengths are iid exp(λ) random variables.
Show that the probability that an n-sided polygon can be formed from these sticks is
n(1/2)n−1 . Hint: A polygon can be formed if and only if the longest stick is shorter
than the sum of the rest.

5.14 Let A, B, and C be iid exp(λ) random variables. Show that the probability
that both the roots of Ax2 + Bx + C = 0 are real is 1/3.

5.15 Suppose U is uniformly distributed over (0, 1). Show that −ln(U ) is an exp(1)
random variable.

5.16 Mary arrives at a single server service facility to find a random number, N , of
customers already in the system. The customers are served in a first come first served
order. Service of all the customers are iid exp(λ) random variables. The pmf of N is
given by P(N = n) = (1 − ρ)ρn , n = 0, 1, 2, · · · , where ρ < 1 is a given constant.
Let W be the time when Mary completes her service. Compute the LST and the pdf
of W .
182 POISSON PROCESSES
5.17 A machine needs a single critical component to operate properly. When that
component fails it is instantaneously replaced by a spare. There is a supply of k
spares. The lifetime of the original component and the k spares are iid exp(λ) random
variables. What is the smallest number of spares that must be provided to guarantee
that the system will last for at least T hours with probability α, a given constant?

5.18 A system consists of two components in series. The lifetime of the first com-
ponent is exp(λ) and that of the second component is Erl(µ, n). The system fails as
soon any one of the two components fails. Assuming the components behave inde-
pendently of each other, compute the expected lifetime of the system.

5.19 Let Xi ∼ exp(λi ) (i = 1, 2) be two independent random variables. Compute

the distribution of X1 /X2 and show that it has infinite expectation.

5.20 Let {N (t), t ≥ 0} be a PP(λ). Compute P(N (t) = k|N (t + s) = k + m) for

s, t ≥ 0, k, m ≥ 0.

5.21 Let Xi be a P(λi ) random variable, and assume that {Xi , 1 ≤ i ≤ r} are
independent. Show that X = X1 + X2 + · · · + Xn is a P(λ) random variable, where
λ = λ1 + λ2 + · · · + λn .

5.22 Customers arrive according to a PP(λ) to a service facility with s ≥ 1 identical

servers in parallel. The service times are iid exp(µ) random variables. At time 0, all
the servers are busy serving one customer each, and no customers are waiting.
(i) Compute the probability that the next arriving customer finds all servers busy.
(ii) Let N be the number of arrivals before the first service completion. Compute
the pmf of N .
(iii) Compute the probability that the next arriving customer finds at least two servers
idle.

5.23 Let Si be the ith event time in a PP(λ) {N (t), t ≥ 0}. Show that
1 − e−λt
E(SN (t) ) = t − .
λ
5.24 Let {N (t), t ≥ 0} be a PP(λ). Suppose a Bernoulli splitting mechanism tags
each event as type one with probability p, and type two with probability 1 − p. Let
Ni (t) be the number of type i events over (0, t]. Let Ti be the time until the first event
in the type i process. Compute the joint pdf of (T1 , T2 ).

5.25 Let {N (t), t ≥ 0} be a PP(λ). Compute the probability that N (t) is odd.

5.26 A system is subject to k types of shocks. Shocks of type i arise according to

a PP(λi ), 1 ≤ i ≤ r. A shock of type i induces a system failure with probability pi
independently of every thing else. Let T be the time when the system fails, and let
S be the type of the shock that causes the failure. Compute P(T > t, S = i), t ≥
0, 1 ≤ i ≤ r.
COMPUTATIONAL EXERCISES 183
5.27 Let {N (t), t ≥ 0} be a PP(λ). The events in this process are registered by
a counter that locks up for a fixed period of time τ > 0 every time it resisters an
event. While it is locked, it cannot register any event. What is the probability that the
counter is not locked at time t, assuming that it is not locked at time 0?

5.28 Suppose customers arrive at a bank according to a Poisson process at a rate of

8 per hour. Compute the following:

(i) The mean and variance of the number of customers who enter the bank during
an 8-hour day.
(ii) Probability that more than four customers enter the bank during an hour long
lunch break.
(iii) Probability than no customers arrive during the last 15 minutes of the day.
(iv) Correlation between the number of customers who enter the bank between 9 am
and 11 am, and those who enter between 10 am and noon.

5.29 Consider a one-way road where the cars form a PP(λ) with rate λ cars/sec.
The road is x feet wide. A pedestrian, who walks at a speed of u feet/sec, will cross
the road if and only if she is certain that no cars will cross the pedestrian crossing
while she is on it. Show that the expected time until she completes the crossing is
(ex/u − 1)/λ.

5.30 A machine is up at time zero. It then alternates between two states: up or down.
(When an up machine fails it goes to the down state, and when a down machine is
repaired it moves to the up state.) Let Un be the nth up-duration, followed by the
nth down-duration, Dn . Suppose {Un , n ≥ 0} is a sequence of iid exp(λ) random
variables. The nth down-duration is proportional to the nth up-duration, i.e., there is
a constant c > 0 such that Dn = cUn .

(i) Let F (t) be the number of failures up to time t. Is {F (t), t ≥ 0} a Poisson

process? If yes, what is its rate parameter? If not, why not?
(ii) Let R(t) be the number of repairs up to time t. Is {R(t), t ≥ 0} a Poisson
process? If yes, what is its rate parameter? If not, why not?

5.31 Suppose customers arrive at a system according to PP(λ). Every customer stays
in the system for an exp(µ) amount of time and then leaves. Customers behave in-
dependently of each other. Show that the number of customers in the system at time
t is µλ (1 − e−µt ).

5.32 Two individuals, 1 and 2, need kidney transplants. Without a transplant the
remaining lifetime of person i is an exp(µi ) random variable, the two lifetimes being
independent. Kidneys become available according to a Poisson process with rate λ.
The first available kidney is supposed to go to person 1 if she is still alive when the
kidney becomes available; else it will go to person 2. The next kidney will go person
2, if he is still alive and has not already received a kidney. Compute the probability
that person i receives a new kidney (i = 1, 2).
184 POISSON PROCESSES
5.33 It has been estimated that meteors entering the earth’s atmosphere over a cer-
tain region form a Poisson process with rate λ = 100 per hour. About 1 percent of
those are visible to the naked eye.

(i) What is the probability that a person is unlucky enough not to see any shooting
stars in a one hour period?
(ii) What is the probability that a person will see two shooting stars in one minute?
More than two in one minute?

5.34 A machine is subject to shocks that arrive according to a PP(λ). The strength of
each shock is non-negative random variable with cdf G(·). If the shock has strength
x it causes the machine to fail with probability p(x). Assuming the successive shock
strengths are independent, what is the distribution of the lifetime of the machine?

5.35 Customers arrive at a bank according to a PP with rate of 10/hour. Forty percent
of them are men, and the rest are women. Given that 10 men have arrived during the
first hour, what the expected number of women who arrived in the first hour?

5.36 Let c > 0 be a constant and define:


c 2n < t < 2n + 1, n ≥ 0
λ(t) =
0 2n + 1 ≤ t ≤ 2n + 2, n ≥ 0.
Let {N (t), t ≥ 0} be a NPP with the above rate function. Compute the distribution
of S1 , the time of occurrence of the first event.

5.37 For the NPP of Computational Exercise 5.36, compute the distribution of N (t).

5.38 For the NPP of Computational Exercise 5.36, compute E(S1 |N (t) = n), for
0 ≤ t ≤ 2.

5.39 Redo Example 5.17 under the assumption that the arrivals to the library follow
an N P P (λ(·)).

5.40 A factory produces items one at a time according a PP(λ). These items are
loaded onto trucks that leave the factory according to a PP(µ) and transport the items
to a warehouse. The truck capacity is large enough so that all the items produced after
the departure of a truck can be loaded onto the next truck. The travel time between the
factory and the warehouse is a constant and can be ignored. Let Z(t) be the number
of items received at the warehouse during (0, t]. Is {Z(t), t ≥ 0} a PP, NPP, or CPP
or none of the above? Show that E(Z(t)) = λt − λ(1 − e−µt )/µ.

5.41 A customer makes deposits in a bank according to a PP with rate λd per week.
The sizes of successive deposits are iid random variables with mean τd and vari-
ance σd2 . Compute the mean and variance of the total amount deposited over [0, t].
Unknown to the customer, the customer’s spouse makes withdrawals from the same
account according to a PP with rate λw . The sizes of successive withdrawals are iid
CONCEPTUAL EXERCISES 185
2
random variables with mean τw and variance σw . Assume that the deposit and with-
drawal processes are independent of each other. Let Z(t) be the account balance at
time t. Show that {Z(t), t ≥ 0} is a CPP. Compute the mean and variance of Z(t).

5.42 Consider a CPP {Z(t), t ≥ 0} with batch size distribution

P(Zn = k) = (1 − α)k−1 α, k ≥ 1.
Compute the LST of Z(t). Hence or otherwise, compute P(Z(t) = k).

5.43 The lifetime of an item is a nonnegative continuous random variable with cdf
F (·) and pdf f (·). Assume that F (0) = 0 and F (t) < 1 for all t < ∞, with F (∞) =
1.

(i) When the item fails at age t, we perform a minimal repair, i.e., we instanta-
neously restore it to functional state, but its failure rate remains unchanged. Thus
after minimal repair of a failed item of age t, the item behaves like a functioning
item of age t. Suppose a new item is put in use at time 0. Let N (t) be the number
of failures up to time t. Is {N (t), t ≥ 0} a non-homogenous Poisson process?
Why or why not? If it is, what is its rate function?
(ii) Suppose the expected cost of performing minimal repair on a failed item of age
t is c(t) dollars. Compute the expected total cost of repairs for this item up to a
given time T > 0.

5.44 Let N (t) be the total number of cameras sold by a store over (0, t] (years).
Suppose {N (t), t ≥ 0} is an NPP with rate function
λ(t) = 200(1 + e−t ), t ≥ 0.
n
Suppose the price of the camera is $350(1.08) in year n = 0, 1, 2, .... The interval
from t = n to t = n + 1 is called year n. Thus sales rate changes continuously with
time, but the price changes from year to year. Compute the mean and variance of the
sales revenue in the nth year.

5.8 Conceptual Exercises

5.1 Derive Equation 5.4.

5.2 Let X ∼ Erl(n, λ) and Y ∼ Erl(m, µ) be two independent Erlang random

variables. Let
F (n, m) = P(X < Y ).
Show that F (n, m) satisfy the following recursion
λ µ
F (n, m) = F (n − 1, m) + F (n, m − 1), n, m ≥ 1,
λ+µ λ+µ
with boundary conditions
F (n, 0) = 0, n ≥ 1; F (0, m) = 1, m ≥ 1.
186 POISSON PROCESSES
Hence or otherwise show that
n+m−1
X n + m − 1  λ k  µ n+m−k−1
F (n, m) = .
k λ+µ λ+µ
k=n

5.3 Let Xi ∼ exp(λi ), i = 1, 2, · · · , k be k independent random variables. Show

that, for 1 ≤ i ≤ k,
λi
P(Xi < Xj , j 6= i) = ,
λ
where
Xk
λ= λj .
j=1

5.4 Prove Equation 5.7.

5.5 Using Laplace transforms (LT) solve Equations 5.10 and 5.9 as follows: denote
the LT of pk (t) by p∗k (s). Using appropriate initial conditions show that
(λ + s)p∗0 (s) = 1
(λ + s)p∗k (s) = λp∗k−1 (s), k ≥ 1.
Solve these to obtain
λk
p∗k (s) = , k ≥ 0.
(λ + s)k+1
Invert this to obtain Equation 5.11.

5.6 Let {Ni (t), t ≥ 0} (i = 1, 2) be two independent Poisson processes with rates
λ1 and λ2 , respectively. At time 0 a coin is flipped that turns up heads with probability
p. Define 
N1 (t) if the coin turns up heads,
N (t) =
N2 (t) if the coin turns up tails.

(i) Is {N (t), t ≥ 0} a PP?

(ii) Does {N (t), t ≥ 0} have stationary increments?
(iii) Does {N (t), t ≥ 0} have independent increments?

5.7 Show that if {N (t), t ≥ 0} is an NPP(λ(·)),

(i) N (0) = 0,
(ii) N (s + t) − N (s) ∼ P (Λ(s + t) − Λ(s)).

Hint: Use generating functions to show part (ii).

5.8 Prove the “if” part of Theorem 5.20 by using the independence of increments of
{N (t), t ≥ 0} and that N (t + h) − N (t) ∼ P (Λ(t + h) − Λ(t)).
CONCEPTUAL EXERCISES 187
5.9 Let {Ni (t), t ≥ 0} (i = 1, 2) be two independent Poisson processes with rates
λ1 and λ2 , respectively. Let Ai be the number of events in the ith process before the
first event in the jth process (i 6= j).

(i) Compute P(Ai = k) for k ≥ 0.

(ii) Are A1 and A2 independent random variables?

5.10 Prove Theorem 5.19.

5.11 Prove Theorem 5.22 by following the proof of Theorem 5.14 on page 163.

5.12 Let {Ni (t), t ≥ 0} (i = 1, 2) be two independent Poisson processes with rates
λ1 and λ2 , respectively. Let T be a non-negative random variable that is independent
of both these processes. Let F be its cdf and φ be its LST. Let Bi = Ni (T ).

(i) Compute the generating function ψi (z) = E(z Bi ) in terms of φ, i = 1, 2.

(ii) Compute the joint generating function ψ(z1 , z2 ) = E(z1B1 z2B2 ) in terms of φ.
(iii) Are B1 and B2 independent? What if T is a constant?

5.13 This is an example of deterministic splitting. Let {N (t), t ≥ 0} be a PP(λ).

Suppose all odd events (i.e., events occurring at time S2n+1 for n ≥ 0) in this process
are classified as type 1 events, and all even events are classified as type 2 events. Let
Ni (t) be the number of type i (i = 1, 2) events that occur during (0, t].

(i) Is {N1 (t), t ≥ 0} a PP?

(ii) Is {N2 (t), t ≥ 0} a PP?
(iii) Are {Ni (t), t ≥ 0} (i = 1, 2) independent of each other?

5.14 Prove Theorem 5.21 by conditioning on Sn .

5.15 Let {Ni (t), t ≥ 0} be an NPP(λi (·)), (1 ≤ i ≤ r). Suppose they are indepen-
dent and define
N (t) = N1 (t) + N2 (t) + · + Nr (t), t ≥ 0.
Show that {N (t), t ≥ 0} is an N P P (λ(·)), where
λ(t) = λ1 (t) + λ2 (t) + · · · + λr (t), t ≥ 0.

5.16 Show that the process {R(t), t ≥ 0} of Theorem 5.18 is an NPP with rate
function λ(t) = λp(t), t ≥ 0.

5.17 Let {N (t), t ≥ 0} be an N P P (λ(·)), and p : [0, ∞) → [0, 1] be a function.

Suppose an event occurring in the NPP at time s is registered with probability p(s),
independent of everything else. Let R(t) be the number of registered events during
(0, t]. Show that {R(t), t ≥ 0} is an NPP with rate function λ(t)p(t), t ≥ 0.
188 POISSON PROCESSES
5.18 Prove Theorem 5.23, by following the initial steps in the proof of Theo-
rem 5.11.

5.19 Prove Theorem 5.25 by using the following identities:

E(Z(t)) = −φ′ (0), E(Z(t)2 ) = φ′′ (0).

5.20 Non-homogeneous compound Poisson process. A process {Z(t), t ≥ 0} of

Equation 5.16 is called a non-homogeneous compound Poisson process if {N (t), t ≥
0} is an N P P (λ(·)). Compute the LST, the mean, and the variance of Z(t).

5.21 Prove or disprove that the process {R(t), t ≥ 0} defined in Example 5.17 a
non-homogeneous Poisson process.

5.22 Derive Equations 5.12, 5.13, and 5.14.

5.23 A two dimensional PP(λ) is a process of points occurring randomly in a Eu-

clidean plane R2 so that (i) the number of points occurring in a region of area A is
P (λA) random variable, and (ii) the number of points in non-overlapping regions
are independent. Let X1 be the Euclidean distance from the origin of the point closest
to the origin. Compute P (X1 > x).

5.24 Derive the variance-covariance matrix of (S1 , S2 , · · · , Sn ) given N (t) = n,

where {N (t), t ≥ 0} is a PP(λ).
 CHAPTER 6

Continuous-Time Markov Chains

The amount of pain involved in solving a mathematical problem is independent of

the route taken to solve it.

Proof: Follows from the general theorem, “there is no such thing as a free lunch.”

6.1 Definitions and Sample Path Properties

In Chapters 2, 3, and 4 we studied DTMCs. They arose as stochastic models of sys-

tems with countable state-space that change their state at times n = 1, 2, · · ·, and
have Markov property at those times. Thus, the probabilistic nature of the future be-
havior of these systems after time n depends on the past only through their state at
time n.

In this chapter we study a system with a countable state-space that can change
its state at any point in time. Let Sn , n ≥ 1, be time of the nth change of state or
transition, Yn = Sn − Sn−1 , (with S0 = 0), be the nth sojourn time, and Xn be the
state of the system after the nth transition. Define
N (t) = sup{n ≥ 0 : Sn ≤ t}, t ≥ 0.
Thus N (t) is the number of transitions the system undergoes over (0, t], and
{N (t), t ≥ 0} is a counting process generated by {Yn , n ≥ 1}. It has piecewise con-
stant sample paths that start with N (0) = 0 and jump up by +1 at times Sn , n ≥ 1.

We make the following important assumption, called the “regularity assumption”:

P( lim Sn = ∞) = 1. (6.1)
n→∞

This ensures that

P(N (t) < ∞) = 1, (6.2)
for finite t. The regularity condition implies that the system undergoes finite number
of transitions during finite intervals of time. This is a critical assumption, and we

189
190 CONTINUOUS-TIME MARKOV CHAINS
always assume that it holds.

Now let X(0) = X0 be the initial state of the system, and X(t) be the state of
the system at time t. Under the regularity assumption of Equation 6.1, N (t) is well
defined for each t ≥ 0, and hence we can write
X(t) = XN (t) , t ≥ 0. (6.3)
The continuous time stochastic system {X(t), t ≥ 0} has piece-wise constant right-
continuous sample paths. A typical sample path of such a system is shown in Fig-
ure 6.1. We see that the {X(t), t ≥ 0} process is in the initial state X0 at time t = 0.

X(t)

X1

X4

X0
S1 S2 S3 S4
0 t
Y1 Y2 Y3 Y4
X3

X2

Figure 6.1 A typical sample path of a CTMC.

It stays there for a sojourn time Y1 and then jumps to state X1 . In general it stays
in state Xn for a duration given by Yn+1 and then jumps to state Xn+1 , n ≥ 0.
Note that if Xn = Xn+1 , there is no jump in the sample path of {X(t), t ≥ 0} at
time Sn+1 . Thus, without loss of generality, we can assume that Xn+1 6= Xn for all
n ≥ 0.

In this chapter we study the case where {X(t), t ≥ 0} belongs to a particular class
of stochastic processes called the continuous-time Markov chains (CTMC), defined
below.

Definition 6.1 CTMC:Definition The stochastic process {X(t), t ≥ 0} as defined

by Equation 6.3 is called a CTMC if it has a countable state-space S, and the se-
quence {X0 , (Xn , Yn ), n ≥ 1} satisfies
P(Xn+1 = j, Yn+1 > y|Xn = i, Yn , Xn−1 , Yn−1 , · · · , X1 , Y1 , X0 )
= P(X1 = j, Y1 > y|X0 = i) = pij e−qi y , i, j ∈ S, n ≥ 0, (6.4)
where P = [pi,j ]i,j∈S is a stochastic matrix with pii = 0 for all i ∈ S, and 0 ≤ qi <
∞, i ∈ S.

Thus a system modeled by a CTMC {X(t), t ≥ 0} evolves as follows: it starts in

DEFINITIONS AND SAMPLE PATH PROPERTIES 191
state X0 = i0 (say). It stays in that state for a random duration of time Y1 ∼ exp(qi0 )
and then jumps to state X1 = X(Y1 +) = i1 6= i0 with probability pi0 ,i1 , indepen-
dent of the amount of time spent in state i0 . In general, after the nth transition at time
Sn , it moves to state Xn = X(Sn +) = in 6= in−1 . It stays in that state for a random
duration of time Yn+1 ∼ exp(qin ) and then jumps at time Sn+1 to state in+1 with
probability pin ,in+1 , independent of the history over [0, Sn+1 ).

From Equation 6.4 it is clear that {Xn , n ≥ 0} is a DTMC on state-space S with

transition probability matrix P . It is called the “embedded DTMC” corresponding to
the CTMC. We illustrate with several examples below.

Example 6.1 Two-State Machine. Consider a machine that can be up or down at

any time. If the machine is up, it fails after an exp(µ) amount of time. If it is down, it
is repaired in an exp(λ) amount of time. The successive up times are iid, and so are
the successive down times, and the two are independent of each other. Define

0 if the machine is down at time t,
X(t) =
1 if the machine is up at time t.
Is {X(t), t ≥ 0} a CTMC?
The state-space of {X(t), t ≥ 0} is {0, 1}. Suppose the machine fails at time t,
i.e., the process enters state 0 at time t. Then the process stays in state 0 for an exp(λ)
amount of time and then jumps to state 1. Thus, q0 = λ and p0,1 = 1. Now suppose
the machine repair completes at time t, i.e., the process enters state 1 at time t. Then
the process stays in state 1 for an exp(µ) amount of time and then jumps to state
0. Thus, q1 = µ and p1,0 = 1. It is easy to see that the independence assumptions
implicit in Equation 6.4 are satisfied. Thus {X(t), t ≥ 0} a CTMC.

Example 6.2 Poisson Process. Show that a P P (λ) is a CTMC.

Let {X(t), t ≥ 0} be a P P (λ). (See Section 5.2.) Its state-space is {0, 1, 2, · · ·}.
Suppose the process enters state i at time t, i.e., the ith event in the PP occurs at
time t. Then the process stays there for an exp(λ) amount of time, independent of
the history, and then jumps to state i + 1. This, along with the properties of the PP,
implies that Equation 6.4 is satisfied with qi = λ and pi,i+1 = 1 for i ∈ S. Hence
{X(t), t ≥ 0} is a CTMC.

Example 6.3 Compound Poisson Process. Let {X(t), t ≥ 0} be a CPP with batch
arrival rate λ and iid integer valued batch sizes with common pmf
αk = P(Batch Size = k), k = 1, 2, 3, · · · .
Is {X(t), t ≥ 0} a CTMC?
See Section 5.6 for the definition of the CPP. The state-space of {X(t), t ≥ 0} is
{0, 1, 2, · · ·}. Suppose the process enters state i at time t, i.e., a batch arrives at time
t and brings the total number of arrivals over (0, t] to i. Then the process stays there
192 CONTINUOUS-TIME MARKOV CHAINS
for an exp(λ) amount of time (time until the next batch arrives), independent of the
history, and then jumps to state j > i if the new batch is of size j − i. This, along
with the properties of the CPP, implies that Equation 6.4 is satisfied with qi = λ,
pi,j = αj−i , j > i. Hence {X(t), t ≥ 0} is a CTMC.

We shall describe many more examples of CTMCs in the next section. In the
remaining section we study general properties of a CTMC. The next theorem shows
that a CTMC has Markov property at all times.

Theorem 6.1 Markov Property. A CTMC {X(t), t ≥ 0} with state-space S has

Markov property at each time point, i.e., for s, t ≥ 0, and i, j ∈ S,
P(X(t+s) = j|X(s) = i, X(u) : 0 ≤ u < s) = P(X(t+s) = j|X(s) = i). (6.5)
Furthermore, it is time homogeneous, i.e.,
P(X(t + s) = j|X(s) = i) = P(X(t) = j|X(0) = i). (6.6)

Proof: Let X(t) be as defined in Equation 6.3. Suppose SN (s) = ν, i.e., the last tran-
sition at or before s takes place at ν ≤ s. Then X(s) = i implies that X(ν) = i, and
Y , the sojourn time in state i that started at ν, ends after s. Thus Y > s − ν, which
implies that the remaining sojourn time in state i at time s, given by Y − (s − ν),
is exp(qi ). Also, Y depends on {X(u) : 0 ≤ u ≤ s} only via X(ν) which equals
X(s). Also, the next stateXN (s)+1 depends on the history only via X(ν) = X(s).
Thus all future evolution of the process after time s depends on the history only via
X(s) = i. This gives Equation 6.5. The same argument yields Equation 6.6 since the
qi and pij do not depend on when the process enters state i.

Intuitively, if a system is modeled by a CTMC the probabilistic nature of the future

behavior of the system after time t depends on the past only through the state of the
system at time t, for all t ≥ 0. We shall concentrate on time-homogeneous CTMCs
in this chapter. Thus, unless otherwise mentioned, when we use the term CTMC we
mean a time-homogeneous CTMC.

Now consider a CTMC {X(t), t ≥ 0} with countable state-space S, and let

pij (t) = P(X(t) = j|X(0) = i), i, j ∈ S, t ≥ 0,
be its transition probabilities and
P (t) = [pij (t)]i,j∈S , t ≥ 0,
be the transition probability matrix of the CTMC. Next, let
ai = P(X(0) = i), i ∈ S,
and
a = [ai ]i∈S ,
DEFINITIONS AND SAMPLE PATH PROPERTIES 193
be a row vector representing the initial distribution of the CTMC. The next theorem
shows how to compute the finite dimensional distributions of a CTMC.

Theorem 6.2 Characterizing a CTMC. A CTMC {X(t), t ≥ 0} is completely

described by its initial distribution a and the transition probability matrices P (t) for
each t ≥ 0.

Proof: We shall prove the theorem by showing that all finite dimensional dis-
tributions of the CTMC are determined by a and {P (t), t ≥ 0}. Let n ≥ 1,
0 ≤ t1 ≤ t2 ≤ · · · ≤ tn and i1 , i2 , · · · in ∈ S be given. We have
P(X(t1 ) = i1 , X(t2 ) = i2 , · · · , X(tn ) = in )
X
= P(X(t1 ) = i1 , X(t2 ) = i2 , · · · , X(tn ) = in |X(0) = i0 )P(X(0) = i0 )
i0 ∈S
X
= ai0 P(X(t2 ) = i2 , · · · , X(tn ) = in |X(0) = i0 , X(t1 ) = i1 )
i0 ∈S
·P(X(t1 ) = i1 |X(0) = i0 )
X
= ai0 pi0 ,i1 (t1 )P(X(t2 ) = i2 , · · · , X(tn ) = in |X(t1 ) = i1 )
i0 ∈S
(from Markov property of the CTMC)
X
= ai0 pi0 ,i1 (t1 )P(X(t2 − t1 ) = i2 , · · · , X(tn − t1 ) = in |X(0) = i1 )
i0 ∈S
(from the time homogeneity of the CTMC).
Continuing in this fashion we get
P(X(t1 ) = i1 , X(t2 ) = i2 , · · · , X(tn ) = in )
X
= ai0 pi0 ,i1 (t1 )pi1 ,i2 (t2 − t1 ) · · · pin−1 ,in (tn − tn−1 ).
i0 ∈S

This proves the theorem.

Next we address the issue of how to describe a CTMC in a tractable fashion.
Theorem 6.2 says that we can do this by specifying the initial distribution a = [ai ],
and the transition probability matrix P (t) for each t ≥ 0. However, the matrix P (t) is
hard to specify even for very simple CTMCs. Hence we look for alternative ways of
describing a CTMC. We defined a CTMC (under regularity conditions) by using the
parameters qi , (i ∈ S) and pij , (i, j ∈ S). If these parameters can determine P (t) for
all t ≥ 0, then we can use them in place of P (·). The regularity condition implies that
the answer is yes. We refer the reader to Chung (1967) and Cinlar (1975) for further
details on this topic. We shall provide a constructive proof in Section 6.3. Right now
we shall assert that in all the applications in this book, the parameters qi , (i ∈ S) and
pij , (i, j ∈ S), along with the initial distribution do describe the CTMCs completely.
Hence we use this method in the rest of this book.
194 CONTINUOUS-TIME MARKOV CHAINS
6.2 Examples

In this section we shall describe an often useful method of constructing a CTMC

model to describe the stochastic evolution of a system. It is based on the following
conceptualization of how the system evolves.

Consider a system with countable state-space. An event Eij is associated with

each pair (i, j) of states (i 6= j) so that if that event occurs while the system is in
state i, the system jumps to state j. Which event actually occurs is determined as
follows. Suppose the system enters state i at time t. Then event Eij is scheduled to
occur at time t + Tij , where Tij ∼ exp(qij ), where qij ≥ 0. If there is no event
whose occurrence can cause the system to jump from state i to state j, we simply set
qij = 0. Furthermore the random variables {Tij , j ∈ S} are mutually independent
and also independent of the history of the system up to time t. Let
Ti = min{Tik : k ∈ S, k 6= i}. (6.7)
Then the system stays in state i until time t+ Ti and then jumps to state j if Ti = Tij ,
i.e., if Eij is the first event that occurs among all the scheduled events. All the other
scheduled events are now canceled, and new events Ejk (k 6= j) are scheduled, and
the process continues.

Now let X(t) be the state of the system at time t, and Sn be the time of the nth
transition. Let Xn = X(Sn +) and Yn = Sn − Sn−1 (with S0 = 0), as before. From
the distributional and independence assumptions about {Tik , k 6= i}, we see that Ti
of Equation 6.7 is an exp(qi ) random variable, where
X
qi = qik , i ∈ S. (6.8)
k6=i

If qi = 0 there are no events that will take the system out of state i, i.e., state i is an
absorbing state. In this case we define pii = 1, since this makes the state i absorbing
in the embedded DTMC as well. If qi > 0, we have
P(Xn+1 = j, Yn+1 > y|Xn = i, Yn , Xn−1 , Yn−1 , · · · , X1 , Y1 , X0 )
= P(Tij = Ti , Ti > y)
qij −qi y
= e , i, j ∈ S, i 6= j, y ≥ 0.
qi
Now define
qij
pij = , i, j ∈ S, j 6= i. (6.9)
qi

From the above derivation it is clear that {X(t), t ≥ 0} is a CTMC with param-
eters {qi } and {pij } as defined in Equations 6.8 and 6.9. Thus we can describe a
CTMC by specifying qij , called the transition rate from state i to j, for all the pairs
EXAMPLES 195
(i, j), with i 6= j. Note that the quantity qii is as yet undefined. For strictly technical
reasons, we define X
qii = − qik = −qi , i ∈ S.
k:k6=i
It is convenient to put all the qij ’s in a matrix form:
Q = [qij ]i,j∈S .
This is called the infinitesimal generator or simply the generator matrix of the CTMC.
It will become clear later that the seemingly arbitrary definition of qii makes it easy
to write many equations of interest in matrix form. An important property of the
generator matrix is that its row sums are zero, i.e.,
X
qij = 0, i ∈ S.
j∈S

The generator matrix of a CTMC plays the same role as the one-step transition prob-
ability matrix of a DTMC.

Analogous to the DTMC case, a CTMC can also be represented graphically by

what are known as rate diagrams. A rate diagram is a directed graph in which each
node represents a state of the CTMC, and there is an arc from node i to node j if
qij > 0. We generally write qij on this arc. The rate diagram helps us visualize the
dynamic evolution of a CTMC.

In the remaining section we present several examples of the CTMCs. Modeling

a system by a CTMC is an art, and like any other art, is perfected by practice. The
first step in the modeling process is to identify the state-space. This step is guided by
intuition and by what kind of information we want the model to produce. The next
step involves identifying the triggering events Eij ’s—events that trigger a transition
from state i to j. The final step is to verify the distributional and independence as-
sumptions about the Tij ’s and obtain the transition rates qij ’s. If the Tij ’s are not
exponential, or the independence assumptions are violated, then either the system
cannot be modeled by a CTMC or the state-space needs to modified.

In the examples that follow we do not explicitly verify the independence assump-
tions. However, we urge the students to do so for themselves.

Example 6.4 Two State Machine. Let X(t) be the state of the machine of Exam-
ple 6.1. The state-space of {X(t), t ≥ 0} is {0, 1}. In state 0 we have E01 = machine
repair completes, and T01 ∼ exp(λ). Hence q01 = λ. Similarly, in state 1 we have
E10 = machine fails, and T10 ∼ exp(µ). Hence q10 = µ. Thus {X(t), t ≥ 0} is a
CTMC with generator matrix
 
−λ λ
Q= , (6.10)
µ −µ
196 CONTINUOUS-TIME MARKOV CHAINS
λ

0 1

Figure 6.2 The rate diagram of a two-state machine.

and the rate diagram as shown in Figure 6.2.

Example 6.5 Two-Machine Two-Repairperson Workshop. Now consider a ma-

chine shop that has two machines that are independent, identical, and behave as de-
scribed in Example 6.1. Each machine has its own repairperson. Let X(t) be the
number of working machines at time t. Is {X(t), t ≥ 0} a CTMC? If it is, what is its
generator matrix?

The state-space of {X(t), t ≥ 0} is {0, 1, 2}. We analyze the states one by one.

State 0. X(t) = 0 implies that both the machines are down at time t. When either
one of them gets repaired the state changes to 1. Hence we have E01 = one of the two
failed machines completes repair. Since the remaining repair times are exponential
due to memoryless property, T01 is the minimum of two independent exp(λ) random
variables. Hence T01 ∼ exp(2λ). Hence
q0 = 2λ, q01 = 2λ.

State 1. X(t) = 1 implies that one machine is up and the other is down at time
t. Now there are two triggering events: E10 = the working machine fails, and E12
= the failed machine completes repair. If E10 occurs before E12 , the process moves
to state 0, else it moves to state 2. Since the remaining repair time and life time
are exponential due to memoryless property, we see that T10 ∼ exp(µ) and T12 ∼
exp(λ). Hence
q1 = λ + µ, q12 = λ, q10 = µ.

State 2. X(t) = 2 implies that both the machines are up at time t. When either one
of them fails the state changes to 1. Hence we have E21 = one of the two working
machines fails. Since the remaining life times are exponential due to memoryless
property, T21 is the minimum of two independent exp(µ) random variables. Hence
T21 ∼ exp(2µ). Hence
q2 = 2µ, q21 = 2µ.
EXAMPLES 197
Thus {X(t), t ≥ 0} is a CTMC with generator matrix
 
−2λ 2λ 0
Q= µ −(λ + µ) λ ,
0 2µ −2µ
and the rate diagram as shown in Figure 6.3.

2λ λ

0 1 2

µ 2µ

Figure 6.3 The rate diagram for the two-machine, two-repairpersons workshop.

Example 6.6 Two-Machine One-Repairperson Workshop. Consider the ma-

chine shop of Example 6.5. Now suppose there is only one repairperson and the
machines are repaired in the order in which they fail. The repair times are iid. Let
X(t) be the number of working machines at time t. Is {X(t), t ≥ 0} a CTMC? If it
is, what is its generator matrix?

The state-space of {X(t), t ≥ 0} is {0, 1, 2}. The analysis of states 1 and 2 is

the same as in Example 6.5. In state 0, both machines are down, but only one is
under repair, and its remaining repair time is exp(λ). Hence the triggering event is
E01 = machine under repair completes repair, and we have T01 ∼ exp(λ). Thus
{X(t), t ≥ 0} is a CTMC with generator matrix
 
−λ λ 0
Q =  µ −(λ + µ) λ ,
0 2µ −2µ
and the rate diagram as shown in Figure 6.4.

λ λ

0 1 2

µ 2µ

Figure 6.4 The rate diagram for the two-machine, one-repairperson workshop.

Example 6.7 Poisson Process. Suppose {X(t), t ≥ 0} is a PP(λ). We saw in Ex-

ample 6.2 that it is a CTMC with state space S = {0, 1, 2, · · ·}. What is its generator
matrix?
198 CONTINUOUS-TIME MARKOV CHAINS
Consider state i. When the next event in the PP occurs, the process moves to state
i = i + 1. Hence we have Ei,i+1 = the next event occurs, and Ti,i+1 ∼ exp(λ).
Hence we have qi = λ, and qi,i+1 = λ, for all i ≥ 0. The generator matrix (where
we shown only non-zero elements) is given below:
 
−λ λ
 −λ λ 
Q= ,
 
 −λ λ 
.. ..
. .

and the rate diagram as shown in Figure 6.5.

λ λ λ λ
0 1 2 3

Figure 6.5 The rate diagram of a PP(λ).

Example 6.8 Pure Birth Process. An immediate extension of the PP(λ) is the pure
birth process, which is a CTMC on S = {0, 1, 2, · · ·} with the following generator
matrix:  
−λ0 λ0
 −λ1 λ1 
Q= .
 
 −λ 2 λ 2 
.. ..
. .
The rate diagram is shown in Figure 6.6. Such a process spends an exp(λi ) amount

0 1 2 3
λ0 λ1 λ2 λ3

Figure 6.6 The rate diagram for a pure birth process.

of time in state i and then jumps to state i + 1. In biological systems, X(t) represents
the number of organisms in a colony at time t, and the transition from i to i + 1
represents birth. Hence the name “pure birth process.” The parameter λi is called
the birth rate in state i. The PP(λ) is a special case of a pure birth process with all
birth rates given by λi = λ. We illustrate with several situations where pure birth
processes can arise.
Yule Process. Let X(t) be the number of amoebae at time t in a colony of amoebae.
Suppose each amoeba lives for an exp(λ) amount of time and then splits into two.
All amoebae in the colony behave independently of each other. Suppose X(t) = i.
When one of the i amoebae splits, the process moves to state i + 1. Hence Ei,i+1 =
one of the i living amoebae splits. This will happen after an exp(iλ) amount of time,
i.e., the minimum of i iid exp(λ) random variables. Thus Ti,i+1 ∼ exp(iλ). Thus
EXAMPLES 199
we get qi,i+1 = iλ, and qii = −iλ. Hence {X(t), t ≥ 0} is a pure birth process with
birth rates λi = iλ, i ≥ 0. Such a process is called the Yule process.
Yule Process with Immigration. Consider the above colony of amoebae. Suppose
amoebae arrive at this colony from outside according to a PP(θ). All amoebae in
the colony, whether native or immigrants, behave independently and identically. Let
X(t) be the number of amoebae in the colony at time t. As before, the state-space of
{X(t), t ≥ 0} is S = {0, 1, 2, · · ·}. Suppose X(t) = i. The system moves to state
i + 1 if one of the existing amoebae splits, which happens after an exp(iλ) amount
of time, or an amoeba arrive from outside, which happens after an exp(θ) amount of
time. Thus Ti,i+1 ∼ exp(iλ + θ). Hence qi,i+1 = iλ + θ and qii = −(iλ + θ). Hence
{X(t), t ≥ 0} is a pure birth process with birth rates λi = iλ + θ, i ≥ 0.
Maintenance. Suppose a machine is brand new at time zero. It fails after an
exp(θ0 ) amount of time, and is repaired instantaneously. The lifetime of a machine
that has undergone n ≥ 1 repairs is exp(θn ) random variable, and is independent
of how old the machine is. Let X(t) be the number of repairs the machine has un-
dergone over time (0, t]. Then {X(t), t ≥ 0} is a pure birth process with birth rates
λi = θi , i ≥ 0.

Example 6.9 Pure Death Process. A CTMC {X(t), t ≥ 0} on S = {0, 1, 2, · · ·}

is called a pure death process if its generator matrix is given by
 
0
 µ1 −µ1 
 
Q=
 µ2 −µ2 .


 µ3 −µ3 

.. ..
. .
The rate diagram is shown in Figure 6.7. Such a process spends an exp(µi ) amount

0 1 2 3
µ1 µ2 µ3 µ4

Figure 6.7 The rate diagram for a pure death process.

of time in state i ≥ 1 and then jumps to state i − 1. In biological systems, X(t) repre-
sents the number of organisms in a colony at time t, and the transition from i to i − 1
represents death. Hence the name “pure death process.” The parameter µi is called
the death rate in state i. Note that state 0 is absorbing, with q00 = 0. Thus once the
colony has no members left in it, it stays extinct forever. We discuss a special case
below.

Multi-Component Machine. Consider a machine consisting of n parts. The life-

times of these parts are iid exp(µ) random variables. The failed parts cannot be
repaired. Once all parts fail, the machine fails, and stays that way forever. Let X(t)
200 CONTINUOUS-TIME MARKOV CHAINS
be the number of functioning parts at time t. Suppose X(t) = i ≥ 1. The system
moves to state i − 1 if one of the i components fails, which happens after an exp(iµ)
time. Hence Ti,i+1 ∼ exp(iµ). Hence {X(t), t ≥ 0} is a pure death process with
death rates µi = iµ. The initial state is X(0) = n.

Example 6.10 Birth and Death Process. A CTMC {X(t), t ≥ 0} on S = {0, 1, 2,

· · ·} is called a birth and death process if its generator matrix is given by
 
−λ0 λ0
 µ1 −(λ1 + µ1 ) λ1 
 
Q=
 µ2 −(λ 2 + µ2 ) λ 2 .


 µ 3 −(λ 3 + µ 3 ) λ 3 

.. ..
. .
The rate diagram is shown in Figure 6.8. In biological systems, X(t) represents the
λ0 λ1 λ2 λ3

0 1 2 3

µ1 µ2 µ3 µ4

Figure 6.8 The rate diagram for a birth and death process.

number of organisms in a colony at time t, and the transition from i to i−1 represents
death and that from i to i + 1 represents birth. Hence the name “birth and death
process.” The parameter µi is called the death rate in state i and λi is called the
birth rate in state i. We define µ0 = 0, since death cannot occur when there are no
organisms to die. A birth and death process spends exp(λi + µi ) amount of time in
state i, and then jumps to state i − 1 with probability µi /(λi + µi ), or to state i + 1
with probability λi /(λi + µi ). A very large number of queuing models give rise to
birth and death processes and we shall study them in detail in Chapter 7. Here we
give a few examples.

Example 6.11 Single-Server Queue. Consider a single-server service system

where customers arrive according to a PP(λ) and request iid exp(µ) service times.
The customers wait in an unlimited waiting area and are served in the order of ar-
rivals. Let X(t) be the number of customers in the system (waiting or in service) at
time t. We shall show that {X(t), t ≥ 0} is a birth and death process.

The state-space is S = {0, 1, 2, · · ·}. Suppose X(t) = 0, that is, there are no
customers in the system at time t. The triggering event is E01 = arrival of a new
customer. From the properties of the PP we have T ∼ exp(λ). Thus q01 = λ. Now
suppose X(t) = i > 0. Then one customer is in service and i − 1 are waiting at
time t. Now there are two triggering events: Ei,i+1 = arrival of a new customer,
and Ei,i−1 = departure of the customer in service. From the memoryless property of
exponentials, we see that Ti,i+1 ∼ exp(λ) and Ti,i−1 ∼ exp(µ). Hence qi,i+1 = λ
EXAMPLES 201
and qi,i−1 = µ for i ≥ 1. Hence {X(t), t ≥ 0} is a birth and death process with birth
parameters λi = λ for i ≥ 0, and death parameters µi = µ for i ≥ 1.

Example 6.12 Infinite-Server Queue. Suppose customers arrive according to

PP(λ) to a service system with infinite servers. The customer service times are iid
exp(µ) random variables. Since there are an infinite number of servers, there is no
waiting and each arriving customer starts getting served at the time of arrival. Such
systems arise as models of self service systems such as parking lots, cafeterias, read-
ing rooms, etc. Let X(t) be the number of customers in the system at time t. We shall
show that {X(t), t ≥ 0} is a birth and death process.

The state-space is S = {0, 1, 2, · · ·}. Suppose X(t) = i ≥ 0. Then i customers

are in service at time t. Now there are two triggering events: Ei,i+1 = arrival of a
new customer, and Ei,i−1 = departure of one of the i customers in service. From
the memoryless property of exponentials, we see that Ti,i+1 ∼ exp(λ) and Ti,i−1 ∼
exp(iµ). Hence qi,i+1 = λ and qi,i−1 = iµ for i ≥ 1. Hence {X(t), t ≥ 0} is a
birth and death process with birth parameters λi = λ for i ≥ 0, and death parameters
µi = iµ for i ≥ 0.

Example 6.13 Linear Growth Model. Consider a colony of individuals whose

lifetimes are independent exp(µ) random variables. During its lifetime an organism
produces offspring according to a PP(λ), independent of everything else. Let X(t)
be the number of organisms in the colony at time t. We shall show that {X(t), t ≥ 0}
is a birth and death process.

The state-space is S = {0, 1, 2, · · ·}. Suppose X(t) = 0, that is, there are no
organisms in the colony at time t. Then X(u) = 0 for all u ≥ t. Thus state 0
is absorbing. Now suppose X(t) = i > 0. Now there are two triggering events:
Ei,i+1 = one of the i organisms gives birth to an individual, and Ei,i−1 = one of
the i organisms dies. From the results about the superposition of Poisson processes,
we see that Ti,i+1 ∼ exp(iλ) and from the memoryless properties of exponential
random variables we get Ti,i−1 ∼ exp(iµ). Hence qi,i+1 = iλ and qi,i−1 = iµ.
Hence {X(t), t ≥ 0} is a birth and death process with birth parameters λi = iλ and
death parameters µi = iµ for i ≥ 0.

Example 6.14 N -Machine Workshop. Consider an extension of Example 6.5 to

N independent and identical machines, each with its own dedicated repairperson. Let
X(t) be the number of working machines at time t. We shall show that {X(t), t ≥ 0}
is a birth and death process.

The state-space of {X(t), t ≥ 0} is {0, 1, · · · , N }. Suppose X(t) = i, i.e., i

machines are up and N − i machines are down at time t. There are two possible trig-
gering events: Ei,i−1 = one of the i functioning machines fails, and Ei,i+1 = one of
202 CONTINUOUS-TIME MARKOV CHAINS
the N − i machines under repair completes its repair. We see that Ti,i−1 is the mini-
mum of i iid exp(µ) random variables, hence Ti,i−1 ∼ exp(iµ). Similarly, Ti,i+1 is
the minimum of N − i iid exp(λ) random variables, hence Ti,i+1 ∼ exp((N − i)λ).
Thus we get qi,i−1 = iµ and qi,i+1 = (N − i)λ, 0 ≤ i ≤ N . Hence {X(t), t ≥ 0} is
a birth and death process with birth parameters λi = (N − i)λ and death parameters
µi = iµ for 0 ≤ i ≤ N .

Example 6.15 Retrial Queue. We describe a simple retrial queue here. Consider
a single-server system where the customers arrive according to a PP(λ) and have iid
exp(µ) service times. Suppose the system capacity is 1. Thus if an arriving customer
finds the server idle, he immediately starts getting served. On the other hand, if an
arriving customer finds the server busy, he goes away (we say he joins an orbit) and
tries his luck again after an exp(θ) amount of time independent of everything else.
He persists this way until he is served. All customers behave in this fashion. We
model this as a CTMC.

Let X(t) be the number of customers in service at time t. Obviously X(t) ∈

{0, 1}. Let R(t) be the number of customers in the orbit. We consider the bivari-
ate process {(X(t), R(t)), t ≥ 0} with state-space S = {(i, k) : i = 0, 1; k =
0, 1, 2, · · ·}.

Suppose (X(t), R(t)) = (1, k), k ≥ 0, i.e., the server is busy and there are k cus-
tomers in the orbit at time t. There are two possible events that can lead to a state
change: E(1,k),(1,k+1) = an external arrival occurs, and E(1,k),(0,k) = the customer
in service departs. Notice that if a customer in orbit conducts a retrial, he simply re-
joins the orbit, and hence there is no change of state. We have T(1,k),(1,k+1) ∼ exp(λ)
and T(1,k),(0,k) ∼ exp(µ). Thus q(1,k),(1,k+1) = λ and q(1,k),(0,k) = µ.

Next suppose (X(t), R(t)) = (0, k), k ≥ 1, i.e., the server is idle and there are
k customers in the orbit at time t. There are two possible events that can lead to a
state change: E(0,k),(1,k) = an external arrival occurs, and E(0,k),(1,k−1) = one of
the k customers in the orbit conducts a retrial, and finding the server idle, joins ser-
vice. We have T(0,k),(1,k) ∼ exp(λ) and T(0,k),(1,k−1) , being the minimum of k iid
exp(θ) random variables, is an exp(kθ) random variable. Thus q(0,k),(1,k) = λ and
q(0,k),(1,k−1) = kθ.

Finally, suppose (X(t), R(t)) = (0, 0). There is only one triggering event in this
state, namely, arrival of an external customer, leading to the new state (1, 0). Hence
T(0,0),(1,0) ∼ exp(λ). Thus {(X(t), R(t)), t ≥ 0} is a CTMC. The rate diagram is
shown in Figure 6.9.
TRANSIENT BEHAVIOR: MARGINAL DISTRIBUTION 203
λ λ λ λ
1,0 1,1 1,2 1,3
λ

θ 2θ 3θ
µ λ µ λ µ λ µ λ 4θ

0,0 0,1 0,2 0,3

Figure 6.9 The rate diagram for the retrial queue.

6.3 Transient Behavior: Marginal Distribution

In this section we study the transient behavior of a CTMC {X(t), t ≥ 0} on state-

space S = {0, 1, 2, · · ·} with initial distribution a = [ai ] and transition probability
matrices {P (t), t ≥ 0}. To be specific, we study the pmf of X(t) for a fixed t. Let
pj (t) = P(X(t) = j), j ∈ S, t ≥ 0,
and
p(t) = [pj (t)]j∈S
be the pmf of X(t). Conditioning on X(0) we get
pj (t) = P(X(t) = j)
X
= P(X(t) = j|X(0) = i)P(X(0) = i)
i∈S
X
= ai pij (t).
i∈S

In matrix form, we get

p(t) = aP (t), t ≥ 0.
Thus it suffices to study P (t) to study p(t). We start with the following theorem.

Theorem 6.3 Chapman-Kolmogorov Equations. Let P (t) be the transition prob-

ability matrix of a CTMC {X(t), t ≥ 0}. Then

(i) pij (t) ≥ 0, i, j ∈ S, t ≥ 0.

X
(ii) pij (t) = 1, i ∈ S, t ≥ 0. (6.11)
j∈S
X
(iii) pij (s + t) = pik (s)pkj (t), i, j ∈ S, s, t ≥ 0. (6.12)
k∈S

Proof: Part (i) is obvious since pij (t)’s are conditional probabilities. To show Equa-
tion 6.11, we assume the regularity condition in 6.1 and use the representation in
Equation 6.3 to get
X X
pij (t) = P(XN (t) = j|X0 = i)
j∈S j∈S
204 CONTINUOUS-TIME MARKOV CHAINS
∞
XX
= P(XN (t) = j|X0 = i, N (t) = n)P(N (t) = n|X0 = i)
j∈S n=0
XX ∞
= P(Xn = j|X0 = i, N (t) = n)P(N (t) = n|X0 = i)
j∈S n=0
X∞ X
= P(Xn = j|X0 = i, N (t) = n)P(N (t) = n|X0 = i)
n=0 j∈S
X
= P(N (t) = n|X0 = i)
j∈S
= P(N (t) < ∞|X0 = i) = 1.
Here the second to last equality is a result of the fact that Xn ∈ S for all n ≥ 0, and
the last equality is the result of Equation 6.2.
To show Equation 6.12, we condition on X(s) to get
pij (s + t) = P(X(s + t) = j|X(0) = i)
X
= P(X(s + t) = j|X(0) = i, X(s) = k)P(X(s) = k|X(0) = i)
k∈S
X
= P(X(s + t) = j|X(s) = k)pik (s)
k∈S
(from Markov property, Equation 6.5)
X
= pik (s)P(X(t) = j|X(0) = k)
k∈S
(from time-homogeneity, Equation 6.6)
X
= pik (s)pkj (t).
k∈S

This completes the proof.

Equation 6.12 is called the Chapman-Kolmogorov equation for the CTMCs. It is
analogous to the Chapman-Kolmogorov equation for the DTMCs, as given in Equa-
tion 2.21. Equation 6.12 can be written in matrix form as
P (s + t) = P (s)P (t), s, t ≥ 0.
By interchanging s and t we get
P (s + t) = P (t)P (s), s, t ≥ 0.
This shows a very important and unusual property of the matrices {P (t), t ≥ 0}:
they commute, i.e.,
P (s)P (t) = P (t)P (s), s, t ≥ 0.

We next develop a set of differential equations satisfied by the transition probabil-

ity matrix P (t) in the next theorem. We say that P (t) is differentiable with respect
to t if pij (t) is differentiable with respect to t for all t > 0 and i, j ∈ S. We define
TRANSIENT BEHAVIOR: MARGINAL DISTRIBUTION 205
the derivative of the matrix P (t) with respect to t as the matrix of the derivatives of
pij (t).

Theorem 6.4 Forward and Backward Equations. Let P (t) be the transition prob-
ability matrix of a CTMC with state-space S = {0, 1, 2, · · ·} and generator matrix
Q. Then P (t) is differentiable with respect t and satisfies
d
P (t) = P ′ (t) = QP (t), (Backward Equations) (6.13)
dt
and
d
P (t) = P ′ (t) = P (t)Q, (Forward Equations) (6.14)
dt
with initial condition
P (0) = I,
where I is an identity matrix of appropriate size.

Proof: We shall first prove that

pij (h) = δij + qij h + o(h), i, j ∈ S, (6.15)
where, δij = 1 if i = j and 0 otherwise, and o(h) is a function such that
o(h)
lim = 0.
h
h→0

From the properties of the sequence {X0 , (Xn , Yn ), n ≥ 1}, we get

P(N (h) = 0|X0 = i) = P(Y1 > h|X0 = i)
= e−qi h = 1 − qi h + o(h)
= 1 + qii h + o(h).
Also,
P(N (h) = 1|X0 = i) = P(Y1 ≤ h, Y1 + Y2 > h|X0 = i)
X Z h
= qi e−qi s e−qj (h−s) pij ds
j∈S,j6=i 0

X 1 − e−(qi −qj )h
= qi pij e−qj h (assuming qi 6= qj )
qi − qj
j∈S,j6=i
= qi h + o(h)
where the last equality follows after a bit of algebra. Similar analysis yields
P(N (h) ≥ 2|X0 = i) = o(h).
Using these we get
pii (h) = P(X(h) = i|X(0) = i)
= P(XN (h) = i|X0 = i)
= P(XN (h) = i|X0 = i, N (h) = 0)P(N (h) = 0|X0 = i)
206 CONTINUOUS-TIME MARKOV CHAINS
+P(XN (h) = i|X0 = i, N (h) = 1)P(N (h) = 1|X0 = i)
+P(XN (h) = i|X0 = i, N (h) ≥ 2)P(N (h) ≥ 2|X0 = i)
= 1 · (1 + qii h + o(h)) + 0 · (qi h + o(h)) + o(h)
= 1 + qii h + o(h).
Also, for j 6= i, we get
pij (h) = P(X(h) = j|X(0) = i)
= P(XN (h) = j|X0 = i)
= P(XN (h) = j|X0 = i, N (h) = 0)P(N (h) = 0|X0 = i)
+P(XN (h) = j|X0 = i, N (h) = 1)P(N (h) = 1|X0 = i)
+P(XN (h) = j|X0 = i, N (h) ≥ 2)P(N (h) ≥ 2|X0 = i)
= 0 · (1 + qii h + o(h)) + P(X1 = j|X0 = i)(qi h + o(h)) + o(h)
= pij (qi h + o(h)) + o(h) = qij h + o(h).
This proves Equation 6.15. Using this we get
X
pij (t + h) = pik (h)pkj (t)
k∈S
(Chapman-Kolmogorov Equations 6.12)
X
= (δik + qik h + o(h))pkj (t) (Equation 6.15).
k∈S

Hence X
pij (t + h) − pij (t) = qik hpkj (t) + o(h).
k∈S
Dividing by h yields
pij (t + h) − pij (t) X o(h)
= qik pkj (t) + .
h h
k∈S

Letting h → 0 we see that the right hand side has a limit, and hence pij (t) is differ-
entiable with respect to t and satisfies
X
p′ij (t) = qik pkj (t).
k∈S

Writing this in matrix form we get the backward equations 6.13. The forward
equations follow similarly by interchanging t and h in applying the Chapman-
Kolmogorov equations.
Thus one may solve forward or backward equations to obtain P (t). Once we have
the matrix P (t), we have the distribution of X(t). We illustrate by means of an
example.

Example 6.16 Two-State Machine. Consider the CTMC of Example 6.4 with the
rate matrix given in Equation 6.10. The forward equations are
p′00 (t) = −λp00 (t) + µp01 (t), p00 (0) = 1,
TRANSIENT BEHAVIOR: OCCUPANCY TIMES 207
p′01 (t) = λp00 (t) − µp01 (t), p01 (0) = 0,
p′10 (t) = −λp10 (t) + µp11 (t), p10 (0) = 0,
p′11 (t) = λp10 (t) − µp11 (t), p11 (0) = 1,
and the backward equations are
p′00 (t) = −λp00 (t) + λp10 (t), p00 (0) = 1,
p′10 (t) = µp00 (t) − µp10 (t), p10 (0) = 0,
p′01 (t) = −λp01 (t) + λp11 (t), p01 (0) = 0,
p′11 (t) = µp10 (t) − µp11 (t), p11 (0) = 1.
Note that we do not need to solve four equations in four unknowns simultaneously,
but only in two equations in two unknowns at a time. We solve the first two forward
equations here. We have
p00 (t) + p01 (t) = P(X(t) = 0 or 1|X(0) = 0) = 1. (6.16)
Substituting for p01 (t) in the first forward equation we get
p′00 (t) = −λp00 (t) + µ(1 − p00 (t)) = −(λ + µ)p00 (t) + µ.
This is a non-homogeneous first order differential equation with constant coeffi-
cients, and can be solved by standard methods (see Appendix I) to get
λ µ
p00 (t) = e−(λ+µ)t + .
λ+µ λ+µ
Using Equation 6.16 we get
λ
p01 (t) = (1 − e−(λ+µ)t ).
λ+µ
Similarly we can solve the last two forward equations to get p10 (t) and p11 (t). Com-
bining all these solutions we get
" #
λ −(λ+µ)t µ λ −(λ+µ)t
λ+µ e + λ+µ λ+µ (1 − e )
P (t) = µ −(λ+µ)t µ λ . (6.17)
λ+µ (1 − e ) λ+µ e−(λ+µ)t + λ+µ

It is easy to check that P (t) satisfies the Chapman-Kolmogorov Equations 6.12 and
that P (t) and P (s) commute. Figure 6.10 shows graphically various pij (t) functions.

6.4 Transient Behavior: Occupancy Times

Following the steps in the study of DTMCs, we now study the occupancy times in
the CTMCs. Let {X(t), t ≥ 0} be a CTMC on state-space S with generator matrix
Q. Let Vj (t) be the amount of time the CTMC spends in state j over (0, t]. Thus
Vj (0) = 0 for all j ∈ S. Define
Mij (t) = E(Vj (t)|X(0) = i), i, j ∈ S, t ≥ 0. (6.18)
208 CONTINUOUS-TIME MARKOV CHAINS

1
P11(t)
λ
λ+µ P00(t)

µ P01(t)
λ+µ P10(t)

Figure 6.10 The transition probability functions for the two-state CTMC.

Mij (t) is called the occupancy time of state j up to time t starting from state i. Define
the occupancy times matrix as
M (t) = [Mij (t)].
The next theorem shows how to compute the occupancy times matrix M (t).

Theorem 6.5 Occupancy Times Matrix. We have

Z t
M (t) = P (u)du, t ≥ 0, (6.19)
0
where the integral of a matrix is defined to be the matrix of the integrals of its ele-
ments.

Proof: Fix a j ∈ S. Let Z(u) = 1 if X(u) = j, and zero otherwise. Then

Z t
Vj (t) = Z(u)du.
0
Hence we get
Mij (t) = E(Vj (t)|X(0) = i)
Z t 
= E Z(u)du|X(0) = i
0
Z t
= E(Z(u)|X(0) = i)du
0
Z t
= P(X(u) = j|X(0) = i)du
0
Z t
= pij (u)du.
0
Writing the above equation in matrix form yields Equation 6.19. This proves the the-
orem.

We illustrate with an example.

COMPUTATION OF P (T ): FINITE STATE-SPACE 209
Example 6.17 Two-State Machine. Consider the CTMC of Example 6.4 with the
rate matrix given in Equation 6.10. The transition probability matrix P (t) of this
CTMC is given in Equation 6.17. Using that, and carrying out the integration, we see
that the occupancy matrix is given by
" µ #
λ −(λ+µ)t λ λ −(λ+µ)t
λ+µ t + (λ+µ)2 (1 − e ) λ+µ t − (λ+µ) 2 (1 − e )
µ µ −(λ+µ)t λ µ −(λ+µ)t . (6.20)
λ+µ t − (λ+µ)2 (1 − e ) λ+µ t + (λ+µ) 2 (1 − e )
Thus, if the CTMC starts in state 1, the expected time it spends in state 2 up to time
t is given by M12 (t).

6.5 Computation of P (t): Finite State-Space

In this section we present several methods of computing the transition matrix P (t) of
a CTMC with finite state-space S = {1, 2, · · · , N } and generator matrix Q. We shall
illustrate the methods with two examples: the two-state machine of Example 6.4
which we analyze algebraically, and the two-machine workshop of Example 6.6
which we analyze numerically.

6.5.1 Exponential of a Matrix

Define the exponential of an N × N square matrix A as follows:

∞
X An
eA = I + .
n=1
n!

Note that eA is an N ×N square matrix. One can show that the series on the right had
side converges absolutely, and hence eA is well defined. Note that if A is a diagonal
matrix
A = diag[a1 , a2 , · · · , aN ],
then
eA = diag[ea1 , ea2 , · · · , eaN ].
In particular e0 = I where 0 is a square matrix with all elements equal to zero. The
next theorem gives the main result. Following the discussion in Section 2.4, we say
that Q is diagonalizable if there exist a diagonal matrix D and an invertible matrix
X such that
Q = XDX −1 . (6.21)
We saw in that section how to obtain the D and X if Q is diagonalizable.

Theorem 6.6 Exponential of Qt. The transition probability matrix of a finite-state

CTMC with generator matrix Q is given by
P (t) = eQt , t ≥ 0. (6.22)
210 CONTINUOUS-TIME MARKOV CHAINS
Furthermore, if Q is diagonalizable and satisfies Equation 6.21
N
X
P (t) = XeDt X −1 = eλi t xi yi , (6.23)
i=1

where λi = Dii , xi is the ith column of X, and yi is the ith row of X −1 .

Proof: We have
∞
X (Qt)n
eQt = I + , t ≥ 0. (6.24)
n=1
n!
Since the infinite series converges uniformly, we can take derivatives term by term to
get
d Qt
e = QeQt = eQt Q.
dt
Thus eQt satisfies the forward and backward equations. Since there is a unique so-
lution to those equations in the finite state-space case, we get Equation 6.22. Now,
from Theorem 2.7 we get
Qn = XDn X −1 , n ≥ 0.
Substituting in Equation 6.24, and factoring out X on the left and X −1 on the right,
we get
∞
!
Qt
X (Dt)n
e =X I+ X −1 , t ≥ 0,
n=1
n!
which yields Equation 6.23. The last representation follows from a similar represen-
tation in Equation 2.32 on page 38.
The next theorem is analogous to Theorem 2.8 on page 39.

Theorem 6.7 Eigenvalues of Q. Let Q be an N × N generator matrix of a CTMC,

with N eigenvalues λi , 1 ≤ i ≤ N . Let
q = max{−qii : 1 ≤ i ≤ M }.
Then

1. At least one of the eigenvalues is zero.

2. |λi + q| ≤ q for all 1 ≤ i ≤ N .

Proof: Define
Q
P =I+ .
q
(We shall see this matrix again in the sub-section on uniformization.) It is easy to
verify that P is a stochastic matrix. Now it can be seen that the N eigenvalues of P
are given by 1 + λi /q , (1 ≤ i ≤ m). From Theorem 2.8 we see that P has at least
one eigenvalue equal to one, thus at least one λi /q + 1 equals 1. Thus at least one λi
COMPUTATION OF P (T ): FINITE STATE-SPACE 211
is zero. Also, all eigenvalues of P lie in complex plane within the unit circle centered
at zero. Thus we must have
λi
|1 + | ≤ 1.
q
Thus each λi must lie in the complex plane within a circle of radius q with center at
−q, which is the second part of the theorem.
Many matrix oriented software programs provide built-in ways to compute the
matrix exponential function. For example, Matlab provides a function expm: we can
compute P (t) by the Matlab statement: P (t) = expm(Q ∗ t). Although simple to use
for reasonable sized matrices with numerical entries, it does not provide any insight
about the behavior of P (t) as a function of t. We get this insight from the above
theorem, which gives P (t) as an explicit function of t. We illustrate with examples.

Example 6.18 Two-State Machine. Consider the two-state CTMC with generator
matrix given in Equation 6.10. We have
Q = XDX −1 ,
where
" #
λ λ λ
1
   
λ+µ 0 0 −1 λ+µ λ+µ
X= µ , D= , X = .
1 − λ+µ 0 −(λ + µ) 1 −1
Substituting in Equation 6.23 we get
 
1 0
P (t) = X X −1 .
0 e−(λ+µ)t
Straightforward calculations show that this reduces to the transition probability ma-
trix given in Equation 6.17.

Example 6.19 Two-Machine One-Repairperson Workshop. Consider the ma-

chine shop of Example 6.6. Suppose that the mean life time of a machine is 10 days,
while the mean repair time is one day. Let X(t) be the number of working machines
at time t. Using days as the unit of time, we see that λ = 1 and µ = .1, and thus
{X(t), t ≥ 0} is a CTMC on state-space {0, 1, 2}and generator matrix
 
−1 1 0
Q =  .1 −1.1 1 .
0 .2 −.2
Numerical calculations show that
Q = XDX −1 ,
where
   
0.9029 0.9844 0.5774 0.3178 −1.4943 1.1765
X =  −0.4245 0.1675 0.5774  , X −1 =  0.7076 1.2041 −1.9117  ,
0.0668 −0.0532 0.5774 0.0284 0.2839 1.4197
212 CONTINUOUS-TIME MARKOV CHAINS
and  
−1.4702 0 0
D= 0 −0.8298 0  .
0 0 0
Hence, from Theorem 6.6, we get
 
0.2870 −1.3493 1.0623
P (t) = e−1.4702t  −0.1349 0.6344 −0.4994 
0.0212 −0.0999 0.0786
 
0.6966 1.1853 −1.8820
+e−0.8298t  0.1185 0.2017 −0.3202 
−0.0376 −0.0640 0.1017
 
0.0164 0.1639 0.8197
+  0.0164 0.1639 0.8197  .
0.0164 0.1639 0.8197
Thus if all machines are initially down, the probability that one machine is working
at the end of the first day is given by
p01 (1) = −1.3493e−1.4702 + 1.1853e−0.8298 + 0.1639 = 0.3707.
From the explicit expression for P (t) we see that
 
0.0164 0.1639 0.8197
lim P (t) =  0.0164 0.1639 0.8197  .
t→∞
0.0164 0.1639 0.8197
Thus P (t) has a limit, and in the limit all its rows are identical. This is similar to the
behavior we had encountered in the study of DTMCs. We shall study this in more
detail in Section 6.10.

6.5.2 Laplace Transforms

Define the Laplace transform (LT) (see Appendix F) of pij (t) as follows:
Z ∞
p∗ij (s) = e−st pij (t)dt, Re(s) > 0,
0

where Re(s) is the real part of the complex number s. Defining the LT of a matrix as
the matrix of the LTs of its elements, we write
P ∗ (s) = [p∗ij (s)]i,j∈S .
The next theorem gives the main result.

Theorem 6.8 The LT of the transition probability matrix P (t) is given by

P ∗ (s) = [sI − Q]−1 , Re(s) > 0.
COMPUTATION OF P (T ): FINITE STATE-SPACE 213
Proof: Using the properties of the Laplace transforms, we get
Z ∞
e−st p′ij (t)dt = sp∗ij (s) − pij (0).
0

In matrix form, this yields that the LT of the derivative matrix P ′ (t) is given by
sP ∗ (s) − I. Now taking the LT on both sides of Equation 6.13 or 6.14, and using the
initial condition P (0) = I, we get
sP ∗ (s) − I = QP ∗ (s) = P ∗ (s)Q. (6.25)
This can be rearranged to get
(sI − Q)P ∗ (s) = P ∗ (s)(sI − Q) = I.
Since sI − Q is invertible for Re(s) > 0, the theorem follows.

Example 6.20 Two-State Machine. Consider the two-state CTMC with generator
matrix given in Equation 6.10. From Theorem 6.8 we have
 
∗ 1 s+µ λ
P (s) = .
s(s + λ + µ) µ s+λ
Using partial fraction expansion, we get, for example
s+µ µ 1 λ 1
p∗00 (s) = = · + · .
s(s + λ + µ) λ+µ s λ+µ s+λ+µ
Using the table of LTs (See Appendix F), we can invert the above transform to get
λ µ
p00 (t) = e−(λ+µ)t + .
λ+µ λ+µ
We can compute other transition probabilities in a similar fashion to obtain the tran-
sition probability matrix given in Equation 6.17.

Example 6.21 Two-Machine One-Repairperson Workshop. Consider the

CTMC of Example 6.19. Using Theorem 6.8 we get
 −1
s+1 −1 0
P ∗ (s) =  −.1 s + 1.1 −1 
0 −.2 s + .2
2
 
50s + 65s + 1 10(5s + 1) 50
1 
= 5s + 1 10(s + 1)(5s + 1) 50(s + 1) ,
D(s) 2
1 10(s + 1) 5 ∗ (10s + 21s + 10)
where
D(s) = s(50s2 + 115s + 61) = s(s + 0.8298)(s + 1.4702).
Thus, using partial fractions, we get
10(5s + 1) 0.1639 1.1853 1.3493
p∗01 (s) = 2
= + − .
s(50s + 115s + 61) s s + 0.8298 s + 1.4702
214 CONTINUOUS-TIME MARKOV CHAINS
This can be inverted easily to get
p01 (t) = 0.1639 + 1.1853e−0.8298t − 1.3493e−1.4702t,
which matches the result in Example 6.19.

6.5.3 Uniformization

We begin with the following theorem.

Theorem 6.9 Let {Xn , n ≥ 0} be a DTMC on state-space S = {1, 2, · · · , N } with

one-step transition probability matrix P = [pij ]. Let {N (t), t ≥ 0} be a PP(λ) that
is independent of {Xn , n ≥ 0}. Define X(t) = XN (t) . Then {X(t), t ≥ 0} is a
CTMC on state-space S with generator matrix
Q = λ(P − I). (6.26)

Proof: Let Ni = min{n ≥ 0 : Xn 6= i}. Then

P(Ni = k|X0 = i) = pk−1
ii (1 − pii ), k ≥ 1.

That is, given X(0) = X0 = i, Ni is a geometric random variable with parameter

1 − pii . Now let Ti = min{t ≥ 0 : X(t) 6= i}. Thus Ti is a sum of a geometric
number of iid exp(λ) random variables. Hence from Theorem 5.7 on page 154 it
follows that Ti ∼ exp(λ(1 − pii )). Also,
 pij
1−pii j 6= i,
P(XNi = j|X0 = i) =
0 j = i.
, Now let

qi = −qii = λ(1 − pii ), qij = λpij . (6.27)

Due to the Markov properties of {Xn , n ≥ 0}, and {N (t), t ≥ 0}, we see that
{X(t), t ≥ 0} stays in state i for an exp(qi ) amount of time and then jumps to state
j 6= i with probability qij /qi , independent of the length of the sojourn in state i.
Hence {X(t), t ≥ 0} is a CTMC with generator matrix Q = [qij ]. Equation 6.27
implies that Q is given by Equation 6.26.
We use the above theorem in reverse to compute P (t) as shown in the following
theorem.

Theorem 6.10 Uniformization. Let {X(t), t ≥ 0} be a CTMC on finite state-space

S = {1, 2, · · · , N } with rate matrix Q. Let
λ = max{−qii },
i∈S

and define
Q
P =I+ . (6.28)
λ
COMPUTATION OF P (T ): FINITE STATE-SPACE 215
Then, P (t), the transition probability matrix of {X(t), t ≥ 0} is given by

∞
X (λt)n n
P (t) = e−λt P , t ≥ 0. (6.29)
n=0
n!

Proof: First, Equation 6.28 implies that P is a stochastic matrix. Now let {Xn , n ≥
0} be a DTMC with one-step transition matrix P and let {N (t), t ≥ 0} be a PP(λ)
that is independent of the DTMC. From Theorem 6.10 it follows that {XN (t) , t ≥ 0}
is a CTMC with generator matrix Q. Hence, for i, j ∈ S and t ≥ 0, we get
pij (t) = P(X(t) = j|X(0) = i)
= P(XN (t) = j|X0 = i)
X∞
= P(XN (t) = j|N (t) = n, X0 = i)P(N (t) = n|X0 = i)
n=0
∞
X (λt)n
= P(Xn = j|N (t) = n, X0 = i)e−λt
n=0
n!
∞
X (λt)n n
= e−λt [P ]ij .
n=0
n!

Putting the above equation in matrix form yields Equation 6.29.

In practice one would computeP (t) by truncating the series in Equation 6.29 after
a finite number of terms. We discuss below how to decide when to truncate the series.
For a given τ > 0, and an ǫ > 0, it is possible to find an M < ∞ such that
∞
X (λτ )n
e−λτ < ǫ.
n!
n=M+1

Choose the smallest such M , and define

M
X (λt)n n
PM (t) = e−λt P .
n=0
n!
We see that, for all 0 ≤ t ≤ τ ,
kP (t) − PM (t)k = max |[P (t)]ij − [PM (t)]ij |
i,j∈S
∞
X (λt)n n
= max e−λt [P ]ij
i,j∈S n!
n=M+1
∞
X (λt)n
≤ e−λt < ǫ.
n!
n=M+1

Thus PM (t) is an ǫ lower bound on P (t) for all 0 ≤ t ≤ τ. In our experience uni-
formization has proved to be an extremely stable and efficient numerical procedure
216 CONTINUOUS-TIME MARKOV CHAINS
for computing P (t). A subtle numerical difficulty arises when λt is so large that
e−λt is numerically computed to be zero. In this case we need more ingenious ways
of computing the series. We shall not go into the details here.

Example 6.22 Two-State Machine. Consider the two-state CTMC with generator
matrix given in Equation 6.10. To avoid confusing notation, we use q instead of λ in
Equation 6.28 to get
q = max(−q00 , −q11 ) = max(λ, µ).
Note that any q larger than the right hand side would do. Thus instead of choosing
q = max(λ, µ), we use q = λ + µ. Then we get
 
Q 1 µ λ
P =I+ = .
q λ+µ µ λ
Clearly, we have P 0 = I and P n = P for n ≥ 1. Thus we have
∞
!
−(λ+µ)t
X ((λ + µ)t)n
P (t) = e I +P ,
n=1
n!

which reduces to Equation 6.17.

Example 6.23 Two-Machine One-Repairperson Workshop. Consider the

CTMC of Example 6.19. From Equation 6.28 we get
λ = max(1, 1.1, .2) = 1.1.
Hence we get  
.1 1 0
Q 1 
P =I+ = .1 0 1  .
1.1 1.1 0 .2 .9
Using Theorem 2.7 on page 38 we get
 
0.2870 −1.3493 1.0623
P n = (−.3365)n  −0.1349 0.6344 −0.4994 
0.0212 −0.0999 0.0786
 
0.6966 1.1853 −1.8820
+(.2456)n  0.1185 0.2017 −0.3202 
−0.0376 −0.0640 0.1017
 
0.0164 0.1639 0.8197
+(1)n  0.0164 0.1639 0.8197  .
0.0164 0.1639 0.8197
Substituting this in Equation 6.29 and simplifying, we get Equation 6.25.

It should be noted that the method of uniformization works even if the CTMC
has infinite state-space as along as λ computed by the Equation 6.28 is finite. Such
COMPUTATION OF P (T ): INFINITE STATE-SPACE 217
CTMCs are called uniformizable. It is easy to see that uniformizable CTMCs are
automatically regular, since they can have at most P (λt) transitions over (0, t].
Finally, one can always solve the backward or forward equations by standard nu-
merical methods of solving differential equations. We refer the readers to any book
on differential equations for more details.

6.6 Computation of P (t): Infinite State-Space

In this section we discuss the computation of the transition matrix P (t) for a CTMC
{X(t), t ≥ 0} on infinite state-space S = {0, 1, 2, · · ·}. This can be done only when
the CTMC has highly specialized structure. In our experience the transform methods
are most useful in such cases. We illustrate with several examples.

Example 6.24 Poisson Process. Let {X(t), t ≥ 0} be a PP(λ). We have X(0) = 0.

To simplify notation we write
pi (t) = p0i (t) = P(X(t) = i|X(0) = 0), i ≥ 0, t ≥ 0.
The forward equations are
p′0 (t) = −λp0 (t),
p′i (t) = −λpi (t) + λpi−1 (t), i ≥ 1,
with initial conditions
p0 (0) = 1, pi (0) = 0, i ≥ 1.
Taking Laplace transforms on both sides of the above differential equations, we get
sp∗0 (s) − 1 = −λp∗0 (s),
sp∗i (s) = −λp∗i (s) + λp∗i−1 (s), i ≥ 1.
These can be solved recursively to obtain
λi
p∗i (s) = , i ≥ 0.
(s + λ)i+1
Inverting this using the table in Appendix F, we get
(λt)i
pi (t) = e−λt , i ≥ 0,
i!
which is as expected.

Example 6.25 Pure Birth Process. Let {X(t), t ≥ 0} be a pure birth process as
described in Example 6.8 with birth parameters λi in state i ≥ 0, and assume that
X(0) = 0. As before we write
pi (t) = p0i (t) = P(X(t) = i|X(0) = 0), i ≥ 0, t ≥ 0.
218 CONTINUOUS-TIME MARKOV CHAINS
The forward equations are
p′0 (t) = −λ0 p0 (t),
p′i (t) = −λi pi (t) + λi−1 pi−1 (t), i ≥ 1
with initial conditions
p0 (0) = 1, pi (0) = 0, i ≥ 1.
Taking Laplace transforms of the above differential equations, we get
sp∗0 (s) − 1 = −λ0 p∗0 (s),
sp∗i (s) = −λi p∗i (s) + λi−1 p∗i−1 (s), i ≥ 1.
These can be solved recursively to obtain
i
1 Y λk
p∗i (s) = , i ≥ 0.
λi s + λk
k=0

Consider the case when all the λi ’s are distinct. Then we can use partial fractions
expansion easily to get
i
X
pi (t) = Aki λk e−λk t , i ≥ 0,
k=0

where
i
1 Y λr
Aki = .
λi λr − λk
r=0,r6=k

As a special case, suppose λi = iλ, (i ≥ 0), but with X(0) = 1. It can be shown that
(see Computational Exercise 6.4) in this case we get
P(X(t) = i|X(0) = 1) = e−λt (1 − e−λt )i−1 .
Thus given X(0) = 1, X(t) is geometrically distributed with parameter e−λt .

Example 6.26 Pure Death Process. Let {X(t), t ≥ 0} be a pure death process as
described in Example 6.9 with death parameters µi in state i ≥ 1, and assume that
X(0) = N . We want to compute
pi (t) = pN i (t) = P(X(t) = i|X(0) = N ), 0 ≤ i ≤ N, t ≥ 0.
Using the fact that pN +1 (t) = pN,N +1 (t) = 0, we can write the forward equations
as
p′N (t) = −µN pN (t),
p′i (t) = −µi pi (t) + µi+1 pi+1 (t), 0 ≤ i ≤ N − 1,
with initial conditions
pN (0) = 1, pi (0) = 0, 0 ≤ i ≤ N − 1.
COMPUTATION OF P (T ): INFINITE STATE-SPACE 219
Taking Laplace transforms on both sides of the above differential equations, we get
sp∗N (s) − 1 = −µN p∗N (s),
sp∗i (s) = −µi p∗i (s) + µi+1 p∗i+1 (s), 0 ≤ i < N.
These can be solved recursively to obtain
N
1 Y µk
p∗i (s) = , 0 ≤ i ≤ N.
µi s + µk
k=i

When all the λi ’s are distinct we can use partial fractions expansion easily to get
N
X
pi (t) = Bki µk e−µk t , 0≤i≤N
k=i

where
N
1 Y µr
Bki = .
µi µr − µk
r=i,r6=k
As a special case, suppose µi = iµ, (i ≥ 0), with X(0) = N . It can be shown that
(see Computational Exercise 6.5) given X(0) = N , X(t) is binomially distributed
with parameters N and e−µt .

Example 6.27 Linear Growth Model. Let X(t) be the birth and death process of
Example 6.13 with birth parameters λi = iλ and death parameters µi = iµ, i ≥ 0.
Suppose X(0) = 1. We shall compute
pi (t) = p1i (t) = P(X(t) = i|X(0) = 1), i ≥ 0, t ≥ 0.
The forward equations are
p′0 (t) = µp1 (t),
p′i (t) = (i − 1)λpi−1 (t) − i(λ + µ)pi (t) + (i + 1)µpi+1 (t), i ≥ 1,
with initial conditions
p1 (0) = 1, pi (0) = 0, i 6= 1.
Now define the generating function
∞
X
p(z, t) = z i pi (t).
i=0

Multiplying the differential equation for pi (t) by z i and summing over all i from 0
to ∞ we get
∞
X ∞
X ∞
X ∞
X
z i p′i (t) = (i − 1)λz i pi−1 (t) − i(λ + µ)z i pi (t) + (i + 1)µz i pi+1 (t),
i=0 i=1 i=1 i=0

which reduces to
∂ ∂
p(z, t) = (λz 2 − (λ + µ)z + µ) p(z, t).
∂t ∂z
220 CONTINUOUS-TIME MARKOV CHAINS
This equation can be written in the canonical form as
∂ ∂
p(z, t) − a(z) p(z, t) = 0,
∂t ∂z
where
a(z) = λz 2 − (λ + µ)z + µ = (z − 1)(λz − µ).
Thus we have a linear first order partial differential equation, which can be solved
by the method of characteristic functions (see Chaudhry and Templeton (1983)). We
first form the total differential equations
dt dz dp
= = .
1 −a(z) 0
The first equation can be integrated to yield
dz
Z Z
dt =
−a(z)
or  
1 λz − µ
t− ln = c,
λ−µ z−1
where c is an arbitrary constant. The last equation gives
p = constant.
Hence the general solution is
  
ˆ 1 λz − µ
p(z, t) = f t− ln
λ−µ z−1
  
λz − µ
= f (λ − µ)t − ln , (6.30)
z−1
where f is a function to be determined by the boundary condition
∞
X
p(z, 0) = z i pi (0) = z.
i=0

Hence   
z−1
f ln = z. (6.31)
λz − µ
Now write  
z−1
u = ln
λz − µ
and invert it to get
µeu − 1
z= .
λeu − 1
Substituting in Equation 6.31 we get
µeu − 1
f (u) = .
λeu − 1
FIRST-PASSAGE TIMES 221
Substituting in Equation 6.30 we get
  
µ exp (λ − µ)t − ln λz−µ
z−1 −1
p(z, t) =   
λ exp (λ − µ)t − ln λz−µ
z−1 −1
which can be simplified to obtain
µ(1 − e(λ−µ)t ) − (λ − µe(λ−µ)t )z
p(z, t) = .
(µ − λe(λ−µ)t ) − λ(1 − e(λ−µ)t )z
The above expression can be expanded in a power series in z, and then the coefficient
of z i will give us pi (t). Doing this, and using ρ = λ/µ, we get
1 − e(λ−µ)t
p0 (t) = ,
1 − ρe(λ−µ)t
(1 − e(λ−µ)t )i−1
pi (t) = ρi−1 (1 − ρ)e(λ−µ)t , i ≥ 1.
(1 − ρe(λ−µ)t )i+1
This completes the solution for the case X(0) = 1. The case X(0) = i ≥ 2 can
be analyzed by treating the linear growth process as the sum of i independent linear
growth processes, each starting in state 1. This example shows that even for highly
structured CTMCs computation of P (t) is a formidable task.

Although computing P (t) is hard, it is relatively easy to compute

∞
X
mi (t) = E(X(t)|X(0) = i) = kpik (t), i ≥ 0, t ≥ 0.
k=0

We have
∞
X
m′i (t) = kp′ik (t).
k=0
By using the forward differential equations for pik (t) and carrying out the above sum
we get
m′i (t) = (λ − µ)mi (t).
Using the initial condition, mi (0) = i, we can solve the above equation to get
mi (t) = ie(λ−µ)t .
Thus if λ, the birth rate per individual, is greater than µ, the death rate per individual,
then the mean population size explodes as t → ∞. If it is less, the mean population
size exponentially reduces to zero. If the two rates are equal, the mean population
size is constant. All these conclusions confirm our intuition.

6.7 First-Passage Times

We follow the developments of DTMCs and first study the first passage times in the
CTMCs before we study their limiting behavior. Specifically, let {X(t), t ≥ 0} be a
222 CONTINUOUS-TIME MARKOV CHAINS
CTMC on S = {0, 1, 2, · · ·} with generator matrix Q = [qij ], and define
T = min{t ≥ 0 : X(t) = 0}. (6.32)
We study the random variable T in this section. Since the first passage times in a
CTMC has a lot of similarity to the first passage times in the embedded DTMC,
many of the results follow from similar results in Chapter 3. Hence we do not spend
as much time on this topic here.

6.7.1 Cumulative Distribution of T

Define
rij (t) = P(T > t, X(t) = j|X(0) = i), i, j ≥ 1, t ≥ 0.
Then the complementary cdf of T , conditioned on X(0) = i, is given by
∞
X
ri (t) = P(T > t|X(0) = i) = rij (t).
j=1

Then next theorem gives the differential equations satisfied by

R(t) = [rij (t)]i,j≥1 .
We shall need the following notation
M = [qij ]i,j≥1 .
Thus the matrix M is obtained by deleting the row and the column corresponding to
the state 0.

Theorem 6.11 The matrix function R(t) satisfies the following set of differential
equations:
R′ (t) = M R(t) = R(t)M, (6.33)
with the initial condition
R(0) = I.

Proof: Follows along the same lines as the proof of the backward and forward equa-
tions in Theorem 6.4.
Thus we can use the methods described in the previous two sections to compute
R(t), and hence the complementary cdf of T .

The next theorem gives a more explicit expression for the cdf of T when the CTMC
has a finite state-space.

Theorem 6.12 Suppose the CTMC has state-space {0, 1, · · · , K} and initial distri-
bution
α = [α1 , α2 , · · · , αK ],
FIRST-PASSAGE TIMES 223
with K
P
i=1 αi = 1.Then
P(T ≤ t) = 1 − αeMt 1, (6.34)
where 1 is a column vector of ones.

Proof: Since M is finite and R(0) = I, we can use Theorem 6.6 to obtain
R(t) = eMt .
Now,
P(T ≤ t) = 1 − P(T > t)
K
X
= 1− αi P(T > t|X(0) = i)
i=1
K X
X K
= 1− αi P(T > t, X(t) = j|X(0) = i)
i=1 j=1
K X
X K
= 1− αi rij (t)
i=1 j=1
= 1 − αR(t)1
= 1 − αeMt 1
as desired.

6.7.2 Absorption Probabilities

Define
vi = P(T = ∞|X(0) = i), i ≥ 1. (6.35)
Thus vi is the probability that the CTMC never visits state zero starting from state i.
The next theorem gives the main result.

Theorem 6.13 Let v = [v1 , v2 , · · ·]′ . Then v is given by the largest solution bounded
above by 1 to
M v = 0. (6.36)

Proof: Let {Xn , n ≥ 0} be the embedded DTMC in the CTMC {X(t), t ≥ 0}. The
transition probability matrix P of the embedded DTMC is related to the generator
matrix Q of the CTMC by Equation 6.9. We have
vi = P({X(t), t ≥ 0} never visits state 0|X(0) = i)
= P({Xn , n ≥ 0} never visits state 0|X0 = i).
Stating Equation 3.8 on page 60 in scalar form we get
∞
X
vi = pij vj , i ≥ 1,
j=1
224 CONTINUOUS-TIME MARKOV CHAINS
where pij ’s are the transition probabilities in the embedded DTMC. Substituting
from Equation 6.9 we get
∞
X qij
vi = vj .
qi
j=1,j6=i
Multiplying on both sides by qi and using qi = −qii we get
∞
X
qij vj = 0,
j=1

which, in matrix form, yields Equation 6.36. The maximality of the solution follows
from Theorem 3.2.

6.7.3 Moments and LST of T

In this section we develop the methods of computing the moments of T . The next
theorem gives a method of computing
mi = E(T |X(0) = i), i ≥ 1. (6.37)
Obviously, mi = ∞ if vi > 0. Hence we consider the case vi = 0 in the next
theorem.

Theorem 6.14 Suppose v = 0. Then m = [m1 , m2 , · · ·]′ is given by the smallest

non-negative solution to
M m + 1 = 0. (6.38)

Proof: We use the first step analysis for CTMCs, which is analogous to the first step
analysis in the DTMCs as explained in Section 3.1. Let X(0) = i > 0 and Y1 be the
first sojourn time in state i. Then Y1 ∼ exp(qi ). Now condition on X1 = X(Y1 ) = j.
Clearly, if j = 0, T = Y1 . If j > 0, then T = Y1 + T ′ , where T ′ has the same
distribution as T conditioned on X0 = j. Hence we get
mi = E(T |X(0) = i)
X∞
= E(T |X(0) = i, X(Y1 ) = j)P(X(Y1 ) = j|X(0) = i)
j=0
= E(Y1 |X(0) = i, X(Y1 ) = 0)P(X(Y1 ) = 0|X(0) = i)
X∞
+ E(Y1 |X(0) = i, X(Y1 ) = j)P(X(Y1 ) = j|X(0) = i)
j=1
X∞
+ E(T ′ |X(0) = i, X(Y1 ) = j)P(X(Y1 ) = j|X(0) = i)
j=1
∞
X
= E(Y1 |X(0) = i) + E(T ′ |X(0) = j)P(X(Y1 ) = j|X(0) = i)
j=1
FIRST-PASSAGE TIMES 225
∞
1 X qij
= + mj .
qi qi
j=1,j6=i

Multiplying on both sides by qi and using qi = −qii we get

∞
X
qij mj + 1 = 0,
j=1

which, in matrix form, yields Equation 6.38. The minimality of the solution follows
from Theorem 3.3.
We next derive the Laplace Stieltjes transform (LST) of T . For s ≥ 0, define
φi (s) = E(e−sT |X(0) = i), i ≥ 1,
and
φ(s) = [φ1 (s), φ2 (s), · · ·]′ .
The main result is given in the next theorem.

Theorem 6.15 {φ(s), s ≥ 0} is given by the smallest non-negative solution to

sφ(s) = w + M φ(s), (6.39)
where w = [q10 , q20 , · · ·]′ .

Proof: Use the terminology and the first step analysis as described in the proof of
Theorem 6.14. Conditioning on (X1 , Y1 ) we get
φi (s) = E(e−sT |X(0) = i)
∞
′
X
= pij E(eY1 |X(0) = i)E(e−sT |X(0) = j)
j=0
 
∞
qi X
= pij φj (s) + pi0  .
s + qi j=1

Multiplying by (s + qi ) and using qi = −qii and qij = qi pij (i 6= j), we get

Equation 6.39. The minimality follows along the same lines as in the proof of Theo-
rem 3.2.
We can use the above theorem to immediately derive the moments of T denoted
by
mi (k) = E(T k |X(0) = i), i ≥ 1, k ≥ 1.

Theorem 6.16 Suppose v = 0. Then m(k) = [m1 (k), m2 (k), · · ·]′ is given by the
smallest non-negative solution to
M m(k) + km(k − 1) = 0, k ≥ 1, (6.40)
with m(0) = 1.
226 CONTINUOUS-TIME MARKOV CHAINS
Proof: Follows from taking successive derivatives of Equation 6.39 and using
dk
mi (k) = (−1)k φi (s)|s=0 .
dsk
We end this section with an example.

Example 6.28 Birth and Death Processes. Let {X(t), t ≥ 0} be the birth and
death process of Example 6.10 with birth parameters λi , i ≥ 0, and death parameters
µi , i ≥ 1. Let T be as defined in Equation 6.32.

Let us first compute the quantities {vi } as defined in Equation 6.35. The Equa-
tion 6.36 can be written in scalar form as
µi vi−1 − (λi + µi )vi + λi vi+1 = 0, i ≥ 1,
with boundary condition v0 = 0. The above equation can be written as
µi λi
vi = vi−1 + vi+1 , i ≥ 1.
λi + µi λi + µi
But these equations are identical to Equations 3.14 on page 63 with
λi µi
pi = , qi = . (6.41)
λi + µi λi + µi
Using the results of Example 3.9 on page 63 we get
 Pi−1
αj P∞
 Pj=0 if j=0 αj < ∞,

∞

αj
vi = j=0
(6.42)

 P∞
0 if j=0 αj = ∞,


where α0 = 1 and
µ1 µ2 · · · µi
αi = , i ≥ 1. (6.43)
λ1 λ2 · · · λi
P∞ we compute mi , as defined in Equation 6.37 under the assumption that
Next
j=0 αj = ∞, so that vi = 0 for all i ≥ 0. The Equation 6.38 can be written
in scalar form as
µi mi−1 − (λi + µi )mi + λi mi+1 + 1 = 0, i ≥ 1,
with boundary condition m0 = 0. The above equation can be written as
1 µi λi
mi = + mi−1 + mi+1 , i ≥ 1.
λi + µi λi + µi λi + µi
But these equations are similar to Equations 3.24 on page 71 with pi and qi as defined
1
in Equation 6.41. The only difference is that we have λi +µ i
on the right hand side
instead of 1. However, we can solve these equations using the same procedure as in
EXPLORING THE LIMITING BEHAVIOR BY EXAMPLES 227
Example 3.15 to get
i−1
! ∞

X X 1 
mi = αk  .
λj αj
k=0 j=k+1

6.8 Exploring the Limiting Behavior by Examples

Let {X(t), t ≥ 0} be a CTMC on S = {0, 1, 2, · · ·} with generator matrix Q. In

the Sections 6.3 and 6.4 we studied two main aspects of the transient behavior of
the CTMCs: the transition probability matrix P (t) and the occupancy matrix M (t).
Theorem 6.6 showed that
P (t) = eQt , t ≥ 0,
and Theorem 6.5 showed that
Z t
M (t) = P (u)du, t ≥ 0. (6.44)
0

In the next several sections we study the limiting behavior of P (t) as t → ∞. Since
the row sums of P (t) are 1, it follows that the row sums of M (t) are t. Hence we
study the limiting behavior of M (t)/t as t → ∞. Note that [M (t)]ij /t can be inter-
preted as the fraction of the time spent by the CTMC in state j during (0, t] starting
from state i. Hence studying this limit makes practical sense. We begin by some ex-
amples illustrating the types of limiting behavior that can arise.

Example 6.29 Two-State Example. Consider the CTMC of Example 6.16 with the
P (t) matrix as given in Equation 6.17. Hence we get
" #
µ λ
λ+µ λ+µ
lim P (t) = µ λ .
t→∞
λ+µ λ+µ

Thus the limit of P (t) exists and its row sums are one. As we had observed in the
DTMCs, the rows of the limiting matrix are the same, implying that the limiting dis-
tribution of X(t) does not depend upon the initial distribution of the CTMC.

Next, the occupancy matrix M (t) for this CTMC is given in Equation 6.20. Hence,
we get " #
µ λ
M (t) λ+µ λ+µ
lim = µ λ .
t→∞ t λ+µ λ+µ

Thus the limit of M (t)/t in this example is the same as that of P (t).
228 CONTINUOUS-TIME MARKOV CHAINS
Example 6.30 Three-state CTMC. Let {X(t), t ≥ 0} be a CTMC on state-space
{0, 1, 2} with generator matrix
 
0 0 0
Q =  µ −(λ + µ) λ .
0 0 0
Direct calculations show that
 
1 0 0
µ λ
P (t) =  λ+µ (1 − e−(λ+µ)t ) e−(λ+µ)t λ+µ (1 − e
−(λ+µ)t 
) ,
0 0 1
and
 
t 0 0
µ p10 (t) 1 p12 (t)
M (t) = λ+µ t− λ+µ λ+µ
(1 − e−(λ+µ)t ) λ
λ+µ
t− λ+µ
.
0 0 t
Hence we get
 
1 0 0
M (t)  µ λ
lim P (t) = lim = λ+µ 0 λ+µ
.
t→∞ t→∞ t
0 0 1
Thus the limiting matrix has distinct rows, implying the that the limiting behavior
depends on the initial state. Furthermore, the row sums are one.

Example 6.31 Linear Growth Model. Consider the CTMC of Example 6.27. Con-
sider the case λ > µ. In this case we can use the transition probabilities derived there
to obtain  µ
i
if j = 0,
lim pij (t) = λ
t→∞ 0 if j > 0.
Thus in this case the limits of pij (t) exist but depends on the initial state i. It is more
tedious to show that Mij (t)/t has the same limit as pij (t). Furthermore, the row
sums of the limiting matrix are less than 1.

Thus we have identified three cases:

Case 1: Limit of P (t) exists, has identical rows, and each row sums to one.

Case 2: Limit of P (t) exists, does not have identical rows, each row sums to one.

Case 3: Limit of P (t) exists, but the rows may not sum to one.

We have also observed that limit of M (t)/t always exists, and it equals the limit
of P (t). We have not seen any examples where the limit of P (t) displays oscillatory
behavior as in the case of periodic DTMCs. In next two sections we develop the
CLASSIFICATION OF STATES 229
necessary theory to help us classify the CTMCs so we can understand their limiting
behavior better.

6.9 Classification of States

Clearly, there is a close connection between the limiting behavior of a CTMC with
state-space S and generator matrix Q and that of the corresponding embedded
DTMC on the same state-space S and one-step transition matrix P that is related
to Q as in Equation 6.9. Thus it seems reasonable that we will need to follow the
same path as we did when we studied the limiting behavior of the DTMCs in Chap-
ter 4.

6.9.1 Irreducibility

This motivates us to begin by introducing the concept of irreducibility for a CTMC.

Note that a CTMC can visit state j from state i in a finite amount of time if and only
if the corresponding embedded DTMC can visit state j from state i in a finite number
of steps. Hence we can make the following definitions.

Definition 6.2 Communicating Classes. A set of states C ⊆ S in a CTMC is said

to be a (closed) communicating class if C is a (closed) communicating class of the
corresponding embedded DTMC.

Definition 6.3 Irreducibility. A CTMC is called irreducible if the corresponding

embedded DTMC is irreducible. Otherwise it is called reducible.

Example 6.32 The CTMC of Example 6.1 is irreducible if λ > 0 and µ > 0. The
CTMC of Example 6.30 is reducible, with {0} and {2} as two closed communicat-
ing classes, and {1} as a communicating class that is not closed. The linear growth
process of Example 6.13, with λ > 0 and µ > 0, has one closed communicating
class, namely {0}. The set {1, 2, · · ·} is a communicating class that is not closed.
The CTMC is reducible.

We see that the issue of periodicity does not arise in the CTMCs, since if a CTMC
can go from state i to j at all, it can do so at any time. Thus, for the two state CTMC
of Example 6.1, the embedded DTMC is periodic. However, the P (t) matrix for the
two-state CTMC does not show a periodic behavior.

6.9.2 Transience and Recurrence

Following the developments in the DTMCs, we now introduce the concepts of re-
currence and transience for the CTMCs. Let Y1 be the first sojourn time in a CTMC
 230 CONTINUOUS-TIME MARKOV CHAINS
{X(t), t ≥ 0} and define the first passage time (in a slightly modified form)
T̃i = inf{t ≥ Y1 : X(t) = i}, i ∈ S. (6.45)
This is a well defined random variable if Y1 < ∞ with probability 1, i.e., if qi > 0.
Note that if qi = 0, i is an absorbing state in the CTMC. Let
ũi = P(T̃i < ∞|X(0) = i), (6.46)
and
m̃i = E(T̃i |X(0) = i). (6.47)
When ũi < 1, m̃i = ∞. However, as in the DTMCs, m̃i can be infinite even if
ũi = 1. With this in mind, we make the following definition, which is analogous to
the corresponding definitions in the case of the DTMCs.

Definition 6.4 Transience and Recurrence. A state i with qi > 0 is said to be

(i) recurrent if ũi = 1,

(ii) transient if ũi < 1.

If qi = 0 we define i to be recurrent. This is consistent with the fact that an

absorbing state in a DTMC is recurrent. The next theorem shows an easy way of
establishing the transience and recurrence of a state in a CTMC.

Theorem 6.17 A state i is recurrent (transient) in a CTMC if and only if it is recur-

rent (transient) in the corresponding embedded DTMC.

Proof: Let {X(t), t ≥ 0} be a CTMC and {Xn , n ≥ 0} be the corresponding em-

bedded DTMC. The implicit assumption of the regularity of the CTMC implies that
{X(t) = i for some t ≥ 0} ⇔ {Xn = i for some n ≥ 0}.
Now,
ũi = 1 ⇔ P(X(t) = i for some t ≥ 0|X(0) = i) = 1.
Hence
ũi = 1 ⇔ P(Xn = i for some n ≥ 0|X0 = i) = 1.
Hence it follows that a state i is recurrent in a CTMC if and only if it is recurrent in
the corresponding embedded DTMC. The statement about the transient case follows
by contraposition.

It follows from the above theorem that recurrence and transience of states in the
CTMCs are class properties, just as they are in the DTMCs. This enables us to call
an irreducible CTMC transient or recurrent if all its states are transient or recurrent,
respectively. Since an irreducible CTMC is recurrent (transient) if and only if the
corresponding embedded DTMC is recurrent (transient), we can use the criteria de-
veloped in Chapter 4 to establish the transience or recurrence of the CTMCs.
 CLASSIFICATION OF STATES 231
Next we define positive and null recurrence.

Definition 6.5 Null and Positive Recurrence. A recurrent state i with qi > 0 is
said to be

(i) positive recurrent if m̃i < ∞,

(ii) null recurrent if m̃i = ∞.

If qi = 0 we define i to be positive recurrent. This is consistent with the fact that

an absorbing state in a DTMC is positive recurrent. Establishing null and positive
recurrence is more complicated than establishing recurrence and transience since a
state i may be positive recurrent in the CTMC but null recurrent in the DTMC, and
vice versa. Clearly, a CTMC with a single state is positive recurrent by definition.
For CTMCs with two or more states we have the following theorem.

Theorem 6.18 Let {X(t), t ≥ 0} be an irreducible CTMC on state-space S with

at least two states, and let Q be its generator matrix. Let {Xn , n ≥ 0} be the corre-
sponding embedded DTMC with transition probability matrix P . Suppose the DTMC
is recurrent and π is a positive solution to
π = πP.
The CTMC is positive recurrent if and only
X πi
< ∞. (6.48)
qi
i∈S

Proof: Since the CTMC is irreducible and has at least two states, we see that none
of the states can be absorbing. Thus qi > 0 for all i ∈ S. We use the first step analysis
as in the proof of Theorem 6.14 to derive a set of equations for
µij = E(T̃j |X(0) = i), i, j ∈ S.
Let X(0) = i and Y1 be the first sojourn time in state i. Then Y1 ∼ exp(qi ). Now
condition on X1 = X(Y1 ) = k. Clearly, if k = j, T̃j = Y1 . If k 6= j, then T̃j =
Y1 + T ′ , where T ′ has the same distribution as T̃j conditioned on X(0) = k. Hence
we get
µij = E(T̃j |X(0) = i)
X
= E(T̃j |X(0) = i, X(Y1 ) = k)P(X(Y1 ) = k|X(0) = i)
k∈S
= E(Y1 |X(0) = i, X(Y1 ) = j)P(X(Y1 ) = j|X(0) = i)
X
+ E(Y1 |X(0) = i, X(Y1 ) = k)P(X(Y1 ) = k|X(0) = i)
k∈S−{j}
X
+ E(T ′ |X(0) = i, X(Y1 ) = k)P(X(Y1 ) = k|X(0) = i)
k∈S−{j}
232 CONTINUOUS-TIME MARKOV CHAINS
X
= E(Y1 |X(0) = i) + pik E(T ′ |X(0) = k)
k∈S−{j}
1 X
= + pik µkj . (6.49)
qi
k∈S−{j}

Now multiply both sides by πi and sum over all i ∈ S. We get

X X πi X X
πi µij = + πi pik µkj
qi
i∈S i∈S i∈S k∈S−{j}
X πi X X
= + πi pik µkj
qi
i∈S k∈S−{j} i∈S
X πi X
= + πk µkj
qi
i∈S k∈S−{j}

since X
πk = πi pik .
i∈S
P
Hence, subtracting k∈S−{j} πk µkj from both sides, we get
X πi
πj µjj = ,
qi
i∈S

which yields !
1 X πi
µjj = . (6.50)
πj qi
i∈S
Since πj > 0, we see that m̃j = µjj < ∞ if and only if Equation 6.48 is satisfied.
The theorem then follows from the definition of positive recurrence.

Theorem 6.18 also implies that null and positive recurrence in the CTMCs are class
properties, just as in the DTMCs. We shall use this theorem to construct examples
of null recurrent CTMCs with positive recurrent embedded DTMCs, and vice versa.
However, note that if the CTMC is positive recurrent, but the embedded DTMC is
null recurrent, the CTMC cannot be regular. Since, this situation would imply that
the CTMC makes infinite number of transitions in a finite amount of time. Thus in
our applications a positive recurrent CTMC will have a positive recurrent embedded
DTMC.

Example 6.33 Success Runs. Consider a CTMC {X(t), t ≥ 0} on S =

{0, 1, 2, · · ·}, with the corresponding embedded DTMC {Xn , n ≥ 0} with transi-
tion probabilities
pi,i+1 = pi , pi,0 = 1 − pi , i ≥ 0.
Thus the embedded DTMC is a success runs Markov chain as described in Exam-
ple 4.13 on page 99. We consider two cases.
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 233
Case 1. Let pi = 1/(i+2), and qi = −qii = (i+1), for i ≥ 0. From Example 4.10
we see that this DTMC is null recurrent, and the solution to
π = πP
is given by
1
πi = , i ≥ 0.
i+1
Now,
X πi ∞
X 1
= < ∞.
qi i=0
(i + 1)2
i∈S
Hence the CTMC is positive recurrent. Clearly such a CTMC cannot be regular.
Case 2. Let pi = p ∈ (0, 1) and qi = −qii = pi for all i ≥ 0. Then the DTMC is
positive recurrent (see Example 4.13) with a positive solution to π = πP given by
πi = pi .
Then
∞
X πi X pi
= = ∞.
qi i=0
pi
i∈S

Hence the CTMC is null recurrent.

6.10 Limiting Behavior of Irreducible CTMCs

In this section we derive the main results regarding the limiting distribution of an
irreducible CTMC {X(t), t ≥ 0} on state-space S = {0, 1, 2 · · ·} and generator
matrix Q. We treat the three types of irreducible CTMCs: transient, null recurrent,
and positive recurrent. We treat the case of reducible CTMCs in the next section.

6.10.1 The Transient Case

We begin with the main result in the following theorem.

Theorem 6.19 Let {X(t), t ≥ 0} be an irreducible transient CTMC. Then

lim pij (t) = 0. (6.51)
t→∞

Proof: Let {Xn , n ≥ 0} be the embedded DTMC corresponding to the irreducible

transient CTMC {X(t), t ≥ 0}. Theorem 6.17 implies that {Xn , n ≥ 0} is irre-
ducible and transient. From Equation 4.21 we see that the expected number of visits
by the DTMC to state j starting from state i over the infinite time horizon is finite.
Every time the DTMC visits state j, the CTMC spends an average of 1/qj time there.
(Note that qj > 0 since state j is transient.) Hence the total expected time spent by
234 CONTINUOUS-TIME MARKOV CHAINS
the CTMC over [0, ∞) in state j, starting from state i is finite. Using the notation of
occupancy times,
lim Mij (t) = Mij (∞) < ∞.
t→∞
From Theorem 6.5 we get
Z ∞
Mij (∞) = pij (t)dt < ∞.
0

Since pij (t) ≥ 0, we get Equation 6.51.

As in the case of the transient DTMCs, a transient CTMC will eventually perma-
nently exit any finite set with probability 1.

6.10.2 The Continuous Renewal Theorem

Following the development in Section 4.5, we start with the statement of the con-
tinuous renewal theorem, which is the continuous analogue of its discrete version in
Theorem 4.14.

Theorem 6.20 Continuous Renewal Theorem. Let f be a probability density

function on [0, ∞) with mean µ. Let h : [0, ∞) → (−∞, ∞) be a monotone function
with Z ∞
|h(u)|du < ∞. (6.52)
0
Suppose the function g : [0, ∞) → (−∞, ∞) satisfies
Z t
g(t) = h(t) + g(t − u)f (u)du, t ≥ 0. (6.53)
0
Then
1 ∞
Z
lim g(t) = h(u)du. (6.54)
t→∞ µ 0
If µ = ∞, the limit on the right in the above equation is to be interpreted as 0.

Proof: As in the proof of Theorem 4.14, the hard part is to prove that g(t) has a limit
as t → ∞. We refer the reader to Karlin and Taylor (1975) or Kohlas (1982) for the
details. Here we assume that the limit exists and show that it is given as stated in
Equation 6.54.

Define the Laplace transform (LT) as follows (see Appendix F). Here s is a com-
plex number with Re(s) ≥ 0.
Z ∞
∗
f (s) = e−st f (t)dt,
0
Z ∞
∗
g (s) = e−st g(t)dt,
Z0 ∞
∗
h (s) = e−st h(t)dt.
0
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 235
Multiplying Equation 6.53 by e−st and integrating from 0 to ∞, and using the prop-
erties of LTs, we get
g ∗ (s) = h∗ (s) + g ∗ (s)f ∗ (s),
which yields
h∗ (s)
g ∗ (s) = . (6.55)
1 − f ∗ (s)
If limt→∞ g(t) exists, we know that it is given by
lim g(t) = lim sg ∗ (s).
t→∞ s→0

Using Equation 6.55 we get

h∗ (s)
lim sg ∗ (s) = lim s
s→0 s→0 1 − f ∗ (s)
1 − f ∗ (s)
= lim h∗ (s)/ lim
s→0 s→0 s
1 ∞
Z
= h(u)du.
µ 0
Here the last equality follows because
1 − f ∗ (s) f ∗ (0) − f ∗ (s) d
lim = lim = lim f ∗ (s) = µ.
s→0 s s→0 s s→0 ds

This proves the theorem.

Equation 6.53 is sometimes called the continuous renewal equation, and is anal-
ogous to its discrete counterpart in Equation 4.24 on page 107. The next theorem
shows that {pjj (t), t ≥ 0} satisfy a continuous renewal equation if qj > 0.

Theorem 6.21 Renewal Equation for pjj (t). Suppose qj > 0 and let fj be the
density of T̃j , as defined in Equation 6.45. Then pjj (t) satisfies the continuous re-
newal equation
Z t
−qj t
pjj (t) = e + fj (u)pjj (t − u)du, t ≥ 0. (6.56)
0

Proof: Suppose X(0) = j and T̃j = u. If u ≤ t, then the CTMC starts all over again
in state j at time u, due to Markov property. If u > t, then X(t) = j if and only if
Y1 > t. Hence we have

P(Y1 > t|T̃j = u, X(0) = j) if u > t,
P(X(t) = j|X(0) = j, T̃j = u) =
P(X(t − u) = j|X(0) = j) if u ≤ t.
Thus we get
pjj (t) = P(X(t) = j|X(0) = j)
Z ∞
= P(X(t) = j|X(0) = j, T̃j = u)fj (u)du
0
236 CONTINUOUS-TIME MARKOV CHAINS
Z t
= P(X(t − u) = j|X(0) = j)fj (u)du
0
Z ∞
+ P(Y1 > t|T̃j = u, X(0) = j)fj (u)du
t
Z ∞ Z t
= P(Y1 > t|T̃j = u, X(0) = j)fj (u)du + pjj (t − u)fj (u)du
0 0
(since P(Y1 > t|T̃j = u, X(0) = j) = 0 if u < t)
Z t
= P(Y1 > t|X(0) = j) + pjj (t − u)fj (u)du
0
Z t
= e−qj t + fj (u)pjj (t − u)du.
0

This proves the theorem.

Using the above theorem we get the next important result.

Theorem 6.22 If qj = 0
lim pjj (t) = 1.
t→∞
If state j is recurrent with qj > 0
1
lim pjj (t) = , (6.57)
t→∞ qj m̃j
where m̃j is as defined in Equation 6.47.

Proof: If qj = 0, we have pjj (t) = 1 for all t ≥ 0. Hence the first equation in
the theorem follows. If qj > 0, we see from Theorem 6.22 that pjj (t) satisfies the
continuous renewal equation. Since j is recurrent, we have
Z ∞
fj (t)dt = 1,
0
and ∞ ∞
1
Z Z
|h(t)|dt = e−qj t dt = .
0 0 qj
We also have Z ∞
µ= tfj (t)dt = E(T̃j |X0 = j) = m̃j .
0
Hence we can apply Theorem 6.20 to get
Z ∞
1 1
lim pjj (t) = e−qj t dt = .
t→∞ m̃j 0 qj m̃j
This proves the theorem.
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 237
6.10.3 The Null Recurrent Case

Now we study the limiting behavior of an irreducible null recurrent CTMC. The main
result is given by

Theorem 6.23 The Null Recurrent CTMC. For an irreducible null recurrent
CTMC
lim pij (t) = 0.
t→∞

Proof: Since the CTMC is null recurrent, we know that

qj > 0, and m̃j = ∞, j ∈ S.
Hence from Theorem 6.21 we see that
lim pjj (t) = 0.
t→∞

Now let i 6= j, and let fij (·) be the conditional probability density of T̃j given
X(0) = i. Using the argument in the proof of Theorem 6.21 we see that
Z t
pij (t) = fij (u)pjj (t − u)du.
0
Since the CTMC is irreducible and recurrent it follows that
P(T̃j < ∞|X(0) = i) = 1.
Hence
lim pij (t) = 0, i, j ∈ S.
t→∞

This proves the theorem.

6.10.4 The Positive Recurrent Case

Now we study the limiting behavior of an irreducible positive recurrent CTMC. Such
CTMCs are also called ergodic. If the state-space is a singleton, say S = {1}, then
q1 = 0, and p11 (t) = 1 for all t ≥ 0. Hence the limting behavior is trivial in this
case. So suppose that S has at least two elements. Then qj must be strictly positive
and m̃j < ∞ for all j ∈ S. Hence, from Theorem 6.21 we get
1
lim pjj (t) = > 0, j ∈ S.
t→∞ qj m̃j
The next theorem yields the limiting behavior of pij (t) as t → ∞.

Theorem 6.24 The Positive Recurrent CTMC. For an irreducible positive recur-
rent CTMC
lim pij (t) = pj > 0, i, j ∈ S (6.58)
n→∞
238 CONTINUOUS-TIME MARKOV CHAINS
where p = [pj , j ∈ S] is given by the unique solution to
pQ = 0, j ∈ S, (6.59)
X
pj = 1. (6.60)
j∈S

Proof: The theorem is true if the CTMC has a single state. Hence assume that the
CTMC has at least two states. Then qj > 0 for all j ∈ S. Equation 6.57 implies
that Equation 6.58 holds when i = j with pj = 1/qj m̃j > 0. Hence assume i 6= j.
Following the proof of Theorem 6.21 we get
Z t
pij (t) = fij (u)pjj (t − u)du, t ≥ 0,
0

where fij (·) is the density of T̃j conditioned on X(0) = i. Since i ↔ j and the
CTMC is positive recurrent, it follows that
Z ∞
fij (u)du = P(X(t) = j for some t ≥ 0 |X(0) = i) = 1.
0
Now let 0 < ǫ < 1 be given. Thus it is possible to pick an N such that
Z ∞
fij (u)du ≤ ǫ/2,
N
and
|pjj (t) − pj | ≤ ǫ/2, for all t ≥ N.
Then, for t ≥ 2N , we get
Z t 
 
|pij (t) − pj | =   fij (u)pjj (t − u)du − pj 
0
Z t−N

=  fij (u)(pjj (t − u)du − pj )du
0
Z t
+ fij (u)(pjj (t − u) − pj )du
t−N
Z ∞ 

− fij (u)pj du
t
Z t−N
≤ fij (u)|pjj (t − u) − pj |du
0
Z t
+ fij (u)|pjj (t − u) − pj |du
Zt−N
∞
+ fij (u)dupj
t
Z t−N Z t Z ∞
≤ fij (u)duǫ/2 + fij (u)du + fij (u)du
0 t−N t−N
≤ ǫ/2 + ǫ/2 ≤ ǫ.
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 239
This proves Equation 6.58. Next we derive Equations 6.59 and 6.60. For any finite
set A ⊂ S, we have X X
pij (t) ≤ pij (t) = 1.
j∈A j∈S
Letting t → ∞ on the left hand side, we get
X
pj ≤ 1.
j∈A

Since the above equation holds for any finite A, we must have
X
pj ≤ 1. (6.61)
j∈S

Now let aj (t) = P(X(t) = j). Then Equation 6.58 implies

lim aj (t) = pj , j ∈ S.
t→∞

Now, the Chapman-Kolmogorov Equations 6.12 yield

X
aj (s + t) = ai (s)pij (t), s, t ≥ 0.
i∈S

Let s → ∞ on both sides. The interchange of the limit and the sum on the right hand
side is justified due to bounded convergence theorem. Hence we get
X
pj = pi pij (t). (6.62)
i∈S

Replacing t by t + h yields
X
pj = pi pij (t + h). (6.63)
i∈S

Subtracting Equation 6.62 from 6.63 we get

X
pi (pij (t + h) − pij (t)) = 0.
i∈S

Dividing the above equation by h and letting h → 0, the above equation in matrix
form yields
pP ′ (t) = 0.
Using Equation 6.13, the above equation reduces to
pQP (t) = 0.
Substituting t = 0 in the above equation and using P (0) = I, we get Equation 6.59.
Again, letting t → ∞ in Equation 6.62 and using bounded convergence theorem to
interchange the sum and the limit on the right hand side we get
!
X
pj = pi pj .
i∈S
240 CONTINUOUS-TIME MARKOV CHAINS
P
But pj > 0. Hence we must have pi = 1, yielding Equation 6.60.

Now suppose {p′i , i ∈ S} is another solution to Equations 6.59 and 6.60. Using
the same steps as before we get
X
p′j = p′i pij (t), t ≥ 0.
i∈S

Letting t → ∞ we get !
X
p′j = p′i pj = pj .
i∈S

Thus the Equations 6.59 and 6.60 have a unique solution.

The next theorem removes the need to first check the positive recurrence of the
CTMC before solving the balance and the normalizing equation. It is analogous to
Theorem 4.21

Theorem 6.25 Let {X(t), t ≥ 0} be an irreducible CTMC. It is positive recurrent

if and only if there is a positive solution to
pQ = 0, (6.64)
X
pj = 1. (6.65)
j∈S

If there is a solution to the above equations, it is unique.

Proof: Let p be a positive solution to the Equations 6.64 and 6.65. Then it is straight-
forward to verify that πj = pj qj solves the balance equations for the embedded
DTMC. Substituting in Equation 6.50 we get
!
1 X πi
µjj =
πj qi
i∈S
!
1 X 1
= pi = < ∞.
pj qj pj qj
i∈S

Hence state j is positive recurrent, and hence the entire CTMC is positive recurrent.
The uniqueness was already proved in Theorem 6.24.
The vector p = [pj ] is called the limiting distribution or the steady state distribu-
tion of the CTMC. It is also the stationary distribution or the invariant distribution of
the CTMC, since if p is the pmf of X(0) then it is also the pmf of X(t) for all t ≥ 0.
See Conceptual Exercise 6.5.

The next theorem shows the relationship between the stationary distribution of the
CTMC and that of the corresponding embedded DTMC.
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 241
Theorem 6.26 Let {X(t), t ≥ 0} be an irreducible positive recurrent CTMC
on state-space S with generator Q and stationary distribution {pj , j ∈ S}. Let
{Xn , n ≥ 0} be the corresponding embedded DTMC with transition probability
matrix P and stationary distribution {πj , j ∈ S}. Then
πj /qj
pj = P , j ∈ S. (6.66)
i∈S πi /qi

Proof: Follows by substituting pij = qij /qi in π = πP and verifying that it reduces
to pQ = 0. We leave the details to the reader.

The last theorem of this section shows that the limiting occupancy distribution of
a CTMC is always the same as its limiting distribution.

Theorem 6.27 Limiting Occupancy Distribution. Let {X(t), t ≥ 0} be an irre-

ducible CTMC. Then
Mij (t)
lim = lim pij (t).
t→∞ t t→∞

Proof: Follows from Equation 6.19 and the fact that pij (t) ≥ 0 and has a limit as
t → ∞.
Thus, if Equations 6.64 and 6.65 have a solution p, it represents the limiting dis-
tribution, the stationary distribution as well as the limiting occupancy distribution
of the CTMC. Equations 6.64 are called the balance equations. Sometimes they are
called global balance equations to distinguish them from the local balance equations
to be developed in Section 6.14. It is more instructive to write them in a scalar form:
X X
pi qij = pj qji .
j:j∈S,j6=i j:j∈S,i6=j

The left hand side equals pi qi , the rate at which transitions take the system out of
state i, and the right hand side equals the rate at which transitions take the system
into state i. In steady state the two rates must be equal. In practice it is easier to write
these “rate out = rate in” equations by looking at the rate diagram, rather than by
using the matrix equation 6.64. We illustrate the theory developed in this section by
several examples below.

Example 6.34 Two-State Machine. Consider the two state CTMC of Exam-
ple 6.29. The balance equations are
λp0 = µp1 ,
µp1 = λp0 .
Thus there is only one independent balance equation. The normalizing equation is
p0 + p1 = 1.
242 CONTINUOUS-TIME MARKOV CHAINS
Solving these we get
µ λ
p0 = , p1 = .
λ+µ λ+µ
This agrees with the result in Example 6.29.

Example 6.35 Birth and Death Processes. Let {X(t), t ≥ 0} be the birth and
death process of Example 6.10 with birth parameters λi > 0 for i ≥ 0, and µi > 0
for i ≥ 1. This CTMC is irreducible. The balance equations are
λ0 p0 = µ1 p1 ,
(λi + µi )pi = λi−1 pi−1 + µi+1 pi+1 , i ≥ 1.
Summing the first i equations we get
λi pi = µi+1 pi+1 , i ≥ 0. (6.67)
These can be solved recursively to get
pi = ρi p0 , i ≥ 0,
where ρ0 = 1, and
λ0 λ1 · · · λi−1
ρi = , i ≥ 1. (6.68)
µ1 µ2 · · · µi
Now, substituting in the normalizing equation we get
∞ ∞
!
X X
pi = ρi p0 = 1.
i=0 i=0

The above equation has a solution

∞
!−1 ∞
X X
p0 = ρi if ρi < ∞.
i=0 i=0

If the infinite sum diverges, there is no solution. Thus the CTMC is positive recurrent
if and only if
X∞
ρi < ∞
i=0
and, when it is positive recurrent, has the limiting distribution given by
ρj
pj = P∞ , j ≥ 0. (6.69)
i=0 ρi
If the CTMC is transient or null recurrent, pj = 0 for all j ≥ 0. We analyze two
special birth and death processes in the next two examples.

Example 6.36 Single-Server Queue. Let X(t) be the number of customers at time
t in the single-server service system of Example 6.11. There we saw that {X(t), t ≥
0} is a birth and death process with birth parameters λi = λ for i ≥ 0, and death
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 243
parameters µi = µ for i ≥ 1. Thus we can use the results of Example 6.35 to study
the limiting behavior of this system. Substituting in Equation 6.68 we get
λ0 λ1 · · · λi−1
ρi = = ρi , i ≥ 0
µ1 µ2 · · · µi
where
λ
ρ= .
µ
Now,
∞  1
X if ρ < 1,
ρi = 1−ρ
∞ if ρ ≥ 1.
i=0
Thus the queue is stable (i.e., the CTMC is positive recurrent) if ρ < 1. In that case
the limiting distribution can be obtained from Equation 6.69 as
pj = ρj (1 − ρ), j ≥ 0. (6.70)
Thus, in steady state, the number of customers in a stable single server queue is a
modified Geometric random variable with parameter 1 − ρ.

Example 6.37 Infinite-Server Queue. Let X(t) be the number of customers at

time t in the infinite-server service system of Example 6.12. There we saw that
{X(t), t ≥ 0} is a birth and death process with birth parameters λi = λ for i ≥ 0,
and death parameters µi = iµ for i ≥ 1. Thus we can use the results of Example 6.35
to study the limiting behavior of this system. Substituting in Equation 6.68 we get
λ0 λ1 · · · λi−1 ρi
ρi = = , i≥0
µ1 µ2 · · · µi i!
where
λ
ρ= .
µ
Now,
∞
X ρi
= eρ .
i=0
i!
Thus the queue is stable (i.e., the CTMC is positive recurrent) if ρ < ∞. In that case
the limiting distribution can be obtained from Equation 6.69 as
ρj
pj = e−ρ , j ≥ 0.
j!
Thus, in steady state, the number of customers in a stable infinite-server queue is a
Poisson random variable with parameter ρ.

Example 6.38 Retrial Queue. Consider the bivariate CTMC {(X(t), R(t)), t ≥
0} of the retrial queue of Example 6.15. From of the rate diagram in Figure 6.9 we
get the following balance equations:
(λ + nθ)p0n = µp1n , n ≥ 0,
244 CONTINUOUS-TIME MARKOV CHAINS
(λ + µ)p10 = λp00 + θp01 ,
(λ + µ)p1n = λp0n + (n + 1)θp0,n+1 + λpi,n−1 , n ≥ 1.

These can be rearranged to yield

(λ + nθ)p0n = µp1n , n ≥ 0,
λp1n = (n + 1)θp0,n+1 , n ≥ 0.
Thus
(n + 1)θ λ + nθ
p0,n+1 = p0n , n ≥ 0,
λ µ
or
λ λ + nθ
p0,n+1 = p0n , n ≥ 0.
µ (n + 1)θ
This can be solved recursively to obtain
 n n−1Y λ 
1 λ
p0n = + k p00 , n ≥ 1. (6.71)
n! µ θ
k=0

Then substituting in the equation for p1n we get

 n+1 Y n  
1 θ λ λ
p1n = + k p00 , n ≥ 0. (6.72)
n! λ µ θ
k=0

Now using
∞
X ∞
X
p0n + p1n = 1
n=0 n=0
we get
∞  n n−1 ∞ n 
"  n+1 Y #
X 1 λ Y λ  X
1 θ λ λ
1+ +k + +k p00 = 1.
n=1
n! µ θ n=0
n! λ µ θ
k=0 k=0

It can be shown that the infinite sums converge if

λ
< 1, (6.73)
µ
and, in that case
  λ +1
λ θ
p00 = 1 − .
µ
Using this in Equations 6.71 and 6.72 we get the complete limiting distribution of the
CTMC. Thus the condition in Equation 6.73 is the condition of positive recurrence.

Example 6.39 Batch Arrival Queue. Consider a single server queue where cus-
tomers arrive in batches. The successive batch sizes are iid with common pmf
P(Btach Size = k) = αk , k ≥ 1.
LIMITING BEHAVIOR OF IRREDUCIBLE CTMCS 245
The batches themselves arrive according to a PP(λ). Thus the customer arrival pro-
cess is a CPP. Customers are served one at a time, service times being iid exp(µ)
random variables. Assume that there is an infinite waiting room. Let X(t) be the
number of customers in the system at time t.

Using the methodology developed in Section 6.2 we can show that {X(t), t ≥ 0}
is a CTMC with state space {0, 1, 2, · · ·} and the following transition rates:
qi,i−1 = µ i ≥ 1,
qi,i+k = λαk , k ≥ 1, i ≥ 0.
Thus we have q00 = −λ and qii = −(λ + µ) for i ≥ 1. Hence the balance equations
for this system are
λp0 = µp1 ,
i−1
X
(λ + µ)pi = λ αi−r pr + µpi+1 , i ≥ 1.
r=0

Now define the generating function of {pi , i ≥ 0} as

∞
X
φ(z) = pi z i .
i=0

Multiplying the balance equation for state i by z i and summing over all i, we get
∞
X ∞
X i−1
X ∞
X
λp0 + (λ + µ) pi z i = µp1 + λ zi αi−r pr + µ z i pi+1 .
i=1 i=1 r=0 i=1

Adding µp0 on both sides, interchanging the i and r sums on the right hand side, and
regrouping terms, we get
∞ ∞ ∞ ∞
!
X µ X X X
(λ + µ) pi z i = µp0 + pi z i + λ z i αi−r pr
i=0
z i=1 r=0 i=r+1
∞ ∞
  ! ∞ !
1 µ X
i
X
r
X
i
= µ 1− p0 + pi z + λ z pr z αi .
z z i=0 r=0 i=1

This can be rewritten as

 
1 µ
(λ + µ)φ(z) = µ 1 − p0 + φ(z) + λφ(z)ψ(z),
z z
where
∞
X
ψ(z) = z r αr .
r=1
This yields
µ(1 − 1/z)
φ(z) = p0 . (6.74)
λ(1 − ψ(z)) + µ(1 − 1/z)
246 CONTINUOUS-TIME MARKOV CHAINS
Note that ψ(z) is known. Thus the only unknown is p0 , which can be computed by
using
lim φ(z) = 1.
z→1
Using L’Hopital’s rule to compute the above limit, we get
µ
lim φ(z) = p0
z→1 µ−c
where
∞
d X
c = lim ψ(z) = iαi
z→1 dz
i=1
is the expected batch size. Writing
λc
ρ= ,
µ
we get
p0 = 1 − ρ.
Thus the queue is stable (i.e., the CTMC is positive recurrent) if
ρ < 1.
If it is stable, the generating function in Equation 6.74 reduces to
(1 − ρ)(1 − z)
φ(z) = .
(1 − z) − ρz(1 − ψ(z))/c
The expected number of customers in the system in steady state is given by
d ρ s2 + c
L = lim φ(z) = · ,
z→1 dz 1−ρ 2c
where s2 is the second moment of the batch size. One needs to apply L’Hopitals’s
rule twice to get the above expression.

6.11 Limiting Behavior of Reducible CTMCs

In this section we derive the main results regarding the limiting distribution of a re-
ducible CTMC {X(t), t ≥ 0} on state-space S = {0, 1, 2 · · ·} and generator matrix
Q. Assume that there are k closed communicating classes Ci , 1 ≤ i ≤ k, and T is
the set of states that do not belong to any closed communicating class. Now, relabel
the states in S by non-negative integers such that i ∈ Cr and j ∈ Cs with r < s
implies that i < j, and i ∈ Cr and j ∈ T implies that i < j. With this relabeling, the
generator matrix Q has the following canonical block structure:
LIMITING BEHAVIOR OF REDUCIBLE CTMCS 247
 
Q(1) 0 ··· 0 0
 0 Q(2) · · · 0 0 
.. .. .. ..
 
Q=
 .. .

(6.75)
 . . . . . 
 0 0 ··· Q(k) 0 
R Q(T )
Here Q(i) is a generator matrix (1 ≤ i ≤ k) of an irreducible CTMC with state-
space Ci , Q(T ) is a |T | × |T | sub-stochastic generator matrix (i.e., all row sums of
Q(T ) being less than or equal to zero, with at least one being strictly less than zero),
and R is a |T | × |S − T | matrix. Then the transition probability matrix P has the
same block structure:
 
P (1)(t) 0 ··· 0 0
 0 P (2)(t) · · · 0 0 
 .. . . .
 
P (t) =  . .. . . . .. .. .

 
 0 0 · · · P (k)(t) 0 
PR (t) P (T )(t)
Since P (r)(t) (1 ≤ r ≤ k) is a transition probability matrix of an irreducible CTMC
with state-space Cr , we already know how P (r)(t) behaves as t → ∞. Similarly,
since all states in T are transient, we know that P (T )(t) → 0 as t → ∞. Thus the
study of the limiting behavior of P (t) reduces to the study of the limiting behavior
of PR (t) as t → ∞. This is what we proceed to do.

Let T (r) be the first passage time to visit the set Cr , i.e.,
T (r) = min{t ≥ 0 : X(t) ∈ Cr }, 1 ≤ r ≤ k.
Let
ui (r) = P(T (r) < ∞|X(0) = i), 1 ≤ r ≤ k, i ∈ T. (6.76)
The next theorem gives a method of computing the above probabilities.

Theorem 6.28 Absorption Probabilities. The quantities {ui (r), i ∈ T, 1 ≤ r ≤

k} are given by the smallest non-negative solution to
X qij X qij
ui (r) = + uj (r). (6.77)
qi qi
j∈Cr j∈T

Proof: Equation 6.77 can be derived as in the proof of Theorem 6.13. The rest of the
proof is similar to the proof of Theorem 3.2 on page 60.
Using the quantities {ui (r), i ∈ T, 1 ≤ r ≤ k} we can describe the limit of PR (t)
as t → ∞. This is done in the theorem below.

Theorem 6.29 Limit of PR (t). Let {ui (r), i ∈ T, 1 ≤ r ≤ k} be as defined in

Equation 6.76. Let i ∈ T and j ∈ Cr .
 248 CONTINUOUS-TIME MARKOV CHAINS
(i) If Cr is transient or null recurrent,
lim pij (t) = 0. (6.78)
t→∞

(ii) If Cr is positive recurrent,

lim pij (t) = ui (r)pj ,
t→∞

where {pj , j ∈ Cr } is the unique solution to

X
pm qmj = 0,
m∈Cr
X
pm = 1.
m∈Cr

Proof: Follows along the same lines as the proof of Theorem 4.23.
We discuss two examples of reducible CTMCs below.

Example 6.40 Let {X(t), t ≥ 0} be the CTMC of Example 6.30. This is a re-
ducible CTMC with two closed communicating classes C1 = {0} and C2 = {2}.
The set T = {1} is not closed. We do not do any relabeling. Equations 6.77 yield:
µ λ
u1 (1) = , u1 (2) = .
λ+µ λ+µ
We also have
p0 = 1, p2 = 1
since these are absorbing states. Then Theorem 6.29 yields
 
1 0 0
µ λ
lim P (t) =  λ+µ 0 λ+µ  .
t→∞
0 0 1
This matches the result in Example 6.30.

Example 6.41 Linear Growth Model. Consider the CTMC of Example 6.27. This
is a reducible CTMC with one closed class C1 = {0}. Hence p0 = 1. The rest of the
states form the set T . Thus we need to compute ui (1) for i ≥ 1. From the results of
Example 6.28 we get
1 if λ ≤ µ,

ui (1) = µ i


λ if λ > µ.
Hence, for i ≥ 1,
1 if λ ≤ µ,

lim pi0 = µ i

(6.79)
t→∞
λ if λ > µ.
which agrees with Example 6.31.
Recall that the linear growth model represents a colony of organisms where each
organism produces new ones at rate λ, and each organism dies at rate µ. State 0 is
CTMCS WITH COSTS AND REWARDS 249
absorbing: once all organisms die, the colony is permanently extinct. Equation 6.79
says that extinction is certain if the birth rate is no greater than the death rate. On
the other hand, even if the birth rate is greater than the death rate, there is a positive
probability of extinction in the long run, no matter how large the colony is to begin
with. Of course, in this case, there is also a positive probability, 1 − (µ/λ)i , that the
size of the colony of initial size i will become infinitely large in the long run.

6.12 CTMCs with Costs and Rewards

As in the case of DTMCs, now we study CTMCs with costs and rewards. Let X(t) be
the state of a system at time t. Suppose {X(t), t ≥ 0} is a CTMC with state-space S
and generator matrix Q. Furthermore, the system incurs cost at a rate of c(i) per unit
time it spends in state i. For other cost models, see Conceptual Exercises 6.6 and 6.7.
Rewards can be thought of as negative costs. We consider costs incurred over infinite
horizon. For the analysis of costs over finite horizon, see Conceptual Exercise 6.8.

6.12.1 Discounted Costs

Suppose the costs are discounted continuously at rate α, where α > 0 is a fixed
(continuous) discount factor. Thus if the system incurs a cost of d at time t, its present
value at time 0 is e−αt d, i.e., it is equivalent to incurring a cost of e−αt d at time zero.
Let C be the total discounted cost over the infinite horizon, i.e.,
Z ∞
C= e−αt c(X(t))dt.
0

Let φ(i) be the expected total discounted cost (ETDC) incurred over the infinite
horizon starting with X(0) = i. That is,
φ(i) = E(C|X(0) = i).
The next theorem gives the main result regarding the ETDC. We introduce the fol-
lowing column vectors
c = [c(i)]i∈S , φ = [φ(i)]i∈S .

Theorem 6.30 ETDC. Suppose α > 0, and c is a bounded vector. Then φ is given
by
φ = (αI − Q)−1 c. (6.80)

Proof: We have
φ(i) = E(C|X(0) = i)
Z ∞ 
= E e−αt c(X(t))dt|X(0) = i
0
250 CONTINUOUS-TIME MARKOV CHAINS
Z ∞
= e−αt E(c(X(t))|X(0) = i)dt
0
Z ∞ X
= e−αt pij (t)c(j)dt.
0 j∈S

The interchanges of integrals and expectations are justified since c is bounded. In

matrix form the above equation becomes
Z ∞ 
−αt
φ= e P (t)dt c. (6.81)
0

The right hand side is the Laplace transform of P (t) evaluated at α. From Theo-
rem 6.8, we see that is is given by (αI − Q)−1 . Hence we get Equation 6.80.
Note that there is no assumption of irreducibility or transience or recurrence be-
hind the above theorem. Equation 6.80 is valid for any generator matrix Q. Note that
the matrix αI − Q is invertible for α > 0. Also, the theorem remains valid if the c
vector is bounded from below or above. It does not need to be bounded from both
sides.

Example 6.42 Two-State Machine. Consider the two-state machine of Exam-

ple 6.4 on page 195. It was modeled by a CTMC {X(t), t ≥ 0} with state-space
{0, 1} (0 being down, and 1 being up), and generator matrix
 
−λ λ
Q= ,
µ −µ
where λ, µ > 0. Now suppose the machine produces a revenue of $r per day when
it is up, and it costs $d in repair costs per day when the machine is down. Suppose a
new machine in working order costs $m. Is it profitable to purchase it if the continu-
ous discount factor is α > 0?

Let c(i) be the expected cost incurred per unit time spent in state i. We have
c = [c(0) c(1)]′ = [d − r]′ .
Then, using Theorem 6.30 we get
φ = [φ(0) φ(1)]′ = (αI − Q)−1 c.
Direct calculations yield
 
1 d(α + µ) − rλ
φ= .
α(α + λ + µ) dµ − r(α + λ)
Thus it is profitable to buy a new machine if the expected total discounted net revenue
from a new machine over the infinite horizon is greater than the initial purchase price
of m, i.e., if
r(α + λ) − dµ
m≤ .
α(α + λ + µ)
How much should you be willing to pay for a machine in down state?
CTMCS WITH COSTS AND REWARDS 251
6.12.2 Average Costs

The discounted costs have the disadvantage that they depend upon the discount fac-
tor and the initial state, thus making decision making more complicated. These is-
sues are addressed by considering the long run cost per unit time, called the av-
R t cost. The expected total cost up to time t, starting from state i, is given by
erage
E( 0 c(X(u))du|X(0) = i). Dividing it by t gives the cost per unit time. Hence the
long run expected cost per unit time is given by:
Z t  
1 
g(i) = lim E c(X(u))duX(0) = i ,

t→∞ t 0
assuming that the above limit exists. To keep the analysis simple, we will assume that
the CTMC is irreducible and positive recurrent with limiting distribution given by
{pj , j ∈ S}, which is also the limiting occupancy distribution.
P Intuitively, it makes
sense that the long run cost per period should be given by pj c(j), independent of
the initial state i. This intuition is formally proved in the next theorem:

Theorem 6.31 Average Cost. Suppose {X(t), t ≥ 0} is an irreducible positive

recurrent CTMC with limiting occupancy distribution {pj , j ∈ S}. Suppose
X
pj |c(j)| < ∞.
j∈S

Then X
g(i) = g = pj c(j).
j∈S

Proof: Let Mij (t) be the expected time spent in state j over (0, t] starting from state
i. See Section 6.4. Then, we see that
1X
g(i) = lim Mij (t)c(j)
t→∞ t
j∈S
X Mij (t)
= lim c(j)
t→∞ t
j∈S
X Mij (t)
= lim c(j)
t→∞ t
j∈S
X
= pj c(j).
j∈S

Here the last interchange of sum and the limit is allowed because the CTMC is posi-
tive recurrent. The last equality follows from Theorem 6.27.
We illustrate with an example.

Example 6.43 Two-State Machine. Consider the cost structure regarding the two-
state machine of Example 6.42. Compute the long run cost per unit time.
252 CONTINUOUS-TIME MARKOV CHAINS
The steady state distribution of the two-state machine is given by (see Exam-
ple 6.34)
µ λ
p0 = , p1 = .
λ+µ λ+µ
From Theorem 6.31 we see that the expected cost per unit time in the long run is
given by
cµ − rλ
g = c(0)p0 + c(1)p1 = .
λ+µ
Thus it is profitable to operate this machine if rλ > cµ.

Example 6.44 Single-Server Queue. Let X(t) be the number of customers in the
single server queue with arrival rate λ and service rate µ as described in Exam-
ple 6.11. Now suppose the cost of keeping customers in the system is $c per customer
per hour. The entry fee is $d per customer, paid upon entry. Compute the long run
net revenue per unit time.

We saw in Example 6.36 that {X(t), t ≥ 0} is positive recurrent if ρ = λ/µ < 1

and in that case the limiting distribution is given by
pj = ρj (1 − ρ), j ≥ 0.
The cost structure implies that c(j) = jc, j ≥ 0. Using Theorem 6.31 we see that
the long run expected cost per unit time is given by
∞ ∞
X X cρ
pj c(j) = jρj (1 − ρ)c = .
j=0 j=0
1−ρ

Let N (t) be the number of arrivals over (0, t]. Since the arrival process is PP(λ), we
get r, the long run fees collected per unit time as
E(N (t)) λt
r=d = d = λd.
t t
Thus the net expected revenue per unit time is given by
cρ
λd − .
1−ρ
c
A bit of algebra shows that the entry fee must be at least µ−λ in order to break even.

It is possible to use the results in Section 6.11 to extend this analysis to reducible
CTMCs. However, the long run cost rate may depend upon the initial state in that
case.

6.13 Phase Type Distributions

The distribution given in Equation 6.34 is called a phase type distribution with pa-
rameters (α, M ). Any random variable whose distribution can be represented as in
PHASE TYPE DISTRIBUTIONS 253
Equation 6.34 for a valid α and M is called a phase type random variable. (By
“valid” we mean that α is a pmf, and M is obtained from a generator matrix of
an irreducible CTMC by deleting rows and columns corresponding to a non-empty
subset of states.) It is denoted by P H(α, M ), and the size of M is called the num-
ber of phases in the random variable. Many well-known distributions are phase type
distributions, as is demonstrated in the next example.

Example 6.45 Examples of Phase Type Random Variables.

1. Exponential. The exp(λ) is a P H(α, M ) random variable with

α = [1], M = [−λ].
2. Sums of Exponentials. Let Xi ∼ exp(λi ), 1 ≤ i ≤ K, be independent random
variables. Then
T = X1 + X2 + · · · + XK
is a P H(α, M ) random variable with
α = [1, 0, 0, · · · , 0],
and  
−λ1 λ1
 −λ2 λ2 
.. .. .. ..
 
M =
 .. .

 . . . . . 
 −λK−1 λK−1 
−λK
As a special case we see that Erl(λ, K) is a phase type random variable.
3. Mixtures of Exponentials. Let Xi ∼ exp(λi ), 1 ≤ i ≤ K, be independent
random variables. Let
α = [α1 , α2 , · · · , αK ]
be such that
K
X
αi ≥ 0, αi = 1.
i=1
Let
T = Xi with probability αi , 1 ≤ i ≤ K.
Thus T is a mixture of exponentials. It is also a P H(α, M ) random variable with
α as above, and
 
−λ1 0 ··· 0 0
 0 −λ2 · · · 0 0 
 .. . . ..  .
 
M = . .. . .. ..
 . 

 0 0 · · · −λK−1 0 
0 0 ··· 0 −λK

Examples 2 and 3 are special cases of the general theorem given below.
254 CONTINUOUS-TIME MARKOV CHAINS
Theorem 6.32 Sums and mixtures of a finite number of independent phase type
random variables are phase type random variables.

Proof: We treat the case of two independent random variables, Ti ∼ P H(αi , Mi )

(i = 1, 2). The general case follows similarly. Let T = T1 + T2 . Define
G = M1 e1 α2 .
Then it is possible to show that (see Conceptual Exercise 6.13) T is a P H(α, M )
random variable with
α = [α1 , 0 · α2 ], (6.82)
and  
M1 G
M= . (6.83)
0 M2
Next, let [β1 , β2 ] is such that βi ≥ 0, and β1 + β2 = 1. Let

T1 with probability β1 ,
T = (6.84)
T2 with probability β2 .
Thus T is a [β1 , β2 ] mixture of {T1 , T2 }. Then it is possible to show that (see Con-
ceptual Exercise 6.14) T is a P H(α, M ) random variable with
α = [β1 α1 , β2 α2 ], (6.85)
and  
M1 0
M= . (6.86)
0 M2
This establishes the theorem.
The phase type distributions form a versatile family of distributions. We refer the
reader to the book by Neuts (1981) for further details. It can be shown that the set
of phase type distributions is dense in the set of all continuous distributions over
[0, ∞). Thus, any continuous distribution over [0, ∞) can be approximated arbitrarily
closely by a phase type distribution. However, this denseness is of little practical
value if the approximating distribution has a very large number of phases. Use of PH
type distributions does help in developing tractable algorithms for the performance
evaluation of many stochastic systems.

6.14 Reversibility

In Section 4.9 we studied reversible DTMCs. In this section we study the reversible
CTMCs. Intuitively, if we watch a movie of a reversible CTMC we will not be able to
tell whether the time is running forward or backward. We begin with the definition.

Definition 6.6 A CTMC is called reversible if the corresponding embedded DTMC

is reversible.
REVERSIBILITY 255
Using the definition of reversible DTMCs, we immediately get the following theo-
rem.

Theorem 6.33 A CTMC with state-space S and generator matrix Q is reversible if

and only if for every r ≥ 1, and i0 6= i1 6= · · · =
6 ir ,
qi0 ,i1 qi1 ,i2 · · · qir−1 ,ir qir ,i0 = qi0 ,ir qir ,ir−1 qir−1 ,ir−2 · · · qi1 ,i0 . (6.87)

Proof: Suppose the CTMC is reversible. Then, by definition, so is the corresponding

embedded DTMC. Now, if qik = 0 for any 0 ≤ k ≤ r the Equation 6.87 holds
trivially since both sides are zero. So assume that qik > 0 for all 0 ≤ k ≤ r. Since
the DTMC is reversible, we have, from Definition 4.12,
pi0 ,i1 pi1 ,i2 · · · pir−1 ,ir pir ,i0 = pi0 ,ir pir ,ir−1 pir−1 ,ir−2 · · · pi1 ,i0 . (6.88)
Using
pij = qij /qi , i 6= j,
the above equation reduces to Equation 6.87.
Next suppose Equation 6.87 holds. Then, using the transition probabilities of the
embedded DTMC we see that Equation 6.88 holds. Hence the embedded DTMC is
reversible. Hence the CTMC is reversible.
The next theorem is analogous to Theorem 4.26.

Theorem 6.34 An irreducible, positive recurrent CTMC with state-space S, gener-

ator matrix Q and stationary distribution {pi , i ∈ S} is reversible if and only if
pi qij = pj qji , i, j ∈ S, i 6= j. (6.89)

Proof: Suppose the CTMC is irreducible, positive recurrent, and reversible. Since
we implicitly assume that the CTMC is regular, the embedded DTMC is irreducible,
positive recurrent, and reversible. Let {πi , i ∈ S} be the stationary distribution of the
DTMC. Since it is reversible, Theorem 4.26 yields
πi pij = πj pji , i 6= j ∈ S,
which implies that
qij qji
= πj , i 6= j ∈ S.
πi
qi qj
From Theorem 6.26, we know that pj = Cπj /qj , for some constant C. Hence the
above equation reduces to Equation 6.89.
Now suppose the CTMC is regular, irreducible, positive recurrent, and Equa-
tion 6.89 holds. This implies that
pi qi pij = pj qj pji .
Then, using Theorem 6.26, we see that the above equation reduces to
πi pij = πj pji .
256 CONTINUOUS-TIME MARKOV CHAINS
Thus the embedded DTMC is reversible. Hence the CTMC is reversible, and the the-
orem follows.

The Equations 6.89 are called the local balance or detailed balance equations, as
opposed to Equations 6.64, which are called global balance equations. Intuitively,
the local balance equations say that, in steady state, the rate of transitions from state
i to j is the same as the rate of transitions from j to i. This is in contrast to stationary
CTMCs that are not reversible: for such CTMCs the global balance equations imply
that the rate of transitions out of a state is the same as the rate of transitions into that
state. It can be shown that the local balance equations imply global balance equations,
but not the other way. See Conceptual Exercise 6.16.

Example 6.46 Birth and Death Processes. Consider a positive recurrent birth and
death process as described in Example 6.35. Show that it is a reversible DTMC.
From Equation 6.67 we see that the stationary distribution satisfies the local bal-
ance equations
pi qi,i+1 = pi+1 qi+1,i .
Since the only transitions in a birth and death process are from i to i + 1, and i to
i − 1, we see that Equations 6.89 are satisfied. Hence the CTMC is reversible.

Reversible CTMCs arise in population models, networks of queues, and a variety

of other applications. An excellent account is given in the books by Kelly (1979)
and Whittle (1986). The steady state analysis of reversible CTMCs is usually very
simple due to the simplicity of the local balance equations. In fact, one method of
establishing reversibility is to guess the limiting distribution, and then verify that it
satisfies the local balance equations. In many cases, Equation 6.89 itself is useful in
coming up with a good guess. We illustrate with two examples.

Example 6.47 Database Locking. A database is a collection of information

records stored on a computer. The records are indexed 1 through N . A user can ac-
cess multiple records. We say that the user belongs to class A if he wishes to access
records with indices in the set A ⊆ {1, 2, · · · , N }. When a user of class A arrives, he
checks to see if all the records in A are available. If they are, the user gets access to
them and he immediately locks the records. If a record is locked, it is unavailable to
any other user. When the user is done, he simultaneously unlocks all the records he
was using. If an arriving user finds that some of the records that he needs to access
are locked (and hence unavailable to him), he leaves immediately.

(In practice there are two classes of users: readers (who read the records) or writers
(who alter the records), and correspondingly, the locks are of two types: exclusive and
shared. When a writer wants to access records in the set A, he gets the access if none
of the records in A have any kind of lock from other users, in which case he puts an
exclusive lock on each of them. When a reader wants to access records in the set A,
he gets it if none of the records in A have an exclusive lock on them, and then puts
REVERSIBILITY 257
a shared lock on each of them. This ensures that many readers can simultaneously
read a record, but if a record is being modified by a writer no one else can access a
record. We do not consider this further complication here.)
Now suppose users of type A arrive according to a P P (λ(A)), and the arrival pro-
cesses are independent of each other. A user of type A needs an exp(µ(A)) amount
of time to process these records. The processing times are independent of each other
and the arrival processes. We say that the database is in state (A1 , A2 , · · · , Ak ) if
there is exactly one user of type Ai (1 ≤ i ≤ k) in the system at time t. This implies
that the sets A1 , A2 , · · · , Ak must be mutually disjoint. When there are no users in
the system, we denote that state by φ.

Let X(t) be the state of the database at time t. The process {X(t), t ≥ 0} is a
CTMC with state-space
S = {φ} ∪ {(A1 , A2 , · · · , Ak ) : k ≥ 1, Ai 6= φ, Ai ⊆ {1, 2, · · · , N },
Ai Aj = φ if i 6= j, 1 ≤ i, j ≤ k},
and transition rates given below (we write q(i → j) instead of qij for the ease of
reading):
q(φ → (A)) = λ(A),
q((A1 , A2 , · · · , Ak ) → (A1 , A2 , · · · , Ak , Ak+1 )) = λ(Ak+1 ),
q((A1 , A2 , · · · , Ak ) → (A1 , · · · , Ai−1 , Ai+1 , · · · , Ak )) = µ(Ai ),
q((A) → φ) = µ(A).
The above rates are defined whenever (A1 , A2 , · · · , Ak ) ∈ S and (A1 , A2 , · · · ,
Ak , Ak+1 ) ∈ S. Thus the state-space is finite and the CTMC is irreducible and
positive recurrent. Let
p(φ) = lim P(X(t) = φ),
t→∞
p(A1 , A2 , · · · , Ak ) = lim P(X(t) = (A1 , A2 , · · · , Ak )),
t→∞

for (A1 , A2 , · · · , Ak ) ∈ S. Now suppose, for (A1 , A2 , · · · , Ak ) ∈ S,

λ(A1 )λ(A2 ) · · · λ(Ak )
p(A1 , A2 , · · · , Ak ) = p(φ). (6.90)
µ(A1 )µ(A2 ) · · · µ(Ak )
It is straightforward to verify that this satisfies the local balance equations:
λ(A)p(φ) = µ(A)p(A), (A) ∈ S,
λ(Ak+1 )p(A1 , A2 , · · · , Ak ) = µ(Ak+1 )p(A1 , A2 , · · · , Ak , Ak+1 ),
for (A1 , A2 , · · · , Ak+1 ) ∈ S. The above equations imply that {X(t), t ≥ 0} is a
reversible CTMC with the limiting distribution given in Equation 6.90. Using the
normalizing equation we get
 −1
 X λ(A1 )λ(A2 ) · · · λ(Ak ) 
p(φ) = 1 + .
 µ(A1 )µ(A2 ) · · · µ(Ak ) 
(A1 ,A2 ,···,Ak )∈S
258 CONTINUOUS-TIME MARKOV CHAINS
Of course, the hard part here is to compute the sum in the above expression.

Example 6.48 Restaurant Process. Consider a restaurant with seating capacity

N . Diners arrive in batches (parties) and leave in the same party that they arrived in.
Suppose parties of size i arrive according to P P (λi ), independently of each other.
When a party of size i arrives and the restaurant has i or more seats available, the
party is seated immediately. (We assume that the seating is flexible enough so that no
seats are wasted.) If not enough seats are available, the party leaves immediately. A
party of size i that gets seated stays in the restaurant for an exp(µi ) amount of time,
independent of everything else. This implies that the restaurant has ample service
capacity so that the service is not affected by how busy the restaurant is.

We say that the restaurant is in state (i1 , i2 , · · · , ik ) if there are k parties in the
restaurant and the size of the rth party is ir (1 ≤ r ≤ k). The empty restaurant is
said to be in state φ. Let X(t) be the state of the restaurant at time t. The process
{X(t), t ≥ 0} is a CTMC with state-space
k
X
S = {φ} ∪ {(i1 , i2 , · · · , ik ) : k ≥ 1, ir > 0, 1 ≤ r ≤ k, ir ≤ N },
r=1

and transition rates given below (we write q(i → j) instead of qij for the ease of
reading):
q(φ → (i)) = λi , 1 ≤ i ≤ N,
q((i1 , i2 , · · · , ik ) → (i1 , i2 , · · · , ik , ik+1 )) = λik+1 , (i1 , i2 , · · · , ik+1 ) ∈ S,
q((i1 , i2 , · · · , ik ) → (i1 , · · · , ir−1 , ir+1 , · · · , ik )) = µir , (i1 , i2 , · · · , ik ) ∈ S,
q((i) → φ) = µi 1 ≤ i ≤ N.
Thus the state-space is finite and the CTMC is irreducible and positive recurrent. Let
p(φ) = lim P(X(t) = φ),
t→∞
p(i1 , i2 , · · · , ik ) = lim P(X(t) = (i1 , i2 , · · · , ik )), (i1 , i2 , · · · , ik ) ∈ S.
t→∞

Now suppose, for (i1 , i2 , · · · , ik ) ∈ S,

λi1 λi2 · · · λik
p(i1 , i2 , · · · , ik ) = p(φ). (6.91)
µi1 µi2 · · · µik
It is straightforward to verify that this satisfies the local balance equations:
λi p(φ) = µi p(i), 1 ≤ i ≤ N,
λik+1 p(i1 , i2 , · · · , ik ) = µik+1 p(i1 , i2 , · · · , ik , ik+1 ),
for (i1 , i2 , · · · , ik , ik+1 ) ∈ S. The above equations imply that {X(t), t ≥ 0} is a
reversible CTMC with the limiting distribution given in Equation 6.91 . Using the
normalizing equation we get
 −1
 X λi1 λi2 · · · λik 
p(φ) = 1 + .
 µi1 µi2 · · · µik 
(i1 ,i2 ,···,ik )∈S
MODELING EXERCISES 259
As in Example 6.47, the hard part here is to compute the sum in the above expression
since the number of terms in the sum grows exponentially in N .

6.15 Modeling Exercises

6.1 Consider a workshop with k machines, each with its own repairperson. The
k machines are identical and behave independently of each other as described in
Examples 6.1 and 6.4. Let X(t) be the number of working machines at time t. Show
that {X(t), t ≥ 0} is a CTMC. Display its generator matrix and the rate diagram.

6.2 Do Modeling Exercise 6.1 assuming that there are r repairpersons (1 ≤ r < k).
The machines are repaired in the order of failure.

6.3 Consider the two-machine one repairperson workshop of Example 6.6. Suppose
the repair time of the ith machine is exp(λi ) and the lifetime is exp(µi ), i = 1, 2.
The repairs take place in the order of failure. Construct an appropriate CTMC to
model this system. Assume that all the life times and repair times are independent.

6.4 Do Modeling Exercise 6.3 with three distinct machines. Care is needed to handle
the “order of failure” service.

6.5 A metal wire, when subject to a load of L kilograms, breaks after an exp(µL)
amount of time. Suppose k such wires are used in parallel to hold a load of L kilo-
grams. The wires share the load equally, and their failure times are independent.
Let X(t) be the number of unbroken wires at time t, with X(0) = k. Formulate
{X(t), t ≥ 0} as a CTMC.

6.6 A bank has five teller windows, and customers wait in a single line and are
served by the first available teller. The bank uses the following operating policy: if
there are k customers in the bank, it keeps one teller window open for 0 ≤ k ≤ 5,
two windows for 6 ≤ k ≤ 8, three for 9 ≤ k ≤ 12, four for 13 ≤ k ≤ 15, and all five
are kept open for k ≥ 16. The service times are iid exp(µ) at any of the tellers, and
the arrival process is PP(λ). Let X(t) be the number of customers in the system at
time t. Show that {X(t), t ≥ 0} is a birth and death process and find its parameters.

6.7 Consider a two-server system where customers arrive according to a PP(λ).

Those going to server i require exp(µi ) amount of service time, i = 1, 2. A customer
arriving at an empty system goes to server 1 with probability α and to server 2 with
probability 1 − α. Otherwise a customer goes to the first available server. Formulate
this as a CTMC. Is it a birth and death process?

6.8 Consider a two-server queue that operates as follows: two different queues are
formed in front of the two servers. The arriving customer joins the shorter of the two
queues (the customer in service is counted as part of that queue). If the two queues
are equal, he joins either one with probability .5. Queue jumping is not allowed. Let
260 CONTINUOUS-TIME MARKOV CHAINS
Xi (t) be the number of customers in the ith queue, including any in service with
server i, i = 1, 2. Assume that the arrival process is PP(λ) and the service times are
iid exp(µ) at either server. Model {(X1 (t), X2 (t)), t ≥ 0} as a CTMC and specify
its transition rates and draw the rate diagram.

6.9 Consider a system consisting of n components in series, i.e., it needs all com-
ponents in working order in order to function properly. The lifetime of the ith com-
ponent is exp(λi ) random variable. At time 0 all the components are up. As soon as
any of the components fails, the system fails, and the repair starts immediately. The
repair time of the ith component is an exp(µi ) random variable. When the system is
down, no more failures occur. Let X(t) = 0 if the system is functioning at time t,
and X(t) = i if component i is down (and hence under repair) at time t. Show that
{X(t), t ≥ 0} is a CTMC and display its rate diagram.

6.10 Consider an infinite server queue of Example 6.12. Suppose the customers
arrive according to a CPP with batch arrival rate λ and the successive batch sizes are
iid geometric with parameter 1 − p. Each customer is served by a different server
in an independent fashion, and the service times are iid exp(µ) random variables.
Model the number of customers in the system by a CTMC. Compute the transition
rates.

6.11 Consider a queueing system with s servers. The incoming customers belong
to two classes: 1 and 2. Class 2 customers are allowed to enter the system if and
only if there is a free server immediately available upon their arrival, otherwise they
are turned away. Class 1 customers always enter the system and wait in an infinite
capacity waiting room for service if necessary. Assume that class i customers arrive
according to independent P P (λi ), i = 1, 2. The service times are iid exp(µ) for
both classes. Let X(t) be the number of customers in the system at time t. Show that
{X(t), t ≥ 0} is a birth and death process and find its parameters.

6.12 Consider a bus depot where customers arrive according to a PP(λ), and the
buses arrive according to a P P (µ). Each bus has capacity K > 0. The bus de-
parts with min(x, K) passengers, where x is the number of customers waiting when
the bus arrives. Loading time is insignificant. Let X(t) be the number of customers
waiting at the depot at time t. Show that {X(t), t ≥ 0} is a CTMC and compute its
generator matrix.

6.13 Customers arrive at a single-server facility according to a PP(λ). An arriving

customer, independent of everything else, belongs to class 1 with probability α and
class 2 with probability β = 1 − α. The service times of class i customers are iid
exp(µi ), i = 1, 2, with µ1 6= µ2 . The customers form a single queue. Let X(t) be
the number of customers in the system at time t and Y (t) be the class of the customer
in service. Define Y (t) = 0 if X(t) = 0. Model {(X(t), Y (t)), t ≥ 0} as a CTMC
and specify its transition rates and draw the rate diagram.
MODELING EXERCISES 261
6.14 Consider the single-server queue of Example 6.11. Suppose that every cus-
tomer has “patience time” such that if his wait for service to begin exceeds the pa-
tience time, he leaves the system without service. Suppose the patience times of
customers are iid exp(θ) random variables. Let X(t) be the number of customers in
the system at time t. Show that {X(t), t ≥ 0} is a is a birth and death process and
find its parameters.

6.15 Genetic engineers have come up with a super-amoeba with exp(µ) lifetimes
and following property: at the end of its lifetime it ceases to exist with probability
0.3, or splits into two clones of itself with probability .4, or three clones of itself
with probability .3. Let X(t) be the number of super-amoebae in a colony at time
t. Suppose all the existing amoebae behave independently of each other. Show that
{X(t), t ≥ 0} is a CTMC and compute its generator matrix.

6.16 Consider a single-server queue with the following operating policy: Once the
server becomes idle, it stays idle as long there are less than N (a given positive
integer) customers in the system. As soon as there are N customers in the system,
the server starts serving them one by one and continues until there is no one left in
the system, at which time the server becomes idle. The arrival process is PP(λ) and
the service times are iid exp(µ). Model this as a CTMC by describing its state-space
and the generator matrix. Draw its rate diagram.

6.17 Consider a computer system with five independent and identical central pro-
cessing units (CPUs). The lifetime of each CPU is exp(µ). When a CPU fails, an
automatic mechanism instantaneously isolates the failed CPU, and the system con-
tinues in a reduced capacity with the remaining working CPUs. If this automatic
mechanism fails to isolate the failed CPU, which can happen with probability 1 − c,
the system crashes. If no working CPUs are left, then also the system crashes. Since
the system is aboard a spaceship, once the system crashes it cannot be repaired.
Model the system by a CTMC by giving its state-space, generator matrix, and the
rate diagram.

6.18 Consider the system in Modeling Exercise 6.17. Now suppose there is a robot
aboard the space ship that can repair the failed CPUs one at a time, the repair time
for each CPU being exp(λ). However, the robot uses the computer system for the
repair operation, and hence requires a non-crashed system. That is, the robot works
as long as the system is working. Model this modified system by a CTMC by giving
its state-space, generator matrix, and the rate diagram.

6.19 Consider the s-server model of Modeling Exercise 6.11. Suppose the service
times of the customers of class i are iid exp(µi ), i = 1, 2. Model this modified
system by a CTMC by giving its state-space and the generator matrix.

6.20 Unslotted Aloha. Consider a communications system where the messages ar-
rive according to a PP(λ). As soon as a message arrives it attempts transmission. The
262 CONTINUOUS-TIME MARKOV CHAINS
message transmission times are iid exp(µ). If no other message tries to transmit dur-
ing this time, the message transmission is successful. Otherwise a collision results,
and both the messages involved in the collision are terminated instantaneously. All
messages involved in a collision are called backlogged, and are forced to retransmit.
All backlogged messages wait for an exp(θ) amount of time (independently of each
other) before starting retransmission attempt. Let X(t) be the number of backlogged
messages at time t, Y (t) = 1 if a message is under transmission at time t and zero
otherwise. Model {(X(t), Y (t)), t ≥ 0} as a CTMC and specify its transition rates
and draw the rate diagram.

6.21 Leaky Bucket. Packets arrive for transmission at a node according to a PP(λ)
and join the packet queue. Tokens (permission slips to transmit) arrive at that node
according to a P P (µ) and join the token pool. The node uses a “leaky bucket” trans-
mission protocol to control the entry of the packets into the network, and it operates
as follows: if the token pool is empty, the arriving packets wait in the packet queue.
Otherwise, an incoming packet removes a token from the token pool and is instanta-
neously transmitted. Thus the packet queue and the token pool cannot simultaneously
be non-empty. The token pool size is M , and any tokens generated when the token
pool is full are discarded. Model this modified system by a CTMC by giving its
state-space, generator matrix, and the rate diagram.

6.22 An inventory system is under continuous control as follows. Demands arrive

according to PP(λ) and are satisfied instantly if there are items in the warehouse.
If the warehouse is empty when a demand occurs, the demand is lost. As soon as
the inventory level drops to R (a given positive integer), an order is placed for K
(another given positive integer > R) items from the supplier. It takes an exp(θ)
amount of time for the supplier to deliver the order to the warehouse. Let X(t) be
the number of items in the inventory at time t, and assume that R < X(0) ≤ K + R.
Model {X(t), t ≥ 0} as a CTMC by giving its state-space and the generator matrix.

6.23 This model is of the growth of a macromolecule in a chemical solution. A

chemical solution has two types of molecules, labeled A and T , suspended in it. The
A molecules are active, while the T molecules are terminal. The A molecules can
latch onto each other and form linear macromolecules, e.g., AAA · · · AA. Such a
macromolecule can grow by adding an A or a T molecule at either end. Once there
is a T molecule at an end, no more molecules can be added to that end. Thus a
macromolecule T AAA can grow only at one end, while the macromolecule T AAT
cannot grow any more. Suppose that an A molecule can get attached to another A
molecule at rate λ, while a T molecule can attach to an A with rate θ. The bond
between two A molecules can come unglued at rate µ, while the bond between an A
and a T molecule is permanent. Assume that the length of a macromolecule can only
increase or decrease by one unit at a time. Let X(t) be the length of a macromolecule
at time t. Assume that at time zero a macromolecule consists of a single A molecule.
Model {X(t), t ≥ 0} as a CTMC by giving its state-space and the rate diagram.
MODELING EXERCISES 263
6.24 There are K parking spaces, indexed 1 through K, in front of a retail store.
Space 1 is the closest (and the most preferred) and space K is the farthest (and
the least preferred). Customers arrive according to a PP(λ) and occupy the closest
available space. If all spaces are occupied, the customer goes away and is lost. Each
customer spends an exp(µ) amount of time in the store, independent of the others.
Let Xi (t) = 1 (1 ≤ i ≤ K) if the i-th space is occupied at time t and 0 otherwise.
Also define Zk (t) to be the number of spaces among {1, 2, ..., k} that are occupied
at time t, (1 ≤ k ≤ K). Is {Xi (t), t ≥ 0} a CTMC (1 ≤ i ≤ K)? If yes, show its
generator matrix, or the rate diagram. Is {Zk (t), t ≥ 0} a CTMC (1 ≤ k ≤ K)? If
yes, show its generator matrix, or the rate diagram.

6.25 Customers arrive according to PP(λ) to a queueing system with two servers.
The ith server (i = 1, 2) needs exp(µi ) amount of time to serve one customer. Each
incoming customer is routed either to server 1 with probability p1 or to server 2 with
probability p2 = 1 − p1 , independently. Queue jumping is not allowed, thus there is a
separate queue in front of each server. Let Xi (t) be the number of customers in the i-
th queue at time t (including any in service with the ith server). Show that {Xi (t), t ≥
0} is a birth and death process, i = 1, 2. Are the two processes independent?

6.26 Jobs arrive at a central computer according to a PP(λ). The job processing
times are iid exp(µ). The computer processes them one at a time in the order of
arrival. The computer is subject to failures and repairs as follows: It stays functional
for an exp(α) amount of time and then fails. It takes an exp(β) amount of time to
repair it back to a fully functional state. Successive up and down times are iid and
independent of the number of jobs in the system. When the computer fails, all the
jobs in the system are lost. All jobs arriving at the computer while it is down are also
lost. Model this system as a CTMC by giving its state-space and the rate diagram.

6.27 A single server queue serves K types of customers. Customers of type k arrive
according to a PP(λk ), and have iid exp(µk ) service times. An arriving customer
enters service if the server is idle, and leaves immediately if the server is busy at the
time of arrival. Let X(t) be 0 if the server is idle at time t, and X(t) = k if the
server is serving a customer of type k at time t. Model X(t) as a CTMC by giving
its state-space and the rate diagram.

6.28 Life time of a machine is exponentially distributed with parameter µ. A repair-

person visits this machine periodically, the inter-visit times being iid exp(λ). If the
machine is discovered to be in working condition, the repairperson leaves it alone;
otherwise he repairs it. Repair takes exp(θ) amount of time and makes the machine
as good as new. Repair visits continue after repair completion as before. Model this
system as a CTMC by giving its state-space and the rate diagram.

6.29 Consider the following modification of the machine shop of Example 6.5.
When both machines are down each machine is repaired by one repairperson. How-
ever, if only one machine is down, both repair persons work on it together so that
the repair occurs at twice the speed. Model this system as a CTMC by giving its
state-space and the rate diagram.
264 CONTINUOUS-TIME MARKOV CHAINS
6.30 A machine produces items according to a PP(λ). The produced items are stored
in a buffer. Demands for these items occur according to a PP(µ). If an item is avail-
able in the buffer when a demand occurs, the demand is satisfied immediately. If the
buffer is empty when a demand occurs, the demand is lost. The machine is turned
off when the number of items in the buffer reaches K, a fixed positive number, and
is turned on when the number of items in the buffer falls to a pre-specified number
k < K. Once it is turned on it stays on until the number of items in the buffer reaches
K. Model this system as a CTMC. State the state-space and show the rate diagram.

6.31 Customers of two classes arrive at a single server service station with infinite
waiting room. Class i customers arrive according to PP(λi ), i = 1, 2. Customers
of class 1 are always allowed to enter the system, while those of class 2 can enter
the system if and only if the total number of customers in the system (before this
customer joins) is less than K, where K > 0 is a fixed integer. The service times
are iid exp(µ) for both the classes. Let X(t) be the total number of customers in the
system at time t. Show that {X(t), t ≥ 0} is a birth and death process and state its
parameters.

6.32 Customers arrive at a service station according to a PP(λ). Each customer

brings with him two tasks that can be done in parallel. Each task takes an exp(µ)
amount of time, and the two tasks are independent. There are two servers in the sys-
tem. When a customer enters service, the servers begin working on one task each.
When one of two tasks finishes, the freed up server helps the other server so that the
remaining task is processed at twice the rate. (i.e., the remaining service time of the
task is now exp(2µ) instead of exp(µ).) Model this system as a CTMC. State the
state-space and show the rate diagram.

6.33 Consider a gas station with two pumps and three car-spaces as shown in Fig-
ure 6.11. Potential customers arrive at the gas station according to a PP(λ). An in-

Departures Space Space Space Arrivals

1 2 3

Pump Pump
1 2

Figure 6.11 The 2-pump gas station.

coming customer goes to pump 1 if both pumps are idle. If pump 1 is busy but pump
2 is idle, she goes to pump 2. If pump 2 is occupied she waits in the space 3 (re-
gardless of whether pump 1 occupied or not. If space 3 is occupied she goes away. It
takes an exp(µ) amount of time to fill gas. After finishing filling gas, the customer at
pump 1 leaves, the customer at pump 2 leaves if space 1 is vacant, else she has to wait
until the customer in space 1 is done (due to one way rules and space restrictions). In
that case both customers leave simultaneously when the customer in space 1 is done.
Model this system as a CTMC. State the state-space and show the rate diagram.
 COMPUTATIONAL EXERCISES 265
6.34 A machine shop has three identical machines, at most two of which can be in
use at any time. Initially all three machines are in working condition. The policy of
the shop is to keep a machine in standby mode if and only if there are two working
machines in use. Each machine in use fails after an exp(λ) amount of time, whereas
the standby machine (if in working condition) fails after an exp(θ) amount of time.
The failure of a standby machine can be detected only after it is put in use. There is
a single repair person who repairs the machines in exp(µ) amount of time. Model
this system as a CTMC. Describe the state-space and show the rate diagram. Assume
independence as needed.

6.16 Computational Exercises

6.1 Consider a pure birth process {X(t), t ≥ 0} with birth parameters

λ2n = α > 0, λ2n+1 = β > 0, n ≥ 0.
Compute P(X(t) is odd|X(0) = 0) for t ≥ 0.

6.2 Let mi (t) = E(X(t)|X(0) = i), where X(t) is the number of customers at
time t in an infinite-server queue of Example 6.12. Derive the differential equations
for mi (t) (i ≥ 0) by using the forward equations for pij (t). Solve for m0 (t).

6.3 Let {X(t), t ≥ 0} be the CTMC of Example 6.5. Compute its transition proba-
bility matrix P (t) by

(i) exploiting the independence of the machines,

(ii) using the matrix exponential technique,
(iii) using the Laplace transform technique.

6.4 Consider the Yule process {X(t), t ≥ 0} with λn = nλ for n ≥ 0. Compute

P(X(t) = j|X(0) = i).

6.5 For the pure death process {X(t), t ≥ 0} with µn = nµ for n ≥ 0, compute
P(X(t) = j|X(0) = i).

6.6 Consider the CTMC {X(t), t ≥ 0} of Modeling Exercise 6.15. Let mi (t) =
E(X(t)|X(0) = i). Show that
m′i (t) = 0.7mi (t), mi (0) = i.
Solve it.

6.7 Let {X(t), t ≥ 0} be the pure birth process of Computational Exercise 6.1.
Compute E(X(t)|X(0) = 0).
266 CONTINUOUS-TIME MARKOV CHAINS
6.8 Customers arrive at a service center according to a PP(λ), and demand iid exp(µ)
service. There is a single service person and waiting room for only one customer.
Thus if an incoming customer finds the server idle he immediately enters service, or
else he leaves without service. A special customer, who knows how the system works,
wants to use it in a different way. He inspects the server at times 0, T, 2T, 3T, . . .
(here T > 0 is a fixed real number) and enters the system as soon as he finds the
server idle upon inspection. Compute the distribution of the number of visits the
special customer makes until he enters, assuming that he finds the server busy at
time 0. Also compute the expected time that the special customer has to wait until he
enters.

6.9 A machine is either up or down. The up machine stays up for an exp(µ) amount
of time and then fails. The repairperson takes an exp(λ) time to fix the machine to a
good-as-new state. A repair person is on duty from 8:00am to 4:00pm every day, and
works on the machine if it fails. A working machine can fail at any time, whether
the repair person is on duty or not. If the machine is under repair at 4:00pm on a
given day, or it fails while the repair person is off-duty, it stays down until 8:00am
the next day, when the repairperson resumes (or starts) the repair work. Compute the
steady-state probability that the machine is working when the repairperson reports to
work.

6.10 Compute the steady state distribution of the CTMC of Modeling Exercise 6.1.

6.11 Compute the steady state distribution of the CTMC of Modeling Exercise 6.6.
What is the condition of stability?

6.12 Compute the steady state distribution of the CTMC of Modeling Exercise 6.7.
What is the condition of stability?

6.13 Compute the steady state distribution of the CTMC of Modeling Exercise 6.9.

6.14 Compute the steady state distribution of the CTMC of Modeling Exercise 6.4.

6.15 Consider the Modeling Exercise 6.10. Let pk be the limiting probability that
there are k customers in the systems. Define
∞
X
G(z) = pk z k .
k=0

Using the balance equations derive the following differential equation for G(z):
λ 1
G′ (z) = · · G(z).
µ 1 − pz
Solve for G(z).

6.16 Compute the steady state distribution of the CTMC of Modeling Exercise 6.11.
What is the condition of stability?
COMPUTATIONAL EXERCISES 267
6.17 Consider the Modeling Exercise 6.13. Let
pik = lim P(X(t) = i, Y (t) = k), i ≥ 0, k = 1, 2,
t→∞

and define
∞
X
φk (z) = pik z i , k = 1, 2.
i=0
Show that
αλ(λ(1 − z) + µ2 )p00
φ1 (z) = .
µ1 µ2 /z − λµ1 (1 − α/z) − λµ2 (1 − β/z) − λ2 (1 − z)
(Expression for φ2 is obtained by interchanging α with β and µ1 with µ2 .) Compute
p00 and show that the condition of stability is
 
α β
λ + < 1.
µ1 µ2

6.18 Compute the steady state distribution of the CTMC of Modeling Exercise 6.14.
What is the condition of stability?

6.19 Consider the parking lot of Modeling Exercise 6.24. Compute the long run
probability that the i-th space is occupied. Also compute the long run fraction of the
customers that are lost.

6.20 Consider the queueing system of Modeling Exercise 6.25. When is {Xi (t), t ≥
0} stable, i = 1, 2? Assuming it is stable, what is the mean and the variance of the
number of customers in the entire system in steady state?

6.21 Consider the computer system of Modeling Example 6.26. When is this system
stable? What is its limiting distribution, assuming stability? What fraction of the
incoming jobs are completed successfully in steady state?

6.22 Compute the limiting distribution of the number of items in the inventory sys-
tem described in Modeling Exercise 6.22. In steady state, what fraction of the de-
mands are lost?

6.23 Consider the system described in Modeling Exercise 6.27. What is the limiting
probability that the server is idle? What is the limiting probability that it is serving a
customer of type k, 1 ≤ k ≤ K?

6.24 Compute the steady state distribution of the system described in Modeling
Exercise 6.28.

6.25 Consider the system of Modeling Exercise 6.31. What is the condition of sta-
bility for this process? Compute its limiting distribution assuming it is stable.
268 CONTINUOUS-TIME MARKOV CHAINS
6.26 Consider the single server queue of Example 6.11. Now suppose the system
capacity is K, i.e., an arriving customer who finds K customers already in the system
is turned away. Compute the limiting distribution of the number of customers in such
a system.

6.27 Consider the system described in Modeling Exercise 6.32. State the condition
of stability and compute the expected number of customers in steady state assuming
that the system is stable.

6.28 Consider the gas station of Modeling Exercise 6.33. Compute the limiting dis-
tribution of the state of the system. In steady state, what fraction of the potential
customers actually enter the gas station?

6.29 Consider the three machine workshop of Modeling Exercise 6.34. Compute
the limiting distribution of the state of the system. In steady state, what fraction of
the time is the repair person idle?

6.30 Consider the model of macromolecular growth as described in Modeling Ex-

ercise 6.23. Compute the limiting distribution of the length of the molecule.

6.31 A finite state CTMC with generator matrix Q is called doubly stochastic if
the row as well column sums of Q are zero. Find the limiting distribution of an
irreducible doubly stochastic CTMC with state-space {1, 2, · · · , N }.

6.32 Let {N (t), t ≥ 0} be a PP(λ). Compute

lim P(N (t) is divisible by 3 or 7).
t→∞

You may use the result of Computational Exercise 6.31.

6.33 Consider a birth and death process on N nodes arranged in a circle. The pro-
cess takes a clockwise step with rate λ and a counter-clockwise step with rate µ.
Display the rate diagram of the CTMC. Compute its limiting distribution.

6.34 Consider the inventory system described in Modeling Exercise 6.22. Compute
the expected time between two consecutive orders placed with the supplier.

6.35 Consider the machine shop of Modeling Exercise 6.29. Suppose machine 1
fails at time zero, and machine two is working, so that both repair persons are avail-
able to repair machine 1 at that time. Let T be the time when machine 1 becomes
operational again. Compute E(T ).

6.36 Consider the computer system described in Modeling Exercise 6.17. Suppose
at time 0 all CPUs are functioning. Compute the expected time until the system
crashes.
COMPUTATIONAL EXERCISES 269
6.37 Consider the machine shop described in Modeling Exercise 6.1. Suppose one
machine is functioning at time 0. Show that the expected time until all machines is
down is given by !
 k
1 λ+µ
−1 .
kλ µ

6.38 Do the Computational Exercise 6.36 for the computer system in Modeling
Exercise 6.18.

6.39 Consider the finite capacity single server queue of Computational Exer-
cise 6.26. Suppose at time zero there is a single customer in the system. Let T be
the arrival time of the first customer who sees an empty system. Compute E(T ).

6.40 Consider the computer system of Modeling Exercise 6.26. Suppose initially
the computer is up and there are i jobs in the system. Compute the expected time
until there are no jobs in the system.

6.41 Consider the parking lot of Modeling Exercise 6.24. Suppose initially there is
one car in the lot. What is the expected time until the lot becomes empty?

6.42 Consider the queueing system of Modeling Exercise 6.25. Suppose keeping a
customer for one unit of time in the i-th queue costs hi dollars (i = 1, 2), including
the time spent in service. Find the optimum routing probabilities that will minimize
the expected total holding cost per unit time in the two queues in steady state.

6.43 Consider the system described in Modeling Exercise 6.28. Suppose the ma-
chine generates revenue at a rate of $30 per hour while it is operating. Each visit of
the repairperson costs $100, and the repair itself costs $20 per hour. Compute the
long run net income (revenue - cost) per hour if the mean machine lifetime is 80 hrs
and mean repair time is 3 hrs. Find the optimal rate at which the repairperson should
visit this machine so as to maximize the net income rate in steady state.

6.44 Consider the inventory system described in Modeling Exercise 6.22. Suppose
each item sold produces a profit of $r, while it costs $h to hold one item in the
inventory for one unit of time. Compute the long run net profit per unit time.

6.45 Consider the k machine workshop of Modeling Exercise 6.1. Suppose each
working machine produces revenues at rate $r per unit time. Compute the expected
total discounted revenue over infinite horizon starting with k operating machines.
Hint: Use independence of the machines.

6.46 A machine shop consists of K independent and identical machines. Each ma-
chine works for an exp(µ) amount of time before it fails. During its lifetime each
machine produces revenue at rate $R per unit time. When the machine fails it needs
to be replaced by a new and identical machine at a cost of $Cm per machine. Any
number of machines can be replaced simultaneously. The replacement operation is
270 CONTINUOUS-TIME MARKOV CHAINS
instantaneous. The repair-person charges $Cv per visit, regardless of how many ma-
chines are replaced on a given visit. Consider the following policy: Wait until the
number of working machines falls below k, (where 0 < k ≤ K is a fixed integer)
and then replace all the failed machines simultaneously. Compute g(k), the long run
cost rate of following this policy. Compute the optimal value of the parameter k that
minimizes this cost rate for the following data: Number of machines in the machine
shop is 10, mean lifetime of the machine is 10 days, the revenue rate is $100.00 per
day, the replacement cost is $750 per machine, and the visit charge is $250.00.

6.47 Consider the computer system described in Modeling Exercise 6.17. Suppose
at time 0 all CPUs are functioning. Suppose each functioning CPU executes r in-
structions per unit time. Compute the expected total number of instructions executed
before the system crashes.

6.48 Consider the machine in Modeling Exercise 6.30. Suppose it costs $h dollar
to keep an item in the buffer for one unit of time. Each item sells for $r dollars.
Compute the long run net income (revenue-holding cost) per unit time, as a function
of k and K.

6.49 Consider the system in Computational Exercise 6.8. Suppose it costs the spe-
cial customer $c per unit time to wait for service, and $d to inspect the server. Com-
pute the total expected cost to the special customer until he enters as a function of T .
Suppose λ = 1/hr, µ = 2/hr, c = $10/hr and d = $20. Compute the value of T
that minimizes the expected total cost.

6.50 A small local car rental company has a fleet of K cars. We are interested in
deciding how many of them should be large cars and how many should be mid-size.
The demands for large cars occur according to a PP with rate λl per day, while those
for the mid-size cars occur according to a PP with rate λm . Any demand that can
not be met immediately is lost. The rental durations (in days) for the large (mid-size)
cars are iid exponential random variables with parameter µl (µm ). The rental net
revenue for the large (mid-size) cars is $rl ($rm ) per day. Compute g(k), the long
run average net rental revenue per day, if the fleet has k large cars and K − k mid-
size cars. Compute the optimal fleet mix if the following data is known: the fleet size
is fixed at 10, demand is 2 cars per day for large cars, 3 per day for mid-size cars,
the mean rental duration for large cars is 2 days, while that for the mid-size cars is 3
days. The large cars generate a net revenue of $60 per day, while that for the mid-size
cars is $40 per day.

6.51 Customers arrive at a single server service facility according to PP(λ) and re-
quest iid exp(µ) service times. Let Vi be the value of the service (in dollars) to the
i-th customer. Suppose {Vi , i ≥ 1} are iid U (0, 1) random variables. The customer
has to pay a price p to enter service. An arriving customer enters the system if and
only if the server is idle when he arrives and the service is worth more to him than the
service charge p. Otherwise, the customer leaves without service and is permanently
COMPUTATIONAL EXERCISES 271
lost. Find the optimal service charge p that will maximize the long run expected rev-
enue per unit time to the system. What fraction of the arriving customers join the
system when the optimal charge is used?

6.52 Consider the single server system of Computational Exercise 6.51. An arriving
customer enters the system if and only if the service is worth more to him than the
service charge p, even if the server is busy. The service provider incurs a holding cost
of $h per customer per hour of staying in the system. Find the optimal service charge
p that will maximize the long run expected revenue per unit time to the system.

6.53 Consider the system of Modeling Exercise 6.31. Class i customers pay $ci to
enter the system, c1 > c2 . Assuming the process is stable, compute the long run rate
at which the system collects revenue.

6.54 Let {Xi (t), t ≥ 0} (1 ≤ i ≤ K) be K independent reversible, irreducible,

positive recurrent CTMCs. The ith CTMC has state-space Si and generator matrix
Qi . Let X(t) = (X1 (t), X2 (t), · · · , XK (t)). Show that {X(t), t ≥ 0} is a reversible
CTMC.

6.55 Consider the following special case of the database locking model described
in Example 6.47: N = 3 items in the database, λ(1) = λ(2) = λ(3) = λ > 0,
λ(123) = θ > 0. All other λ(A)’s are zero. Assume µ(A) = µ for all A’s with
λ(A) > 0. Compute the steady state distribution of the state of the database.

6.56 Consider a warehouse that gets shipments from k different sources. Source i
sends items to the warehouse according to a PP(λi ). The items from source i need
Mi (a fixed positive integer) amount of space. The capacity of the warehouse is
B > max(M1 , M2 ,· · · , Mk ). If there is not enough space in the warehouse for an
incoming item, it is sent somewhere else. An item from source i stays in the ware-
house for an exp(µi ) amount of time before it is shipped off. The sojourn times are
independent. Model this system by a CTMC and show that it is reversible. What is
the probability that a warehouse has to decline an item from source i, in steady state?

6.57 A particle moves on an undirected connected network of N nodes as follows:

it stays at the ith node for an exp(qi ) amount of time and then moves to any one of
its immediate neighbors with equal probability. Let X(t) be the index of the node
occupied by the particle at time t. Show that {X(t), t ≥ 0} is a reversible CTMC,
and compute its limiting distribution.

6.58 Is the retrial queue of Example 6.15 reversible?

6.59 A system consists of N urns containing k balls. Each ball moves indepen-
dently among these urns according to a reversible CTMC with rate matrix Q.
Let Xi (t) be the number of balls in the ith urn at time t. Show that {X(t) =
(X1 (t), X2 (t), · · · , XN (t)), t ≥ 0} is a reversible CTMC.
272 CONTINUOUS-TIME MARKOV CHAINS
6.17 Conceptual Exercises

6.1 Let {X(t), t ≥ 0} be an irreducible CTMC on state-space S = {1, 2, ..., N }

with generator matrix Q. Let
T = min{t ≥ 0 : X(t) = N }.
Derive a set of simultaneous linear equations for
Mi,j = E{time spent in state j during [0, T )|X(0) = i}, i, j ∈ S.

6.2 Complete the proof of Theorem 6.16.

6.3 Let {X(t), t ≥ 0} be a CTMC on state-space S and generator matrix Q. Let T

be the first time the CTMC undergoes a transition from state i to j (here i and j are
fixed). Derive a method to compute E(T ).

6.4 Let {X(t), t ≥ 0} be a CTMC on state-space {0, 1, 2, ...}. Suppose the system
earns reward at rate ri per unit time when the system is in state i. Let Z be the total
reward earned by the system until it reaches state 0, and let g(i) = E(Z|X(0) = i).
Using first step analysis derive a set of equations satisfied by {g(i), i = 1, 2, 3, ...}.

6.5 Let {X(t), t ≥ 0} be an irreducible and positive recurrent CTMC with state-
space S and limiting distribution p = [pj , j ∈ S]. Suppose the initial distribution is
given by
P(X(0) = j) = pj , j ∈ S.
Show that
P(X(t) = j) = pj , j ∈ S, t ≥ 0.

6.6 Consider a CTMC with state-space S and generator matrix Q. Suppose the
CTMC earns reward at rate ri per unit time it spends in state i (i ∈ S). Further-
more it earns a lump sum reward rij whenever it undergoes a transition from state i
to state j. Let gα (i) be the total expected discounted (with continuous discount factor
α > 0) reward earned over an infinite horizon if the initial state is i. Show that the
vector gα = [gα (i), i ∈ S] satisfies
[αI − Q]gα = r,
where r = [r(i), i ∈ S] is given by
X
r(i) = ri + qij rij .
j6=i

Note that no lump sum reward is obtained at time 0.

6.7 Consider Conceptual Exercise 6.6 and assume that the CTMC is irreducible and
positive recurrent. Let g be the long run average reward per unit time. Show that
X
g= r(i)pi ,
i∈S
CONCEPTUAL EXERCISES 273
where pi is the limiting probability that the CTMC is in state i, and r(i) is as given
in Conceptual Exercise 6.6.

6.8 Consider the cost structure of Section 6.12. Let gi (t) be the expected total cost
incurred over (0, t] by the CTMC starting from state i at time 0. Let g(t) = [gi (t), i ∈
S]. Show that
g(t) = M (t)c,
where M (t) is the occupancy times matrix, and c is the cost rate vector.

6.9 In Conceptual Exercise 6.4, suppose r(i) > 0 for all 1 ≤ i < N . Construct a
new CTMC {Y (t), t ≥ 0} on {0, 1, 2, · · · , N } with the same initial distribution as
that of X(0), such that the reward Z equals the first passage time until the {Y (t), t ≥
0} process visits state N .

6.10 Let {X(t), t ≥ 0} be an irreducible positive recurrent CTMC on state-space

S, generator matrix Q, and limiting distribution p = [pj , j ∈ S]. Let Nij (t) be the
number of times the CTMC undergoes a transition from state i to j during (0, t].
Show that
E(Nij (t))
lim = pi qij .
t→∞ t
6.11 Consider a computer system modeled by a CTMC {X(t), t ≥ 0} on state-
space {1, 2, · · · , N } and generator matrix Q. Whenever the system is in state i it
executes ri > 0 instructions/time. A programming job requires x instructions to be
executed. Since the numbers involved are in the millions, we shall treat ri ’s and x as
real numbers. Suppose the job starts executing at time 0, with the system in state i.
Let T be the time when the job completes. Let
Z ∞
−sT ∗
φi (s, x) = E(e |X(0) = i), φi (s, w) = e−wx φi (s, x)dx.
0
Show that
N
ri X qij
φ∗i (s, w) = + φ∗ (s, w).
s + qi + ri w s + qi + ri w i
j=1,j6=i

6.12 Consider the computer system described in Conceptual Exercise 6.11, with the
following modification: whenever the system changes state, all the work done on the
job is lost, and the job starts from scratch in the new state. Now show that
N
X qij
φi (s, x) = e−(s+qi )x/ri + (1 − e−(s+qi )x/ri φj (s, x)).
s + qi
j=1,j6=i

6.13 Let Ti ∼ PH(αi , Mi ), i = 1, 2, be two independent phase type random vari-

ables. Show that T = T1 + T2 is a PH(α, M ) random variable with α and M as given
in Equations 6.82 and 6.83.
274 CONTINUOUS-TIME MARKOV CHAINS
6.14 Let Ti ∼ PH(αi , Mi ), i = 1, 2, be two independent phase type random vari-
ables. Let T be as defined in Equation 6.84. Show that T is a PH(α, M ) random
variable with α and M as given in Equations 6.85 and 6.86.

6.15 Let Ti ∼ PH(αi , Mi ), i = 1, 2, be two independent phase type random vari-

ables. Show that T = min(T1 , T2 ) is a phase type random variable. Find its parame-
ters.

6.16 Suppose a vector of probabilities p satisfies the local balance Equation 6.89.
Show that is satisfies the global balance Equation 6.64.

6.17 Show that an irreducible CTMC on {1, 2, · · · , N } with a symmetric generator

matrix is reversible.
 CHAPTER 7

Queueing Models

An operations research professor gets an offer from another university that gives him
better research opportunities, better students, and better pay, but it means uprooting
his family and moving to a new place. When he describes this quandary to his friend,
the friend says, “Well, why don’t you use the methodology of operations research?
Set up an optimization model with an objective function and constraints to maximize
your expected utility.” The professor objects, saying, “No, no! This is serious!”

7.1 Introduction

Queues are an unavoidable aspect of modern life. We do not like queues because
of the waiting involved. However, we like the fair service policies that a queueing
system imposes. Imagine what would happen if an amusement park did not enforce
a first-come first-served queueing discipline at its attractions!

Knowingly or unknowingly, we face queues every day. We stand in queues at gro-

cery store checkout counters, for movie tickets, at the baggage claim areas in airports.
Many times we may be in a queue without physically being there - as when we are
put on hold for the “next available service representative” when we call the customer
service number during peak times. Many times we do not even know that we are in
a queue: when we pick up the phone we are put in a queue to get a dial tone. Since
we generally get the dial tone within a fraction of a second, we do not realize that we
went through a queue. But we did, nonetheless.

Finally, waiting in queues is not a uniquely human fate. All kinds of systems en-
force queueing for all kinds of non-human commodities. For example, a modern
computer system manages queues of computer programs at its central processing
unit, its input/output devices, etc. A telephone system maintains a queue of calls
and serves them by assigning circuits to them. A digital communication network
transmits packets of data in a store-and-forward fashion, i.e., it maintains a queue of
packets at each node before transmitting them further towards their destinations. In
manufacturing setting queues are called inventories. Here the items are produced at

275
276 QUEUEING MODELS
a factory and stored in a warehouse, i.e., they form a queue in the warehouse. The
items are removed from the warehouse whenever a demand occurs.

Managing queue - whether human or non-human - properly is critical to a smooth

(and profitable) operation of any system. In a manufacturing system, excessive
queues (i.e., inventories) of manufactured products are expensive - one needs larger
warehouses to store them, enormous capital is idled in the inventory, there are costs
of deterioration, spoilage, etc. In a computer setting, building large queues of pro-
grams inevitably means slower response times. Slow response times from the central
computers can have a disastrous effect on modern banks, and stock exchanges, for
example. The modern communication networks can carry voice and video traffic ef-
ficiently only if the queue of data packets are managed so that delays do not exceed
a few milliseconds.
For human queues there is an additional cost to queueing - the psychological stress
generated by having to wait, the violent road rage being the most extreme manifesta-
tion of this. Industries that manage human queues (airlines, retail stores, amusement
parks, etc.) employ several methods to reduce the stress. For example, knowing be-
forehand that there is a half-hour wait in a queue for a ride in an amusement park
helps reduce the stress. Knowing the reason for delay also reduces stress - hence the
airline pilot’s announcement about being tenth in a queue for takeoff after the plane
leaves from the gate but sits on the tarmac for 20 minutes. Diverting the attention of
the waiting public can make the wait more bearable - this explains the presence of
tabloids and TVs near the grocery store checkout counters. Airports use an ingenious
method - there is generally a long walk from the gate to the baggage claim area. This
makes the total wait seem smaller. Finally, there is the famous story of the manager
of a skyscraper who put mirrors in the elevator lobbies and successfully reduced the
complaints about slow elevators!

Another aspect of queues mentioned earlier is the “fair service discipline.” In hu-
man queues this generally means first-come first-served (or first-in, first-out, or head
of the line). In non-human queues many other disciplines can be used. For example,
blood banks may manage their blood inventory by last-in first-out policy. This en-
sures better quality blood for most clients. The bank may discard blood after it stays
in the bank for longer than a certain period. Generally, queues (or inventories) of per-
ishable commodities use last-in first-out systems. Computers use many specialized
service disciplines. A common service discipline is called time sharing, under which
each job in the system gets ∆ milliseconds from the CPU in a round-robin fashion.
Thus all jobs get some service in reasonable intervals of time. In the limit, as ∆ → 0,
all jobs get served continuously in parallel, each job getting a fraction of the CPU
processing capacity. This limiting discipline is called processor sharing. As a last
example, the service discipline may be random: the server picks one of the waiting
customers at random for the next service. Such a discipline is common in statistical
experiments to avoid bias. It is also common to have priorities in service, although
we do not consider such systems in this chapter.
INTRODUCTION 277
A block diagram of a simple queueing system is shown in Figure 7.1. There are
several key aspects to describing a queueing system. We discuss them below.

Arrivals Departures

Queue

Figure 7.1 A typical queueing system.

1. The Arrival Process. The simplest model is when the customers arrive one at a
time and the times between two consecutive arrivals are iid non-negative random
variables. We use special symbols to denote these inter-arrival times as follows:
M for exponential (stands for memoryless or Markovian), Ek for Erlang with k
phases, D for deterministic, P H for phase type, G for general (sometimes we
use GI to emphasize independence). This list is by no means exhaustive, and
new notation keeps getting introduced as newer applications demand newer arrival
characteristics. For example, the applications in telecommunication systems use
what are known as the Markovian arrival processes, or Markov modulated Poisson
processes, denoted by MAP or MMPP.
2. The Service Times. The simplest model assumes that the service times are iid non-
negative random variables. We use the notation of the inter-arrival times for the
service times as well.
3. The Number of Servers. It is typically denoted by s (for servers) or c (for chan-
nels in telecommunication systems). It is typically assumed that all servers are
identical.
4. The Holding Capacity. This is the maximum number of customers that can be in
the system at any time. We also call it the system capacity, or just capacity. If
the capacity is K, an arriving customer who sees K customers in the system is
permanently lost. If no capacity is mentioned, it is assumed to be infinity.
5. The Service Discipline. This describes the sequence in which the waiting cus-
tomers are serviced. As described before, the possible disciplines are FCFS (first-
come first-served), LCFS (last-come first-served), random, PS (processor shar-
ing), etc.

We shall follow the symbolic representation introduced by G. Kendall to represent

a queueing system as inter-arrival time distribution/service time distribution/number
of servers/capacity/service discipline. Thus M/G/3/15/LCF S represents a queue-
ing system with Poisson arrivals, generally distributed service times, three servers,
a capacity to hold 15 customers, and last-come first-served service discipline. If the
capacity and the service discipline are not mentioned, we assume infinite capacity
and FCFS discipline. Thus an M/M/s queue has Poisson arrivals, exponential ser-
vice times, s servers, infinite waiting room and FCFS discipline.
278 QUEUEING MODELS
Several quantities are of interest in the study of queueing systems. We introduce
the relevant notation below:

X(t) = the number of customers in the system at time t,

Xn = the number of customers in the system just after the n-th customer departs,
Xn∗ = the number of customers in the system just before the n-th customer enters,
X̂n = the number of customers in the systems just before the n-th customer arrives,
Wn = time spent in the system by the n-th customer.

Note that we distinguish between an arriving customer and an entering customer,

since an arriving customer may not enter because the system is full, or the customer
decides not to join since the system is too congested. We are interested in

pj = lim P(X(t) = j),

t→∞
πj = lim P(Xn = j),
n→∞
πj∗ = lim P(Xn∗ = j),
n→∞

π̂j = lim P(X̂n = j),

n→∞
F (x) = lim P(Wn ≤ x),
n→∞
L = lim E(X(t)),
t→∞
W = lim E(Wn ).
n→∞

We shall also find it useful to study the customers in the queue (i.e., those who are
in the system but not in service). We define
X q (t) = the number of customers in the queue at time t,
Xnq = the number of customers in the queue just after the n-th customer departs,
Xn∗q = the number of customers in the queue just before the n-th customer enters,
X̂nq = the number of customers in the queue just before the n-th customer arrives,
Wnq = time spent in the queue by the n-th customer,
and
pqj = lim P(X q (t) = j),
t→∞
πjq = lim P(Xnq = j),
n→∞
πj∗q = lim P(Xn∗q = j),
n→∞

π̂jq = lim P(X̂nq = j),

n→∞
F q (x) = lim P(Wnq ≤ x),
n→∞
PROPERTIES OF GENERAL QUEUEING SYSTEMS 279
Lq = lim E(X q (t)),
t→∞
Wq = lim E(Wnq ).
n→∞

With this introduction we are ready to apply the theory of DTMCs and CTMCs to
queueing systems. In particular we shall study queueing systems in which {X(t), t ≥
0} or {Xn , n ≥ 0} or {Xn∗ , n ≥ 0} is a Markov chain. We shall also study queueing
systems modeled by multi-dimensional Markov chains. There is an extremely large
and growing literature on queueing theory and this chapter is by no means an exhaus-
tive treatment of even the Markovian queueing systems. Readers are encouraged to
refer to one of the several excellent books that are devoted entirely to queueing the-
ory.

It is obvious that {X(t), t ≥ 0}, {Xn , n ≥ 0}, {Xn∗ , n ≥ 0}, and {X̂n , n ≥ 0}
are related processes. The exact relationship between these processes is studied in
the next section.

7.2 Properties of General Queueing Systems

In this section we shall study several important properties of general queueing sys-
tems. They are discussed in Theorems 7.1, 7.2, 7.3, and 7.5.

In Theorem 7.1 we state the connection between the state of the system as seen
by a potential arrival and an entering customer. In Theorem 7.2 we show that under
mild conditions on the sample paths of {X(t), t ≥ 0}, the limiting distributions of
Xn and Xn∗ , if they exist, are identical. Thus, in steady state, the state of the system
as seen by an entering customer is identical to the one seen by a departing customer.
In Theorems 7.3 and 7.4, we prove that, if the arrival process is Poisson, and certain
other mild conditions are satisfied, the limiting distributions of X(t) and X̂n (if they
exist) are identical. Thus in steady state the state of the system as seen by an arriving
customer is the same as the state of the system at an arbitrary point of time. This
property is popularly known as PASTA – Poisson Arrivals See Time Averages. The
last result (Theorem 7.5) is called Little’s Law, and it relates limiting averages L and
W defined in the last sections. All these results are very useful in practice, but their
proofs are rather technical. We suggest that the reader should first read the statements
of the theorems, and understand their implications, before reading the proofs.

7.2.1 Relationship between πj∗ and π̂j

As pointed out in the last section, we make a distinction between the arriving cus-
tomers and the entering customers. One can think of the arriving customers as the
potential customers and the entering customers as the actual customers. The potential
customers become an actual customers when they decide to enter. The relationship
280 QUEUEING MODELS
between the state of the system as seen by an arrival (potential customer) and as seen
by an entering customer depends on the decision rule used by the arriving customer
to actually enter the system. To make this more precise, let In = 1 if the n-th arriving
customer enters, and 0 otherwise. Suppose the following limits exist
lim P(In = 1) = α, (7.1)
n→∞

and
lim P(In = 1|X̂n = j) = αj , j ≥ 0. (7.2)
n→∞
Note that α is the long run fraction of the arriving customers that enter the system.
The next theorem gives a relationship between the state of the system as seen by an
arrival and by an entering customer.

Theorem 7.1 Arriving and Entering Customers. Suppose α > 0, and one of the
two limiting distributions {πj∗ , j ≥ 0} and {π̂j , j ≥ 0} exist. Then the other limiting
distribution exists, and the two are related to each other by
αj
πj∗ = π̂j , j ≥ 0. (7.3)
α
Proof: Define
n
X
N (n) = Ii , n ≥ 1.
i=1
Thus N (n) be the number of customers who join the system from among the first n
arrivals. This implies that
∗
P(X̂n = j|In = 1) = P(XN (n) = j).

The assumption α > 0 implies that N (n) → ∞ as n → ∞ with probability 1. Now

P(X̂n = j) = P(X̂n = j, In = 1) + P(X̂n = j, In = 0)
∗
= P(XN (n) = j)P(In = 1) + P(In = 0|X̂n = j)P(X̂n = j).

Rearranging this yields

∗
P(In = 1|X̂n = j)P(X̂n = j) = P(XN (n) = j)P(In = 1).

Letting n → ∞ on both sides, and using Equations 7.31 and 7.2, we get
αj π̂j = απj∗
if either P(X̂n = j) or P(Xn∗ = j) has a limit as n → ∞. The theorem follows from
this.
Note that if we further know that
∞
X
πj∗ = 1
j=0

we can evaluate α as
∞
X
α= αi π̂i .
i=0
PROPERTIES OF GENERAL QUEUEING SYSTEMS 281
Hence Equation 7.3 can be written as
αj π̂j
πj∗ = P∞ , j ≥ 0.
i=0 αi π̂i
Finally, if every arriving customer enters, we have αj = 1 for all j ≥ 0 and α = 1.
Hence the above equation reduces to πj∗ = π̂j for all j ≥ 0, as expected.

Example 7.1 M/M/1/K System. Consider an M/M/1/K system. An arriving

customer enters the system if he finds less than K customers ahead of him, else he
leaves. Thus we have

1 if 0 ≤ j < K,
P(In = 1|X̂n = j) =
0 if j ≥ K.
From Equation 7.2 
1 if 0 ≤ j < K,
αj =
0 if j ≥ K.
Now suppose the limiting distribution {πj∗ , 0 ≤ j ≤ K − 1} exists. Then clearly we
must have
K−1
X
πj∗ = 1.
j=0

The limiting distribution {π̂j , 0 ≤ j ≤ K} exists, and we must have

K
X
π̂j = 1.
j=0

Hence we get
π̂j π̂j
πj∗ = PK−1 = , 0 ≤ j ≤ K − 1.
i=0 π̂i 1 − π̂K

7.2.2 Relationship between πj∗ and πj

The next theorem gives the relationship between the state of the system as seen by
an entering customer and a departing customer.

Theorem 7.2 Entering and Departing Customers. Suppose the customers enter
and depart a queueing system one at a time, and one of the two limiting distributions
{πj∗ , j ≥ 0} and {πj , j ≥ 0} exists. Then the other limiting distribution exists, and
πj = πj∗ , j ≥ 0. (7.4)

Proof: We follow the proof as given in Cooper [1981]. Suppose that there are i
customers in the system at time 0. We shall show that
∗
{Xn+i ≤ j} ⇔ {Xn+j+1 ≤ j}, j ≥ 0. (7.5)
282 QUEUEING MODELS
First suppose Xn+i = k ≤ j. This implies that there are exactly k + n entries before
the (n + i)th departure. Thus there can be only departures between the (n + i)th
departure and (n + k + 1)st entry. Hence
∗
Xn+k+1 ≤ k.
Using k = j we see that
∗
{Xn+i ≤ j} ⇒ {Xn+j+1 ≤ j}, j ≥ 0.
∗
To go the other way, suppose Xn+j+1 = k ≤ j. This implies that there are exactly
n + i + j − k departures before the (n + j + 1)st entry. Thus there no entries between
the (n + i + j − k)th departure and (n + j + 1)st entry. Hence
∗
Xn+i+j−k ≤ Xn+j+1 = k ≤ j, j ≥ 0.
Setting k = j we get Xn+i ≤ j. Hence
∗
{Xn+j+1 ≤ j} ⇒ {Xn+i ≤ j}, j ≥ 0.
This proves the equivalence in Equation 7.5. Hence we have
∗
P(Xn+i ≤ j) = P(Xn+j+1 ≤ j), j ≥ 0.
Letting n → ∞, and assuming one of the two limits exist, the theorem follows.
Two comments are in order at this point. First, the above theorem is a sample path
result and does not require any probabilistic assumptions. As long as the limits exist,
they are equal. Of course, we need probabilistic assumption to assert that the limits
exist. Second, the above theorem can be applied even in the presence of batch arrivals
and departures, as long as we sequence the entries (or departures) in the batch and
observe the system after every entry and every departure in the batch. Thus, suppose
n customers have entered so far, and a new batch of size 2 enters when there are i
customers in the system. Then we treat this as two single entries occurring one after
∗ ∗
another, thus yielding Xn+1 = i + 1 and Xn+2 = i + 2.

Example 7.2 Entering and Departing Customers. The {X(t), t ≥ 0} processes

in the queueing systems M/M/1, M/M/s, M/G/1, G/M/1, etc. satisfy the hy-
pothesis of Theorem 7.2. Since every arrival enters the system, we can combine the
result of Theorem 7.1 with that of Theorem 7.2, to get
πj = πj∗ = π̂j , j ≥ 0,
if any one of the three limiting distributions exist. At this point we do not know how
to prove that they exist.

7.2.3 Relationship between π̂j and pj

Now we discuss the relationship between the limiting distribution of the state of the
system as seen by an arrival and that of the state of the system at any time point. This
will lead us to an important property called PASTA – Poisson Arrivals See Time
PROPERTIES OF GENERAL QUEUEING SYSTEMS 283
Averages. Roughly speaking, we shall show that, the limiting probability that an ar-
riving customer sees the system in state j is the same as the limiting probability that
the system is in state j, if the arrival process is Poisson. Once can think of pj as the
time average (that is, the occupancy distribution): the long run fraction of the time
that the system spends in state j. Similarly, we can interpret π̂j as the long run frac-
tion of the arriving customers that see the system in state j. PASTA says that, if the
arrival process is Poisson, these two averages are identical.

We shall give a proof of this under the restrictive setting when the stochastic pro-
cess {X(t), t ≥ 0} describing the system state is a CTMC on {0, 1, 2, · · ·}. However,
PASTA is a very general result that applies even though the process is not Markovian.
For example, it applies to the queue length process in an M/G/1 queue, even if that
process is not a CTMC. However, its general proof is rather technical and will not be
presented here. We refer the reader to Wolff [1989] or Heyman and Sobel [1982] for
the general proof.

Let X(t) be the state of queueing system at time t. Let N (t) be the number of
customers who arrive at this system up to time t, and assume that {N (t), t ≥ 0} is a
PP(λ), and Sn is the time of arrival of the nth customer. Assume that {X(t), Sn ≤
t < Sn+1 } is a CTMC with state-space S and rate matrix G = [gij ], n ≥ 0. When the
nth arrival occurs at time Sn , it causes an instantaneous transition in the X process
from i to j with probability rij , regardless of the past history up to time t. That is,
P(X(Sn +) = j|X(Sn −) = i, (X(u), N (u)), 0 ≤ u < Sn ) = ri,j , i, j ≥ 0.
Now the nth arrival sees the system in state X̂n = X(Sn −). Hence we have, assum-
ing the limits exist,

π̂j = lim P(X̂n = j) = lim P(X(Sn −) = j), j ≥ 0 (7.6)

n→∞ n→∞
and
pj = lim P(X(t) = j), j ≥ 0. (7.7)
t→∞
The next theorem gives the main result.

Theorem 7.3 PASTA. If the limits in Equations 7.6 and 7.7 exist,
π̂j = pj , j ≥ 0.

The proof proceeds via several lemmas, and is completely algebraic. We provide the
following intuition to strengthen the feel for the result. Figure 7.2(a) shows a sample
path {X(t), t ≥ 0} of a three-state system interacting with a PP(λ) {N (t), t ≥ 0}.
In the figure, the Ti s are the intervals of time (open on the left and closed on the
right) when the system is in state 3. The events in the PP {N (t), t ≥ 0} that see the
system in state 3 are numbered consecutively from 1 to 12. Figure 7.2(b) shows the
Ti s spliced together, essentially deleting all the intervals of time when the system is
not in state 3. Thus we count the Poisson events that trigger the transitions out of
284 QUEUEING MODELS
X(t)

× ×× × × × × × ×× × × × × × × × × × × × × ×
1 2 3 4 56 7 8 9 10 11 12
T1 T2 T3 T4 T5 T6
(a)

123 456 78 9 10 11 12
××× ××× ×× × × × ×

T1 T2 T3 T4 T5 T6

(b)

Figure 7.2 (a) Sample path of a CTMC interacting with a PP. (b) Reduced sample path of a
CTMC interacting with a PP.

state 3 to states 1 and 2, but not those that trigger a transition into state 3 from state
1 and 2. We also count all the Poisson events that do not generate any transitions.

Now, the times between consecutive events in Figure 7.2(a) are iid exp(λ), due
to the model of the interaction that we have postulated. Hence the process of events
in Figure 7.2(b) is a PP(λ). Following the notation of Section 6.4, let Vj (t) be the
amount of time the system spends in state j over (0, t]. Hence the expected number
of Poisson events up to time t that see the system in state 3 is λE(V3 (t)). By the same
argument, the expected number of Poisson events up to time t that see the system in
state j is λE(Vj (t)). The expected number of events up to time t in a PP(λ) is λt.
Hence the fraction of the Poisson events that see the system in state j is

λE(Vj (t)) E(Vj (t))

= .
λt t

If the {X(t), t ≥ 0} process is a CTMC, the above fraction, as t → ∞ goes to pj ,

the long run fraction of the time the CTMC spends in state j. Hence, the limiting
probability (if the limits exist) the system is in state j just before an event in the PP,
PROPERTIES OF GENERAL QUEUEING SYSTEMS 285
is the same as the long run probability that the system is in state j. The hard part is
to prove that the limits exist.

We now continue with the proof of Theorem 7.3. First we study the structure of
the process {X(t), t ≥ 0}.

Lemma 7.1 {X(t), t ≥ 0} is a CTMC on S with rate matrix Q given by

Q = G + λ(R − I),
where R = [rij ], and I is the identity matrix.

Proof: That {X(t), t ≥ 0} is a CTMC is a consequence of how the system interacts

with the Poisson process, and the system behavior between the Poisson events. The
rate at which it moves to state j from state i 6= j is given by
qij = λrij + gij .
Hence we have
qii = gii − λ(1 − rii ).
The above two relations imply Equation 7.8.

The next lemma describes the probabilistic structure of the {X̂n , n ≥ 0} process.

Lemma 7.2 {X̂n , n ≥ 0} is a DTMC on S with transition probability matrix P =

[pij ] given by
P = RB (7.8)
where R = [rij ] and B = [bij ] where
bij = P(X(Sn+1 −) = j|X(Sn +) = i), i, j ∈ S.

Proof: That {X̂n , n ≥ 0} is a DTMC is clear. We have

pij = P(X̂n+1 = j|X̂n = i)
= P(X(Sn+1 −) = j|X(Sn −) = i)
X
= P(X(Sn+1 −) = j|X(Sn −) = i, X(Sn +) = k)
k∈S
·P(X(Sn +) = k|X(Sn −) = i)
X
= P(X(Sn +) = k|X(Sn −) = i)P(X(Sn+1 −) = j|X(Sn +) = k)
k∈S
X
= rik pkj ,
k∈S

which yields Equation 7.8 in matrix form.

The next lemma relates the B and G matrices in an algebraic fashion.

286 QUEUEING MODELS
Lemma 7.3 The matrix B satisfies
(λI − G)B = λI. (7.9)

Proof: Let {Y (t), t ≥ 0} be a CTMC with generator matrix G, and let

aij (t) = P(Y (t) = j|Y (0) = i).
Let a∗ij (s) be the Laplace transform of aij (·). From Equation 213 on page 6.25 we
see that A∗ (s) = [a∗ij (s)] satisfies
sA∗ (s) − I = GA∗ (s).
Since the {X(t), Sn ≤ t < Sn+1 } is a CTMC with generator G, and Sn+1 − Sn ∼
exp(λ), we get
bij = P(X(Sn+1 −) = j|X(Sn +) = i)
Z ∞
= λe−λt aij (t)dt
0
= λa∗ij (λ).
Thus
B = λA∗ (λ) = I + GA∗ (λ) = I + GB/λ,
which yields Equation 7.9.

With these three lemmas we can now give the proof.

The Proof of Theorem 7.3: Suppose {X(t), t ≥ 0} is an irreducible positive re-
current CTMC with limiting distribution p = [pj , j ∈ S]. Then, we have, from
Theorems 4.21 and 6.25, we get
X X
pQ = 0, pj = 1, π̂P = π̂, π̂j = 1.
This yields
0 = pQ
= pQB
= p(G + λ(R − I))B (from Lemma 7.1)
= p((G − λI)B + λRB)
= p(−λI + λP ) (from Lemma 7.3)
= −λ(p − pP ).
Hence we have X
p = pP, pj = 1.
Thus p is the stationary distribution of the DTMC {X̂n , n ≥ 0}. However, since
{X(t), t ≥ 0} is an irreducible CTMC, {X̂n , n ≥ 0} is irreducible and aperi-
odic. Hence it has a unique limiting distribution π̂. Hence, we must have p = π̂, as
desired.
PROPERTIES OF GENERAL QUEUEING SYSTEMS 287
We explain with three examples.

Example 7.3 M/M/1/1 Queue. Verify Theorem 7.3 directly for the M/M/1/1
queue.

Let X(t) be the number of customers at time t in an M/M/1/1 queue with

arrival rate λ and service rate µ. Let N (t) be the number of arrivals over (0, t].
Then {N (t), t ≥ 0} is a PP(λ). Let Sn be the nth arrival epoch. We see that
{X(t), Sn ≤ t < Sn+1 } is a CTMC with generator matrix
 
0 0
G= .
µ −µ
If the system is empty at an arrival instant, the arrival enters, else the arrival leaves.
Thus the R matrix is given by
 
0 1
R= .
0 1
From Lemma 7.1 we see that {X(t), t ≥ 0} is a CTMC with generator matrix
 
−λ λ
Q = G + λ(R − I) = .
µ −µ
This is as expected. Next we compute the B matrix. We have
b10 = P(X(Sn+1 −) = 0|X(Sn +) = 1)
= P(next departure occurs before next arrival)
µ
= .
λ+µ
Similar calculations yield
 
1 0
B= µ λ .
λ+µ λ+µ

Using Lemma 7.2 we see that {X̂n , n ≥ 0} is a DTMC on {0, 1} with transition
probability matrix " #
µ λ
λ+µ λ+µ
P = RB = µ λ .
λ+µ λ+µ
Hence its limiting distribution is given by
µ λ
π̂0 = , π̂1 = .
λ+µ λ+µ
Using the results of Example 6.34 on page 241 we see that the limiting distribution
of the CTMC {X(t), t ≥ 0} is given by
µ λ
p0 = , p1 = .
λ+µ λ+µ
Thus Theorem 7.3 is verified.
288 QUEUEING MODELS
Example 7.4 PASTA for M/M/1 and M/M/s Queues. Let X(t) be the number
of customers at time t in an M/M/1 queue with arrival rate λ and service rate µ > λ.
Let N (t) be the number of arrivals over (0, t]. Then {N (t), t ≥ 0} is a PP(λ). We
see that X and N processes satisfy the assumptions described in this section. Hence
we can apply PASTA (Theorem 7.3) to get
π̂j = pj , j ≥ 0.
Furthermore, the X process satisfies the conditions for Theorem 7.1. Thus
πj = πj∗ , j ≥ 0.
Finally, every arriving customer enters the system, hence
π̂j = πj∗ , j ≥ 0.
From Example 6.36 we see that this queue is stable, and using Equation 6.70 and the
above equations, we get
π̂j = πj∗ = πj = pj = (1 − ρ)ρj , j ≥ 0, (7.10)
where ρ = λ/µ < 1.

Similar analysis for the M/M/s queue shows that

π̂j = πj∗ = πj = pj , j ≥ 0.

Example 7.5 M/M/1/K System. Let X(t) be the number of customers at time
t in an M/M/1/K queue with arrival rate λ and service rate µ. Let N (t) be the
number of arrivals over (0, t]. An arriving customer enters if and only if the number
in the system is less than K when he arrives at the system. Then {N (t), t ≥ 0} is
a PP(λ). We see that X and N processes satisfy the assumptions described in this
section. Hence we can apply PASTA (Theorem 7.3) to get
π̂j = pj , 0 ≤ j ≤ K.
Furthermore, the X process satisfies the conditions for Theorem 7.1. Thus
πj = πj∗ , 0 ≤ j ≤ K − 1.
Finally, from Example 7.1 we get
π̂j
πj∗ = , 0 ≤ j ≤ K − 1.
1 − π̂K
Thus the arriving customers see the system in steady state, but not the entering cus-
tomers. This is because the arriving customers form a PP, but not the entering cus-
tomers. We shall compute the limiting distribution {pj , 0 ≤ j ≤ K} for this system
in Section 7.3.2.

We have proved PASTA in Theorem 7.3 for a {X(t), t ≥ 0} process that interacts
with a PP in a specific way, and behaves like a CTMC between the events of the PP.
PROPERTIES OF GENERAL QUEUEING SYSTEMS 289
However, PASTA is a very general property. We state the general result here, but omit
the proof. In almost all applications the version of PASTA given here is sufficient,
since almost all the processes we study in this book can be turned into CTMCs by
using phase-type distributions as we shall see in Section 7.6.

Let X(t) be the state of a system at time t, and let {N (t), t ≥ 0} be a PP that may
depend on the {X(t), t ≥ 0} process. However, we assume the following:

“Lack of Anticipation” Property: {N (t + s) − N (s), t ≥ 0} and {X(u), 0 ≤

u ≤ s} are independent.

Note that {N (t + s) − N (s), t ≥ 0} is independent of {N (u), 0 ≤ u ≤ s} due to

the independence of increments property of a PP. Thus the lack of anticipation prop-
erty says that the future arrivals after time s are independent of the system history up
to time s.

Now let B be a given set of states and let VB (t) be the time spent by the system
in the set B over (0, t], and AB (t) be the number of arrivals over (0, t] that see the
system in the set B,
N (t)
X
AB (t) = 1{X(Sn −)∈B} , t ≥ 0
n=1

where Sn is the time of the nth arrival.

Now suppose the sample paths of the {X(t), t ≥ 0} are almost surely piecewise
continuous and have finite number of jumps in finite intervals of time. Then the
following theorem gives the “almost sure” version of PASTA.

Theorem 7.4 PASTA. VBt(t) converges almost surely if and only if AB (t)
N (t) converges
almost surely, and both have the same limit.

Proof: See Wolff (1989), or Heyman and Sobel (1982).

Note that PASTA, which equates the limit of VBt(t) to that of ANB(t)
(t)
, holds when-
ever the convergence holds. We use the tools developed in this book to show that
VB (t)
lim = lim P(X(t) ∈ B),
t→∞ t t→∞

and
AB (t)
lim = lim P(X(Sn −) ∈ B).
t→∞ N (t) n→∞
290 QUEUEING MODELS
Thus according to PASTA, whenever the limits exist,
lim P(X(t) ∈ B) = lim P(X(Sn −) ∈ B),
t→∞ n→∞

or, when B = {j},

pj = π̂j .

Example 7.6 PASTA and the M/G/1 and G/M/1 Queues. Theorems 7.2 and
7.4 imply that for the M/G/1 queue
π̂j = πj∗ = πj = pj , j ≥ 0.
On the other hand, for the G/M/1 queue, we have
π̂j = πj∗ = πj , j ≥ 0.
However, PASTA does not apply unless the inter-arrival times are exponential. Thus
in general, for an G/M/1 queue, we do not have π̂j = pj .

7.2.4 Little’s Law

Recall the definitions of X(t), Wn , L, and W on Page 278. In this subsection we

study another important theorem of queueing theory: The Little Law. It states that
L, the average number of customers in the system, λ, the average arrival rate of
customers to the system, and W , the average time a customer spends in the system
(all in steady state), are related to each other by the following simple relation:
L = λW. (7.11)
The above relation is in fact a sample path property, rather than a probabilistic one.
We need probabilistic assumption to assert that the above averages exist. An intu-
itive explanation of Little’s Law is as follows: Suppose each arriving customer pays
the system $1 per unit time the customer spends in the system. The long run rate at
which the system earns revenue can be computed in two equivalent ways. First, the
nth customer pays the system $Wn , the time spent by that customer in the system.
Hence the average amount paid by a customer in steady state is $W . Since the av-
erage arrival rate is λ, the system earns $λW per unit time in the long run. Second,
since each customer in the system pays at rate $1 per unit time, the system earns
revenue at the instantaneous rate of $X(t) per unit time at time t. Hence the long run
rate of revenue is seen to be L, the steady state expected value of X(t). Since these
two calculations must provide the same answer, Equation 7.11 follows.

In the rest of the section we shall make the statement of Little’s law more precise.
Consider a general queueing system where the customers arrive randomly, get served,
and then depart. Let X(t) the number of customers in the system at time t, A(t) be
the number of arrivals over (0, t], and Wn be the time spent in the system by the
nth customer. We also refer to it as the waiting time or sojourn time. Now define the
PROPERTIES OF GENERAL QUEUEING SYSTEMS 291
following limits, whenever they exist. Note that these are defined for every sample
path of the queueing system.
1 t
Z
L = lim X(u)du, (7.12)
t→∞ t 0

A(t)
λ = lim , (7.13)
Pt n
t→∞

k=1 Wk
W = lim . (7.14)
n→∞ n
The next theorem states the relationship that binds the three limits defined above.

Theorem 7.5 Little’s Law. Suppose that for a fixed sample path the limits in Equa-
tions 7.13 and 7.14 exist and are finite. Then the limit in Equation 7.12 exists, and is
given by
L = λW.

Proof: Let S0 = 0, D0 = 0, and Sn be the arrival time and Dn ≥ Sn be the departure

time of the nth customer, n ≥ 1, and assume that 0 ≤ S1 ≤ S2 ≤ · · · . A(t) is
already defined to be the number of arrivals over (0, t]. Define D(t) to be the number
of departures over (0, t]. Without loss of generality we assume that X(0) = 0, since
we can always assume that X(0) customers arrived at time 0+. Then we have
Wn = Dn − Sn , n ≥ 1,
∞
X
X(t) = 1Sn ≤t<Dn , t ≥ 0,
n=1
A(t) = sup{n ≥ 0 : Sn ≤ t}, t ≥ 0,
D(t) = sup{n ≥ 0 : Dn ≤ t}, t ≥ 0,
X(t) = A(t) − D(t), t ≥ 0.
Now the existence of the limits implies that (we omit the details of the proof of this
assertion, and refer the readers to El-Taha and Stidham (1998))
X(t)
lim = 0. (7.15)
t→∞ t
This implies that
D(t) A(t) − X(t)
= → λ.
t t
The above relations also imply that
D(t) Z t A(t)
X X
Wn ≤ X(u)du ≤ Wn .
n=1 0 n=1

Dividing by t on both sides we get

D(t) A(t)
1X 1 t 1X
Z
Wn ≤ X(u)du ≤ Wn ,
t n=1 t 0 t n=1
292 QUEUEING MODELS
which can be written as
D(t) A(t)
D(t) 1 X 1 t A(t) 1 X
Z
Wn ≤ X(u)du ≤ Wn . (7.16)
t D(t) n=1 t 0 t A(t) n=1

Now, assume that A(t) → ∞ as t → ∞. Then D(t) → ∞ as t → ∞, and we have

D(t) A(t)
1 X 1 X
lim Wn = lim Wn = W.
t→∞ D(t) t→∞ A(t)
n=1 n=1

Now let t → ∞ in Equation 7.16. We get

λW ≤ L ≤ λW.
If A(t) remains bounded as t → ∞, we necessarily have L = λ = W = 0. The
theorem follows.

There are many results that prove Little’s Law under less restrictive conditions.
We refer the readers to Wolff (1989) and Heyman and Sobel (1982) and El-Taha
and Stidham (1998). The condition in Equation 7.15 usually holds since we concen-
trate on stable queueing systems, where the queue length has non-defective limiting
distribution.
Note that as long as the service discipline is independent of the service times,
the {X(t), t ≥ 0} process does not depend on the service discipline. Hence L is
also independent of the service discipline. On the other hand, the {Wn , n ≥ 0}
process does depend upon the service discipline. However, Little’s Law implies that
the average wait is independent of the service discipline, since the quantities L and
λ are independent of the service discipline.

Example 7.7 The M/G/∞ Queue. Consider the infinite server queue of Exam-
ple 5.17 on page 173. In the queueing nomenclature introduce in Section 7.1 this is
an M/G/∞ queue. Verify Little’s law for this system.

From Example 5.17, we see that in steady state the number of customers in this
system is a P(λτ ) random variable, where λ is the arrival rate of customers, and τ is
the mean service time. Thus
L = λτ.
Since the system has infinite number of servers, Wn , the time spent by the n customer
in the system equals his service time. Hence
W = E(Wn ) = τ.
Thus Equation 7.11 is satisfied.
BIRTH AND DEATH QUEUES 293
7.3 Birth and Death Queues

Many queueing systems where customers arrive one at a time, form a single queue,
and get served one at a time, can be modeled by birth and death processes. See
Example 6.10 on page 200 for the definition, and Example 6.35 on page 242 for the
limiting behavior. We shall use these results in the rest of this section.

7.3.1 M/M/1 Queue

Consider an M/M/1 queue. Such a queue has a single server, infinite capacity wait-
ing room, and the customers arrive according to a PP(λ) and request iid exp(µ)
service times. Let X(t) be the number of customers in the system at time t. We have
seen in Example 6.11 on page 200 that {X(t), t ≥ 0} is a birth and death process
with birth rates
λn = λ, n ≥ 0
and death rates
µn = µ, n ≥ 1.
We saw in Example 6.36 on page 242 that this queue is stable if
ρ = λ/µ < 1.
The parameter ρ is called the traffic intensity of the queue, and it can be interpreted
as the expected number of arrivals during one service time. The system serves one
customer during one service times, and gets ρ new customers during this time on the
average. Thus if ρ < 1, the system can serve more customers than it gets, so it should
be stable. The stability condition can also be written as λ < µ. In this form it says that
the rate at which customers arrive is less than the rate at which they can be served, and
hence the system should be stable. Note that the system is perfectly balanced when
λ = µ, but it is unstable, since it has no spare service capacity to handle random
variation in arrivals and services. Example 6.36 shows that the limiting distribution
of X(t) in a stable M/M/1 queue is given by
pj = (1 − ρ)ρj , j ≥ 0. (7.17)
From Example 7.4 we get
π̂j = πj∗ = πj = pj = (1 − ρ)ρj , j ≥ 0.
We have
∞
X ρ
L= jpj = . (7.18)
j=0
1−ρ
Thus as ρ → 1, L → ∞. This is a manifestation of increasing congestion as ρ → 1.

Next we shall compute F (·), the limiting distribution of Wn , assuming that the
294 QUEUEING MODELS
service discipline is FCFS. We have
F (x) = lim P(Wn ≤ x)
n→∞
∞
X
= lim P(Wn ≤ x|Xn∗ = j)P(Xn∗ = j)
n→∞
j=0
∞
X
= lim πj∗ P(Wn ≤ x|Xn∗ = j).
n→∞
j=0

From Equation 7.10 we see that

πj∗ = pj = (1 − ρ)ρj , j ≥ 0.
Now suppose an entering customer sees j customers ahead in the system. Due to
the FCFS discipline, these customers will be served before his service starts. The
remaining service time of the customer in service (if any) is an exp(µ) random vari-
able. Hence this customer’s time in the system is the sum of j + 1 iid exp(µ) random
variables. Thus
j
X (µx)r
P(Wn ≤ x|Xn∗ = j) = 1 − e−µx .
r=0
r!
Substituting, we get
∞ j
!
r
−µx (µx)
X X
j
F (x) = (1 − ρ)ρ 1− e ,
j=0 r=0
r!

which, after some algebra, reduces to

F (x) = 1 − e−(µ−λ)x , x ≥ 0.
Thus the steady state waiting time is an exp(µ − λ) random variable. Thus we have
1
W = E(waiting time in steady state) = .
µ−λ
Using equation 7.18 we see that L = λW . Thus Little’s Law is verified for the
M/M/1 queue.

7.3.2 M/M/1/K Queue

In an M/M/1/K system, customers arrive according to a PP(λ) and receive iid

exp(µ) service times from a single server. If an arriving customer finds K persons
in the system, he leaves immediately without service. Let X(t) be the number of
customers in the system at time t. One can show that {X(t), t ≥ 0} is a birth and
death process on {0, 1, 2, · · · , K} with birth rates
λn = λ, 0≤n<K
BIRTH AND DEATH QUEUES 295
and death rates
µn = µ, 1 ≤ n ≤ K.
Note that λK = 0 implies that the number in the system will not increase from K to
K + 1. We can use the results of Example 6.35 on page 242 to compute the limiting
distribution of the X(t). Substituting in Equation 6.68 we get
ρj = ρj , 0 ≤ j ≤ K,
where ρ = λ/µ is the traffic intensity. We have
K
(
1−ρK+1
X
1−ρ if ρ 6= 1,
ρj =
j=0
K + 1 if ρ = 1.

This is always finite, hence the queue is always stable. Substituting in Equation 6.69
we get  1−ρ j
pj = 1−ρK+1 ρ if ρ 6= 1,
(7.19)
1
K+1 if ρ = 1.
Finally, from Example 7.5 we have
π̂j = pj , 0 ≤ j ≤ K
and
pj
πj = πj∗ = , 0 ≤ j ≤ K − 1.
1 − pK
The mean number of customers in the system in steady state is given by
∞
1 − ρK
 
X ρ
L= jpj = − Kp K . (7.20)
j=0
1 − ρ 1 − ρK+1

7.3.3 M/M/s Queue

Consider an M/M/s queue. Such a queue has s identical servers, infinite capacity
waiting room, and the customers arrive according to a PP(λ) and request iid exp(µ)
service times. The customers form a single line and the customer at the head of the
line is served by the first available server. If more than one server is idle when a
customer arrives, he goes to any one of the available servers. Let X(t) be the number
of customers in the system at time t. One can show that {X(t), t ≥ 0} is a birth and
death process with birth rates
λn = λ, n ≥ 0
and death rates
µn = min(n, s)µ, n ≥ 0.
We can use the results of Example 6.35 on page 242 to compute the limiting distri-
bution of the X(t). Using the traffic intensity parameter
λ
ρ=
sµ
296 QUEUEING MODELS
in Equation 6.68 we get
smin(n,s) n
ρn = ρ , n ≥ 0.
min(n, s)!
We have
∞  Ps−1 ss ρs
X
n=0 ρn + s! 1−ρ if ρ < 1,
ρn =
n=0
∞ if ρ ≥ 1.
Hence the stability condition for the M/M/s queue is
ρ < 1.
This condition says that the queue is stable if the arrival rate λ is less than the maxi-
mum service rate sµ, which makes intuitive sense. From now on we assume that the
queue is stable. Using Equation 6.69 we get
∞
!−1 " s−1 #−1
X X ss ρs
p0 = ρn = ρn +
n=0 n=0
s! 1 − ρ
and
smin(n,s) n
p n = ρn p 0 = ρ p0 , n ≥ 1.
min(n, s)!
The limiting probability that all servers are busy is given by
∞
X ss ρ s ps
pn = p0 = .
n=s
s! 1 − ρ 1−ρ
The mean number of customers in the system can be shown to be
λ ρ
L= + ps . (7.21)
µ (1 − ρ)2

7.3.4 M/M/∞ Queue

The M/M/∞ queue is the limit of the M/M/s queue as s → ∞. It arises as a

model of self-service queues and was modeled in Example 6.12 as a birth and death
process with birth parameters
λn = λ, n ≥ 0
and death parameters
µn = nµ n ≥ 0.
This queue is stable as long as
λ
ρ= < ∞.
µ
Its limiting distribution is given in Example 6.37 as
ρj
pj = e−ρ , j ≥ 0.
j!
BIRTH AND DEATH QUEUES 297
As in the case of the M/M/1 queue, we have
ρj
π̂j = πj∗ = πj = pj = e−ρ , j ≥ 0.
j!
The transient analysis of the M/M/∞ queue was done in Example 5.17, from which
we see that if X(0) = 0, then X(t) is a Poisson random variable with mean ρ(1 −
e−µt ). From this we get
E(X(t)|X(0) = 0) = ρ(1 − e−µt ).
In comparison, the transient analysis of the M/M/1 and M/M/s queues is quite
messy. We refer the readers to one of the books on queueing theory for details: Saaty
(1961) or Gross and Harris (1985).

7.3.5 Queues with Finite Populations

In Example 6.6 on page 197 we modeled a workshop with two machines and one
repairperson. Here we consider a more general workshop with N machines and s
repairpersons. The life times of the machines are iid exp(µ) random variables, and
the repair times are iid exp(λ) random variables, and are independent of the life
times. The machines are as good as new after repairs. The machines are repaired in
the order in which they fail. Let X(t) be the number of working machines at time
t. One can show that {X(t), t ≥ 0} is a birth and death process on {0, 1, 2, · · · , N }
with birth parameters
λn = (N − n)λ, 0 ≤ n ≤ N − 1
and death parameters
µn = min(n, s)µ, 0 ≤ n ≤ N.
Since this is a finite state queue, it is always stable. One can compute the limiting
distribution using the results of Example 6.35 on page 242.

7.3.6 M/M/1 Queue with Balking and Reneging

Consider an M/M/1 queue of Section 7.3.1. Now suppose an arriving customer who
sees n customers in the system ahead of him joins the system with probability αn .
This is called the balking behavior. Once he joins the system, he displays reneging
behavior as follows: He has a patience time that is an exp(θ) random variable. If his
service does not begin before his patience time expires, he leaves the system without
getting served (i.e., he reneges); else he completes his service and then departs. All
customer patience times are independent of each other.

Let X(t) be the number of customers in the system at time t. We can show that
{X(t), t ≥ 0} is a birth and death process with birth rates
λn = αn λ, n≥0
298 QUEUEING MODELS
and death rates
µn = µ + (n − 1)θ, n ≥ 1.
(Why do we get (n − 1)θ and not nθ in the death parameters?) If θ > 0, such a queue
is always stable and its steady state distribution can be computed by using the results
of Example 6.35 on page 242.

The above examples illustrate the usefulness of the birth and death processes in
modeling queues. Further examples are given in Modeling Exercises 7.1 - 7.5.

7.4 Open Queueing Networks

The queueing models in Section 7.3 are for single-station queues, i.e., there is a sin-
gle place where the service is rendered. Customers arrive at this facility, which may
have more than one server, get served once, and then depart.

In this and the next section we consider more complicated queueing systems called
queueing networks. A typical queueing network consists of several service stations,
or nodes. Customers form queues at each of these queueing stations. After complet-
ing service at a station the customer may depart from the system or join a queue at
some other service station. In open queueing networks, customers arrive at the nodes
from outside the system, visit the nodes in some order, and then depart. Thus the
total number of customers in an open queueing network varies over time. In closed
queueing networks there are no external arrivals or departures, thus the total number
of customers in a closed queueing network remains constant. We shall study open
queueing networks in this section and closed queueing networks in the next.

Open queueing networks arise in all kinds of situations: hospitals, computers,

telecommunication networks, assembly lines, and supply chains, to name a few. Con-
sider a simple model of patient flow in a hospital. Patients arrive from outside at the
admitting office or the emergency ward. Patients from the admitting office go to vari-
ous clinics, which we have lumped into one node for modeling ease. In the clinics the
patients are diagnosed and routed to the intensive care unit, or are dismissed, or are
given reappointment for a follow-up visit. Patients with reappointments go home and
return to the admitting office at appropriate times. Patients from the emergency ward
are either dismissed after proper care, or are sent to the intensive care unit. From the
intensive care unit they are either discharged or given reappointments for follow-up
visits.

This simple patient flow model can be set up as a queueing network with five
nodes as shown in Figure 7.3. The arrows interconnecting the nodes show the patient
routing pattern. In this figure the customers can arrive at the system at two nodes: ad-
mitting and emergency. Customers can depart the system from three nodes: clinics,
 OPEN QUEUEING NETWORKS 299
Home
Returning
patients Reappointments

New Admis- Discharged

Clinics Reappointments
arrivals sions patients

Intensive Discharged
Care patients

Emergency Emerg- Discharged

arrivals ency patients

Figure 7.3 Schematic diagram of patient flow in a hospital.

emergency, and intensive care. Note that a customer can visit a node a random num-
ber of times before departing the system, i.e., a queueing network can have cycles.

It is extremely difficult to analyze a queueing network in all its generality. In this

section we shall concentrate on a special class of queueing networks called the Jack-
son networks. Jackson introduced this class in a seminal paper in 1957, and it has
become a standard queueing network model since then.

A queueing network is called a Jackson network if it satisfies the following as-

sumptions:

A1. It has N service stations (nodes).

A2. There are si servers at node i, 1 ≤ si ≤ ∞, 1 ≤ i ≤ N . Service times of
customers at node i are iid exp(µi ) random variables. They are independent of
service times of customers in other nodes.
A3. There is an infinite waiting room at each node.
A4. External arrivals at node i form a PP(λi ). All the arrival processes are independent
of each other and the service times.
A5. After completing service at node i, the customer departs the system with prob-
ability ri , or joins the queue at node j with probability rij , independent of the
number of customers at any node in the system. rii can be positive. We have
N
X
ri + rij = 1, 1 ≤ i ≤ N. (7.22)
j=1

A6. The routing matrix R = [rij ] is such that I − R is invertible.

All the above assumptions are crucial to the analysis of Jackson networks. Some as-
sumptions can be relaxed, but not the others. For example, in practice we would like
to consider finite capacity waiting rooms (assumption 3) or state-dependent routing
(assumption 5). However such networks are very difficult to analyze. On the other
hand certain types of state-dependent service and arrival rates can be handled fairly
300 QUEUEING MODELS
easily. See Subsections 7.4.1 and 7.4.2.

Now let us study a Jackson network described above. Let Xi (t) be the number of
customers at node i at time t, (1 ≤ i ≤ N, t ≥ 0), and let

X(t) = [X1 (t), X2 (t), · · · , XN (t)]

be the state of the queueing network at time t. The state-space of the {X(t), t ≥ 0}
process is S = {0, 1, 2, · · ·}N . To understand the transitions in this process we need
some notation.

Let ei be an N -vector with a 1 in the ith coordinate and 0 in all others. Suppose
the system is in state x = [x1 , x2 , · · · , xN ] ∈ S. If an external arrival takes place at
node i, the system state changes to x + ei . If a customer completes service at node
i (this can happen only if xi > 0) and departs the system, the system state changes
from x to x − ei , and if the customer moves to state j, the system state changes from
x to x − ei + ej . It can be seen that {X(t), t ≥ 0} is a multi-dimensional CTMC
with the following transition rates:
q(x, x + ei ) = λi ,
q(x, x − ei ) = min(xi , si )µi ri ,
q(x, x − ei + ej ) = min(xi , si )µi rij , i 6= j.
Hence we get
N
X N
X
q(x, x) = −q(x) = − λi − min(xi , si )µi (1 − rii ).
i=1 i=1

Now let

p(x) = lim P(X(t) = x)

t→∞
= lim P(X1 (t) = x1 , X2 (t) = x2 , · · · , XN (t) = xN )
t→∞

be the limiting distribution, assuming it exists.

We now study node j in isolation. The input to node j consists of two parts: the
external input that occurs at rate λj , and the internal input originating from other
nodes (including node j) in the network. Let aj be the total arrival rate (external +
internal) to node j. In steady state (assuming it exists) the departure rate from node i
must equal the total input rate ai to node i. A fraction rij of the departing customers
goes to node j. Thus the internal input from node i to node j is ai rij . Thus in steady
state we must have
XN
aj = λj + ai rij , 1 ≤ j ≤ N, (7.23)
i=1
OPEN QUEUEING NETWORKS 301
which can be written in matrix form as

a(I − R) = λ,

where a = [a1 , a2 , · · · , aN ] and λ = [λ1 , λ2 , · · · , λN ]. Now, from assumption 6,

I − R is invertible. Hence we have
a = λ(I − R)−1 . (7.24)

Note that the invertibility of I − R implies that no customer stays in the network
indefinitely.

Now consider an M/M/si queue with arrival rate ai and service rate µi . From the
results of Section 7.3.3 we see that such a queue is stable if ρi = ai /si µi < 1, and
φi (n), the steady state probability that there are n customers in the system is by
min(n,s )
i
si
φi (n) = ρn φi (0), n ≥ 0,
min(n, si )! i

where
"s −1 #−1
Xi
sni n ssi ρsi i
φi (0) = ρi + i .
n=0
n! si ! 1 − ρ i

With these preliminaries we are ready to state the main result about the Jackson
networks below.

Theorem 7.6 Open Jackson Networks. The CTMC {X(t), t ≥ 0} is positive re-
current if and only if
ai < si µi , 1 ≤ i ≤ N,

where a = [a1 , a2 , · · · , aN ] is given by Equation 7.24. When it is positive recurrent,

its steady state distribution is given by
N
Y
p(x) = φi (xi ). (7.25)
i=1

Proof: The balance equations for {X(t), t ≥ 0} are

N
X N
X
q(x)p(x) = λi p(x − ei ) + ri µi min(xi + 1, si )p(x + ei )
i=1 i=1
N X
X
+ rij µi min(xi + 1, si )p(x + ei − ej ), x ∈ S.
j=1 i:i6=j
302 QUEUEING MODELS
Assume that p(x) = 0 if x has any negative coordinates. Substitute p(x) of Equa-
tion 7.25 in the above equation to get
N N N
Y X Y φi (xi − 1)
q(x) φk (xk ) = λi φk (xk )
i=1
φi (xi )
k=1 k=1
N N
X Y φi (xi + 1)
+ ri µi min(xi + 1, si ) φk (xk )
i=1
φi (xi )
k=1
N X N
X Y φi (xi + 1) φj (xj − 1)
+ rij µi min(xi + 1, si ) φk (xk ) .
j=1 i:i6=j
φi (xi ) φj (xj )
k=1
QN
Canceling k=1 φk (xk ) from both sides and using the identity
φi (xi − 1) µi min(xi , si )
=
φi (xi ) ai
the above equality reduces to
N N
X µi min(xi , si ) X
q(x) = λi + ri ai
i=1
ai i=1
N X
X µj min(xj , sj )
+ (ai rij ) .
j=1 i:i6=j
aj

Using Equation 7.23 and simplifying the above equation we get

N
X N
X
λi = ri ai .
i=1 i=1

However, the above equation holds in steady state since the left hand side is the
rate at which the customers enter the network, and the right hand side is the rate at
which they depart the network. We can also derive this equation from Equations 7.23
and 7.22. Thus the solution in Equation 7.25 satisfies the balance equation. Since
φi (0) > 0 if and only if ai < si µi , the condition of positive recurrence follows.
Also, the CTMC is irreducible. Hence there is a unique limiting distribution. Hence
the theorem follows.

The form of the distribution in Equation 7.25 is called the product form, for the
obvious reason. Theorem 7.6 says that, in steady state, the queue lengths at the N
nodes are independent random variables. Furthermore, node i behaves as if it is an
M/M/si queue with arrival rate ai and service rate µi . The phrase “behaves as if”
is important, since the process {Xi (t), t ≥ 0} is not a birth and death process of an
M/M/si queue. For example, in general, the total arrival process to node i is not a
PP(ai ). It just so happens that the steady distribution of {Xi (t), t ≥ 0} is the same
as that of an M/M/si queue. We illustrate the result of Theorem 7.6 with a few
examples.
OPEN QUEUEING NETWORKS 303
Example 7.8 Single Queue with Feedback. The simplest queueing network is a
single station queue with feedback as shown in Figure 7.4. Customers arrive from

1–α
λ s
Servers α

Figure 7.4 An s-server queue with Bernoulli feedback.

outside at this service station according to a PPλ) and request iid exp(µ) services.
The service station has s identical servers. When a customer completes service, he
leaves the system with probability α. With probability 1 − α he rejoins the queue and
behaves as a new arrival.

This is a Jackson network with N = 1, s1 = s, µ1 = µ, λ1 = λ, r1 = α, r11 =

1 − α. From Equation 7.23, we get
a1 = λ + (1 − α)a1 ,
which yields
a1 = λ/α.
Thus the queue is stable if
λ
ρ= < 1.
αsµ
The steady state distribution of X(t) = X1 (t) is given by
smin(n,s) n
pn = lim P(X(t) = n) = ρ p0 , n ≥ 0,
t→∞ min(n, s)!
where " s−1 #−1
X sn ss ρ s
n
p0 = ρ + .
n=0
n! s! 1 − ρ

Example 7.9 Tandem Queue. Consider N single server queues in tandem as

shown in Figure 7.5. External customers arrive at node 1 according to a PP(λ). The
λ µ1 µ2 µN–1 µN

Figure 7.5 A tandem queueing network.

service times at the ith node are iid exp(µi ) random variables. Customers complet-
ing service at node i join the queue at node i + 1, 1 ≤ i ≤ N − 1. Customers leave
304 QUEUEING MODELS
the system after completing service at node N .

This is a Jackson network with N nodes and λ1 = λ, λi = 0, for 2 ≤ i ≤ N ,

ri,i+1 = 1 for 1 ≤ i ≤ N − 1, ri = 0 for 1 ≤ i ≤ N − 1, and rN = 1. The traffic
Equations 7.23 yield
ai = λ, 1 ≤ i ≤ N.
Hence the tandem queueing system is stable if λ < µi for all 1 ≤ i ≤ N , i.e.,
λ < min{µ1 , µ2 , · · · , µN }.
Thus the slowest server determines the traffic handling capacity of the tandem net-
work. Assuming stability, the limiting distribution is given by
N  x  
Y λ i λ
lim P(Xi (t) = xi , 1 ≤ i ≤ N ) = 1− .
t→∞
i=1
µi µi

Example 7.10 Patient Flow. Consider the queueing network model of patient flow
as shown in Figure 7.3. Suppose external patients arrive at the admitting ward at a
rate of 4 per hour and at the emergency ward at a rate of 1/hr. The admissions desk
is managed by one secretary who processes an admission in five minutes on the av-
erage. The clinic is served by k doctors, (here k is to be decided on), and the average
consultation with a doctor takes 15 minutes. Generally, one out of every four patients
going through the clinic is asked to return for another check up in two weeks (336
hours). One out of every ten is sent to the intensive care unit from the clinic. The rest
are dismissed after consultations. Patients arriving at the emergency room requires
on the average one hour of consultation time with a doctor. The emergency room is
staffed by m doctors, where m is to be decided on. One out of two patients in the
emergency ward goes home after treatment, whereas the other is admitted to the in-
tensive care unit. The average stay in the intensive care unit is four days, and there
are n beds available, where n is to decided on. From the intensive care unit, 20% of
the customers go home, and the other 80% are given reappointments for follow up in
two weeks. Analyze this system assuming that the assumptions of Jackson networks
are satisfied.

The parameters of the N = 5 node Jackson network (using time units of hours)
are
λ1 = 4, λ2 = 1, λ3 = 0, λ4 = 0, λ5 = 0,
s1 = 1, s2 = m, s3 = k, s4 = n, s5 = ∞,
µ1 = 12, µ2 = 1, µ3 = 4, µ4 = 1/96, µ5 = 1/336,
r1 = 0, r2 = 0.5, r3 = 0.65, r4 = 0.2, r5 = 0.
OPEN QUEUEING NETWORKS 305
The routing matrix is given by
 
0 0 1 0 0

 0 0 0 0.5 0  
R=
 0 0 0 0.1 0.25 
.
 0 0 0 0 0.80 
1 0 0 0 0
Equation 7.23 are given by
a1 = 4 + a5
a2 = 1
a3 = a1
a4 = .5a2 + .1a3
a5 = .25a3 + .8a4 .
Solving the above equations we get
a1 = 6.567, a2 = 1, a3 = 6.567, a4 = 1.157, a5 = 2.567.
We use Theorem 7.6 to establish the stability of the network. Note that a1 < s1 µ1
and a5 < s5 µ5 . We must also have
1 = a2 < s2 µ2 = m,
6.567 = a3 < s3 µ3 = 4k,
1.157 = a4 < s4 µ4 = n/96.
These are satisfied if we have
m > 1, k > 1.642, n > 111.043.
Thus the hospital must have at least two doctors in the emergency room, at least two
in the clinics, and have at least 112 beds in the intensive care unit. So let us assume
the hospital uses two doctors each in the emergency room and the clinics, and has
120 beds. With these parameters, the steady state analysis of the queueing network
can be done by treating (1) the admissions queue as an M/M/1 with arrival rate
6.567 per hour, and service rate of 12 per hour; (2) the emergency ward queue as an
M/M/2 with arrival rate 1 per hour, and service rate of 1 per hour; (3) the clinic
queue as an M/M/2 queue with arrival rate 6.567 per hour and service rate of 4 per
hour per server; (4) the intensive care queue as an M/M/120 with arrival rate 1.157
per hour, and service rate of 1/96 per hour; and (5) the home queue as an M/M/∞
with arrival rate 2.567 per hour, and service rate of 1/336 per hour. Furthermore these
five queues are independent of each other in steady state.

Next we discuss two important generalizations of the Jackson networks.

306 QUEUEING MODELS
7.4.1 State-Dependent Service

In the Jackson network model we had assumed that the service rate at node i when
there are n customers at that node is given by min(si , n)µi . We define Jackson net-
works with state-dependent service by replacing assumption A2 by A2’ as follows:
A2’. The service rate at node i when there are n customers at that node is given by
µi (n), with µi (0) = 0 and µi (n) > 0 for n ≥ 0, 1 ≤ i ≤ N .

Note that the service rate at node i is not allowed to depend on the state of node
j 6= i. Now define
n  
Y ai
φi (0) = 1, φi (n) = , n ≥ 1, 1 ≤ i ≤ N (7.26)
j=1
µi (j)

where ai is the total arrival rate to node i as given by Equation 7.24. Jackson networks
with state-dependent service also admit a product form solution as shown in the next
theorem.

Theorem 7.7 Jackson Networks with State-Dependent Service. A Jackson net-

work with state-dependent service is stable if and only if
∞
X
ci = φi (n) < ∞ for all 1 ≤ i ≤ N.
n=0
If the network is stable, the limiting state distribution is given by
N
Y φi (xi )
p(x) = , x ∈ S.
i=1
ci

Proof: Follows along the same lines as the proof of Theorem 7.6.

Thus, in steady-state, the queues at various nodes in a Jackson network with state-
dependent service are independent.

7.4.2 State-Dependent Arrivals and Service

It is also possible to further generalize the model of Subsection 7.4.1 by allowing the
instantaneous arrival rate to the network to depend on the total number of customers
in the network. Specifically, we replace assumption A4 by A4’ as follows:

A4’. External customers arrive at the network at rate λ(n) when the total number
of customers in the network is n. An arriving customer joins node i with probability
ui , where
XN
ui = 1.
i=1
OPEN QUEUEING NETWORKS 307
The above assumption implies that the instantaneous external arrival rate to node
i is ui λ(n) if the total number of customers in the network is n. To keep the
{X(t), t ≥ 0} process irreducible, we assume that there is a K ≤ ∞ such that
λ(n) > 0 for 0 ≤ n < K, and λ(n) = 0 for n ≥ K. We call the Jackson networks
with assumptions A2 and A5 replaced by A2’ and A5’ “Jackson networks with state-
dependent arrivals and service.”

We shall see that such Jackson networks with state-dependent arrival and service
rates continue to have a kind of product form limiting distribution. However, the
queue at various nodes are not independent any more. The results are given in the
next theorem. First, we need the following notation.

Let {ai , 1 ≤ i ≤ N } be the unique solution to

N
X
aj = u j + ai pij , 1 ≤ j ≤ N.
i=1

Let φi (n) be as defined in Equation 7.26 using the above {ai , 1 ≤ i ≤ N }. Also, for
x = [x1 , x2 , · · · , xN ] ∈ S, let
N
X
|x| = xi .
i=1

Thus if the state of the network is X(t), the total number of customers in it at time t
is |X(t)|.

Theorem 7.8 Jackson Networks with State-Dependent Arrivals and Service.

The limiting state distribution in a Jackson network with state-dependent arrivals
and service is given by
N |x|
Y Y
p(x) = c · φi (xi ) · λ(j), x ∈ S,
i=1 j=1

where c is the normalizing constant given by

 −1
N |x|
XY Y
c= φi (xi ) · λ(j) .
x∈S i=1 j=1

The network is stable if and only if c > 0.

Proof: Follows along the same lines as that of Theorem 7.6.

Computation of the constant c is the hard part. There is a large literature on “prod-
uct form” queueing networks, and it is more or less completely understood now as
what enables a network to have “product form” solution. See Kelly (1979) and Wal-
rand (1988).
308 QUEUEING MODELS
7.5 Closed Queueing Networks

In this section we consider the closed queueing networks. In these networks there
are no external arrivals to the network, and there are no departures from the network.
Thus the total number of customers in the network is constant. Closed queueing net-
works have been used to study population dynamics, multi-programmed computer
systems, telecommunication networks with window flow control, etc. We start with
the definition.

A queueing network is called a closed Jackson network if it satisfies the following

assumptions:

B1. It has N service stations (nodes) and a total of K customers.

B2. The service rate at node i when there are n customers at that node is given by
µi (n), with µi (0) = 0 and µi (n) > 0 for 1 ≤ n ≤ K, 1 ≤ i ≤ N .
B3. After completing service at node i, the customer joins the queue at node j with
probability rij , independent of the number of customers at any node in the
system. rii can be positive.
B4. The routing matrix R = [rij ] is a transition probability matrix of an irreducible
DTMC.

Now let us study a closed Jackson network described above. Let Xi (t) be the
number of customers at node i at time t, (1 ≤ i ≤ N, t ≥ 0), and let
X(t) = [X1 (t), X2 (t), · · · , XN (t)]
be the state of the queueing network at time t. As in the case of open Jackson net-
works, we see that {X(t), t ≥ 0} is a CTMC on state-space
N
X
S = {x = [x1 , x2 , · · · , xN ] : xi ≥ 0, xi = K}
i=1

with transition rates given by

q(x, x − ei + ej ) = µi (xi )rij , i 6= j, x ∈ S.
Hence we get
N
X
q(x, x) = −q(x) = − µi (xi )(1 − rii ), x ∈ S.
i=1

Since the CTMC has finite state-space and is irreducible, it is positive recurrent. Let
p(x) = lim P(X(t) = x)
t→∞
= lim P(X1 (t) = x1 , X2 (t) = x2 , · · · , XN (t) = xN )
t→∞

be the limiting distribution. We need the following notation before we give the result
CLOSED QUEUEING NETWORKS 309
about p(x). Let π = [π1 , π2 , · · · , πN ] be the limiting distribution of the DTMC with
transition matrix R. Since R is assumed to be irreducible, π is the unique solution to
N
X
π = πR, πi = 1. (7.27)
i=1

Next, define
n  
Y πi
φi (0) = 1, φi (n) = , 1 ≤ n ≤ K, 1 ≤ i ≤ N. (7.28)
j=1
µi (j)

Theorem 7.9 Closed Jackson Networks. The limiting distribution of the CTMC
{X(t), t ≥ 0} is given by
N
Y
p(x) = GN (K) φi (xi ), (7.29)
i=1

where the normalizing constant GN (K) is chosen so that

X
p(x) = 1.
x∈S

Proof: Follows by verifying that the solution in Equation 7.29 satisfies the balance
equation X
q(x)p(x) = p(y)q(y, x).
y∈s:y6=x

The verification proceeds along the same lines as that in the proof of Theorem 7.6.

Thus the closed Jackson network has a “product form” limiting distribution. The
hard part is the evaluation of G(K), the normalizing constant. The computation is
difficult
since the size of the state-space grows exponentially in N and K: it has
N +K−1
K elements. A recursive method of computing G(K) for closed Jackson net-
works of single-server queues is described in the next example.

Example 7.11 Tandem Closed Network. Consider a closed Jackson network of N

single server nodes as shown in Figure 7.6. The service rate at node i is µi (n) = µi

µ1 µ2 µN–1 µN

Figure 7.6 A tandem closed network.

for n ≥ 1. The routing probabilities are ri,i+1 = 1 for 1 ≤ i ≤ N − 1, and rN,1 = 1.

Thus the solution to Equation 7.27 is given by
1
πi = , 1 ≤ i ≤ N.
N
310 QUEUEING MODELS
We have
φi (n) = ρni , n ≥ 0,
where
πi 1
ρi = = .
µi N µi
From Theorem 7.9 we get
N
Y
p(x) = GN (K) ρxi i , x ∈ S.
i=1

We leave it to the readers to verify that the generating function of GN (K) is given
by
∞ N
X Y 1
G̃N (z) = GN (K)z K = .
i=1
1 − ρi z
K=0

Now we see that G̃0 (z) = 1 and

G̃N (z)(1 − ρN z) = G̃N −1 (z), N ≥ 1,
which can be written as
G̃N (z) = ρN z G̃N (z) + G̃N −1 (z).
From this we can derive the following recursion
GN (K) = GN −1 (K) + ρN GN (K − 1),
with boundary conditions
G0 (0) = 1, GN (0) = 1, N ≥ 1, G0 (K) = 0, K ≥ 1.
We can use this recursion to compute GN (K) in O(N K) steps.

Example 7.12 Multi-Programming Systems. Consider the following model of a

multi-programming computer system. It consists of a central processing unit (CPU)
(node 1) a printer (node 2), and a disk drive (node 3). A program starts in the CPU.
When the computing part is done, it goes to the printer with probability α or the disc
drive with probability 1 − α. From the printer the program terminates with probabil-
ity β or goes back to the CPU for further computing with probability 1 − β. After
completing the operation at the disc drive the program returns to the CPU with prob-
ability 1. Suppose the time required at the CPU phase is exp(µ1 ), the printer stage
is exp(µ2 ), and the disc drive stage is exp(µ3 ). Suppose these times are indepen-
dent. When a program departs the system from the printer queue, a new program is
instantaneously admitted to the CPU queue, so that the total number of programs in
the system remains constant, say K. The parameter K is called the degree of multi-
programming.

This system can be modeled by a closed Jackson network as shown in Figure 7.7.
The parameters of this network are
CLOSED QUEUEING NETWORKS 311

Printer

CPU

1–α

Disc
Drive

Figure 7.7 Multi-programming system.

µi (n) = µi , i = 1, 2, 3; n ≥ 1,
 
0 α 1−α
R= 1 0 0 .
1 0 0
Thus the solution to Equation 7.27 is given by
π1 = 0.5, π2 = 0.5α, π3 = 0.5(1 − α).
Using
1 α 1−α
ρ1 = , ρ2 = , ρ3 = ,
2µ1 2µ2 2µ3
we get
φi (n) = ρni .
Thus
p(x1 , x2 , x3 ) = G3 (K)ρx1 1 ρx2 2 ρx3 3 , x1 + x2 + x3 = K.
The constant G3 (K) can be computed by using the method of Example 7.11.
The throughput of the system is defined as the rate at which jobs get completed in
steady state. In our system, if the system is in state (x1 , x2 , x3 ) with x2 > 0, jobs get
completed at rate µ2 β. Hence we have
X
throughput = µ2 β p(x1 , x2 , x3 ).
x∈S:x2 >0

The closed queueing systems have been found to be highly useful models for com-
puting systems and there is a large literature in this area. See Gelenbe and Pujolle
(1987) and Saur and Chandy (1981).
312 QUEUEING MODELS
7.6 Single Server Queues

So far we have studied queueing systems that are described by CTMCs. In this sec-
tion we study single server queues where either the service times or the interarrival
times are non-exponential, making the queue length process non-Markovian.

7.6.1 M/G/1 Queue

We study an M/G/1 queue where customers arrive according to a PP(λ) and form
a single queue in an infinite waiting room in front of a single server and demand iid
service times with common cdf G(·), with mean τ and variance σ 2 . Let X(t) be the
number of customers in the system at time t. The stochastic process {X(t), t ≥ 0}
is a CTMC if and only if the service times are exponential random variables. Thus in
general we cannot use the theory of CTMCs to study
pj = lim P(X(t) = j), j ≥ 0,
t→∞

in an M/G/1 queue.

Recall the definitions of Xn , Xn∗ , X̂n , πj , πj∗ , and π̂j from Section 7.1. Since
every arriving customer joins the system, the {X(t), t ≥ 0} process jumps up and
down by one at a time, and the arrival process is Poisson, we can use Theorems 7.3,
7.2 and 7.4 to get
π̂j = πj = πj∗ = pj , j ≥ 0.
Thus we can compute the limiting distribution of {X(t), t ≥ 0} by studying the
limiting distribution of {Xn , n ≥ 0}. This is possible to do, since the next theorem
shows that {Xn , n ≥ 0} is a DTMC.

Theorem 7.10 Embedded DTMC in an M/G/1 Queue. {Xn , n ≥ 0} is an irre-

ducible and aperiodic DTMC on S = {0, 1, 2, · · ·} and one-step transition probabil-
ity matrix  
α0 α1 α2 α3 · · ·
 α0 α1 α2 α3 · · · 
 
 0 α0 α1 α2 · · · 
P = 0 , (7.30)
 
 0 α0 α1 · · ·  
 0 0 0 α 0 · · · 
.. .. .. ..
 
..
. . . . .
where
∞
(λt)i
Z
αi = e−λt dG(t), i ≥ 0. (7.31)
0 i!

Proof: Let An be the number of arrivals to the queueing system during the nth ser-
vice time. Since the service times are iid random variables with common distribution
SINGLE SERVER QUEUES 313
G(·), and the arrival process is PP(λ), we see that {An , n ≥ 1} is a sequence of iid
random variables with common pmf
P(An = i) = P(i arrivals during a service time)
Z ∞
= P(i arrivals during a service time of duration t)dG(t)
0
Z ∞
(λt)i
= e−λt dG(t)
0 i!
= αi .
Now, if Xn > 0, the (n+ 1)st service time starts immediately after the nth departure,
and during that service time An+1 customers join the system. Hence after the (n +
1)st departure there are Xn + An+1 − 1 customers are left in the system. On the other
hand, if Xn = 0, the (n + 1)st service time starts immediately after the (n + 1)st
arrival, and during that service time An+1 customers join the system. Hence after the
(n + 1)st departure there are An+1 customers are left in the system. Combining these
two observations, we get

An+1 if Xn = 0,
Xn+1 = (7.32)
Xn − 1 + An+1 if Xn > 0.
This is identical to Equation 2.9 derived in Example 2.16 on page 19 if we define
Yn = An+1 . The result then follows from the results in Example 2.16. The DTMC
is irreducible and aperiodic since αi > 0 for all i ≥ 0.

The next theorem gives the result about the limiting distribution of {Xn , n ≥ 0}.

Theorem 7.11 Limiting Distribution of an M/G/1 Queue. The DTMC {Xn , n ≥

0} is positive recurrent if and only if
ρ = λτ < 1.
If it is positive recurrent, its limiting distribution has the generating function given
by
∞
X (1 − z)G̃(λ − λz)
φ(z) = πj z j = (1 − ρ) , (7.33)
j=0
G̃(λ − λz) − z
where Z ∞
G̃(s) = e−st dG(t).
0

Proof: Since {Xn , n ≥ 0} is the DTMC studied in Example 2.16 on page 19, we
can use the results about its limiting distribution from Example 4.23 on page 120.
From there we see that the DTMC is positive recurrent if and only if
∞
X
kαk < 1.
k=0
314 QUEUEING MODELS
Substituting from Equation 7.31 we get
∞ ∞ ∞
(λt)k
X X Z
kαk = k e−λt dG(t)
0 k!
k=0 k=0
∞
!
∞
(λt)k
Z X
= e−λt k dG(t)
0 k!
k=0
Z ∞
= λtdG(t) = λτ = ρ.
0
Thus the DTMC is positive recurrent if and only if ρ < 1. From Equation 4.44 (using
ρ in place of µ) we get
ψ(z)(1 − z)
φ(z) = (1 − ρ) , (7.34)
ψ(z) − z
where
∞
X
ψ(z) = αk z k .
k=0
Substituting for αk from Equation 7.31 in the above equation
∞ Z ∞
X (λt)k
ψ(z) = zk e−λt dG(t)
0 k!
k=0
∞
Z ∞ !
k
k (λt)
X
−λt
= e z dG(t)
0 k!
k=0
Z ∞
= e−λt eλzt dG(t)
Z0 ∞
= e−λ(1−z)t dG(t) = G̃(λ − λz).
0

Substituting in Equation 7.34 we get Equation 7.33. This proves the theorem.

One immediate consequence of Equation 7.33 is that the probability that the server
is idle in steady state can be computed as
p0 = π0 = φ(0) = 1 − ρ. (7.35)
Also, since pj = πj for all j ≥ 0, φ(z) in Equation 7.33 is also the generating func-
tion of the limiting distribution of the {X(t), t ≥ 0} process. Using Equation 7.33
we can compute the expected number of customers in the system in steady state as
given in the following theorem.

Theorem 7.12 Expected Number in an M/G/1 Queue. The expected number in

steady state in a stable M/G/1 queue is given by

ρ2 σ2
 
1
L=ρ+ · 1+ 2 , (7.36)
2 1−ρ τ
SINGLE SERVER QUEUES 315
where τ and σ 2 are the mean and variance of the service time.

Proof: We have
L = lim E(X(t))
t→∞
X∞
= jpj
j=0
X∞
= jπj
j=0
dφ(z)
= |z=1 .
dz
The theorem follows after evaluating the last expression in straight forward fashion.
This involves using L’Hopital’s rule twice.

The Equation 7.36 is called the Pollaczek-Khintchine formula. It is interesting to

note that the first moment of the queue length depends on the second moment (or
equivalently, the variance) of the service time. This has an important implication.
We can decrease the queue length by making the server more consistent, that is, by
reducing the variability of the service times. Since the variance is zero for constant
service times, it follows that among all service times with the same mean, the de-
terministic service time will minimize the expected number in the system in steady
state!

Example 7.13 The M/M/1 Queue. If the service time distribution is

G(x) = 1 − e−µx , x ≥ 0
the M/G/1 queue reduces to M/M/1 queue with τ = 1/µ and ρ = λτ = λ/µ. In
this case we get
µ
G̃(s) = .
s+µ
Substituting in Equation 7.33 we get
(1 − z)G̃(λ − λz)
φ(z) = (1 − ρ)
G̃(λ − λz) − z
(1 − z)µ/(λ + µ − λz)
= (1 − ρ)
µ/(λ + µ − λz) − z
(1 − z)µ
= (1 − ρ)
(1 − z)(µ − λz)
1−ρ
= .
1 − ρz
316 QUEUEING MODELS
By expanding the last expression in a power series in z we get
∞
X ∞
X ∞
X
φ(z) = πj z j = pj z j = (1 − ρ) ρj z j .
j=0 j=0 j=0

Hence we get
pj = (1 − ρ)ρj , j ≥ 0.
This matches with the result in Example 6.36 on page 242, as expected.

Example 7.14 The M/Ek /1 Queue. Suppose the service times are iid Erl(k, µ).
Then the M/G/1 queue reduces to M/Ek /1 queue. In this case we get
k k
τ= , σ2 = 2 .
µ µ
The queue is stable if
kλ
ρ = λτ = < 1.
µ
Assuming the queue is stable, the expected number in steady state can be computed
by using Equation 7.36 as
1 ρ2 k + 1
L=ρ+ · .
2 1−ρ k

A large number of variations of the M/G/1 queue have been studied in literature.
See Modeling Exercises 7.11 and 7.13.

Next we study the waiting times (this includes time in service) in an M/G/1 queue
assuming FCFS service discipline. Let Fn (·) be the cdf of Wn , the waiting time of
the nth customer. Let
F̃n (s) = E(e−sWn ).
The next theorem gives the Laplace Stieltjes transform (LST) of the waiting time in
steady state, defined as
F̃ (s) = lim F̃n (s).
n→∞

Theorem 7.13 Waiting Times in an M/G/1 Queue. The LST of the waiting time
in steady state in a stable M/G/1 queue with FCFS service discipline is given by
sG̃(s)
F̃ (s) = (1 − ρ) . (7.37)
s − λ(1 − G̃(s))

Proof: Let An be the number of arrivals during the n customer’s waiting time in the
system. Since Xn is the number of customers left in the system after the nth depar-
ture, the assumption of FCFS service discipline implies that Xn = An . The Poisson
assumption implies that (see the derivation of ψ(z) in the proof of Theorem 7.11)
E(z An ) = F̃n (λ − λz).
SINGLE SERVER QUEUES 317
Hence
φ(z) = lim E(z Xn ) = lim E(z An ) = lim F̃n (λ − λz) = F̃ (λ − λz).
n→∞ n→∞ n→∞

Substituting λ − λz = s we get Equation 7.37.

Equation 7.37 is also known as the Pollaczec-Khintchine formula. Using the

derivatives of F̃ (s) at s = 0 we get
λS 2
W =τ+ ,
2(1 − ρ)
where S 2 = σ 2 + τ 2 is the second moment of the service time. Using Equation 7.36
we can verify directly that Little’s Law L = λW holds for the M/G/1 queue.

7.6.2 G/M/1 Queue

Now we study a G/M/1 queue where customers arrive one at a time and the inter-
arrival times are iid random variables with common cdf G(·), with G(0) = 0 and
mean 1/λ. The arriving customers form a single queue in an infinite waiting room in
front of a single server and demand iid exp(µ) service times. Let X(t) be the num-
ber of customers in the system at time t. The stochastic process {X(t), t ≥ 0} is a
CTMC if and only if the interarrival times are exponential random variables. Thus
in general we cannot use the theory of CTMCs to study the limiting behavior of X(t).

Recall the definitions of Xn , Xn∗ , X̂n , πj , πj∗ , and π̂j from Section 7.1. Since
every arriving customer joins the system, and the {X(t), t ≥ 0} process jumps up
and down by one at a time, we can use Theorems 7.3 and 7.2 to get
π̂j = πj = πj∗ , j ≥ 0.
However, unless the interarrival times are exponential, the arrival process is not a PP,
and hence π̂j 6= pj . The next theorem shows that {Xn∗ , n ≥ 0} is a DTMC.

Theorem 7.14 Embedded DTMC in a G/M/1 Queue. {Xn∗ , n ≥ 0} is an irre-

ducible and aperiodic DTMC on S = {0, 1, 2, · · ·} with one-step transition proba-
bility matrix  
β0 α0 0 0 0 ···
 β1 α1 α0 0 0 ··· 
 
 β2 α2 α1 α0 0 · · · 
P = , (7.38)
 β3 α3 α2 α1 α0 · · · 
.. .. .. .. ..
 
..
. . . . . .
where
∞
(µt)i
Z
αi = e−µt dG(t), i ≥ 0, (7.39)
0 i!
318 QUEUEING MODELS
and
∞
X
βi = aj , i ≥ 0.
j=i+1

Proof: Let Dn be the number of departures that can occur (assuming there are
enough customers in the system) during the nth interarrival time. Since the interar-
rival times are iid random variables with common distribution G(·), and the service
times are iid exp(µ), we see that {Dn , n ≥ 1} is a sequence of iid random variables
with common pmf
P(Dn = i) = P(i possible departures during an interarrival time)
Z ∞
(µt)i
= e−µt dG(t)
0 i!
= αi .
Now, the nth arrival sees Xn∗ customers in the system. Hence there are Xn∗ + 1
customers in the system after the nth customers enters. If Dn+1 < Xn∗ + 1, the
(n + 1) arrival will see Xn∗ + 1 − Dn+1 customers in the system, else there will be
no customers in the system when the next arrival occurs. Hence we get
∗
Xn+1 = max{Xn∗ + 1 − Dn+1 , 0}.
This is identical to Equation 2.11 derived in Example 2.17 on page 20 if we define
Yn = Dn+1 . The result then follows from the results in Example 2.17. The DTMC
is irreducible and aperiodic since αi > 0 for all i ≥ 0.

The next theorem gives the result about the limiting distribution of {Xn∗ , n ≥ 0}.

Theorem 7.15 G/M/1 Queue at Arrival Times. The DTMC {Xn∗ , n ≥ 0} is

positive recurrent if and only if
ρ = λ/µ < 1.
If it is positive recurrent, its limiting distribution is given by
πj∗ = lim P(Xn∗ = j) = (1 − α)αj , j ≥ 0 (7.40)
n→∞

where α is the unique solution in (0, 1) to

Z ∞
α= e−µ(1−α)t dG(t) = G̃(µ(1 − α)). (7.41)
0

Proof: Since {Xn∗ , n ≥ 0} is the DTMC studied in Example 2.17 on page 20, we
can use the results about its limiting distribution from Example 4.24 on page 121.
From there we see that the DTMC is positive recurrent if and only if
∞
X
kαk > 1.
k=0
SINGLE SERVER QUEUES 319
Substituting from Equation 7.39 we get
∞
X µ
kαk = .
λ
k=0
µ
Thus the DTMC is positive recurrent if and only if λ > 1, i.e., ρ < 1. Let
∞
X
ψ(z) = z i αi .
i=0

Following the derivation in the proof of Theorem 7.11, we get

ψ(z) = G̃(µ − µz).
From Equation 4.46 (using α in place of ρ) we get
πj∗ = (1 − α)αj , j ≥ 0,
where α is the unique solution in (0, 1) to
α = ψ(α) = G̃(µ − µα).
This proves the theorem.

Example 7.15 The M/M/1 Queue. If the service time distribution is

G(x) = 1 − e−λx , x ≥ 0
the G/M/1 queue reduces to M/M/1 queue. In this case we get
λ
G̃(s) = .
s+λ
Substituting in Equation 7.41 we get
λ
α= .
µ(1 − α) + λ
Solving for α we get
λ
α= = ρ, or α = 1.
µ
If ρ < 1, the α = ρ is the only solution in (0, 1). In this case Equation 7.40 reduces
to
πj∗ = (1 − ρ)ρj , j ≥ 0.
Since the arrival process in this queue is Poisson, we have pj = π̂j = πj∗ . Thus we
have
pj = (1 − ρ)ρj , j ≥ 0.
This matches with the result in Example 6.36 on page 242, as expected.

The next theorem relates {pj , j ≥ 0} and {πj∗ , j ≥ 0}.

Theorem 7.16 Limiting Distribution of a G/M/1 Queue. For a G/M/1 queue

320 QUEUEING MODELS
with ρ = λ/µ < 1 the limiting distributions {pj , j ≥ 0} and {πj∗ , j ≥ 0} are related
as follows:
p0 = 1 − ρ, (7.42)
∗
pj = ρπj−1 , j ≥ 1. (7.43)

Proof: Postponed to Theorem 9.22 on page 437.

In the next theorem we study the limiting distribution of waiting times (this in-
cludes time in service) in a G/M/1 queue assuming FCFS service discipline.

Theorem 7.17 Waiting Times in a G/M/1 Queue. The limiting distribution of the
waiting time in a stable G/M/1 queue with FCFS service discipline is given by
F (x) = lim P(Wn ≤ x) = 1 − e−µ(1−α)x , x ≥ 0. (7.44)
n→∞

Proof: The waiting time of a customer who sees j customers ahead of him is an
Erlang(j + 1, µ) random variable. Using this we get
∞
X
F (x) = lim P(Wn ≤ x|Xn∗ = j)P(Xn∗ = j)
n→∞
j=0
∞
X
= πj∗ P(Erl(j + 1, µ) ≤ x).
j=0

The rest of the proof follows along the same lines as in the case of the M/M/1
queue.

From Equation 7.44 we get

1
W = .
µ(1 − α)
Using Little’s Law we get
λ
L= ,
µ(1 − α)
which can be verified by computing L directly from the limiting distribution of X(t)
given Theorem 7.16.

7.7 Retrial Queue

We have already seen the M/M/1/1 retrial queue in Example 6.15 on page 202. In
this section we generalize it to M/G/1/1 queue. We describe the model below.

Customers arrive from outside to a single server according to a PP(λ) and require
RETRIAL QUEUE 321
iid service times with common distribution G(·) and mean τ . There is room only for
the customer in service. Thus the capacity is 1, hence the M/G/1/1 nomenclature. If
an arriving customer finds the server idle, he immediately enters service. Otherwise
he joins the “orbit,” where he stays for an exp(θ) amount of time (called the retrial
time) independent of his past and the other customers. At the end of the retrial time
he returns to the server, and behaves like a new customer. He persists in conducting
retrials until he is served, after which he exits the system. A block diagram of this
queueing system is shown in Figure 7.8.

Orbit

Server busy

Single Server

Arrivals No Waiting Space Departures

Figure 7.8 Schematic diagram of a single server retrial queue.

Let X(t) be the number of customers in the system (those in service + those in
orbit) at time t. Note that {X(t), t ≥ 0} is not a CTMC. It has jumps of size +1
when a new customer arrives, and of size -1 when a customer completes service.
Since every arriving customer enters the system (either service or the orbit), and the
arrival process is Poisson, we have
π̂j = πj∗ = πj = pj , j ≥ 0.
Thus we can study the limiting behavior of the {X(t), t ≥ 0} by studying the
{Xn , t ≥ 0} process at departure points, since, as the next theorem shows, it is a
DTMC.

Theorem 7.18 Embedded DTMC in an M/G/1/1 Retrial Queue. {Xn , n ≥ 0}

is an irreducible and aperiodic DTMC on S = {0, 1, 2, · · ·}.

Proof: Let An be the number of arrivals to the queueing system during the nth ser-
vice time. Since the service times are iid random variables with common distribution
G(·), and the arrival process is PP(λ), we see that {An , n ≥ 1} is a sequence of iid
random variables. Now, immediately after a service completion, the server is idle.
Hence Xn represents the number of customers in the orbit when the nth service
completion occurs. Each of these customers will conduct a retrial after iid exp(θ)
times. Also, a new arrival will occur after an exp(λ) amount of time. Hence the next
service request will occur after an exp(λ + θXn ) amount of time. With probability
θXn /(λ + θXn ) this request is from a customer from the orbit, and with probability
λ/(λ + θXn ), it is from a new customer. The (n + 1)st service starts when this re-
quest arrives, during which An+1 new customers arrive and join the orbit. Hence the
322 QUEUEING MODELS
system dynamics is given by

Xn + An+1 with probability λ/(λ + θXn )
Xn+1 = (7.45)
Xn + An+1 − 1 with probability θXn /(λ + θXn ).
Since An+1 is independent of history, the above recursion implies that {Xn , n ≥ 0}
is a DTMC. Irreducibility and aperiodicity is obvious.

The next theorem gives the generating function of the limiting distribution of
{Xn , n ≥ 0}.

Theorem 7.19 Limiting Distribution of an M/G/1/1 Retrial Queue. The DTMC

{Xn , n ≥ 0} with θ > 0 is positive recurrent if and only if
ρ = λτ < 1.
If it is positive recurrent, its limiting distribution has the generating function given
by
∞ Z 1 !
X (1 − z)G̃(λ − λz) λ 1 − G̃(λ − λu)
φ(z) = πj z j = (1 − ρ) · exp − du ,
j=0
G̃(λ − λz) − z θ z G̃(λ − λu) − u
(7.46)
where Z ∞
G̃(s) = e−st dG(t).
0

Proof: From Equation 7.45 we get

   
Xn+1 Xn +An+1 λ Xn +An+1 −1 θXn
E(z ) = E z +E z
λ + θXn λ + θXn
Xn Xn −1
    
λz θX n z
= E(z An+1 ) E +E . (7.47)
λ + θXn λ + θXn
Now let
λz Xn
 
Xn
φn (z) = E(z ) and ψn (z) = E .
λ + θXn
Then
λz Xn λXn z Xn −1
   
θz θz
ψn (z) + ψn′ (z) = E + E
λ λ + θXn λ λ + θXn
Xn
 
(λ + θXn )z
= E (7.48)
λ + θXn
= E(z Xn ) = φn (z). (7.49)
Also, from the proof of Theorem 7.11, we get
E(z An ) = G̃(λ − λz), n ≥ 1.
RETRIAL QUEUE 323
Using Equation 7.49 in the Equation 7.47 we get
θz ′ θ
ψn+1 (z) + ψ (z) = G̃(λ − λz)(ψn (z) + ψn′ (z)). (7.50)
λ n+1 λ
Now let
φ(z) = lim φn (z), and ψ(z) = lim ψn (z).
n→∞ n→∞
Letting n → ∞ in Equation 7.50 and rearranging, we get
θ
(z − G̃(λ − λz))ψ ′ (z) = (G̃(λ − λz) − 1)ψ(z)
λ
which can be easily integrated to obtain
!
λ z G̃(λ − λu) − 1
Z
′
ψ(z) = C exp du ,
θ u − G̃(λ − λu)
where C ′ is a constant of integration. By choosing
!
λ 1 G̃(λ − λu) − 1
Z
′
C = C exp − du ,
θ u − G̃(λ − λu)
we get !
1
λ G̃(λ − λu) − 1
Z
ψ(z) = C exp − du . (7.51)
θ z u − G̃(λ − λu)
Letting n → ∞ in Equation 7.49 we get
θz ′
φ(z) = ψ(z) + ψ (z).
λ
Using Equation 7.50 and 7.51 we get
!
1
(1 − z)G̃(λ − λz) λ 1 − G̃(λ − λu)
Z
φ(z) = C · exp − du .
G̃(λ − λz) − z θ z G̃(λ − λu) − u
The unknown constant C can be evaluated by using φ(1) = 1. This yields (after
applying L’Hopital’s rule)
C = 1 − ρ. (7.52)
Hence the theorem follows.

One immediate consequence of Equation 7.46 is that the probability that the sys-
tem is empty in steady state can be computed as
!
λ 1 1 − G̃(λ − λu)
Z
p0 = π0 = φ(0) = (1 − ρ) exp − du .
θ 0 G̃(λ − λu) − u
However, this is not the same as the server being idle, since the server can be idle
even if the system is not empty. We can use Little’s Law type argument to show that
the server is idle in steady state with probability 1 − ρ, independent of θ! But this
324 QUEUEING MODELS
simple fact cannot be deduced by using the embedded DTMC.

Now, since pj = πj for all j ≥ 0, φ(z) in Equation 7.46 is also the generating func-
tion of the limiting distribution of the {X(t), t ≥ 0} process. Using Equation 7.46
we can compute the expected number of customers in the system in steady state as
given in the following theorem.

Theorem 7.20 Expected Number in an M/G/1/1 Retrial Queue. The expected

number in steady state in a stable M/G/1/1 retrial queue with θ > 0 is given by
ρ2 σ2
 
1 λ ρ
L=ρ+ · 1+ 2 + , (7.53)
2 1−ρ τ θ 1−ρ
where τ and σ 2 are the mean and variance of the service time.

Proof: We have
L = lim E(X(t))
t→∞
∞ ∞
X X dφ(z)
= jpj = jπj = |z=1 .
j=0 j=0
dz

The theorem follows after evaluating the last expression in straight forward fashion.
This involves using L’Hopital’s rule twice.

Note that L is a decreasing function of θ. In fact, as θ → ∞, L of the above

theorem converges to the L of a standard M/G/1 as given Theorem 7.12. This makes
intuitive sense, since in the limit as θ → ∞, every customer is always checking to
see if the server is idle. Thus as soon as the service is complete a customer from the
orbit (if it is not empty) enters service. However, the service discipline is in “random
order,” rather than in FCFS fashion, although this does not affect the queue length
process. As expected, the generating function of the retrial queue reduces to that of
the M/G/1 queue as θ → ∞.

7.8 Infinite Server Queue

We have seen the M/M/s queue in Subsection 7.3.3 and the M/M/∞ queue in
Subsection 7.3.4. Unfortunately, the M/G/s queue proves to be intractable. Surpris-
ingly, M/G/∞ queue can be analyzed very easily. We present that analysis here.

Consider an infinite server queue where customers arrive according to a PP(λ), and
request iid service times with common cdf G(·) and mean τ . Let X(t) be the number
of customers in such a queue at time t. Suppose X(0) = 0. We have analyzed this
process in Example 5.17 on page 173. Using the analysis there we see that X(t) is a
MODELING EXERCISES 325
Poisson random variable with mean λm(t), where
Z t
m(t) = (1 − G(u))du.
0
We have
lim m(t) = τ.
t→∞
Hence the limiting distribution of X(t) is P(λτ ). Note that this limiting distribution
holds even if X(0) > 0, since all the initial customers will eventually leave, and do
not affect the newly arriving customers.
We conclude this chapter with the remark that it is possible to analyze a G/M/s
queue with an embedded DTMC chain. The G/M/∞ queue can be analyzed by the
methods of renewal processes, to be developed in the next chapter. Note that the
{X(t), t ≥ 0} processes studied in the last three sections are not CTMCs. What kind
of processes are these? The search for the answer to this question will lead us into
renewal theory, regenerative processes, and Markov regenerative processes. These
topics will be covered in the next two chapters.

7.9 Modeling Exercises

7.1 Customers arrive at a taxi stand according to a PP(λ). If a taxi is waiting at the
taxi stand, the customer immediately hires it and leaves the taxi stand in the taxi. If
there are no taxis available, the customer waits. There is essentially infinite waiting
room for the customers. Independently of the customers, taxis arrive at the taxi stand
according to a PP(µ). If a taxi arriving at the taxi stand finds that no customer is
waiting, it leaves immediately. Model this system as an M/M/1 queue, and specify
its parameters.

7.2 A machine produces items one at a time according to a PP(λ). These items are
stored in a warehouse of infinite capacity. Demands arise according to a PP(µ). If
there is an item in the warehouse when a demand arises, an item is immediately
removed to satisfy the demand. Any demand that occurs when the warehouse is
empty is lost. Let X(t) be the number of items in the warehouse at time t. Model the
{X(t), t ≥ 0} process as a birth and death process.

7.3 Customers arrive at a bank according to a PP(λ). The service times are iid
exp(µ). The bank follows the following policy: when there are fewer than four cus-
tomers in the bank, only one teller is active; for four to nine customers, the bank uses
two tellers; and beyond nine customers there are three tellers. Model the number of
customers in the bank as a birth and death process.

7.4 Customers arrive according to a PP(λ) at a single-server station and demand iid
exp(µ) service times. When a customer completes his service, he departs with proba-
bility α, or rejoins the queue instantaneously with probability 1 − α, and behaves like
a new customer. The service times are iid exp(µ). Model the number of customers in
the system as a birth and death process.
326 QUEUEING MODELS
7.5 Consider a grocery store checkout queue. When the number of customers in the
line is three or less, the checkout person does the pricing as well as bagging, taking
exp(µ1 ) time. When there are three or more customers in the line, a bagger comes
to help, and the service rate increases to µ2 > µ1 , i.e., the reduced service times are
now iid exp(µ2 ). Assume that customers join the checkout line according to a PP(λ).
Model the number of customers in the checkout line as a birth and death process.

7.6 Consider a single server queue subject to break downs and repairs as follows: the
worker stays functional for an exp(θ) amount of time and then fails. The repair time
is exp(α). The successive up and down times are iid. However, the server is subject
to failures only when it is serving a customer. The service times are iid exp(µ).
Assume that the failure does not cause any loss of work. Thus if a customer service
is interrupted by failure, the service simply resumes after the server is repaired. Let
X(t) be the number of customers in this system at time t. Model {X(t), t ≥ 0} as
an M/G/1 queue by identifying the correct service time distribution G.

7.7 Consider a single server queue that serves customers from k independent
sources. Customers from source i arrive according to a PP(λi ) and need iid exp(µi )
service times. They form a single queue and are served in an FCFS fashion. Let X(t)
be the number of customers in the system at time t. Show that {X(t), t ≥ 0} is the
queue-length process in an M/G/1 queue. Identify the service distribution G.

7.8 Redo the problem in Modeling Exercise 7.6 assuming that the server can fail
even when it is not serving any customers. Is {X(t), t ≥ 0} the queue length process
of an M/G/1 queue? Explain. Let Xn be the number of customers in the system
after the nth departure. Show that {Xn , n ≥ 0} is a DTMC and display its transition
probability matrix.

7.9 Consider the {X(t), t ≥ 0} process described in Modeling Exercise 7.2 with
the following modification: the machine produces items in a deterministic fashion at
a rate of one item per unit time. Model {X(t), t ≥ 0} as the queue length process in
a G/M/1 queue.

7.10 Customers arrive according to a PP(λ) at a service station with s distinct

servers. Service times at server i are iid exp(µi ), with µ1 > µ2 > · · · > µs . There
is no waiting room. Thus there can be at the most s customers in the system. An
incoming customer goes to the fastest available server. If all the servers are busy, he
leaves without service. Model this as a CTMC.

7.11 Consider an M/G/1 queue where the server goes on vacation if the system is
empty upon service completion. If the system is empty upon return from the vacation,
the server goes on another vacation; else he starts serving the customers in the system
one by one. Successive vacation times are iid. Let Xn be the number of customers in
the system after the nth customer departs. Show that {Xn , n ≥ 0} is a DTMC.
COMPUTATIONAL EXERCISES 327
7.12 A service station is staffed with two identical servers. Customers arrive ac-
cording to a PP(λ). The service times are iid with common distribution exp(µ) at
either server. Consider the following two routing policies

1. Each customer is randomly assigned to one of the two servers with equal proba-
bility.
2. Customers are alternately assigned to the two servers.

Once a customer as assigned to a server he stays in that line until served. Let Xi (t) be
the number of customers in line for the ith server. Is {Xi (t), t ≥ 0} the queue-length
process of an M/M/1 queue or an G/M/1 queue under the two routing schemes?
Identify the parameters of the queues.

7.13 Consider the following variation of an M/G/1 queue: All customers have iid
2
service times with common cdf G, with mean τG and variance σG . However the
customers who enter an empty system have a different service time cdf H with mean
2
τH and variance σH . Let X(t) be the number of customers at time t. Is {X(t), t ≥ 0}
a CTMC? If yes, give its generator matrix. Let Xn be the number of customers in the
system after the nth departure. Is {Xn , n ≥ 0} a DTMC? If yes, give its transition
probability matrix.

7.14 Consider a communication node where packets arrive according to a PP(λ).

The node is allowed to transmit packets only at times n = 0, 1, 2 · · · , and transmis-
sion time of a packet is one unit of time. If a packet arrives at an empty system, it
has to wait for the next transmission time to start its transmission. Let X(t) be the
number of packets in the system at time t, Xn be the number of packets in the system
after the completion of the nth transmission, and X̄n be the number of packets avail-
able for transmission at time n. Is {Xn , n ≥ 0} a DTMC? If yes, give its transition
probabilities. Is {X̄n , n ≥ 0} a DTMC? If yes, give its transition probabilities.

7.15 Suppose the customers that cannot enter an M/M/1/1 queue (with arrival
rate λ and service rate µ) enter service at another single server queue with infinite
waiting room. This second queue is called an overflow queue. The service times
at the overflow queue are iid exp(θ) random variables. Let X(t) be the number of
customers at the overflow queue at time t. Model the overflow queue as a G/M/1
queue. What is the LST of the interarrival time distribution to the overflow queue?

7.10 Computational Exercises

7.1 Show that the variance of the number of customers in steady state in a stable
M/M/1 system with arrival rate λ and service rate µ is given by
ρ
σ2 = ,
(1 − ρ)2
where ρ = λ/µ.
328 QUEUEING MODELS
7.2 Let X q (t) be the number of customers in the queue (not including any in
service) at time t in an M/M/1 queue with arrival rate λ and service rate µ. Is
{X q (t), t ≥ 0} a CTMC? Compute the limiting distribution of X q (t) assuming
λ < µ. Show that the expected number of customers in the queue (not including the
customer in service) is given by
ρ2
Lq = .
1−ρ
7.3 Let Wnq be the time spent in the queue (not including time in service) by the
nth arriving customer in an M/M/1 queue with arrival rate λ and service rate µ.
Compute the limiting distribution of Wnq assuming λ < µ. Compute W q , the limiting
expected value of Wnq as n → ∞. Using the results of Computational Exercise 7.2
show that Lq = λW q . Thus little’s law holds when applied to the customers in the
queue.

7.4 Let X(t) be the number of customers in the system at time t in an M/M/1
queue with arrival rate λ and service rate µ > λ. Let
T = inf{t ≥ 0 : X(t) = 0}.
T is called the busy period. Compute E(T |X(0) = i).

7.5 Let T be as in Computational Exercise 7.4. Let N be the total number of cus-
tomers served during (0, T ]. Compute E(N |X(0) = i).

7.6 Customers arrive according to P P (λ) to a queueing system with two servers.
The ith server (i = 1, 2) needs exp(µi ) amount of time to serve one customer. Each
incoming customer is routed to server 1 with probability p1 or to server 2 with prob-
ability p2 = 1 − p1 , independently. Queue jumping is not allowed. Find the optimum
routing probabilities that will minimize the expected total number of customers in
the system in steady state.

7.7 Consider a stable M/M/1 queue with the following cost structure. A customer
who sees i customers ahead of him when he joins the system costs $ci to the system.
The system charges every customer a fee of $f upon entry. Show that the long run
net revenue is given by
∞
X
λ(f − ci ρi (1 − ρ)).
i=0

7.8 This is a generalization of Computational Exercise 7.6. A queueing system con-

sists of K servers, each with its own queue. Customers arrive at the system according
to a PP(λ). A system controller routes an incoming customer to server k with proba-
bility αk , where α1 + α2 + · · · + αK = 1. Customers assigned to server k receive
iid exp(µk ) service times. Assume that µ1 + µ2 + · · · + µK > λ. It costs hk dollars
to hold a customer for one unit of time in queue k.

1. What are the feasible values of αk ’s so that the resulting system is stable?
COMPUTATIONAL EXERCISES 329
2. Compute the the expected holding cost per unit time as a function of the routing
probabilities αk (1 ≤ k ≤ K) in the stable region.
3. Compute the optimal routing probabilities αk that minimize the holding cost per
unit time for the entire system.

7.9 Compute the long run fraction of customers who cannot enter the M/M/1/K
system described in Subsection 7.3.2.

7.10 Compute W , the expected time spent in the system by an arriving customers
in steady state in an M/M/1/K system, by using Little’s Law and Equation 7.20.
(If an arriving customer does not enter, his time in the system is zero.) What is the
correct value of λ in L = λW as applied to this example?

7.11 Compute W , the expected waiting time of entering customers in steady state in
an M/M/1/K system, by using Little’s Law and Equation 7.20. What is the correct
value of λ in L = λW as applied to this example?

7.12 Suppose there are 0 < i < K customers in an M/M/1/K queue at time 0.
Compute the expected time when the queue either becomes empty or full.

7.13 Consider the M/M/1/K system of Subsection 7.3.2 with the following cost
structure. Each customer waiting in the system costs $c per unit time. Each customer
entering the system pays $a as an entry fee to the system. Compute the long run rate
of net revenue for this system.

7.14 Consider the system of Modeling Exercise 7.2 with production rate of 10 per
hour and demand rate of 8 per hour. Suppose the machine is turned off when the
number of items in the warehouse reaches K, and is turned on again when it falls
to K − 1. Any demand that occurs when the warehouse is empty is lost. It costs 5
dollars to produce an item, and 1 dollar to keep an item in the warehouse for one
hour. Each item sells for ten dollars.

1. Model this system as an M/M/1/K queue. State the parameters.

2. Compute the long run net income (revenue-production and holding cost) per unit
time, as a function of K.
3. Compute numerically the optimal K that maximizes the net income per unit time.

7.15 Consider the M/M/1 queue with balking (but no reneging) as described in
Subsection 7.3.6. Suppose the limiting distribution of the number of customers in this
queue is {pj , j ≥ 0}. Using PASTAP show that in steady state an arriving customer
enters the system with probability ∞ j=0 αj pj .

7.16 Consider the M/M/1 queue with balking (but no reneging) as described in
Subsection 7.3.6. Suppose the limiting distribution of the number of customers in
this queue is P(ρ), where ρ = λ/µ. What balking probabilities will produce this
limiting distribution?
330 QUEUEING MODELS
7.17 Show that the expected number of busy servers in a stable M/M/s queue is
λ/µ.

7.18 Derive Equation 7.21. Hence or otherwise compute the expected waiting time
of a customer in the M/M/s system in steady state.

7.19 Show that for a stable M/M/s queue of Subsection 7.3.3

ps ρ
Lq = .
(1 − ρ)2
Compute W q explicitly and show that Little’s Law Lq = λW q is satisfied.

7.20 Compute the limiting distribution of the time spent in the queue by a customer
in an M/M/s queue. Hence or otherwise compute the limiting distribution of the
time spent in the system by a customer in an M/M/s queue.

7.21 Consider two queueing systems. System 1 has s servers, each serving at rate
µ. System 2 has a single server, serving at rate sµ. Both systems are subject to PP(λ)
arrivals. Show that in steady state, the expected number of customers in the queue
(not including those in service) System 2 is less than in System 1. This shows that it
is better to have a single efficient server than many inefficient ones.

7.22 Consider the finite population queue of Subsection 7.3.5 with two machines
and one repairperson. Suppose every working machine produces revenue at a rate of
$r per unit time. It costs $C to repair a machine. Compute the long run rate at which
the system earns profits (revenue - cost).

7.23 When is the system in Modeling Exercise 7.2 stable? Assuming stability, com-
pute the limiting distribution of the number of items in the warehouse. What fraction
of the incoming demands are satisfied in steady state?

7.24 Compute the limiting distribution {pi , 0 ≤ i ≤ s} of the number of customers

in an M/M/s/s queue with arrival rate λ and service rate µ for each server.

7.25 The quantity ps in the Computational Exercise 7.24 is called the blocking prob-
ability, and is denoted by B(s, ρ) where ρ = λ/µ. Show that the long run rate at
which the customers enter the system is given by λ(1 − B(s, ρ)). Also, show that
B(s, ρ) satisfies the recursion
ρB(s − 1, ρ)
B(s, ρ) = ,
s + ρB(s − 1, ρ)
with initial condition B(0, ρ) = 1.

7.26 When is the system in Modeling Exercise 7.3 stable? Assuming stability, com-
pute the limiting distribution of the number of customers in the bank. What is the
steady state probability that three tellers are active?
COMPUTATIONAL EXERCISES 331
7.27 When is the system in Modeling Exercise 7.4 stable? Assuming stability, com-
pute the limiting distribution of the number of customers in the system.

7.28 When is the system in Modeling Exercise 7.5 stable? Assuming stability, com-
pute the expected number of customers in the system in steady state.

7.29 Consider the single server queue with N -type control described in Modeling
Exercise 6.16. Let X(t) be the number of customers in the system at time t, and Y (t)
be 1 if the server busy and 0 if it is idle at time t. Show that {(X(t), Y (t)), t ≥ 0} is
a CTMC and that it is stable if ρ = λ/µ < 1. Assuming it is stable, show that
pi,j = lim P(X(t) = i, Y (t) = j), i ≥ 0, j = 0, 1,
t→∞

is given by
1−ρ
pi,0 = , 0 ≤ i < N,
N
ρ
pi,1 = (1 − ρi ), 1 ≤ i < N
N
ρ
pN +n,1 = (1 − ρN )ρn , n ≥ 0.
N
7.30 Consider the queueing system of Computational Exercise 7.29. Suppose it
costs $f to turn the server on from the off position, while turning the server off
is free of cost. It costs $c to keep one customer in the system for one unit of time.
Compute the long run operating cost per unit of the N -type policy. Show how one
can optimally choose N to minimize this cost rate.

7.31 Consider the system of Modeling Exercise 6.31. What is the limiting distribu-
tion of the number of customers in the system as seen by an arriving customer of
type i? By an entering customer of type i? (i = 1, 2)

7.32 Compute the limiting distribution of the CTMC in modeling Exercise 7.10 for
the case of s = 3. What fraction of the customers are turned away in steady state?

7.33 Consider the Jackson network of single server queues as shown in Figure 7.9.
Derive the stability condition. Assuming stability compute

λ 1–p
µ1 µ2 µN–1 µN

Figure 7.9 Queueing network for Computational Exercise 7.33.

1. the expected number of customers in steady state in the network,

2. the fraction of the time the network is completely empty in steady state.
332 QUEUEING MODELS
7.34 Do Computational Exercise 7.33 for the network in Figure 7.10.
λ 1 – p1 1 – p2 1 – pN–1 1 – pN
µ1 µ2 µN–1 µN

p1 p2 pN–1 pN

Figure 7.10 Queueing network for Computational Exercise 7.34.

7.35 Do Computational Exercise 7.33 for the network in Figure 7.11.

λ 1–P1 1–P2 1–PN–1 1–PN

µ1 µ2 µN–1 µN

P1 P2 PN–1 PN
µN+1

Figure 7.11 Queueing network for Computational Exercise 7.35.

7.36 North Carolina State Fair has 35 rides, and it expects to get about 60,000 vis-
itors per day (12 hours) on the average. Each visitor is expected to take 5 rides on
the average during his/her visit. Each ride lasts approximately 1 minute and serves
an average of 30 riders per batch. Construct an approximate Jackson network model
of the rides in the state fair that incorporates all the above data in a judicious fashion.
State your assumptions. Is this network stable? Show how to compute the average
queue length at a typical ride.

7.37 Consider a network of two nodes in series that operates as follows: customers
arrive at the first node from outside according to a PP(λ), and after completing service
at node 1 move to node 2, and exit the system after completing service at node 2. The
service times at each node are iid exp(µ). Node 1 has one server active as long
as there are five or fewer customers present at that node, and two servers active
otherwise. Node 2 has one server active for up to two customers, two servers for
three through ten customers, and three servers for any higher number. If an arriving
customer sees a total of i customers at the two nodes, he joins the first node with
probability 1/(i + 1) and leaves the system without any service with probability
i/(i + 1). Compute

1. the condition of stability,

2. the expected number of customers in the network in steady state.

7.38 A 30 mile long stretch of an interstate highway in Montana has no inlets or

exits. This stretch is served by 3 cell towers, stationed at milepost numbers 5, 15, and
25. Each tower serves calls in the ten mile section around it. Cars enter the highway at
milepost zero according to a PP(λ), with λ = 60/hr. (Ignore the traffic in the reverse
direction.) They travel at a constant speed of 100 miles per hour. Each entering car
COMPUTATIONAL EXERCISES 333
initiates a phone call at rate θ = .2 per minute, i.e., the time until the initiation of a
call is an exp(θ) random variable. The call duration is exponentially distributed with
mean 10 minutes. Once the call is finished the car does not generate any new calls.
(Thus each car generates at most one call.) Suppose there is enough channel capacity
available that no calls are blocked. When the calling car crosses from the area of
one station to the next, the call is seamlessly handed over to the next station. Model
this as a Jackson network with five nodes, each having infinite servers. Node 1 is for
the first tower, nodes 2 and 3 are for the second tower, and nodes 4 and 5 are for
the third tower. Nodes 1, 2, and 4 handle newly initiated calls, while nodes 3 and 5
handle handed-over calls. Tower 1 does not handle any handed-over calls. Note that
for infinite server nodes the service time distribution can be general. Let Xi (t) be the
number of calls at time t in node i, 1 ≤ i ≤ 5. Compute

1. the service time distribution of the calls in node i,

2. the routing matrix,
3. the expected number of calls handled by the ith station in steady state,
4. the expected number of calls that are handed over from station i to station i + 1
per unit time (i = 1, 2).

7.39 Consider an open Jackson network with N single-server nodes. Customers

arrive from outside the network to the ith node with rate λi . A fraction pi of the
customers completing service at node i join the queue at node i + 1 and the rest
leave the network permanently, i = 1, 2, ..., N − 1. Customers completing service at
node N join the queue at node 1 with probability pN , and the rest leave the network
permanently. The service times at node i are exp(µi ) random variables.

1. State the assumptions to model this as a Jackson network.

2. What are the traffic equations for the Jackson network? Solve them.
3. What is the condition of stability?
4. What is the expected number of customers in the network in steady state, assum-
ing the network is stable?

7.40 Show that the probability that a customer in an open Jackson network of Sec-
tion 7.4 stays in the network forever is zero if I − R is invertible.

7.41 For a closed Jackson network of single server queues, show that

1. limt→∞ P(Xi (t) ≥ j) = ρji GGNN(K−j)

(K)
, 0 ≤ j ≤ K.
PK j GN (K)
2. Li = limt→∞ E(Xi (t)) = j=1 ρi GN (K−j) , 0 ≤ j ≤ K.

7.42 Generalize the method of computing GN (K) derived in Example 7.11 to gen-
eral closed Jackson networks of single-server queues with N nodes and K customers.
334 QUEUEING MODELS
7.43 A simple communications network consists of two nodes labeled A and B
connected by two one-way communication links: line AB from A to B, and line BA
from line from B to A. There are N users at each node. The ith user (1 ≤ i ≤ N ) at
node A (B) is denoted by Ai (Bi ). User Ai has an interactive session set up with user
Bi and it operates as follows: User Ai sends a message to user Bi . All the messages
generated at node A wait in a buffer at node A for transmission to the appropriate
user at node B on line AB in an FCFS fashion. When user Bi receives the message
from user Ai , she spends a random amount of time, called think time, to generate a
response to it. All the messages generated at node B wait in a buffer at node B for
transmission to the appropriate user at node A on line BA in an FCFS fashion. When
user Ai receives the message from user Bi , she spends a random amount of time to
generate a response to it. This process of messages going back and forth between
the pairs of users Ai and Bi continues forever. Suppose all the think times are iid
exp(θ) random variables, and the message transmission times are iid exp(µ) random
variables. Model this as a closed Jackson network. What is the expected number of
messages in the buffers at nodes A and B in steady state?

7.44 For the closed Jackson network of Section 7.5, define the throughput T H(i)
of node i as the rate at which customers leave node i in steady state, i.e.,
K
X
T H(i) = µi (n) lim P(Xi (t) = n).
t→∞
n=0

Show that
GN (K)
T H(i) = πi .
GN (K − 1)

7.45 When is the system in Modeling Exercise 7.7 stable? Assuming stability, com-
pute the expected number of customers in the system in steady state.

7.46 When is the system in Modeling Exercise 7.6 stable? Assuming stability, com-
pute the generating function of the limiting distribution of the number of customers
in the system.

7.47 Compute the expected number of customers in steady state in an M/G/1 sys-
tem where the arrival rate is one customer per hour and the service time distribution
is PH(α, M ) where
α = [0.5 0.5 0]
and  
−3 1 1
M =  0 −3 2  .
0 0 −3

7.48 Compute the expected queue length in an M/G/1 queue with the following
service time distributions (all with mean 1/µ):

1. Exponential with parameter µ,

COMPUTATIONAL EXERCISES 335
2. Uniform over [0, 2/µ],
3. Deterministic with mean 1/µ,
4. Erlang with parameters (k, kµ).

Which distribution produces the largest congestion? Which produces the smallest?

7.49 Consider the {X(t), t ≥ 0} and the {Xn , n ≥ 0} processes defined in Mod-
eling Exercise 7.8. Show that the limiting distribution of the two (if they exist) are
identical. Let pn (qn ) be the limiting probability that there are n customers in the
system and the server is up (down). Let p(z) and q(z) be the generating functions of
{pn , n ≥ 0} and {qn , n ≥ 0}. Show that this system is stable if
λ α
< .
µ α+θ
Assuming that the system is stable show that

µ α λ

z α+θ − µ
q(z) = µ α
,
+ λθ (1 − z) − λ



z −λ θ
and  
α λ
p(z) = + (1 − z) q(z).
θ θ

7.50 Show that the DTMC {Xn , n ≥ 0} in the Modeling Exercise 7.11 is positive
recurrent if ρ = λτ < 1, where λ is the arrival rate and τ is the mean service time.
Assuming the DTMC is stable, show that the generating function of the limiting
distribution of Xn is given by
1−ρ G̃(λ − λz)
φ(z) = · · (ψ(z) − 1),
m z − G̃(λ − λz)
where G̃ is the LST of the service time, m is the expected number of arrivals during a
single vacation, and ψ(z) is the generating function of the number of arrivals during
a single vacation.

7.51 Let X(t) be the number of customers at time t in the system described in Mod-
eling Exercise 7.11. Show that {Xn , n ≥ 0} and {X(t), t ≥ 0} have the same limit-
ing distribution, assuming it exists. Using the results of Computational Exercise 7.50
show that the expected number of customers in steady state is given by
ρ2 σ2 m(2)
 
1
L=ρ+ · 1+ 2 + ,
2 1−ρ τ 2m
where σ 2 is the variance of the service time, m(2) is the second factorial moment of
the number of arrivals during a single vacation.

7.52 Let X(t) be the number of customers at time t in an M/G/1 queue under N -
type control as explained in Modeling Exercise 6.16 for an M/M/1 queue. Using the
336 QUEUEING MODELS
results of Computational Exercises 7.50 and 7.51 establish the condition of stability
for this system and compute the generating function of the limiting distribution of
X(t) as t → ∞.

7.53 When is the queueing system described in Modeling Exercise 7.12 stable? As-
suming stability, compute the expected number of customers in the system in steady
state under the two policies. Which policy is better at minimizing the expected num-
ber in the system in steady state?

7.54 Analyze the stability of the {X(t), t ≥ 0} process in Modeling Exercise 7.9.
Assuming stability, compute the limiting distribution of the number of items in the
warehouse. What fraction of the demands are lost in steady state?

7.55 Show that the DTMC {Xn , n ≥ 0} in the Modeling Exercise 7.12 is positive
recurrent if ρ = λτG < 1. Assuming the DTMC is stable, show that the generating
function of the limiting distribution of Xn is given by

1 − λτG z H̃(λ − λz) − G̃(λ − λz)

φ(z) = · .
1 − λτG + λτH z − G̃(λ − λz)

Hint: Use the results of Computational Exercise 4.24.

7.56 Let X(t) be the number of customers at time t in the system described in Mod-
eling Exercise 7.12. Show that {Xn , n ≥ 0} and {X(t), t ≥ 0} have the same limit-
ing distribution, assuming it exists. Using the results of Computational Exercise 7.55
show that the expected number of customers in steady state is given by

λτH λ2 σH
2
+ τH2 2
− σG 2
− τG λ2 σG
2 2
+ τG
L= + · + · .
1 − λτG + λτH 2 1 − λτG + λτH 2 1 − λτG

7.57 Show that the DTMC {Xn , n ≥ 0} in the Modeling Exercise 7.14 is positive
recurrent if λ < 1. Assuming the DTMC is stable, compute φ(z), the generating
function of the limiting distribution of Xn as n → ∞.

7.58 Show that the DTMC {X̄n , n ≥ 0} in the Modeling Exercise 7.14 is positive
recurrent if λ < 1. Assuming the DTMC is stable, compute φ̄(z), the generating
function of the limiting distribution of X̄n as n → ∞.

7.59 In the Modeling Exercise 7.14, is the limiting distribution of {X(t), t ≥ 0}

same as that of {Xn , n ≥ 0} or {X̄n , n ≥ 0}? Explain.

7.60 Consider an M/G/1 queue where the customers arrive according to a PP(λ)
and request iid service times with common mean τ , and variance σ 2 . After service
completion, a customer leaves with probability p, or returns to the end of the queue
with probability 1 − p, and behaves like a new customer.
COMPUTATIONAL EXERCISES 337
1. Compute the mean and variance of the amount of time a customer spends in ser-
vice during the sojourn time in the system.
2. Compute the condition of stability.
3. Assuming stability, compute the expected number of customers in the system as
seen by a departure (from the system) in steady state.
4. Assuming stability, compute the expected number of customers in the system at a
service completion (customer may or may not depart at each service completion)
in steady state.

7.61 Compute the limiting distribution of the number of customers in a G/M/1

queue with the interarrival times

G(x) = r(1 − e−λ1 x ) + (1 − r)(1 − e−λ2 x ),

where 0 < r < 1, λ1 > 0, λ2 > 0. The service times are iid exp(µ).

7.62 Let X(t) be the number of customers in a G/M/2 queue at time t. Let Xn∗
be the number of customers as seen by the nth arrival. Show that {Xn∗ , n ≥ 0} is a
DTMC, and compute its one-step transition probability matrix. Derive the condition
of stability and the limiting distribution of Xn∗ as n → ∞.

7.63 Consider the overflow queue of Modeling Exercise 7.15.

1. Compute the condition of stability for the overflow queue.

2. Assuming the overflow queue is stable, compute the pmf of the number of cus-
tomers in the overflow queue in steady state.

7.64 Consider the following modification to the M/G/1/1 retrial queue of Sec-
tion 7.7. A new customer joins the service immediately if he finds the server free
upon his arrival. If the server is busy, the arriving customer leaves immediately with
probability c, or joins the orbit with probability 1 − c, and conducts retrials until he
is served. Let Xn and X(t) be as in Section 7.7. Derive the condition of stability and
compute the generating function of the limiting distribution of Xn and X(t). Are
they the same?

7.65 Consider the retrial queue of Section 7.7 with exp(µ) service times. Show that
the results of Section 7.7 are consistent with those of Example 6.38.

7.66 A warehouse stocks Q items. Orders for these items arrive according to a
PP(µ). The warehouse follows a (Q, Q − 1) replenishment policy with back orders
as follows: If the warehouse is not empty, the incoming demand is satisfied from the
existing stock and an order is placed with the supplier for replenishment. If the ware-
house is empty, the incoming demand is back-logged and an order is placed with the
supplier for replenishment. The lead time, i.e., the amount of time it takes for the
order to reach the warehouse from the supplier, is a random variable with distribu-
tion G(·). The lead times are iid, and orders may cross, i.e., the orders placed at the
338 QUEUEING MODELS
supplier may be received out of order. Let X(t) be the number of outstanding orders
at time t.

1. Model {X(t), t ≥ 0} as an M/G/∞ queue.

2. Compute the long run fraction of the time the warehouse is empty.
 CHAPTER 8

Renewal Processes

“Research is seeing what everyone else has seen and thinking what no one else has
thought.”
—Anonymous

8.1 Introduction

This chapter is devoted to the study of a class of stochastic processes called renewal
processes (RPs), and their applications. The RPs play several important roles in the
grand scheme of stochastic processes. First, they help remove the stringent distribu-
tional assumptions that were needed to build Markov models, namely, the geometric
distributions for the DTMCs, and the exponential distributions for the CTMCs. RPs
provide us with important tools such as the key renewal theorem (Section 8.5) to deal
with general distributions.

Second, RPs provide a unifying theoretical framework for studying the limiting
behavior of specialized stochastic processes such as the DTMCs and CTMCs. Recall
that we have seen the discrete renewal theorem in Subsection 4.5.2 on page 106 and
continuous renewal theorem in Subsection 6.10.2 on page 234. The discrete renewal
theorem was used in the study of the limiting behavior of the DTMCs, and the contin-
uous renewal theorem was used in the study of the limiting behavior of the CTMCs.
We shall see that RPs appear as embedded processes in the DTMCs and the CTMCs,
and the key-renewal theorem provides the unifying tool to obtain convergence results.

Third, RPs lead to important generalizations such as Semi-Markov processes (Sec-

tion 8.9), renewal-reward processes (Section 8.10), and regenerative processes (Sec-
tion 8.11), to name a few. Renewal-reward processes are very useful in the com-
putation of important performance measures such as long run cost and reward rates.
Regenerative processes are the most general class of processes that encompass all the
classes of stochastic processes studied in this book. Of course this generality makes
them difficult to use in computations. This difficulty is somewhat removed by the
Markov-regenerative processes studied in the next chapter. With this overview we

339
340 RENEWAL PROCESSES
begin the study of RPs.

Consider a process of events occurring over time. Let S0 = 0, and Sn be the time
of occurrence of the nth event, n ≥ 1. Assume that
0 ≤ S1 ≤ S2 ≤ S3 ≤ · · · ,
and define
Xn = Sn − Sn−1 , n ≥ 1.
Thus {Xn , n ≥ 1} is a sequence of inter-event times. Clearly, Xn ≥ 0. Since we
allow Xn = 0, multiple events can occur simultaneously. Next define
N (t) = sup{n ≥ 0 : Sn ≤ t}, t ≥ 0. (8.1)

Thus N (t) is simply the number of events up to time t. The process {N (t), t ≥ 0}
is called the counting process generated by {Xn , n ≥ 1}. A typical sample path of
{N (t), t ≥ 0} is shown in Figure 8.1. With this notation we are ready to define a
Renewal Process.

N(t)
6

0 t
S0 S1 S2 S3 S4 S5 S6

Figure 8.1 Sample path of a counting process.

Definition 8.1 Renewal Sequence and Renewal Process. The sequence {Sn , n ≥
1} is called the renewal sequence and the process {N (t), t ≥ 0} is called the renewal
process generated by {Xn , n ≥ 1} if {Xn , n ≥ 1} is a sequence on non-negative iid
random variables.

The study of the stochastic process {N (t), t ≥ 0} is renewal theory. Below we

give several examples where RPs are encountered.

Example 8.1 Poisson Process. From the definition of a Poisson process, we see
that the RP generated by a sequence of iid exp(λ) random variables is a Poisson
process with parameter λ. Thus an RP is a direct generalization of a Poisson process
INTRODUCTION 341
when the inter-event times are iid non-negative random variables that have a general
distribution.

Example 8.2 Arrival Process in a G/M/1 Queue. Let N (t) be the number of
arrivals in a G/M/1 queue up to time t. By definition (See Section 7.6.2) the inter-
arrival times {Xn , n ≥ 1} in a G/M/1 queue are iid random variables. Hence
{N (t), t ≥ 0}, the counting process generated by {Xn , n ≥ 1}, is an RP.

Example 8.3 RPs in CTMCs. Let {X(t), t ≥ 0} be a CTMC on {0, 1, 2, · · ·} with

P(X(0) = 0) = 1. Let Sn be the time of the nth entry into state 0, n ≥ 1, and let
N (t) be the number of entries into state 0 up to t. (Note that we do not count the
entry at time 0, since the process was already in state 0 at time 0.) Since Xn = Sn −
Sn−1 is the time between two successive entries into state 0, the Markov property of
the CTMC implies that {Xn , n ≥ 1} is a sequence of iid random variables. Hence
{N (t), t ≥ 0}, the counting process generated by {Xn , n ≥ 1}, is an RP.

Example 8.4 Machine Maintenance. Suppose a manufacturing process requires

continuous use of a machine. We begin with a new machine at time 0. Suppose we
follow the age-replacement policy: replace the machine in use when it fails or when
it reaches an age of T years. This policy is useful in reducing the number of on-
job failures, which tend to be more costly. Let Li be the lifetime of the ith machine
and assume that {Li , i ≥ 1} is a sequence of iid non-negative random variables.
This process produces several renewal sequences (the reader should convince him-
self/herself that these are indeed renewal sequences by showing that inter-event time
Xn = Sn − Sn−1 are iid):

1. Sn = time of the nth failure,

2. Sn = time of the nth planned replacement,
3. Sn = time of the nth replacement (planned or otherwise).

Example 8.5 RPs in an M/G/1 Queue. Let X(t) be the number of customers at
time t in an M/G/1 system. Suppose the system is initially empty, i.e., X(0) = 0.
Let Sn be the time of departure of the nth customer who leaves behind an empty
system, i.e., it is the completion time of the nth busy cycle. Since the Poisson process
has independent increments, and the service times are iid, we see that successive busy
cycles are iid, i.e., {Xn = Sn − Sn−1 , n ≥ 1} is a sequence of iid random variables.
It generates a counting process {N (t), t ≥ 0} that counts the number of busy cycles
completed by time t. It is thus an RP.
If we change the initial state of the system so that an arrival has occurred to an empty
system at time 0-, we can come up with a different RP by defining Sn as the arrival
time of the nth customer who enters an empty system. In this case we get an RP
where N (t) is the number of busy cycles that start by time t. (Note that we do not
count the initial busy cycle.) In general, we can come up with many RPs embedded
in a given stochastic process. What about a G/G/1 queue?
342 RENEWAL PROCESSES
Example 8.6 RPs in a DTMC. Following Example 8.3 we can identify RPs in a
DTMC {Xn , n ≥ 0} on {0, 1, 2, · · ·} with P(X0 = 0) = 1. Let Sn be the time of
the nth visit to state 0, n ≥ 1, and let N (t) be the number of entries into state 0 up to
t. (Note that we do not count the visit at time 0, since the process was already in state
0 at time 0.) Then, by the same argument as in Example 8.3, {N (t), t ≥ 0} is an RP.
Note that in this case renewals can occur only at integer times n = 0, 1, 2, · · · .

The above examples show how pervasive RPs are. Next we characterize an RP.

Theorem 8.1 Characterization of an RP. An RP generated by a sequence of iid

random variables {Xn , n ≥ 1} with common cdf G(·) is completely characterized
by G(·).

Proof: The renewal sequence {Sn , n ≥ 1} is clearly completely described by G.

Now, for any integer n ≥ 1, and 0 < t1 < · · · < tn and integers 0 ≤ k1 ≤ kn ≤
· · · ≤ kn , we have
P(N (t1 ) = k1 , N (t2 ) = k2 , · · · , N (tn ) = kn )
= P(Sk1 ≤ t1 , Sk1 +1 > t1 , Sk2 ≤ t2 , Sk2 +1 > t2 , · · · , Skn ≤ tn , Skn +1 > tn ).
Thus all finite dimensional joint distributions of the RP are determined by those of
the renewal sequence, which are determined by G. This proves the theorem.

The next theorem shows that the RP “renews” at time X1 = S1 , i.e., it essentially
starts anew at time S1 .

Theorem 8.2 The RP {N (t), t ≥ 0} generated by {Xn , n ≥ 1} is stochastically

identical to {N (t + X1 ) − 1, t ≥ 0}.

Proof: To show that they are stochastically identical, we have to show that their finite
dimensional distributions are identical. Now, for any integer n ≥ 1, and 0 < t1 <
· · · < tn and integers 0 ≤ k1 ≤ kn ≤ · · · ≤ kn , we have
P(N (t1 + X1 ) − 1 = k1 , N (t2 + X1 ) − 1 = k2 , · · · , N (tn + X1 ) − 1 = kn )
= P(N (t1 + X1 ) = k1 + 1, N (t2 + X1 ) = k2 + 1, · · · , N (tn + X1 ) = kn + 1)
= P(Sk1 +1 ≤ t1 + X1 , Sk1 +2 > t1 + X1 , Sk2 +1 ≤ t2 + X1 , Sk2 +2 > t2 + X1 ,
· · · , Skn +1 ≤ tn + X1 , Skn +2 > tn + X1 )
= P(Sk1 +1 − X1 ≤ t1 , Sk1 +2 − X1 > t1 , Sk2 +1 − X1 ≤ t2 , Sk2 +2 − X1 > t2 ,
· · · , Skn +1 ≤ tn + X1 , Skn +2 − X1 > tn )
= P(Sk1 ≤ t1 , Sk1 +1 > t1 , Sk2 ≤ t2 , Sk2 +1 > t2 , · · · , Skn ≤ tn , Skn +1 > tn )
= P(N (t1 ) = k1 , N (t2 ) = k2 , · · · , N (tn ) = kn ).
Here the second to the last equality follows because (S2 − X1 , S3 − X1 , · · · , Sn+1 −
X1 ) has the same distribution as (S1 , S2 , · · · , Sn ). The last equality follows from the
PROPERTIES OF N (T ) 343
proof of Theorem 8.1.

The above theorem provides a justification for calling S1 the first renewal epoch,
since the process essentially restarts at that point of time. By the same reason, we
call Sn as the nth renewal epoch.

8.2 Properties of N (t)

Now that we know how to characterize an RP, we will study it further: its transient
behavior, limiting behavior, etc. We shall use the following notation throughout this
chapter:
G(t) = P(Xn ≤ t), τ = E(Xn ), s2 = E(Xn2 ), σ 2 = Var(Xn ).
We shall assume that
G(0−) = 1, G(0+) = G(0) < 1.
Thus Xn ’s are non-negative, but not identically zero with probability 1. This assump-
tion implies that
τ > 0,
and is necessary to avoid trivialities.

Theorem 8.3
P(N (t) < ∞) = 1, t ≥ 0.

Proof: From strong law of large numbers we have

Sn
P( → τ ) = 1.
n
Since τ > 0, this implies that
P(Sn → ∞) = 1.
Thus, for 0 ≤ t < ∞,
P(N (t) = ∞) = P( lim Sn ≤ t)
n→∞
= 1 − P(Sn → ∞) = 0.
Hence the theorem follows.

The next theorem gives the exact distribution of N (t) for a given t.

Theorem 8.4 Marginal Distribution of N (t). Let

Gk (t) = P(Sk ≤ t), t ≥ 0. (8.2)
Then
pk (t) = P(N (t) = k) = Gk (t) − Gk+1 (t), t ≥ 0. (8.3)
344 RENEWAL PROCESSES
Proof: Follows along the same lines as the proof of Theorem 5.8 by using Equa-
tion 8.2.

Example 8.7 Poisson Process. Let G(t) = 1 − e−λt , t ≥ 0. Then Gk (t) is the cdf
of an Erl(k, λ) random variable. Hence
k−1
X (λt)r
Gk (t) = 1 − e−λt , t ≥ 0.
r=0
r!
Hence, from Theorem 8.4, we get
(λt)k
pk (t) = Gk (t) − Gk+1 (t) = e−λt , t ≥ 0.
k!
Thus N (t) ∼ P(λt). This is to be expected, since G(t) = 1 − e−λt implies the RP
{N (t), t ≥ 0} is a PP(λ).

The function Gk (·) of Equation 8.2 is called the k-fold convolution of G with
itself. It can be computed recursively as
Z t Z t
Gk (t) = Gk−1 (t − u)dG(u) = G(t − u)dGk−1 (u), t ≥ 0
0 0

with initial condition G0 (t) = 1 for t ≥ 0. Let G̃(s) be the Laplace-Stieltjes trans-
form (LST) of G, defined as
Z ∞
G̃(s) = e−st dG(t).
0
Since the LST of a convolution is the product of the LSTs (see Appendix E), it
follows that
G̃k (s) = G̃(s)k .
Hence
Z ∞
p̃k (s) = e−st dpk (t) = G̃k (s) − G̃k+1 (s) = G̃(s)k (1 − G̃(s)). (8.4)
0

One of the most important tools of renewal theory is the renewal argument. It is a
method of deriving an integral equation for a probabilistic quantity by conditioning
on S1 , the time of the first renewal. We explain it by deriving an equation for pk (t) =
P(N (t) = k). Now fix a k > 0 and t ≥ 0. Suppose S1 = u. If u > t, we must
have N (t) = 0. If u ≤ t, we already have one renewal at time u, and the process
renews at time u. Hence we can get k renewals by time t if and only if we get k − 1
additional renewals in this new renewal process starting at time u up to time t, which
is P(N (t − u) = k − 1). Combining these observations we get

0 if u > t,
P(N (t) = k|S1 = u) =
P(N (t − u) = k − 1) if u ≤ t.
PROPERTIES OF N (T ) 345
Hence,
Z ∞
pk (t) = P(N (t) = k|S1 = u)dG(u)
0
Z t
= P(N (t − u) = k − 1)dG(u)
0
Z t
= pk−1 (t − u)dG(u). (8.5)
0
We also have

p0 (t) = P(N (t) = 0) = P(S1 > t) = 1 − G(t). (8.6)

Unfortunately, computing pk (t) by using Equation 8.5 is no easier than using Equa-
tion 8.3. However, when the inter-event times are integer valued random variables,
the renewal argument provides a simple recursive method of computing pk (t). To see
this, let
αi = P(Xn = i), i = 0, 1, 2, · · · .
Since all renewals take place at integer time points, we study pk (n) for n =
0, 1, 2, · · · . Equation 8.6 reduces to
n
X
p0 (n) = 1 − αi , n = 0, 1, 2, · · · , (8.7)
i=0

and Equation 8.5 reduces to

n
X
pk (n) = αi pk−1 (n − i). (8.8)
i=0

The above equation can be used to compute pk (n) for increasing values of k starting
with p0 (n) of Equation 8.7.

This completes our study of the transient behavior of the RP. Next we study its
limiting behavior. Unlike in the case of DTMCs and CTMCs, where most of limiting
results were about convergence in distribution, the results in renewal theory are about
convergence with probability one (w.p. 1), or almost sure (a.s.) convergence. See
Appendix G for relevant definitions. Let N (∞) be the almost sure limit of N (t) as
t → ∞. That is,
N (∞) = lim N (t), with probability 1.
t→∞
In other word, N (∞) is the sample-path wise limit of N (t). It exists since the sample
paths of a RP are non-decreasing functions of time.

Theorem 8.5 Almost Sure Limit of N (t). Let {N (t), t ≥ 0} be an RP with com-
mon inter-event time cdf G(·), with almost sure limit N (∞).

1. G(∞) = 1 ⇒ P(N (∞) = ∞) = 1.

346 RENEWAL PROCESSES
2. G(∞) < 1 ⇒ P(N (∞) = k) = G(∞)k (1 − G(∞)), k = 0, 1, 2, · · · .

Proof: We have
N (∞) = k < ∞ ⇔ {Xn < ∞, 1 ≤ n ≤ k, Xk+1 = ∞}.
The probability of the event on the right is 0 if G(∞) = P(Xn < ∞) = 1, and
G(∞)k (1 − G(∞)) if G(∞) < 1. The theorem follows from this.

Thus if G(∞) = 1 the renewals recur infinitely often, while if G(∞) < 1, the
renewals stop occurring after a while. We use this behavior to make the following
definition:

Definition 8.2 Recurrent and Transient RP. A renewal process {N (t), t ≥ 0} is

called
1. recurrent if P(N (∞) = ∞) = 1,
2. transient if P(N (∞) = ∞) < 1.

Example 8.8 Transient and Recurrent RPs in a CTMC. Let {N (t), t ≥ 0} be

the renewal process described in Example 8.3. It counts the number of entries into
state 0 up to time t. If state 0 is recurrent, the RP {N (t), t ≥ 0} is recurrent, and if the
state 0 is transient, the RP {N (t), t ≥ 0} is transient. This is because the number of
visits to state 0 over [0, ∞) is infinity if state 0 is recurrent and finite if it is transient.

From now on we shall concentrate on recurrent RPs, i.e., we shall assume that
G(∞) = 1. In this case N (t) → ∞ with probability 1 as t → ∞. The next theorem
gives us the rate at which it approaches infinity.

Theorem 8.6 Elementary Renewal Theorem: Almost-Sure Version. Let

{N (t), t ≥ 0} be a recurrent RP with mean inter-event time τ > 0. Then
N (t) 1
→ , w.p.1. (8.9)
t τ
The right hand side is to be 0 if τ = ∞.

Proof: From the definition of the RP and Figure 8.2, we see that SN (t) ≤ t <
SN (t)+1 . Hence, when N (t) > 0,
Time
0 SN(t) t SN(t)+1

Figure 8.2 Relationship between t, SN(t) and SN(t)+1 .

SN (t) t SN (t)+1
≤ < .
N (t) N (t) N (t)
PROPERTIES OF N (T ) 347
Since the RP is recurrent, N (t) → ∞ as t → ∞. Thus, using the strong law of large
numbers, we get
SN (t) Sn
lim = lim = τ, w.p.1,
t→∞ N (t) n→∞ n

and
SN (t)+1 Sn+1 Sn+1 n + 1
lim = lim = lim = τ, w.p.1.
t→∞ N (t) n→∞ n n→∞ n+1 n
Hence
t SN (t)+1
lim sup ≤ lim = τ,
t→∞ N (t) t→∞ N (t)
and
t SN (t)
lim inf ≥ lim = τ.
t→∞ N (t) t→∞ N (t)

Hence
t t
τ ≥ lim sup ≥ lim inf ≥ τ.
t→∞ N (t) t→∞ N (t)

Thus we have
t t
lim sup = lim inf = τ.
t→∞ N (t) t→∞ N (t)
Thus N (t)/t has a limit given by
t
lim = τ.
t→∞ N (t)
Now, if τ < ∞, we can use the continuity of the function f (x) = 1/x for x > 0 to
get Equation 8.9. If τ = ∞, we first construct a new renewal process with inter-event
times given by min(Xn , T ), for a fixed T , and then let T → ∞. We leave the details
to the reader.

Theorem 8.6 makes intuitive sense because the limiting value of N (t)/t is the long
run number of renewals per unit time. Thus if the mean inter-event time is 10 min-
utes, it makes sense that in the long run we should see one renewal every 10 minutes,
or 1/10 renewals every minute.

The following theorem gives more detailed distributional information about how
the RP approaches infty.

Theorem 8.7 Central Limit Theorem for N (t). Let {N (t), t ≥ 0} be a recurrent
RP with mean inter-event time 0 < τ < ∞ and variance σ 2 < ∞. Then
! Z x −u2 /2
N (t) − t/τ e
lim P p ≤ x = Φ(x) = √ du. (8.10)
t→∞ 2
σ t/τ 3 2π
−∞

Proof: We have
P(N (t) ≥ k) = P(Sk ≤ t)
348 RENEWAL PROCESSES
and hence
!  
N (t) − t/τ k − t/τ Sk − kτ t − kτ
P p ≥ p =P √ ≤ √ . (8.11)
2
σ t/τ 3 σ 2 t/τ 3 σ k σ k
Now let t and k both grow to ∞ so that
t − kτ
√ → x,
σ k
where x is a fixed real number. The above equation implies that
k − t/τ
p → −x.
σ 2 t/τ 3
Letting k, t → ∞ appropriately in Equation 8.11, and using the central limit theorem,
we get
!  
N (t) − t/τ Sk − kτ
lim P p ≥ −x = lim P √ =≤ x Φ(x). (8.12)
t→∞ σ 2 t/τ 3 k→∞ σ k
Hence !
N (t) − t/τ
lim P p ≤ x = 1 − Φ(−x) = Φ(x).
t→∞ σ 2 t/τ 3
This completes the proof.

The above theorem says that for large t, N (t) approaches a normal random vari-
able with mean t/τ and variance σ 2 t/τ 3 .

Example 8.9 RPs in DTMCs. Let {Xn , t ≥ 0} be an irreducible, aperiodic, and

positive recurrent DTMC with X0 = 0, and consider the RP {N (t), t ≥ 0} of Exam-
ple 8.6. We see that the first renewal time S1 is the same as T̃0 , the time of the first
return to state 0, as defined in Equation 4.10. It has mean m̃0 . From Theorems 4.16
and 4.18 on pages 109 and 111, we see that
π0 = 1/m̃0 ,
where π0 is the limiting probability that the DTMC is in state 0. Now, from Theo-
rem 8.6, we see that the long run rate of visits to state 0 is given by
N (t)
lim = 1/m̃0 = π0 ,
t→∞ t
with probability 1.

Example 8.10 Machine Maintenance. Suppose a part in a machine is available

from two different suppliers, A and B. When the part fails it is replaced by a new
one from supplier A with probability .3 and supplier B with probability .7. A part
from supplier A lasts for an exponential amount of time with a mean of 8 days, and
it takes 1 day to install it. A part from supplier B lasts for an exponential amount of
THE RENEWAL FUNCTION 349
time with a mean of 5 days, and it takes 1/2 a day to install it. Installation times are
deterministic. Assume that a failure has occurred at time 0-. Compute the approxi-
mate distribution of the number of failures in the first year (not counting the one at
time 0-).

Let Xn be the time between the nth and (n − 1)st failure. {Xn , n ≥ 1} are iid
random variables with common distribution given by

1 + exp(1/8) with probability .3,
Xn ∼
0.5 + exp(1/5) with probability .7.
Let N (t) be the number of failures up to time t. Then {N (t), t ≥ 0} is an RP
generated by {Xn , n ≥ 1}. Straightforward calculations yield
τ = E(Xn ) = 6.55, σ 2 = Var(Xn ) = 39.2725.
Hence, from Theorem 8.7, N (t) is approximately normally distributed with mean
t/6.55 and variance (39.2725/6.553)t. Thus, in the first year (t = 365), the number
of failures is approximately normal with mean 55.725 and variance 51.01.

8.3 The Renewal Function

We begin with the definition.

Definition 8.3 Let {N (t), t ≥ 0} be an RP. The renewal function M (·) is defined
as
M (t) = E(N (t)), t ≥ 0. (8.13)

The next theorem gives a method of computing the renewal function.

Theorem 8.8 Renewal Function. Let {N (t), t ≥ 0} be an RP with common inter-

event cdf G. The renewal function is given by
∞
X
M (t) = Gk (t), t ≥ 0, (8.14)
k=1

where Gk (t) is as defined in Equation 8.2.

Proof: We have
∞
X
M (t) = kP(N (t) = k)
k=0
X∞
= k(Gk (t) − Gk+1 (t)) (Eq. 8.3)
k=0
X∞
= Gk (t).
k=1
350 RENEWAL PROCESSES
This proves the theorem.

The next theorem introduces an important equation known as the renewal equation,
and gives a simple expression for the LST of the renewal function defined by
Z ∞
M̃ (s) = e−st dM (t), Re(s) > 0.
0

Theorem 8.9 The Renewal Equation. The renewal function {M (t), t ≥ 0} of

a renewal process {N (t), t ≥ 0} with common inter-event cdf G(·) satisfies the
renewal equation
Z t
M (t) = G(t) + M (t − u)dG(u), t ≥ 0. (8.15)
0
The LST of the renewal function is given by
G̃(s)
M̃ (s) = . (8.16)
1 − G̃(s)

Proof: We use the renewal argument. Fix a t, and suppose S1 = u. If u > t the very
first renewal is after t, hence N (t) = 0, and hence M (t) = 0. On the other hand, if
u ≤ t, then we get one renewal at u, and a new renewal process starts at u, which
produces additional M (t − u) expected number of events up to t. Hence we have

0 if u > t,
E(N (t)|S1 = u) =
1 + M (t − u) if u ≤ t.
Hence we get
Z ∞
M (t) = E(N (t)|S1 = u)dG(u)
0
Z t
= (1 + M (t − u))dG(u)
0
Z t
= G(t) + M (t − u)dG(u).
0
This gives the renewal equation. Taking LSTs on both sides of Equation 8.15 we get,
M̃ (s) = G̃(s) + M̃ (s)G̃(s),
which yields Equation 8.16.

As in the case of pk (t), when the inter-event times are integer valued random
variables, the renewal argument provides a simple recursive method of computing
M (t). To derive this, let
αi = P(Xn = i), i = 0, 1, 2, · · · ,
THE RENEWAL FUNCTION 351
and
i
X
βi = P(Xn ≤ i) = αk .
k=0
Since all renewals take place at integer time points, it suffices to study M (n) for
n = 0, 1, 2, · · · . We leave it to the reader to show that Equation 8.15 reduces to
n
X
M (n) = βn + αk M (n − k), n = 0, 1, 2, · · · . (8.17)
k=0

The above equation can be used to compute M (n) recursively for increasing values
of n.

The renewal function plays a very important role in the study of renewal processes.
The next theorem shows why.

Theorem 8.10 The renewal function {M (t), t ≥ 0} completely characterizes the

renewal process {N (t), t ≥ 0}.

Proof: Equation 8.16 can be rewritten as

M̃ (s)
G̃(s) = . (8.18)
1 + M̃ (s)
Now, the renewal function {M (t), t ≥ 0} determines its LST M̃ (s), which deter-
mines the LST G̃(s) by the above relation, which determines the cdf G(·), which
characterizes the renewal process {N (t), t ≥ 0}, according to Theorem 8.1. Hence
the theorem follows.

Example 8.11 Renewal Function for a PP. Consider a renewal process with inter-
event time distribution G(x) = 1 − e−λx for x ≥ 0. We have G̃(s) = λ/(s + λ).
From Equation 8.16 we get
λ
M̃ (s) = .
s
Inverting this we get
M (t) = λt, t ≥ 0.
This is as expected, since the renewal process in this case is a PP(λ). Theorem 8.10
implies that if a renewal process has a linear renewal function, it must be a Poisson
process!

Example 8.12 Compute the renewal function for an RP {N (t), t ≥ 0} generated

by {Xn , n ≥ 0} with common pmf
P(Xn = 0) = 1 − α, P(Xn = 1) = α,
where 0 < α < 1.
352 RENEWAL PROCESSES
From Equation 8.17 we get
M (0) = 1 − α + (1 − α)M (0),
M (n) = 1 + (1 − α)M (n) + αM (n − 1), n ≥ 1.
The above equations can be solved to get
M (n) = (n + 1 − α)/α, n ≥ 0.
Thus
M (t) = ([t] + 1 − α)/α, t ≥ 0,
where [t] is the largest integer not exceeding t.

Next we study the asymptotic behavior of the renewal function.

Theorem 8.11 Let M (t) be the renewal function of an RP with mean inter-event
time τ > 0. Then
M (t) < ∞ for all t ≥ 0.

Proof: Let G be the cdf of the inter-event times. Since τ > 0, G(0) < 1. Thus there
is a δ > 0 such that G(δ) < 1. Define

0 if Xn < δ,
Xn∗ =
δ if Xn ≥ δ.
Let {Sn∗ , n ≥ 1} and {N ∗ (t), t ≥ 0} be the renewal sequence and the renewal
process generated by {Xn∗ , n ≥ 1}. Since Xn∗ ≤ Xn for all n ≥ 1, we see that
Sn∗ ≤ Sn for all n ≥ 1, which implies that N (t) ≤ N ∗ (t) for all t ≥ 0. Hence
M (t) = E(N (t)) ≤ E(N ∗ (t)) = M ∗ (t).
We can modify Example 8.12 slightly to get
[t/δ] + G(δ)
M ∗ (t) = < ∞.
1 − G(δ)
Hence
M (t) ≤ M ∗ (t) < ∞,
and the result follows.

Theorem 8.12 Elementary Renewal Theorem.

M (t) 1
lim = , (8.19)
t→∞ t τ
where the right hand side is to be interpreted as zero if τ = ∞.

Before we prove the theorem, it is worth noting that this theorem does not follow
from Theorem 8.6, since almost sure convergence does not imply convergence of the
THE RENEWAL FUNCTION 353
expected values. Hence we need to establish this result independently. We need the
following result first.

Theorem 8.13
E(SN (t)+1 ) = τ (M (t) + 1). (8.20)

Proof: We prove this by using renewal argument. Let

H(t) = E(SN (t)+1 ).
We have 
u if u > t,
E(SN (t)+1 |S1 = u) =
u + H(t − u) if u ≤ t.
Hence,
Z ∞
H(t) = E(SN (t)+1 |S1 = u)dG(u)
0
Z t Z ∞
= (u + H(t − u))dG(u) + udG(u)
0 t
Z ∞ Z t
= udG(u) + H(t − u)dG(u)
0 0
Z t
= τ+ H(t − u)dG(u). (8.21)
0
Taking LSTs on both sides of the above equation, we get
H̃(s) = τ + H̃(s)G̃(s).
Hence,
τ
H̃(s) =
1 − G̃(s)
" #
G̃(s)
= τ 1+
1 − G̃(s)
= τ (1 + M̃ (s)).
Inverting the above equation yields Equation 8.20.

A quick exercise in renewal argument shows that E(SN (t) ) 6= τ M (t). Thus Equa-
tion 8.20 is unusual indeed! With this result we are now ready to prove the elementary
renewal theorem.
Proof of Theorem 8.12. Assume 0 < τ < ∞. By definition, we have
SN (t)+1 > t.
Hence
E(SN (t)+1 ) > t.
354 RENEWAL PROCESSES
Using Equation 8.20, the above inequality yields
M (t) 1 1
> − .
t τ t
Hence
M (t) 1
lim inf ≥ . (8.22)
t→∞ t τ
Now, fix a 0 < T < ∞ and define
Xn′ = min(Xn , T ), n ≥ 1.
Let {N ′ (t), t ≥ 0} be an RP generated by {Xn′ , n ≥ 1}, and M ′ (t) be the corre-
sponding renewal function. Now,
SN ′ (t)+1 ≤ t + T.
Hence, taking expected values on both sides, and using Equation 8.20, we get
τ ′ (M ′ (t) + 1) ≤ t + T,
where τ ′ = E(Xn′ ). Hence we get
M ′ (t) 1 T − τ′
≤ ′+ .
t τ t
Since Xn′ ≤ Xn , we see that M (t) ≤ M ′ (t). Hence we get
M (t) 1 T − τ′
≤ ′+ .
t τ t
This implies
M (t) 1
lim sup ≤ ′.
t→∞ t τ
Now as T → ∞, τ ′ → τ. Hence letting T → ∞ on both sides af the above inequality,
we get
M (t) 1
lim sup ≤ . (8.23)
t→∞ t τ
Combining Equations 8.22 and 8.23 we get Equation 8.19. We leave the case of
τ = ∞ to the reader.

This theorem has the same intuitive explanation as Theorem 8.6. We end this sec-
tion with results about the higher moments of N (t).

Theorem 8.14 Higher Moments of N (t). Let

Mk (t) = E(N (t)(N (t) − 1) · · · (N (t) − k + 1)), k = 1, 2, · · · ; t ≥ 0.
Then
M̃k (s) = k!M̃ (s)k . (8.24)
RENEWAL-TYPE EQUATION 355
Proof: From Equation 8.4 we get
∞
X
M̃k (s) = r(r − 1) · · · (r − k + 1)p̃k (s)
r=1
X∞
= r(r − 1) · · · (r − k + 1)G̃k (s)(1 − G̃(s))
r=1
!k
G̃(s)
= k!
1 − G̃(s)
= k!M̃ (s)k .
Thus Mk (t)/k! is a k-fold convolution of M (t) with itself.

8.4 Renewal-Type Equation

In this section we study the type of integral equations that arise from the use of
renewal argument. We have already seen two instances of such equations: Equa-
tion 8.15 for M (t), and Equation 8.21 for E(SN (t)+1 ). All these equations have the
following form: Z t
H(t) = D(t) + H(t − u)dG(u), (8.25)
0
where G(·) is a cdf of a random variable with G(0−) = 0 and G(∞) = 1, D(·) is a
given function, and H is to be determined. When D(t) = G(t), the above equation is
the same as Equation 8.15, and is called the renewal equation. When D(·) is a func-
tion other than G(·), Equation 8.25 is called the renewal-type equation. Thus, when
D(t) = τ , we get the renewal-type equation for E(SN (t)+1 ), namely, Equation 8.21.

We will need to solve the renewal-type equations arising in applications. The fol-
lowing theorem gives the conditions for the existence and uniqueness of the solution
to a renewal-type equation.

Theorem 8.15 Solution of the Renewal-Type Equation. Suppose

|D(t)| < ∞ for all t ≥ 0.
Then there exists a unique solution to the renewal-type equation
Z t
H(t) = D(t) + H(t − u)dG(u),
0
such that
|H(t)| < ∞ for all t ≥ 0,
given by Z t
H(t) = D(t) + D(t − u)dM (u), (8.26)
0
356 RENEWAL PROCESSES
where M (·) is the renewal function associated with G(·).

Proof: Since M (t) < ∞ and Gk (t) is a decreasing function of k, Equation 8.14
implies that
lim Gn (t) = 0.
n→∞
We introduce the following convenient notation for convolution:
Z t
A ∗ B(t) = A(t − u)dB(u).
0

By recursive use of the renewal type equation, and writing H for H(t) for compact-
ness, we get
H = D + H ∗ G = D + (D + H ∗ G) ∗ G = D + D ∗ G + H ∗ G2 = · · ·
n−1
X
= D+D∗ Gk + H ∗ Gn . (8.27)
k=1

Letting n → ∞ and using Gn → 0 we get

∞
X
H =D+D∗ Gk = D + D ∗ M,
k=1

which is Equation 8.26. Thus we have shown that H as given in Equation 8.26 is a
solution to Equation 8.25.
Since D is assumed to be bounded we get
c = sup |D(x)| < ∞.
0≤x≤t

Then Equation 8.26 yields

|H(t)| ≤ |D(t)| + |D ∗ M (t)| ≤ c + cM (t) < ∞.
This shows the boundedness of H(t).
To show uniqueness, suppose H1 and H2 are two solutions to Equation 8.25. Since
both H1 and H2 satisfy Equation 8.27, we get
H = H1 − H2 = (H1 − H2 ) ∗ Gn = H ∗ Gn .
Letting n → ∞, we can use the boundedness of H1 and H2 to get
H = lim H ∗ Gn = 0.
n→∞

Hence we must have H1 = H2 . This shows uniqueness.

We illustrate with an example below.

Example 8.13 Two-State Machine. We revisit our two state machine with iid
RENEWAL-TYPE EQUATION 357
exp(µ) up-times. However, when the machine fails after an up-time U , the ensu-
ing down time is cU , for a fixed c > 0. Suppose the machine is up at time 0. Let
X(t) = 1 if the machine is up at time t, and 0 otherwise. Compute
H(t) = P(X(t) = 1).

We approach the problem by first deriving a renewal-type equation for H(t) by

using the renewal argument.
Let U1 ∼ exp(µ) be the first up-time, and D1 = cU1 be the following down
time. The machine becomes as good as new at time S1 = U1 + D1 = (1 + c)U1 ∼
exp(µ/(1 + c)). Now fix a t ≥ 0 and suppose S1 = u. If u > t, then X(t) = 1 if
U1 > t. If u ≤ t, the machine regenerates at time u, hence it will be up at time t with
probability H(t − u). Combining these observations, we get

P(U1 > t|S1 = u) if u > t,
P(X(t) = 1|S1 = u) =
H(t − u) if u ≤ t.
Hence,
Z ∞
H(t) = P(X(t) = 1|S1 = u)dG(u)
0
Z t Z ∞
= H(t − u)dG(u) + P(U1 > t|S1 = u)dG(u), (8.28)
0 t
where G is the cdf of S1 . Since P(U1 > t|S1 = u) = 0 if u ≤ t, we get
Z ∞ Z ∞
P(U1 > t|S1 = u)dG(u) = P(U1 > t|S1 = u)dG(u) = P(U1 > t) = e−µt .
t 0
Substituting in Equation 8.28, we get
Z t
−µt
H(t) = e + H(t − u)dG(u),
0

which is a renewal-type equation with D(t) = e−µt , and G being the cdf of an
exp(µ/(1 + c)) random variable. The RP corresponding to this G is a PP(µ/(1 + c)).
Hence the renewal function is given by (see Example 8.11)
µ
M (t) = t.
1+c
Hence the solution given by Equation 8.26 reduces to
Z t
µ 1 + ce−µt
H(t) = e−µt + e−µ(t−u) du = .
0 1+c 1+c
Note that we could not have derived the above expression by the method of CTMCs
since {X(t), t ≥ 0} is not a CTMC due the dependence of the up and down times.

The solution in Equation 8.26 is not easy to use in practice since, unlike in the
previous example, M (t) is generally not explicitly known. The next theorem gives a
transform solution that can be more useful in practice.
358 RENEWAL PROCESSES
Theorem 8.16 If D(·) has an LST D̃(s), then H(·) has an LST H̃(s) given by
D̃(s)
H̃(s) = . (8.29)
1 − G̃(s)

Proof: Follows by taking LST on both sides of Equation 8.25.

We illustrate with an example. It is an unusual application of renewal-type equa-

tions, since generally D(t) is known, and H(t) is unknown in Equation 8.25. How-
ever, in the following example, H(t) is known, and D(t) and G(t) are unknown!

Example 8.14 Busy Period in an M/G/∞ Queue. Consider an M/G/∞ queue

where customers arrive according to a PP(λ) and require iid service times with com-
mon cdf B(·). Let X(t) be the number of customers in the system at time t. Now
suppose a customer enters an empty system at time 0. Compute the cdf F (·) of the
busy period of the system, defined as
T = min{t ≥ 0 : X(t) = 0}.

Define Sn be the nth time when a customer enters an empty system. Since the
arrival process is Poisson, and the service times are iid, we see that {Sn , n ≥ 1} is a
renewal sequence. S1 is called the busy cycle. Let H(t) = P(X(t) = 0). Using the
renewal argument, we get

P(T ≤ t|S1 = u) if u > t,
P(X(t) = 0|S1 = u) =
H(t − u) if u ≤ t.
Hence, using G as the cdf of S1 , we get
Z t
H(t) = P(X(t) = 0|S1 = u)dG(u)
0
Z t Z ∞
= H(t − u)dG(u) + P(T ≤ t|S1 = u)dG(u). (8.30)
0 t
Since P(T ≤ t|S1 = u) = 1 if u ≤ t, we get
Z ∞ Z ∞ Z t
P(T ≤ t|S1 = u)dG(u) = P(T ≤ t|S1 = u)dG(u) − dG(u)
t 0 0
= P(T ≤ t) − G(t).
Substituting in Equation 8.28, we get
Z t
H(t) = P(T ≤ t) − G(t) + H(t − u)dG(u). (8.31)
0
We see that S1 = T + M where M ∼ exp(λ) is the time until a customer arrives to
the system once it becomes empty at time T , and is independent of T . Since F is the
cdf of T and G is cdf of S1 , we see that
λ
G̃(s) = F̃ (s) .
s+λ
KEY RENEWAL THEOREM 359
Also, from the results in Section 7.8, and using the fact that the M/G/∞ queue starts
with an arrival to an empty queue at time 0, we see that
Z t
H(t) = exp(−λ (1 − B(u))du)B(t).
0

Let H̃(s) be the LST of H(·). Taking the LST of Equation 8.31 we get
λ λ
H̃(s) = F̃ (s) − F̃ (s) + H̃(s)F̃ (s) .
s+λ s+λ
Solving for F̃ (s) we get
(s + λ)H̃(s)
F̃ (s) = ,
s + λH̃(s)
which is the result we desired. Differentiating F̃ (s) with respect to s and using
lim H̃(s) = H(∞) = e−λτ
s→0
we get
eλτ − 1
E(T ) = .
λ

8.5 Key Renewal Theorem

In the previous section we saw several examples of the renewal type equation, and
methods of solving it. In general explicit solutions to renewal type equations are hard
to come by. Hence in this section we study the asymptotic properties of the solution
H(t) as t → ∞. We have done such asymptotic analysis for a discrete renewal type
equation in our study of DTMCs (Equation 4.24 on page 107), and a continuous re-
newal type equation in the study of CTMCs (Equation 6.53 on page 234). In this
section we shall deal with the general case. We begin with a definition.

Definition 8.4 Periodicity. A non-negative, non-defective random variable X is

called periodic (or arithmetic, or lattice) if there is a d > 0 such that
∞
X
P(X = kd) = 1.
k=0

The largest such d is called the period (or span) of X. If there is no such d, X is
called aperiodic (or non-arithmetic, or non-lattice).

We say that a distribution of X is periodic, (or arithmetic, or lattice) if X is pe-

riodic. Similarly we say that an RP is aperiodic if it is generated by aperiodic inter-
event times. Note that all continuous random variables on [0, ∞) are aperiodic, and
all non-negative integer valued random variables are periodic. However, a discrete
random variable is not necessarily periodic. Thus a random variable taking values
360 RENEWAL PROCESSES
√ √
{1, 2} is aperiodic, but one taking values {0, 2} is periodic. (What about one that
takes values {e, π}?) With the help of this concept, we shall prove the main result in
the next theorem.

Theorem 8.17 Key Renewal Theorem. Let H be a solution to the following re-
newal type equation
Z t
H(t) = D(t) + H(t − u)dG(u). (8.32)
0
Suppose D is a difference of two non-negative bounded monotone functions and
Z ∞
|D(u)|du < ∞. (8.33)
0

1. If G is aperiodic with mean τ > 0,

∞
1
Z
lim H(t) = D(u)du. (8.34)
t→∞ τ 0
2. If G is periodic with period d and mean τ > 0,
∞
dX
lim H(kd + x) = D(kd + x). (8.35)
k→∞ τ
k=0

The proof of this theorem is beyond the scope of this book, and we omit it. The
hard part is proving that the limit exists. If we assume the limit exists, evaluating
the limits is relatively easy, as we have done in the proofs of the discrete renewal
theorem (Theorem 4.14 on page 107) and the continuous renewal theorem (Theo-
rem 6.20 on page 234). Feller proves the result under the assumption that D is a
“directly Riemann integrable” function. The condition on D assumed above is a suf-
ficient condition for direct Riemann integrability, and is adequate for our purposes.
We refer the readers to Feller(1971), Kohlas (1982), or Heyman and Sobel (1982) for
proofs. We shall refer to key renewal theorem as KRT from now on.

In the remainder of this section we shall illustrate the usefulness of KRT by means
of several examples. As a first application, we shall prove the following theorem.

Theorem 8.18 Blackwell’s Renewal Theorem. Let M (·) be the renewal function
of an RP with mean inter-renewal time τ > 0.

1. If the RP is aperiodic,
h
lim [M (t + h) − M (t)] = , h ≥ 0. (8.36)
t→∞ τ
2. If the RP is periodic with period d,
kd
lim [M (t + kd) − M (t)] = , k = 0, 1, 2, · · · . (8.37)
t→∞ τ
KEY RENEWAL THEOREM 361
Proof: We consider the aperiodic case. For a given h ≥ 0, consider the renewal type
equation 8.25 with the following D function:

1 if 0 ≤ t ≤ h,
D(t) =
0 if t > h.
The solution is given by Equation 8.26. For t > h this reduces to
Z t
H(t) = dM (u) = M (t) − M (t − h).
t−h

From the KRT we get

lim [M (t + h) − M (t)] = lim H(t + h)
t→∞ t→∞
∞
1 h
Z
= lim H(t) = D(u)du = .
t→∞ τ 0 τ
This proves Equation 8.36. Equation 8.37 follows similarly.

It is interesting to note that one can prove that the above theorem is in fact equiv-
alent to the KRT, although the proof is not simple!

Example 8.15 We verify Blackwell’s renewal theorem for the two renewal func-
tions we derived in Examples 8.11 and 8.12. In Example 8.11 we have a Poisson
process, which is an aperiodic RP with M (t) = λt. In this case the mean inter-
renewal time is τ = 1/λ. We get
h
lim [M (t + h) − M (t)] = lim λh = ,
t→∞ t→∞ τ
thus verifying Equation 8.36. In Example 8.12 we have a periodic RP with period 1,
with M (t) = ([t] + 1 − α)/α. In this case the mean inter-renewal time is τ = α.
Thus, if h is a nonnegative integer,
[t + h] − [t] h
lim [M (t + h) − M (t)] = lim = .
t→∞ t→∞ α τ
If h is not an integer, the above limit does not exist. This verifies Equation 8.37.

Example 8.16 Asymptotic Behavior of M (t). Let M (t) be the renewal function
of an aperiodic RP with inter-event time with finite variance (σ 2 < ∞). Show that
t σ2 − τ 2
M (t) = + + o(1), (8.38)
τ 2τ 2
where o(1) is a function that approaches 0 as t → ∞.

From Theorem 8.12 we know that

M (t) 1
lim = .
t→∞ t τ
362 RENEWAL PROCESSES
We now derive a renewal type equation for
H(t) = M (t) − t/τ.
Using renewal argument we get

−t/τ if u > t
E(N (t) − t/τ |S1 = u) =
1 + M (t − u) − t/τ if u ≤ t

−t/τ if u > t
=
H(t − u) + 1 − u/τ if u ≤ t.
Hence we get
Z ∞
H(t) = E(N (t) − t/τ |S1 = u)dG(u)
0
Z t
= D(t) + H(t − u)dG(u),
0
where Z t Z ∞
D(t) = (1 − u/τ )dG(u) − (t/τ )dG(u).
0 t
This can be rearranged in the following form
1 ∞
Z
D(t) = (1 − G(u))du − (1 − G(t)) = D1 (t) − D2 (t).
τ t
Thus D(t) is a difference of two monotone functions, and we have
Z ∞ Z ∞ Z ∞
1
D1 (t)dt = (1 − G(u))dudt
0 t=0 τ u=t
1 ∞
Z Z u
= (1 − G(u))dtdu
τ u=0 t=0
Z ∞
1
= u(1 − G(u))du
τ u=0
σ2 + τ 2
= < ∞.
2τ
Also, Z ∞ Z ∞
D2 (t)dt = (1 − G(t))dt = τ < ∞.
0 0
Hence
∞
σ2 + τ 2 σ2 − τ 2
Z
D(t)dt = −τ = .
0 2τ 2τ
We can now apply the aperiodic KRT to get
σ2 − τ 2
 
t
lim M (t) − = .
t→∞ τ 2τ 2
This implies Equation 8.38. Note that if the coefficient of variation σ 2 /τ 2 < 1, then
t/τ overestimates M (t) for large t, and if σ 2 /τ 2 > 1, then t/τ underestimates M (t)
RECURRENCE TIMES 363
for large t. For the exponential inter-event times the coefficient of variation is one. In
this case t/τ is the exact estimate of M (t), as the RP is a PP in this case.

8.6 Recurrence Times

For an aperiodic RP {N (t), t ≥ 0}, let

A(t) = t − SN (t) , t ≥ 0,
B(t) = SN (t)+1 − t, t ≥ 0,
C(t) = SN (t)+1 − SN (t) = XN (t)+1 = A(t) + B(t), t ≥ 0.
Thus A(t) is the total time elapsed since the most recent renewal at or before t; it is
called backward recurrence time or current life, or age. B(t) is the time from t until
the first renewal after t; it is called the forward recurrence time, or remaining life,
or excess life. C(t) is the length of the inter-renewal interval that covers time t; it is
called the total life. Figure 8.3 shows the relationships between A(t), B(t), and C(t)
graphically.
The stochastic process {A(t), t ≥ 0} is called the age process, {B(t), t ≥ 0} is

A(t) B(t)

SN(t) t SN(t)+1

C(t)

Figure 8.3 The random variables A(t), B(t), and C(t).

called the remaining life (or excess life) process, and {C(t), t ≥ 0} is called the total
life process. Figures 8.4, 8.5, and 8.6 show the sample paths of the three stochastic

A(t)

t
S0 S1 S2 S3 S4

Figure 8.4 A typical sample path of the age process.

processes. The age process increases linearly with rate 1, jumps down to zero at ev-
ery renewal epoch Sn . The remaining life process starts at X1 , and decreases linearly
at rate 1. When it reaches zero at time Sn , it jumps up to Xn+1 . The total life pro-
cess has piecewise constant sample paths with upward or downward jumps at Sn . We
364 RENEWAL PROCESSES
B(t)

t
S0 S1 S2 S3 S4

Figure 8.5 A typical sample path of the excess life process.

C(t)

t
S0 S1 S2 S3 S4

Figure 8.6 A typical sample path of the total life process.

shall use the renewal argument and the KRT to study these three stochastic processes.

Theorem 8.19 Remaining Life Process. Let B(t) be the excess life at time t in an
aperiodic RP with inter-event time distribution G. Then, for a given x > 0,
H(t) = P(B(t) > x), t ≥ 0,
satisfies the renewal type equation
Z t
H(t) = 1 − G(x + t) + H(t − u)dG(u), (8.39)
0
and ∞
1
Z
lim P(B(t) > x) = (1 − G(u))du. (8.40)
t→∞ τ x

Proof: Conditioning on S1 we get


 H(t − u) if u ≤ t
P(B(t) > x|S1 = u) = 0 if t < u ≤ t + x
1 if u > t + x.


Hence we get
Z ∞
H(t) = P(B(t) > x|S1 = u)dG(u)
0
RECURRENCE TIMES 365
Z t
= 1 − G(x + t) + H(t − u)dG(u),
0

which is Equation 8.39. Now D(t) = 1 − G(x + t) is monotone, and satisfies the
condition in Equation 8.33. Since G is assumed to be aperiodic, we can use KRT to
get
1 ∞ 1 ∞
Z Z
lim H(t) = (1 − G(x + u))du = (1 − G(u))du,
t→∞ τ 0 τ x
which gives Equation 8.40.

Next we give the limiting distribution of the age process.

Theorem 8.20 Age Process. Let A(t) be the age at time t in an aperiodic RP with
inter-event time distribution G. Its limiting distribution is given by
1 ∞
Z
lim P(A(t) ≥ x) = (1 − G(u))du. (8.41)
t→∞ τ x

Proof: It is possible to prove this theorem by first deriving a renewal type equation
for H(t) = P(A(t) ≥ x) and then using the KRT. We leave that to the reader and
show an alternate method here. For t > x, we have
{A(t) ≥ x} ⇔ {No renewals in (t − x, t]} ⇔ {B(t − x) > x}.
Hence
lim P(A(t) ≥ x) = lim P(B(t − x) > x)
t→∞ t→∞
= lim P(B(t) > x)
t→∞
1 ∞
Z
= (1 − G(u))du,
τ x
which is Equation 8.41.

Note that in the limit, A(t) and B(t) are continuous random variables with density
(1 − G(u))/τ , even if the inter-event times may be discrete (but aperiodic). Thus
the presence of strict or weak inequalities in Equations 8.40 and 8.41 have do not
make any difference. In the limit the age and excess life are identically distributed!
However, they are not independent, as shown in the next theorem.

Theorem 8.21 Joint Distribution of Age and Excess Life. Suppose the hypotheses
of Theorems 8.19 and 8.20 hold. Then
1 ∞
Z
lim P(A(t) ≥ y, B(t) > x) = (1 − G(u))du. (8.42)
t→∞ τ x+y

Proof: For t > y we have

{A(t) ≥ y, B(t) > x} ⇔ {No renewals in (t − y, t + x]} ⇔ {B(t − y) > x + y}.
366 RENEWAL PROCESSES
Hence
lim P(A(t) ≥ y, B(t) > x) = lim P(B(t − y) > x + y)
t→∞ t→∞
= lim P(B(t) > x + y)
t→∞
1 ∞
Z
= (1 − G(u))du,
τ x+y
which is Equation 8.42.

The complementary cdf given in Equation 8.40 occurs frequently in the study of
renewal processes. Hence we introduce the following notation:
1 x
Z
Ge (x) = (1 − G(u))du. (8.43)
τ 0
R∞
Ge is a proper distribution of a continuous random variable on [0, ∞), since 0 (1 −
G(u))du = τ. It is called the equilibrium distribution corresponding to G. Let Xe be
a random variable with cdf Ge . One can show that
Z ∞
σ2 + τ 2
E(Xe ) = (1 − Ge (u))du = . (8.44)
0 2τ
Unfortunately, since convergence in distribution does not imply convergence of the
means, and we cannot conclude
σ2 + τ 2
lim E(A(t)) = lim E(B(t)) =
t→∞ t→∞ 2τ
based on Theorems 8.19 and 8.20. We can, however, show the validity of the above
limit directly by using KRT, as shown in the next theorem.

Theorem 8.22 Limiting Mean Excess Life. Suppose the hypothesis of Theo-
rems 8.19 holds, and σ 2 < ∞. Then
σ2 + τ 2
lim E(B(t)) = . (8.45)
t→∞ 2τ
Proof: Let H(t) = E(B(t)). Conditioning on S1 we get

H(t − u) if u ≤ t,
E(B(t)|S1 = u) =
u−t if u > t.
Hence we get
Z ∞
H(t) = E(B(t)|S1 = u)dG(u)
0
Z t
= D(t) + H(t − u)dG(u),
0
where Z ∞
D(t) = (u − t)dG(u).
t
DELAYED RENEWAL PROCESSES 367
Now D is monotone since
d
D(t) = −(1 − G(t)) ≤ 0.
dt
Also, we can show that
Z ∞
D(u)du = (σ 2 + τ 2 )/2 < ∞.
0
Hence the KRT can be applied to get
∞
1 σ2 + τ 2
Z
lim E(B(t)) = D(u)du = .
t→∞ τ 0 2τ
This proves the theorem.

We can derive the next two results in an analogous manner.

Theorem 8.23 Limiting Mean Age. Suppose the hypothesis of Theorem 8.19
holds, and σ 2 < ∞. Then
σ2 + τ 2
lim E(A(t)) = . (8.46)
t→∞ 2τ
Theorem 8.24 Limiting Mean Total Life. Suppose the hypothesis of Theorem 8.19
holds, and σ 2 < ∞. Then
σ2 + τ 2
lim E(C(t)) = . (8.47)
t→∞ τ

Equation 8.47 requires some fine tuning of our intuition. Since C(t) = XN (t)+1 ,
the above theorem implies that
σ2 + τ 2
lim E(XN (t)+1 ) = ≥ τ = E(Xn ).
t→∞ τ
Thus, for large t, the inter-renewal time covering t, namely, XN (t)+1 , is longer in
mean than a generic inter-renewal time, say Xn . This counter-intuitive fact is called
the inspection paradox. One way to rationalize this “paradox” is to think of picking
a t uniformly over a very long interval [0, T ] over which we have observed an RP.
Then it seems plausible the probability that our randomly picked t will lie in a given
inter-renewal interval is directly proportional to the length of that interval. Hence
such a random t is more likely to fall in a longer interval. This fact is quantified by
Equation 8.47.

8.7 Delayed Renewal Processes

In this section we study a seemingly trivial but a useful generalization of an RP. We

begin with a definition.
368 RENEWAL PROCESSES
Definition 8.5 Delayed Renewal Process. A counting process {N (t), t ≥ 0} gen-
erated by {Xn , n ≥ 1} as defined by Equation 8.1 is called a delayed RP if X1 has
cdf F , {Xn , n ≥ 2} is a sequence of iid random variables with common cdf G, and
they are independent of X1 .

To distinguish an RP from a delayed RP we use the notation {N D (t), t ≥ 0} to

denote a delayed RP. Clearly, if F = G, the delayed renewal process reduces to the
standard renewal process. Also, the shifted delayed RP {N D (t + X1 ) − 1, t ≥ 0} is
a standard RP. We illustrate with examples where we encounter delayed RPs.

Example 8.17 Let {N (t), t ≥ 0} be a standard RP. For a fixed s > 0 define
Ns (t) = N (s + t) − N (s). Then {Ns (t), t ≥ 0} is a delayed RP. Here F is the
distribution of B(s), the excess life at time s in the original RP.

Example 8.18 Delayed RPs in CTMCs. Let {X(t), t ≥ 0} be an irreducible and

recurrent CTMC on state-space {0, 1, 2, · · ·}. Let Nj (t) be the number of entries
into state j up to t. Following the argument in Example 8.3 we see that, if X(0) = j,
{Nj (t), t ≥ 0} is an RP. If X(0) 6= j, it is a delayed renewal process, since the time
of the first entry into state j has a different distribution than the subsequent inter-visit
times.

Example 8.19 Delayed RPs in an M/G/1 Queue. Let X(t) be the number of
customers at time t in an M/G/1 system. Let N (t) be the number of busy cycles
completed by time t. We saw in Example 8.5 that {N (t), t ≥ 0} is an RP if the
system starts empty, that is, if X(0) = 0. For any other initial distribution, {N (t), t ≥
0} is a delayed RP.

In our study of the delayed renewal process we shall assume that

G(0) < 1, F (∞) = G(∞) = 1. (8.48)
The delayed renewal process inherits almost all the properties of the standard renewal
process. We leave many of them to the reader to prove.

Next we study the renewal function for a delayed RP defined as

M D (t) = E(N D (t)), t ≥ 0.
Let M (t) be the renewal function for the standard RP. The next theorem gives the
main result about M D (t).

Theorem 8.25 The Renewal Function for the Delayed RP. The renewal function
{M D (t), t ≥ 0} of a renewal process {N D (t), t ≥ 0} satisfies the integral equation
Z t
D
M (t) = F (t) + M (t − u)dG(u), t ≥ 0. (8.49)
0
DELAYED RENEWAL PROCESSES 369
Its LST is given by
F̃ (s)
M̃ D (s) = . (8.50)
1 − G̃(s)

Proof: Conditioning on X1 we get


D 0 if u > t,
E(N (t)|S1 = u) =
1 + M (t − u) if u ≤ t.
Hence we get
Z ∞
M D (t) = E(N D (t)|S1 = u)dF (u)
0
Z t
= (1 + M (t − u))dF (u)
0
Z t
= F (t) + M (t − u)dF (u).
0
This gives Equation 8.49. Taking LSTs on both sides of it we get,
M˜D (s) = F̃ (s) + M̃ (s)G̃(s).
Substituting for M̃ (s) from Equation 8.16, we get Equation 8.50.

The renewal function {M D (t), t ≥ 0} also inherits almost all the properties of the
renewal function of a standard RP, except that its does not uniquely characterize the
delayed RP.

When we use renewal argument in a delayed RP, we generally get an equation of

the following type:
Z t
H D (t) = C(t) + H(t − x)dF (x), t ≥ 0. (8.51)
0

This is not a renewal type equation, since H D appears on the left, but not on the
right. The next theorem gives a result about the limiting behavior of the function H D
satisfying the above equation.

Theorem 8.26 Let C(t) and H(t) be bounded functions with finite limits as t →
∞. Let H D be given as in Equation 8.51, where F is a cdf of a non-negative non-
defective random variable. Then
lim H D (t) = lim C(t) + lim H(t). (8.52)
t→∞ t→∞ t→∞

Proof: Since H(·) is a bounded function with a finite limit h (say), we see that the
functions Ht (·) defined by

H(t − x) if 0 ≤ x ≤ t
Ht (x) =
0 if x > t
370 RENEWAL PROCESSES
form a sequence of bounded functions with
lim Ht (x) = h, x ≥ 0.
t→∞

Hence by bounded convergence theorem

Z t Z ∞ Z ∞
lim H(t − x)dF (x) = lim Ht (x)dF (x) = hdF (x) = h.
t→∞ 0 t→∞ 0 0

The theorem follows from this.

As an application of the above theorem we study the asymptotic behavior of

M D (t) the following example.

Example 8.20 Asymptotic Behavior of M D (t). Let M D (t) be the renewal func-
tion of a delayed RP. Let
τF = E(X1 ), σF2 = Var(X1 ) < ∞,
τ = E(Xn ), σ 2 = Var(Xn ) < ∞, n ≥ 2.
Show that
t σ 2 + τ 2 − 2τ τF
M D (t) = + + o(1), (8.53)
τ 2τ 2
where o(1) is a function that approaches 0 as t → ∞.

We first derive an integral equation for

H D (t) = M D (t) − t/τ.
Using renewal argument we get

−t/τ if u > t,
E(N D (t) − t/τ |S1 = u) =
1 + M (t − u) − t/τ if u ≤ t.
Hence, after rearrangements, we get
Z t
H D (t) = C(t) + H(t − u)dG(u),
0
where
H(t) = M (t) − t/τ,
and Z t Z ∞
C(t) = (1 − u/τ )dF (u) − (t/τ )dF (u).
0 t
Now, from Example 8.16 we have
σ2 − τ 2
lim H(t) = ,
t→∞ 2τ 2
and
lim C(t) = 1 − τF /τ.
t→∞
DELAYED RENEWAL PROCESSES 371
We can now apply Theorem 8.26 to get
σ2 − τ 2 τF
lim H D (t) = +1− .
t→∞ 2τ 2 τ
This implies Equation 8.53.

Next we study a special case of a delayed RP called the equilibrium renewal pro-
cess as defined below.

Definition 8.6 Equilibrium Renewal Process. A delayed RP is called an equilib-

rium RP if the cdf F is related to G as follows:
1 x
Z
F (x) = Ge (x) = (1 − G(u))du, x ≥ 0.
τ 0

An equilibrium RP is denoted by {N e (t), t ≥ 0}. Recall that Ge is the limiting

distribution of the excess life in a standard renewal process, see Theorem 8.19. Thus
{N e (t), t ≥ 0} can be thought of as the limit of the shifted RP {Ns (t), t ≥ 0} of
Example 8.17 as s → ∞. We prove two important properties of the equilibrium RP
below.

Theorem 8.27 Renewal Function of an Equilibrium RP. Let {N e (t), t ≥ 0} be

an equilibrium RP as in Definition 8.6. Then
t
M e (t) = E(N e (t)) = , t ≥ 0. (8.54)
τ
Proof: From the definition of an equilibrium RP, we get
1 − G̃(s)
F̃ (s) = G̃e (s) = .
τs
Substituting in Equation 8.50, we get
F̃ (s) 1 1 − G̃(s) 1
M̃ e (s) = = = .
1 − G̃(s) τ s 1 − G̃(s) τ s
Inverting this we get Equation 8.54.

Note that M e (t) is a linear function of t, but {N e (t), t ≥ 0} is not a PP. Does this
contradict our statement in Example 8.11 that linear renewal function implies that
the RP is a PP? Not at all, since {N e (t), t ≥ 0} is a delayed RP, and not an RP. If it
was an RP, i.e., if Ge = G, then it would be a PP. This indeed is the case, since it is
possible to show that G = Ge if and only if G is exponential!

Theorem 8.28 Excess Life in an Equilibrium RP. Let B e (t) be the excess life at
time t in an equilibrium RP as in Definition 8.6. Then,
P(B e (t) ≤ x) = Ge (x), t ≥ 0, x ≥ 0.
372 RENEWAL PROCESSES
Proof: Let H e (t) = P(B e (t) > x) and H(t) = P(B(t) > x) where B(t) is the
excess life in an RP with common inter-event cdf G. Using Ge as the cdf of S1 and
following the proof of Theorem 8.19 we get
Z t
e
H (t) = 1 − Ge (x + t) + H(t − u)dGe (u).
0
Taking the LST on both sides, we get
H̃ e (s) = 1 − c(s) + H̃(s)G̃e (s), (8.55)
where Z ∞
c(s) = e−st dGe (x + t).
0
On the other hand, from Equation 8.39, we have
Z t
H(t) = 1 − G(x + t) + H(t − u)dG(u).
0
Taking LST of the above equation we get
1 − b(s)
H̃(s) = , (8.56)
1 − G̃(s)
where Z ∞
b(s) = e−st dG(x + t).
0
Taking into account the jump of size Ge (x) at t = 0 in the function Ge (x + t), we
can show that
1 − b(s)
c(s) = Ge (x) + . (8.57)
sτ
Substituting Equation 8.57 and 8.56 in 8.55, and simplifying, we get
H̃ e (s) = 1 − Ge (x).
Hence we must have
H e (t) = 1 − Ge (x),
for all t ≥ 0. This proves the theorem.

Thus the distribution of the excess life in an equilibrium RP does not change with
time. This is another manifestation of the “equilibrium”! We leave it to the reader to
derive the next result from this.

Theorem 8.29 {N e (t), t ≥ 0} has stationary increments.

An interesting application of an equilibrium RP to G/M/∞ queue is described in

Computational Exercise 8.52.
ALTERNATING RENEWAL PROCESSES 373
8.8 Alternating Renewal Processes

Consider a stochastic process {X(t), t ≥ 0} with state-space {0, 1}. Suppose the
process starts in state 1 (also called the “up” state). It stays in that state for U1 amount
of time and then jumps to state 0 (also called the “down” state). It stays in state 0 for
D1 amount of time and then goes back to state 1. This process repeats forever, with
Un being the nth up-time, and Dn the nth down-time. The nth up time followed by
the nth down time is called the nth cycle. A sample path of such a process is shown
in Figure 8.7.

Z(t)

Up

Down

t
U1 D1 U2 D2

Figure 8.7 A typical sample path of an alternating renewal process.

Definition 8.7 Alternating Renewal Process. The stochastic process {X(t), t ≥

0} as described above is called an alternating renewal process if {(Un , Dn ), n ≥ 1}
is a sequence of iid bivariate random variables.

We abbreviate “alternating renewal process” as ARP. Note that an ARP does not
count any events even though it has the term “renewal process” in its name. The next
theorem gives the main result about the ARPs.

Theorem 8.30 Distribution of ARP. Suppose the ARP {X(t), t ≥ 0} starts in

state 1 at time zero. Then H(t) = P(X(t) = 1) satisfies the renewal equation
Z t
H(t) = P(U1 > t) + H(t − u)dG(u), (8.58)
0

where G(x) = P(U1 + D1 ≤ x). If G is aperiodic

E(U1 )
lim P(X(t) = 1) = . (8.59)
t→∞ E(U1 ) + E(D1 )
If G is periodic with period d, then the above limit holds if t = nd and n → ∞.

Proof: Let Sn be the nth time the ARP enters state 1, i.e.,
n
X
Sn = (Ui + Di ), n ≥ 1.
i=1
374 RENEWAL PROCESSES
Then {Sn , n ≥ 1} is a renewal sequence. Since the ARP renews at time S1 =
U1 + D1 , we get

H(t − u) if u ≤ t,
P(X(t) = 1|S1 = u) =
P(U1 > t|S1 = u) if u > t.
Using G(u) = P(S1 ≤ u), we get
Z ∞
H(t) = P(X(t) = 1|S1 = u)dG(u)
0
Z t Z ∞
= H(t − u)dG(u) + P(U1 > t|S1 = u)dG(u). (8.60)
0 t

We use the same trick as in Example 8.13: since P(U1 > t|S1 = u) = 0 if u ≤ t, we
get
Z ∞ Z ∞
P(U1 > t|S1 = u)dG(u) = P(U1 > t|S1 = u)dG(u) = P(U1 > t).
t 0

Substituting in Equation 8.60 we get Equation 8.58. Now, P(U1 > t) is bounded and
monotone, and Z ∞
P(U1 > u)du = E(U1 ) < ∞.
0
Hence we can use KRT to get, assuming S1 is aperiodic,
E(U1 ) E(U1 )
lim H(t) = lim P(X(t) = 1) = = .
t→∞ t→∞ E(S1 ) E(U1 ) + E(D1 )
This proves the theorem. The periodic case follows similarly.

The above theorem is intuitively obvious if one interprets the limiting probability
that X(t) = 1 as the fraction of the time the ARP is up. Thus, if the successive up-
times are 30 minutes long on the average, and the down-times are 10 minutes long
on the average, then it is reasonable to expect that the ARP will be up 75% of the
time in the long run. What is non-intuitive about the theorem is that it is valid even
if the Un and Dn are dependent random variables. All we need to assume is that
successive cycles of the ARP are independent. This fact makes the ARP a powerful
tool. We illustrate its power with several examples.

Example 8.21 Two-State Machine. The CTMC {X(t), t ≥ 0} developed in Ex-

ample 6.4 on page 195 to describe a two state machine is an ARP with Un ∼ exp(µ)
and Dn ∼ exp(λ). In this example Un and Dn are also independent. Since S1 =
U1 + D1 is aperiodic, we can use Equation 8.59 to get
E(U1 ) λ
lim P(X(t) = 1) = = ,
t→∞ E(U1 ) + E(D1 ) λ+µ
and hence
E(D1 ) µ
lim P(X(t) = 0) = = .
t→∞ E(U1 ) + E(D1 ) λ+µ
ALTERNATING RENEWAL PROCESSES 375
This matches with the limiting distribution of {X(t), t ≥ 0} computed in Exam-
ple 6.34 on page 241.

Example 8.22 Excess Life. In this example we use ARPs to derive the limting dis-
tribution of the excess life in a standard RP given in Equation 8.40.

For a fixed x > 0, define


1 if B(t) > x,
X(t) =
0 if B(t) ≤ x.
Then {X(t), t ≥ 0} is a stochastic process with state-space {0, 1} (see Figure 8.8)
with
B(t)

t
S1 S2 S3 S4

Z(t)

Up
Down

t
S1 S2 S3 S4

Figure 8.8 The excess life process and the induced ARP.

Un = max(Xn − x, 0),
Dn = min(Xn , x),
Un + Dn = Xn .
Note that Un and Dn are dependent, but {(Un , Dn ), n ≥ 0} is a sequence of iid
bivariate random variables. Hence {X(t), t ≥ 0} is an ARP. Then, assuming Xn is
aperiodic, we can use Theorem 8.30 to get
lim P(B(t) > x) = lim P(X(t) = 1)
t→∞ t→∞
E(U1 )
=
E(U1 ) + E(D1 )
E(max(X1 − x, 0))
=
E(X1 )
376 RENEWAL PROCESSES
∞
1
Z
= (1 − G(u))du
τ x

which is Equation 8.40, as desired.

Example 8.23 M/G/1/1 Queue. Consider an M/G/1/1 queue with arrival rate
λ and iid service times with mean τ . Let X(t) be the number of customers in the
system at time t. Compute the limiting distribution of X(t) as t → ∞.

The stochastic process {X(t), t ≥ 0} has state-space {0, 1}, where the down-
times {Dn , n ≥ 1} are iid exp(λ) and the up-times {Un , n ≥ 1} are iid service
times. Hence {X(t), t ≥ 0} is an aperiodic ARP (i.e., S1 = U1 + D1 is aperiodic).
Hence
p1 = lim P(X(t) = 1)
t→∞
E(U1 )
=
E(U1 ) + E(D1 )
τ
=
τ + 1/λ
ρ
= ,
1+ρ
where ρ = λτ . Hence
1
p0 = lim P(X(t) = 0) = .
t→∞ 1+ρ

Example 8.24 G/M/1/1 Queue. Consider a G/M/1/1 queue with iid inter-arrival
times with common cdf G and mean 1/λ, and iid exp(µ) service times. Let X(t) be
the number of customers in the system at time t. Compute the limiting distribution
of X(t) as t → ∞.

The stochastic process {X(t), t ≥ 0} has state-space {0, 1}. The up-times
{Un , n ≥ 1} are iid exp(µ) and the cycle lengths {Un +Dn , n ≥ 1} form a sequence
of iid random variables, although Un and Dn are dependent. Hence {X(t), t ≥ 0} is
an ARP. Now, let N (t) be the number of arrivals (who may or may not enter) up to
time t and An be the time of the nth arrival. Then
U1 + D1 = AN (U1 )+1 .
Hence, from Theorem 8.13, we get
1
E(U1 + D1 |U1 = t) = (1 + M (t)),
λ
where 1/λ is the mean inter-arrival time, and M (t) = E(N (t)). Hence
Z ∞
1
E(U1 + D1 ) = (1 + M (t))µe−µt dt
0 λ
ALTERNATING RENEWAL PROCESSES 377
 Z ∞ 
1 −µt
= 1+ M (t)µe dt
λ 0
 Z ∞ 
1 −µt
= 1+ e dM (t)
λ 0
1 
= 1 + M̃ (µ)
λ !
1 G̃(µ)
= 1+
λ 1 − G̃(µ)
1
= .
λ(1 − G̃(µ))
Assuming G is aperiodic, we get
E(U1 )
p1 = lim P(X(t) = 1) = = ρ(1 − G̃(µ)),
t→∞ E(U1 + D1 )
where ρ = λ/µ. Hence
p0 = lim P(X(t) = 0) = 1 − ρ(1 − G̃(µ)).
t→∞

Next we define a delayed ARP, along the same lines as in the definition of the
delayed RP.

Definition 8.8 Delayed Alternating Renewal Process. The stochastic process

{X(t), t ≥ 0} with state-space {0, 1} with Un being the nth up-time, and Dn the
nth down-time, is called a delayed alternating renewal process if {(Un , Dn ), n ≥ 2}
is a sequence of iid bivariate random variables, and is independent of (U1 , D1 ).

Thus a delayed ARP behaves like a standard ARP from the second cycle onward.
The main result about the delayed ARP is given in the next theorem.

Theorem 8.31 Distribution of Delayed ARP. Suppose {X(t), t ≥ 0} is a delayed

ARP that enters in state 1 at time S1 = U1 + D1 . If P(S1 < ∞) = 1 and U2 + D2
is aperiodic with E(U2 + D2 ) < ∞,
E(U2 )
lim P(X(t) = 1) = . (8.61)
t→∞ E(U2 ) + E(D2 )
If U1 + D1 and U2 + D2 both have a common period d, then the above limit holds if
t = nd and n → ∞.

Proof: Let F be the cdf of S1 . Conditioning on S1 we get

Z t
P(X(t) = 1) = H(t − u)dF (u),
0
378 RENEWAL PROCESSES
where H(t) is as defined in Theorem 8.30 for a standard ARP with sojourn times
{(Un , Dn ), n ≥ 2}. Then from the proof of Theorems 8.26, we get
E(U2 )
lim P(X(t) = 1) = lim H(t) = .
t→∞ t→∞ E(U2 ) + E(D2 )
Here we used the facts that F (∞) = 1, H(t) ≤ 1, and U2 + D2 is aperiodic.

This theorem says that the limiting behavior of a delayed ARP is not affected by
its behavior over the first cycle, as long as it terminates with probability 1. Finally,
one can prove the validity of Theorems 8.30 and 8.31 even if the sojourn in states 1
and 0 during one cycle are not contiguous intervals.

8.9 Semi-Markov Processes

In this section we study a class of stochastic processes obtained by relaxing the expo-
nential sojourn time assumption in the CTMCs. Specifically, we consider a stochastic
process {X(t), t ≥ 0} of Section 6.1 with a countable state-space S. It starts in the
initial state X0 at time t = 0. It stays there for a sojourn time Y1 and then jumps to
state X1 . In general it stays in state Xn for a duration given by Yn+1 and then jumps
to state Xn+1 , n ≥ 0. Let N (t) be the number of jumps up to time t. We assume
that the condition in Equation 6.1 on page 189 holds. Then {N (t), t ≥ 0} is well
defined, and the sequence {X0 , (Xn , Yn ), n ≥ 1} can be used to define the process
{X(t), t ≥ 0} by
X(t) = XN (t) , t ≥ 0. (8.62)
In this chapter we study the case where {X(t), t ≥ 0} belongs to a particular class
of stochastic processes called the semi-Markov processes (SMP), as defined below.

Definition 8.9 Semi-Markov Process. The stochastic process {X(t), t ≥ 0} as

defined by Equation 8.62 is called an SMP if it has a countable state-space S, and
the sequence {X0 , (Xn , Yn ), n ≥ 1} satisfies
P(Xn+1 = j, Yn+1 ≤ y|Xn = i, Yn , Xn−1 , Yn−1 , · · · , X1 , Y1 , X0 )
= P(X1 = j, Y1 ≤ y|X0 = i) = Gij (y), i, j ∈ S, n ≥ 0. (8.63)

Thus the semi-Markov process has Markov property at every jump epoch, hence
the name “semi”-Markov. In comparison, a CTMC has Markov property at every
time t. Note that unlike in the CTMCs, we allow pii > 0. Also, unlike in the CTMCs,
the variables Xn+1 and Yn+1 in an SMP are allowed to depend on each other.

Define the kernel of an SMP as the matrix

G(y) = [Gij (y)]i,j∈S , y ≥ 0.
An SMP is completely described by giving its kernel and the initial distribution
ai = P(X(0) = i), i ∈ S,
SEMI-MARKOV PROCESSES 379
if we further assume that the SMP enters state i with probability ai at time 0.

Clearly {Xn , n ≥ 0} is a DTMC (called the embedded DTMC in the SMP) with
transition probabilities
pij = Gij (∞) = P(Xn+1 = j|Xn = i), i, j ∈ S.
If there is a state i ∈ S for which Gij (y) = 0 for all j ∈ S and all y ≥ 0, then state
i must be absorbing . Hence we set pii = 1 in such a case. With this convention, we
see that X
pij = 1, i ∈ S.
j∈S

Next, let
X
Gi (y) = Gij (y) = P(S1 ≤ y|X0 = i), i ∈ S, y ≥ 0.
j∈S

Thus Gi is the cdf of the sojourn time in state i. We illustrate with several examples.

Example 8.25 Alternating Renewal Process. An ARP {X(t), t ≥ 0} is an SMP

if the sojourn time Un is independent of Dn . In this case the kernel is given by
 
0 P(Dn ≤ y)
G(y) = .
P(Un ≤ y) 0

Example 8.26 CTMC as SMP. A CTMC {X(t), t ≥ 0} with generator matrix Q

is an SMP with kernel G(y) = [Gij (y)] with Gii (y) = 0 and
Gij (y) = (1 − e−qi y )pij , i 6= j,
where qi = −qii and pij = qij /qi for i 6= j. If qi = 0 we define pii = 1. Thus
in a CTMC the sojourn time in the current state and the state after the jump are
independent.

Example 8.27 Series System. Consider a system consisting of N components in

series, i.e., it needs all components in working order in order to function properly.
The life time of the ith component is exp(λi ) random variable. At time 0 all the
components are up. As soon as any of the components fails, the system fails, and the
repair starts immediately. The repair time of the ith component is a non-negative ran-
dom variable with cdf Hi (·). Once repaired, a component is as good as new. When
the system is down, no more failures occur. Let X(t) = 0 if the system is functioning
at time t, and X(t) = i if component i is down (and hence under repair) at time t.
We shall show that {X(t), t ≥ 0} is an SMP and compute its kernel.

The state-space is {0, 1, 2, · · · , N }. Suppose the system is in state 0 at time t. Then

all components are up and the system state changes to i ∈ {1, 2, · · · , N } if the first
380 RENEWAL PROCESSES
item to fail is item i. Thus the sojourn time in state 0 is exp(λ) where
N
X
λ= λi ,
i=1

and the next state is i with probability λi /λ. Once the system enters state i ∈
{1, 2, · · · , N }, repair starts on component i. Thus the sojourn time in this state has
cdf Hi , at the end of which the system jumps to state 0. Combining all these obser-
vations we see that {X(t), t ≥ 0} is an SMP with kernel
λ1 −λy
) λλ2 (1 − e−λy ) · · · λλN (1 − e−λy )
 
0 λ (1 − e
 H1 (y) 0 0 ··· 0 
 
 H2 (y) 0 0 ··· 0
G(y) =  .

 .. .. .. .. .. 
 . . . . . 
HN (y) 0 0 ··· 0

We had seen this system in Modeling Exercise 6.9 on page 260 where the repair-
times were assumed to be exponential random variables. Thus the CTMC there is a
special case of the SMP developed here.

Armed with these examples of SMPs we now study the limiting behavior of the
SMPs. We need the following preliminaries. Define
Tj = min{t ≥ Y1 : X(t) = j}, j ∈ S,
where Y1 is the first sojourn time. Thus if the SMP starts in state j, Tj is the first time
it returns to state j (after leaving it at time S1 .) If it starts in a state i 6= j, then Tj is
the first time it enters state j. Now let
τi = E(Y1 |X(0) = i), i ∈ S,
and
τij = E(Tj |X(0) = i), i, j ∈ S.
The next theorem shows how to compute the τij ’s. It also extends the concept of first
step analysis to SMPs.

Theorem 8.32 First Passage Times in SMPs. The mean first passage times {τij }
satisfy the following equations:
X
τij = τi + pik τkj . (8.64)
k6=j

Proof: Follows along the same line as the derivation of Equation 6.49 on page 232,
and using τi in place of 1/qi as E(Y1 |X(0) = i).

Theorem 8.33 Suppose the embedded DTMC {Xn , n ≥ 0} in an SMP {X(t), t ≥

0} has transition probability matrix P = [pij ] that is irreducible and recurrent. Let
SEMI-MARKOV PROCESSES 381
π be a positive solution to
π = πP.
Then P
i∈S πi τi
τjj = , j ∈ S. (8.65)
πj

Proof: Follows along the same line as the derivation of Equation 6.50 on page 232,
and using τi in place of 1/qi as E(Y1 |X(0) = i).

Next we introduce the concept of irreducibility and periodicity for SMPs.

Definition 8.10 An SMP {X(t), t ≥ 0} is said to be irreducible (and recurrent) if

its embedded DTMC is irreducible (and recurrent).

It follows from Theorem 8.33 that if the mean return time to any state j in an irre-
ducible recurrent SMP is finite, it is finite for all states in the SMP. Hence we can
make the following definition.

Definition 8.11 An irreducible SMP {X(t), t ≥ 0} is said to be positive recurrent

if the mean return time to state j is finite for any j ∈ S.

It follows from Theorem 8.33 that a necessary and sufficient condition for positive
recurrence of an irreducible recurrent SMP is
X
πi τi < ∞,
i∈S

where πi and τi are as in Theorem 8.33. Note that if the first return time is aperiodic
(periodic with period d) for any one state, it is aperiodic (periodic with period d) for
all states in an irreducible and recurrent SMP. This motivates the next definition.

Definition 8.12 An irreducible and recurrent SMP is called aperiodic if the first
passage time Ti , starting with X0 = i, is an aperiodic random variable for any state
i. If it is periodic with period d, the SMP is said to be periodic with period d.

With these preliminaries we can now state the main result about the limiting be-
havior of SMPs in the next theorem. Intuitively, we consider a (possibly delayed)
ARP associated with the SMP such that the ARP is in state 1 whenever the SMP is
in state j, and zero otherwise. The length of a cycle in this ARP is the time between
two consecutive visits by the SMP to state j, hence the expected length of the cycle is
τjj . During this cycle the SMP spends an expected time τj in state j. Hence the long
run fraction of the time spent in state j is given by τj /τjj , which gives the limiting
distribution of the SMP. Since the behavior in the first cycle does not matter (as long
as it is finite with probability 1), the same result holds no matter what state the SMP
starts in, as long as the second and the subsequent cycles are defined to start with an
entry into state j.
382 RENEWAL PROCESSES
Theorem 8.34 Limiting Behavior of SMPs. Let {X(t), t ≥ 0} be an irreducible,
positive recurrent and aperiodic SMP with kernel G. Let π be a positive solution to
π = πG(∞).
Then {X(t), t ≥ 0} has a limiting distribution [pj , j ∈ S], and it is given by
πj τj
pj = lim P(X(t) = j|X(0) = i) = P , j ∈ S. (8.66)
k∈S πk τk
t→∞

If the SMP is periodic with period d, then the above limit holds if t = nd and n → ∞.

Proof: Fix a j ∈ S and define


1 if X(t) = j,
Z(t) =
0 if X(t) 6= j.
Let Sn be the nth entry into state j. The probabilistic structure of the SMP {X(t), t ≥
0} implies that {Z(t), t ≥ 0} is an ARP if X(0) = j, else it is a delayed ARP. In any
case consider the cycle [S1 , S2 ). We have X(S1 ) = j. The sojourn time U2 of the
delayed ARP in state 1 has mean τj . Let D2 be the time spent by the delayed ARP in
state 0 in the second cycle. The second cycle ends as soon as the SMP re-enters state
j. Hence we have
E(U2 + D2 ) = E(S2 − S1 ) = E(Tj |X(0) = j) = τjj .
Suppose i ∈ S is such that Ti is aperiodic. Since the embedded chain is irreducible,
the SMP has a positive probability of visiting every state k ∈ S during (0, Tj ) starting
from state j. This implies that every Tj must be aperiodic. Also, since Gi (∞) = 1
for all i, and the DTMC is recurrent, Tj < ∞ with probability 1 starting from any
state X(0) = i. Thus the hypothesis of Theorem 8.31 is satisfied. Hence
lim P(X(t) = j|X(0) = i) = lim P(Z(t) = 1)
t→∞ t→∞
E(U2 ) τj πj τj
= = =P ,
E(U2 ) + E(D2 ) τjj k∈S πk τk

where the last equality follows from Equation 8.65. This proves the theorem.

Equation 8.66 implies that the limiting distribution of the SMP depends on the
sojourn time distributions only through their means! This insensitivity of the lim-
iting distribution is very interesting and useful. It has generated a lot of literature
investigating similar insensitivity results in other contexts.
We illustrate with examples.

Example 8.28 Alternating Renewal Process. Let {X(t), t ≥ 0} be the ARP of

Example 8.25 with independent Un and Dn . The embedded DTMC has the P matrix
given by  
0 1
P = G(∞) = .
1 0
SEMI-MARKOV PROCESSES 383
This is an irreducible recurrent matrix, and π = [1 1] satisfies π = πP . Hence
equation 8.66 yields
E(D1 ) E(U1 )
p0 = , p1 = .
E(U1 ) + E(D1 ) E(U1 ) + E(D1 )
This matches with Equation 8.59, although that equation was derived without assum-
ing independence of Un and Dn .

Example 8.29 CTMC as SMP. Let {X(t), t ≥ 0} be an irreducible positive re-

current CTMC on state-space S with generator matrix Q. Then it has a limiting
distribution p = [pj , j ∈ S] that is given by the unique solution to
X
pQ = 0, pi = 1.
i∈S

We saw in Example 8.26 {X(t), t ≥ 0} is an SMP with an embedded DTMC with

transition probability matrix P = [pij ] where pij = qij /qi if i 6= j, pii = 0, and
qi = −qii . Let π be the solution to π = πP . From Theorem 6.26 on page 241 we
have
πj /qj
pj = P , j ∈ S.
i∈S πi /qi
This is same as Equation 8.66 since the expected sojourn time in state i is given
by τi = 1/qi . Thus the theory of SMP produces consistent results when applied to
CTMCs.

Example 8.30 Series System. Compute the limiting distribution of the series sys-
tem of Example 8.27.

The embedded DTMC has transition probability matrix given by

0 λλ1 λλ2 · · · λλN
 
 1 0 0 ··· 0 
 
 1 0 0 ··· 0 
P = G(∞) =  .
 . .. .. .. .. 
 .. . . . . 
1 0 0 ··· 0
A solution to π = πP is given by
π0 = λ, πi = λi , 1 ≤ i ≤ N.
Let ri be the mean repair time of component i. Then we have
τ0 = 1/λ, τi = ri , 1 ≤ i ≤ N.
Using Equation 8.66 we get
π0 τ0 1
p0 = PN = PN
i=0 πi τi 1 + i=0 λi ri
384 RENEWAL PROCESSES
and
πj τj λj rj
pj = PN = PN , 1 ≤ j ≤ N.
π
i=0 i iτ 1 + i=0 λi ri

Unlike in the study of CTMCs, the study of the limiting distribution of the SMPs
cannot stop at the limiting distribution of X(t) as t → ∞, since knowing the value
of X(t) at time t is, in general, not enough to determine the future of an SMP. We
also need to know the distribution of the remaining sojourn time at that time. We will
have to wait for the theory of Markov regenerative processes in the next chapter to
settle this question completely.

8.10 Renewal Processes with Costs/Rewards

We studied cost/reward models for the DTMCs in Section 4.8 and for the CTMCs in
Section 6.12. Following that tradition we now study cost/reward models associated
with RPs.

Let {N (t), t ≥ 0} be a standard RP generated by an iid sequence {Xn , n ≥ 0}.

We think of Xn as the length of the nth cycle between renewals. Suppose a reward
Rn is earned at the end of the nth cycle, i.e., at time Sn = X1 + · · · + Xn . Define
Z(t) to be the total reward earned up to time t, i.e.,
N (t)
X
Z(t) = Rn , t ≥ 0. (8.67)
n=1

Note that R(t) = 0 if N (t) = 0. With this notation we are ready to define a renewal
reward process.

Definition 8.13 Renewal Reward Process. The stochastic process {Z(t), t ≥ 0}

defined by Equation 8.67 is called a renewal reward process if {(Xn , Rn ), n ≥ 1} is
a sequence of iid bivariate random variables.

We say that the process {Z(t), t ≥ 0} is generated by {(Xn , Rn ), n ≥ 1}. Note

that Xn is a non-negative random variable, representing the length of the nth cycle,
but Rn can be positive or negative. Thus the sample paths of a renewal reward process
may go up or down, as shown in Figure 8.9. We illustrate with several examples.

Example 8.31 RP as a Renewal Reward Process. Suppose Rn = 1 for n ≥ 1

Then Z(t) = N (t). Thus an RP is a special case of a renewal reward process.

Example 8.32 CPP as a Renewal Reward Process. Let {Z(t), t ≥ 0} be a CPP

with batch arrival rate λ and P(Batch Size = k) = ak , k ≥ 1. Then {Z(t), t ≥ 0} is
a renewal reward process where {Rn , n ≥ 1} is a sequence of iid random variables
with common pmf [ak , k ≥ 1]. It is independent of {Xn , n ≥ 1}, which is a sequence
of iid exp(λ) random variables.
RENEWAL PROCESSES WITH COSTS/REWARDS 385
Z(t)

R2

R1
t
X1 X2

Figure 8.9 A typical sample path of a renewal reward process.

Example 8.33 Machine Maintenance. Consider the age replacement policy de-
scribed in Example 8.4, where we replace a machine upon failure or upon reaching
age T . Recall that Li is the lifetime of the ith machine and {Li , i ≥ 1} is a sequence
of iid non-negative random variables. Replacing a machine by a new one costs $Cr ,
and failure costs $Cf . Let Z(t) be the total cost incurred up to time t. Show that
{Z(t), t ≥ 0} is a renewal reward process.

Let Sn be the time when nth replacement occurs (S0 = 0), and N (t) be the
number of replacements up to time t. Then
Sn − Sn−1 = Xn = min(Ln , T ).
Thus {Xn , n ≥ 1} is a sequence of iid random variables, and {N (t), t ≥ 0} is an
RP generated by it. The cost Rn , incurred at time Sn , is given by

Cr if Ln > T,
Rn =
Cf + Cr if Ln ≤ T.
Here we have implicitly assumed that if Ln = T , then we pay for the failure and
then replace the machine. With the above expression for the cost Rn , we see that
{(Xn , Rn ), n ≥ 1} is a sequence of iid bivariate random variables. It is clear that
{Z(t), t ≥ 0} is generated by {(Xn , Rn ), n ≥ 1}, and hence it is a renewal reward
process.

Computing the distribution of Z(t) is rather difficult. Hence we study its asymp-
totic properties as t → ∞.

Theorem 8.35 Almost-Sure ERT for Renewal Reward Processes. Let {Z(t), t ≥
0} be a renewal reward process generated by {(Xn , Rn ), n ≥ 1}, and suppose
r = E(Rn ) < ∞, τ = E(Xn ) < ∞.
386 RENEWAL PROCESSES
Then
Z(t) r
lim = , w.p.1. (8.68)
t→∞ t τ

Proof: Using Equation 8.67, we get

PN (t)
Z(t) n=1 Rn N (t)
= · .
t N (t) t
Since N (t) → ∞ with probability 1, we can use the strong law of large numbers to
get
PN (t)
Rn
lim n=1 = r, w.p.1.
t→∞ N (t)
Furthermore, Theorem 8.6 yields
N (t) 1
→ , w.p.1.
t τ
Combining these two assertions we get Equation 8.68.

Next we derive the expected value version of the above result, i.e., lim E(Z(t))/t =
r/τ. As in the case of the elementary renewal theorem, this conclusion does not fol-
low from Theorem 8.35, and has to be established independently. We need two results
before we can prove this.

Theorem 8.36 Suppose the hypothesis of Theorem 8.35 holds. Then

 
N (t)+1
X
E Rn  = r(M (t) + 1), (8.69)
n=1

where M (t) = E(N (t)).

Proof: Let  
N (t)+1
X
H(t) = E  Rn  .
n=1

Using the renewal argument we get

 
N (t)+1
 
E(R1 |X1 = u) if u > t,
X 
E Rn  X1 = u =

 E(R1 |X1 = u) + H(t − u) if u ≤ t.
n=1

Hence, using G(u) = P(Xn ≤ u), we get

Z ∞ Z t
H(t) = E(R1 |X1 = u)dG(u) + H(t − u)dG(u)
0 0
Z t
= r+ H(t − u)dG(u).
0
RENEWAL PROCESSES WITH COSTS/REWARDS 387
Solving this renewal type equation by using Equation 8.26, we get
Z t
H(t) = r + rdM (u) = r(1 + M (t)).
0

This proves the theorem.

Theorem 8.37 Suppose the hypothesis of Theorem 8.35 holds. Then

E(RN (t)+1 )
lim = 0. (8.70)
t→∞ t
Proof: Note that RN (t)+1 depends upon XN (t)+1 , which has a different distribu-
tion than X1 as we had seen in our study of total life process in Section 8.6. Thus
E(RN (t)+1 ) 6= E(R1 ), and hence this theorem does not follow trivially. We prove it
by deriving a renewal equation for
H(t) = E(RN (t)+1 ).
Using the renewal argument we get

E(R1 |X1 = u) if u > t,
E(RN (t)+1 |X1 = u) =
H(t − u) if u ≤ t.
Hence, using G(u) = P(Xn ≤ u), we get
Z t
H(t) = D(t) + H(t − u)dG(u),
0
where Z ∞
D(t) = E(R1 |X1 = u)dG(u).
t
Now
Z ∞ Z ∞
|D(t)| = | E(R1 |X1 = u)dG(u)| ≤ |E(R1 |X1 = u)|dG(u)
t t
Z ∞
≤ |E(R1 |X1 = u)|dG(u) ≤ E(|R1 |) < ∞. (8.71)
0
Hence Theorem 8.15 yields
Z ∞
H(t) = D(t) + D(t − u)dM (u).
0

Also, Equation 8.71 implies that |D(t)| → 0 as t → ∞. Hence, for a given ǫ > 0,
there exists a T < ∞ such that |D(t)| < ǫ for t ≥ T . Then, for all t ≥ T , we have
Z t−T Z t
H(t) D(t) D(t − u) D(t − u)
= + dM (u) + dM (u)
t t 0 t t−T t
ǫ M (t − T ) M (t) − M (t − T )
≤ +ǫ + E(|R1 |) .
t t t
388 RENEWAL PROCESSES
Now, as t → ∞, M (t) − M (t − T ) → T /τ . Hence we get
H(t) ǫ M (t − T ) M (t) − M (t − T ) ǫ
lim ≤ lim + ǫ lim + lim E(|R1 |) = ,
t→∞ t t→∞ t t→∞ t t→∞ t τ
since the first and the third limit on the right hand are zero, and the second limit is
1/τ from the elementary renewal theorem. Since ǫ > 0 was arbitrary, we get Equa-
tion 8.70.

Now we are ready to prove the next theorem.

Theorem 8.38 ERT for Renewal Reward Processes. Suppose the hypothesis of
Theorem 8.35 holds. Then
E(Z(t)) r
lim = . (8.72)
t→∞ t τ
Proof: Write
N (t)+1
X
Z(t) = Rn − RN (t)+1 .
n=1
Hence
P 
N (t)+1
E(Z(t)) E n=1 Rn RN (t)+1
= −
t t t
1 + M (t) RN (t)+1
= r − ,
t t
where we have used Theorem 8.36. Now let t → ∞. Theorem 8.37 and the elemen-
tary renewal theorem yield
E(Z(t)) r
lim =
t→∞ t τ
as desired.

The above theorem is very intuitive and useful: it says that the long run expected
rate of reward is simply the ratio of the expected reward in one cycle and the expected
length of that cycle. What is surprising is that the reward does not have to be inde-
pendent of the cycle length. This is what makes theorem so useful in applications.
We end this section with two examples.

Example 8.34 Machine Maintenance. Compute the long run expected cost per
unit time of the age replacement policy described in the machine maintenance model
of Example 8.33.

Suppose the lifetimes {Li , i ≥ 1} are iid random variables with common cdf F (·).
Then we have
Z T
τ = E(Xn ) = E(min(Ln , T )) = (1 − F (u))du.
0
RENEWAL PROCESSES WITH COSTS/REWARDS 389
r = E(Rn ) = Cr + Cf F (T ).
Hence the long run cost rate is given by
E(Z(t)) r Cr + Cf F (T )
lim = = RT .
t→∞ t τ
0 (1 − F (u))du
Clearly, as T increases, the cost rate of the planned replacements decreases, but the
cost rate of the failures increases. Hence one would expect that there is an optimal
T which minimizes the total cost rate. The actual optimal value of T depends upon
Cr , Cf and F . For example, if Ln ∼ exp(λ), we get
E(Z(t)) λCr
lim = λCf + .
t→∞ t 1 − e−λT
This is a monotonically decreasing function of T , implying that the optimal T is
infinity, i.e., the machine should be replaced only upon failure. In retrospect, this is
to be expected, since a machine with exp(λ) life time is always as good as new!

In our study of renewal reward processes, we have assumed that the reward Rn is
earned at the end of the nth cycle. This is not necessary. The results of this section
remain valid no matter how the reward is earned over the cycle as long as the total
reward earned over the nth cycle is Rn and {(Xn , Rn ), n ≥ 1} is a sequence of iid
bivariate random variables. We use this fact in the next example.

Example 8.35 Total Up-Time. Let {X(t), t ≥ 0} be the ARP as defined in Sec-
tion 8.8. Let W (t) be the total time spent in state 1 by the ARP up to time t. Show
that
W (t) E(U1 )
lim = .
t→∞ t E(U1 ) + E(D1 )

{W (t), t ≥ 0} can be seen to be a renewal reward process with Un +Dn as the nth
cycle duration and Un as the reward earned over the nth cycle. Note that the reward
is earned continuously at rate X(t) during the cycle. Thus,
τ = E(U1 + D1 ), r = E(U1 ).
Hence the result follows from Theorem 8.38.

The results of this section remain valid for “delayed” renewal reward processes,
i.e., when {(Xn , Rn ), n ≥ 2} is a sequence of iid bivariate random variables, and is
independent of (X1 , R1 ). We shall omit the proofs.

Theorems 8.35 and 8.38 deal with what we had earlier called the “average cost
case.” It is possible to study the “discounted cost” case as well. The results are left as
Computational Exercise 8.49.
390 RENEWAL PROCESSES
8.11 Regenerative Processes

We study a class of stochastic processes {X(t), t ≥ 0} called “regenerative pro-

cesses” in this section. Intuitively, a regenerative process has “regeneration points”
{Sn , n ≥ 1}, so that the probabilistic behavior of the stochastic process during the
nth cycle, i.e., {X(t), Sn−1 ≤ t < Sn }, is the same for each n ≥ 1, and independent
from cycle to cycle. This idea is made precise in the following definition.

Definition 8.14 Regenerative Process. A stochastic process {X(t), t ≥ 0} is

called a regenerative process (RGP) if there exists a non-negative random variable
S1 so that
1. P(S1 = 0) < 1, P(S1 < ∞) = 1,
2. {X(t), t ≥ 0} and {X(t + S1 ), t ≥ 0} are stochastically identical, and
3. {X(t + S1 ), t ≥ 0} is independent of {X(t), 0 ≤ t < S1 }.

The above definition has several important implications. Existence of S1 implies

the existence of an infinite sequence of increasing random variables {Sn , n ≥ 1}
so that {X(t), t ≥ 0} and {X(t + Sn ), t ≥ 0} are stochastically identical, and
{X(t + Sn ), t ≥ 0} is independent of {X(t), 0 ≤ t < Sn }. We say that Sn is the nth
regeneration epoch, since the stochastic process loses all its memory and starts afresh
at times Sn . The interval [Sn−1 , Sn ) is called the nth regenerative cycle (S0 = 0).
The probabilistic behavior of the RGP over consecutive regenerative cycles is inde-
pendent and identical. In particular, {X(Sn ), n ≥ 0} are iid random variables .

Definition 8.15 Delayed Regenerative Process. A stochastic process {X(t), t ≥

0} is called a delayed RGP if there exists a non-negative random variable S1 so that
1. P(S1 < ∞) = 1,
2. {X(t + S1 ), t ≥ 0} is independent of {X(t), 0 ≤ t < S1 }, and
3. {X(t + S1 ), t ≥ 0} is a regenerative process.

Thus for a delayed RGP we get a sequence of regenerative epochs {Sn , n ≥ 0}

(S0 = 0). The behavior of the process over different regenerative cycles is indepen-
dent. However, the behavior of the first cycle may be different than the behavior of
the later cycles. We now give several examples of RGPs.

Example 8.36 Alternating Renewal Process. Consider an ARP {X(t), t ≥ 0}

generated by the bivariate sequence {(Un , Dn ), n ≥ 1} with U1 + D1 < ∞ with
probability 1. It is an RGP with regenerative epochs
n
X
Sn = (Ui + Di ), n ≥ 1,
i=1

since every time the process enters state 1, it regenerates. If {X(t), t ≥ 0} is a

delayed ARP, it is a delayed RGP.
REGENERATIVE PROCESSES 391
Example 8.37 Recurrence Times. The renewal process {N (t), t ≥ 0} with re-
newal sequence {Sn , n ≥ 1} is not an RGP. However the age process {A(t), t ≥ 0}
and the excess life process {B(t), t ≥ 0} are RGPs with regenerative epochs
{Sn , n ≥ 1}. If {N (t), t ≥ 0} is a delayed RP, then {A(t), t ≥ 0} and {B(t), t ≥ 0}
are delayed RGPs.

Example 8.38 CTMCs as RGPs. Let {X(t), t ≥ 0} be an irreducible recurrent

CTMC with X(0) = 0. Let Sn be the nth entry into state 0. The Markov property
of the CTMC implies that it regenerates at Sn , for n ≥ 1. Hence it is an RGP with
regeneration epochs {Sn , n ≥ 1}. If X(0) 6= 0, then it is a delayed RGP with this
sequence of regenerative epochs. Now suppose Sn is the time of nth entry into a set
of states {0, 1, · · · , k}. Are {Sn , n ≥ 1} the regenerative epochs of {X(t), t ≥ 0}?

Example 8.39 SMPs as RGPs. Let {X(t), t ≥ 0} be an SMP as defined in Sec-

tion 8.9. Suppose it has entered state 0 at time 0. Let Sn be the nth entry into state 0.
The Markov property of the SMP at all points of transition implies that it is an RGP
that regenerates at Sn , for n ≥ 1. If X(0) 6= 0, then it is a delayed RGP with this
sequence of regenerative epochs.

Example 8.40 GI/GI/1 Queue. Let X(t) be the number of customers in a GI/GI
/1 queue. Let Sn be the nth time when a customer enters an empty system. From
the independence of the inter-arrival times and the service times, it follows that the
system loses dependence on the history at times S1 , S2 , S3 , · · ·. A sufficient condition
for P(S1 < ∞) = 1 is that the mean service time be less than the mean inter-
arrival time. If the process starts with a customer entering an empty system at time
0, {X(t), t ≥ 0} is an RGP, otherwise it is a delayed RGP.

Next we study the limiting behavior of the RGPs. The main result is given in the
next theorem.

Theorem 8.39 Limiting Distribution for RGP. Let {X(t), t ≥ 0} be an RGP with
state-space (−∞, ∞) with right continuous sample paths with left limits. Let S1 be
the first regeneration epoch and U1 (x) be the time that the process spends in the
interval (−∞, x] during [0, S1 ). If S1 is aperiodic with E(S1 ) < ∞,
E(U1 (x))
F (x) = lim P(X(t) ≤ x) = . (8.73)
t→∞ E(S1 )
If S1 is periodic with period d, the above limit holds if t = nd and n → ∞.

Proof: Fix an x ∈ (−∞, ∞), and define H(t) = P(X(t) ≤ x). Since the process
regenerates at time S1 , we have

H(t − u) if u ≤ t,
P(X(t) ≤ x|S1 = u) =
P(X(t) ≤ x|S1 = u) if u > t.
392 RENEWAL PROCESSES
Using G(u) = P(S1 ≤ u), we get
Z ∞
H(t) = P(X(t) ≤ x|S1 = u)dG(u)
0
Z t
= D(t) + H(t − u)dG(u), (8.74)
0
where
D(t) = P(X(t) ≤ x, S1 > t).
It can be shown that the assumptions about the sample paths ensure that D(·) satisfies
the conditions in KRT (Theorem 8.17). Now define

1 if X(t) ≤ x,
Z(t) =
0 if X(t) > x.
Then we have
Z S1 
E(U1 (x)) = E Z(t)dt
0
Z ∞ Z u 


= E Z(t)dt  S1 = u dG(u)
Z0 ∞ Z u 0
= E(Z(t)|S1 = u)dtdG(u)
Z0 ∞ Z0 u
= P(X(t) ≤ x|S1 = u)dtdG(u)
Z0 ∞ Z0 ∞
= P(X(t) ≤ x|S1 = u)dG(u)dt
0 t
Z ∞
= P(X(t) ≤ x, S1 > t)dt.
0
Using the KRT we get
Z ∞
1
lim P(X(t) ≤ x) = lim H(t) = D(t)dt
t→∞ t→∞ E(S1 ) 0
Z ∞
1 E(U1 (x))
= P(X(t) ≤ x, S1 > t)dt = .
E(S1 ) 0 E(S1 )
This proves the theorem.

Several observations about the above theorem are in order. Let U1 (∞) =
limx→∞ U1 (x). Since the state-space of the RGP is (−∞, ∞) and S1 < ∞ with
probability 1, we have U1 (∞) = S1 and E(U1 (∞)) = E(S1 ). This implies
E(U1 (∞))
F (∞) = lim P(X(t) < ∞) = = 1.
t→∞ E(S1 )
Thus the RGP satisfying the hypothesis of the above theorem has a proper limiting
distribution.
REGENERATIVE PROCESSES 393
If the state-space of the RGP is discrete, say {0, 1, 2 · · ·}, we can define U1,j as
the time the RGP spends in state j over the first regenerative cycle. In that case the
above theorem implies that the RGP has a proper limiting pmf given by
E(U1,j )
pj = lim P(X(t) = j) = . (8.75)
t→∞ E(S1 )

Finally, Theorem 8.39 continues to hold if {X(t), t ≥ 0} is a delayed RGP, ex-

cept we now use the second cycle to do the computation. Thus, for a delayed RGP
satisfying the hypothesis of Theorem 8.39, we have
U2 (x)
lim P(X(t) ≤ x) = . (8.76)
t→∞ E(S2 − S1 )

We illustrate with an example.

Example 8.41 Suppose a coin is tossed repeatedly and independently with p =

P(H), and q = 1 − p = P(T ) on any one toss. What is the expected number of
tosses needed to observe the sequence HHTT?

Let Yn be the outcome of the nth toss. Let X0 = 1 and for n ≥ 1 define

1 if Yn = H, Yn+1 = H, Yn+2 = T, Yn+3 = T,
Xn =
0 otherwise.
Let S0 = 0 and define Sk+1 = min{n > Sk : Xn = 1}, for k ≥ 0. Note that since
the coin tosses are iid, {Sk+1 − Sk , k ≥ 1} are iid and have the same distribution
as S1 + 3. Let X(t) = X[t] , where [t] is the largest integer less than or equal to t.
Thus {X(t), t ≥ 0} is a periodic delayed RGP with period 1, and {Sk , k ≥ 1} are
the regeneration epochs. During each regenerative cycle it spends 1 unit of time in
state 1. Hence
1
lim P(X(n) = 1) = lim P(Xn = 1) = .
n→∞ n→∞ E(S2 − S1 )
However we know that
P(Xn = 1) = P(Yn = H, Yn+1 = H, Yn+2 = T, Yn+3 = T ) = p2 q 2 .
Hence
1
E(S2 − S1 ) = E(S1 + 3) = .
p2 q 2
Since the number of tosses needed to observe HHTT is S1 + 3, we see that the
expected number of tosses needed to observe HHTT is 1/p2 q 2 . Will this method
work for the sequence HTTH?

8.11.1 RGPs with Costs/Rewards

We next consider the cost/reward models in RGPs. Let X(t) be the state of a system
at time t, and assume that {X(t), t ≥ 0} is a regenerative process. Suppose the
394 RENEWAL PROCESSES
system earns rewards at rate r(x) whenever it is in state x. Then the total reward
obtained by the system up to time t is given by
Z t
r(X(u))du,
0
and the reward rate up to time t is given by
1 t
Z
r(X(u))du.
t 0
The next theorem gives the limiting behavior of the reward rate as t → ∞.

Theorem 8.40 Reward Rates in RGPs. Let {X(t), t ≥ 0} be a regenerative pro-

cess on S : (−∞, ∞) with proper limiting distribution F (·). Let r : S → (−∞, ∞)
be bounded from either above or below. Then
Rt R∞
1. limt→∞ 1t 0 r(X(u))du = −∞ r(u)dF (u), w.p. 1,
R  R
t ∞
2. limt→∞ 1t E 0 r(X(u))du = −∞ r(u)dF (u).

Proof: Let {Sn , n ≥ 1} (with S0 = 0) be a sequence of regenerative epochs in

{X(t), t ≥ 0}. Now define
Z t
Z(t) = r(X(u))du
0
and Z Sn
Rn = r(X(u))du, n ≥ 1.
Sn−1
Since {(Sn − Sn−1 , Rn ), n ≥ 1} is a sequence of iid bivariate random variables, we
see that {Z(t), t ≥ 0} is a renewal reward process. From Theorems 8.35 and 8.38
we get
1 t E(R1 )
Z
lim r(X(u))du = , w.p. 1
t→∞ t 0 E(S1 )
and Z t 
1 E(R1 )
lim E r(X(u))du = .
t→∞ t 0 E(S1 )
Now, from Theorem 8.39, we see that the time spent in the infinitesimal interval
(x, x + dx] by the system over the first regenerative cycle is given by E(S1 )dF (x).
Then Z S1  Z ∞
E(R1 ) = E r(X(u))du = r(x)E(S1 )dF (x).
0 −∞
Hence ∞
E(R1 )
Z
= r(x)dF (x).
E(S1 ) −∞
Thus the theorem follows.
REGENERATIVE PROCESSES 395
If the state-space of the RGP is discrete, say {0, 1, 2 · · ·}, and the limiting pmf of
the RGP is given by {pj , j ≥ 0} as in Equation 8.75, the long run reward rate in
Theorem 8.40 reduces to
X∞
r(j)pj . (8.77)
j=0

We illustrate the above theorem with an example below.

Example 8.42 Optimal Clearing. Suppose a manufacturing facility produces fin-

ished good one at a time according to a renewal process with mean production time
τ < ∞ per item and stores them in a warehouse. As soon as there are k items in
the warehouse we clear it by shipping them to the retailers. The clearing is instan-
taneous. It costs $h to hold an item in the warhouse per unit time, and it costs $c to
clear the warehouse. What is optimal value of k that minimizes the long run holding
plus clearing cost per unit time?

Let X(t) be the number of items in the warehouse at time t. Suppose X(0) = 0
and Sn be the time of the nth clearing time of the warehouse. It is obvious that
{X(t), t ≥ 0} is a regenerative process on state-space {0, 1, · · · , k − 1} with regen-
eration epochs {Sn , n ≥ 1}. A typical sample path of the {X(t), t ≥ 0} process is
shown in Figure 8.10. We see that
P(S1 < ∞) = 1, E(S1 ) = kτ.

Z(t)

4
3
2
1

t
S1 S2

Figure 8.10 A typical sample path of a regenerative process.

Hence, from Equation 8.75 we get

E(U1,j ) τ 1
pj = lim P(X(t) = j) = = = , 0 ≤ j ≤ k − 1.
t→∞ E(S1 ) kτ k

We compute Ch , the long run expected holding costs per unit time first. The system
incurs waiting costs at a rate jh per unit time if there are j items in the warehouse.
We can use Equation 8.77 to get
k−1 k−1
X hX 1
Ch = jhpj = j = h(k − 1).
j=0
k j=0 2
396 RENEWAL PROCESSES
To compute Cc , the long run clearing cost per unit time, we use Theorem 8.38 for
the renewal reward processes to get
c
Cc = .
kτ
Thus the C(k), the long run total cost per unit time is given by
1 c
C(k) =
h(k − 1) + .
2 kτ
This is a convex function of k and is minimized at
r
∗ 2c
k = .
hτ
Since k ∗ must be an integer we check the two integers near the above solution to see
which is optimal.

8.11.2 Little’s Law

We gave a sample path proof of Little’s Law as given in Theorem 7.5 on page 291.
Here we give an alternate proof under the assumption that the queueing system is
described by an RGP {X(t), t ≥ 0}, where X(t) is the number of customers in the
system at time t.

Theorem 8.41 Little’s Law for RGPs. Suppose {X(t), t ≥ 0} is an RGP on

{0, 1, 2, · · ·} with regeneration epochs {Sn , n ≥ 1} where Sn is the nth time it jumps
from state 0 to 1 (i.e., a customer enters an empty system). Suppose E(S1 ) < ∞. Then
the limits L, λ, and W be as defined by Equations 7.12, 7.13, and 7.14 on page 291,
exist and are related by
L = λW.

Proof: Let N be the number of customers who enter the system over [0, S1 ). Thus
it includes the arrival at time 0, but not the one at time S1 . By the definition of S1 ,
the number of departures up to time S1 is also N . Then using the renewal reward
theorem (Theorem 8.38) we see that the limits L, λ, and W exist, and are given by
Z S1 
L = E X(u)du /E(S1 ), (8.78)
0
λ = E(N )/E(S1 ), (8.79)
N
!
X
W = E Wn /E(N ). (8.80)
n=1

Now define

1 if the nth customer is in the system at time t,
In (t) =
0 otherwise.
COMPUTATIONAL EXERCISES 397
Then for 1 ≤ n ≤ N we have
Z S1
Wn = In (t)dt,
0
since these customers depart by time S1 . Also, for 0 ≤ t < S1 , we have
N
X
X(t) = In (t),
n=1

since the right hand side simply counts the number of customers in the system at time
t. Combining these equations we get
X N XN Z S1 N
Z S1 X Z S1
Wn = In (t)dt = In (t)dt = X(t)dt.
n=1 n=1 0 0 n=1 0

Substituting in Equation 8.78 we get

Z S1 
L = E X(u)du /E(S1 )
0
N
!
X
= E Wn /E(S1 )
n=1
P 
N
E(N ) n=1 Wn
= ·
E(S1 ) E(S1 )
= λW
as desired.

We end this chapter with the observation that the difficulty in using Theorem 8.39
as a computational tool is in the computation of E(U1 (x)) and E(S1 ). This is gen-
erally the result of the fact that the sample paths of the RGP over [0, S1 ) can be
quite complicated with multiple visits to the interval (−∞, x]. In the next chapter we
shall study the Markov RGPs, which alleviate this problem by using “smaller” S1 ,
but in the process giving up the assumption that the system loses dependence on the
history completely at time S1 . This leads to a richer structure and a more powerful
computational tool. So we march on to Markov RGPs!

8.12 Computational Exercises

8.1 Consider an M/M/1/K queue that is full at time 0. Let S0 = 0, and Sn be

the arrival time of the nth customer who sees the system full upon arrival. Show that
{Sn , n ≥ 1} is a renewal sequence. What does the corresponding RP count? Do the
same analysis for the M/G/1/K and the G/M/1/K queue.
398 RENEWAL PROCESSES
8.2 Consider a birth and death process on S = {· · · , −2, −1, 0, 1, 2, · · ·} with λn =
λ and µn = µ for all n ∈ S. Let Sn be the time of the nth downward jump. Show
that {Sn , n ≥ 1} is a renewal sequence. What is the corresponding RP?

8.3 Consider the RP of Computational Exercise 8.2. Is this renewal process transient
or recurrent?

8.4 Let X(t) be the number of customers at time t in a G/G/1 queue with iid inter-
arrival times and iid service times. Suppose at time 0 a customer enters an empty
system and starts service. Find an embedded renewal sequence in {X(t), t ≥ 0}.

8.5 Let {N (t), t ≥ 0} be an RP with iid inter-renewal times with common pdf
g(t) = λ2 te−λt , t ≥ 0.
Compute P(N (t) = k) for k = 0, 1, 2, · · ·.

8.6 Let {N (t), t ≥ 0} be an RP with integer valued inter-renewal times with com-
mon pmf
P(Xn = 0) = 1 − α, P(Xn = 1) = α, n ≥ 1,
where 0 < α < 1. Compute P(N (t) = k) for k = 0, 1, 2, · · ·.

8.7 Let {N (t), t ≥ 0} be an RP with common inter-renewal time pmf

P(Xn = i) = αi−1 (1 − α), i ≥ 1,
where 0 < α < 1. Compute P(N (t) = k) for k = 0, 1, 2, · · ·.

8.8 Let {N (t), t ≥ 0} be an RP with common inter-renewal time pmf

P(Xn = 1) = .8, P(Xn = 2) = .2, n ≥ 1.
Compute P(N (t) = k) for k = 0, 1, 2, · · ·.

8.9 Let {N (t), t ≥ 0} be an RP with common inter-renewal time pmf

P(Xn = 0) = .2, P(Xn = 1) = .3, P(Xn = 2) = .5, n ≥ 1.
Compute P(N (t) = k) for k = 0, 1, 2, · · ·.

8.10 Let {N (t), t ≥ 0} be an RP with common inter-renewal time pdf

g(t) = rλ1 e−λ1 t + (1 − r)λ2 e−λ2 t , t ≥ 0.
Compute the LST p̃k (s) of pk (t) = P(N (t) = k) for k = 0, 1, 2, · · ·. Compute p0 (t)
and p1 (t).

8.11 Consider the machine maintenance problem of Example 8.4. Suppose the ma-
chine lifetimes (in years) are iid U (2, 5), and they are replaced upon failure or upon
reaching age 3. Compute

1. the long run rate of replacements,

COMPUTATIONAL EXERCISES 399
2. the long run rate of planned replacements,
3. the long run rate of failures.

8.12 In Computational Exercise 8.11 compute the asymptotic distribution of the

number of replacements.

8.13 Let {Yn , n ≥ 0} be a DTMC on {0, 1} with the following transition probabil-
ity matrix  
α 1−α
.
1−β β
Analyze the asymptotic behavior of N (n) = number of visits to state 0 during {1, 2
, · · · , n}, assuming that Y0 = 0.

8.14 Let {Y (t), t ≥ 0} be a DTMC on {0, 1} with the following generator matrix
 
−λ λ
.
µ −µ
Analyze the asymptotic behavior of N (t) = number of visits to state 0 during (0, t],
assuming that Y (0) = 0.

8.15 Compute the renewal function for the RP in Computational Exercise 8.10.

8.16 Compute the renewal function for the RP in Computational Exercise 8.8.

8.17 Compute the renewal function for the RP in Computational Exercise 8.14.

8.18 Derive a renewal type equation for E(SN (t)+k ), k ≥ 1, and solve it.

8.19 Compute M ∗ (t) = E(N ∗ (t)) in terms of M (t) = E(N (t)), where N ∗ (t) and
N (t) are as defined in Conceptual Exercise 8.2.

8.20 Let Mi (t) = E(Ni (t)) (i = 1, 2) where Ni (t) is as defined in Conceptual

Exercise 8.3. Compute M̃i (s) in terms of G̃(s) for i = 1, 2.

8.21 Compute the renewal type equation for P(A(t) ≤ x), where A(t) is the age at
time t in an RP. Show that the KRT is applicable and compute the limiting distribution
of A(t) as t → ∞, assuming that the RP is aperiodic.

8.22 Compute the renewal type equation for E(A(t)B(t)), where A(t) (B(t)) is the
age (excess-life) at time t in an RP. Show that the KRT is applicable and compute the
limiting value of E(A(t)B(t)) as t → ∞, assuming that the RP is aperiodic. Using
this compute the limiting covariance of A(t) and B(t) as t → ∞.

8.23 Derive an integral equation for P(N (t) is odd) for an RP {N (t), t ≥ 0} by
conditioning on the first renewal time. Is this a renewal type equation? Solve it ex-
plicitly when the RP is PP(λ).
400 RENEWAL PROCESSES
8.24 Consider the two-state CTMC of Computational Exercise 8.14. Let W (t) be
the time spent by this process in state 0 during (0, t]. Derive a renewal type equation
for E(W (t)), and solve it using the transform method.

8.25 Let {N (t), t ≥ 0} be an RP with inter-renewal time distribution G. Using the

renewal argument derive the following renewal type equation for Mk (t) = E(N (t)
(N (t) − 1) · · · (N (t) − k + 1)), (k ≥ 1), with M0 (t) = 1, and M1 (t) = M (t):
Z t Z t
Mk (t) = k Mk−1 (t − u)dG(u) + Mk (t − u)dG(u).
0 0

Hence derive an expression for M̃k (s) in terms of G̃(s).

8.26 Compute the renewal type equation for E(A(t)k ), where A(t) is the age at
time t in an RP. Assume that the inter-renewal times are aperiodic with finite (k +
1)st moment. Show that the KRT is applicable and compute the limiting value of
E(A(t)k ) as t → ∞.

8.27 Compute the renewal type equation for E(C(t)k ), where C(t) is the total life
at time t in an RP. Assume that the inter-renewal times are aperiodic with finite
(k + 1)st moment. Show that the KRT is applicable and compute the limiting value
of E(C(t)k ) as t → ∞.

8.28 Compute the renewal type equation for E(A(t)k C(t)m ), where A(t) is the age
and C(t) is the total life at time t in an RP. Assume that the inter-renewal times are
aperiodic with finite (k + m + 1)st moment. Show that the KRT is applicable and
compute the limiting value of E(A(t)k C(t)m ) as t → ∞.

8.29 Compute an integral equation for P(B D (t) > x), where B D (t) is the excess
life at time t in a delayed RP. Compute its limiting value.

8.30 Consider a G/G/1/1 queue with inter-arrival time cdf G and service time cdf
F . Using ARPs, compute the limiting probability that the server is busy.

8.31 Compute the expected busy period started by a single customer in a stable
M/G/1 queue of Section 7.6.1 of Chapter 7, by constructing an appropriate ARP.
(Use Equation 7.35 on page 314.)

8.32 Consider an M/M/1/K queue of Section 7.3.2 of Chapter 7 with limiting

distribution given by Equation 7.19. Construct an appropriate ARP to compute the
expected time when the system becomes full (i.e., it enters state K from K − 1) for
the first time after it becomes non-full (i.e., after it enters state K − 1 from K).

8.33 A particle moves on n sites arranged in a circle as follows: it stays at the ith site
for a random amount of time with cdf Fi and mean µi and then moves to the adjacent
site in the clockwise direction. Let H be the cdf of the time it takes to complete the
circle, and assume that it is aperiodic with mean µ. Furthermore, assume that the
successive sojourn times are independent. Construct an appropriate ARP to compute
the limiting probability that the particle is on site i.
COMPUTATIONAL EXERCISES 401
8.34 For the series system in Example 8.27 define Y (t) = 0 if the system is up at
time t and 1 if it is down at time t. Assume that the system is up at time 0, and show
that {Y (t), t ≥ 0} is an ARP, and compute the long run probability that the system
is up. Verify the result with result in Example 8.30.

8.35 Do Computational Exercise 8.34 by constructing an appropriate two-state

SMP.

8.36 Let {Y (t), t ≥ 0} be as in Computational Exercise 8.14. Suppose Y (0) = 0.

Show that {Y (t), t ≥ 0} is an ARP. Compute P(Y (t) = 0) by solving Equation 8.58
by using the transform methods.

8.37 In the machine maintenance model of Example 8.34, suppose the machine
lifetimes are iid U (0, a) random variables. Compute the optimal age replacement
parameter T that minimizes the long run expected total cost per unit time.

8.38 Demands occur according to a PP(λ) at a warehouse that has S items in it

initially. As soon as the number of items in the warehouse decreases to s(< S), the
inventory is instantaneously replenished to S. Let X(t) be the number of items in
the warehouse at time t. Is {X(t), t ≥ 0} a (1) a CTMC, (2) an SMP, (3) a RGP?
Compute its limiting distribution.

8.39 Let X(t) be the number of customers in a queueing system. In which of the
following systems is {X(t), t ≥ 0} an SMP? Why or why not?

1. An M/G/1/1 system,
2. An G/M/1/1 system,
3. An G/G/1/1 system.

8.40 Do Computational Exercise 8.38 if the demands arrive according to an RP with

common inter-demand time cdf G.

8.41 A machine is subject to shocks that arrive according to PP(λ). Each shock
causes a damage that can represented by an integer-valued random variable with
pmf {αj , j ≥ 1}. The damages are additive, and when the total damage exceeds a
threshold K, the machine breaks down. The repair time has cdf A(·), and successive
repair times are iid. Shocks have no effect during repair, and the machine is as good
as new once the repair completes. Model this system by an appropriate SMP. Show
its kernel.

8.42 Let {W (t), t ≥ 0} be as defined in Example 8.35 and let U and D be the
E(U)
generic up and down times. Define H(t) = E(W (t)) − E(U+D) t. Show that H
satisfies the renewal type equation
Z t
E(min(U + D, t)
H(t) = E(min(U, t)) − + H(t − u)dG(u),
E(U + D) 0
402 RENEWAL PROCESSES
where G is the cdf of U + D. Assuming that G is aperiodic, show that
1 E(U )E((U + D)2 ) − E(U + D)E(U 2 )
lim H(t) = · .
t→∞ 2 (E(U + D))2

8.43 Let Xn be the amount of inventory in a warehouse at the beginning of day n.

Suppose X0 = S > 0, a given number. Let Yn be the size of the demand on day n
and assume that {Yn , n ≥ 1} is a sequence of iid random variables with common cdf
G. As soon as the inventory goes below s < S, it is instantaneously replenished to
S. Is {Xn , n ≥ 0} a (1) DTMC, (2) SMP, (3) RGP? Explain why or why not. Show
that
1 + M (S − x)
lim P(Xn ≥ x) = , s ≤ x ≤ S,
n→∞ 1 + M (S − s)
where M (·) is the renewal corresponding to the cdf G.

8.44 Use the renewal equation derived in the proof of Theorem 8.37 to compute the
limit of E(RN (t)+1 ) as t → ∞.

8.45 What is the long run fraction of customers who are turned away in an
M/G/1/1 queue? In a G/M/1/1 queue?

8.46 The patients in a hospital are classified as belonging to the following units:
(1) coronary care unit, (2) intensive care unit, (3) ambulatory unit, (4) extended care
unit, and (5) home or dead. As soon as a patient goes home or dies, a new patient is
admitted to the coronary unit. The successive units the patient visits form a DTMC
with transition probability matrix given below:
 
0 1 0 0 0
 0 0 1 0 0 
 
 0.1 0 0 0.9 0 .
 
 0.1 0.1 0.1 0.5 0.2 
0 0 0 0 1
The patient spends on the average 1.7 days in the coronary care unit, 2.2 days in
the intensive care unit, 8 days in the ambulatory unit and 16 days in the extended
care unit. Let X(t) be the state of the patient in the hospital (assume it has exactly
one patient at all times. Extension to more than one patient is easy if the patient
movements are iid.) at time t. Model it as an SMP with four states and compute the
long run probability that patient is in the ambulatory unit.

8.47 A company classifies its employees into four grades, labeled 1, 2, 3, and 4. An
employee’s stay in grade i is determined by two random variables: the promotion
time Ai , and the tolerance time Bi . The employee stays in grade i for min(Ai , Bi )
amount of time. If Ai ≤ Bi , he moves to grade i + 1. If Ai > Bi , he quits, and is
instantaneously replaced by a new employee in grade 1. Since there is no promotion
from grade 4, we set A4 = ∞. Let X(t) be the grade of the employee working at
time t.
COMPUTATIONAL EXERCISES 403
1. Model {X(t), t ≥ 0} as an SMP. Explicitly state any assumptions needed to do
this, and display the kernel.
2. Assume Ai ∼ exp(λi ) and Bi ∼ Erl(2, µi ), and that they are independent. Com-
pute the limiting distribution of X(t).

8.48 Sixteen underground bunkers are connected by tunnels as shown in Fig-

ure 8.11. This complex serves as a residence of a military despot, who spends a

1 2 3 4

5 6 7 8

9 10 11 12

13 14 15 16

Figure 8.11 The underground maze of 16 bunkers.

random amount of time in a bunker and then moves to any of the adjacent ones
with equal probability. Suppose the mean time spent in bunker i during one stay is
τi . Compute the long run probability that the despot is in bunker i. (You may use
symmetry to analyze the embedded DTMC.)

8.49 Consider the renewal reward process of Section 8.10. Suppose the rewards are
discounted with continuous discount factor α > 0. Let D(t) be the total discounted
reward earned up to time t, i.e.,
N (t)
X
D(t) = e−αSn Rn .
n=1

Show that
E(R1 e−αS1 )
lim E(D(t)) = .
t→∞ 1 − E(e−αS1 )

8.50 Suppose demands arise according to a PP(λ) at a warehouse that initially has
S items. A demand is satisfied instantly if there are items in the warehouse, else the
demand is lost. When the warehouse becomes empty, it places an order for S items
from the supplier. The order is fulfilled after a random amount of time (called the
404 RENEWAL PROCESSES
lead-time) with mean L days. Suppose it costs $h to store an item in the warehouse
for one day. The warehouse manager pays $c to buy an item and sells it for $p. The
order processing cost is $d, regardless of the size of the order. Using an appropriate
renewal reward process, compute the long run cost rate for this policy. Find the value
of S that will minimize this cost rate for the following parameters: λ = 2 per day,
h = 1, c = 70, p = 80, d = 50, L = 3.

8.51 Consider the parking lot of Modeling Exercise 6.24, and let N0 (t) be the num-
ber of customers who arrived there up to time t. Suppose {N0 (t), t ≥ 0} is an RP
with common inter-arrival time cdf A(·). Let Nk (t) be the number of customers
who arrived up to time t and found spaces 1 through k occupied (1 ≤ k ≤ K).
{Nk (t), t ≥ 0} is called the overflow process from space k.
1. Show that {Nk (t), t ≥ 0} is an RP and that the LST of its inter-renewal times is
given by
φk−1 (s + µ)
φk (s) = ,
1 − φk−1 (s) + φk−1 (s + µ)
where φ0 (s) is the LST of A(·).
2. Let Xk (t) be 1 if space k is occupied, and zero otherwise. Show that
{Xk (t), t ≥ 0} is the queue-length process in a G/M/1/1 queue with arrival
process{Nk−1 (t), t ≥ 0} and iid service times with LST φk (s). Compute the lim-
iting distribution of Xk (t) as t → ∞, and show that long run fraction of the time
the kth space is occupied (called its utilization) is given by
1 − φk−1 (µ)
,
µτk−1
where τk is the mean inter-renewal time in {Nk (t), t ≥ 0}. Show that
τk = τk−1 /φk−1 (µ),
where τ0 is the mean of A(·).
3. Compute the space utilizations for each space if there are six parking spaces and
customers arrive every 2 minutes in a deterministic fashion and stay in the lot for
an average of 15 minutes.

8.52 Consider a G/M/∞ queue with infinite number of servers, common inter-
arrival time cdf G with mean 1/λ and exp(µ) service times. Suppose that at time
0 the system is in steady state. In particular the arrival process is assumed to have
started at −∞ so that at time 0 it is in equilibrium. Let {Tn , n ≥ 0} be the arrival
times of customers who arrived before time 0, indexed in reverse, so that 0 > T1 >
T2 > · · ·. Now, let Xi be 1 if the customer
P who arrived at time Ti is still in the system
at time 0, and 0 otherwise. Thus X = ∞ i=1 Xi is the total number of customers in
the system at time 0 (i.e., in steady state.)
1. Show that cdf of −T1 is Ge , the equilibrium cdf associated with G, and {Ti −
Ti+1 , i ≥ 1} is a sequence of iid random variables with common cdf G, and is
independent of −T1 .
COMPUTATIONAL EXERCISES 405
2. Show that
λ
E(Xi ) = G̃(µ)i−1 (1 − G̃(µ)), i ≥ 1.
µ
3. Using the above, show that the expected number of customers in steady state in a
G/M/∞ queue is given by
λ
E(X) = .
µ
Show this directly by using Little’s Law.
4. Show that for i > j,
λ
E(Xi Xj ) = G̃(µ)i−j G̃(2µ)j−1 (1 − G̃(2µ)).
2µ
5. Using the above, show that
λ G̃(µ)
E(X(X − 1)) = · .
µ 1 − G̃(µ)

8.53 Functionally identical machines are available from N different vendors. The
lifetimes of the machines from vendor i are iid exp(λi ) random variables. The ma-
chines from different vendors are independent of each other. We use a “cyclic” re-
placement policy parameterized by a fixed positive number T as follows: suppose
we are currently using a machine from vendor i. If it is less than T time units old
upon failure, it is replaced by machine from vendor i + 1 if i < N and vendor 1 if
i = N . If the machine is at least T time units old upon failure, the replacement is
from vendor i. Replacements are instantaneous. Let X(t) = i if the machine in use
at time t is from vendor i.

1. Is {X(t), t ≥ 0} an SMP? If it is, give its kernel G.

2. What is the long run fraction of the time that a machine from vendor i is in use?
3. Suppose the machines from vendor i cost $ci . What is the long run cost per unit
time of operating this policy?

8.54 April One Computers provides the following warranty on all its hard drives for
its customers that sign a long term contract with it: it will replace a malfunctioning
drive with a new one for free any number of times for up to one year after the initial
purchase. If the hard drive fails after one year, the customer must purchase a new one
with a new one year free replacement warranty. Suppose the lifetimes of these drives
are iid exponential variables with common mean 1/λ. The customer pays c for each
new hard drive that is not replaced for free. It costs the company d to make the drive,
(d < c). All failed hard drives are discarded. Suppose a customer has signed a long
term contract with April One Computers for his hard drive. Let Z(t) be the total cost
(excluding the initial purchase cost) to the customer over the interval (0, t], and Y (t)
be the total profits to the April One Computers from this contract over the period
(0, t]. Assume that replacement is instantaneous.

1. Show that {Z(t), t ≥ 0} and {Y (t), t ≥ 0} are renewal reward processes.

406 RENEWAL PROCESSES
2. Compute the expected cost per year to the customer and the expected profit per
year to the producer.

8.13 Conceptual Exercises

8.1 Suppose {Xn , n ≥ 0} is a DTMC on {0, 1, 2, · · ·}. Suppose we define S0 = 0

and Sn = min{k > Sn−1 : Xk = XSk−1 }, n ≥ 1. Is {Sn , n ≥ 1} a renewal
sequence? (Hint: Consider if X0 is a constant or not.)

8.2 Let {N (t), t ≥ 0} be an RP. Suppose each event is registered with probability
p, independent of everything else. Let N ∗ (t) be the number of registered events up
to t. Is {N ∗ (t), t ≥ 0} an RP?

8.3 Let {N (t), t ≥ 0} be an RP. Define

 
N (t)
N1 (t) = ,
2
 
N (t) + 1
N2 (t) = ,
2
where [x] is the largest integer less than or equal to x. Is {N1 (t), t ≥ 0} an RP? What
about {N2 (t), t ≥ 0}?

8.4 Complete the proof of Theorem 8.6 when τ = ∞, by showing that the limsup
and liminf of N (t)/t are both zero.

8.5 Derive Equation 8.17 by directly using the renewal argument.

8.6 Complete the proof of Theorem 8.12 when τ = ∞, by showing that the limsup
and liminf of M (t)/t are both zero.

8.7 Show that a random variable with the following pmf is aperiodic:
P(X = e) = .5, P(X = π) = .5.

8.8 Let {N D (t), t ≥ 0} be a delayed RP satisfying Equation 8.48. Show that

P(N D (t) < ∞) = 1, for all t ≥ 0.

8.9 Let {N D (t), t ≥ 0} be a delayed RP satisfying Equation 8.48. Show that

P(N D (t) = 0) = 1 − F (t),
P(N D (t) = k) = F ∗ (Gk−1 − Gk )(t), k ≥ 1.

8.10 Let {N D (t), t ≥ 0} be a delayed RP satisfying Equation 8.48. Show that

P( lim N D (t) = ∞) = 1.
t→∞
CONCEPTUAL EXERCISES 407
8.11 Let {N D (t), t ≥ 0} be a delayed RP satisfying Equation 8.48. Show that
N D (t) 1
lim = .
t→∞ t τ
where τ = E(X2 ).

8.12 Let {M D (t), t ≥ 0} be the renewal function of a delayed RP satisfying Equa-

tion 8.48. Show that
M D (t) 1
lim = .
t→∞ t τ
where τ = E(X2 ).

8.13 Let {X(t), t ≥ 0} be an ARP as given in Definition 8.7. Let Ni (t) be the
number of entries into state i (i = 1, 2) over (0, t]. Is {Ni (t), t ≥ 0} an RP?

8.14 Let Nj (t) be as in Example 8.18. Show that the limiting values of Nj (t)/t and
E(Nj (t))/t are independent of the initial state of the CTMC.

8.15 Let {N e (t), t ≥ 0} is an equilibrium renewal process. Show that the distribu-
tion of N e (s + t) − N e (s) is independent of s, i.e., {N e (t), t ≥ 0} has stationary
increments. Are the increments independent?

8.16 Let {Xn , n ≥ 0} be an irreducible, aperiodic, positive recurrent DTMC with

limiting distribution {πj , j ∈ S}. Use regenerative processes to show that the mean
inter-visit time to state j is given by 1/πj .

8.17 Let {(Xn , Bn , Cn ), n ≥ 1} be a sequence of iid tri-variate random variables.

Suppose Xn ’s are non-negative, representing the length of the nth cycle, Bn the
benefit accrued at the end of the nth cycle, and Cn the cost incurred at the end of the
nth cycle. Let Zb (t) (Zc (t)) be the total benefit (cost) up to time t. Show that the
long run cost to benefit ratio is given by
Zc (t) E(Zc (t)) E(C1 )
lim = lim = .
t→∞ Zb (t) t→∞ E(Zb (t)) E(R1 )

8.18 Derive the following analog of Theroem 8.41:

Lq = λWq ,
where Wq is the expected time in the queue (time in the system - the time in service),
and Lq is the expected number in the queue (number in the system - number in
service).

8.19 Let {X(t), t ≥ 0} be a positive recurrent and aperiodic SMP with state-space
S, kernel G, and limiting distribution {pi , i ∈ S}. Suppose we incur cost at rate ci
wheneverP the SMP is in state i. Show that the long-run rate at which we incur cost is
given by i∈S pi ci .
408 RENEWAL PROCESSES
8.20 Consider the system in Conceptual Exercise 8.19. Suppose the costs are dis-
counted continuously with a discount factor α > 0. Let φ(i) be the total expected
discounted cost incurred over the infinite horizon given that the SMP enters state i at
time 0. Let
ci
γi = (1 − G̃i (α)),
α
where G̃i (·) is the LST of the sojourn time in state i. Show that the φ(i)’s satisfy the
following equation: X
φ(i) = γi + G̃ij (α)φ(j).
j∈S
 CHAPTER 9

Markov Regenerative Processes

Statistics can be used to reach any conclusion you want: The odds of getting in an
accident are directly proportional to the time spent on the road. The time spent on
the road is inversely proportional to the speed at which you drive. One-third of traffic
accidents are caused by drunk drivers, while two-thirds are caused by drivers who
are not drunk. Clearly, the odds of getting into a traffic accident are minimized by
driving drunk at a high speed!

9.1 Definitions and Examples

Markov renewal theory is a natural generalization of renewal theory, and as the name
suggests, it combines the concepts from Markov chains and renewal processes. We
begin with a definition.

Definition 9.1 Markov Renewal Sequence. A sequence of bivariate random vari-

ables {(Xn , Sn ), n ≥ 0} is called a Markov renewal sequence (MRS) if
1. S0 = 0, Sn+1 ≥ Sn , Xn ∈ I, where I is a discrete set,
2. For all n ≥ 0,
P(Xn+1 = j, Sn+1 − Sn ≤ x|Xn = i, Sn , Xn−1 , Sn−1 , · · · , X0 , S0 )
= P(Xn+1 = j, Sn+1 − Sn ≤ x|Xn = i)
= P(X1 = j, S1 ≤ x|X0 = i).

Now define Yn = Sn − Sn−1 for n ≥ 1. There is a one to one correspondence

between the sequence {(Xn , Sn ), n ≥ 0} and the sequence {X0 , (Xn , Yn ), n ≥
1}. In fact we have used the sequence {X0 , (Xn , Yn ), n ≥ 1} earlier to define a
CTMC in Definition 6.1 on page 190 and an SMP in Definition 8.9 on page 378.
In a similar vein we use Markov renewal sequences to build a class of processes
called the Markov regenerative processes (MRGP) that encompasses CTMCs and
SMPs. We allow the state-spaces of such a process to be discrete, say {0, 1, 2, · · ·},
or continuous, say [0, ∞).

Definition 9.2 Markov Regenerative Process. A stochastic process {Z(t), t ≥

0} with state-space S is called a Markov regenerative process if there is a Markov

409
410 MARKOV REGENERATIVE PROCESSES
renewal sequence {(Xn , Sn ), n ≥ 0} such that {Z(t + Sn ), t ≥ 0} given {Z(u) :
0 ≤ u < Sn , Xn = i} is stochastically identical to {Z(t), t ≥ 0} given X0 = i and
is conditionally independent of {Z(u) : 0 ≤ u < Sn , Xn = i} given Xn = i.

Several comments on the implications of the above definition are in order. In most
applications we find that Xn = Z(Sn +) or Xn = Z(Sn −). One can see that if
{(Xn , Sn ), n ≥ 0} is Markov renewal sequence then {Xn , n ≥ 0} is a DTMC. Thus
in most applications {Z(Sn +), n ≥ 0} or {Z(Sn −), n ≥ 0} is a DTMC. Hence
sometimes an MRGP is also called a process with an embedded Markov chain. We
also see that
P(Z(t + Sn ) ≤ x|Z(u) : 0 ≤ u ≤ Sn , Xn = i) = P(Z(t) ≤ x|X0 = i).
Also, the future of the MRGP from t = Sn depends on its past up to time Sn only
through Xn . This distinguishes an MRGP from a regenerative process, whose future
from t = Sn is completely independent of its past up to time Sn . A typical sample
path of an MRGP is shown in Figure 9.1, where we have assumed that the state-space
of the MRGP is discrete, and Xn = Z(Sn −).

Z(t)

X4

X3
X1

X2

t
S0 S1 S2 S3 S4

Figure 9.1 A typical sample path of a Markov regenerative process.

In the examples below we verify that a given process is an MRGP by identifying

the embedded MRS. To do this we need to know how to describe an MRS. The next
theorem answers this question. We need the following notation. Let
ai = P(X0 = i), i ∈ I.
The vector a = [ai ]i∈I is called the initial distribution. Also, let
Gij (y) = P(X1 = j, Y1 ≤ y|X0 = i), i, j ∈ I. (9.1)
The matrix G(y) = [Gij (y)]i,j∈I is called the kernel of the MRS.

Theorem 9.1 Characterization of an MRS. An MRS {(Xn , Sn ), n ≥ 0} is com-

pletely described by the initial distribution a and the kernel G.
DEFINITIONS AND EXAMPLES 411
Proof: Describing the sequence {(Xn , Sn ), n ≥ 0} is equivalent to describing the
sequence {X0 , (Xn , Yn ), n ≥ 1}. Thus it suffices to show that
P(X0 = i0 , X1 = i1 , Y1 ≤ y1 , · · · , Xn = in , Yn ≤ yn )
= ai0 Gi0 ,i1 (y1 ) · · · Gin−1 ,in (yn ), (9.2)
for n ≥ 1, ik ∈ I and yk ≥ 0 for 1 ≤ k ≤ n. We leave it to the reader to show this
by induction.

We get the following as a corollary to the above theorem. We leave the proof to
the reader.

Theorem 9.2 Let {(Xn , Sn ), n ≥ 0} be an MRS with initial distribution a and

kernel G(y). Then {Xn , n ≥ 0} is a DTMC with the same initial distribution a and
transition probability matrix
P = G(∞),
P
with the convention that pii = 1 if j∈I Gij (∞) = 0.

We illustrate all the concepts introduced so far by examples.

Example 9.1 CTMCs as MRGPs. Let {Z(t), t ≥ 0} be a CTMC with state-space

{0, 1, 2, · · ·} and generator matrix Q. Show that {Z(t), t ≥ 0} is an MRGP.

Let S0 = 0 and Sn be the nth jump epoch in the CTMC. Define Xn = Z(Sn +).
Then, from Definition 6.1 on page 190, it follows that {(Xn , Sn ), n ≥ 0} is an MRS
with kernel
qij
Gij (y) = (1 − e−qi y ), i, j ∈ I, y ≥ 0.
qi
If qi = 0, we define Gij (y) = 0 for all j ∈ I. The Markov property of the CTMC
implies that {Z(t + Sn ), t ≥ 0} depends on {Z(t), 0 ≤ t < Sn , Xn } only through
Xn . Thus {Z(t), t ≥ 0} is an MRGP.

Example 9.2 SMPs as MRGPs. Let {Z(t), t ≥ 0} be an SMP with state-space

{0, 1, 2, · · ·} and kernel G(y). Show that {Z(t), t ≥ 0} is an MRGP.

We assume that the SMP has entered the initial state at time 0. Let S0 = 0 and Sn
be the nth jump epoch in the SMP. Define Xn = Z(Sn +). Then, from Definition 8.9
on page 378, it follows that {(Xn , Sn ), n ≥ 0} is an MRS with kernel G(y). The
Markov property of the SMP at jump epochs implies that {Z(t+Sn ), t ≥ 0} depends
on {Z(t), 0 ≤ t < Sn , Xn } only through Xn . Thus {Z(t), t ≥ 0} is an MRGP.

Example 9.3 M/G/1 Queue. Let Z(t) be the number of customers in an M/G/1
queue with PP(λ) arrivals and common service time (s.t.) cdf B(·). Show that
{Z(t), t ≥ 0} is an MRGP.
412 MARKOV REGENERATIVE PROCESSES
Assume that either the system is empty initially, or a service has just started at
time 0. Let Sn be the time of the nth departure, and define Yn = Sn − Sn−1 , and
Xn = Z(Sn +). Note that if Xn > 0, Yn is the nth service time. If Xn = 0, then
Yn is the sum of an exp(λ) random variable and the nth service time. Then, due to
the memoryless property of the PP and the iid service times, we get, for i > 0 and
j ≥ i − 1,
Gij (y) = P(Xn+1 = j, Yn+1 ≤ y|X0 = i0 , X1
= i1 , Y1 ≤ y1 , · · · , Xn = i, Yn ≤ yn )
= P(Xn+1 = j, Yn+1 ≤ y|Xn = i)
= P(j − i + 1 arrivals during a s.t. and s.t. ≤ y)
Z y
(λt)j−i+1
= e−λt dB(t).
0 (j − i + 1)!
For i = 0 and j ≥ 0 we get
G0j (y) = P(Xn+1 = j, Yn+1 ≤ y|X0 = i0 , X1
= i1 , Y1 ≤ y1 , · · · , Xn = 0, Yn ≤ yn )
= P(Xn+1 = j, Yn+1 ≤ y|Xn = 0)
= P(j arrivals during a s.t. and s.t. + idle time ≤ y)
Z y
(λt)j
= (1 − e−λ(y−t) )e−λt dB(t).
0 j!
Thus {(Xn , Sn ), n ≥ 0} is an MRS with kernel G(y). The queue length process
from time Sn onwards depends on the history up to time Sn only via Xn . Hence
{(Z(t), t ≥ 0} is an MRGP. Note that we had seen in Section 7.6.1 that {Xn , n ≥ 0}
is an embedded DTMC with transition probability matrix P = G(∞). One can show
that this is consistent with Equation 7.30 on page 312.

Example 9.4 G/M/1 Queue. Let Z(t) be the number of customers in a G/M/1
queue with common inter-arrival time (i.a.t.) cdf A(·), and iid exp(µ) service times.
Show that {Z(t), t ≥ 0} is an MRGP.

Assume that an arrival has occurred at time 0. Let Sn be the time of the nth arrival,
and define Xn = Z(Sn −), and Yn = Sn − Sn−1 , the nth inter-arrival time. Due to
the memoryless property of the PP and the iid service times, we get, for 0 < j ≤ i+1,
Gij (y) = P(i + 1 − j departures during an i.a.t. and i.a.t. ≤ y)
Z y
(λt)i+1−j
= e−λt dA(t),
0 (i + 1 − j)!
MARKOV RENEWAL PROCESS AND MARKOV RENEWAL FUNCTION 413
and
i+1
X
Gi0 (y) = A(y) − Gij (y).
j=1
Thus {(Xn , Sn ), n ≥ 0} is an MRS. The queue length process from time Sn onwards
depends on the history up to time Sn only via Xn . Hence {(Z(t), t ≥ 0} is an MRGP.
Note that we had seen in Section 7.6.2 that {Xn , n ≥ 0} is an embedded DTMC with
transition probability matrix P = G(∞). One can show that this is consistent with
Equation 7.38 on page 317.

Example 9.5 M/G/1/1 Retrial Queue. Consider an M/G/1/1 retrial queue as

described in Section 7.7 of Chapter 7. Let R(t) be the number of customers in the or-
bit and I(t) be the number of customers in service at time t. Let X(t) = R(t) + I(t)
be the number of customers the system at time t. The service times are iid with com-
mon cdf B(·). The arrival process is PP(λ), and retrial times are iid exp(θ). Show
that {(R(t), I(t)), t ≥ 0} and {X(t), t ≥ 0} are MRGPs.

Assume that the server is idle initially. Let Sn be the time of the nth service com-
pletion, and define Xn = X(Sn −), and Yn = Sn − Sn−1 , the nth idle time plus the
following service time. Since the server is idle after a service completion, Xn is also
the number of customers in the orbit at the nth service completion. Thus, if Xn = i,
the (n+1)st idle time is exp(λi ), where λi = λ+iθ. Due to the memoryless property
of the PP and the iid service times, we get, for 0 ≤ i ≤ j − 1,
λ
Gij (y) = P(j − i arrivals during a s.t. and s.t. + idle time ≤ y)
λi
iθ
+ P(j − i − 1 arrivals during a s.t. and s.t. + idle time ≤ y)
λi
λ y (λt)j−i
Z
= (1 − e−λi (y−t) )e−λt dB(t)
λi 0 (j − i)!
Z y
iθ (λt)j−i−1
+ (1 − e−λi (y−t) )e−λi t dB(t).
λi 0 (j − i − 1)!
Thus {(Xn , Sn ), n ≥ 0} is an MRS with kernel G(y). The {(R(t), I(t)), t ≥ 0} and
{X(t), t ≥ 0} processes from time Sn onwards depend on the history up to time Sn
only via Xn . Hence both are MRGPs with the same embedded MRS. Note that we
had seen in Theorem 7.18 on page 321 that {Xn , n ≥ 0} is an embedded DTMC
in the {X(t), t ≥ 0} process. It is an embedded DTMC in the {(R(t), I(t)), t ≥ 0}
process as well.

9.2 Markov Renewal Process and Markov Renewal Function

Let {(Xn , Sn ), n ≥ 0} be a given Markov renewal sequence with kernel G. Define

the counting process
N (t) = min{n ≥ 0 : Sn ≤ t}. (9.3)
414 MARKOV REGENERATIVE PROCESSES
We say that the process {N (t), t ≥ 0} is regular if N (t) < ∞ with probability 1 for
all 0 ≤ t < ∞. We derive a sufficient condition for regularity next. First we need the
following notation:
X
Gi (y) = P(Y1 ≤ y|X0 = i) = Gij (y). (9.4)
j∈I

Theorem 9.3 Sufficient Condition for Regularity. The process {N (t), t ≥ 0}

defined by Equation 9.3 is regular if there exists an ǫ > 0 and a δ > 0 such that
Gi (δ) < 1 − ǫ, i ∈ I. (9.5)

Proof: Let {Yn∗ , n ≥ 1} be a sequence of iid random variables with common pmf
P(Yn∗ = 0) = 1 − ǫ, P(Yn∗ = δ) = ǫ. (9.6)
Let {Sn∗ , n ≥ 1} be renewal sequence and {N ∗ (t), t ≥ 0} be the RP generated by
{Yn∗ = Sn∗ − Sn−1
∗
, n ≥ 1}. Equations 9.5 and 9.6 imply that
P(Yn ≤ t) ≤ P(Yn∗ ≤ t), t ≥ 0, n ≥ 1.
Hence
P(Sn ≤ t) ≤ P(Sn∗ ≤ t), t ≥ 0, n ≥ 1,
which implies
P(N (t) ≥ k) ≤ P(N ∗ (t) ≥ k), k ≥ 1.
However N ∗ (t) < ∞ with probability 1, from Theorem 8.3 on page 343. Hence
N (t) < ∞ with probability 1.

The condition in Equation 9.5 is not necessary, and can be relaxed. We refer the
reader to Cinlar (1975) for weaker conditions. For a regular process {N (t), t ≥ 0},
define
X(t) = XN (t) . (9.7)
It can be seen that {X(t), t ≥ 0} is an SMP with kernel G. We say that {X(t), t ≥ 0}
is the SMP generated by the MRS {(Xn , Sn ), n ≥ 0}. Define Nj (t) be the number
of entries into state j by the SMP over (0, t], and
Mij (t) = E(Nj (t)|X0 = i), i, j ∈ I, t ≥ 0. (9.8)
Thus Mij (t) is the expected number of transitions into state j over (0, t] starting
from X(0) = i.

Definition 9.3 Markov Renewal Process. The vector valued process {N(t) =
[Nj (t)]j∈I , t ≥ 0} is called the Markov renewal process generated by the Markov
renewal sequence {(Xn , Sn ), n ≥ 0}.

Next, following the development in the renewal theory, we define the Markov re-
newal function.
MARKOV RENEWAL PROCESS AND MARKOV RENEWAL FUNCTION 415
Definition 9.4 Markov Renewal Function. The matrix M (t) = [Mij (t)]i,j∈I
is called the Markov renewal function generated by the Markov renewal sequence
{(Xn , Sn ), n ≥ 0}.

Theorem 9.4 Suppose the condition in Equation 9.5 holds. Then

 
X 1 t
Mij (t) ≤ + 1 < ∞, t ≥ 0.
ǫ δ
j∈I

∗
Pthe process {N (t), t ≥ 0} introduced in the proof of Theorem 9.3.
Proof: Consider
Since N (t) = j∈I Nj (t), we have
X X
Mij (t) = E(Nj (t)|X0 = i)
j∈I j∈I
= E(N (t)|X0 = i)
 
1 t
≤ E(N ∗ (t)) ≤ + 1 < ∞.
ǫ δ
This yields the theorem.

We define the concept of matrix convolution next. We shall find it very useful in
the study of Markov renewal processes. Let A(t) = [Aij (t)] and B(t) = [Bij (t)]
be two matrices of functions for which the product A(t)B(t) is defined. A matrix
C(t) = [Cij (t)] is called the convolution of A and B, written C(t) = A ∗ B(t), if
XZ t XZ t
Cij (t) = Aik (t − u)dBkj (u) = dAik (u)Bkj (t − u).
k 0 k 0

The proof of the next theorem introduces the concept of Markov renewal argu-
ment, and we urge the reader to become adept at using with it.

Theorem 9.5 Markov Renewal Equation. Let M (t) be the Markov renewal func-
tion generated by the Markov renewal sequence {(Xn , Sn ), n ≥ 0} with kernel G.
M (t) satisfies the following Markov renewal equation:
M (t) = G(t) + G ∗ M (t). (9.9)

Proof: Condition on X1 and S1 to get


δkj + Mkj (t − u) if u ≤ t,
E(Nj (t)|X0 = i, X1 = k, S1 = u) =
0 if u > t.
Here the Kronecker delta function is defined as δij = 1 if i = j, and 0 otherwise.
Hence we get
Z ∞X
Mij (t) = E(Nj (t)|X0 = i, X1 = k, S1 = u)dGik (u)
0 k∈I
416 MARKOV REGENERATIVE PROCESSES
Z tX
= (δkj + Mkj (t − u))dGik (u)
0 k∈I
Z t Z tX
= dGij (u) + Mkj (t − u)dGik (u)
0 0 k∈I

= Gij (t) + [G ∗ M (t)]ij .

The above equation, in matrix form, is Equation 9.9.

A k-fold convolution of matrix A with itself is denoted by A∗k . Using this notation
we get the following analog of Theorem 8.8 on page 349.

Theorem 9.6 Suppose the condition in Equation 9.5 holds. Then

∞
X
M (t) = G∗k (t), t ≥ 0.
k=1

Proof: Iterating Equation 9.9, and writing M for M (t) etc, we get
M = G+G∗M
= G + G ∗ (G + G ∗ M ) = G + G∗2 + G∗2 ∗ M
Xn
= G∗k + G∗n ∗ M.
k=1

The theorem follows by letting n → ∞ since Theorem 9.4 implies that

lim G∗n ∗ M = 0.
n→∞

Note that one consequence of the regularity condition of Equation 9.5 is that
lim G∗n e = 0, (9.10)
n→∞

where e is a vector of all ones. Define the LST of a matrix as follows

Z ∞ 
˜
Ã(s) = [Aij (s)] = −st
e dAij (t) .
0

Let C(t) = A ∗ B(t) for t ≥ 0. Then one can show that

C̃(s) = Ã(s)B̃(s). (9.11)
Using this notation we get the following analog of Theorem 8.9 on page 350.

Theorem 9.7 Suppose the condition in Equation 9.5 holds. The LST of the Markov
renewal function is given by
M̃ (s) = [I − G̃(s)]−1 G̃(s). (9.12)
KEY RENEWAL THEOREM FOR MRPS 417
Proof: Taking the LST on both sides of Equation 9.9, we get
M̃ (s) = G̃(s) + G̃(s)M̃ (s).
Thus
(I − G̃(s))M̃ (s) = G̃(s).
The regularity condition is one sufficient condition under which the inverse of (I −
G̃(s)) exists. Hence the theorem follows.
Note that if the SMP is recurrent and X0 = j then {Nj (t), t ≥ 0} is a standard
renewal process, and Mjj (t) is its renewal function; while {Ni (t), t ≥ 0} (i 6= j)
is a delayed renewal process, and Mij (t) is the delayed renewal function. Thus the
limiting behavior of the Markov renewal processes and functions can be derived from
the corresponding theorems from Chapter 8.

9.3 Key Renewal Theorem for MRPs

Equation 9.9 is called the Markov renewal equation and plays the same role as the
renewal equation in renewal theory. The Markov renewal argument, namely the tech-
nique of conditioning on X1 and S1 , generally yields an integral equation of the
following form
H(t) = D(t) + G ∗ H(t), (9.13)
where G is the kernel of the Markov renewal sequence. Such an equation is called a
Markov renewal type equation. We study such an equation in this section. The next
theorem gives the solution to the Markov renewal type equation.

Theorem 9.8 Solution of Markov Renewal Type Equation. Suppose that

sup |Dij (t)| ≤ dj (t) < ∞, j ∈ I, t ≥ 0, (9.14)
i∈I

and G satisfies the condition in Theorem 9.3. Then there exists a unique solution to
the Markov renewal type Equation 9.13 such that
sup |Hij (t)| ≤ hj (t) < ∞, j ∈ I, t ≥ 0, (9.15)
i∈I

and is given by
H(t) = D(t) + M ∗ D(t), (9.16)
where M (t) is the Markov renewal function associated with G.

Proof: First we verify that the solution in Equation 9.16 satisfies Equation 9.13.
Dropping (t) for ease of notation we get
H = D + M ∗ D = D + (G + G ∗ M ) ∗ D = D + G ∗ (D + M ∗ D) = D + G ∗ H.
Here the first equality follows from Equation 9.16, the second from Equation 9.9,
the third from associativity of matrix convolutions, and the last from Equation 9.13.
Thus any H satisfying Equation 9.16 satisfies Equation 9.13.
418 MARKOV REGENERATIVE PROCESSES
Next we establish Equation 9.15. Suppose Equation 9.14 holds and let
cj (t) = sup dj (u).
0≤u≤t

From Equation 9.16 we get

Z tX
|Hij (t)| ≤ |Dij (t) + dMik (u)Dkj (t − u)|
0 k∈I

XZ t
≤ dj (t) + cj (t) dMik (u)
k∈I 0
X
≤ cj (t)(1 + Mik (t))
k∈I
  
1 t
≤ cj (t) 1 + +1 ,
ǫ δ
where the last inequality follows from Theorem 9.4. Thus Equation 9.15 follows if
we define the right hand side above as hj (t).
To show uniqueness, suppose H1 and H2 are two solutions to Equation 9.13 sat-
isfying Equation 9.15. Then H = H1 − H2 satisfies Equation 9.15 and H = G ∗ H.
Iterating this n times we get
H = G∗n ∗ H.
Using
ĥj (t) = sup hj (u)
0≤u≤t
we get
Z tX
|Hij (t)| ≤ | dG∗n
ik (u)Hkj (t − u)|
0 k∈I
X
≤ ĥj (t) G∗n
ik (t).
k∈I

Letting n → ∞ and using Equation 9.10 we see that the right hand side goes to 0.
Hence H = 0, or, H1 = H2 . This shows uniqueness.

In general solving the generalized Markov renewal equation is a formidable task,

and we rarely attempt it. Hence we turn our attention to its limiting behavior as
t → ∞. The next theorem is the key renewal theorem for Markov renewal processes.
We shall use the following notation from Section 8.9:
Tj = min{t ≥ S1 : X(t) = j}, j ∈ I,
τi = E(S1 |X(0) = i), i ∈ I,
τij = E(Tj |X(0) = i), i, j ∈ I.
KEY RENEWAL THEOREM FOR MRPS 419
Theorem 9.9 Key Markov-Renewal Theorem. Let H be a solution to the follow-
ing Markov renewal type equation
H(t) = D(t) + G ∗ H(t). (9.17)

1. Suppose the condition of Theorem 9.3 is satisfied.

2. The SMP generated by the Markov renewal sequence is irreducible, aperiodic,
positive recurrent, and has the limiting distribution {pj , j ∈ I}.
3. Suppose D(t) = [Dij (t)] satisfies
3a. |Dij (t)| ≤ dj (t) < ∞, i ∈ I, t ≥ 0,
3b. Dij (t) (i, j ∈ I) is a difference of two non-negative bounded monotone func-
tions,
R∞P pk
3c. 0 k∈I τk |Dkj (u)|du < ∞, j ∈ I.

Then ∞
Z X pk
lim Hij (t) = Dkj (u)du. (9.18)
t→∞ 0 τk
k∈I

Proof: Since the condition of Theorem 9.3 is satisfied, and the condition [3a] holds,
we see that the unique solution to Equation 9.17 is given by Equation 9.16. In scalar
form we get
XZ t
Hij (t) = Dij (t) + dMik (u)Dkj (t − u)
k∈I 0
Z t
= Dij (t) + dMii (u)Dij (t − u)
0
XZ t
+ dMik (u)Dkj (t − u). (9.19)
k6=i 0

Conditions [3a], [3b], and [3c], and the fact that Mii (t) is a standard renewal function
of a renewal process with mean inter-renewal time τii , implies that we can use the
key renewal theorem (Theorem 8.17 on page 360) to show that
Z t Z ∞
1
lim [Dij (t) + dMii (u)Dij (t − u)] = Dij (u)du. (9.20)
t→∞ 0 τii 0
Since Mik (t) is a delayed renewal function with common mean inter-renewal time
τkk , we can use an argument similar to that in the proof of Theorem 8.26 on page 369,
to get Z t Z ∞
1
lim dMik (u)Dkj (t − u) = Dkj (u)du. (9.21)
t→∞ 0 τkk 0
Letting t → ∞ in Equation 9.19, and using Equations 9.20 and 9.21, we get
XZ t
lim Hij (t) = lim Dij (t) + lim dMik (u)Dkj (t − u)
t→∞ t→∞ t→∞ 0
k∈I
420 MARKOV REGENERATIVE PROCESSES
X 1 Z ∞
= Dkj (u)du
τkk 0
k∈I
X pk Z ∞
= Dkj (u)du,
τk 0
k∈I

where the last equality follows from Equation 8.66 on page 382.

This theorem appears in Theorem 4.17 of Chapter 10 of Cinlar (1975), and later
in Kohlas (1982) and Heyman and Sobel (1982). We do not consider the periodic
case here, since it is more involved in terms of notation. We refer the reader to Cinlar
(1975) for further details.

9.4 Extended Key Renewal Theorem

The reader can safely skip over the rest of this section without any loss of continuity.
In this section we remove the implicit assumption
lim Dij (t) = 0, i, j ∈ I
t→∞

made in Theorem 9.9. In some applications this may not be satisfied. We need to
develop a few preliminary results before we give the main result in Theorem 9.13
below.

Let {(Xn , Sn ), n ≥ 0} be a given Markov renewal sequence with kernel G, and

let {X(t), t ≥ 0} be the SMP generated by it, and M (t) be the Markov renewal
function associated with it. Let Tj , τj and τij be as defined in the previous section.
Also define
µik = E(S1 |X0 = i, X1 = k), i, k ∈ I,
s2i = E(S12 |X0 = i), i ∈ I,
s2ij = E(Tj2 |X0 = i), i, j ∈ I.
Theorem 8.32 showed how to compute the first moments τij of the first passage time
Tj . The next theorem shows how to compute the second moments.

Theorem 9.10 The second moments {s2ij , i ∈ I}, for a given j ∈ I, satisfy
X X
s2ij = s2i + 2 Gik (∞)µik τkj + Gik (∞)s2kj .
k6=j k6=j

Proof: Left as Conceptual Exercise 9.6.

Next two theorems describe the asymptotic behavior of the Markov renewal func-
tion.
EXTENDED KEY RENEWAL THEOREM 421
Theorem 9.11 Let {X(t), t ≥ 0} be an irreducible and recurrent SMP. Then
Mij (t) 1
lim = .
t→∞ t τjj

Proof: Left as Conceptual Exercise 9.7.

Theorem 9.12 Let {X(t), t ≥ 0} be an irreducible, aperiodic, and recurrent SMP.

Then
s2jj − 2τjj
2
 
t
αjj = lim Mjj (t) − = 2
t→∞ τjj 2τjj
s2jj − 2τjj τij
 
t
αij = lim Mij (t) − = 2 , i 6= j.
t→∞ τjj 2τjj

Proof: Left as Conceptual Exercise 9.8.

With these preliminary results we are ready to state the extension of Theorem 9.9
below.

Theorem 9.13 Extended Key Markov-Renewal Theorem. Let H be a solution to

the following Markov renewal type equation
H(t) = D(t) + G ∗ H(t). (9.22)

1. Suppose the condition of Theorem 9.3 is satisfied.

2. The SMP generated by the Markov renewal sequence is irreducible, aperiodic,
positive recurrent, and has limiting distribution {pj , j ∈ I}.
3. Suppose D(t) = [Dij (t)] satisfies
3a. |Dij (t)| ≤ dj (t) < ∞, i ∈ I, t ≥ 0,
3b. Dij (t) (i, j ∈ I) is a difference of two non-negative bounded monotone func-
tions,
3c. dij = limt→∞ Dij (t) exists for all i, j ∈ I,
R∞
3d. 0 (Dij (t) − dij )dt < ∞, i, j ∈ I,
R∞P pk
3e. 0 k∈I τk Dkj (u)du < ∞, j ∈ I.

Then ∞
X Z X pk
lim Hij (t) = dij + αik dkj + Dkj (u)du, (9.23)
t→∞ 0 τk
k∈I k∈I
where αik are as given in Theorem 9.12.

Before we prove this theorem we make several observations. The limit of Hij (t) has
three terms: the first two depend on i, while the third one does not. If dij = 0 for all
422 MARKOV REGENERATIVE PROCESSES
i, j ∈ I, the first two terms disappear, and we are left with a limit that is independent
of i, and we get Theorem 9.9. Clearly we do not need to assume this. All we need to
assume is that X pk
dkj = 0, j ∈ I,
τk
k∈I
which is implicit in the condition [3e] of the above theorem. As before, we do not
consider the periodic case here, since it is more involved in terms of notation. We
refer the reader to Cinlar (1975) for further details.
Proof: Since the condition of Theorem 9.3 is satisfied, and the condition [3a]
holds, we see that the unique solution to Equation 9.17 is given by Equation 9.16. In
scalar form we get
XZ t
Hij (t) = Dij (t) + dMik (u)Dkj (t − u)
k∈I 0
Z t
= Dij (t) − dij + dMii (u)(Dij (t − u) − dij )
0
XZ t
+ dMik (u)(Dkj (t − u) − dkj ) (9.24)
k6=i 0
X
+dij + Mik (t)dij . (9.25)
k∈S

Following the proof of Theorem 9.9 we get

Z t Z ∞
1
lim [Dij (t) − dij + dMii (u)(Dij (t − u) − dij )] = (Dij (u) − dij )du,
t→∞ 0 τii 0
(9.26)
and
Z t Z ∞
1
lim dMik (u)(Dkj (t − u) − dkj ) = (Dkj (u) − dkj )du. (9.27)
t→∞ 0 τkk 0
P
Finally we consider the term k Mik (t)dkj appearing in Equation 9.25. By using
pk = τk /τkk in Theorem 9.12 we get
pk
Mik (t) = t + αik + o(1)
τk
where o(1) is a function of t that converges to zero as t → ∞. Hence we have
X X pk X
Mik (t)dkj = t dkj + αik dkj + o(1).
τk
k k k

Conditions [3e] and [3e] imply that

X pk
dkj = 0.
τk
k

Hence X X
lim Mik (t)dkj = αik dkj .
t→∞
k k
EXTENDED KEY RENEWAL THEOREM 423
This yields the theorem.

We illustrate with a simple example.

Example 9.6 Two-State SMP. Consider the two-state SMP of Example 8.25 on
page 379 with kernel given by
 
0 H0 (y)
G(y) = .
H1 (y) 0
Here Hi is the cdf of the sojourn time in state i = 0, 1. Assume that Hi is a non-
defective cdf with mean τi , second moment s2i , and variance σi2 . We see that
τ00 = τ11 = τ0 + τ1 ,
τ01 = τ0 , τ10 = τ1 ,
s200 = s211 = s20 + 2τ0 τ1 + s21 ,
s210 = s21 , s201 = s20 .
Using the above in Theorem 9.12, and simplifying, we get
s20 + 2τ0 τ1 + s21 − 2(τ0 + τ1 )2
α00 = α11 =
2(τ0 + τ1 )2
σ02 + σ12 − (τ0 + τ1 )2
= ,
2(τ0 + τ1 )2
s2jj − 2τjj τij
αij = 2
2τjj
σ02 + σ12 − τi2 + τj2
= , i 6= j.
2(τ0 + τ1 )2
Let
t
Hij (t) = Mij (t) − .
τjj
We saw in Theorem 9.12 that
lim Hij (t) = αij , i, j = 0, 1.
t→∞

We shall derive the same limits using the extended key Markov-renewal theorem.
First we ask the reader to derive the following Markov renewal type equation:
H(t) = G(t) + G ∗ H(t) (9.28)
with D(t) = [Dij (t)] given by
t
1
Z
Dij (t) = (1 − δij )Hi (t) − (1 − Hi (u))dt, (9.29)
τ0 + τ1 0
where δij = 1 if i = j and 0 otherwise. Hence we have
τi
dij = (1 − δij ) − .
τ0 + τ1
424 MARKOV REGENERATIVE PROCESSES
We also have (see Example 8.28 on page 382)
τi
pi = , i = 0, 1.
τ0 + τ1
The reader should verify that
p0 p1
d0j + d1j = 0, j = 0, 1.
τ0 τ1
Thus the conditions in Theorem 9.13 are satisfied. We leave it to the reader to alge-
braically verify that Equation 9.23 reduces to
lim Hij (t) = αij . (9.30)
t→∞

This shows that Theorem 9.13 produces consistent results. Note that we cannot get
this result from Theorem 9.9.

9.5 Semi-Markov Processes: Further Results

Let {(Xn , Sn ), n ≥ 0} be a Markov renewal sequence with kernel G that satisfies the
regularity condition of Theorem 9.3. Let {X(t), t ≥ 0} be the semi-Markov process
generated by this Markov renewal sequence. We have studied the limiting behavior
of this SMP in Section 8.9 by means of an alternating renewal process. In this sec-
tion we derive the results of Theorem 8.34 by using the key Marov-renewal theorem
(Theorem 9.9).

Define
pij (t) = P(X(t) = j|X(0) = i), i, j ∈ I, t ≥ 0.
Note that by X(0) = i we implicity mean that the SMP has just entered state i at time
0. We use the Markov-renewal argument to derive a Markov-renewal type equation
for
P (t) = [pij (t)]
in the theorem below.

Theorem 9.14 Markov-Renewal Type Equation for P (t). The matrix P (t) satis-
fies the following Markov-renewal type equation:
P (t) = D(t) + G ∗ P (t), (9.31)
where D(t) is a diagonal matrix with
X
Dii (t) = 1 − Gi (t) = 1 − Gij (t), i ∈ I. (9.32)
j∈I

Proof: Condition on X1 and S1 to get


pkj (t − u) if u ≤ t,
P(X(t) = j|X0 = i, X1 = k, S1 = u) =
δij if u > t.
SEMI-MARKOV PROCESSES: FURTHER RESULTS 425
Hence we get
Z ∞ X
pij (t) = P(X(t) = j|X0 = i, X1 = k, S1 = u)dGik (u)
0 k∈I
Z tX Z ∞ X
= pkj (t − u)dGik (u) + δij dGik (u)
0 k∈I t k∈I
= [G ∗ P (t)]ij + δij (1 − Gi (t)),
which, in matrix form, yields the desired Markov-renewal type equation.

Next we use the key Markov-renewal theorem 9.9 to study the limiting behavior
of P (t). This may seem circular, since the the key Markov-renewal theorem uses
{pj , j ∈ I}, the limiting distribution of the SMP, in the limit! We assume that this
limiting distribution is obtained by using Theorem 8.34 on page 382. Then we show
that the limiting distribution produced by applying the key Markov-renewal theorem
produces the same limiting distribution.

Theorem 9.15 Limiting Distribution of an SMP. Let {X(t), t ≥ 0} be an irre-

ducible, positive recurrent, and aperiodic SMP with kernel G. Let π be a positive
solution to
π = πG(∞).
Then {X(t), t ≥ 0} has a limiting distribution [pj , j ∈ S], and it is given by
πj τj
pj = lim P(X(t) = j|X(0) = i) = P , j ∈ S. (9.33)
k∈S πk τk
t→∞

Proof: Theorem 9.14 shows that P (t) satisfies the Markov-renewal type equa-
tion 9.31 with the D(t) matrix as given in Equation 9.32. It is easy to verify that
all the conditions of Theorem 9.9 are satisfied. The {pj , j ∈ I} in the condition 2 is
given by Equation 8.66 on page 382. Note that
Z ∞ Z ∞
Dii (t)dt = (1 − Gi (t))dt = τi .
0 0
Hence Equation 9.18 reduces to
Z ∞ X pk
lim Hij (t) = Dkj (t)dt
t→∞ 0 τk
k∈I
Z ∞
pj
= Dii (u)du
τj 0
pj
= τj = pj .
τj
This proves the theorem.

We had remarked in Section 8.9 that the study of the limiting distribution of the
426 MARKOV REGENERATIVE PROCESSES
SMPs cannot stop at the limiting distribution of X(t) as t → ∞, since knowing the
value of X(t) at time t is, in general, not enough to determine the future of an SMP.
We also need to know the distribution of the remaining sojourn time at that time. We
proceed to do that here. Towards this end, define
B(t) = SN (t)+1 − t, Z(t) = XN (t)+1 . (9.34)
Thus B(t) is the time until the next transition after t (or the remaining sojourn time
in the current state at time t), and Z(t) is the state after the next transition after t. To
complete the study of the limiting distribution, we study
Hij (t) = P(X(t) = j, B(t) > x, X(t) = k|X(0) = i), (9.35)
where x ≥ 0 and k ∈ I is fixed. The next theorem gives the limit of the above
quantity as t → ∞.

Theorem 9.16 Limiting Behavior. Let {X(t), t ≥ 0} be an irreducible, positive

recurrent, and aperiodic SMP with kernel G with limiting distribution [pj , j ∈ I].
Then
pj ∞
Z
lim Hij (t) = (Gjk (∞) − Gjk (u))du, (9.36)
t→∞ τj x
where Hij (t) is as defined in Equation 9.35.

Proof: Condition on X1 and S1 to get

P(X(t) = j, B(t) > x, Z(t) = k|X0 = i, X1 = r, S1 = u)

 Hrj (t − u) if u ≤ t,
= 0 if t < u ≤ t + x
δij δrk if u > t + x.


Hence we get
Z ∞ X
Hij (t) = P(X(t) = j, B(t) > x, Z(t) = k|X0 = i, X1 = r, S1 = u)dGir (u)
0 r∈I
Z tX Z ∞ X
= Hrj (t − u)dGir (u) + δij δrk dGir (u)
0 r∈I t+x r∈I
Z ∞ Z tX
= δij dGik (u) + Hrj (t − u)dGir (u)
t+x 0 r∈I

= [G ∗ H(t)]ij + δij (Gik (∞) − Gik (t + x)),

which, in matrix form, yields
H(t) = D(t) + G ∗ H(t),
where D(t) is a diagonal matrix with
Dii (t) = Gik (∞) − Gik (t + x).
SEMI-MARKOV PROCESSES: FURTHER RESULTS 427
It is easy to see that all the conditions of Theorem 9.9 are satisfied. Hence Equa-
tion 9.18 yields
Z ∞X
pr
lim Hij (t) = Drj (u)du
t→∞ 0 r∈I rrτ
pj ∞
Z
= (Gjk (∞) − Gjk (u + x))du,
τj x
as desired.

The next theorem follows immediately from the above theorem.

Theorem 9.17 Under the conditions of Theorem 9.16

1 ∞
Z
lim P(X(t) = j, B(t) > x|X(0) = i) = pj (1 − Gj (u))du, (9.37)
t→∞ τj x
and ∞
1
Z
lim P(B(t) > x|X(t) = j) = (1 − Gj (u))du, (9.38)
t→∞ τj x

Proof: Follows from Equation 9.36

X
P(X(t) = j, B(t) > x|X(0) = i) = P(X(t) = j, B(t) > x, Z(t) = k|X(0) = i)
k∈I

and recognizing that

X
(Gjk (∞) − Gjk (t)) = 1 − Gj (t).
k∈I

Now, we can write

P(X(t) = j, B(t) > x|X(0) = i) = P(X(t) = j)P(B(t) > x|X(0) = i, X(t) = j).
Taking limits as t → ∞ on both sides we see that
1 ∞
Z
pj (1 − Gj (u))du = pj lim P(B(t) > x|X(0) = i, X(t) = j).
τj x t→∞

Hence we get
∞
1
Z
lim P(B(t) > x|X(0) = i, X(t) = j) = (1 − Gj (u))du.
t→∞ τj x

Since the right hand side is independent of i we get the desired result in the
theorem.

The above theorem has an interesting interpretation: in the limit the state of the
SMP is j with probability pj , and given that the current state j, the remaining sojourn
time in that state has the same distribution as the equilibrium distribution associated
with Gj . In hindsight, this is to be expected. We close this section with an example.
428 MARKOV REGENERATIVE PROCESSES
Example 9.7 Remaining Service Time in an M/G/1 Queue. Let Z(t) be the
number of customers at time t in an M/G/1 queue with PP(λ) arrivals and iid service
times with common cdf F (·) and common mean τ . Define U (t) to be the remaining
service time of the customer in service at time t, if Z(t) > 0. If Z(t) = 0, define
U (t) = 0. Show that
1 ∞
Z
lim P(U (t) > x|Z(t) > 0) = (1 − F (u))du.
t→∞ τ x

Let S0 = 0. If Z(Sn +) = 0 define Sn+1 be the time of arrival of the next

customer, and if Z(Sn +) > 0 define Sn+1 be the time of the next departure. Let
Xn = Z(Sn +). Then it can be seen that {(Xn , Sn ), n ≥ 0} is a Markov renewal
sequence and 
F (t) if i > 0,
Gj (t) =
1 − e−λt if i = 0.
Let {X(t), t ≥ 0} be the SMP generated by the Markov renewal sequence
{(Xn , Sn ), n ≥ 0}. Let B(t) be as defined in Equation 9.34. Then it is clear that if
X(t) > 0, then B(t) = U (t). From Equation 9.38 we see that, for a stable queue,
1 ∞
Z
lim P(B(t) > x|X(t) = j) = (1 − F (u))du, j > 0. (9.39)
t→∞ τ x
This yields the desired result, since Z(t) > 0 is equivalent to X(t) > 0. Thus, in
steady state when the server is busy, the remaining service time distribution is given
by the equilibrium distribution associated with the service time distribution. Thus
the expected remaining service time in steady state, given that the server is busy, is
given by E(S 2 )/2E(S). This fact is very important in many waiting time calculations.
Another curious fact to note is that Equation 9.39 fails if X(t) = j on the left hand
side is replaced by Z(t) = j!

9.6 Markov Regenerative Processes

We began this chapter with the definition of an MRGP in Definition 9.2. Markov
regenerative processes (MRGPs) are to the Markov renewal sequence what regener-
ative processes are to renewal sequences. Let {Z(t), t ≥ 0} be an MRGP with an
embedded MRS {(Xn , Sn ), n ≥ 0}. We begin by assuming that the Z(t) and Xn
both take values in a discrete set I. We have seen several examples of MRGPs in
Section 9.1.
As with the SMPs we study the transient behavior of the MRGPs with countable
state-space I by concentrating on
Hij (t) = P(Z(t) = j|Z(0) = i), i, j ∈ I.
The next theorem gives the Markov renewal type equation satisfied by H(t) =
[Hij (t)].

Theorem 9.18 Transient Distribution of an MRGP. Let {Z(t), t ≥ 0} be an

MARKOV REGENERATIVE PROCESSES 429
MRGP with embedded MRS {(Xn , Sn ), n ≥ 0} with kernel G. Assume that Z(t) ∈ I
for all t ≥ 0 and Xn ∈ I for all n ≥ 0, where I is a discrete set. Let
Dij (t) = P(Z(t) = j, S1 > t|Z(0) = i), i, j ∈ I,
and D(t) = [Dij (t)]. Then H(t) satisfies the following Markov renewal type equa-
tion:
H(t) = D(t) + G ∗ H(t).

Proof: Condition on X1 and S1 to get

P(Z(t) = j|X0 = i, X1 = k, S1 = u)

Hkj (t − u) if u ≤ t,
=
P(Z(t) = j|X0 = i, X1 = k, S1 = u) if u > t.
Hence we get
Z ∞ X
Hij (t) = P(Z(t) = j|X0 = i, X1 = k, S1 = u)dGik (u) +
tk∈I
XZ t
Hkj (t − u))dGik (u)
k∈I 0

= P(Z(t) = j, S1 > t|X0 = i) + [G ∗ H(t)]ij ,

which yields the Markov renewal equation in the theorem.

Note that D(t) contains the information about the behavior of the MRGP over
the first cycle (0, S1 ). Thus the above theorem relates the behavior of the process at
time t to its behavior over the first cycle. In practice computing D(t) is not easy, and
solving the Markov renewal equation to obtain H(t) is even harder. Hence, following
the now well trodden path, we study its limiting behavior in the next theorem.

Theorem 9.19 Limiting Behavior of MRGPs. Let {Z(t), t ≥ 0} be an MRGP

with embedded MRS {(Xn , Sn ), n ≥ 0} with kernel G satisfying conditions of The-
orem 9.3. Assume that Z(t) ∈ I for all t ≥ 0 and Xn ∈ I for all n ≥ 0, where I
is a discrete set. Let αkj be the expected time spent by the MRGP in state j during
(0, S1 ) starting with X0 = k. Furthermore, suppose that the sample paths of the
MRGP are right continuous with left limits, the SMP {X(t), t ≥ 0} generated by the
MRS {(Xn , Sn ), n ≥ 0} is irreducible, aperiodic, positive recurrent, with limiting
distribution [pj , j ∈ I]. Then
X αkj
lim P(Z(t) = j|X0 = i) = pk , i, j ∈ I. (9.40)
t→∞ τk
k∈I

Proof: Consider the Markov renewal type equation derived in Theorem 9.18. It is
straightforward to verify that the conditions of Theorem 9.9 are satisfied. Thus Equa-
tion 9.18 can be used to compute the limit of Hij (t) = P(Z(t) = j|X0 = i) as
430 MARKOV REGENERATIVE PROCESSES
follows: ∞
Z X pk
lim Hij (t) = Dkj (u)du. (9.41)
t→∞ 0 τk
k∈I
Now,
αkj = E(time spent by the MRGP in state j during (0, S1 )|X0 = k)
Z S1  

= E 1{Z(t)=j} dt X0 = k
0
Z ∞ Z S1  

= E 1{Z(t)=j} dt X0 = k, S1 = u dGk (u)
Z0 ∞ Z s 0
= P(Z(t) = j|X0 = k, S1 = u)dtdGk (u)
0 0
Z ∞Z ∞
= P(Z(t) = j|X0 = k, S1 = u)dGk (u)dt
Z0 ∞ t
= P(Z(t) = j, S1 > t|X0 = k)dt
Z0 ∞
= Dkj (t)dt.
0

Substituting in Equation 9.41 we get Equation 9.40.

Note that the limiting distribution of the MRGP given in Equation 9.40 is indepen-
dent of the initial distribution of the MRS, or, initial value of X0 . The distribution
itself can be intuitively explained as follows: Since every time the SMP {Z(t), t ≥ 0}
visits state k it spends τk amount of time there, we can interpret αk /τk as the time
time spent by the MRGP in state j per unit time spent in state k by the SMP. Since
pk is the fraction of the time spent in state k by the SMP in the long run, we can
compute
lim P(Z(t) = j|X0 = i)
t→∞
= Long run fraction of the time spent by the MRGP in state j
X
= [Long run fraction of the time spent by the SMP in state k] ×
k∈I
[Long run time spent by the MRGP in state j per unit time spent by
the SMP in state k]
X αkj
= pk .
τk
k∈I

Now let us relax the assumption that the Z(t) and Xn take values in the same
discrete set I. Suppose Z(t) ∈ S for all t ≥ 0 and Xn ∈ I for all n ≥ 0. We can see
that 9.40 remains valid even in this case if S is also discrete. If S is continuous, say
(−∞, ∞), we can proceed as follows: fix an x ∈ S, and define
Y (t) = 1{Z(t)≤x} , t ≥ 0.
MARKOV REGENERATIVE PROCESSES 431
Then it is clear that {Y (t), t ≥ 0} is also an MRGP with the same embedded MRS
{(Xn , Sn ), n ≥ 0}. We do need to assume that the sample paths of {Z(t), t ≥ 0}
are sufficiently nice so that the sample paths of {Y (t), t ≥ 0} are right continuous
with left limits. Next define αk (x) as the expected time spent by the {Z(t), t ≥ 0}
process in the set (−∞, x] during (0, S1 ) starting with X0 = k. Then we can show
that
X αk (x)
lim P(Z(t) ≤ x|X0 = i) = pk , i ∈ I, x ∈ S.
t→∞ τk
k∈I
We illustrate with two examples.

Example 9.8 SMPs as MRGPs. Let {X(t), t ≥ 0} be an SMP with kernel G. We

saw in Example 9.2 that an SMP is a special case of an MRGP. Since
X(t) = X0 , 0 ≤ t < S1 ,
we have 
τj if k = j,
αkj =
0 if k =
6 j.
Substituting in Equation 9.40 we get
X αkj
lim P(X(t) = j|X0 = i) = pk = pj ,
t→∞ τk
k∈I

as expected.

Example 9.9 Machine Maintenance. Consider a machine consisting of two iden-

tical components in series. The lifetimes of the components are iid exp(µ) random
variables. A single repairperson repairs these components, the repair times being iid
random variables with common cdf A(·) and mean τ . The components are as good
as new after repairs. Assume that the functioning component can fail even if the ma-
chine fails. Compute the long run fraction of the time that the machine is working.

Let Z(t) be the number of functioning components at time t. Let S0 = 0. If

Z(Sn +) = 2, define Sn+1 as the time of first failure after Sn . If Z(Sn +) = 0 or
1, define Sn+1 as the time of completion of repair of the item under repair at time
Sn +. Define Xn = Z(Sn +). Note that Xn ∈ {1, 2} even if Z(t) ∈ {0, 1, 2}. Then
{(Xn , Sn ), n ≥ 0} is a Markov renewal sequence with kernel
 
G11 (x) G12 (x)
G(x) = ,
G21 (x) G22 (x)
where
G11 (x) = P(Repair time ≤ x, one failure during repair|X0 = Z(0) = 1)
Z x
= (1 − e−µt )dA(t),
0
G12 (x) = P(Repair time ≤ x, no failure during repair|X0 = Z(0) = 1)
432 MARKOV REGENERATIVE PROCESSES
Z x
= e−µt dA(t),
0
G21 (x) = P(Failure time ≤ x|X0 = Z(0) = 2)
Z x
= (1 − e−µt )dA(t),
0
G22 (x) = 0.
Thus
G1 (x) = G11 (x) + G12 (x) = A(x), τ1 = τ
and
G2 (x) = G21 (x) + G22 (x) = 1 − e−2µx , τ2 = 1/2µ.
The transition probability matrix of {Xn , n ≥ 0} is
 
1 − Ã(µ) Ã(µ)
P = G(∞) = ,
1 0
R∞
where Ã(µ) = 0 e−µt dA(t). To use Equation 9.40 we need to compute the limiting
distribution of the SMP {X(t), t ≥ 0} generated by the MRS {(Xn , Sn ), n ≥ 0}
described above. We do that using Theorem 8.34 on page 8.34. We see that
π = [π1 π2 ] = [1 Ã(λ)]
satisfies π = πP . Substituting in Equation 8.66 we get
π1 τ1 2µτ
p1 = lim P(X(t) = 1) = = ,
t→∞ π1 τ1 + π2 τ2 2µτ + Ã(µ)
π2 τ2 Ã(µ)
p2 = lim P(X(t) = 2) = = .
t→∞ π1 τ1 + π2 τ2 2µτ + Ã(µ)
From Equation 9.41, we see that the long run fraction of the time the machine is
up is given by
α12 α22
lim P(Z(t) = 2) = p1 + p2 .
t→∞ τ1 τ2
Thus we need to compute α12 and α22 . We have
1
α12 = 0, α22 = .
2µ
Substituting in the previous equation and simplifying, we see that the long run frac-
tion of the time the machine is up is given by
Ã(µ)
.
2µτ + Ã(µ)
Using the same procedure we can compute the entire limiting distribution of the
MRGP.
APPLICATIONS TO QUEUES 433
9.7 Applications to Queues

In this section we apply the theory of MRGPs to the queueing systems. Specifically
we apply it to the birth and death queues, the M/G/1 queue, the G/M/1 queue, and
the M/G/1/1 retrial queue.

9.7.1 The Birth and Death Queues

Let Z(t) be the number of customers in a queueing system at time t. Assume that
{Z(t), t ≥ 0} is a birth and death process on {0, 1, 2, · · ·} with birth parameters
λi > 0 for i ≥ 0, and µi > 0 for i ≥ 1. We studied the limiting behavior of this
process in Example 6.35 on page 242. There we saw that such a queueing system is
stable if
X∞
ρi < ∞,
i=0
where ρ0 = 1, and
λ0 λ1 · · · λi−1
ρi = , i ≥ 1,
µ1 µ2 · · · µi
and its limiting distribution is given by
ρj
pj = P∞ , j ≥ 0.
i=0 ρi
Thus pj is the steady-state probability that there are j customers in the system. From
Section 7.1 recall that p∗j is the steady-state probability that an entering customer sees
j customers ahead of him at the time of entry. We saw in Theorem 7.3 that if λi = λ
for all i ≥ 0, we can apply PASTA (Theorem 7.3 on page 283, and noting that all
arriving customers enter) to see that p∗j = pj . We use the theory of MRGPs to derive
a relationship between πj∗ and pj in the general case in the next theorem.

Theorem 9.20 The Birth and Death Queue. For a stable birth and process queue,
λj pj
πj∗ = P∞ , j = 0, 1, 2, · · · . (9.42)
k=0 λk pk

Proof: Let S0 = 0, and Sn be the time of the nth upward jump in the {Z(t), t ≥ 0}.
Thus the nth entry to the queueing system takes place at time Sn . Let Xn = Z(Sn −),
the number of customers as seen by the nth entering customer. Note that Xn = k
implies that Z(Sn +) = k + 1. Since {Z(t), t ≥ 0} is a CTMC, {(Xn , Sn ), n ≥ 0}
is a Markov renewal sequence. Note that {Z(t), t ≥ 0} decreases over [0, S1 ), and
{Xn , n ≥ 0} is a DTMC with transition probabilities
λj
pkj = mkj , 0 ≤ j ≤ k + 1, (9.43)
λj + µj
where
k+1
Y µr
mk,k+1 = 1, mkj = , 0 ≤ j ≤ k.
λ + µr
r=j+1 r
434 MARKOV REGENERATIVE PROCESSES
Since
πj∗ = lim P(Xn = j), j ≥ 0, (9.44)
n→∞
we see that {πj∗ , j ≥ 0} satisfy
∞ ∞
X X λj
πj∗ = πk∗ pkj = πk∗ mkj , j ≥ 0. (9.45)
λj + µj
k=j−1 k=j−1

We can interpret mkj as the probability that the {Z(t), t ≥ 0} process visits state j
over [0, S1 ) starting with X0 = k. We can treat {Z(t), t ≥ 0} as an MRGP with the
embedded MRS {(Xn , Sn ), n ≥ 0}. Let {X(t), t ≥ 0} be the SMP generated by the
MRS {(Xn , Sn ), n ≥ 0}. We change the notation slightly and use
pj = lim P(Z(t) = j), j ≥ 0, (9.46)
t→∞

and
p̄j = lim P(X(t) = j), j ≥ 0. (9.47)
t→∞
We use Theorem 8.34 on page 382 to get
πj∗ τj
p̄j = P∞ ∗ , j ≥ 0.
k=0 πk τk
Now, let αkj be as defined in Theorem 9.19. Note that the MRGP can visit state j
at most once during [0, S1 ), mkj is the probability that the MRGP visits state j over
(0, S1 ) starting with X0 = k, and 1/(λj + µj ) is the average time it spends in state
j once its reaches state j. These observations can be combined to yield
1
αkj = mkj , 0 ≤ j ≤ k + 1.
λj + µj
Using the above results in Theorem 9.19 we get
∞ P∞ ∗
αkj k=j−1 πk αkj
X
pj = p̄k = P∞ ∗ , j ≥ 0.
τk k=0 πk τk
k=j−1

Now we have
∞ ∞
X X 1
πk∗ αkj = πk∗ mkj
λj + µj
k=j−1 k=j−1
∞
1X λj
= πk∗ mkj
λj λj + µj
k=j−1
∞
1 X
= πk∗ pkj (from Eq. 9.43)
λj
k=j−1
πj∗
= (from Eq. 9.45).
λj
APPLICATIONS TO QUEUES 435
Hence pj ∝ πj∗ /λj , or πj∗ ∝ λj pj . Since the πj∗ must add up to 1, we get Equa-
tion 9.42 as desired.

The relation in Equation 9.42 has a simple intuitive explanation: λj pj is the rate at
which state j + 1 is entered from state j in steady state. Hence it must be proportional
to the probability that an entering customer sees j customers ahead of him.

Now let πj be the limiting probability that a departing customer leaves behind j
customers in the system. One can define an appropriate Markov renewal sequence
and follow the steps of the proof of Theorem 9.20 to show that
µj+1 pj+1
πj = P∞ , j = 0, 1, 2, · · · . (9.48)
k=1 µk pk
However, for a positive recurrent birth and death process we have
λj pj = µj+1 pj+1 , j ≥ 0.
Hence we get
λj pj
πj = P∞ = πj∗ , j = 0, 1, 2, · · · . (9.49)
k=0 λ k p k
Thus, in steady state, the distribution of the number of customers as seen by an ar-
rival is the same as seen by a departure. This is a probabilistic proof of the general
Theorem 7.2 on page 281.

9.7.2 The M/G/1 Queue

Let Z(t) the number of customers in a stable M/G/1 queue at time t with arrival rate
λ and service time cdf B(·) with mean τ . We showed in Example 9.3 that {Z(t), t ≥
0} is an MRGP with the embedded Markov renewal sequence {(Xn , Sn ), n ≥ 0} as
defined there. Let {X(t), t ≥ 0} be the SMP generated by this MRS. Let pj and p̄j
be as defined in Equations 9.46 and 9.47. Also define
πj = lim P(Xn = j), j ≥ 0.
n→∞

Thus πj is the probability that a departing customer leaves behind j customers in

steady-state. We had used PASTA and other properties of general queueing systems
to show that πj = pj in Example 7.6. Here we derive this result by using the theory
of MRGPs.

Theorem 9.21 M/G/1 Queue. For a stable M/G/1 queue

pj = πj , j ≥ 0.

Proof: We have shown in Theorem 7.10 that {Xn , n ≥ 0} is a DTMC with transi-
tion probability matrix P as given there. We need the following quantities to apply
436 MARKOV REGENERATIVE PROCESSES
Theorem 8.34 on page 382:
1
τ0 = + τ, τj = τ, j ≥ 1.
λ
Using Equation 8.66 we get
πj τj
p̄j = P∞ , j ≥ 0.
k=0 πk τk
From Equation 7.35 on page 314 we get
π0 = 1 − λτ.
Hence we have
∞
X
πk τk = τ + π0 /λ = 1/λ.
k=0
Thus we get
p̄j = λπj τj . (9.50)
In order to use Theorem 9.19 we need to compute αkj , the expected time spent by
the MRGP in state j over [0, S1 ) starting with X0 = k. Note that the sojourn time of
the SMP in state k > 0 is given by Gk (x) = B(x). For j ≥ k > 0 we have
Z S1  

αkj = E 1Z(t)=j dt X0 = k

0
Z ∞ Z S1
 

= E 1Z(t)=j dt X0 = k, S1 = u dGk (u)

Z0 ∞ Z u 0 
= P(Z(t) = j  X0 = k, S1 = u)dtdB(u)

Z0 ∞ Z0 u
= P(j − k arrivals in [0, t))dtdB(u)
0 0
Z ∞Z u
(λt)j−k
= e−λt dtdB(u)
0 0 (j − k)!
j−k
" #
1 X
= 1− αr
λ r=0

where
∞
(λt)i
Z
αi = e−λt dB(t), i ≥ 0.
0 i!
In a similar way we can show that
α00 = 1/λ, α0j = α1j , j ≥ 1.
Substituting in Equation 9.41 we get
∞
X αkj
pj = p̄k
τk
k=0
APPLICATIONS TO QUEUES 437
j
X
= λ πk αkj (Use Eq. 9.50)
k=0
j
!
X
= λ π0 α0j + πk αkj
k=1
= πj ,
where the last equality follows by using the balance equation π = πP repeatedly to
simplify the right hand side.

We cannot use theory of MRGP to prove πj∗ = pj in a similar way. However, we

have seen in Example 7.6 that πj∗ = πj , and hence we have a direct proof of PASTA
for the M/G/1 queue.

9.7.3 The G/M/1 Queue

Let Z(t) the number of customers in a stable G/M/1 queue at time t with common
inter-arrival time cdf A(·) with mean 1/λ and iid exp(µ) service times. We showed
in Example 9.4 that {Z(t), t ≥ 0} is an MRGP with the embedded Markov renewal
sequence {(Xn , Sn ), n ≥ 0} as defined there. Let {X(t), t ≥ 0} be the SMP gener-
ated by this MRS. Let pj , p̄j and πj∗ be as defined in Equations 9.46, 9.47, and 9.44.
Here we use the theory of MRGPs to give a computational proof of Theorem 7.16.

Theorem 9.22 Limiting Distribution of a G/M/1 Queue. For a G/M/1 queue

with ρ = λ/µ < 1 the limiting distributions {pj , j ≥ 0} and {πj∗ , j ≥ 0} are related
as follows:
p0 = 1 − ρ,
∗
pj = ρπj−1 , j ≥ 1.

Proof: We have shown in Theorem 7.14 on page 317 that {Xn , n ≥ 0} (it was
denoted by {Xn∗ , n ≥ 0} there) is a DTMC with transition probability matrix P as
given in Equation 7.38. We need the following quantities to apply Theorem 8.34 on
page 382:
τj = 1/λ, j ≥ 0.
Substituting in Equation 8.66 we get
p̄j = πj∗ , j ≥ 0. (9.51)
In order to use Theorem 9.19 we need to compute αkj . Note that the sojourn time
of the SMP in state k ≥ 0 is given by Gk (x) = A(x). Going through the same
calculations as in the proof of Theorem 9.21 we get, for 1 ≤ j ≤ k + 1
k+1−j
" #
1 X
αkj = 1− αr ,
µ r=0
438 MARKOV REGENERATIVE PROCESSES
where
∞
(µt)i
Z
αi = e−µt dA(t), i ≥ 0.
0 i!
Substituting in Equation 9.41 we get, for j ≥ 1,
∞
X αkj
pj = p̄k
τk
k=j
∞
X
= λ πk∗ αkj (Use Eq. 9.51)
k=j
∞ k+1−j
" #
λ X ∗ X
= πk 1 − αr
µ r=0
k=j−1
∗
= ρπj−1 ,
where the last equality follows by using the balance equation π ∗ = π ∗ P repeatedly
to simplify the right hand side. Finally, we have
∞
X ∞
X
∗
p0 = 1 − pj = 1 − ρ πj−1 = 1 − ρ.
j=1 j=1

This completes the proof.

9.7.4 The M/G/1/1 Retrial Queue

Consider the M/G/1/1 retrial queue as described in Example 9.5. Using the no-
tation there we see that {Z(t) = (R(t), I(t)), t ≥ 0} is an MRGP with the MRS
{(Xn , Sn ), n ≥ 0} defined there. Let {X(t), t ≥ 0} be the SMP generated by this
MRS. Note that this is different than the X(t) defined in Example 9.5. Let p̄j be as
defined in Equations 9.46. We have derived the limiting distribution of the embedded
DTMC {Xn , n ≥ 0} in Theorem 7.19. We use those results to derive the limiting
distribution of (R(t), I(t)) as t → ∞.

Let
p(j,i) = lim P((R(t), I(t)) = (j, i)), j ≥ 0, i = 0, 1, (9.52)
t→∞
and define the generating functions
∞
X
φi (z) = z k p(j,i) , i = 0, 1.
j=0
APPLICATIONS TO QUEUES 439
Theorem 9.23 M/G/1/1 Retrial Queue. Suppose ρ = λτ < 1. The
!
λ 1 1 − G̃(λ − λu)
Z
φ0 (z) = (1 − ρ) exp − du , (9.53)
θ z G̃(λ − λu) − u
!
G̃(λ − λz) − 1 λ 1 1 − G̃(λ − λu)
Z
φ1 (z) = (1 − ρ) · exp − du . (9.54)
z − G̃(λ − λz) θ z G̃(λ − λu) − u

Proof: We have shown in Theorem 7.18 on page 321 that {Xn , n ≥ 0} is a DTMC.
Let
πk = lim P(Xn = k), k ≥ 0.
n→∞
In Theorem 7.19 we computed the generating function
∞
X
φ(z) = z k πk .
k=0

We also derived a peculiar, but useful, generating function

∞
X πk
ψ(z) = λ zk
λ + kθ
k=0

in Equation 7.51 on page 323. From there we see that

ψ(1) = 1 − ρ.
Next we compute the relevant quantities.
1
τk = + τ, k ≥ 0,
λ + kθ
1
αk,(k,0) = , k ≥ 0,
λ + kθ
αk,(j,0) = 0, if k 6= j.
The quantities αk,(j,1) are more complicated, however, we do not need them, as we
shall see. We have
∞ ∞
X X πk
πk τk = τ +
λ + kθ
k=0 k=0
ψ(1) 1
= τ+ = .
λ λ
This is to be expected, since the left hand side is the expected time between two
successive departures in steady state, and hence must equal 1/λ, the expected time
between two consecutive arrivals. Substituting in Equation 8.66 we get
p̄j = λπj τj . (9.55)
Substituting in Equation 9.41 we get
∞
X αk,(j,0)
p(j,0) = p̄k
τk
k=0
440 MARKOV REGENERATIVE PROCESSES
∞
X
= λ πk αk,(j,0) (Use Eq. 9.55)
k=0
λ
= πj .
λ + jθ
Thus
∞
X λ
φ0 (z) = zk πj = ψ(z).
j=0
λ + jθ
The Equation 9.53 now follows from Equations 7.51 and 7.52.

Now let φ(z) be the limiting generating function of Xn as derived in Equation 7.46
on page 322. We saw in Section 7.7 that it is also the limiting generating function of
R(t) + I(t) as t → ∞. Hence we get
φ(z) = φ0 (z) + zφ1 (z).
Substituting for φ(z) and φ0 (z) we get the expression for φ1 (z) in Equation 9.54.

Note that the limiting probability that the server is idle is given by
φ0 (1) = 1 − ρ,
as expected. It is a bit surprising that this is independent of the retrial rate θ as long
as it is positive! We could not derive this result in our earlier analysis of the retrial
queue in Section 7.7.

9.8 Modeling Exercises

9.1 Let Z(t), t ≥ 0} be a birth and death process. Let Sn be the nth downward jump
epoch and define Xn = X(Sn +). Assume that S0 = 0. Show that {Z(t), t ≥ 0} is
an MRGP with {(Xn , Sn ), n ≥ 0} as the embedded MRS. Compute its kernel.

9.2 Let Z(t) be the number of customers in an M/G/1/K queue with PP(λ) ar-
rivals and common service time cdf B(·). Assume that either the system is empty
initially, or a service has just started at time 0. Let Sn be the time of the nth de-
parture, and define Xn = Z(Sn +). Show that {Z(t), t ≥ 0} is an MRGP with
{(Xn , Sn ), n ≥ 0} as the embedded MRS. Compute its kernel.

9.3 Let Z(t) be the number of customers in an G/M/1/K queue with common
inter-arrival time cdf A(·), and iid exp(µ) service times. Assume that an arrival
has occurred at time 0. Let Sn be the time of the nth arrival (who may or or may
not enter), and define Xn = Z(Sn −). Show that {Z(t), t ≥ 0} is an MRGP with
{(Xn , Sn ), n ≥ 0} as the embedded MRS. Compute its kernel.
MODELING EXERCISES 441
9.4 Consider a closed queueing network with two single-server nodes and N cus-
tomers. After completing service at node 1 (2) a customer joins node 2 (1). The
service times at node i are iid exp(µi ), (1, 2). Let Z(t) be the number of customers
at node 1 at time t, and Sn be the time of nth service completion at node 2, which is
also an arrival instant at node 1. Assume S0 = 0, i.e., a service completion at node
2 has occurred at time 0. Let Xn = Z(Sn −). Show that {Z(t), t ≥ 0} is an MRGP
with {(Xn , Sn ), n ≥ 0} as the embedded MRS. Compute its kernel.

9.5 Consider a workshop with N machines and a single repairperson. Lifetimes of

the machines are iid exp(µ), and the repair times are iid with common cdf R(·). Let
Z(t) be the number of down machines at time t, Sn the nth repair completion time,
and Xn = Z(Sn +). Assume that a repair has just completed at time 0. Show that
{Z(t), t ≥ 0} is an MRGP with {(Xn , Sn ), n ≥ 0} as the embedded MRS. Compute
its kernel.

9.6 Consider the queueing network of Modeling Exercise 9.4. Suppose one of the
N customer is tagged, and let Sn be the time of his nth return to node 1. Assume
S0 = 0, i.e., the tagged customer joins node 1 at time 0. Let Xn = Z(Sn −). Show
that {Z(t), t ≥ 0} is an MRGP with {(Xn , Sn ), n ≥ 0} as the embedded MRS.
Compute its kernel.

9.7 A machine can exist in one of N + 1 states labeled {0, 1, 2, · · · , N }, with state
0 representing a new machine, and state N representing a failed machine, and the in-
between states indicating increasing levels of deterioration. Left to itself the machine
changes states according to a DTMC with transition probability matrix H = [hij ].
The maintenance policy calls for repairing the machine whenever it reaches a state
k or more, where 0 < k ≤ N is a given integer. Suppose the repair takes one
unit of time and transforms the machine from state i(≥ k) to state j(< k) with
probability aij . Let Sn be the nth repair completion time and Xn the state of the
machine immediately after the nth repair completion. Assume that S0 = 0. Let Z(t)
be the state of the machine at time t. Show that {Z(t), t ≥ 0} is an MRGP with
{(Xn , Sn ), n ≥ 0} as the embedded MRS. Compute its kernel.

9.8 A machine is maintained by a robot. The lifetime of a new machine is an Exp(µ)

random variable. When the machine fails, it is repaired by a robot in a fixed amount
of time r. After repair the machine is as good as new. After repair, the robot needs a
down time of a fixed amount d. If the machine fails when the robot is down, it has to
wait for the robot to become available again. Let Z(t) be 0 if the machine is down
and waiting for repair, 1 if the machine is under repair, and 2 if the machine is up.
Show that {Z(t), t ≥ 0} is an MRGP.

9.9 A high-speed network transmits cells (constant length packets of data) over
communication channels. At its input ports it exercises access control by dropping
incoming cells if the input rate gets too high. One such control mechanism is de-
scribed here. The controller generates r tokens at times n = 0, 1, 2, · · · into a token
pool of size M . Tokens that exceed the capacity are lost. The cells arrive according
442 MARKOV REGENERATIVE PROCESSES
to a PP(λ). If there is a token available when a cell arrives, it grabs one token and
enters the network immediately, else the cell is prohibited from entering the network
and is permanently lost. Let X(t) be the number of tokens in the token pool at time
t. Model {X(t), t ≥ 0} as an MRGP.

9.10 Consider following modification to the access control mechanism of Model-

ing Exercise 9.9. The incoming cells are queued up in a buffer of size L. When an
incoming cell finds no tokens, it waits in the buffer and enters the network as soon
as the next token becomes available. Thus the token pool and the buffer cannot be
simultaneously non-empty. Let Y (t) be the number of cells in the buffer at time t,
and define Z(t) = X(t) − Y (t). Model {Z(t), t ≥ 0} as an MRGP.

9.11 Customers arrive at a service station according to a Poisson process with rate
λ. Servers arrive at this station according to an independent renewal process with
iid inter-arrival times with mean τ and second moment s2 . Each incoming server
removes each of the waiting customer with probability α > 0 in an independent
fashion, and departs immediately. Let X(t) be the number of customers in the system
at time t. Model {X(t), t ≥ 0} as an MRGP by identifying an appropriate MRS
embedded in it.

9.9 Computational Exercises

9.1 Consider a system that cycles through N states labeled 1, 2, · · · , N starting in

state 1. It stays in state i for a random amount of time with cdf Hi and then jumps to
state i + 1 if i < N and state 1 if i = N . Let Sn be the time of the nth transition,
with S0 = 0, and Xn be the state of the system after the nth jump. Compute the LST
M̃ (s) of the Markov renewal function M (t) for the MRS {(Xn , Sn ), n ≥ 0}.

9.2 Let {Z(t), t ≥ 0} be a CTMC with generator matrix Q. For the MRS described
in Example 9.1, compute the LST M̃ (s) of the Markov renewal function M (t).

9.3 Compute the limiting probability that 0 or 1 components are functional in the
machine in Example 9.9.

9.4 Consider a two state CTMC {X(t), t ≥ 0} on {0, 1} with generator matrix
 
−λ λ
Q= .
µ −µ
Let Hij (t) be the expected time spent in state j by the CTMC up to time t starting
in state i at time 0. Derive a Markov renewal type equation for H(t). Solve it by
inverting its LST H̃(s).

9.5 Consider an M/M/1 queue with balking as follows: an incoming customer

joins the system with probability 1/(j + 1) if he sees j customers ahead of him.
Compute the steady state probability that
CONCEPTUAL EXERCISES 443
1. an entering customer sees j customers ahead of him when he joins,
2. a potential arrival joins the system.

9.6 Let pj be the limiting probability that there are j customers in an M/G/1/K
queue, and πj∗ be the limiting probability that an entering customer sees j customers
ahead of him when he enters. Using the theory of MRGPs, establish the relationship
between the pj ’s and the πj∗ ’s.

9.7 Compute the long run behavior of the access control scheme described in Mod-
eling Exercise 9.9 for the special case when M = r.

9.8 Consider the following variation of the M/G/1/1 retrial queue. Suppose that
after a service completion a customer departs with probability 1 − p, or rejoins the
orbit with probability p and behaves like a new customer. Study the limiting distri-
bution of this system using appropriate MRGPs.

9.9 Consider yet another variation of the M/G/1/1 retrial queue. Suppose that an
arriving customer joins service immediately if he finds the server free upon arrival.
Else he departs with probability 1 − p, or joins the orbit with probability p and con-
ducts retrials until getting served. Study the limiting distribution of this system using
appropriate MRGPs.

9.10 Using the MRGP developed in Modeling Exercise 9.8, compute the long run
fraction of the time that the machine is up.

9.11 Consider the MRGP developed in Modeling Exercise 9.11. Let Xn be the num-
ber of customers left behind after the nth server departs. Compute the limiting value
of E(Xn ) as n → ∞. Compute the limiting value of E(X(t)) as t → ∞.

9.10 Conceptual Exercises

9.1 Complete the proof of Theorem 9.1.

9.2 Prove Theorem 9.2.

9.3 Consider the Markov renewal sequence embedded in a CTMC as described in

Example 9.1. Show that the regularity condition in Equation 9.5 is satisfied if an only
if the CTMC is uniformizable.

9.4 Derive Equation 9.11.

9.5 Let {(Xn , Sn ), n ≥ 0} be a given Markov renewal sequence with kernel G.

Show that the process {X(t), t ≥ 0} defined by Equation 9.7 is an SMP with kernel
G.
444 MARKOV REGENERATIVE PROCESSES
9.6 Prove Theorem 9.10 by using the first step analysis to compute the second mo-
ments.

9.7 Using the fact that Mjj is a standard renewal function and Mij (i 6= j) is a
delayed renewal function with common inter-renewal time τjj , prove Theorem 9.11.

9.8 Use the results of Examples 8.16 on page 361 and 8.20 on page 370 to prove
Theorem 9.12.

9.9 Derive the Markov-renewal type equation 9.28 with D(t) as in Equation 9.29.
(Use the derivational steps in Example 8.16 on page 361.)

9.10 Complete the algebraic calculations leading to Equation 9.30.

 CHAPTER 10

Diffusion Processes

Suppose you’re on a game show, and you’re given the choice of three doors. Behind
one door is a car, behind the others, goats. You pick a door, say #1, and the host, who
knows what’s behind the doors, opens another door, say #3, which has a goat. He
says to you, “Do you want to pick door #2?” Is it to your advantage to switch your
choice of doors?

A question asked of Marilyn vos Savant by a reader. Marilyn said yes, while count-
less others said, “It does not matter.” This created a heated controversy, and pro-
duced several papers, with one reader telling Marilyn: “You are the goat!”

10.1 Brownian Motion

In this chapter we study a class of stochastic processes called the diffusion processes.
Intuitively speaking these processes are continuous-time, continuous state-space pro-
cesses and their sample paths are everywhere continuous but nowhere differentiable.
The history of diffusion processes begins with the botanist Brown, who in 1827 ob-
served that grains of pollen suspended in a liquid display a kind of erratic motion.
This motion came to be known as the Brownian motion. Einstein later used physical
principles to do a mathematical analysis of this motion. Wiener later provided rigor-
ous probabilistic foundation for the Brownian motion, and hence some times it is also
called the Wiener process. Diffusion processes are built upon the simpler process of
Brownian motion.

We begin with a formal definition. It uses the concept of stationary and indepen-
dent increments introduced in Definition 5.5 on page 157. We also use the notation R
to denote the real line (−∞, ∞), and N (µ, σ 2 ) to denote a Normal random variable
(or its distribution) with mean µ and variance σ 2 .

Definition 10.1 Brownian Motion (BM). A stochastic process {X(t), t ≥ 0} with

state-space R is called a Brownian motion (BM) with drift parameter µ ∈ R and
variance parameter σ > 0 if

445
446 DIFFUSION PROCESSES
1. for each t ≥ 0, X(t) ∼ N (µt, σ 2 t),
2. it has stationary and independent increments.

We shall denote a BM with parameters µ and σ by BM(µ, σ).

Definition 10.2 Standard Brownian Motion (SBM). A BM(0,1) is called a stan-

dard Brownian motion (SBM).

We reserve the notation {B(t), t ≥ 0} for the standard BM. Some important prop-
erties of a Brownian motion are given in the following theorem.

Theorem 10.1 Basic Properties of a BM. Let {B(t), t ≥ 0} be an SBM, and define
X(t) = µt + σB(t).

1. {X(t), t ≥ 0} is a BM(µ, σ).

2. X(0) = B(0) = 0 with probability 1.
3. {X(t), t ≥ 0} is a Markov process.
4. {−X(t), t ≥ 0} is a BM(−µ, σ).

Proof: Parts 1 and 2 follow from part 1 of the definition of a BM. To see part 3, we
have
P(X(t + s) ≤ x|X(s) = y, X(u) : 0 ≤ u ≤ s)
= P(X(t + s) − X(s) ≤ x − y|X(s) = y, X(u) : 0 ≤ u ≤ s)
= P(X(t + s) − X(s) ≤ x − y) (Independent Increments)
= P(X(t) − X(0) ≤ x − y) (Stationary Increments)
 
x − y − µt
= Φ √
σ t
where Φ(·) is the cdf of a standard normal random variable, and the last equality
follows from part 1 of the definition of a BM. Part 4 follows from the definition of a
BM(µ, σ).

The last property mentioned in the above theorem implies that if {B(t), t ≥ 0} is
an SBM, then so is {−B(t), t ≥ 0}. We say that an SBM is symmetric.

We next compute the joint pdf of an SBM. By definition, B(t) is a N (0, t) random
variable, and has the density φ(t, x) given by
1 − x2
φ(t, x) = √ e 2t , t > 0, x ∈ R. (10.1)
2πt
Since Brownian motion has such a nice structure we can give much more detailed
results about its finite dimensional joint pdfs.
BROWNIAN MOTION 447
Theorem 10.2 Joint Distributions of an SBM. Let {B(t), t ≥ 0} be an SBM,
and 0 < t1 < t2 < · · · < tn . Then [B(t1 ), B(t2 ), · · · , B(tn )] is a multi-variate
N (µ, Σ) random variable with the mean vector µ = [µi ], 1 ≤ i ≤ n, and variance-
covariance matrix Σ = [σij ], 1 ≤ i, j ≤ n, given by
µi = 0, 1 ≤ i ≤ n,
and
σij = min{ti , tj }, 1 ≤ i, j ≤ n.

Proof: Let t0 = 0 and define Yi = B(ti ) − B(ti−1 ), 1 ≤ i ≤ n. Then from the

definition of the SBM we see that Y = [Y1 , Y2 , · · · , Yn ] is a multivariate normal
with mean vector µ(Y ) = 0 and variance-covariance matrix Σ(Y ) = diag(t1 −
t0 , t2 − t1 , · · · , tn − tn−1 ). We see that B = [B(t1 ), B(t2 ), · · · , B(tn )] = AY where
A = [aij ] is given by 
1 if i ≥ j,
ai,j =
0 if i < j.
Hence B is a multivariate normal random variable. Its mean vector and variance-
covariance matrix can be computed by
µ = Aµ(Y )0, Σ = AΣ(Y )AT .
The result follows from this.

Since all the finite dimensional joint pdfs in an SBM are multi-variate normal,
explicit expressions can be computed for many probabilistic quantities. For example,
the joint density f (x, y) of B(s) and B(t) (0 < s < t) is
1 x2 (y−x)2
f (x, y) = φ(s, x)φ(t − s, y − x) = p e− 2s − 2(t−s) . (10.2)
2π s(t − s)
One can show that, given B(t) = x, B(s) is a normal random variable with mean
xs/t and variance s(t−s)/t. A special case of this arises when we consider the SBM
conditioned on B(1) = 0. Such a process is called the Brownian bridge, since it is
anchored at 0 at times 0 and 1. We can see that B(t) (0 ≤ t ≤ 1) in a Brownian
bridge is a normal random variable with mean 0 and variance t(1 − t).
We next state two more basic properties of the BM without proof. First we need
two definitions.

Definition 10.3 Stopping Times. A random variable T defined on the same proba-
bility space as a stochastic process {X(t), t ≥ 0} is called a stopping time for the
{X(t), t ≥ 0} if the event {T ≤ t} is completely determined by {X(u) : 0 ≤ u ≤ t}.

For example,
T = min{t ≥ 0 : X(t) ≥ 2}
is a stopping time, but
T = min{t ≥ 0 : X(t) ≥ X(1)}
448 DIFFUSION PROCESSES
is not. In the latter case we cannot determine if T ≤ 1/2 without observing X(1),
which is not part of {X(u) : 0 ≤ u ≤ 1/2}.

Definition 10.4 Strong Markov Property. A Markov process {X(t), t ≥ 0} is said

to be a strong Markov process (or to have a strong Markov property) if, for any
stopping time T , the finite dimensional distributions of {X(t), t ≥ T }, given {X(u) :
0 ≤ u ≤ T }, depend only on X(T ).

The above definition is mathematically imprecise as it stands, but its meaning is clear:
The future of the process from any stopping time onwards depends on the history up
to that stopping time only via the value of the process at that stopping time. We will
need an enormous mathematical apparatus from measure theory to make it precise,
and we shall not do so here. For the same reason, it is difficult to come up with an
example of a Markov process that is not strong Markov. The DTMCs and the CTMCs
that we have studied so far have strong Markov property. We state the following
theorem without proof.

Theorem 10.3 A BM(µ, σ) is a strong Markov process.

10.2 Sample Path Properties of BM

In this section we shall study several important properties of the sample paths of a
BM. We state the first property in the next theorem.

Theorem 10.4 The sample paths of {X(t), t ≥ 0} are everywhere continuous and
nowhere differentiable with probability 1.

This is one of the deep results about Brownian motion. It is much stronger than as-
serting that the sample paths as a function of t are continuous for almost all values
of t ≥ 0. This property is valid even for the sample paths of a Poisson process since
they have a countable number of jumps with probability one. But none of the sample
paths of a Poisson process is continuous everywhere. What the above theorem as-
serts is that, with probability one, a randomly occurring sample path of a Brownian
motion is a continuous function of t for all t ≥ 0. Even more surprisingly, it asserts
that, with probability one, a randomly occurring sample path of a Brownian motion
is nowhere differentiable. Thus the sample paths of a Brownian motion are indeed
very crazy functions of t.

We will not give formal proof of the above theorem. Instead we shall provide a way
to understand such a bizarre behavior by proposing an explicit method of construct-
ing the sample paths of an SBM as a limit of a sequence of simple symmetric random
walks. Recall from Example 2.10 on page 16 that {Xn , n ≥ 0} is called a simple
symmetric random walk if it is a DTMC on all integers with transition probabilities
pi,i+1 = 1/2, pi,i−1 = 1/2, −∞ < i < ∞.
SAMPLE PATH PROPERTIES OF BM 449
Using the notation [x] to denote the integer part of x, define a sequence of continuous
time stochastic processes {X k (t), t ≥ 0}, indexed by k ≥ 1, as follows:
X[kt]
X k (t) = √ , t ≥ 0. (10.3)
k
Figure 10.1 shows the typical sample paths of the {X k (t), 0 ≤ t < 1} processes
0.4

X 9(t)
0.2

t
0

–0.2

–0.4

X 4(t)
–0.6

–0.8

–1

Figure 10.1 Typical sample paths of the {X k (t), 0 ≤ t < 1} processes for k = 4 and 9.

for k = 4 and k = 9, corresponding to the sample path [X0 , X1 · · · , X9 ] =

[0, −1, −2, −1, 0, 1, 0, −1, −2, 0]. Now define
X ∗ (t) = lim X k (t), t ≥ 0, (10.4)
k→∞

where the limit is in the almost sure sense. (See Appendix H for the relevant defi-
nitions.) Thus a sample path of {Xn , n ≥ 0} produces a unique (in the almost sure
sense) sample path of {X ∗ (t), t ≥ 0}. We have the following result:

Theorem 10.5 Random Walk to Brownian Motion. Suppose |X0 | < ∞ with
probability 1. Then limiting process {X ∗ (t), t ≥ 0} defined by Equation 10.3 exists,
and is an SBM.

Proof: We will not show the existence. Let {Yn , n ≥ 0} be a sequence of iid random
variables with common pmf
P(Yn ) = 1 = .5, P(Yn = −1) = .5, n ≥ 1.
We have
E(Yn ) = 0, Var(Yn ) = 1, n ≥ 1.
Since {Xn , n ≥ 0} is a simple random walk, we see that
n
X
Xn = X0 + Yi . (10.5)
i=1
450 DIFFUSION PROCESSES
Then
P[kt]
X0 + Y
∗
X (t) = lim √ i=1 i
k→∞ k
[kt] X0 + [kt]
p P
Yi
= lim √ p i=1
k→∞ k [kt]
√
= tN (0, 1) (in distribution)
= N (0, t) (in distribution).
This shows part 1 of the Definition 10.2.

Next we show that {X ∗ (t), t ≥ 0} has stationary and independent increments.

From Equation 10.5 we see that {X k (t), t ≥ 0} has stationary and independent
increments as long as the end-points of the intervals over which the increments are
computed are integer multiples of 1/k. Hence in the limit, as k → ∞, {X ∗ (t), t ≥ 0}
has stationary increments as long as the intervals have rational start and end points,
and hence for all intervals. Thus {X ∗ (t), t ≥ 0} satisfies both parts of the Defini-
tion 10.2.

This explicit construction of an SBM is useful in getting an intuitive understanding

k
of the behavior of its sample paths. For example, the sample paths√ of {X (t), t ≥ 0}
are piecewise constant functions of time t with jumps of ±1/ k at all times t that
are integer multiples of 1/k. Hence as k → ∞ the sample paths become continuous
everywhere. Furthermore, for any t the finite derivative of the {X k (t), t ≥ 0} process
is given by
X k (t + 1/k) − X k (t) √
= ± k.
1/k
Thus in the limit the finite derivative has limsup of +∞ and liminf of −∞. Thus in
the limit the sample paths of an SBM are nowhere differentiable.

10.3 Kolmogorov Equations for Standard Brownian Motion

Let p(t, x) be the density of B(t). Since the SBM is a Markov process, we can expect
to derive differential equations for p(t, x), much along the lines of the Chapman-
Kolmogorov Equations we derived in Theorem 6.4 on page 205. However, since
the state-space of the SBM is continuous, we need a different machinery to derive
these equations. The formal derivation of these equations is rather involved, and we
refer the reader to advanced books on the subject, such as Chung (1967). Here we
present an “engineering” derivation, which glosses over some of the technicalities.
The following theorem gives the main differential equation, which is equivalent to
the forward equations derived in Theorem 6.4. We shall use the following notation
KOLMOGOROV EQUATIONS FOR STANDARD BROWNIAN MOTION 451
for the partial derivatives:
∂p(t, x)
pt (t, x) = ,
∂t
∂p(t, x)
px (t, x) = ,
∂x
∂ 2 p(t, x)
pxx (t, x) = .
∂x2

Theorem 10.6 Kolmogorov Equation for SBM. The density p(t, x) satisfies the
following partial differential equation:
1
pt (t, x) = pxx (t, x), x ∈ R, t ≥ 0, (10.6)
2
with the boundary condition
p(0, x) = fB(0) (x).

Proof: We assume that p(t, x) is continuously differentiable in t and twice continu-

ously differentiable in x. Write
B(t) = (B(t) − B(t − h)) + B(t − h).
Note that B(t − h) is independent of B(t) − B(t − h) due to the independence of
the increments. Furthermore, stationarity of increments implies that B(t) − B(t − h)
is identical to B(h) − B(0) = B(h) ∼ N (0, h) in distribution. Conditioning on
B(t − h) we get
Z
p(t, x) = p(t − h, y)fB(h) (x − y)dy, (10.7)
R

where fB(h) (·) is the density of a B(h). Using Taylor series expansion for p(t− h, y)
around (t, x), we get
(y − x)2
p(t − h, y) = p(t, x) − hpt (t, x) + (y − x)px (t, x) + pxx (t, x) + · · · .
2
Substituting in Equation 10.7 we get
Z
p(t, x) = [p(t, x) − hpt (t, x) + (y − x)px (t, x)
R
(y − x)2
+ pxx (t, x) + o((y − x)2 )]fB(h) (x − y)dy
2
E(B(h)2 )
= p(t, x) − hpt (t, x) − E(B(h))px (t, x) + pxx (t, x) + o(h)
2
h
= p(t, x) − hpt (t, x) + pxx (t, x) + o(h).
2
Here we have used the fact that all odd moments of B(h) are zero, and
(2k)! k
E(B(h)2k ) = h ,
2k k!
452 DIFFUSION PROCESSES
to obtain the o(h) term. Dividing by h we get
1 o(h)
pt (t, x) = pxx (t, x) + .
2 h
Letting h → 0 we get equation 10.6.

Equation 10.6 is the forward differential equation. It is also possible to derive

the backward differential equations, but we do not present it here. Under the initial
condition B(0) = 0, the above equation has a unique solution given by
p(t, x) = φ(t, x), t > 0, x ∈ R, (10.8)
which is the density of a N (0, t) random variable. We shall find this method of de-
riving partial differential equations quite useful when dealing with Brownian motion
and related processes. The following theorem gives an extension of the above theo-
rem to a BM(µ, σ).

Theorem 10.7 Kolmogorov Equation for BM. Let {X(t), t ≥ 0} be a BM(µ, σ)

and let p(t, x) be the density of X(t). It satisfies the following partial differential
equation:
σ2
pt (t, x) = −µpx (t, x) + pxx (t, x), x ∈ R, t ≥ 0, (10.9)
2
with the boundary condition
p(0, x) = fX(0) (x).

Proof: See Computational Exercise 10.8.

10.4 First Passage Times

Following our plan of studying any class of stochastic processes, we now study the
first passage times in an SBM. Define
Ta = min{t ≥ 0 : B(t) = a} (10.10)
as the first passage time to the state a. Note that this is a well defined random variable
since the sample paths of the SBM are everywhere continuous with probability 1. It
is clear that Ta is a stopping time, since one can tell if the SBM has visited state a by
time t by looking at the sample path of the SBM over [0, t]. The next theorem gives
the pdf of Ta . It uses a clever argument called ”reflection principle,” which uses the
symmetry of the SBM and its strong Markov property.

Theorem 10.8 Pdf of Ta . Let Ta be as defined in Equation 10.10. If a 6= 0, the pdf

of Ta is given by
|a| − a2
fTa (t) = √ e 2t , t > 0. (10.11)
2πt3
FIRST PASSAGE TIMES 453
If a = 0, Ta = 0 with probability 1.

Proof: Suppose a > 0. We have

P(Ta ≤ t) = P(Ta ≤ t, B(t) ≥ a) + P(Ta ≤ t, B(t) < a).
Since the paths of an SBM are continuous, if a sample path is above a at time t, it
must have crossed a at or before t. Thus {B(t) ≥ a} ⇒ {Ta ≤ t}. Hence
P(Ta ≤ t, B(t) ≥ a) = P(B(t) ≥ a) = P(B(t) > a).
Next, if Ta ≤ t, then due to strong Markov property, the {B(t + Ta ) − B(Ta ), t ≥ 0}
is an SBM and is independent of {B(u) : 0 ≤ u ≤ Ta }. Thus
P(Ta ≤ t, B(t) < a)
= P(Ta ≤ t, B(Ta + t − Ta ) − B(Ta ) < a)
= P(Ta ≤ t, B(Ta + t − Ta ) − B(Ta ) > a) (SBM is symmetric)
= P(Ta ≤ t, B(t) > a) = P(B(t > a).
Combining the above observations we get
Z ∞
P(Ta ≤ t) = 2P(B(t) > a) = 2 φ(t, u)du. (10.12)
a
Taking derivatives on both sides with respect to t (this is a bit tedious), we get the
pdf given in Equation 10.11. The case of a < 0 follows in a symmetric fashion. If
a = 0, the above equation implies
P(T0 ≤ t) = 2P(B(t) > 0) = 1, t ≥ 0.
Hence T0 = 0 with probability 1.

Note that we can do direct computations to show that

Z ∞
E(Ta ) = tfTa (t)dt = ∞, (a 6= 0).
0
Thus the SBM reaches any state a eventually with probability 1, but the expected time
to do so is infinity, except when a = 0. This should remind the reader the definitions
of null recurrence of DTMCs and CTMCs. In some sense, an SBM is a null recurrent
Markov process, although we have not formally defined this concept for continuous
state-space Markov processes. In retrospect, this is not surprising, since an SBM is a
limit of a simple symmetric random walk which is a null recurrent DTMC. From the
above theorem we immediately get the distribution of the maximum and the mini-
mum of an SBM.

Theorem 10.9 The Max and Min of an SBM. The densities of

U (t) = max{B(u) : 0 ≤ u ≤ t}, L(t) = min{B(u) : 0 ≤ u ≤ t},
are given by r
2 − a2
fU(t) (a) = fL(t) (−a) = e 2t , a ≥ 0.
πt
454 DIFFUSION PROCESSES
Proof: We see that, for a ≥ 0,
{U (t) ≥ a} ⇔ {Ta ≤ t}.
From Equation 10.12 we get
Z ∞
P(U (t) ≥ a) = P(Ta ≤ t) = 2 φ(t, u)du.
a
The result follows from taking derivatives with respect to a. The result about the
minimum follows by using the fact that {−B(t), t ≥ 0} is also an SBM.

Extending this analysis to a BM(µ, σ) with µ 6= 0 is a not straightforward, since

we do not have symmetry property when µ 6= 0. We follow a different, and more
general approach to analyze this problem. To be precise, let {X(t), t ≥ 0} be a
BM(µ, σ), and, for a > 0, define
Ta = min{t ≥ 0 : X(t) = a},
and let
ψ(s) = E(e−sTa ),
be its LST. The main result is given in the next theorem. It uses the infinitesimal
version of the first step analysis that we have used before in the CTMCs and the
DTMCs. In this analysis we will need to consider a BM starting from any state x ∈
R. If {X(t), t ≥ 0} is a BM (starting in state 0, by definition), then {x+X(t), t ≥ 0}
is a BM starting in state x.

Theorem 10.10 LST of Ta . Let a > 0. Then

p
ψ(s) = exp(a(µ − µ2 + 2σ 2 s)/σ 2 ). (10.13)

1 if µ ≥ 0,
P(Ta < ∞) = 2 (10.14)
e2µa/σ if µ < 0,

a/µ if µ > 0,
E(Ta ) = (10.15)
∞ if µ ≤ 0.

Proof: We shall do an infinitesimal first step analysis and derive a differential equa-
tion for the LST
ψ(s, x) = E(e−sTa |X(0) = x).
Note that, since a > 0 and X(0) = 0, we can redefine the above first passage time as
Ta = min{t ≥ 0 : X(t) ≥ a}.
We know that X(h)−X(0) ∼ N (µh, σ 2 h). Suppose X(0) = x and X(h)−X(0) =
y. If x + y > a, Ta ≈ h. Else, due to the Markov property of the BM, Ta has the
same distribution as h + Ta starting from state x + y. Using this argument and fh (·)
as the pdf of X(h) − X(0) we get
ψ(s, x) = E(e−sTa |X(0) = x)
FIRST PASSAGE TIMES 455
Z
= E(e−sTa |X(h) − X(0) = y, X(0) = x)fh (y)dy
y∈R
Z
= E(e−s(Ta +h) |X(0) = x + y)fh (y)dy
y∈R
Z
= e−sh ψ(s, x + y)fh (y)dy.
y∈R

Now, for a fixed s, the Taylor expansion of ψ(s, x + y) around x yields

y 2 ′′
ψ(s, x + y) = ψ(s, x) + yψ ′ (s, x) + ψ (s, x) + · · · ,
2
where the superscripts ′ and ′′ denote the first and second derivative with respect to
x. Substituting in the previous equation we get
y2
Z
ψ(s, x) = e−sh [ψ(s, x) + yψ ′ (s, x) + ψ ′′ (s, x) + · · ·]fh (y)dy
y∈R 2
= e−sh [ψ(s, x) + E(X(h) − X(0))ψ ′ (s, x) +
E((X(h) − X(0))2 ) ′′
ψ (s, x) + · · ·]
2
σ 2 h ′′
= e−sh [ψ(s, x) + µhψ ′ (s, x) + ψ (s, x)] + o(h).
2

Using e−sh = 1 − sh + o(h) and collecting the o(h) terms we get

σ 2 h ′′
ψ(s, x) = (1 − sh)ψ(s, x) + µhψ ′ (s, x) + ψ (s, x) + o(h).
2
Dividing by h and letting h → 0, we get
σ 2 ′′
sψ(s, x) − µψ ′ (s, x) − ψ (s, x) = 0. (10.16)
2
This is a second order differential equation with constant coefficients. Hence it has
a solution
ψ(s, x) = Aeθx , −∞ < x < a,
where A is an arbitrary constant. Substituting in Equation 10.16, we get
σ2 θ2
s − µθ − = 0.
2
This has two solutions:
p p
−µ + µ2 + 2σ 2 s −µ − µ2 + 2σ 2 s
θ1 = , θ2 = .
σ2 σ2
Now, one expects that as x → −∞, Ta → ∞, and hence ψ(s, x) → 0. Thus we
should use the solution θ = θ1 . Thus we get
ψ(s, x) = A exp(θ1 x).
456 DIFFUSION PROCESSES
Finally, we have the boundary condition
ψ(s, a) = 1,
since x = a ⇒ Ta = 0. Using this we get
ψ(s, x) = exp(θ1 (x − a)).
The required LST is given by
ψ(s) = ψ(s, 0).
This yields equation 10.13. Equations 10.14 and 10.15 can be derived by using the
properties of the LST.

The case of a < 0 can be handled by simply studying the −a case in a BM(−µ, σ).
Next let a < 0 and b > 0 and define
Tab = min{t ≥ 0 : X(t) ∈ {a, b}}. (10.17)

Thus Tab is the first time the BM(µ, σ) reaches either a or b. Following Theo-
rem 10.10 we give the main result in the following theorem.

Theorem 10.11 Mean of Tab . Let

2µ
θ=− . (10.18)
σ2
Then
exp(θa) − 1
P(X(Tab ) = b) = , (10.19)
exp(θa) − exp(θb)
b(exp(θa) − 1) − a(exp(θb) − 1)
E(Tab ) = . (10.20)
µ(exp(θa) − exp(θb))
In particular, when µ = 0,
|a|
P(X(Tab ) = b) = , (10.21)
|a| + b
E(Tab ) = |a|b/σ 2 . (10.22)

Proof: Let
v(x) = P(X(Tab ) = b|X(0) = x).
Using the notation from the proof of Theorem 10.10, we get, for a < x < b,
Z
v(x) = v(x + y)fh (y)dy
y∈R
y 2 ′′
Z
= [v(x) + yv ′ (x) + v (x) + · · ·]fh (y)dy
y∈R 2
σ 2 h ′′
= v(x) + µhv ′ (x) + v (x) + o(h).
2
FIRST PASSAGE TIMES 457
Dividing by h and letting h → 0, we get
σ 2 ′′
v (x) + µv ′ (x) = 0.
2
The solution to the above equation is
v ′ (x) = Ceθx ,
where
2µ
θ=− .
σ2
This yields
v(x) = Aeθx + B,
where A and B are arbitrary constants. Using the boundary conditions
v(a) = 0, v(b) = 1,
we get the complete solution as
exp(θa) − exp(θx)
v(x) = . (10.23)
exp(θa) − exp(θb)
Substituting x = 0 gives Equation 10.19.

To derive Equation 10.20, let

m(x) = E(Tab |X(0) = x).
Then, for a < x < b, we have
Z
m(x) = [h + m(x + y)]fh (y)dy
y∈R
y 2 ′′
Z
= h+ [m(x) + ym′ (x) + m (x) + · · ·]fh (y)dy
y∈R 2
σ 2 h ′′
= h + m(x) + µhm′ (x) + m (x) + o(h).
2
Simplifying and dividing by h and letting h → 0, we get
σ 2 ′′
m (x) + µm′ (x) = −1.
2
The solution to the above equation is
m′ (x) = Ceθx − 1/µ,
which yields
m(x) = Aeθx + B − x/µ,
where A and B are arbitrary constants. Using the boundary conditions
m(a) = m(b) = 0,
458 DIFFUSION PROCESSES
we get the complete solution as
(a − b) exp(θx) + b exp(θa) − a exp(θb) x
m(x) = − .
µ(exp(θa) − exp(θb)) µ
The required solution is given by m(0) and is given in Equation 10.20. This proves
the theorem. The results for the case µ = 0 are obtained by taking the limits of Equa-
tions 10.19 and 10.20 as µ → 0.

We shall show an alternate method of deriving the results of the above theorem by
using the concept of Martingales in Section 10.7.

10.5 Reflected SBM

We begin with the definition of the reflected standard Brownian motion.

Definition 10.5 Reflected SBM. Let {B(t), t ≥ 0} be an SBM. The process

{Y (t), t ≥ 0} defined by
Y (t) = |B(t)|, t ≥ 0, (10.24)
is called an SBM reflected at the origin, or simply a reflected SBM.

The nomenclature makes intuitive sense because one can obtain a sample path of
{Y (t), t ≥ 0} by reflecting the parts of the sample path of an SBM that lie below
zero around the horizontal axis x = 0 in the (t, x) plane, as shown in Figure 10.2.

Y(t)

Figure 10.2 A typical sample path of an SBM reflected at 0.

We also say that x = 0 is a reflecting boundary. The following theorem lists the
important properties of a reflected SBM.

Theorem 10.12 Properties of Reflected SBM. Let {Y (t), t ≥ 0} be a reflected

Brownian motion. Then

1. The state-space of {Y (t), t ≥ 0} is [0, ∞).

2. {Y (t), t ≥ 0} is a Markov process.
3. The increments of {Y (t), t ≥ 0} are stationary, but not independent.
REFLECTED SBM 459
4. For a fixed t ≥ 0, Y (t) has density given by
fY (t) (y) = 2φ(t, y), t ≥ 0, y ≥ 0.
q
2t 2


5. E(Y (t)) = π , Var(Y (t)) = 1 − π t.

Proof: Part 1 is obvious from the definition. To show part 2, we exploit the following
consequence of the symmetry of the SBM
P(−y ≤ B(t + s) ≤ y|B(t) = x) = P(−y ≤ B(t + s) ≤ y|B(t) = −x).
Now let 0 ≤ t1 < t2 < · · · < tn < tn + s. We have
P(Y (tn + s) ≤ y|Y (ti ) = yi , 1 ≤ i ≤ n)
= P(−y ≤ B(tn + s) ≤ y|B(ti ) = ±yi , 1 ≤ i ≤ n)
= P(−y ≤ B(tn + s) ≤ y|B(ti ) = yi , 1 ≤ i ≤ n) (by symmetry)
= P(−y ≤ B(tn + s) ≤ y|B(tn ) = yn ) (SBM is Markov)
= P(−y − yn ≤ B(tn + s) − B(tn ) ≤ y − yn |B(tn ) = yn )
Z y
= φ(s, x − yn )dx (SBM has ind. inc.).
−y

The last equality follows because B(tn + s) − B(tn ) is independent of B(tn ) and is
Normally distributed with mean 0 and variance s. This proves the Markov property.
It also implies that given Y (t) = x, the density of the increment Y (t + s) − Y (t)
is φ(s, y) + φ(s, −y − 2x) = φ(s, y) + φ(s, y + 2x). This shows the stationarity of
increments. Since the density depends on x, the increments are clearly not indepen-
dent. That proves part 3. Part 4 follows by taking the derivatives with respect y on
both sides of
Z y
P(Y (t) ≤ y) = P(−y ≤ B(t) < y) = φ(u, t)du,
−y

and using the symmetry of the φ density. Part 5 follows by direct integrations.

The symmetry of the SBM can be used to study its reflection across any horizontal
line x = a, not just across the t−axis x = 0. In case a > 0, we consider the process
that is reflected downward, so that the reflected process {Ya (t), t ≥ 0} has state-
space (−∞, a] and is defined by
Ya (t) = a − |a − B(t)|, t ≥ 0.
Similarly, in case a < 0, we consider the process that reflected upward, so that the
reflected process {Ya (t), t ≥ 0} has state-space [a, ∞) and is defined by
Ya (t) = |B(t) − a| + a, t ≥ 0.
For a = 0, we could use either of the above two definitions. The Definition 10.6
corresponds to using the upward reflection. A typical sample path of such a process
is shown in Figure 10.3.
460 DIFFUSION PROCESSES
Ya(t)

Figure 10.3 A typical sample path of a BM reflected downward at a > 0.

Theorem 10.13 Let Sa = [a, ∞) if a ≤ 0, and (−∞, a] if a > 0. The density

p(t, y) of Ya (t) is given by

p(t, y) = φ(t, y) + φ(t, 2a − y), y ∈ Sa . (10.25)

Proof: Consider the case a > 0. Using symmetry, we get for x ≥ 0,

Z a+x
P(a − Ya (t) ≤ x) = P(a − x ≤ B(t) ≤ a + x) = φ(t, u)du.
a−x

Making the change of variables y = a − x, the above equation yields

Z 2a−y
P(Ya (t) ≥ y) = φ(t, u)du, y ∈ Sa .
y

Taking the derivatives with respect to y we can derive the density given in Equa-
tion 10.25. The case a ≤ 0 is similar.

How does the partial differential equation of Theorem 10.6 account for the reflect-
ing boundaries? We state the result in the following theorem.

Theorem 10.14 Kolmogorov Equation for the Reflected SBM. Let p(t, y) be the
density Ya (t). It satisfies the partial differential equation and the boundary condi-
tions of Theorem 10.6, and it satisfies an additional boundary condition:
py (t, a) = 0, t ≥ 0.

Proof: The proof of the partial differential equation is as in Theorem 10.6. The proof
of the boundary condition is technical, and beyond the scope of this book.

Note that the partial derivative in the boundary condition of the above theorem is
to be interpreted as the right derivative if a ≤ 0 and the left derivative if a > 0. The
reader should verify that the density in Equation 10.25 satisfies the partial differential
equation and the boundary condition.
REFLECTED BM AND LIMITING DISTRIBUTIONS 461
10.6 Reflected BM and Limiting Distributions

Analysis of a reflected BM(µ, σ) is much harder when µ 6= 0 due to the lack of

symmetry. In fact a reflected BM cannot be defined as in Equation 10.24, and we
need a new definition given below.

Definition 10.6 Reflected BM. Let {X(t), t ≥ 0} be a BM(µ, σ). The process
{Y (t), t ≥ 0} defined by
Y (t) = X(t) − inf X(u), t ≥ 0, (10.26)
0≤u≤t

is called a BM reflected at the origin, or simply a reflected BM.

Note that |X(t)| is a very different process than the reflected BM defined above.
The two are identical (in distribution) if µ = 0. Since we do not have symmetry,
we derive the marginal distribution of a reflected BM by deriving the Kolmogorov
equation satisfied by its density in the next theorem.

Theorem 10.15 Kolmogorov Equation for a Reflected BM. Let {Y (t), t ≥ 0} be

a BM(µ, σ) reflected at the origin, and let p(t, y) be the density of Y (t). It satisfies
the following partial differential equation:
σ2
pt (t, y) = −µpy (t, y) + pyy (t, y), y > 0, t ≥ 0, (10.27)
2
with boundary conditions
p(0, y) = fY (0) (y),
σ2
µp(t, 0) = py (t, 0), t > 0.
2
Proof: The proof of the partial differential equation is as in Theorem 10.7. The first
boundary condition is as in Theorem 10.7. The proof of the second boundary condi-
tion is technical, and beyond the scope of this book.

One can similarly study a Brownian motion constrained to lie in the interval [a, b]
with X(0) ∈ [a, b]. A typical sample path such a BM is shown in Figure 10.4. Thus
it is reflected up at a and down at b. Giving the functional form of such a BM along
the same lines as Equation 10.26 is not possible. However, we give the Kolmogorov
equation satisfied by its density in the next theorem. The proof is omitted.

Theorem 10.16 Kolmogorov Equation for a BM on [a, b]. Let {Y (t), t ≥ 0} be

a BM(µ, σ) reflected up at a and down at b, with X(0) ∈ [a, b]. Let p(t, y) be the
density of Y (t). It satisfies the following partial differential equation:
σ2
pt (t, y) = −µpy (t, y) + pyy (t, y), t ≥ 0, y ∈ [a, b], (10.28)
2
462 DIFFUSION PROCESSES
X(t)

0 t

Figure 10.4 A typical sample path of a BM constrained to lie in [a, b].

with boundary conditions

σ2
µp(t, a) = py (t, a), t > 0
2
σ2
µp(t, b) = py (t, b), t > 0.
2

Note that py (t, a) is to be interpreted as the right derivative, and py (t, b) as the
left derivative. Although, one can solve the above equation analytically, the result is
complicated, and hence we do not give it here. Below we study the solution when
t → ∞.

Theorem 10.17 Limiting Distribution. Let p(t, y) be as in Theorem 10.16. The

limiting distribution
p(y) = lim p(t, y), y ∈ [a, b],
t→∞
is given by
θ exp(θy)
p(y) = , y ∈ [a, b], (10.29)
exp(θb) − exp(θa)
where
2µ
θ= .
σ2

Proof: We shall assume that the limiting distribution [p(y), a ≤ y ≤ b] exists. Hence
we expect
lim pt (t, y) = 0, y ∈ [a, b].
t→∞
Substituting in Equation 10.28 we get
d2 d
2
p(y) − θ p(y) = 0,
dy dy
with θ = 2µ/σ 2 . The solution to the above equation is given by
p(y) = A exp(θy) + B, y ∈ [a, b].
BM AND MARTINGALES 463
The boundary conditions in Theorem 10.16 reduce to
θp(a) = p′ (a),
θp(b) = p′ (b).
Both these conditions yield only one equation: B = 0. The remaining constant A
can be evaluated by using
Z b
p(y)dy = 1.
a
This yields the solution given in Equation 10.29.

10.7 BM and Martingales

We begin with the definition of Martingale.

Definition 10.7 Martingales. A discrete time real valued stochastic process

{Xn , n ≥ 0} is called a Martingale if
E(Xn+m |Xn , Xn−1 , · · · , X0 ) = Xn , n, m ≥ 0. (10.30)
A continuous time real valued stochastic process {X(t), t ≥ 0} is called a Martin-
gale if
E(X(t + s)|X(u) : 0 ≤ u ≤ s) = X(s), s, t ≥ 0. (10.31)

Example 10.1 Random Walk Martingale. Let {Yn , n ≥ 1} be a sequence of iid

random variables with E(Yn ) = 0. Define
X0 = 0, Xn = Y1 + Y2 + · · · + Yn , n ≥ 1.
Then
Xm
E(Xn+m |Xn , Xn−1 , · · · , X0 ) = Xn + E( Yn+k ) = Xn , n ≥ 0.
k=1

Hence {Xn , n ≥ 0} is a Martingale.

Example 10.2 Linear Martingale. Let {X(t), t ≥ 0} be a BM(µ, σ), and define
Y (t) = X(t) − µt, t ≥ 0.
Show that {Y (t), t ≥ 0} is a Martingale.
Note that conditioning on {Y (u) : 0 ≤ u ≤ s} is equivalent to conditioning on
{X(u) : 0 ≤ u ≤ s}. From the properties of BM we see that X(t + s) − X(s) is
independent of {X(u) : 0 ≤ u ≤ s} and has N (µt, σ 2 t) distribution. Using this we
get
E(Y (t + s)|Y (u) : 0 ≤ u ≤ s)
464 DIFFUSION PROCESSES
= E(X(t + s) − µ(t + s)|X(u) : 0 ≤ u ≤ s)
= E(X(t + s) − X(s) − µt + X(s) − µs|X(u) : 0 ≤ u ≤ s)
= E(X(t + s) − X(s) − µt|X(u) : 0 ≤ u ≤ s)
+E(X(s) − µs|X(u) : 0 ≤ u ≤ s)
= X(s) − µs = Y (s).
Thus {Y (t), t ≥ 0} is a Martingale. As a special case we see that the SBM {B(t), t ≥
0} is a Martingale.

Example 10.3 Quadratic Martingale. Let {B(t), t ≥ 0} be the SBM, and define
Y (t) = B 2 (t) − t, t ≥ 0.
Show that {Y (t), t ≥ 0} is a Martingale.
Using the same arguments as in Example 10.2, we get
E(Y (t + s)|Y (u) : 0 ≤ u ≤ s)
= E(B 2 (t + s) − (t + s)|B 2 (u) − u : 0 ≤ u ≤ s)
= E((B(t + s) − B(s) + B(s))2 − (t + s)|B(u) : 0 ≤ u ≤ s)
= E((B(t + s) − B(s))2 |B(u) : 0 ≤ u ≤ s)
+E(2(B(t + s) − B(s))B(s)|B(u) : 0 ≤ u ≤ s)
+E(B(s)2 |B(u) : 0 ≤ u ≤ s) − (t + s)
= t + B 2 (s) − (t + s)
= B 2 (s) − s = Y (s).
Thus {B 2 (t) − t, t ≥ 0} is a Martingale.

Example 10.4 Exponential Martingale. Let {X(t), t ≥ 0} be a BM(µ, σ). For a

θ ∈ R define
 
1 2 2
Y (t) = exp θX(t) − (θµ + θ σ )t , t ≥ 0.
2
Show that {Y (t), t ≥ 0} is a Martingale.
Using the same reasoning as in Example 10.2, we get
E(Y (t + s)|Y (u) : 0 ≤ u ≤ s)
1
= E(exp(θX(t + s) − (θµ + θ2 σ 2 )(t + s))|X(u) : 0 ≤ u ≤ s)
2
= E(exp(θ(X(t + s) − X(s) + X(s)) −
1
(θµ + θ2 σ 2 )(t + s))|X(u) : 0 ≤ u ≤ s)
2
1
= exp(−(θµ + θ2 σ 2 )(t + s)) ·
2
E(exp(θ(X(t + s) − X(s))) exp(θX(s))|X(u) : 0 ≤ u ≤ s)
1
= exp(θX(s) − (θµ + θ2 σ 2 )(t + s)) ·
2
BM AND MARTINGALES 465
E(exp(θ(X(t + s) − X(s)))|B(u) : 0 ≤ u ≤ s)
1
= exp(θX(s) − (θµ + θ2 σ 2 )(t + s))E(exp(θX(t)))
2
1 2 2
= exp(θX(s) − (θµ + θ σ )s) = Y (s).
2
The penultimate equality follows from the expression for the LST of a N (µt, σ 2 t)
random variable. Thus {Y (t), t ≥ 0} is a Martingale. As a special case we see that
exp(θB(t) − θ2 t/2) is a Martingale.

One of the main results about Martingales is the optional sampling theorem (also
called the stopping theorem), stated below for the continuous time case.

Theorem 10.18 Optional Sampling Theorem. Let {X(t), t ≥ 0} be a Martingale

and T be a stopping time. If P(T < ∞) = 1 and |X(min(T, t))| ≤ K < ∞ for all
t ≥ 0, then
E(X(T )) = E(X(0)).

We do not include the proof of this theorem here. Similar theorem holds for the
discrete time Martingales as well. We illustrate the theorem by using it to derive the
results of Theorem 10.11.

Example 10.5 Let {X(t), t ≥ 0} be a BM(µ, σ). Let a < 0 and b > 0 be given,
and Tab be the first time the BM visits a or b. From Example 10.4 we conclude that
exp(θX(t)) is a Martingale if we choose
2µ
θ=− .
σ2
Since X(t) ∼ N (µt, σ 2 t), we see that X(t) will eventually go below a or above b
with probability 1. Hence P(Tab < ∞) = 1. Also a ≤ X(min(Tab , t)) ≤ b for all
t ≥ 0. Hence we can apply Theorem 10.18. We have
E(exp(θX(Tab ))) = E(exp(θX(0))) = 1.
Now let α = P(X(Tab ) = b) be the probability that the BM visits b before it visits
a. Then
1 = E(exp(θX(Tab ))) = exp(θb)α + exp(θa)(1 − α).
Solving for α we get the result in Equation 10.19.

To derive E(Tab ) we use the linear Martingale X(t) − µt of Example 10.2. Using
Theorem 10.18 we get
E(X(Tab ) − µTab ) = E(X(0) − µ0) = 0.
Hence
E(X(Tab )) bα + a(1 − α)
E(Tab ) = = .
µ µ
Simplifying this we get Equation 10.20.
466 DIFFUSION PROCESSES
10.8 Cost/Reward Models

Let X(t) be the state of a system at time t, and suppose {X(t), t ≥ 0} is a BM(µ, σ).
Suppose the system incurs costs at rate f (x) whenever it is in state x. Using α ≥ 0 as
a continuous discount factor, we see that the expected total discounted cost (ETDC)
over [0, t] starting in state x is given by
Z t 
−αu
c(t, x) = E e f (X(u))du|X(0) = x , x ∈ R.
0

The next theorem gives the partial differential equation satisfied by the function c.

Theorem 10.19 ETDC Over [0, t]. The function c(t, x) satisfies the following par-
tial differential equation:
σ2
ct (t, x) = f (x) − αc(t, x) + µcx (t, x) + cxx (t, x). (10.32)
2
The boundary condition is c(0, x) = 0.

Proof: We shall do an infinitesimal first step analysis and derive a differential equa-
tion for c(t, x). We know that X(h) − X(0) ∼ N (µh, σ 2 h). Using fh (·) as the pdf
of X(h) − X(0) we get
c(t, x)
t
Z  
−αu

= E e f (X(u))du X(0) = x

0
Z Z t  
−αu

= E e f (X(u))du X(h) − X(0) = y, X(0) = x fh (y)dy

y∈R 0
Z  Z t   
e−αu f (X(u))du X(h) = x + y + o(h) fh (y)dy

= f (x)h + E
y∈R h
Z
= [f (x)h + e−αh c(t − h, x + y) + o(h)]fh (y)dy.
x∈R

Now the Taylor expansion of c(t − h, x + y) around (t, x) yields

y2
c(t − h, x + y) = c(t, x) + ycx (t, x) + cxx (t, x) − hct (t, x) + · · · .
2
Substituting in the previous equation we get
c(t, x)
y2
Z
= [f (x)h + e−αh (c(t, x) + ycx (t, x) + cxx (t, x) − hct (t, x))
x∈R 2
+o(h)]fh (y)dy
= f (x)h + e−αh [c(t, x) + E(X(h) − X(0))cx (t, x)
E((X(h) − X(0))2 )
+ cxx (t, x) − hct (t, x)] + o(h)
2
COST/REWARD MODELS 467
2
σ h
= f (x)h + e−αh [c(t, x) + µhcx (t, x) + cxx (t, x) − hct (t, x)] + o(h).
2

Using e−αh = 1 − αh + o(h) and collecting the o(h) terms we get

σ2 h
c(t, x) = f (x)h + (1 − αh)c(t, x) + µhcx (t, x) + cxx (t, x) − hct (t, x) + o(h).
2
Dividing by h and letting h → 0, we get
σ2
f (x) − αc(t, x) + µcx (t, x) + cxx (t, x) − ct (t, x) = 0. (10.33)
2
Rearranging this yields Equation 10.32.

If f is a reasonably simple function, we do not need to solve the partial differential

equation to compute the ETDCs, as shown in the following example.

Example 10.6 Quadratic Cost Model. Suppose f (x) = βx2 , where β > 0 is a
fixed constant. Compute c(t, 0).
We have
Z t  
−αu 2

c(t, 0) = E e βX (u)du X(0) = 0
0
Z t 
e−αu βE(X 2 (u) X(0) = 0)du

=
0
Z t
= e−αu β(σ 2 u + µ2 u2 )du
0
1
= βσ 2 (1 − e−αt (1 + αt))/α2 + 2βµ2 (1 − e−αt (1 + αt + α2 t2 ))/α3 .
2
If we let t → ∞, we get the ETDC over the infinite horizon as
α
c(∞, 0) = 2β(µ2 + σ 2 )/α3 .
2
If we let α → 0 in the expression for c(t, 0) we get the expected total cost over [0, t]
as  
1 2 2 1 2 3
β σ t + µ t .
2 3
This implies that the average cost per unit time c(t, 0)/t goes to infinity as t → ∞.

In many applications the cost rate at time t is given by f (t, X(t)). In this case we
define the ETDC over [0, t] as
Z t  
−αu

c(t, x) = E e f (u, X(u))du X(0) = x , x ∈ R.
0
468 DIFFUSION PROCESSES
In this case we cannot derive a differential equation for c(t, x). In stead, we use the
brute force formula
Z t
c(t, x) = e−αu E(f (u, X(u)) | X(0) = x)du, x ∈ R.
0

Since we know the distribution of X(u), E(f (u, X(u)) is available in closed form if
f is reasonably simple. Hence the integral can be evaluated as a standard Riemann
integral.
In c(t, x) we computed the ETDC over [0, t], where t is a fixed constant. In many
applications we are interested in the ETDC over [0, T ], where T is a random vari-
able, typically a stopping time for the underlying process. Here we consider the case
where T = Tab , where Tab is the first passage time to the set {a, b}, as defined in
Equation 10.17. Let
Z Tab  
e−αu f (X(u))du X(0) = x , a ≤ x ≤ b.

c(x) = E
0

The next theorem describes the differential equation satisfied by c(x) and also the
solution.

Theorem 10.20 ETDC over [0, Tab ]. The function c(x) satisfies the following par-
tial differential equation:
dc(x) σ 2 d2 c(x)
−αc(x) + µ + = −f (x). (10.34)
dx 2 dx2
The boundary condition is c(a) = c(b) = 0.

Proof: As in the proof of Theorem 10.19 we do an infinitesimal first step analysis

and derive a differential equation for c(x). We have
Z Tab  
e−αu f (X(u))du X(0) = x

c(x) = E
0
Z Z Tab  
−αu

= E e f (X(u))du X(h) − X(0) = y, X(0) = x fh (y)dy

y∈R 0
Tab
Z  Z   
−αu

= f (x)h + E e f (X(u))du X(h) = x + y + o(h) fh (y)dy

y∈R h
Z
= [f (x)h + e−αh c(x + y) + o(h)]fh (y)dy.
x∈R

Now, for a fixed h, the Taylor expansion of c(x + y) around x yields

y 2 ′′
c(x + y) = c(x) + yc′ (x) + c (x) + · · · .
2
Substituting in the previous equation we get
y2
Z
c(x) = [f (x)h + e−αh (c(x) + yc′ (x) + c′′ (x) + · · ·)]fh (y)dy
y∈R 2
COST/REWARD MODELS 469
= f (x)h + e−αh [c(x) + E(X(h) − X(0))c′ (x)
E((X(h) − X(0))2 ) ′′
+ c (x) + · · ·]
2
σ 2 h ′′
= f (x)h + e−αh [c(t, x) + µhc′ (x) + c (x)] + o(h).
2

Using e−αh = 1 − αh + o(h) and collecting the o(h) terms we get

σ 2 h ′′
c(x) = f (x)h + (1 − αh)c(x) + µhc′ (x) + c (x) + o(h).
2
Dividing by h and letting h → 0, we get
σ 2 ′′
f (x) − αc(x) + µc′ (x) + c (x) = 0. (10.35)
2
Rearranging this yields Equation 10.34. The boundary conditions follow since Tab =
0 if X(0) = a or X(0) = b.

Recall the following notation from the proof of Theorem 10.11:

θ = −2µ/σ 2 ,
s(x) = exp(θx),
s(a) − s(x)
v(x) = .
s(a) − s(b)
One can verify by direct substitution that, when α = 0, the solution to Equation 10.34
is given by
 Z b Z x 
1
c(x) = − v(x) (s(b − u) − 1)f (u)du + (1 − v(x)) (1 − s(a − u))f (u)du ,
µ x a
(10.36)
for a ≤ x ≤ b. When µ = 0, the above reduces to
2 x−a b b−x x
 Z Z 
c(x) = 2 (b − u)f (u)du + (u − a)f (u)du , a ≤ x ≤ b.
σ b−a x b−a a
(10.37)
We apply the above results to a simple control policy in the example below.

Example 10.7 Control Policy for an SBM. Suppose the state of a system evolves
according to an SBM. When the system is in state x, the system incurs holding cost
at rate cx2 . We have an option of instantaneously changing the state of the system to
zero at any time by paying a fee of K. Once the system state is reset, it evolves as
an SBM until the next intervention. Since the stochastic evolution is symmetric, we
use the following control policy parameterized by a single parameter b > 0: reset the
system state back to state 0 when it reaches state b or −b. Compute the value of b that
minimizes the long run cost per unit time of operating the system.
470 DIFFUSION PROCESSES
Let Y (t) be the state of the system at time t, and let
T = min{t ≥ 0 : Y (t) = b or Y (t) = −b}.
Then Y (T +) = Y (0) = 0, and the system regenerates at time T . Let c(0) be the
expected holding cost over (0, T ] starting from state 0. From Equation 10.37 we get
Z b Z 0 
2
c(0) = c (b − u)u du + (u + b)u du = cb4 /6.
2
(10.38)
0 −b

Thus the total cost over a cycle (0, T ] is K + cb4 /6. The expected length of the
cycle is given by Equation 10.22 as E(T ) = b2 . From the results on renewal reward
processes we get the long run cost per unit time as
K + cb4 /6
.
b2
This is a convex function over b > 0 and is minimized at
 1/4
6K
b∗ = .
c
Note that b∗ increases as c decreases or K increases, as expected.

10.9 Stochastic Integration

In the previous section we saw integrals of the form

Z t
f (u, X(u))du
0

where {X(t), t ≥ 0} is a BM. If f is a continuous function, we see that f (u, X(u))

is a continuous function everywhere on [0, t] with probability 1. Hence, the above
integral can be defined as the standard Riemann integral for almost every sample
path. In this section we define integrals of the form
Z t
f (u, X(u))dX(u). (10.39)
0

If the sample paths of {X(t), t ≥ 0} were differentiable functions, with derivative

X ′ (t), we could define the above integral as
Z t
f (u, X(u))X ′ (u)du.
0
However, we have seen in Section 10.2 that, with probability one, the sample paths of
a BM are nowhere differentiable. Hence we cannot use this definition. If the sample
paths of the BM had bounded variation (i.e., finite total variation), we could use the
Stieltjes integral to define the integral. However, we shall show below that the total
variation of an SBM (and hence of any BM) is infinite. We begin with the definition
of the total variation and the quadratic variation of a standard Brownian motion.
STOCHASTIC INTEGRATION 471
Definition 10.8 Total and Quadratic Variation. For an SBM {B(t), t ≥ 0}, and
p > 0, define the p-th variation process of B by
k
n n−1 p
VBp (t) = lim
X
|B(t ) − B(t )| (10.40)
k→∞
n=1
k k
if the limit exists. The cases p = 1 and p = 2 are referred to as the total variation
and quadratic variation of the SBM, respectively.

Although we have defined the concept of p-variation for an SBM, the same definition
can be used to define the p-variation of any continuous time stochastic process with
piecewise continuous sample paths.

Theorem 10.21 With probability one, an SBM satisfies

VB1 (t) = ∞, t > 0,
VB2 (t) = t, t ≥ 0,
VBp (t) = 0, t ≥ 0, p > 2.

Proof: Let X k (t) be as defined in Equation 10.3. We begin by computing the to-
tal variation of the {X k (t), t ≥ 0} process over [0, t]. Since the sample paths of
{X k (t), t ≥ 0} are piecewise constant functions of time t, we see that the total vari-
ation of X k over [0, t] is the just the sum of the absolute values of all the jumps√over
[0, t]. Since the jumps in the sample paths of {X k (t), t ≥ 0} are of size ±1/ k at
all integer multiples of 1/k, we get
[kt]
X n n−1
VX1 k (t) = |X k (t ) − X k (t )|
n=1
k k
[kt]
X 1
= √
n=1 k
[kt]
= √ .
k
Similarly, the quadratic variation of X k is given by
[kt]
X n n−1 2
VX2 k (t) = |X k (t ) − X k (t )|
n=1
k k
[kt]
X 1
= ( √ )2
n=1 k
[kt]
= .
k
Now, we know that {X k (t), t ≥ 0} converges to {B(t), t ≥ 0} as k → ∞. Hence
total and quadratic variations of {X k (t), t ≥ 0} converges to that of {B(t), t ≥ 0}
472 DIFFUSION PROCESSES
as k → ∞. (This needs to be proved, but we omit this step.) Hence we get
[kt]
VB1 (t) = lim VX1 k (t) = lim √ = ∞,
k→∞ k→∞ k
and
[kt]
VB2 (t) = lim VX2 k (t) = lim= t.
k→∞ k→∞ k
The result about higher-order variation follows similarly. This “proves” the
theorem.

Thus we cannot use Stieltjes integrals to define the integral in Equation 10.39.
What we are facing is a new kind of integral, and it was properly defined first by Ito.
We shall use a severely simplified version of that definition here.

Definition 10.9 Ito Integral. Let f (t, x) be a continuous function in t and x. Sup-
pose Z t
E(f 2 (u, X(u)))du < ∞.
0
Then the Ito integral of f (t, B(t)) with respect to B(t) is defined as
Z t k   
X n−1 n−1 n n−1
f (u, B(u))dB(u) = lim f t , B(t ) B(t ) − B(t ) .
0 k→∞
n=1
k k k k
(10.41)
Here the limit is defined in the mean-squared sense.

Note that the above definition is very similar to the definition of Stieltjes integral,
except that we insist on using the value of the function f at the left ends of the
intervals. Thus, for the n-th interval (t n−1 n n−1 n−1
k , t k ], we use the value f (t k , B(t k ))
at the left end of the interval, and multiply it by the increment in the SBM over the
interval, and then sum these products over all the intervals. This choice is very critical
in the definition of Ito integral and has very important implications.

Theorem 10.22 Linearity of Ito Integral. Ito integral is a linear operator, i.e.,
Z t
(af (u, B(u)) + bg(u, B(u)))dB(u)
0
Z t Z t
=a f (u, B(u))dB(u) + b g(u, B(u))dB(u).
0 0

Proof: Follows directly from the definition.

Note that the Ito integral of Equation 10.41 is a random variable, and hence it
makes sense to compute its moments. The next theorem gives the first two moments.
STOCHASTIC INTEGRATION 473
Theorem 10.23 Moments of the Ito Integral.
Z t 
E f (u, B(u))dB(u) = 0, (10.42)
"Z 0 2 #
t Z t
E f (u, B(u))dB(u) = E(f 2 (u, X(u)))du. (10.43)
0 0

Proof: An SBM has independent increments, hence fn = f (t n−1 n−1

k , B(t k )) is in-
n n−1
dependent of ∆n = B(t k ) − B(t k ). (We suppress the dependence of fn and ∆n
on k to simplify notation.) This independence implies that
E(fn ∆n ) = E(fn )E(∆n ) = 0.
Thus we have
k
!
Z t  X
E f (u, B(u))dB(u) = E lim fn ∆n
0 k→∞
n=1
k
X
= lim E(fn )E(∆n )
k→∞
n=1
= 0.
The interchange of the limit and the expected value is allowed since the convergence
is in mean square. This yields Equation 10.42.

To obtain Equation 10.43, we use


k
!2  k k
!
X X X
E fn ∆n  = E fn ∆n fm ∆m
n=1 n=1 m=1
k X
X k
= E(fn fm ∆n ∆m )
n=1 m=1
k
X
= E(fn2 ∆2n )
n=1
k
X
= E(fn2 )E(∆2n )
n=1
k
X n−1 n−1 1
= E(f 2 (t , B(t ))) .
n=1
k k k

Here we have used the fact that E(fn fm ∆n ∆m ) = 0 if n 6= m. Finally,

"Z
t 2 # k  
X n−1 n−1 1
E f (u, B(u))dB(u) = lim E f 2 (t , B(t ))
0 k→∞
n=1
k k k
474 DIFFUSION PROCESSES
Z t
= E(f 2 (u, X(u)))du.
0

This yields Equation 10.43.

We illustrate the computation of Ito integrals with an example.

Example 10.8 Show that

Z t
dB(u) = B(t), t ≥ 0,
0
t
1 2
Z
B(u)dB(u) = (B (t) − t), t ≥ 0.
0 2
Let Bn = B(t nk ). Using f (u, B(u)) = 1 in Equation 10.41 we get
Z t k
X
dB(u) = lim (Bn − Bn−1 )
0 k→∞
n=1
= lim Bk
k→∞
k
= lim B(t ) = B(t).
k→∞ k

To compute the next integral, we use the identity

(Bn − Bn−1 )2 = Bn2 − Bn−1
2
− 2Bn−1 (Bn − Bn−1 ), 1 ≤ n ≤ k.
Summing over all n from 1 to k we get
k
X k
X k
X
(Bn −Bn−1 )2 = Bk2 −2 Bn−1 (Bn −Bn−1 ) = B 2 (t)−2 Bn−1 (Bn −Bn−1 ).
n=1 n=1 n=1

Now let k → ∞. The sum on the left hand side reduces to the squared variation of
R SBM over [0, t], while the sum on the right hand side reduces to the Ito integral
the
BdB. Hence, using Theorem 10.21 we get
Z t
VB2 (t) = t = B 2 (t) − 2 B(u)dB(u).
0
This gives
t
1 2
Z
B(u)dB(u) =
(B (t) − t).
0 2
Note that there is an unexpected t/2 term in the integral! This is the contribution of
the non-zero quadratic variation that is absent in the standard calculus, but plays an
important part in the Ito calculus!

We have seen the terms B(t) and B 2 (t) − t as examples of Martingales, see Ex-
amples 10.2 and 10.3. Is this just a coincidence that the two Ito integrals turned out
STOCHASTIC DIFFERENTIAL EQUATIONS 475
to be Martingales? The next theorem shows that this is a general property of the Ito
integrals. We omit the proof, since it is too technical for this book.

Theorem 10.24 Ito Integral as a Martingale. Let

Z t
X(t) = f (u, B(u))dB(u), t ≥ 0.
0

Then {X(t), t ≥ 0} is a Martingale.

One can extend the class of functions f for which the Ito integral can be defined
to the functions of the type f (t, X[0 : t]) where X[0 : t] is short for {X(u) : 0 ≤
u ≤ t}. However, we shall not use this generality in this book. We refer the reader to
many excellent texts on this subject for further information.
We develop three peculiar integrals below that will help us in the development of
stochastic differential calculus in the next section. Following Equation 10.41 we have
Z t k
X n n−1 n n−1
dB(u)dB(u) = lim (B(t ) − B(t ))(B(t ) − B(t ))
0 k→∞
n=1
k k k k
= VB2 (t) = t.

t k
1 n n−1
Z X
dudB(u) = lim (B(t ) − B(t ))
0 k→∞
n=1
k k k
B(t)
= lim = 0.
k→∞ k

t k
1
Z X
dudu = lim ( )2
0 k→∞
n=1
k
k
= lim = 0.
k→∞ k2

In differential notation, we write the above “integrals” as

dB(t)dB(t) = dt, (10.44)
dtdB(t) = dB(t)dt = dtdt = 0. (10.45)
The above differentials form the basis of the Ito stochastic differential calculus de-
veloped in the next section.

10.10 Stochastic Differential Equations

Theorem 10.24 shows that we can think of the Ito integral as a continuous time
stochastic process. In fact it motivates us to define more general stochastic processes
as defined below.
476 DIFFUSION PROCESSES
Definition 10.10 Diffusion Process. Let µ(t, x) and σ(t, x) be continuous func-
tions of t and x. Suppose
Z t
E(σ 2 (u, B(u)))du < ∞.
0
Define
Z t Z t
X(t) = X(0) + µ(u, B(u))du + σ(u, B(u))dB(u), t ≥ 0. (10.46)
0 0

Then {X(t), t ≥ 0} is called a diffusion process, µ is called its drift function, and σ
the diffusion function.

Note that the first integral is a simple Riemann integral, while the second inte-
gral is an Ito integral. Diffusion processes are a special case of Ito processes, which
are defined as in Equation 10.46 with more general functions µ(t, X[0 : t]) and
σ(t, X[0 : t]). We shall restrict ourselves to diffusion processes for simplicity. It is
customary to write the integral equation in Equation 10.46 in a notationally equiva-
lent stochastic differential equation form as follows
dX(t) = µ(t, X(t))dt + σ(t, X(t))dB(t). (10.47)
One can even interpret the above stochastic differential equation as follows: if
X(t) = x, the increment X(t+dt)−X(t) in the X process over the interval (t, t+dt)
is a sum of two components: (1) a deterministic drift component given by µ(t, x)dt,
and (2) a random diffusion component σ(t, x)dB(t) that is a N (0, σ 2 (t, x)dt) ran-
dom variable.

Example 10.9 Suppose X(0) = 0, µ(t, x) = 0, and σ(t, x) = 1. Substituting in

Equation 10.46 we see that
X(t) = B(t).
Thus the SBM is a diffusion process.
Next suppose X(0) = x, µ(t, x) = µ, and σ(t, x) = σ. Substituting in Equa-
tion 10.46 we see that
X(t) = x + µt + σB(t).
From Theorem 10.1 we see that {X(t), t ≥ 0} is a BM(µ, σ) with initial state x.
Thus a BM is a diffusion process.

Next we consider a process {Z(t), t ≥ 0} defined by

Z(t) = g(t, X(t)),
where X(t) is as defined in Equation 10.46. The next theorem shows that {Z(t), t ≥
0} is a diffusion process itself, and relates its drift and diffusion functions to the drift
and diffusion functions of the X process.

Theorem 10.25 Functions of a Diffusion Process. Let {X(t), t ≥ 0} satisfy Equa-

tion 10.46. Let g(t, x) be a function that is continuously differentiable in t and twice
STOCHASTIC DIFFERENTIAL EQUATIONS 477
continuously differentiable in x. The stochastic process {g(t, X(t)), t ≥ 0} satisfies
the following stochastic integral equation
g(t, X(t)) = g(0, X(0))
Z t 
1 2
+ gt (u, X(u)) + µ(u, X(u))gx (u, X(u)) + σ (u, X(u))gxx(u, X(u)) dt
0 2
Z t
+ σ(u, X(u))gx (u, X(u))dB(u) (10.48)
0
or, equivalently, the stochastic differential equation
dg(t, X(t)) =
 
1
gt (t, X(t)) + µ(t, X(t))gx (t, X(t)) + σ 2 (t, X(t))gxx (t, X(t)) dt
2
+σ(t, x)gx (t, X(t))dB(t). (10.49)

Proof: Let ∆h X(t) = X(t+h)−X(t). Taylor expansion of g(t+h, X(t)+∆h X(t))

around (t, X(t)) yields :
g(t + h, X(t) + ∆h X(t)) = g(t, X(t)) + hgt (t, X(t)) + ∆h X(t)gx (t, X(t))
1
+ (∆h X(t))2 gxx (t, X(t)) + Rh
2
where Rh is the remainder error term that goes to zero as h → 0. Hence we have
∆h g(t, X(t)) = hgt (t, X(t))
1
+∆h X(t)gx (t, X(t)) + (∆h X(t))2 gxx (t, X(t)) + Rh ,
2
or in differential form,
1
dg(t, X(t)) = gt (t, X(t))dt + gx (t, X(t))dX(t) + gxx (t, X(t))dX(t)dX(t).
2
Now {X(t), t ≥ 0} satisfies Equation 10.47. Substituting in the previous equation
we get
dg(t, X(t)) = gt (t, X(t))dt
1
+(µ(t, X(t))dt + σ(t, X(t))dB(t))gx (t, X(t)) + (µ(t, X(t))dt
2
+σ(t, X(t))dB(t))2 gxx (t, X(t))
= (gt (t, X(t)) + µ(t, X(t))gx (t, X(t)))dt
1
+ σ 2 (t, X(t))gxx (t, X(t))(dB(t))2 + σ(t, X(t))gx (t, X(t))dB(t)
2
1
+ (µ2 (t, X(t))dtdt + 2µ(t, X(t))σ(t, X(t))dtdB(t))gxx (t, X(t)).
2
Now we use Equations 10.44 and 10.45 to get
dg(t, X(t)) =
478 DIFFUSION PROCESSES
 
1 2
gt (t, X(t)) + µ(t, X(t))gx (t, X(t)) + σ (t, X(t))gxx (t, X(t)) dt
2
+σ(t, X(t))gx (t, X(t))dB(t),
which is Equation 10.49. The integral representation of this equation is Equa-
tion 10.48.

Equations 10.48 and 10.49 are known as Ito’s formula. The formula remains valid
if we replace the µ(t, X(t)) and σ(t, X(t)) functions by the path dependent functions
µ(t, X[0 : t]) and σ(t, X[(0 : t]). However, we do not prove it in that generality here.
If X(t) = B(t), i.e., if µ(t, x) = 0 and σ(t, x) = 1, Ito’s formula reduces to
Z t 
1
g(t, B(t)) = g(0, 0) + gt (u, B(u)) + gxx (u, B(u)) du
0 2
Z t
+ gx (u, B(u))dB(u). (10.50)
0
We can write the above in an equivalent differential form as follows:
 
1
dg(t, B(t)) = gt (t, B(t)) + gxx (t, B(t)) dt + gx (t, B(t))dB(t). (10.51)
2
The above formula can be used to compute stochastic integrals defined by Equa-
tion 10.41. Let Z x
G(t, x) = g(t, y)dy.
0
Then using g(t, x) in place of gx (t, x), Equation 10.50 can be written in a more
useful form as
Z t Z t 
1
g(u, B(u))dB(u) = G(t, B(t)) − Gt (u, B(u)) + gx (u, B(u)) du.
0 0 2
(10.52)
We illustrate the use of the above formula by several examples below.
Rt
Example 10.10 Compute 0 B(u)dB(u) using Equation 10.52.

Let g(t, x) = x. Then G(t, x) = x2 /2, and gx (t, x) = 1, and Gt (t, x) = 0.

Hence, Equation 10.52 yields
Z t Z t
B 2 (t) 1 1
B(u)dB(u) = − du = (B 2 (t) − t).
0 2 0 2 2
This matches with our brute force calculation of this integral in Example 10.8.

Example 10.11 Geometric Brownian Motion. Solve the stochastic differential

equation
dX(t) = µX(t)dt + σX(t)dB(t).
STOCHASTIC DIFFERENTIAL EQUATIONS 479
First of all, what does it mean to solve a stochastic differential equation? We take
it to mean that we need to find a function g(t, ·) such that X(t) = g(t, B[0 : t])
satisfies the above differential equation. In this case we first try to see if can find a
solution of a simpler type X(t) = g(t, B(t)). From Equation 10.51, we see that the
g function has to satisfy:
1
gt (t, x) + gxx (t, x) = µg(t, x), (10.53)
2
and
gx (t, x) = σg(t, x). (10.54)
The last equation implies that
g(t, x) = f (t) exp(σx),
where f (t) is a function of t to be determined. Substituting in Equation 10.53 we get
1
f ′ (t) exp(σx) + f (t)σ 2 exp(σx) = µf (t) exp(σx).
2
Canceling exp(σx) from both sides, we get
1
f ′ (t) = (µ − σ 2 )f (t),
2
which yields
1
f (t) = c exp((µ − σ 2 )t),
2
where c is an arbitrary constant. Putting these equations together, we get
1
g(t, x) = c exp((µ − σ 2 )t + σx).
2
Thus the solution is
1
X(t) = c exp((µ − σ 2 )t + σB(t)).
2
Substituting t = 0, and using B(0) = 0, we get c = X(0). Thus the final solution is
 
1 2
X(t) = X(0) exp (µ − σ )t + σB(t) . (10.55)
2
{X(t), t ≥ 0} is called the geometric Brownian motion and is a common model of
stock returns used in financial literature.

Example 10.12 Solve the stochastic differential equation

dZ(t) = −µZ(t)dt + σdB(t). (10.56)

As in the previous example, we attempt to find a solution of the type Z(t) =

g(t, B(t)). Thus we try to find a g satisfying
1
gt (t, x) + gxx (t, x) = −µg(t, x), (10.57)
2
480 DIFFUSION PROCESSES
and
gx (t, x) = σ. (10.58)
The last equation yields
g(t, x) = f (t) + σx,
for an arbitrary function f (t). Substituting in Equation 10.57 we get
f ′ (t) = −µ(f (t) + σx)
which has a solution
f (t) = A exp(−µt) − σx,
where A is a constant. However, this implies
g(t, x) = A exp(−µt),
which does not satisfy Equation 10.58. Thus there is no function g(t, x) that satisfies
both Equations 10.57 and 10.58. Thus there is no stochastic process of the form
Z(t) = g(t, B(t)) that solves Equation 10.56.
Instead we assume that {X(t), t ≥ 0} satisfies Equation 10.47, where we have
complete freedom to choose the functions µ(t, x) and σ(t, x), and look for a solution
of the form Z(t) = g(t, X(t)). Using Equation 10.49 we see that g(t, x) satisfies
1
gt (t, x) + µ(t, x)gx (t, x) + σ 2 (t, x)gxx (t, x) = −µg(t, x), (10.59)
2
and
σ(t, x)gx (t, x) = σ. (10.60)
The simplest choice is to use σ(t, x) = σ exp(µt), and µ(t, x) = 0, i.e., choose
dX(t) = σ exp(µt)dB(t),
or Z t
X(t) = X(0) + eµu dB(u). (10.61)
0
Then Equations 10.59 and 10.60 reduce to
1
gt (t, x) + σ 2 exp(2µt)gxx (t, x) = −µg(t, x),
2
and
σ exp(µt)gx (t, x) = σ.
By following the earlier procedure, we see that these two equations admit the solution
g(t, x) = exp(−µt)(c + x),
where c is an arbitrary constant. Thus the solution to Equation 10.56 is given by
Z(t) = exp(−µt)(c + X(t)).
Setting t = 0 we get c = Z(0). Using Equation 10.61, we get the final solution as
Z t
Z(t) = exp(−µt)(Z(0) + exp(µu)dB(u)).
0
APPLICATIONS TO FINANCE 481
The process defined above is known as the Ornstein-Uhlenbeck process. This ex-
ample illustrates the complexity of the solutions exhibited by even simple stochastic
differential equations.

10.11 Applications to Finance

Let X(t) be the price of a stock at time t. One of the simplest financial derivative
is called the European call option. (American call options are discussed at the end
of this section.) A broker sells this option at a price C to a buyer. It gives the buyer
(called the owner) the right (but not the obligation) to buy one share of this stock at a
pre-specified time T in the future (called the maturity time, or expiry date), at a pre-
specified price K (called the strike price). Clearly, if the stock price X(T ) at time
T is greater than K, it makes sense for the owner to exercise the option, since the
owner can buy the stock at price K and immediately sell it at price X(T ) and realize
a net profit of X(T ) − K. On the other hand, if X(T ) ≤ K, it makes sense to let the
option lapse. Thus, the payout to the owner of this contract is max(X(T ) − K, 0)
at time T . How much should the broker sell this option for, i.e., what should be the
value of C? This is the famous option pricing problem.

As mentioned in Example 10.11, the geometric Brownian motion as defined in

Equation 10.55 is commonly used as a model of stock market price evolution. We
can rewrite Equation 10.55 to see that the stock price X(t) under this model can be
represented as
1 √
X(t) = exp(lnX(0) + (µ − σ 2 )t + σ tZ), (10.62)
2
where Z ∼ N (0, 1). Let 0 ≤ t1 < t2 . The ratio X(t2 )/X(t1 ) is called the return
over the period [t1 , t2 ]. It can be seen that the geometric Brownian motion model of
the stock price implies that the returns over non-overlapping intervals are indepen-
dent. This is one of the most compelling reason for the wide use of this model. The
other reason is the analytical tractability of this process.

Before we can settle the question of evaluating the proper value of C, we need
to know what else we can do with the money. We assume that we can either invest
it in the stock itself, or put it in a risk-free savings account that yields a continuous
fixed rate of return r. Thus one dollar invested in the stock at time 0 will be worth
$X(t)/X(0) at time t, while one dollar invested in the savings account will be worth
ert at time t.

One would naively argue that the value of C should be given by the expected
discounted (discount factor r) value of the option payout at time T , i.e.,
C = e−rT E(max(X(T ) − K, 0)). (10.63)
One can show (see any book on mathematical finance) that if µ 6= r, the broker can
482 DIFFUSION PROCESSES
make positive profit with probability one by judiciously investing the above proceeds
of $C in the stock and the savings account. Such a possibility is called an arbitrage
opportunity, and it cannot exist in a perfect market. Hence we need to evaluate C
assuming that µ = r. The next theorem gives the expression for C when µ = r. This
is the celebrated Black-Scholes’ formula.

Theorem 10.26 Black-Scholes’ Formula: European Call Option. The value of

the European call option with maturity date T and strike price K is given by:
C = X(0)Φ(d1 ) − Ke−rT Φ(d2 ), (10.64)
where
ln(X(0)/K) + (r + σ 2 /2)T
d1 = √ , (10.65)
σ T
ln(X(0)/K) + (r − σ 2 /2)T
d2 = √ , (10.66)
σ T
and Φ is the cdf of a N(0,1) random variable.

Proof: Define 
+ X(T ) if X(T ) > K,
X(T ) =
0 if X(T ) ≤ K.
and 
+ K if X(T ) > K,
K =
0 if X(T ) ≤ K.
Then
max(X(T ) − K, 0) = X(T )+ − K + .
The value of the option as given in Equation 10.63 reduces to
C = e−rT [E(X(T )+ ) − E(K + )], (10.67)
where we use µ = r to compute the expectations. Next we compute the two expec-
tations above. Using Equation 10.62 we get
E(X(T )+ ) =
1 √
= E(exp((lnX(0) + (r − σ 2 )T + σ T Z)+ )
2
√
 
1
= E exp(lnX(0) + (r − σ 2 )T + σ T Z) · 1{lnX(0)+(r− 1 σ2 )T +σ√T Z>lnK}
2 2

1 2 √
= exp(lnX(0) + (r − σ )T )E(exp(σ T Z)1{Z>c1 } ),
2
where
ln(K/X(0)) − (r − 21 σ 2 )T
c1 = √ .
σ T
Now we can show by direct integration that
2
E(exp(aZ)1{Z>b} ) = ea /2
Φ(a − b), (10.68)
APPLICATIONS TO FINANCE 483
for positive a. Substituting in the previous equation, we get
1 1
E(X(T )+ ) = X(0) exp((r − σ 2 )T ) exp( σ 2 T )Φ(d1 ) = X(0)erT Φ(d1 ),
2 2
√
where d1 = σ T − c1 is as given in Equation 10.65. Next, we have
E(K + ) = E(K1X(T )>K ) = KP(X(T ) > K)
1 √
= KP(lnX(0) + (r − σ 2 )T + σ T Z > lnK)
2
= KP(Z > c1 ) = KP(Z ≤ d2 ),
where d2 = −c1 is as given in Equation 10.66. Substituting in Equation 10.67 we
get Equation 10.64.

One can also study the European put option in a similar way. It gives the buyer the
right (but not the obligation) to sell one share of the stock at time T at price K. The
payout of this option at time T is max(K − X(T ), 0). We leave it to the reader to
prove the following theorem.

Theorem 10.27 Black-Scholes’ Formula: European Put Option. The value of

the European put option with maturity date T and strike price K is given by:
C = Ke−rT Φ(−d2 ) − X(0)Φ(−d1 ), (10.69)
where d1 , d2 , and Φ are as in Theorem 10.27.

There are options called the American and call and put options with the maturity
time T and strike price K. These are the same as the European options, except that
they may be exercised at any time t ∈ [0, T ]. If an American call option is exercised at
time t, then its payout is max(X(t) − K, 0). Similarly the payout from an American
put option exercised at time t is max(K − X(t), 0). One can show that under the
assumption that the stock price satisfies Equation 10.62, it is optimal to exercise an
American call option at maturity. Thus its value is same as that of the European
option. Pricing the American put option is much harder for the finite maturity date.
However, if the maturity date is infinity (this case is called the perpetual American
put option) one can evaluate it analytically. We state the main result in the following
theorem.

Theorem 10.28 American Perpetual Put Option. Consider a perpetual American

put option with strike price K, and assume that the stock price process is as given
in Equation 10.62. It is optimal to exercise the option as soon as the stock price falls
below
2r
a∗ = K.
2r + σ 2
The optimal expected payout under this policy, starting with X(0) = x > a∗ is
2r
(K − a∗ )(a∗ /x) σ2 .
484 DIFFUSION PROCESSES
Proof: See Conceptual Exercises 10.11 and 10.13.

Note that the result of the above theorem does not depend upon µ. As we have seen
before, this is because we need to do the calculations under the assumption µ = r in
order to avoid arbitrage.

10.12 Computational Exercises

10.1 Let {B(t), t ≥ 0} be an SBM. Show that Cov(B(s), B(t)) = min(s, t).

10.2 Let {X(t), t ≥ 0} be a BM(µ, σ). Compute Cov(X(t), X(t + s)).

10.3 Let {X(t), t ≥ 0} be a BM(µ, σ). Compute the joint density of [X(t1 ), X(t2 )],
where 0 < t1 < t2 .

10.4 Let {B(t), t ≥ 0} be an SBM. Let 0 < s < t. Show that, given B(t) = y,
B(s) is a normal random variable with mean ys/t and variance s(t − s)/t.

10.5 Let {X(t), t ≥ 0} be a BM(µ, σ). Let 0 < s < t. Compute the conditional
density of B(s) given B(t) = y.

10.6 Let t ∈ (0, 1). Show that B(t) − tB(1) has the same distribution as the condi-
tional density of B(t) given B(1) = 0.

10.7 Verify that the solution given in Equation 10.8 satisfies the partial differential
equation given in Equation 10.6.

10.8 Prove Theorem 10.7. Show that the pdf of a N (µt, σ 2 t) random variable sat-
isfies this equation.

10.9 Derive Equation 10.11 from Equation 10.12.

10.10 Let {X(t), t ≥ 0} be a BM(µ, σ). If we stop the process at time t, we earn a
discounted reward equal to e−αt X(t). Suppose we use the following policy: stop the
process as soon as it reaches a state a. Find the optimal value of a that maximizes the
expected discounted reward of this policy.

10.11 Let {X(t), t ≥ 0} be BM(µ, σ) reflected at 0. Assume X(0) ≥ 0, and µ < 0.

Show that the limiting distribution of X(t), as t → ∞, is exp(−2µ/σ 2). Hint: Set
a = 0 and let b → ∞ in Theorem 10.17.

10.12 Let {X(t), t ≥ 0} be a BM(µ, σ). Find functions a(t) and b(t) such that
X 2 (t) + a(t)X(t) + b(t) is a Martingale. Hint: Use Example 10.3.
COMPUTATIONAL EXERCISES 485
10.13 Let {X(t), t ≥ 0} be a BM(µ, σ). Find the functions a(t), b(t), and c(t) such
that X 3 (t) + a(t)X 2 (t) + b(t)X(t) + c(t) is a Martingale. Hint: Use Conceptual
Exercise 10.3.
2
10.14 Let m2 (x) = E(Tab |X(0) = x) where Tab is the first passage time as defined
in Equation 10.17 in a BM(µ, σ). Show that m2 (·) satisfies the following differential
equation

σ 2 ′′
m (x) + µm′2 (x) = −2m(x), a < x < b
2 2
where m(x) = E(Tab |X(0) = x). The boundary conditions are m2 (a) = m2 (b) =
0.
10.15 Let {X(t), t ≥ 0} be a BM(µ, σ) with X(0) = 0 and µ < 0, and define
M = maxt≥0 X(t). Show that M is an exp(−2µ/σ 2) random variable. Hint: Argue
that
P(M < y) = lim P(X(Tay ) = a),
a→−∞
and use Theorem 10.11.

10.16 Let {X(t), t ≥ 0} be a BM(µ, σ 2 ) with X(0) = 0 and µ > 0, and define
L = mint≥0 X(t). Use the hint in Computational Exercise 10.15 to compute the
distribution of L.

10.17 Use the exponential Martingale of Example 10.4 to derive Equation 10.13.
Hint: Set
θµ + θ2 σ 2 /2 = s
and use Theorem 10.18.

10.18 Let X(t) be the price of a stock at time t. Suppose {X(t), t ≥ 0} is the
geometric Brownian motion defined by Equation 10.55. Thus a dollar invested in this
stock at time u will be worth X(t)/X(u) at time t > u. Consider a static investment
strategy by which we invest in this stock fresh money at a rate $d at all times t ≥ 0.
Let Y (t) be total value of the investment at time t, assuming that Y (0) = 0. Compute
E(Y (t)).

10.19 Consider the optimal policy derived in Example 10.7. Using the regenerative
nature of {Y (t), t ≥ 0} defined there, compute the limiting mean and variance of
Y (t) as t → ∞.

10.20 Let Y (t) be the level of inventory at time t. Assume that Y (0) = q > 0.
When the inventory reaches 0, an order of size q is placed from an outside source.
Assume that the order arrives instantaneously, so that the inventory level jumps to
q. Between two orders the Y process behaves like an BM(µ, σ) starting in state q,
with µ < 0. Suppose it costs h dollars to hold one unit of the inventory for one
unit of time, and the restoration operation costs K. Find the optimal value of q that
minimizes the long run expected cost per unit time.
486 DIFFUSION PROCESSES
10.21 Using Definition 10.9 show that
Z t Z t
2 1 3
B (u)dB(u) = B (t) − B(u)du.
0 3 0

10.22 Using Equation 10.52 show that

Z t
1 n t n−1
Z
n n+1
B (u)dB(u) = B (u) − B (u)du.
0 n+1 2 0

10.23 Using Equation 10.52 show that

Z t Z t
−αu n 1 −αt n+1 n n−1 
e B (u)dB(u) = e B (t)− B (u) − αe−αu B n (u) du.
0 n+1 0 2

10.24 Solve the stochastic differential equation

dY (t) = −αY (t)dt + e−αt dB(t),
assuming a solution of the form Y (t) = g(t, B(t)).

10.25 Solve the stochastic differential equation

Y (t) 1
dY (t) = − dt + dB(t),
1+t 1+t
assuming a solution of the form Y (t) = g(t, B(t)).

10.13 Conceptual Exercises

10.1 Let T be a stopping time for the stochastic process {X(t), t ≥ 0}. Prove or
disprove the following statements:

1. The event {T > t} is completely described by {X(u) : 0 ≤ u ≤ t}.

2. The event {T > t} is completely described by {X(u) : u > t}.
3. The event {T < t} is completely described by {X(u) : 0 ≤ u < t}.

10.2 Let {X(t), t ≥ 0} be a BM(µ, σ). Are the following random variables stopping
times for the BM?
Z t
T1 = min{t ≥ 0 : X(u)du ≥ 1},
0

T2 = min{t ≥ 0 : X(t) > 1}.

10.3 Show that B 3 (t) − 3tB(t) is a Martingale.

10.4 Show that B 4 (t) − 6tB 2 (t) + 3t2 is a Martingale.

CONCEPTUAL EXERCISES 487
Rt
10.5 Let L = 0 B(u)du. Show that
E(L) = 0, E(L2 ) = t3 /3.

10.6 Let L be as in Conceptual Exercise 10.5. Show that

E(exp(θL)) = exp(θ2 t3 /6).
Rt
10.7 Let L = 0 exp(θB(u))du. Show that
2
E(L) = (exp(θ2 t/2) − 1).
θ2
10.8 Let L be as in Conceptual Exercise 10.7. Show that
2 8 2
E(L2 ) = exp(2θ2 t) − 4 exp(θ2 t/2)) − 4 .
3θ4 3θ θ
10.9 Using Equation 10.52 show that, for a differentiable function f ,
Z t Z t
f (u)dB(u) = f (t)B(t) − B(u)f ′ (u)du.
0 0

10.10 Solve the stochastic differential equation:

dY (t) = a(t)Y (t)dt + b(t)Y (t)dB(t), Y (0) = 1,
where a(·) and b(·) are differentiable functions of t. Hint: Try a solution of the type
Y (t) = g(t, X(t)), where X(t) satisfies dX(t) = b(t)dB(t).

10.11 Proof of Theorem 10.28. Consider the perpetual American put option with
strike price K for the stock price process given in Theorem 10.28. Suppose X(0) >
K, for otherwise the buyer would exercise the option right away with a payout of
K − X(0). Let Π(a) be a policy that exercises the option as soon as the stock price
falls below a given constant 0 < a < K. Thus the payout under policy Π(a) is
C(a, x) = E(e−rTa (K − a)|X(0) = x), x>K
where
Ta = min{t ≥ 0 : X(t) = a}
and the expected value is computed under the assumption that µ = r. Show that
2r
C(a, x) = (K − a)(a/x) σ2 , x > a.
Hint: First show that Ta is the same as the first time a BM(r − σ 2 /2, σ), starting from
ln(X(0)), reaches ln(a). Then use Theorem 10.10 to compute the LST of Ta , and
then use Computational Exercise 10.10 to compute the expectations.

10.12 Prove Theorem 10.27.

10.13 Proof of Theorem 10.28, continued. Starting with the result in the Conceptual
Exercise 10.11 show that the value of a that maximizes C(a, x) is given by the a∗ of
Theorem 10.28. Also, C(a∗ , x) is as given in the theorem.
 Epilogue

Here ends our journey. Congratulations! Now is the time to look back to see what we
have learned and to look ahead to see what uncharted territory lies ahead.

We started with a basic knowledge of probability and built classes of increasingly

powerful stochastic processes, each class providing a stepping stone to the next. We
started with Markov chains and ended with Markov regenerative processes. The the-
ories of Markov chains and renewal processes emerged as two main corner stones
of the entire development. At the end we dipped our toes in diffusion processes, and
saw what is involved once we step away from the constraints of discrete state-space
and discrete time. However, we stayed within the confines of Markov processes.

We saw a large number of examples of modeling a given system by an appropriate

class of processes and then performing the transient analysis, first passage time anal-
ysis, steady-state analysis, cost/reward analysis, etc. Indeed we now have a rich bag
of tools to tackle the uncertain world.

What lies ahead? There is far more to diffusion processes than what we have cov-
ered here. Also, we have not covered the class of stationary processes: these pro-
cesses look the same from any point of time. We saw them when studying the Markov
chains starting in their stationary distributions. Stationary processes play an impor-
tant role in statistics and forecasting.

We have also ignored the topic of controlling a stochastic system. The models
studied here are descriptive – they describe the behavior of a system. They do not
tell us how to control it. Of course, a given control scheme can be analyzed using
the descriptive models. However, these models will not show how to find the optimal
control scheme. This direction of inquiry will lead us to Markov decision processes.

Each of these topics merits a whole new book by itself. We stop here by wishing
our readers well in the future as they explore these new and exciting areas.

489
 APPENDIX A

Probability of Events

“All business proceeds on beliefs, on judgement of probabilities, and not on

certainties.”
– Charles W. Eliot

This appendix contains the brief review of probability and analysis topics that we
use in the book. Its main aim is to act as a resource for the main results. It is not
meant to be a source for the beginners to learn these topics.

Probability Model

A random phenomenon or a random experiment is mathematically described by a

probability model (Ω, F , P). Ω is called the sample space: the set of all possible out-
comes of the random experiment. A subset of Ω is called an event. F is the set of all
possible events of interest about the random experiment. P is a consistent description
of the likelihood of the occurrence of the events in F .
Properties of F . F is called the σ−algebra of events and has the following proper-
ties:
1. Ω ∈ F,
2. E ∈ F ⇒ E c ∈ F,
3. E1 , E2 , · · · ∈ F ⇒ ∪∞
i=1 Ei ∈ F.

Properties of P(·). P is called the probability function or the probability measure. It

is a function P : F → [0,1] with the following properties:
1. P(Ω) = 1,
2. If E1 , E2 , · · · are mutually exclusive events in F
∞
X
P(∪∞
i=1 Ei ) = P(Ei ).
i=1

One can deduce the following important properties of the probability function:
P(φ) = 1, P(E ∪ F ) = P(E) + P(F ) − P(EF ).

491
492 PROBABILITY OF EVENTS
Conditional Probability

Suppose E, F ∈ F, and P(F ) > 0. Then P(E|F ), the conditional probability of E

given F , is given by
P(EF )
P(E|F ) = .
P(F )
If P(F ) = 0, the above conditional probability is undefined.
Independence of Events. E and F are called independent if
P(EF ) = P(E)P(F ).
The events E1 , E2 , · · · , En ∈ F are called mutually independent if for any A ⊆
{1, 2, · · · , n} Y
P(∩i∈A Ei ) = P(Ei ).
i∈A
Law of Total Probability. Let E1 , E2 , · · · ∈ F be mutually exclusive and exhaustive
events. Then, for F ∈ F,
∞
X
P(F ) = P(F |Ei )P(Ei ).
i=1

Bayes’ Rule. One consequence of the law of total probability is the Bayes’ rule:
P(F |Ei )P(Ei )
P(Ei |F ) = P∞ .
i=1 P(F |Ei )P(Ei )

Limits of Sets

Let E1 , E2 , · · · ∈ F. We define
lim sup En = ∩m≥1 ∪n≥m En , lim inf En = ∪m≥1 ∩n≥m En .
In words, lim sup En is the event that the events En occur infinitely often, and
lim inf En is the event that one of the events En occurs eventually. We have the
following inequalities, Fatou’s lemma for sets:
P(lim sup En ) ≥ lim sup P(En ),
P(lim inf En ) ≤ lim inf P(En ).
The following result is called the Borel-Cantelli lemma:
∞
X
P(En ) < ∞ ⇒ P(lim sup En ) = 0.
n=1

The second Borel-Cantelli lemma states: If {En , n ≥ 1} are independent events,

then
X∞
P(En ) = ∞ ⇒ P(lim sup En ) = 1.
n=1
 APPENDIX B

Univariate Random Variables

A real valued random variable is a function X : Ω → R = (−∞, ∞) such that

{ω ∈ Ω : X(ω) ≤ x} ∈ F for all −∞ < x < ∞. It is described by its cumulative
distribution function (cdf)
FX (x) = P(X ≤ x) = P({ω ∈ Ω : X(ω) ≤ x}), −∞ < x < ∞.
The cdf FX is a non-decreasing right-continuous function of x and is bounded above
by 1 and below by 0. X is said to be non-defective if
FX (∞) = lim FX (x) = 1,
x→∞
and
FX (−∞) = lim FX (x) = 0.
x→−∞
Otherwise it is called defective.

Discrete Random Variables

X is called discrete if FX is a piecewise constant function with jumps in a discrete

set A of real numbers. When A is a set of integers X is called an integer-valued
random variable. It can be described in an equivalent way by giving its probability
mass function (pmf) defined as
pX (x) = P(X = x) = FX (x) − FX (x−), x ∈ A.
A pmf satisfies X
pX (x) ≥ 0, pX (x) = 1.
x∈A

Continuous Random Variables

X is said to be a continuous random variable if FX is absolutely continuous, that

is, if there exists a function fX (·), called the probability density function (pdf), such
that Z x
FX (x) = fX (u)du, x ∈ R.
−∞

493
494 UNIVARIATE RANDOM VARIABLES
A pdf satisfies Z ∞
fX (x) ≥ 0, fX (u)du = 1.
−∞
It is possible for a random variable to have a discrete part and a continuous part.
There is a third possibility: FX (x) can be continuous, but not absolutely continuous.
Such random variables are called singular, and we will not encounter them in this
book.

Expectation of a Random Variables

The expectation of a random variable X is defined as

 P
E(X) = R ∞x∈A xpX (x) if X is discrete
−∞ xf X (x)dx if X is continuous,
provided the sums and integrals are well defined. When a random variable has both
discrete and continuous components, we employ the Stieltjes integral notation and
write Z ∞
E(X) = xdFX (x).
−∞
For a non-negative random variable we can also write
Z ∞
E(X) = (1 − FX (x))dx.
0

Functions of Random Variables. Let g be a function such that Y = g(X) is also a

random variable. This is the case if g : R → R is a piecewise continuous function.
Computing the expected value of g(X) can be done without first computing the cdf
of Y = g(X) by using the following formula:
Z ∞
E(g(X)) = g(x)dFX (x).
−∞

Suppose X is non-negative, and g(·) is a differentiable function such that

lim g(x)(1 − FX (x)) = 0.
x→∞

Then Z ∞
E(g(X)) = g ′ (x)(1 − FX (x))dx.
0
Expectations of special function of X have special names:

1. E(X n ) : nth moment of X,

2. E((X − E(X))n ) : nth central moment of X,
3. E((X − E(X))2 ) or Var(X) : variance of X,
p
4. Var(X) : standard deviation of X,
√
Var(X)
5. E(X) : coefficient of variation of X,
UNIVARIATE RANDOM VARIABLES 495
6. E(e−sX ) : Laplace-Stieltjes transform of X, used primarily for non-negative ran-
dom variables,
7. E(z X ) : generating function of X, used primarily for non-negative integer valued
random variables,
8. E(eiθX ): characteristic function of X, ( θ ∈ R).

Properties of the expectation:

E(aX + b) = aE(X) + b,
Var(aX + b) = a2 Var(X).

Table of Common Discrete Random Variables

The facts about the commonly occurring integer valued random variables are given
in the tables below.

Random Variable, X Symbol Parameter Range pX (k)

Bernoulli Ber(p) 0≤p≤1 pk (1 − p)1−k , k = 0, 1
Bin(n, p) n ≥ 0, 0 ≤ p ≤ 1 nk pk (1 − p)n−k , 0 ≤ k ≤ n


Binomial
Geometric G(p) 0≤p≤1 (1 − p)k−1 p, k ≥ 1
Modified Geometric MG(p) 0≤p≤1 (1 − p)k p, k ≥ 0
k
Poisson P(λ) λ≥0 e−λ λk! , k ≥ 0

X Mean Variance Generating Function

Ber(p) p p(1 − p) pz + 1 − p
Bin(n, p) np np(1 − p) (pz + 1 − p)n
pz
G(p) 1/p (1 − p)/p2 1−(1−p)z
p
MG(p) (1 − p)/p (1 − p)/p2 1−(1−p)z

P(λ) λ λ e−λ(1−z)
496 UNIVARIATE RANDOM VARIABLES
Table of Common Continuous Random Variables

The facts about the commonly occurring real valued continuous random variables
are given in the tables below.

Random Variable, X Symbol Parameter Range fX (x)

1
Uniform U(a, b) −∞ < a < b < ∞ b−a , a≤x≤b
Exponential exp(λ) λ≥0 λe−λx , x ≥ 0
k−1
Erlang Erl(k, λ) λ ≥ 0, k ≥ 1 λe−λx (λx)
(k−1)! , x ≥ 0

2 
Normal 1
N(µ, σ 2 ) −∞ < µ < ∞, σ 2 > 0 √2πσ 2
exp − 21 x−µσ

X Mean Variance LST(X)

a+b (b−a)2 e−sa −e−sb
U(a, b) 2 12 s(b−a)
1 1 λ
exp(λ) λ λ2 λ+s
 k
k k λ
Erl(k, λ) λ λ2 λ+s
 
σ2 s2
N(µ, σ 2 ) µ σ2 exp −µs + 2
 APPENDIX C

Multivariate Random Variables

Let Xi : Ω → R (i = 1, 2, · · · , n) be n real valued random variables. The X =

(X1 , X2 , · · · , Xn ) is called a multivariate random variable. When n = 2 it is called
a bivariate random variable. It is described by its joint cdf
FX (x) = P(Xi ≤ xi , 1 ≤ i ≤ n), x = (x1 , x2 , · · · , xn ) ∈ Rn .
The marginal cdf FXi (xi ) of Xi is given by
FXi (xi ) = FX (∞, · · · , ∞, xi , ∞, · · · , ∞).
X is called discrete if each Xi (1 ≤ i ≤ n) is a discrete random variable taking
values in a discrete set Ai ⊂ R. A discrete X can be described in an equivalent way
by giving its joint pmf
pX (x) = P(Xi = xi , 1 ≤ i ≤ n), xi ∈ Ai .
The marginal pmf pXi (xi ) of Xi is given by
X
pXi (xi ) = pX (x1 , · · · , xi−1 , xi , xi+1 , · · · , xn ).
xj ∈Aj :j6=i

X is said to be a jointly continuous random variable if FX is absolutely continuous,

that is, if there exists a function fX (x1 , x2 , · · · , xn ), called the joint pdf, such that
Z
FX (x) = fX (u1 , u2 , · · · , un )du1 du2 · · · dun , x ∈ Rn .
u≤x

It is possible that for each Xi is a continuous random variable but X is not a jointly
continuous random variable. The marginal pdf fXi (xi ) of Xi from a jointly contin-
uous X is given by
Z
fXi (xi ) = fX (u1 , · · · , ui−1 , xi , ui+1 , · · · , un )du1 · · · dui−1 dui+1 · · · dun .
xj ∈R:j6=i

497
498 MULTIVARIATE RANDOM VARIABLES
Independent Random Variables

The random variables (X1 , X2 , · · · , Xn ) are said to be independent if

n
Y
FX (x) = FXi (xi ), x = (x1 , x2 , · · · , xn ) ∈ Rn .
i=1

The jointly discrete random variables (X1 , X2 , · · · , Xn ) are said to be independent

if n
Y
pX (x) = pXi (xi ), xi ∈ Ai .
i=1
The jointly continuous random variables (X1 , X2 , · · · , Xn ) are said to be indepen-
dent if
Yn
fX (x) = fXi (xi ), xi ∈ R.
i=1

Sums of Random Variables

Let (X1 , X2 ) be a discrete bivariate random variable. Then the pmf of Z = X1 + X2

is given by X
pZ (z) = pX (x1 , x2 ).
(x1 ,x2 ):x1 +x2 =z
If X1 and X2 are independent, we have
X
pZ (z) = pX1 (x1 )pX2 (z − x2 ).
x1

This is called a discrete convolution. As a special case, if X1 and X2 are non-negative

integer valued random variables, we have
n
X
pZ (n) = pX1 (i)pX2 (n − i), n ≥ 0.
i=0

Let (X1 , X2 ) be a jointly continuous bivariate random variable. Then the pdf of
Z = X1 + X2 is given by
Z
fZ (z) = fX (x1 , x2 )dx1 dx2 .
(x1 ,x2 ):x1 +x2 =z

If X1 and X2 are independent, we have

Z
fZ (z) = fX1 (x1 )fX2 (z − x2 )dx1 .
x1 ∈R

This is called the convolution of fX1 and fX2 . As a special case, if X1 and X2 are
non-negative real valued random variables, we have
Z z
fZ (z) = fX1 (x)fX2 (z − x), z ≥ 0.
x=0
MULTIVARIATE RANDOM VARIABLES 499
If X1 and X2 are non-negative real valued random variables (discrete, continuous,
or mixed), Z z
FZ (z) = dFX1 (x)FX2 (z − x), z ≥ 0.
x=0
The following facts about the sums of independent random variables are useful:

1. Suppose (X1 , X2 , · · · , Xn ) are iid Ber(p). Then X1 + X2 + · · · + Xn is Bin(n, p).

2. Suppose (X1 , X2 , · · · , Xn ) are iid exp(λ). Then X1 + X2 + · · · + Xn is Erl(n, λ).
3. Suppose Xi ∼ P(λi ) (1 ≤ i ≤ n) are independent. Then X1 + X2 + · · · + Xn is
P(λ) where λ = λ1 + λ2 + · · · + λn .

Functions of Multi-Variate Random Variables

Let g : Rn → R be a function such that Z = g(X) is a real valued random variable.

This is true if g is a piecewise differential function, for example. The expectation of
Z can be computed without computing its distribution as follows:
Z
E(Z) = g(u)dFX (u).
u∈Rn

In particular we have
n n
!
X X
E Xi = E(Xi ),
i=1 i=1
which holds even if the Xi ’s are dependent. If they are independent, we also have
n n
!
Y Y
E Xi = E(Xi ).
i=1 i=1

We define the covariance of two random variables as

Cov(X1 , X2 ) = E(X1 X2 ) − E(X1 )E(X2 ).
If X1 and X2 are independent, their covariance is zero. The converse is not true in
general. We have
n n n n
!
X X X X
Var Xi = Var(Xi ) + 2 Cov(Xi , Xj ).
i=1 i=1 i=1 j=i+1

In particular, when the Xi ’s are independent,

n n
!
X X
Var Xi = Var(Xi ).
i=1 i=1
500 MULTIVARIATE RANDOM VARIABLES
Conditional Distributions and Expectations

Let (X1 , X2 ) be a discrete bivariate random variable. Then the conditional pmf of
X1 given X2 = x2 is given by
pX (x1 , x2 )
pX1 |X2 (x1 |x2 ) = ,
pX2 (x2 )
and the conditional expected value of X1 given X2 = x2 is given by
X
E(X1 |X2 = x2 ) = x1 pX1 |X2 (x1 |x2 ).
x1

We have the following useful formula for computing expectations by conditioning:

X
E(X1 ) = E(X1 |X2 = x2 )pX2 (x2 ),
x2

Let (X1 , X2 ) be a jointly continuous bivariate random variable. Then the conditional
pdf of X1 given X2 = x2 is given by
fX (x1 , x2 )
fX1 |X2 (x1 |x2 ) = ,
fX2 (x2 )
and the conditional expected value of X1 given X2 = x2 is given by
Z
E(X1 |X2 = x2 ) = x1 fX1 |X2 (x1 |x2 )dx1 .
x1

We have the following useful formula for computing expectations by conditioning:

Z
E(X1 ) = E(X1 |X2 = x2 )fX2 (x2 )dx2 .
x2

In general we can define E(X1 |X2 ) to be a random variable that takes value
E(X1 |X2 = x2 ) with “probability dFX2 (x2 ).” With this interpretation we get
Z
E(X1 ) = E(E(X1 |X2 )) = E(X1 |X2 = x2 )dFX2 (x2 ).
x2

Order Statistics

Let X1 , X2 , · · · , Xn be iid random variables with common cdf F (·). Let Y1 ≤

Y2 ≤ · · · ≤ Yn be the X1 , X2 , · · · , Xn ordered in an ascending order, i.e., Y1 =
min(X1 , X2 , · · · , Xn ), · · · , Yn = max(X1 , X2 , · · · , Xn ). Then (Y1 , Y2 , · · · , Yn ) is
called the order statistics of (X1 , X2 , · · · , Xn ). The marginal distribution of Yk is
given by
n  
X n
FYk (t) = P(Yk ≤ t) = F (t)j (1 − F (t))n−j , 1 ≤ k ≤ n.
j
j=k

In particular
FYn (t) = F (t)n , FY1 (t) = 1 − (1 − F (t))n .
MULTIVARIATE RANDOM VARIABLES 501
In addition, when (X1 , X2 , · · · , Xn ) are jointly continuous with common pdf f (·),
the (Y1 , Y2 , · · · , Yn ) are also jointly continuous with joint pdf given by
fY (y1 , y2 , · · · , yn ) = n!f (y1 )f (y2 ) · · · f (yn ), y1 ≤ y2 ≤ · · · ≤ yn .
The density is zero outside the above region.

Multivariate Normal Random Variable

Let µ be an n-dimensional vector, and Σ = [σij ] be an n × n positive definite matrix.

A random vector X = [X1 , X2 , · · · , Xn ] is called a multi-variate normal variable
N (µ, Σ) if it has the joint pdf given by
 
1 1 −1 ⊤
fX (x) = p exp − (x − µ)Σ (x − µ) , x = (x1 , . . . , xn ) ∈ Rn .
(2π)n det(Σ) 2
In this case we have
E(Xi ) = µi , Cov(Xi , Xj ) = σij .
If Σ is diagonal, the [X1 , X2 , · · · , Xn ] are independent normal random variables and
Xi ∼ N (µi , σii ). If Σ is the identity matrix and µ = 0, the n components of X
are iid standard normal random variables. Now let Z = AX + c where A is an
m × n matrix of real numbers and c is an m-dimensional vector of reals. The Z is a
N (µZ , ΣZ ) random variable where
µZ = Aµ + c, ΣZ = AΣA⊤ .
Now suppose the n-dimensional multivariate vector X ∼ N (µ, Σ) is partitioned as
[X1, X2] where Xi is ni dimensional (n1 + n2 = n). Partition the µ vector and the
Σ matrix in a commensurate fashion as
   
µ1 Σ11 Σ12
µ= , Σ= .
µ2 Σ21 Σ22
Then the marginal distribution of X1 is N (µ1, Σ11), and the distribution of X1
given X2 = a is multivariate normal N (µ̄, Σ) where
µ̄ = µ1 + (Σ12)(Σ22−1 )(a − µ2),
and covariance matrix
Σ = Σ11 − (Σ12)(Σ22−1 )(Σ21).
 APPENDIX D

Generating Functions

Let X be a non-negative integer valued random variable with pmf {pk , k ≥ 0}. The
generating function (GF) of X (or its pmf) is defined as
∞
X
gX (z) = E(z X ) = z k pk , |z| ≤ 1.
k=0

The GFs for common random variables in Appendix B. Important and useful prop-
erties of the GF are enumerated below:

1. gX (1) = P(X < ∞).

2. A random variable or its pmf is uniquely determined by its GF.
3. The pmf can be obtained from its GF as follows:
1 dk
pk = gX (z)|z=0 , k ≥ 0.
k! dz k
4. The factorial moments of X are given by
dk
E(X(X − 1) · · · (X − k + 1)) = gX (z)|z=1 , k ≥ 1.
dz k
5. Let X1 and X2 be independent random variables. Then
gX1 +X2 (z) = gX1 (z)gX2 (z).
6. Let {Xn , n ≥ 1} be a given sequence of random variables and let X be such that
lim P(Xn = k) = P(X = k), k ≥ 0.
n→∞

Then
lim gXn (z) = gX (z), |z| ≤ 1.
n→∞
The converse also holds.

Let {pk , k ≥ 0} be a sequence of real numbers, not necessarily a pmf. Define its GF
as
X∞
p(z) = z k pk .
k=0

503
504 GENERATING FUNCTIONS
Let R be its radius of convergence. We list some important and useful properties of
the GF:

1. Let q(z) be a GF of a sequence {qk , k ≥ 0}. If p(z) = q(z) for |z| < r ≤ R, then
pk = qk for all k ≥ 0.
2. Let rk = apk + bqk , k ≥ 0, where a and b are constants. Then
r(z) = ap(z) + bq(z).
3. Generating function of {kpk } is zp′ (z).
4. Let q0 = p0 , qk = pk − pk−1 for k ≥ 1. Then
q(z) = (1 − z)p(z).
Pk
5. qk = r=0 pr for k ≥ 0. Then
q(z) = p(z)/(1 − z).
1
Pk
6. limk→∞ k+1 r=0 pr = limz→1 (1 − z)p(z) if the limit on either side exists.
7. limk→∞ pk = limz→1 (1 − z)p(z) if the limit on the left hand side exists.

Suppose the GF of {pk , k ≥ 0} is given by

P (z)
p(z) = ,
Q(z)
where P (z) is a polynomial of degree r, and Q(z) is a polynomial of degree s > r.
Suppose Q(z) has distinct roots so that we can write
s
Y
Q(z) = c (1 − zαi ),
i=1

where {αi , 1 ≤ i ≤ s} are distinct. Then

s
X
pk = ci αki ,
i=1

where
P (1/αi )
ci = −αi , 1 ≤ i ≤ s.
Q′ (1/αi )
 APPENDIX E

Laplace-Stieltjes Transforms

Let X be a non-negative real valued random variable with cdf FX (·). The Laplace-
Stieltjes transform (LST) of X (or its cdf) is defined as
Z ∞
−sX
φX (s) = E(e )= e−sx dFX (x), Re(s) ≥ 0.
x=0
The LSTs for common random variables in Appendix B. Important and useful prop-
erties of the LST are enumerated below:

1. φX (0) = P(X < ∞).

2. A random variable or its cdf is uniquely determined by its LST.
3. The moments of X are given by
dk
E(X k ) = (−1)k φX (s)|s=0 , k ≥ 1.
dsk
4. Let X1 and X2 be independent random variables
φX1 +X2 (s) = φX1 (s)φX2 (s).
5. Let {Xn , n ≥ 1} be a given sequence of random variables and let X be such that
lim P(Xn ≤ x) = P(X ≤ x) = F (x), k ≥ 0,
n→∞

at all points of continuity of F . Then

lim φXn (s) = φX (s), Re(s) > 0.
n→∞

The converse also holds.

Let F : [0, ∞) → (−∞, ∞), not necessarily a cdf. Define its LST as
Z ∞
LST (F ) = F̃ (s) = e−sx dF (x),
x=0

if the integral exists for some complex s with Re(s) > 0. It is assumed F (0−) = 0
so that there is a jump of size F (0+) at x = 0. We list some important and useful
properties of the LST:

505
506 LAPLACE-STIELTJES TRANSFORMS
1. Let a and b be given constants. Then
LST (aF + bG) = aF̃ (s) + bG̃(s).
Rt
2. H(t) = 0 F (t − u)dG(u), t ≥ 0 ⇔ H̃(s) = F̃ (s)G̃(s).
3. Assuming the limits exist
lim F (t) = lim F̃ (s),
t→∞ s→0

lim F (t) = lim F̃ (s),

t→0 s→∞
F (t)
lim = lim sF̃ (s).
t→∞ t s→0
 APPENDIX F

Laplace Transforms

Let f : [0, ∞) → (−∞, ∞). Define its Laplace transform (LT) as

Z ∞
∗
LT (f ) = f (s) = e−sx f (x)dx,
x=0
if the integral exists for some complex s with Re(s) > 0. We list some important and
useful properties of the GF:

1. LT (af + bg) = af ∗ (s) + bg ∗ (s).

2. F̃ (s) = sF ∗ (s).
Rt
3. h(t) = 0 f (t − u)g(u)du, t ≥ 0 ⇔ h∗ (s) = f ∗ (s)g ∗ (s).
4. LT (f ′(t)) = sf ∗ (s) − f (0).
5. LT (e−at f (t)) = f ∗ (s + a).
n
d
6. LT (tn f (t)) = (−1)n ds ∗
n f (s).

Table of Laplace Transforms.

f (t) f ∗ (s)
1 1/s
t 1/s2
tn n!/sn+1
e−at 1/(s + a)
e−at tn−1 /(n − 1)! 1/(s + a)n
(e−at − e−bt )/(b − a) 1/(s + a)(s + b)

507
 APPENDIX G

Modes of Convergence

Let the random variables X1 , X2 , X3 , · · · be defined on a common probability space

(Ω, F , P). Let X be another random variable defined on the same probability space.

1. Almost Sure Convergence.

Xn is said to converge almost surely to X, written Xn → X (a.s.), if
P( lim Xn = X) = P({ω ∈ Ω : lim Xn (ω) = X(ω)}) = 1.
n→∞ n→∞

This mode of convergence is also called convergence with probability 1, or al-

most everywhere convergence, or sample-path convergence. If Xn converges to
X almost surely and f is a continuous function, then f (Xn ) converges to f (X)
almost surely.
2. Convergence in Probability.
Xn is said to converge to X in probability, written Xn → X (p), if
lim P(|Xn − X| > ǫ) = 0
n→∞

for any ǫ > 0. If Xn converges to X in probability and f is a continuous function,

then f (Xn ) converges to f (X) in probability.
3. Convergence in Distribution.
Xn is said to converge to X in distribution, written Xn → X (d), if
lim FXn (x) = FX (x)
n→∞

at all points of continuity of FX . If Xn converges to X in distribution and f is a

continuous function, then f (Xn ) converges to f (X) in distribution.
4. Convergence in Mean.
Xn is said to converge to X in mean, written Xn → X (m), if
lim E(|Xn − X|) = 0.
n→∞

5. Convergence in Mean Square.

Xn is said to converge to X in mean square, written Xn → X (m.s.), if
lim E((Xn − X)2 ) = 0.
n→∞

509
510 MODES OF CONVERGENCE
The various modes of convergence are related as follows: Almost sure convergence
implies convergence in probability which implies convergence in distribution. Mean
square convergence implies convergence in probability. Convergence in distribution
or with probability one does not imply convergence in mean. We need an additional
condition called uniform integrability defined as follows: A sequence of random vari-
ables {Xn , n ≥ 0} is called uniformly integrable if for a given ǫ > 0, there exists a
K < ∞ such that
Z
E(|X|; |X| > K) = xdFXn (dx) < ǫ, n ≥ 1.
{x:|x|>K}

Then we have the following result: If Xn converges to X in distribution (or almost

surely), and {Xn , n ≥ 0} is uniformly integrable, the Xn converges to X in mean.

The three main convergence results associated with a sequence {Xn , n ≥ 1} of

iid random variables with common mean τ and variance σ 2 are given below.

1. Weak Law of Large Numbers. Let X̄n = (X1 + X2 + · · · + Xn )/n. Suppose

τ < ∞. Then
X̄n → τ (p).
2. Strong Law of Large Numbers. Assume τ < ∞. Then
X̄n → τ (a.s.).

3. Central Limit Theorem. Assume σ 2 < ∞. Then

X̄n − τ
√ → N (0, 1) (d).
σ/ n
 APPENDIX H

Results from Analysis

We frequently need to interchange the operations of sums, integrals, limits, etc. Such
interchanges are in general not valid unless certain conditions are satisfied. In this
section we collect some useful sufficient conditions which enable us to do such in-
terchanges.
1. Monotone Convergence Theorem for Sums. Let π(i) ≥ 0 for all i ≥ 0 and
{gn (i), n ≥ 1} be a non-decreasing non-negative sequence for each i ≥ 0. Then
∞ ∞ 
!
X X 
lim gn (i)π(i) = lim gn (i) π(i).
n→∞ n→∞
i=0 i=0

2. Suppose {gn (i), n ≥ 1} is a non-negative sequence for each i ≥ 0 and π(i) ≥ 0

for all i ≥ 0. Then
X∞ X ∞ ∞ X
X ∞
gn (i)π(i) = gn (i)π(i).
n=0 i=0 i=0 n=0

3. Fatou’s Lemma for Sums. Suppose {gn (i), n ≥ 1} is a non-negative sequence for
each i ≥ 0 and π(i) ≥ 0 for all i ≥ 0. Then
∞  ∞
 !
X X
lim inf gn (i) π(i) ≤ lim inf gn (i)π(i) .
n≥1 n≥1
i=0 i=0

4. Bounded Convergence Theorem for Sums. P Suppose π(i) ≥ 0 for all i ≥ 0, and
there exists a {g(i), i ≥ 0} such that g(i)π(i) < ∞ and |gn (i)| ≤ g(i) for all
n ≥ 1 and i ≥ 0. Then
∞
X ∞ 
X 
lim gn (i)π(i) = lim gn (i) π(i).
n→∞ n→∞
i=0 i=0

5. Let {πn (i), n ≥ 1} be a non-negative sequence for each i ≥ 0 and let

π(i) = lim πn (i), i ≥ 0.
n→∞
Suppose
∞
X X
lim πn (i) = π(i) < ∞.
n→∞
i=0

511
 512 RESULTS FROM ANALYSIS
Suppose {g(i), i ≥ 0} is a non-negative bounded function with 0 ≤ g(i) ≤ c for
all i ≥ 0 for some c < ∞. Then
∞
X ∞
X  
lim g(i)πn (i) = g(i) lim πn (i) .
n→∞ n→∞
i=0 i=0

6. Monotone Convergence Theorem for Integrals. Let π(x) ≥ 0 for all x ∈ R and
{gn (x), n ≥ 1} be a non-decreasing non-negative sequence for each x ∈ R. Then
Z  Z  
lim gn (x)π(x)dx = lim gn (x) π(x)dx.
n→∞ x∈R x∈R n→∞

7. Suppose {gn (x), n ≥ 1} is a non-negative sequence for each x ∈ R and π(x) ≥ 0

for all x ∈ R. Then
X∞ Z Z ∞
X
gn (x)π(x)dx = gn (x)π(x)dx.
n=0 x∈R x∈R n=0

8. Fatou’s Lemma for Integrals. Suppose {gn (x), n ≥ 1} is a non-negative sequence

for each x ∈ R and π(x) ≥ 0 for all x ∈ R. Then
Z   Z 
lim inf gn (x) π(x)dx ≤ lim inf gn (x)π(x)dx .
x∈R n≥1 n≥1 x∈R

9. Bounded Convergence Theorem for Integrals. R Let π(x) ≥ 0 for all x ∈ R and
there exists a function g : R → R such that g(x)π(x)dx < ∞, and |gn (x)| ≤
g(x) for each x ∈ R and all n ≥ 1. Then
Z  Z  
lim gn (x)π(x)dx = lim gn (x) π(x)dx.
n→∞ x∈R x∈R n→∞

10. Let {πn (x), n ≥ 1} be a non-negative sequence for each x ∈ R and let
π(x) = lim πn (x), x ∈ R.
n→∞

Suppose Z Z
lim πn (x)dx = π(x)dx < ∞.
n→∞ x∈R x∈R
Suppose g : R → [0, ∞) is a non-negative bounded function with 0 ≤ g(x) ≤ c
for all x ∈ R for some c < ∞. Then
Z  Z  
lim g(x)πn (x)dx = g(x) lim πn (x) dx.
n→∞ x∈R x∈R n→∞
 APPENDIX I

Difference and Differential Equations

A sequence {xn , n ≥ 0} is said to satisfy an nth order difference equation with

constant coefficients {ai , 0 ≤ i ≤ n − 1} if
n−1
X
ai xk+i + xk+n = rk , k ≥ 0.
i=0
If the right hand side rk is zero for all k, the equation is called homogeneous, else it
is called non-homogeneous. The polynomial
n−1
X
ai αi + αn = 0,
i=0
is called the characteristic polynomial corresponding to the difference equation. Let
{α(i), 1 ≤ i ≤ d} be the d distinct roots of the characteristic polynomial, and let
m(i) be the multiplicity of α(i). ( di=1 m(i) = n.) Then any solution to the homo-
P
geneous equation
n−1
X
ai xk+i + xk+n = 0, k ≥ 0,
i=0
is of the form
d m(i)−1
X X
xn = cij nj α(i)n , n ≥ 0.
i=1 j=0
The constants cij are to be determined by using the initial conditions. Now consider
the non-homogeneous equation
n−1
X
ai xk+i + xk+n = rk , k ≥ 0
i=0
Any solution of this equation is of the form
d m(i)−1
X X
xn = xpn + cij nj α(i)n
i=1 j=0

where {xpn , n ≥ 0} is any one solution (called the particular solution) of the non-
homogeneous equation. The constants cij are to be determined by using the initial

513
514 DIFFERENCE AND DIFFERENTIAL EQUATIONS
conditions.

A function {x(t), t ≥ 0} is said to satisfy an nth order differential equation with

constant coefficients {ai , 0 ≤ i ≤ n − 1} if
n−1
X di dn
ai x(t) + x(t) = r(t), t ≥ 0.
i=0
dti dtn

If the right hand side r(t) is zero for all t ≥ 0, the equation is called homogeneous,
else it is called non-homogeneous. The polynomial
n−1
X
ai αi + αn = 0,
i=0

is called the characteristic polynomial corresponding to the differential equation. Let

{α(i), 1 ≤ i ≤ d} be the d distinct roots of the characteristic polynomial, and let
m(i) be the multiplicity of α(i). ( di=1 m(i) = n.) Then any solution to the homo-
P
geneous equation
n−1
X di dn
ai i x(t) + n x(t) = 0, t ≥ 0
i=0
dt dt
is of the form
d m(i)−1
X X
x(t) = cij tj eα(i)t .
i=1 j=0
The constants cij are to be determined by using the initial conditions. Now consider
the non-homogeneous equation
n−1
X di dn
ai x(t) + x(t) = r(t), t ≥ 0.
i=0
dti dtn
Any solution of this equation is of the form
d m(i)−1
X X
p
x(t) = x (t) + cij tj eα(i)t
i=1 j=0
p
where {x (t), t ≥ 0} is any one solution (called the particular solution) of the non-
homogeneous equation. The constants cij are to be determined by using the initial
conditions.
 Answers to Selected Problems

Chapter 2

MODELING EXERCISES

2.1 State space = {0, 1, 2, · · ·}.

∞
X
pi,0 = pi+1 / pj , pi,i+1 = 1 − pi,0 , i ≥ 0.
j=i+1

2.3 State space = {b, b + 1, b + 2, · · ·}.


Xn + k − 1 w.p. Xn /(w + b + n(k − 1))
Xn+1 =
Xn w.p. 1 − Xn /(w + b + n(k − 1))
Thus {Xn , n ≥ 0} is a DTMC, but it is not time-homogeneous.

2.5 Xn = 1 if the weather is sunny on day n, 0 otherwise. Yn = (Xn−1 , Xn ), n ≥ 1.

{Yn , n ≥ 1} is a DTMC on S = {(0, 0), (0, 1), (1, 0), (1, 1)} with the following
transition probability matrix:
 
.6 0.4 0 0
 0 0 .25 .75 
P =  .5 .5
.
0 0 
0 0 .2 .8

2.7 {Xn , n ≥ 0} is a space homogeneous random walk on S = {..., −2, −1, 0, 1, 2, ...}
with
pi = p1 (1 − p2 ), qi = p2 (1 − p1 ), ri = 1 − pi − qi .

2.9 State space = S = {1, 2, 3, 4, 5, 6}. Transition probabilities:

pi,j = 1/6, i 6= j, pii = 0.

2.11 Bn (Gn ) = the bar the boy (girl) is in on the nth night. {(Bn , Gn ), n ≥ 0} is a
DTMC on S = {(1, 1), (1, 2), (2, 1), (2, 2)} with the following transition probability

515
516 ANSWERS TO SELECTED PROBLEMS
matrix:
 
1 0 0 0
 a(1 − d) ad (1 − a)(1 − d) (1 − a)d 
P =
 (1 − b)c (1 − b)(1 − c)
.
bc b(1 − c) 
0 0 0 1

The story ends in bar k if the DTMC gets absorbed in state (k, k), for k = 1, 2.

2.13 pii = pi , pi,i+1 = 1 − pi , 1 ≤ i ≤ k − 1, pk,k = pk , pk,1 = 1 − pk .

2.15 The state space is S = {(1, 2), (2, 3), (3, 1)}. The transition probability matrix
is  
0 b21 b12
P =  b23 0 b32  .
b13 b31 0

2.17 {Xn , n ≥ 0} is a DTMC on S = {0, 1, · · · , M } with

pi,j = αj , 0 ≤ i < j ≤ M,
i
X
pi,i = αk , 0 ≤ i ≤ M.
k=0

2.19 {Xn , n ≥ 0} is a DTMC since {Yn , n ≥ 0} are iid and Xn+1 = Yn +

Bin(Xn , 1 − p).

2.21. Xn+1 = min(Bin(Xn , 1 − p) + Yn+1 , B). Hence {Xn , n ≥ 0} is a DTMC

with transition probabilities:
i  
X i k
pi,j = p (1 − p)i−k αk+j−i , 0 ≤ i ≤ B, 0 ≤ i < B,
k
k=0

and
B−1
X
pi,B = 1 − pij ,
j=0
where we use the convention that αk = 0 if k < 0.

2.23 Since {Dn , n ≥ 0} are iid, and


Xn − Dn if Xn − Dn ≥ s,
Xn+1 =
S if Xn − Dn < s,
{Xn , n ≥ 0} is a DTMC on state space {s, s + 1, ..., S − 1, S}. The transition
probabilities are
pi,j = αi−j , s ≤ j ≤ i ≤ S, j 6= S,
X∞
pi,S = αk , s ≤ i < S, j = S
k=i−s+1
ANSWERS TO SELECTED PROBLEMS 517
∞
X
pS,S = α0 + αk .
k=S−s+1

2.25 {Xn , n ≥ 0} is a DTMC general simple random walk on {0, 1, 2, ...} with
transition probabilities
p0,0 = 1, pi,i+1 = βi α2 , pi,i−1 = βk α0 , pii = βi α1 + 1 − βi .

2.27 {Xn , n ≥ 0} is a DTMC with state space S = {rr, dr, dd} and transition
probability matrix  
0 1 0
P =  0 .5 .5  .
0 0 1

2.29. {Xn , n ≥ 0} is a branching process with common progeny distribution

P(Yi,n = 0) = 1 − α; P(Yi,n = 20) = α.
The number of recipients in the (n + 1)st generation are given by
Xn
X
Xn+1 = Yi,n .
i=1

2.31 Xn+1 = max(Xn −1+Yn , Yn ). This is the same as the DTMC in Example 2.16.

COMPUTATIONAL EXERCISES

2.1 (a).E(X20 |X0 = 8) = 5.0346, E(X20 |X0 = 5) = 5.0000, E(X20 |X0 = 3) =

4.9769.
2.5 Bin(k, .3187).
2.7 E(Xn ) = (1 − d)n (pα + 1 − p)n , E(Xn2 ) = (1 − d)2n (pα2 + 1 − p)n , where
α = (1 + u)/(1 − d).
2.9 E(X5 ) = 14.9942, E(X10 ) = 14.5887.
2.13 E(X1 ) = 9.9972, E(X10 ) = 19.0538.
(n)
2.19 p11 = .4 + 0.5236(−0.2236)n + 0.0764(0.2236)n, n ≥ 0.

2.21
21−n 1 − 21−n
   
0 1 0 0
P =  0 .5 .5  , P n = XDn X −1 =  0 2−n 1 − 2−n  , n ≥ 1.
0 0 1 0 0 1
518 ANSWERS TO SELECTED PROBLEMS
2.23
E(Xn |X0 = i) = iµn
inσ 2 if µ = 1

Var(Xn |X0 = i) = n .
iσ 2 µn−1 µµ−1
−1
if µ 6= 1.

2.25 E(Xn ) = i, Var(Xn ) = (1 − an )i(N − i), where a = (N − 1)/N .

CONCEPTUAL EXERCISES

2.1 First show that

P(Xn+2 = k, Xn+1 = j|Xn = i, Xn−1 , · · · , X0 ) = P(X2 = k, X1 = j|X0 = i).
Sum over all j ∈ A and k ∈ B to get the desired result.

2.3 (a) False (b) True (c) False.

2.5 No.

2.7 P(Ni = k|X0 = i) = (pi,i )k−1 (1 − pi,i ), k ≥ 1.

2.9 Use induction.

2.11 {|Xn |, n ≥ 0} is a random walk on {0, 1, 2, ...} with p0,1 = 1, and, for i ≥ 1,
pi+1 + q i+1
pi,i+1 = = 1 − pi,i−1 .
pi + q i

Chapter 3

COMPUTATIONAL EXERCISES

3.1 (a) .87, (b) 5.7895, (c) 16.5559.

3.3 14.5771.

3.5 2.6241.

3.7 18.

3.9 2(e-1) = 3.4366.

pr−1 (1−qm )
3.11 1−(1−pr−1 )(1−qm−1 )
ANSWERS TO SELECTED PROBLEMS 519
3.13 7.

3.15 1/p2 q 2 .

3.17 If p = q the formula reduces to i(N − i).

(2−pr )pr
3.25 .5 1−(1−pr2)(1−p
1
r) .
1 2

3.27 .2.

3.29 M + 1.

3.33 1/(a + b − 2ab).

CONCEPTUAL PROBLEMS

P∞
3.3 ṽi = vi , i > 0, ṽ0 = j=1 pij vj .

3.7 For B ⊂ A, and i ≥ 1, let u(i, B) be the conditional probability that the process
visits all states in A before visiting state 0, given that currently it is in state i and it
has visited the states in set B so far. Then
X X
u(i, B) = pi,j u(j, B ∪ {j}) + pi,j u(j, B).
j∈A−B j6=0,j ∈
/ A−B

3.9
∞
X
wi = δi,j pi,0 + pi,k wk .
k=1

Chapter 4

COMPUTATIONAL EXERCISES

M (n)
4.1 All rows of P n and n+1 converge to [.132 .319 .549].

M (n)
4.3 All rows of P n and n+1 converge to [1/(N + 1), 1/(N + 1), · · · , 1/(N + 1)].

4.5 p > q → Transient, p = q → Null recurrent, p < q → Positive recurrent.

4.7 Results of special case 2 of Example 4.22 continue to hold.

4.9 Communicating class: {A, B, C} All states are aperiodic, positive recurrent.
520 ANSWERS TO SELECTED PROBLEMS
4.13 (a)

1. Communicating Classes {1}, {2}, {3}, transient, aperiodic.

2. Communicating Class {4}, positive recurrent, aperiodic.

(b)

1. Class 1: {1, 2}, positive recurrent, aperiodic.

2. Class 2: {3}, transient, aperiodic.
3. Class 3: {4}, positive recurrent, aperiodic.

4.15 (i) Positive recurrent, (ii) Null recurrent, (iii) Positive recurrent.

4.17 Positive recurrent if ∞

P P∞
n=0 npn < ∞, null recurrent if n=0 npn = ∞.

4.19 Use Pakes’s lemma.

4.21 (a) Limiting distribution: [.25 .25 .25 .25]. (b) Limiting occupancy distribu-
tion: [ 16 31 13 16 ].

4.23 Limiting probability that the bus is full = 0.6822.

4.25 Pkmi , where mi = 1/(1 − pi ).

j=1
mj

2α−1 1−α j
4.27 (i) πj = α ( α ) , j ≥ 0.

4.29 1/r.
2
(N )
4.31 πn = PN n N 2 , 0 ≤ n ≤ N.
( )
j=0 j

P∞
4.33 πn = i=n+1 αi /τ, n ≥ 0.

4.35 φ(z) = ∞ n n
Q
n=0 ψ((1 − p) z + 1 − (1 − p) ).

4.37 π0 α0 + πB−1 α2 , where πj is the long run probability that the buffer has j bytes
in it.

4.39 τ1 /(τ1 + τ2 ).

4.41 [.25, .5, .25].

4.43
 
4/11 7/11 0 0
 4/11 7/11 0 0 
(a) 
 4/11
,
7/11 0 0 
4/11 7/11 0 0
ANSWERS TO SELECTED PROBLEMS 521
 
90/172 27/172 55/172 0 0 0
 90/172 27/172 55/172 0 0 0 
 
 90/172 27/172 55/172 0 0 0 
(b)  .

 0 0 0 1 0 0 

 1170/3956 351/3956 715/3956 10/23 0 0 
540/3956 162/3956 330/3956 17/23 0 0

K−1
P K−1
P
4.45 (C1 r + C2 (1 − r))/T , where r = pi , T = ipi + K(1 − r).
i=1 i=1

4.47 d(1 − 2p) + cr/(1 − r) where r = (p/(1 − p))2 .

4.51 3.005 dollars per day.

4.53 Optimum B = 12, maximum probability = .9818.

CONCEPTUAL EXERCISES

4.1 (i) Reflexive, symmetric, transitive.

(iii) Reflexive, symmetric, not transitive.

4.17 The results follows from the fact that c(i) is the expected cost incurred at time
n if Xn = i.

4.21 Global balance equations are obtained by summing the local equation over all
j.

4.29 Suppose the DTMC earns one dollar every time it undergoes a transition from
state i to j, and 0 otherwise. Then use Conceptual Exercise 4.18.

4.31 g(i) = ri + ∞
P
j=1 pij g(j), i > 0.

Chapter 5

COMPUTATIONAL EXERCISES
h i
λ2 −λ1 x λ1 −λ2 x
5.1 P(Length of the shortest path > x) = exp(−λ3 x) λ2 −λ1 e + λ1 −λ2 e .
 
λ2 −λ1 x λ1 −λ2 x
5.3 P(Length of the longest path ≤ x) = 1− λ2 −λ1 e − λ1 −λ2 e (1 −
−λ3 x
e ).

5.5 (1 − e−λx )n .
T
5.7 1−e−λT
− λ1 .
522 ANSWERS TO SELECTED PROBLEMS
1
5.9 λ+µ 2p(1 + 3q 2 ) where p = µ/(λ + µ) and q = 1 − p.

5.11 Process job 1 first if C1 λ1 > C2 λ2 , otherwise process job 2 first.

Pn−1 (λT )i
5.17 k = min{n ≥ 1 : e−λT i=0 i! ≥ α}.

5.25 12 (1 − e−2λt ).

5.27 e−λτ .

5.33 (i) e−1 , (ii) e−1 /2, 1 − 2.5e−1

5.35 6.

Rt c(t − k) if 2k ≤ t < 2k + 1
5.37 N (t) ∼ P(Λ(t)) where Λ(t) = 0 λ(u)du =
c(k + 1) if 2k + 1 ≤ t < 2k + 2.
Rt
5.39R(t) ∼ P( 0 λ(u)(1 − G(t − u))du).

5.41 E(Z(t)) = λτ and Var(Z(t)) = λs2 t, where

λd λw
τ= τd + τw ,
λd + λw λd + λw
λd λw
s2 = (τd2 + σd2 ) + (τ 2 + σw
2
).
λd + λw λd + λw w
5.43 (i) {N (t), t ≥ 0} ∼ NPP(r(t)) where r(t) = f (t)/(1 − F (t))). (ii)
RT
0 c(t)r(t)dt.

CONCEPTUAL EXERCISES

5.1 Let H(x) = P(X > x). Definition of hazard rate implies H ′ (x) = −r(x)H(x).
The solution is Equation 5.4.

5.9Ai is a modified geometric ( λ1λ+λ

i
2
) random variable.

5.13 (i) No, (ii) No, (iii) No.

5.21 It is not an NPP since it does not have independent increments property.
2
5.23 e−λπx .

Chapter 6
MODELING EXERCISES
ANSWERS TO SELECTED PROBLEMS 523
6.1 State space = {0, 1, ..., k}.
qi,i+1 = (k − i)λ, 0 ≤ i ≤ k − 1,
qi,i−1 = iµ, 1 ≤ i ≤ k.

6.3 State space = {0, 1, 2, 12, 21}. The state represents the queue of failed machines.
 
−(µ1 + µ2 ) µ1 µ2 0 0

 λ1 −(λ1 + µ2) 0 µ2 0  
Q=  λ2 −(λ2 + µ1) 0 0 µ1  .
 0 0 λ1 −λ1 0 
0 λ2 0 0 −λ2

6.5 Pure death process on {0, 1, ..., k} with

µi = µM, 1 ≤ i ≤ k.

6.7 The state space = {0, 1A, 1B, 2, 3, 4, ...}. State 1A (1B) = one customer in the
system and he is being served by server A (B). State i = i customers in the system.
q0,1A = λα, q0,1B = λ(1 − α),
q1A,0 = µ1 , q1A,2 = λ,
q1B,0 = µ2 , q1B,2 = λ,
q2,1A = µ2 , q2,1B = µ1 , q2,3 = λ,
qi,i+1 = λ, qi,i−1 = µ1 + µ2 , i ≥ 3.

6.9 q0,i = λi , qi,0 = µi , 1 ≤ i ≤ n.


λ1 + λ2 if 0 ≤ i < s
6.11 λi = , µi = min(i, s)µ, i ≥ 0.
λ1 if i ≥ s,

6.13 q(0,0),(1,j) = λαj , q(1,j),(0,0) = µj , j = 1, 2,

q(i,j),(i+1,j) = λ, i ≥ 1, j = 1, 2,
q(i,j),(i−1,k) = µj αk , i ≥ 2, j, k = 1, 2.

6.15 qi,i+1 = .4iλ, qi,i+2 = .3iλ, qi,i−1 = .3iµ, i ≥ 0.

6.17 State space = {0, 1, 2, 3, 4, 5}, 0 = failed, i = i CPUs working.

qi0 = iµ(1 − c), qi,i−1 = iµc, 2 ≤ i ≤ 5, q10 = µ.

6.19 Let Xi (t) be the number of customers of type i in the system at time t.
{(X1 (t), X2 (t)), t ≥ 0} is a CTMC on S = {(i, j) : i ≥ 0, 0 ≤ j ≤ s} with
transition rates
q(i,j),(i+1,j) = λ1 , (i, j) ∈ S, q(i,j),(i,j+1) = λ2 , (i, j) ∈ S, j < s,
q(i,j),(i−1,j) = min(i, s − j)µ1 , q(i,j),(i,j−1) = jµ2 , (i, j) ∈ S.
524 ANSWERS TO SELECTED PROBLEMS
6.21 Y (t) = the number of packets in the buffer, Z(t) = the number of tokens in
the token pool at time t. X(t) = M − Z(t) + Y (t). {X(t), t ≥ 0} is a birth and
death process with birth rates λi = λ, i ≥ 0 and death rates µ0 = 0, µi = µ, i ≥ 1.

6.23 q2,1 = µ, qi,i−1 = 2µ, i ≥ 3,

qi,i+1 = 2λ, qi,iT = 2θ, i ≥ 2,
q1T,1 = µ, qiT,(i−1)T = µ, i ≥ 2,
qiT,T iT = θ, i ≥ 1.
6.25 {Xk (t), t ≥ 0} k = 1, 2 are two independent is a birth and death processes on
{0, 1, · · · , } with birth parameters λi = λpk for i ≥ 0, and death parameters µk for
i ≥ 1.
6.27 q0,k = λk , qk,0 = µk 1 ≤ k ≤ K.
6.29 State space = {0, 1, 2}. Transition rates:
q0,1 = 2λ, q1,0 = µ, q1,2 = 2λ, q2,0 = 2µ.

6.31 λi = λ1 + λ2 for 0 ≤ i ≤ K − 1, λi = λ1 for i ≥ K, µi = µ for i ≥ 1.

6.33 Denote a space as E if it is empty, B if it is occupied by a car in service, and W
if it is occupied by a car that is waiting to begin service or has finished service. The
state space is S = {1 = EEE, 2 = BEE, 3 = BBE, 4 = EBE, 5 = BBW, 6 =
BW E, 7 = EBW, 8 = BW W }. The rate matrix is:
 
−λ λ 0 0 0 0 0 0
 µ −(λ + µ) λ 0 0 0 0 0 
 
 0 0 −(λ + 2µ) µ λ µ 0 0 
 
 µ 0 0 −(λ + µ) 0 0 λ 0 
Q=  0
.
 0 0 0 −2µ 0 µ µ 
 µ 0 0 0 0 −(λ + µ) 0 λ 
 
 0 µ 0 0 0 0 −µ 0 
0 µ 0 0 0 0 0 −µ

COMPUTATIONAL EXERCISES

α
6.1 α+β (1 − e−(α+β)t ).
λi µi −(λi +µi )t
6.3 Let αi (t) = λi +µi + λi +µi e . Then
P(X(t) = 0|X(0) = 2) = (1 − α1 (t))(1 − α2 (t)),
P(X(t) = 1|X(0) = 2) = α1 (t)(1 − α2 (t)) + α2 (t)(1 − α1 (t)),
P(X(t) = 2|X(0) = 2) = α1 (t)α2 (t).
ANSWERS TO SELECTED PROBLEMS 525
6.5 P (X(t) = j|X(0) = i) = ji e−jµt (1 − e−µt )i−j .

2αβ α(α−β)
6.7 α+β t + (α+β)2 (1 − e−(α+β)t ).

λ(e−16µ −e−(8λ+24µ) )
6.9 (λ+µ)(1−e−(8λ+24µ) )
.

6.11 Stable iff λ < 5µ.

 λ i

 (µ) for 0 ≤ i ≤ 5
1 λ i
 ( 2i−5 µ ) for 6 ≤ i ≤ 8



ρi = ( 8·31i−8 µλ )i for 9 ≤ i ≤ 12
1 λ i
 ( 648·4i−12 µ ) for 13 ≤ i ≤ 15



( 41472·5i−15 µλ )i
1

for i ≥ 16.


pi = ρi p0 , i ≥ 0.
λi

6.13 p0 = P1n λi , pi = Pµni λi , i = 1, 2, ..., n.

1+ i=1 µi
1+ i=1 µi

 λ
 µp
1−p
6.15 G(z) = 1−pz .
Pk−1 i Pk−2 i
ρ /i! ρ /i!
6.19 Long run probability that the kth space is occupied = ρ k i −ρ i=0
P i=0
P k−1
i
.
i=0
ρ /i! i=0
ρ /i!
ρK /K!
Long run fraction of the customers lost = PK .
i=0
ρi /i!

6.21 The system is stable if α > 0. State R = computer under repair. State j =
computer is functioning and there are j jobs in the system. The limiting distribution:
θ
pR = θ+α
α
pj = θ+α (1 − b)bj , j ≥ 0
where q
λ θ λ
1+ µ + µ − (1 + µ + µθ )2 − 4 µλ
b= .
2
Long run fraction of job that are completed successfully = λµ (1 − pR − p0 ).

6.23 p0 = (1 + K λk −1 λk
P
k=1 µk ) , pk = µk p0 , 1 ≤ k ≤ K.

6.25 Let αi = λi /µ, i = 1, 2

ρi = (α1 + α2 )i , 0 ≤ i ≤ K,
ρi = (α1 + α2 )K αi−K
1 , i > K.
The system is stable if λ1 < µ. The limiting distribution is given by
∞
X
p j = ρj / ρi , j ≥ 0.
i=0
526 ANSWERS TO SELECTED PROBLEMS
3ρ
6.27 Condition of stability: λ < µ. Expected number in the system = 4(1−ρ) + ρ/4.

2λµ2 (µ+θ+2λ)
6.29 Idle time fraction = 8λ4 +4λ3 θ+8λ3 µ+6λ2 µθ+4λµ2 θ+2λµ3 +µ3 θ+4λ2 µ2 .

6.31 pi = 1/N, 1 ≤ i ≤ N.
2λ+µ
6.35 λ(4λ+µ) .

1 1−(λ/µ)K+1
6.39 λ 1−λ/µ .

6.43 Optimal λ = .065.

kr(α+λ)
6.45 α(α+λ+µ) .

r 1−c5
6.47 µ 1−c .

6.49 T = .74 approximately.

√ √
6.51 Optimal p = √ 1+ρ√ . Fraction joining = 1/ 1 + ρ.
1+ρ+ ρ

Use solution to Computational Exercise 6.25. The rate of revenue is λ1 c1 +

6.53 P
λ2 c2 K−1
j=0 pj .

6.55 p(φ) = [1 + 3 µλ + 3( λµ )2 + ( λµ )3 + µθ ]−1 .

6.57 Yes. pi ∝ di /qi where di is the number of neighbors of node i.

CONCEPTUAL EXERCISES

6.1 Let B be the submatrix of Q obtained by deleting row and column for the state
N . Then the matrix M = [Mi,j ] satisfies BM = −I.

6.3 Let mk = E(T |X(0) = k). Then

X X
qkn mn = −1, k 6= i, qin mn = −1.
n∈S n∈S−{j}

′
6.9 Let {Y (t), t ≥ 0} have transition rates qi,j = qi,j /r(i).

6.11
e−sx/ri

−sT if y > x/ri
E(e |X(0) = i, S1 = y, X(y) = j) =
e−sy φj (s, x − yri ) if y ≤ x/ri .
The result follows by unconditioning.
ANSWERS TO SELECTED PROBLEMS 527
Chapter 7

MODELING EXERCISES

7.1 X(t) = number of waiting customers at time t. {X(t), t ≥ 0} is the queue length
process in an M/M/1 queue with arrival rate λ and service rate µ.

7.3 λi = λ, i ≥ 0, µi = µ, 1 ≤ i ≤ 3, 2µ, 4 ≤ i ≤ 9, 3µ, i ≥ 10.

7.5 λi = λ, i ≥ 0, µi = µ1 , 1 ≤ i ≤ 3, µ2 , i ≥ 4.

7.7 Arrival process is P(λ) where λ = ki=1 λi . Service times are iid with common
P
Pk
cdf i=1 (λi /λ)(1 − e−µi x ).

7.9 Interarrival times are deterministic (equal to 1), and the service times are iid
exp(µ).

7.13 {X(t), t ≥ 0} not a CTMC. {Xn , n ≥ 0} is a DTMC of the type given in

Computational Exercise 4.24.
λ(λ+s)
7.15 (λ+s)(λ+µ+s)−λµ .

COMPUTATIONAL EXERCISES

1 ρ
7.3 W q = µ 1−ρ .

i
7.5 1−ρ .

7.9 PK from Equation 7.19.

7.11 L/(λ(1 − pK )).

7.13 λa(1 − pK ) − cL.

7.23 1 − ρ.

7.27 M/M/1 queue with ρ = λ/αµ.

7.31 See answer to Computational Exercise 6.25. An arriving customer of either

type, and an entering customer of type 1, sees j people with probability pj , j ≥ 0.
An entering customer of type 2 see j people with probability pj / K−1
P
k=0 pk , 0 ≤ j ≤
K − 1.
 
7.33 1. N λ
2. N λ
P Q
i=1 (1−p)µi −λ , i=1 1 − (1−p)µi .
528 ANSWERS TO SELECTED PROBLEMS
n

(1/µ) if 1 ≤ n ≤ 5,
7.37 φ1 (n) =
(1/µ)5 (1/2µ)n−5 if n ≥ 6,

 (1/µ)n if 1 ≤ n ≤ 2,
φ2 (n) = (1/µ)2 (1/2µ)n−2 if 3 ≤ n ≤ 10,
(1/µ)2 (1/2µ)8 (1/3µ)n−10 if n ≥ 11.

The network is always stable.
∞ n
X λn X
L= n φ1 (n1 )φ2 (n2 ).
n=0
n! n1 +n2 =0

7.39 a1 = λ1 + aN pN , ai = λi + ai−1 pi−1 , 2 ≤ i ≤ N.

Solution:  
N
X N
Y −1 YN
aN =  λi pj  /(1 − pj ).
i=1 j=i j=1

Other ai ’s are obtained by symmetry.

7.45 Condition of stability: ρ = ki=1 λi

P
µi < 1. Expected number in steady state
λ
Pk λi
L = ρ + 1−ρ i=1 µ2 .
i

7.47 .9562.

7.53 Condition of stability: λ < 2µ. The second system has smaller L.

7.59 X(t) and Xn have the same limiting distribution, which differs from that of
X̄n .
ρ
7.63 Stability condition: 1+ρ < θ. The solution to Equation 7.41:
1 p
α= ((2λ + µ + θ) − (2λ + µ + θ)2 − 4(θ + λ)).
2
Use Theorem 7.16.

Chapter 8

COMPUTATIONAL EXERCISES

8.1 {Sn , n ≥ 0} is a renewal sequence in the M/M/1/K and G/M/1/K system,

but not in the M/G/1/K system.

8.3 Recurrent.
2k 2k+1
8.5 P{N (t) = k} = e−λt (λt)
(2k)! + e
−λt (λt)
(2k+1)! .
ANSWERS TO SELECTED PROBLEMS 529
8.7 P (N (n) = k) = nk (1 − α)k αn−k .

8.9 pk (n) = .2pk−1 (n) + .3pk−1 (n − 1) + .5pk−1 (n − 2), k ≥ 1, n ≥ 2.

8.11 1. 6/17, 2. 4/17, 3. 2/17.

8.13 τ = m0 = 1+ 1−α 2 2 2 3
1−β , s = 2+α−β+2/(1−β), N (n) ∼ N (t/τ, (s −τ )t/τ ).

λ1 λ2 r(1−r)(λ1 −λ2 )2
8.15 M (t) = (1−r)λ1 +rλ2 t + ((1−r)λ1 +rλ2 )2 (1 − e−((1−r)λ1 +rλ2 )t ).
λµ λµ
8.17 M (t) = λ+µ t − (λ+µ)2 (1 − e−(λ+µ)t ).

8.19 M ∗ (t) = pM (t).


Rt 1 − G(t) t≤x
8.21 H(t) = D(t) + 0 H(t − u)dG(u) where D(t) = .
0 otherwise
Rx
limt→∞ H(t) = τ1 0 (1 − G(t))dt.
Rt
8.23 p(t) = 1 − 0 p(t − x)dG(x). Not a renewal equation. For a PP(λ) p(t) =
1 −2λt
2 (1 − e ).
h ik
G̃(s)
8.25 M̃k (s) = k! 1−G̃(s)
.
Rt R∞
8.27 H(t) = D(t) + 0 H(t − u) dG(u) where D(t) = t uk dG(u).
E(X k+1 )
limt→∞ E(C(t)k ) = τ .
τ
8.31 1−ρ .

8.33 µi /µ.
q
8.37 a min(1, C
C1 [ 1 +
2 2C1
C2 − 1]).

8.39 1. SMP, 2. Not an SMP, 3. Not an SMP.

8.41 Let X(t) = −1 if the machine is under repair at time t. If the machine is up
at time t, let X(t) be the cumulative damage at time t (since the last repair). Then
{X(t), t ≥ 0} is an SMP with state space {−1, 0, 1, 2, · · · , K}. The non-zero entries
of the kernel G(x) = [Gij (x)] are given by
G−1,0 (x) = A(x),
Gi,j (x) = (1 − exp(−λx))αj−i , 0 ≤ i < j ≤ K,
X∞
Gi,0 (x) = (1 − exp(−λx)) αj , 0 ≤ i ≤ K.
j=K+1−i

τ
8.45 M/G/1/1 : 1+λτ , G/M/1/1 : G̃(µ).
530 ANSWERS TO SELECTED PROBLEMS
8.47 Let αi = (λ2i + 2λi µi )/(λi + µi )2 , ρj := j−1
Q P4
i=1 αi ,, πj = ρj / i=1 ρi ,
τj = (λi + 2µi )/(λi + µi )2 . Then
4
X
pj = ρj τj / ρi τi .
i=1

8.51 3. Space 6 is occupied for 76.24% of the time in steady state.

CONCEPTUAL EXERCISES

8.1 Yes if X0 = i with probability 1 for some fixed i, and no otherwise.

8.3 N1 is an RP, N2 is a delayed RP.
8.13 N1 is an ARP, N0 is a delayed ARP.

Chapter 9

MODELING EXERCISES

9.1 Let pij = j+1 λr −(λi +µi )x

Q
r=i+1 λr +µr , j ≥ i − 1, Ai (x) = 1 − e . Then Gij (x) =
pij · Ai ∗ Ai+1 ∗ · · · ∗ Aj ∗ Aj+1 (x).
9.3 Gi,j (x) is as given in Example 9.4 for 0 ≤ j ≤ i + 1 < K. For i = K, 0 ≤ j ≤
K, GK,j = GK−1,j .
R x N −i

9.5 Gij (x) = 0 j+1−i (1 − e−µt )j+1−i e−(N −j−1)µt dG(t), N > j ≥ i − 1 ≥ 0.
Rx −N µy
G0j (x) = 0 N µe G1j (x − y)dy, N > j ≥ 0.
9.7 A = [hij ], 0 ≤ i, j < k, B = [hij ], 0 ≤ i < k ≤ j ≤ N, C = [hij ], 0 ≤
j < k ≤ i ≤ N. Then Gnij = [An−2 BC]ij , n ≥ 2.
9.9 Use Sn = n and Xn = X(Sn +).
9.11 Use Sn time of departure of the nth server, and Xn = X(Sn +).

COMPUTATIONAL EXERCISES

9.1

Hi ∗ Hi+1 ∗ · · · ∗ Hj−2 ∗ Hj−1 (t) 1≤i<j≤N
Fi,j (t) =
Hi ∗ Hi+1 ∗ · · · ∗ HN ∗ H1 ∗ · · · ∗ Hj−2 ∗ Hj−1 (t) 1 ≤ j ≤ i ≤ N
F̃i,j (s)
Mi,j (s) = .
1 − F̃i,i (s)
ANSWERS TO SELECTED PROBLEMS 531
2(1−Ã(µ))
9.3 p1 = 2µτ +Ã(µ)
.
 
ρj+1 e−ρ
9.5 (1). (j+1)! 1−e−ρ , (2). (1 − e−ρ )/ρ.
h i
1
1 − e−λ r−j k
P
9.7 Long run fraction of the time the MRGP spends in state j = λ k=0 (λ /k!) .

λτ λτ λs2
9.11 limn→∞ E(Xn ) = α , limt→∞ E(X(t)) = α + 2τ .

CONCEPTUAL EXERCISES

9.7 Follows from the elementary renewal theorem for the renewal processes (Theo-
rem 8.12) and for the delayed renewal processes (Conceptual Exercise 8.12).

Chapter 10

COMPUTATIONAL EXERCISES

10.1 For 0 < s < t,

E(B(s)B(t)) = E(B(s)(B(t)−B(s)+B(s))) = E(B(s)(B(t)−B(s)))+E(B(s)2 ).

10.3
1
f (x1 , x2 ) = p ×
2π t1 (t2 − t1 )
 
1 2 2
exp − [(x1 − µ1 t1 ) t2 − 2(x1 − µ1 t1 )(x2 − µ2 t2 ) + (x2 − µ2 t2 ) t1 ] .
2t1 (t2 − t1 )

10.5 N(sy/t, σ 2 s(t − s)/t).

10.13 a(t) = −3µt, b(t) = 3(µt)2 − 3σ 2 t, c(t) = 3σ 2 µt − (µt)3 .

10.19 Limiting mean = 0, and limiting variance = b2 /6.

10.25 Y (t) = (B(t) + Y (0))/(1 + t).

CONCEPTUAL EXERCISES

10.1 1. True, 2. False, 3. True.

Rt
C(t)+b(t)B(t)− B(u)b′ (u)du
10.10 Y (t) = Y (0)eC(t)+X(t) = Y (0)e 0 ,
Rt
where C(t) = 0 (a(u) − b2 (u)/2)du.
 References

Assmussen, S. (1987). Applied Probability and Queues, Wiley, NY.

Bartholomew, D. J. (1982). Stochastic Models for Social Processes, Wiley, NY.

Bartholomew, D. J. (1991). Statistical Techniques for Manpower Planning, Wiley,

NY.

Bartlett, M.S. (1978). An Introduction to Stochastic Processes, Cambridge University

Press, Cambridge, UK.

Beard, R. E., T. Pentikeinen, and E. Pesonen (1984). Risk Theory: The Stochastic
Basis of Insurance, Chapman & Hall, London.

Bertsekas, D., and R. Gallager (1987). Data Networks, Prentice Hall, Englewood
Cliffs, NJ.

Billingsley, P. (1968). Convergence of Probability Measures, Wiley, NY.

Burghes, D. N., and A. D. Wood (1980). Mathematical Models in Social, Manage-

ment, and Life Sciences, Halstead Press, NY.

Borodin, A. and P. Salminen (1996). Handbook of Brownian Motion - Facts and

Formulae, Birkhauser, Boston.

Buzzacott, J. A., and J. G. Shanthikumar (1993). Stochastic Models of Manufacturing

Systems, Prentice Hall, Englewood Cliffs, NJ.

Chaudhry, M. I., and J. G. C. Templeton (1983). A First Course in Bulk Queues,

Wiley, NY.

Chung, K. I. (1967). Markov Chains with Stationary Transition Probabilities,

Springer-Verlag, NY.

Cinlar, E. (1975). Introduction to Stochastic Processes, Prentice Hall, Englewood

Cliffs, NJ.

Cooper, R. E. (1981). Introduction to Queueing Theory, North Holland, NY.

Cox, D. R. (1962). Renewal Theory, Metheun, London.

533
534 REFERENCES
Cox, D. R. (1980). Point Processes, Chapman & Hall, London.

Cox, D. R., and H. D. Miller (1965). The Theory of Stochastic Processes, Chapman
& Hall, London.

Cox, D. R., and W. L. Smith (1961). Queues, Metheun, London.

Daigle, J. N. (2005). Queueing Theory with Applications to Packet Telecommunica-

tions, Springer-Verlag, NY.

Durbin, R., S. Eddy, A. Krogh, and G. Mitchison (2001). Biological Sequence Analy-
sis: Probabilistic Models of Proteins and Nucleic Acids, Cambridge University Press,
Cambridge, UK.

Durett, R. (2001). Essentials of Stochastic Processes, Springer-Verlag, NY.

Durett, R. (1996). Stochastic Calculus: A Practical Introduction, CRC Press, Boca

Raton, FL.

Elmaghraby, S. E. (1977). Activity Networks: Project Planning and Control by Net-

work Models, Wiley, NY.

El-Taha, M. and S. Stidham (1998). Sample Path Analysis of Queueing Systems,

Springer-Verlag, NY.

Feller, W. (1971). An Introduction to Probability Theory and Its Applications, Vols.

I and II, Wiley, NY.

Fuller, L. E. (1962). Basic Matrix Theory, Prentice Hall, Englewood Cliffs, NJ.

Gallager, R. G. (1996). Discrete Stochastic Processes, Kluwer Academic Publishers,

The Netherlands.

Gantmacher, F. R. (1960). Matrix Theory, Vols. I and II, Chelsea Publishing, NY.

Gelenbe, E. and G. Pujolle (1987). Introduction to Queueing Networks, Wiley, NY.

Goel, S. N. and N. Richter-Dyn (1974). Stochastic Models in Biology, Academic

Press, NY.

Golub, V. K. and C. F. van Loan (1983). Matrix Computations, Johns Hopkins Uni-
versity Press, Baltimore, MD.

Gradshteyn, I. S. and I. M. Ryzhik (1990). Table of Integrals, Series, and Products,

Academic Press, NY.

Griffiths, A. J. F., J. H. Miller, D. T. Suzuki, R. C. Lewontin, and W. M. Gelbert

(1993). An Introduction to Genetic Analysis, W. H. Freeman and Co., NY.

Grinold, R. C. and K. T. Marshall (1977). Manpower Planning Models, North-

Holland, NY.
REFERENCES 535
Gross, D. and C. M. Harris (1985). Fundamentals of Queueing Theory, Wiley, NY.

Heyman, D. P. and M. J. Sobel (1982). Stochastic Models in Operations Research,

Vols. I and II, McGraw-Hill, NY.

Hoel, P. G., S. C. Port, and C. J. Stone (1972). Introduction to Stochastic Process,

Houghton-Mifflin, Boston, MA.

Horn, R. A. and C. R. Johnson (1990). Matrix Analysis, Cambridge University Press,

Cambridge, UK.

Hull, J. C. (1997). Options, Futures, and Other Derivatives, Prentice Hall, Engle-
wood Cliffs, NJ.

Jagers, P. (1975). Branching Processes with Biological Applications, Wiley, NY.

Jaiswal, N. K. (1968). Priority Queues, Academis Press, NY.

Karatzas, I. and S. E. Shreve (1991). Brownian Motion and Stochastic Calculus,

Springer-Verlag, NY.

Karlin, S. and H. Taylor (1975). A First Course in Stochastic Processes, Academic

Press, NY.

Karlin, S. and H. Taylor (1981). A Second Course in Stochastic Processes, Academic

Press, NY.

Kelly, F. P. (1979). Reversibility and Stochastic Networks, Wiley, NY.

Kemeny, J. G. and L. J. Snell (1960). Finite Markov Chains, Van Nostrand, Princeton,
NJ.

Kemeny, J. G., L. J. Snell, and A. W. Knapp (1960). Denumerable Markov Chains,

Van Nostrand, Princeton, NJ.

Kempthorne, O. (1969). An Introduction to Genetic Statistics, The Iowa University

Press, IA.

Kingman, J. F. C. (1993). Poisson Processes, Oxford University Press, Oxford, UK.

Kingman, J. F. C. (1972). Regenerative Phenomena, Wiley, London.

Kleinrock, L. (1976). Queueing Systems, Vol. I: Theory, Wiley, NY.

Kleinrock, L. (1976). Queueing Systems, Vol. II: Computer Applications, Wiley, NY.

Kohlas, J. (1982). Stochastic Methods of Operations Research, Cambridge Univer-

sity Press, Cambridge, UK.

Koski, T. (2001). Hidden Markov Models of Bioinformatics, Kluwer Academic Pub-

lishers, The Netherlands.
536 REFERENCES
Kreyszig, E. (1973). Advanced Engineering Mathematics, Wiley, NY.

Malliaris, A.g. (1982). Stochastic Methods in Economics and Finance, North-

Holland.

Marsden, J. E. (1974). Elementary Classical Analysis, W. H. Freeman and Co., San

Francisco.

Moran, P. A. P. (1962). Statistical Processes of Evolutionary Theory, Oxford Univer-

sity Press, UK.

Neuts, M. F. (1981). Matrix Geometric Solutions in Stochastic Models: An Algorith-

mic Approach, Johns Hopkins University Press, Baltimore, MD.

Neuts, M. F. (1989). Structured Stochastic Matrices of M/G/1 Type and Their Appli-
cations, Marcel Dekker, NY.

Nelson, R. (1995). Probability, Stochastic Processes and Queueing Theory, Springer-

Verlag, NY.

Prabhu, N. U. (1985). Queues and Inventories, Wiley, NY.

Prabhu, N. U. (1980). Stochastic Storage Processes: Queues Insurance Risk and

Dams, Springer-Verlag, NY.

Oksendal, B. (1995). Stochastic Differential Equations, Springer-Verlag, NY.

Resnick, S. I. (1992) Adventures in Stochastic Processes, Birkhauser, Boston.

Rolski, T., H. Schmidli, V. Schmidt, and J. Teugels (1999). Stochastic Processes for
Insurance and Finance, Wiley, NY.

Ross, S. M. (1983). Stochastic Processes, Wiley, NY.

Ross, S. M. (2002). An Elementary Introduction to Mathematical Finance: Options

and Other Topics, Cambridge University Press, Cambridge, UK.

Saaty, T. L. (1961). Elements of Queueing Theory, McGraw-Hill, NY.

Saur, C. H. and K. M. Chandy (1981). Computer Systems Performance Modeling,

Prentice Hall, Englewood Cliffs, NJ.

Seneta, E. (1981). Non-negative Matrices and Markov Chains, Springer-Verlag, NY.

Serfozo, R. (1999). Introduction to Stochastic Networks, Springer-Verlag, NY.

Shreve, S. E. (2000). Stochastic Calculus for Finance I: The Binomial Asset Pricing
Model, Springer-Verlag, NY.

Shreve, S. E. (2000). Stochastic Calculus for Finance II: Continuous-Time Models,

Springer-Verlag, NY.
REFERENCES 537
Spitzer, F. (1964). Priciples of Random Walk, Van Nostrand, NJ.

Steele, J. M. (2001). Stochastic Calculus and Financial Applications, Springer-

Verlag, NY.

Stoyan, D. (1983). Comparison Methods for Queues and Other Stochastic Mod-
els,Wiley, NY.

Syski, R. (1992). Passage Times for Markov Chains, IOS Press, The Netherlands.

Takacs, L. (1962). Introduction to the Theory of Queues, Oxford University Press,

UK.

Tijms, H. C. (1986). Stochastic Modeling and Analysis: A Computational Approach,

Wiley, NY.

Trivedi, K. S. (1982). Probability and Statistics with Reliability, Queueing, and Com-
puter Science Applications, Prentice Hall, Englewood Cliffs, NJ.

Vajda, S. (1978). Mathematics of Manpower Planning, Wiley, NY.

Varga, R. S. (1962). Matrix Iterative Analysis, Prentice Hall, Englewood Cliffs, NJ.

Walrand, J. (1988). An Introductions to Queueing Networks, Prentice Hall, Engle-

wood Cliffs, NJ.

Whittle, P. (1986). Systems in Stochastic Equilibrium, Wiley, NY.

Wilmott, P., S. Howison, and J. Dewynne (1995). The Mathematics of Financial

Derivatives: A Student Introduction, Cambridge University Press, Cambridge, UK.

Wolff, R. W. (1989). Stochastic Modeling and the Theory of Queues, Prentice Hall,
Englewood Cliffs, NJ.
 Index

Absorption Probability Average Costs, 251

CTMC, 223 Backward Equations, 205
DTMC, 60 Chapman-Kolmogorov Equations, 203
Age Process, 363 Communicating Class, 229
Definition, 190
Discounted Costs, 249
Bayes’ Rule, 492
First Passage Times, 221
Birth and Death Process, 200, 242, 256
CDF, 222
Pure Birth, 198, 217
LST, 225
Pure Death, 199, 218
Moments, 224
Black Scholes’ Formula, 482
Forward Equations, 205
Borel-Cantelli lemma, 492
Generator Matrix, 195
Bounded Convergence Theorem, 511
Eigenvalues, 210
Branching Process, 26, 35, 64
Irreducibility, 229
Brand Switching Model, 18, 129
Laplace Transforms, 212
Brownian Motion
Limiting Behavior, 233
Constrained on [a, b], 461
Marginal Distribution, 203
Definition, 445
Occupancy Times, 207
Discounted Cost, 466, 468
Rate Diagram, 195
Exponential Martingale, 464
Recurrence, 230
First Passage Time, 454
Transience, 230
Geometric, 478
Transition Probability Matrix, 192
Joint Distribution, 447
Uniformization, 214
Kolmogorov Equation, 451, 452
Convergence
Limiting Distribution, 462
Almost Sure, 509
Linear Martingale, 463
in Distribution, 509
Reflected, 461
in Mean, 509
Sample Path Properties, 448
in Mean Square, 509
Standard, 446
in Probability, 509
First Passage Time, 452
Costs and Rewards, 8
Maximum and Minimum, 453
Quadratic Martingale, 464
Quadratic Variation, 471 Database Locking Model, 256
Reflected, 458 Diagonalization, 38
Total Variation, 471 Difference Equations, 513
Differential Equations, 514
Diffusion Process, 476
Cambell’s Theorem, 163
Function of, 476
Central Limit Theorem, 510
Discrete Time Markov Chain
Clinical Trials, 14
Balance Equation, 113
Continuous Time Markov Chain, 189, 205
Chapman-Kolmogorov Equations, 32

539
540 INDEX
Characterization, 11 First Passage Times, 8
Communicating Class, 90 Foster’s Criterion, 103
Communication, 89
Definition, 10
G/M/1 Queue, 437
First Passage Times, 55
Gambler’s Ruin, 17, 61
CDF, 56
Generating Functions, 503
Expectation, 69
Generating Functions, 74
Moments, 74 Hessenberg Matrix
Higher Order, 23 Lower, 21
in Finance, 27 Upper, 20
in Genealogy, 25 Hidden Markov Models, 22
in Genetics, 23
in Genomics, 22 Infinite Server Queue, 173, 201, 243, 324
in Manpower Planning, 28 Inspection Paradox, 367
in Telecommunications, 30 Ito Integral, 472
Initial Distribution, 11 Ito Process, 476
Irreducibility, 89, 91 Ito’s Formula, 478
Limiting Behavior, 85
Limiting Distribution, 113
Marginal Distributions, 31 Laplace Stieltjes Transform, 505
Occupancy Distribution, 116 Laplace Transform, 507
Occupancy Times, 36 Law of Large Numbers
Periodicity, 92 Strong, 510
Recurrence, 94 Weak, 510
Stationary Distribution, 113 Limiting Distribution, 7
Steady State distribution, 113 Linear Growth Model, 201, 219, 228, 248
Time Homogeneous, 10 Little’s Law, 396
Transience, 94
Transient Behavior, 9 M/G/∞ queue Queue
Transition Diagram, 12 Busy Period, 358
Discrete-Time Queue, 17 M/G/1 Queue, 435
Average Costs, 128 M/M/∞ Queue, 296
Batch Arrivals, 19, 67, 72, 101, 120 Markov Property, 10
Batch Service, 20, 68, 72, 102, 121 Markov Regenerative Process, 409
Discounted Costs, 126 Transient Distribution, 428
Drunkard’s Walk, 17 Markov Regenerative process
Limiting Distribution, 429
Ehrenfest Model, 18 Markov Renewal Equation, 415
Excess Life Process, 363 Markov Renewal Function, 415
Exponential Random Variable, 145 Markov Renewal Process
Distribution, 145 Definition, 414
Hazard Rate, 147 Markov Renewal Sequence
Memoryless Property, 146 Characterization, 410
Minimum, 149 Definition, 409
Sums, 152, 153 Regularity, 414
Markov Renewal Theorem
Extended, 421
Fatou’s Lemma, 511 Key, 419
INDEX 541
Markov Renewal Type Equation, 417 Departures, 281
Solution, 417 Entries, 280
Martingale, 463 Little’s Law, 291
Matrix Exponential, 209 Nomenclature, 277
Monotone Convergence Theorem, 511 PASTA, 283, 289
Moran Model, 25
Random Variable
o(h) Functions, 160 CDF, 493
Option Conditional Distributions, 500
American Call, 483 Conditional Expectations, 500
American Perpetual Put, 483 Continuous, 493
American Put, 483 Discrete, 493
European Call, 481 Expectation, 494
European Put, 483 Independence, 498
Optional Sampling Theorem, 465 Multi-Variate, 497
Order Statistics, 500 Multi-Variate
Ornstein-Uhlenbeck process, 481 Normal, 501
Sums, 498
Univariate, 493
Pakes’ Lemma, 104
Random Walk, 6, 16
Parallel System, 151
General, 63, 71, 100, 119, 131
Phase Type Distributions, 252
Simple, 16, 33, 62, 87, 91, 93, 99
Poisson Process, 145, 191, 197, 217
State-Dependent, 16
Characterization, 160, 161
Recurrence Time
Compound, 177, 191
Backward, 363
Covariance, 159
Forward, 363
Definition, 155
Regenerative Process, 390
Distribution, 155
Costs and Rewards, 394
Event Times, 162
Delayed, 390
Non-Homogeneous, 173
Limiting Distribution, 391
Event Times, 177
Renewal Argument, 344
Shifted, 157
Renewal Equation, 350
Splitting, 169
Renewal Function, 349
Bernoulli, 169
Asymptotic Behavior, 361
Non-Homogeneous, 171
Delayed, 368
Superposition, 166
Asymptotic Behavior, 370
Probability
Renewal Process, 339
Axioms, 491
Age, 365, 367
Conditional, 492
Alternating, 373
Model, 491
Delayed, 377
Asymptotic Behavior, 385, 388
Queueing Networks Central Limit Theorem, 347
Closed, 308 Characterization, 342
Open, 298 Definition, 340
State-Dependent Arrivals, 306 Delayed, 368
State-Dependent Service, 306 Limiting Behavior, 369
Queues Equilibrium, 371
Arrivals, 280 Limiting Distribution, 345
Birth and Death, 293, 433 Marginal Distribution, 343
542 INDEX
Moments, 354 Transient Distribution, 7
Periodicity, 359 Transition Probability Matrix
Recurrence and Transience, 346 n-Step, 32
Remaining Life, 364, 366 Eigenvalues, 39
Total Life, 367 Generating Function, 41
Renewal Reward Process, 384 One Step, 10
Renewal Sequence, 340
Renewal Theorem
Weather Model, 13
Almost-Sure, 346
Wright-Fisher Model, 24
Blackwell’s, 360
Continuous, 234
Discrete, 106 Yule Process, 198
Elementary, 352 with Immigration, 199
Key, 360
Renewal-Type Equation, 355
Solution, 355
Restaurant Process, 258
Retrial Queue, 202, 243, 320, 438
Reversibility
CTMC, 254
DTMC, 129
Rumor Model, 48

Sample Path, 2
Semi-Markov Process, 378, 424
First Passage Times, 380
Limiting Behavior, 426
Limiting Distribution, 382, 425
Single Server Queue, 200, 242, 252
Batch Arrivals, 244
Slotted ALOHA, 48
Stationary and Independent Increments, 157
Stochastic Differential Equation, 476
Stochastic Integral, 472
Martingale, 475
Moments, 473
Stochastic Matrix, 10
Stochastic Process, 1
Continuous Time, 1
Discrete Time, 1
Parameter Set, 1
State-Space, 1
Stopping Time, 447
Strong Markov Property, 448
Success Runs, 19, 58, 62, 94, 99, 118, 232

Time Division Multiplexing, 49

Total Life Process, 363