DMS625: Introduction to stochastic processes and their

applications
Sourav Majumdar

Continuous-time Markov chain

October 27, 2024

1 Introduction
So far we have discussed Discrete-time Markov chains (DTMC) with discrete state spaces. We
will now consider a generalization where time will become continuous, however the state space will
continue to stay discrete.
We say X(t), t ∈ [0, ∞), is a continuous-time Markov chain (CTMC) if it satisfies the Markov
property given by,

P(X(t + s) = xt+s |X(s) = xs , X(u) = xu , 0 ≤ u < s) = P(X(t + s) = xt+s |X(s) = xs )

Here xi ∈ S, 0 ≤ i ≤ t + s, are the values X(t) attains on the state space S at each time. This
definition of the Markov property is similar to the one for DTMC that we considered earlier. Since
in a CTMC time is continuous, this has been accounted for here in the conditioning event on the
LHS. The advantages of CTMC are obvious in that now we can model continuously over time. Let
us look at an example.
Example 1.1 (Poisson Process). The Poisson process N (t) with rate λ is a CTMC. We need to
show that N (t) satisfies the Markov property,

P(N (t + s) = xt+s |N (s) = xs , N (u) = xu , 0 ≤ u < s)

=P(N (t + s) = xt+s |N (s) = xs , N (tk ) = xtk , N (tk−1 ) = xtk−1 , . . . , N (t1 ) = xt1 , N (0) = 0)
(For an arbitrary k, consider any sequence of times such that 0 < t1 < t2 < . . . < tk < s)
=P(N (t + s) − N (s) = xt+s − xs |N (s) − N (tk ) = xs − xtk , N (tk ) − N (tk−1 ) = xtk − xtk−1 , . . . , N (t1 ) − N (0) = xt1 − 0)
=P(N (t + s) − N (s) = xt+s − xs )
(Follows from the independent increments property)
=P(N (t + s) = xt+s |N (s) = xs )

Therefore, N (t) is a CTMC.

Example 1.2 (Non-homogeneous Poisson process). Consider a Non-homogeneous Poisson process
N(t) with rate λ(t). One can show that it satisfies the Markov property as above. We note that to show
the Poisson process satisfied the Markov property above, we only required the independent increments
property. Since the non-homogeneous Poisson process possesses the independent increments property,
the proof will follow similarly here.

1
 We say X(t) is a time-homogeneous CTMC, if,

P(X(t + s) = xt+s |X(s) = xs ) = P(X(t) = xt+s |X(0) = xs )

This is similar to the time-homogeneity condition for DTMC, where we say the probability of going
from one state to another only depends on the difference of the times of their transition.

Exercises
1. Verify that the Poisson process N (t) with rate λ is a time-homogeneous CTMC.
2. Verify that the non-homogeneous Poisson process with rate λ(t) need not be a time-homogeneous
CTMC.

Remark 1. All CTMCs henceforth will be assumed to be time-homogeneous CTMCs unless other-
wise stated.

2 Time spent in a state

Let Ti denote the time spent by a CTMC in state i. Ti is also referred to as the sojourn time.
Consider a Poisson process N (t) that is modelling the number of Earthquakes. Let’s say the first
Earthquake occurred on year t = 1, and the second earthquake occurred on year t = 3. Then the
time spent by Poisson process on state i = 1 is Ti = 3 − 1 = 2 years. We want to understand the
distribution of Ti for CTMCs in general.

P(Ti > t + s|Ti > s) = P(X(v) = i, v ∈ (s, s + t]|X(u) = i, i ∈ [0, s])

= P(X(v) = i, v ∈ (0, t]|X(0) = i)
(Follows from time-homogeneity)
= P(Ti > t)

Therefore, Ti follows the memoryless property and hence Ti is exponentially distributed.

Similarly consider,

P(Ti > ti , Tj > tj ) = P(Tj > tj |Ti > ti )P(Ti > ti )

(Without loss of generality assume that the chain visits i before j)
= P(X(v) = j, v ∈ (s, s + tj ]|X(s) = j, X(u) = i, u ∈ (r, r + ti ])P(Ti > ti )
= P(X(v) = j, v ∈ (s, s + tj ]|X(s) = j)P(Ti > ti )
(Markov property)
= P(Tj > tj )P(Ti > ti )

We have shown that Ti , Tj are also independent. We can therefore claim the following result.

Theorem 2.1. In a CTMC, Ti ’s are independent and exponentially distributed, ∀i ∈ S.

Remark 2. Note that even though Ti ’s are exponentially distributed, the rate of the exponential
distribution could vary across i. We will see such examples next.

2
3 Birth and Death process
We will now look at a class of processes known as the Birth and Death processes. These are
continuous-time generalisations of the Birth and Death chains we looked at earlier during our study
of Markov chains. The state space of the Birth and Death process is S = {0, 1, 2, . . .}.
This means that if the CTMC is at state n, it may either move to state n − 1 or state n + 1 with
some probability for each, as in the case of Birth and Death chains.
Suppose the CTMC is at state n, then it could move to state n + 1 after a time Bn ∼ Exp(λn ),
i.e, another unit will be born after a random time Bn which is exponentially distributed with rate
λn . Alternatively, it could move to state n − 1 after a time Dn ∼ Exp(µn ), i.e., a unit will die after
a random time Dn which is exponentially distributed with rate µn . We shall assume that the time
for a birth and death at state n are independent, i.e., Bn and Dn are independent.

Let’s answer a few questions about this system,

• What is the distribution of the time spent by the chain at state n?
Let us denote this as Tn (time spent in a state, that we introduced above). There could be a
birth or a death, therefore the time spent in state n is the time until either a birth or death
happens. This is nothing but,
Tn = min(Bn , Dn )

since Bn , Dn are independent exponential random variables, it follows that Tn ∼ Exp(λn + µn )

(see the section on Exponential distribution, in the Poisson process notes).
• What is the probability that the chain moves to state n + 1?
Let us denote the probability of moving to state n + 1 from state n as pn,n+1 . If the chain
moves to state n + 1 from n, this implies that a birth happened before death. Therefore,
λn
pn,n+1 = P(Bn < Dn ) =
λn + µn
By similar logic,
µn
pn,n−1 = P(Dn < Bn ) =
λn + µn
Suppose you record observations from a Birth and Death process such as the one we described
above. Each time a transition happens we record the state, say one such realization could be

3
0, 1, 2, 1, 2, 3, 2, where the chain started at 0, followed by a birth to 1, then 2, then a death to 1,
followed by births to 2 and 3, and again death to 2. The chain naturally would have spent some time
at each of these states but we choose to ignore these times for the moment. Then the transitions
from each state to another is called the embedded Markov chain with transition matrix for this
embedded Markov chain given by pi,j as defined above.
Remark 3. The idea of the embedded Markov chain can be applied to CTMCs in general. What is
implied is that each CTMC can be thought of as a Markov chain (the embedded Markov chain) along
with some random time for transition between the states.
So where do these Birth and Death processes arise?
Example 3.1 (M/M/1 Queue). Consider a shop with a clerk and customers line up at the shop
to get their requirements serviced. Customers arrive at a rate of λn and customers are serviced by
the clerk at a rate of µn . Here n denotes the number of customers in the system, i.e., the customer
getting serviced and those waiting in the queue to get serviced.
Note here that the arrival rate, λn , is state dependent (depends on n). This in application means
that if a customer were to see a queue with 100 people, they would be less likely to join it, than
if there were 5 people in the queue (given that customers generally prefer to wait less). A similar
interpretation can be made for the service rate being state dependent.
This is the simplest kind of queue, referred to in the queuing notation (also referred to as the
Kendall notation) as M/M/1 queue. The first “M” denotes that the arrival rate has the Memoryless
or Markovian property. The second “M” denotes that the service rate has the Memoryless or Marko-
vian property. The third “1” denotes the number of servers in the system. Several other choices and
parameters can be used to describe more general queues, but we shall not cover them in this course.
Example 3.2 (M/M/s Queue). A simple extension of the M/M/1 queue is one with s servers
instead of just 1. Customers arrive at rate λn . Suppose each of the s servers can service with rate
µ. Then convince yourself that, (
nµ 1 ≤ n ≤ s
µn =
sµ n > s
Calculate pn,n+1 and pn,n−1 .
Example 3.3 (Pure Birth process). Suppose µn = 0, ∀n. Then this is called a pure birth process.
The Birth and Death processes are a general framework for modelling in continuous-time. They
can be applied to model queues as considered above, stock prices, limit order books, epidemics and
many more phenomena.
Let Tii+1 denote the time taken by the Birth and Death process to go from state i to i + 1. We
want to understand E[Tii+1 ]. The idea behind the subsequent calculations is the following. There
are two possible cases,
• There is a direct transition from state i to i + 1.
• There is a transition first from i to i − 1, in which case the CTMC would first have to return
to i and then go to i + 1.
Define a random variable Ii ,
(
1 i→i+1
Ii =
0 i→i−1
Ii takes the value 1, if the chain first moves from i to i + 1, and 0 if it moves to i − 1. Note also that
P(Ii = 1) = pi,i+1 and P(Ii = 0) = pi,i−1 . Therefore,

4
 • E[Tii+1 |Ii = 1] = E[Ti ] = 1
λi +µi

• E[Tii+1 |Ii = 0] = E[Ti ] + E[Ti−1

i
] + E[Tii+1 ] = 1
λi +µi
i
+ E[Ti−1 ] + E[Tii+1 ]
And,
E[Tii+1 ] = P(Ii = 1)E[Tii+1 |Ii = 1] + P(Ii = 0)E[Tii+1 |Ii = 0]
Also, note that E[T01 ] = λ10 .
Therefore, you can show that (Verify!) for λn = λ and µn = µ ∀n,

 1−(µ/λ)i+1 λ =
̸ µ
i+1 λ−µ
E[Ti ] = (i+1)

λ λ=µ

Transition Probability Function

The transition probability function of a CTMC is defined as,

Pi,j (t) = P(X(t + s) = j|X(s) = i)

P
Note that j Pij (t) = 1.
The transition probability function can be thought of as the analogous quantity for a CTMC to
the n−step transition probability function, P(n) (i, j) that we came across while studying Markov
chains.

Consider a pure birth process defined as above. If λ1 = λ2 = . . . = λ, then the pure birth process
is a Poisson process (Verify!).
The transition probability function if X(t) is a Poisson process can be calculated,

P(X(t + s) = j|X(s) = i) = P(X(t + s) − X(s) = j − i|X(s) = i)

= P(X(t + s) − X(s) = j − i)
(From the independent increments property)
e−λt (λt)(j−i)
=
(j − i)!

Now let us consider the general case of a pure birth process, where λ′i s need not all be identical.
It is easy to see that for a pure birth process,

Ti ∼ Exp(λi )

5
 We now want to understand the transition probability function of a pure birth process.

Pi,j (t)
=P(X(t + s) = j|X(s) = i)
=P(X(t + s) < j + 1|X(s) = i) − P(X(t + s) < j|X(s) = i)
=P(Ti + Ti+1 + . . . + Tj > t) − P(Ti + Ti i + 1 + . . . + Tj−1 > t)

We know Ti ’s are independent, but are not identically distributed since λi ’s could be different.
The sum of exponential distribution with non-identical parameters follows what is called as the Hypo-
exponential distribution. We shall consider a more general approach towards studying transition
probability function and return to this question for the pure birth process.
Remark 4. Note that pi,j that we discussed earlier is the probability of the CTMC going to j from
i before any other state. However Pij (t) is the probability of going from i to j in some time t. These
both are different probabilities, don’t confuse them.

Chapman-Kolmogorov Equation
Just like we had the Chapman-Kolmogorov equation for the Markov chain, we have an analogous
result for CTMCs.
P∞
Proposition 3.1. Pi,j (t + s) = k=0 Pi,k (t)Pk,j (s)
Proof. Left as an exercise. The proof follows similarly to that of Chapman-Kolmogorov equation
for the Markov chain.
Recall that if we wanted to calculate the probability of going from state i to j in n steps, i.e.,
P(n) (i, j), we would multiply the transition matrix P(i, j) to itself n times and consider its (i, j) − th
entry. This result was a consequence of the Chapman-Kolmogorov equation for Markov chain.
For CTMCs, unfortunately there is no such simple way for evaluating Pi,j (t). However, like the
Markov chain we will use the Chapman-Kolmogorov equation to study the transition probability
function for CTMCs.

Instantaneous transition rates

We define the instantaneous transition rate of a CTMC as,

qi,j = vi pi,j

Here vi is the rate parameter of the distribution of Ti , i.e.,

Ti ∼ Exp(vi )

It will be shortly clear why qi,j is referred as the instantaneous transition rate.

Example 3.4. 1. Poisson process

Ti ∼ Exp(λ), therefore, vi = λi and pi,i+1 = 1 (Why?). Therefore,

qi,i+1 = λ

Find qi,i−1

6
 2. Pure birth process
Find qi,i+1 and qi,j , where j > i + 1.
3. Birth and Death process
λi
Ti ∼ Exp(λi + µi ), therefore, vi = λi + µi and pi,i+1 = λi +µi . Therefore,

qi,i+1 = λi

Find qi,i−1 .
Lemma 3.1.
1 − Pi,i (h)
lim = vi
h→0 h
Proof. Assume that h is small, such that after time h, the CTMC which starts at state i continues
to be in state i.
Then,
Z h
1 − Pi,i (h) = P(Ti < h) = vi e−vi t dt = 1 − e−vi h
0

It follows that 1 − Pi,i (h) = vi h + o(h) (Verify!).

Therefore,
1 − Pi,i (h) vi h + o(h)
lim = lim = vi
h→0 h h→0 h

and the following Lemma is why we refer to qi,j as the instantaneous transition rate.
Lemma 3.2.
Pi,j (h)
lim = qi,j
h→0 h
Proof. Again assume that h is small, such that you reach j from i in one jump in some time h and
stay at j.

Pi,j (h) = P(X(h) = j|X(0) = i) = pi,j P(Ti < h)

This therefore becomes the probability of making the jump (pi,j ) multiplied by the probability
that the jump happens in some time less than h (P(Ti < h)). Hence,

Pi,j (h) = pi,j P(Ti < h)

= pi,j (vi h + o(h))
(See the previous proof) = qi,j h + o(h)

Therefore,
Pi,j (h)
lim = qi,j
h→0 h

7
Kolmogorov Backward Equation
We will now establish an important tool for computing the transition probability function.
Theorem 3.3. The Kolmogorov Backward equation for a CTMC is given by,
X
′
Pi,j (t) = qi,k Pk,j (t) − vi Pi,j (t)
k̸=i

Proof.
∞
X
Pi,j (t + h) − Pi,j (t) = Pi,k (h)Pk,j (t) − Pi,j (t)
k=0
(Applying the Chapman-Kolmogorov equation to the first term)
X
= Pi,i (h)Pi,j (t) + Pi,k (h)Pk,j (t) − Pi,j (t)
k̸=i
X
= Pi,k (h)Pk,j (t) − Pi,j (t)(1 − Pi,i (h))
k̸=i

Now divide by h on both sides and take the limit h → 0 to obtain the result of the theorem.
Example 3.5. Consider the following two-state CTMC with states {0} and {1}. The transitions
and their rates are shown in the diagram below. Find P0,0 (t).

Note that q0,1 = v0 × p0,1 = λ × 1 and q1,0 = v1 × p1,0 = µ × 1.

Now the Kolmogorov Backward equations are given by,
′
P0,0 (t) = q0,1 P1,0 (t) − v0 P0,0 (t)
= λP1,0 (t) − λP0,0 (t) (1)
′
P1,0 (t) = q1,0 P0,0 (t) − v1 P1,0 (t)
= µP0,0 (t) − µP1,0 (t) (2)

Multiplying (1) by µ and multiplying (2) by λ, and adding them,

′ ′
µP0,0 (t) + λP1,0 (t) = 0

Integrating the above we get,

µP0,0 (t) + λP1,0 (t) = c
P0,0 (0) = 1, i.e., the probability of starting at the state {0} and staying at state {0} when no
time has passed is 1. P1,0 (0) = 0, i.e., the probability of starting at state {0} and moving to state
{1} when no time has passed is 0.
Therefore, c = µ. And rearranging the above we obtain,

λP1,0 (t) = µ[1 − P0,0 (t)]

8
Substituting in (1),
′
P0,0 (t) = µ[1 − P0,0 (t)] − λP0,0 (t)
′
P0,0 (t) + (µ + λ)P0,0 (t) = µ
 
′ µ
P0,0 (t) + (µ + λ) P0,0 (t) − =0
µ+λ
µ
(Let h(t) = P0,0 (t) − )
µ+λ
h′ (t) + (µ + λ)h(t) = 0
h(t) = Ke−(µ+λ)t
Therefore,
µ
P0,0 (t) = Ke−(µ+λ)t +
µ+λ
λ
Since P0,0 (0) = 1. Hence, K = µ+λ and therefore,
λ −(µ+λ)t µ
P0,0 (t) = e +
µ+λ µ+λ

Exercises
1. Verify that the Kolmogorov Backward equation for the Pure Birth Process is,
′
Pi,j = λi Pi+1,j (t) − λi Pi,j (t)

2. Verify that the Kolmogorov Backward equations for the Birth and Death process are,
′
P0,j (t) = λ0 Pi,j (t) − λ0 P0,j (t)
′
Pi,j (t) = λi Pi+1,j (t) + µi Pi−1,j (t) − (λi + µi )Pi,j (t), ∀i > 0

Kolmogorov Forward equation

The Kolmogorov Forward equation is given by,
X
′
Pi,j (t) = qk,j Pi,k (t) − vj Pi,j (t)
k̸=j

The Kolmogorov Forward equation is only valid for certain CTMCs like the Birth-Death processes
and finite state CTMCs, this would cover all CTMCs considered in this course. The Kolmogorov
Backward equation on the other hand is well defined for all CTMCs.

Exercises
1. Verify that the Kolmogorov Forward equation for the Pure birth process is,
′
Pi,j (t) = λj−1 Pi,j−1 (t) − λj Pi,j (t)

2. Using the forward equation, show that for a Pure Birth process,
Pi,i (t) = e−λi t

Is there an alternate way to show the above?

3. Verify that the Kolmogorov Forward equations for the Birth and Death process are,
′
Pi,0 (t) = µ1 Pi,1 (t) − λ0 Pi,0 (t)
′
Pi,j (t) = λj−1 Pi,j−1 (t) + µj+1 Pi,j+1 (t) − (λj + µj )Pi,j (t), ∀j > 0

9
4 Limiting Probability
The notion of limiting probability is the analogue of the stationary distribution for a CTMC. It
is defined as follows,
Pj ≡ lim Pi,j (t)
t→∞
Pj can be interpreted as the probability of a CTMC to be in state {j} after a long time has
passed. In other words, it is the long-run proportion of the CTMC to be in state {j}.
One can consider a more explicit example. Consider a M/M/1 queue. Suppose this queue has
a limiting probability. Then P5 denotes the probability of this queue to have 5 people after a long
time has passed.

Exercises
′
1. Show that if the limt→∞ Pi,j (t) exists, then limt→∞ Pi,j (t) = 0.
We will use the result from the above exercise to obtain conditions for the existence of limiting
probability. Consider the Kolmogorov Forward equation,
X
′
Pi,j (t) = qk,j Pi,k (t) − vj Pi,j (t)
k̸=j

(Taking limit)
X
′
lim Pi,j (t) = lim qk,j Pi,k (t) − vj Pi,j (t)
t→∞ t→∞
k̸=j
X
0= qk,j Pk − vj Pj
k̸=j

Therefore, the set of equations below can be used to determine the limiting probabilities of a
CTMC, X
qk,j Pk = vj Pj , ∀j (3)
k̸=j
P
and j Pj = 1. Let us apply this to evaluate the limiting probability of a CTMC.

Limiting probability of Birth and Death process

Applying (3) to the Birth and Death process leads to the following set of equations (Verify!),
µ1 P1 = λ0 P0
µ2 P2 + λ0 P0 = (λ1 + µ1 )P1
µ3 P3 + λ1 P1 = (λ2 + µ2 )P2
..
.
µn+1 Pn+1 + λn−1 Pn−1 = (λn + µn )Pn , n ≥ 1
It follows from the above that,
λ0 P0 = µ1 P1
λ1 P1 = µ2 P2
λ2 P2 = µ3 P3
..
.
λn Pn = µn+1 Pn+1

10
 Therefore, verify that the following also holds,
λn−1 λn−2 . . . λ1 λ0
Pn = P0
µn µn−1 . . . µ2 µ1
P∞
and since, n=0 Pn = 1. It follows that,

1
P0 = P∞ λn−1 λn−2 ...λ1 λ0
(4)
1+ n=1 µn µn−1 ...µ2 µ1

Hence, it follows that the limiting probabilities in a Birth and Death chain exists if and only if,
∞
X λn−1 λn−2 . . . λ1 λ0
<∞ (5)
n=1
µn µn−1 . . . µ2 µ1

Example 4.1 (Limiting probability of a M/M/1 queue). Recall that a M/M/1 queue is a Birth and
Death process where λ0 = λ1 = . . . = λn−1 = λ and µ1 = µ2 = . . . = µn = µ. From above it follows
that,
1 λ
P0 = P∞ λn
=1−
1+ n=1 µn
µ
 n  n  
λ λ λ
Pn = P0 = 1−
µ µ µ

Note that the above probabilities are well defined if and only if,

λ<µ

Therefore, the limiting probabilities in a M/M/1 queue exists if and only if, λ < µ. That is the clerk
services the customers at a faster rate than the rate at which the customers join the queue.

Example 4.2 (Limiting probability of a M/M/s queue). Recall that in a M/M/s queue there are s
servers (
nµ 1 ≤ n ≤ s
µn =
sµ n > s
and λn = λ, ∀n. From (5) it follows that the limiting probabilities exist if and only if,
∞ s ∞
X λn−1 λn−2 . . . λ1 λ0 X λn−1 λn−2 . . . λ1 λ0 X λn−1 λn−2 . . . λ1 λ0
= + <∞
n=1
µn µn−1 . . . µ2 µ1 n=1
µn µn−1 . . . µ2 µ1 n=s+1
µn µn−1 . . . µ2 µ1

Therefore,
∞
X λn−1 λn−2 . . . λ1 λ0
<∞
n=s+1
µn µn−1 . . . µ2 µ1

Now,
∞ ∞ ∞
X λn−1 λn−2 . . . λ1 λ0 X λn λs X λn−s
= = <∞
n=s+1
µn µn−1 . . . µ2 µ1 n=s+1
(sµ)n−s (sµ)((s − 1)µ)((s − 2)µ) . . . µ s!µs n=s+1 (sµ)n−s

This sum exists when, λ < sµ. That is all the s clerks in the queue service the customers faster
than the rate at which the customers join the queue.

11
Example 4.3. There are M machines and 1 serviceman. Each machine runs for an exponentially
distributed amount of time with mean λ1 before breaking down. The serviceman takes an exponentially
distributed amount of time to repair each broken machine with mean µ1 .
First define the states of the system as S = {0, 1, 2, . . . , M }, which denotes the number of broken
machines. It follows that this system is a CTMC. See the figure below to understand the transitions.

In state {0} there are M functioning machines, the time the first machine breaks down is
min(T1 , T2 , . . . , TM ), where Ti denotes the time the i − th machine broke down. Note that,

min(T1 , T2 , . . . , TM ) ∼ Exp(M λ)

Therefore, the rate from {0} to {1} is M λ. Verify the other rates in the figure given above. Now
from (4) it follows that for this system,
1
P0 = PM  λ n M!
1+ n=1 µ (M −n)!

Find Pn for this system.

Uniformization
Knowing Pi,j (t) for a CTMC leads to an entire understanding of the CTMC. The tools to obtain
CTMC so far that we have seen are the Kolmogorov Backward and Kolmogorov Forward equations.
Solving them analytically is not always possible. Uniformization is a technique to computationally
evaluate Pi,j (t) for a CTMC.
Consider a CTMC, X(t), in which mean time spent in a state is identically distributed across all
states, i.e.,
Ti ∼ Exp(v), ∀i ∈ S
Let N (t) denote the number of transitions in this CTMC by time t. Can you guess what kind of
a process N (t) is?
We know that for a Poisson process the interarrival times are independent and identically dis-
tributed as exponential distribution. It turns out the converse is also true, that if the interarrival
times are iid exponential, then the arrivals follow a Poisson process. For a proof of this fact refer
Durrett (1999) chapter on Poisson processes. Therefore, N (t) is a Poisson process with rate v.

12
Pi,j (t) = P(X(t) = j|X(0) = i)
X∞
= P(X(t) = j|X(0) = i, N (t) = n)P(N (t) = n|X(0) = i)
n=0
∞
X (vt)n
= P(X(t) = j|X(0) = i, N (t) = n)e−vt
n=0
n!
(Note that the term in blue is the just the n-step transition probability of the embedded Markov chain)
∞
X (n) (vt)n
= pi,j e−vt (6)
n=0
n!

Note that (6) holds only for CTMCs where vi = v, ∀i ∈ S.

However, we can adapt this for CTMCs where vi ’s are not all identical. What we do is we will
uniformize all states to have the same rate for the time spent in each state. We will uniformize it
by rate v such that,
vi ≤ v, ∀i
This will require us to adjust the transition matrix of the embedded Markov chain of the CTMC.
The new transition matrix p∗i,j is given by,
(
1 − vvi j=i
p∗i,j = vi
v pi,j j ̸= i

Now (6) can be rewritten as,

∞
X ∗(n) (vt)n
Pi,j (t) = pi,j e−vt (7)
n=0
n!

Note that (7) is computationally feasible always, i.e., it can be readily evaluated on a computer
to get a numerical estimate of Pi,j (t). This is the advantage of uniformization, that it gives us a
numerical procedure to understand the transition function of the CTMC.
Remark 5. The readjustment of pi,j to p∗i,j is based on the idea that in the embedded Markov chain
of the original CTMC pi,i = 0, i.e., the probability of going from the state to itself is 0. This is a
consequence of the way we define the embedded Markov chain.
In the uniformized CTMC since we allot “extra”(v ≥ vi ) v − vi to every state for the time spent,
we account for this by introducing a fictitious possibly non-zero probability of going from state {i} to
{i}. This is given by 1 − vvi . The other states are adjusted accordingly.
Example 4.4 (2-state CTMC). Consider the following two-state CTMC with states {0} and {1}
which we considered earlier. To recall, the transitions and their rates are shown in the diagram
below.

Verify the following, v0 = λ, v1 = µ, p0,1 = 1, p1,0 = 1.

13
 µ µ
Verify also, p∗0,0 = ∗
λ+µ , p0,1 = λ ∗
λ+µ , p1,1 = λ ∗
λ+µ , p1,0 = λ+µ .
" #
µ λ
∗ λ+µ λ+µ
p = µ λ
λ+µ λ+µ

Verify that,
" #2 " #
µ λ µ λ
∗(2) λ+µ λ+µ λ+µ λ+µ
p = µ λ = µ λ
λ+µ λ+µ λ+µ λ+µ

Therefore, " #
µ λ
p∗(n) = λ+µ
µ
λ+µ
λ
λ+µ λ+µ

From (7),

∞
X [(λ + µ)t]n
∗(n)
P0,0 (t) = P0,0 e−(λ+µ)t
n=0
n!
∞
[(λ + µ)t]n
 
X µ
= e−(λ+µ)t + e−(λ+µ)t
n=1
λ+µ n!
∞
[(λ + µ)t]n
 
µ X
= e−(λ+µ)t + e−(λ+µ)t
λ+µ n=1
n!
 
µ  
= e−(λ+µ)t + e−(λ+µ)t e(λ+µ)t − 1
λ+µ
(Follows from the Taylor expansion of e(λ+µ)t )
λ −(µ+λ)t µ
= e + (Rearranging)
µ+λ µ+λ

We obtained the same solution for P0,0 (t) by solving the Backward equation, we obtain the same
answer through uniformization here. Note that here we were lucky to be able to obtain an analyt-
ical solution through uniformization itself, however that need not always be the case. But through
uniformization we can always evaluate (7) on a computer to numerically obtain Pi,j (t).

References
1. Sheldon Ross, Introduction to Probability Models, Academic Press, 2024.

2. Rick Durrett, Essentials of Stochastic Processes, Springer, 1999.

14