IE 325: Stochastic Models

Homework Assignment 6
Solutions
Spring 2020

Question 1. Consider a job shop that consists of three machines and two repairmen. Sup-
pose that the amount of time that machine i runs before breaking down is exponentially
distributed with parameter µi , for i = 1, 2, 3. Moreover, the amount of time that it takes for
a repairman to fix a machine is exponentially distributed with mean λ1 (the same for both
repairmen and for all machines). Whenever a machine breaks down, it goes under repair as
long as there is an available repairman. If both repairmen are busy, then the broken machine
waits until one of the repairmen becomes available.

(a) Describe this system as a CTMC. Draw the transition diagram and write the transition
rate matrix Λ.

(b) Assume µ1 = 2, µ2 = 3, µ3 = 1 and λ = 2. Determine the proportion of time the

process is in each of the states. (You may use a computer to solve the system of
equations.)

(c) Assume µi are as given in part b for i = 1, 2, 3. Find the proportion of time the process
is in each of the states for different values of λ. Solve it for at least three different values
of λ. Briefly explain your observation.

Solution
( of Question 1. (a)
1 if machine 1 is working
•i :
0 if machine 1 is not working
(
1 if machine 2 is working
•j :
0 if machine 2 is not working
(
1 if machine 3 is working
•k :
0 if machine 3 is not working


 N if all not working machines are being repaired

I if machine 1 is not working but not being repaired
•l :


 II if machine 2 is not working but not being repaired

III if machine 3 is not working but not being repaired

1
 Hence defining states by (i, j, k, l), X(t)t≥0 is a CTMC where state space is as follows:

E = {(1, 1, 1, N ), (0, 1, 1, N ), (1, 0, 1, N ), (1, 1, 0, N ), (0, 0, 1, N ),

(0, 1, 0, N ), (1, 0, 0, N ), (0, 0, 0, I), (0, 0, 0, II), (0, 0, 0, III)}.

The transition rate matrix can be found on the next page and the transition diagram is as
follows:

2
3
 (b) Letting µ1 = 2, µ2 = 3, µ3 = 1 and λ = 2, if we solve the system of equations
Π · Λ = ~0 and Π · ~1 = 1 we obtain the following results:

π(1,1,1,N ) = 0.1270 π(0,1,1,N ) = 0.1270 π(1,0,1,N ) = 0.1905 π(1,1,0,N ) = 0.0635

π(0,0,1,N ) = 0.1905 π(0,1,0,N ) = 0.0635 π(1,0,0,N ) = 0.0952 π(0,0,0,I) = 0.0476
π(0,0,0,II) = 0.0476 π(0,0,0,III) = 0.0476

(c) Taking µ1 = 2, µ2 = 3, µ3 = 1, solutions for λ = 1, λ = 3 and λ = 4 can be seen

below. It can be observed that as λ increase, meaning that the repairmen work faster, the
proportion of time that the machines work increases.
For λ = 1
π(1,1,1,N ) = 0.0370 π(0,1,1,N ) = 0.0741 π(1,0,1,N ) = 0.1111 π(1,1,0,N ) = 0.0370
π(0,0,1,N ) = 0.2222 π(0,1,0,N ) = 0.0741 π(1,0,0,N ) = 0.1111 π(0,0,0,I) = 0.1111
π(0,0,0,II) = 0.1111 π(0,0,0,III) = 0.1111

For λ = 3
π(1,1,1,N ) = 0.2195 π(0,1,1,N ) = 0.1463 π(1,0,1,N ) = 0.2195 π(1,1,0,N ) = 0.0732
π(0,0,1,N ) = 0.1463 π(0,1,0,N ) = 0.0488 π(1,0,0,N ) = 0.0732 π(0,0,0,I) = 0.0244
π(0,0,0,II) = 0.0244 π(0,0,0,III) = 0.0244

For λ = 4
π(1,1,1,N ) = 0.3005 π(0,1,1,N ) = 0.1502 π(1,0,1,N ) = 0.2254 π(1,1,0,N ) = 0.0751
π(0,0,1,N ) = 0.1127 π(0,1,0,N ) = 0.0376 π(1,0,0,N ) = 0.0563 π(0,0,0,I) = 0.0141
π(0,0,0,II) = 0.0141 π(0,0,0,III) = 0.0141

Question 2. There is a maintenance worker who is responsible for two machines in a work-
station, machine A and machine B. The machines have exponential running times with rates
λA and λB before they breakdown. When a machine breaks down, the reparation takes
exponential time with rate µA and µB for machine A and machine B, respectively. Since
machine A is the bottleneck of the workstation, it is the priority of the maintenance worker,
i.e, even if machine A breaks down while he repairs machine B, he stops working on machine
B and starts repairing machine A. Write the system of equations to solve for the limiting
probabilities.
(
1 if machine 1 is working
Solution of Question 2. •i :
0 if machine 1 is not working
(
1 if machine 2 is working
•j :
0 if machine 2 is not working
Hence defining states by (i, j), X(t)t≥0 is a CTMC where state space is as follows:

E = {(1, 1), (1, 0), (0, 1), (0, 0)}.

4
The balance equations can be written as follows:

(λ1 + λ2 )π(1,1) = µ1 π(0,1) + µ2 π(1,0)

(µ1 + λ2 )π(0,1) = λ1 π(1,1) + µ1 π(0,0)
(λ1 + µ2 )π(1,0) = λ2 π(1,1)
µ1 π(0,0) = µ2 π(0,1) + µ1 π(1,0)
π(0,0) + π(1,0) + π(0,1) + π(1,1) = 1

This set of equations can be solved for π(1,1) , π(0,1) , π(1,0) and π(0,) using λ1 , λ2 , µ1 and µ2 .
However, the results are too complicated and cannot be simplified.

Question 3. A town has total number of N people living in it and there are some individuals
whom have infection. Two individuals are involved in an interaction according to a Poisson
process with rate λ. The interaction may involve any two individual of the town with equal
probabilities. The infection spreads only if one of the individuals is infected and the other
one is non-infected and if so, the infection spreads to non-infected individual with probability
p. Assume there is no treatment for this infection and individuals stay infected.

(a) Letting X(t) denote the number of infected individuals, notice that {X(t), t ≥ 0} is a
continuous-time Markov chain. Draw its transition diagram.

(b) Specify the type of CTMC in part a with reasoning.

(c) Starting with a single infected individual, what is the expected time until all members
are infected?

(d) Now assume that infected individuals gain their health back with exponential healing
rate µ. Draw the transition diagram of this CTMC.

Solution of Question 3. (a) Notice that the infection spreads only if infected and non-
infected individuals interact and if
so, the infection spreads to the non-infected individual
with probability p. Each of the N2 pairs of individuals are independently in contact with
probability p. Given that there are n infected people, each of the (N − n) susceptible people
will become infected at the rate of npλ and
( (N −n)npλ
n = 1, 2 · · · , N − 1
λn = (N2 )
0 otherwise

Therefore the transition diagram can be constructed as follows:

5
(b) Since there is no transition from state i to state i − 1, it is a pure birth process.
(c) Let T denote the time until all the population is infected, then
N
X −1
T = Ti
i=1

where Ti is the time to go from i to i + 1 infected people. Ti ’s are independent exponential

random variables with rates λi = (N −i)ipλ , for all i = 1, 2, · · · , N − 1. Hence,
(N2 )
N −1 N −1
X X 1
E[T ] = E[Ti ] =
i=1 i=1
λi
N −1
X N (N − 1)
=
i=1
2i(N − i)pλ
N −1
N (N − 1) X 1
=
2pλ i=1
i(N − i)

(d) Individuals gain their health back with exponential rate µ. Given that there are n
infected people, µn = nµ.

Question 4. Potential customers arrive at a single-server station with respect to a Poisson

process with rate λ. If an arriving customer finds the system empty then s/he enters the
system. Otherwise if there are n ≥ 1 customers that are already in the station (including
the one being served, then s/he enters with probability α/n. Assume that the service rate
is exponential with mean 1/µ.

(a) Construct a birth and death process to model the problem and determine the birth
and death rates.

(b) Give an expression for the expected number of customers waiting in the queue (exclud-
ing the one being served).

Solution of Question 4. Let X(t) denote the number of customers at time t. The corre-
sponding state space would be E = {0, 1, 2, 3, . . . }
(a) The transition diagram is as follows:

6
 αλ
• The birth rate in state n is λn =
n
• The death rate in state n is µn = µ
αλ µi
• Also, the state transition rates for state n is p0,1 = 1, pi,i+1 = and pi,i−1 = .
αλ + µi αλ + µi

(b) To find the expected number of customers waiting in the queue, we need to find the
limiting probabilities for this birth and death process example. Balance equations to calculate
the limiting probabilities are as follows:
λ
State 0: λπ0 = µπ1 =⇒ π1 = π0
µ
αλ2
State 1: (αλ + µ)π1 = λπ0 + µπ2 =⇒ π2 = π0
µ2
αλ α 2 λ3
State 2: ( + µ)π2 = αλπ1 + µπ3 =⇒ π3 = π0
2 2µ3
αλ αλ α 3 λ4
State 3: ( + µ)π3 = π2 + µπ4 =⇒ π4 = π0
3 2 3!µ4
..
.
∞
αλ α2 λ2 α3 λ3
 
X λ
1= πn = π0 + π 0 1 + + + + ···
n=0
µ µ 2µ2 3!µ3
x2 x3
 
αλ λ
If we take = x the equation becomes 1 = π0 + π0 1 + x + + + ···
µ µ 2! 3!

7
 2 3
By Taylor series for ex , we know ex = (1 + x + x2! + x3! + · · · ). Hence the equation can be
solved as follows:  
λ x
1 = π0 1 + e
µ
µ
π0 = αλ
µ + λe µ
Letting N be the number of customers waiting in the queue, the expected number of cus-
tomers waiting in the queue can be found as follows:
∞
X αλ2 α2 λ3 α 3 λ4
E[N ] = (i − 1)πi = 1 · π 0 + 2 · π 0 + 3 · π0 + · · ·
i=2
µ2 2µ3 3!µ4
αλ2 αλ α2 λ2 α3 λ3
 
= 2 π0 1 + + + + ···
µ µ 2!µ2 3!µ3
αλ
Taking µ
= x, the equation becomes
∞
X αλ2 x2 x3
(i − 1)πi = 2 π0 (1 + x + + + ···)
i=2
µ 2! 3!

Using Taylor series for ex again, the following calculations hold:

∞ αλ
X αλ2 x αλ2 αλ αλ 2
µ αλ αλ 2
e µ
E[N ] = (i − 1)πi = 2 π0 e = 2 π0 e µ = 2 αλ e
µ =

i=2
µ µ µ µ + λe µ µ (µ + λe αλ
µ )

Question 5. Customers arrive at a three-server system at a Poisson rate λ. An arrival

finding the system empty is equally likely to enter service with three servers. An arrival
finding one customer in the system will enter service with the one of the idle servers with
equal probability. An arrival finding two others in the system will enter service with the idle
server. An arrival finding three others in the system will wait in line for the first free server.
An arrival finding four in the system will not enter. All service times are exponential with
rate µ, and once a customer is served (by any of the servers), he departs the system.

(a) Define the states.

(b) Find the long-run probabilities.

(c) Suppose a customer arrives and finds three others in the system. What is the expected
time he spends in the system?

(d) What proportion of customers enter the system?

(e) What is the average time an entering customer spends in the system?

Solution of Question 5. (a) Let X(t) denote the number of customers in the system at
time t. Therefore the state space is E = {0, 1, 2, 3, 4}.
(b) The transition rate diagram of this system can be constructed as follows:

8
 To find the long-run probabilities, one needs to solve the balance equations, which are
given below:
λ
State 0: λπ0 = µπ1 =⇒ π1 = π0
µ
λ2
State 1: (λ + µ)π1 = λπ0 + 2µπ2 =⇒ π2 = 2 π0
2µ
λ3
State 2: (λ + 2µ)π2 = λπ1 + 3µπ3 =⇒ π3 = 3 π0
6µ
λ4
State 3: (λ + 3µ)π3 = λπ2 + 3µπ4 =⇒ π4 = π0
18µ4
State 4: 3µπ4 = λπ3
π0 + π1 + π 2 + π3 + π4 = 1

9
 Solving the balance equations, we find the following long-run probabilities
1
π0 = λ λ2 λ3 λ4
1+ µ
+ 2µ2
+ 6µ3
+ 18µ4
λ
µ
π1 = λ λ2 λ3 λ4
1+ µ
+ 2µ2
+ 6µ3
+ 18µ4
λ2
2µ2
π2 = λ λ2 λ3 λ4
1+ µ
+ 2µ2
+ 6µ3
+ 18µ4
λ3
6µ3
π3 = λ λ2 λ3 λ4
1+ µ
+ 2µ2
+ 6µ3
+ 18µ4
λ4
18µ4
π4 = λ λ2 λ3 λ4
1+ µ
+ 2µ2
+ 6µ3
+ 18µ4

(c)

E[Time spent in the system] = E[Time spent in the queue] + E[Time spent in the service]
1 1
= +
3µ µ
(d) Since arriving customers do not enter when there are four customers in the system, the
proportion of customers that enter to the system can be found by 1 − π4 .
(e)
1 4
µ
(π0 + π1 + π2 ) + 3µ π3
W =
1 − π4
Question 6. Consider an M/M/1 queue with the following modification: after the comple-
tion of a service, the customer will depart the system with probability 0.7 or join to the end
of the queue with probability 0.3. Therefore, a customer may be serviced more than once.
Assume that the arrival rate is λ and rate of service time is µ.

(a) Write the balance equations and solve them to find the steady-state probabilities.

(b) State the condition that is necessary for the existence of steady-state probabilities.

(c) What proportion of time is the server busy?

(d) What is the average number of customers in the system?

(e) What is the average queue length?

(f) Find the expected waiting time of customer.

(g) What is the expected amount of time that a customer spends in service (time spent in
the queue is not included)?

10
Solution of Question 6. Let X(t) be the number of customers in the system at time
t. Service of a customer takes exponential time with rate µ, and customer departs the
system with probability 0.7. By decomposition theorem, departures from the system follow
a Poisson process with rate 0.7µ. Therefore the transition rate diagram of this system can
be constructed as follows:

(a)The balance equations are as follows:

λ
State 0: λπ0 = 0.7µπ1 =⇒ π1 = π0
0.7µ
 2
λ
State 1: (λ + 0.7µ)π1 = λπ0 + 0.7µπ2 =⇒ π2 = π0
0.7µ
 3
λ
State 2: (λ + 0.7µ)π2 = λπ1 + 0.7µπ3 =⇒ π3 = π0
0.7µ
..
.
 n
λ
πn = π0
0.7µ
∞ ∞  i
X X λ
1= πi = π0 + π 0
i=1 i=1
0.7µ

11
 Using the geometric series formula, the steady-state probabilities can be found as follows:
1 λ
π0 = i = 1 −
0.7µ

P∞ λ
1+ i=1 0.7µ
 i  
λ λ
πi = 1− for i = 1, 2, 3, . . .
0.7µ 0.7µ

(b)
∞  i
X λ λ
Steady state probabilities exist if < ∞ or equivalently if <1
i=1
0.7µ 0.7µ

(c)The proportion of the time that the server is busy can be found as follows:
∞  
X λ λ
πi = 1 − π 0 = 1 − 1 − =
i=1
0.7µ 0.7µ

(d) Let L denote the average number of customers in the system. Note that the system is
an M/M/1 queue with arrival rate λ and departure rate 0.7µ. Therefore since the traffic
λ
intensity is given by ρ = 0.7µ , L can be found as follows:

λ
ρ 0.7µ
L= = λ
1−ρ 1 − 0.7µ

(e) Let Lq denote the average number of customers in the queue. Lq can be found as follows:
 2
λ
ρ2 0.7µ
Lq = = λ
1−ρ 1− 0.7µ

(f ) Let Wq denote the expected waiting time of customers. Using Little’s Law, Wq can be
found as follows:
Lq Lq
Lq = RWq =⇒ Wq = =⇒ Wq = (Since system has infinite capacity)
R λ
 2
λ
0.7µ
Wq =  
λ
λ 1− 0.7µ

(g)The expected amount of time that a customer spends in service can be found as follows:
1
E[W − Wq ] =
0.7µ

12
Question 7. Analyze the M/M/2/5 queuing model.

(a) Draw the transition rate diagram.

(b) Write balance equations.

(c) Compute the limiting probabilities.

(d) Write expressions for the system performance measures (L, Lq, B, U, W, Wq) in terms
of the limiting probabilities.

Solution of Question 7. (a) The transition rate diagram can be constructed as follows:

(b) Balance equations of the system can be written as follows:

State 0: λπ0 = µπ1

State 1: (λ + µ)π1 = λπ0 + 2µπ2
State 2: (λ + 2µ)π2 = λπ1 + 2µπ3
State 3: (λ + 2µ)π3 = λπ2 + 2µπ4
State 4: (λ + 2µ)π4 = λπ3 + 2µπ5
State 5: 2µπ5 = λπ4
π0 + π 1 + π2 + π3 + π 4 + π5 = 1

(c) Using the equations found in part (b), the limiting probabilities can be computed as
follows:
λ
π1 = π0
µ
λ λ2
π2 = π1 = 2
2µ 2µ
λ λ3
π3 = π2 = 3
2µ 4µ
λ λ4
π4 = π3 = 4
2µ 8µ
λ λ5
π5 = π4 =
2µ 16µ5
λ2 λ3 λ4 λ5
 
λ
1 = π0 1 + + 2 + 3 + 4 +
µ 2µ 4µ 8µ 16µ5

13
 Therefore the limiting probabilities are as follows:
1
π0 =  
λ λ2 λ3 λ4 λ5
1+ µ
+ 2µ2
+ 4µ3
+ 8µ4
+ 16µ5

λ
µ
π1 =  
λ λ2 λ3 λ4 λ5
1+ µ
+ 2µ2
+ 4µ3
+ 8µ4
+ 16µ5

λ2
2µ2
π2 =  
λ λ2 λ3 λ4 λ5
1+ µ
+ 2µ2
+ 4µ3
+ 8µ4
+ 16µ5

λ3
4µ3
π3 =  
λ λ2 λ3 λ4 λ5
1+ µ
+ 2µ2
+ 4µ3
+ 8µ4
+ 16µ5

λ4
8µ4
π4 =  
λ λ2 λ3 λ4 λ5
1+ µ
+ 2µ2
+ 4µ3
+ 8µ4
+ 16µ5

λ5
16µ5
π5 =  
λ λ2 λ3 λ4 λ5
1+ µ
+ 2µ2
+ 4µ3
+ 8µ4
+ 16µ5

(d)
L = 0π0 + 1π1 + 2π2 + 3π3 + 4π4 + 5π5
Lq = 0(π0 + π1 + π2 ) + 1π3 + 2π4 + 3π5
B = 0π0 + 1π1 + 2(π2 + π3 + π4 + π5 )
B B
U= =
S 2
L L
W = =
R λ(1 − π5 )
Lq Lq
Wq = =
R λ(1 − π5 )

14