Introduction to Hidden Markov Models

Alperen Degirmenci

This document contains derivations and algorithms for im- Z1 Z2 Zt ZN

plementing Hidden Markov Models. The content presented
here is a collection of my notes and personal insights from
two seminal papers on HMMs by Rabiner in 1989 [2] and
Ghahramani in 2001 [1], and also from Kevin Murphy’s book X1 X2 Xt XN
[3]. This is an excerpt from my project report for the MIT
6.867 Machine Learning class taught in Fall 2014. Fig. 1. A Bayesian network representing a first-order HMM. The hidden
states are shaded in gray.
I. H IDDEN M ARKOV M ODELS (HMM S )
HMMs have been widely used in many applications, such
1) Number of states in the model, K: This is the number
as speech recognition, activity recognition from video, gene
of states that the underlying hidden Markov process has.
finding, gesture tracking. In this section, we will explain what
The states often have some relation to the phenomena being
HMMs are, how they are used for machine learning, their
modeled. For example, if a HMM is being used for gesture
advantages and disadvantages, and how we implemented our
recognition, each state may be a different gesture, or a part
own HMM algorithm.
of the gesture. States are represented as integers 1, . . . , K.
A. Definition We will encode the state Zt at time t as a K × 1 vector of
A hidden Markov model is a tool for representing prob- binary numbers, where the only non-zero element is the k-th
ability distributions over sequences of observations [1]. In element (i.e. Ztk = 1), corresponding to state k ∈ K at time
this model, an observation Xt at time t is produced by a t. While this may seem contrived, it will later on help us in
stochastic process, but the state Zt of this process cannot be our computations. (Note that [2] uses N instead of K).
directly observed, i.e. it is hidden [2]. This hidden process 2) Number of distinct observations, Ω: Observations are
is assumed to satisfy the Markov property, where state Zt at represented as integers 1, . . . , Ω. We will encode the observa-
time t depends only on the previous state, Zt−1 at time t−1. tion Xt at time t as a Ω × 1 vector of binary numbers, where
This is, in fact, called the first-order Markov model. The nth - the only non-zero element is the l-th element (i.e. Xtl = 1),
order Markov model depends on the n previous states. Fig. 1 corresponding to state l ∈ Ω at time t. While this may seem
shows a Bayesian network representing the first-order HMM, contrived, it will later on help us in our computations. (Note
where the hidden states are shaded in gray. that [2] uses M instead of Ω, and [1] uses D. We decided
We should note that even though we talk about “time” to use Ω since this agrees with the lecture notes).
to indicate that observations occur at discrete “time steps”, 3) State transition model, A: Also called the state transi-
“time” could also refer to locations within a sequence [3]. tion probability distribution [2] or the transition matrix [3],
The joint distribution of a sequence of states and observa- this is a K × K matrix whose elements Aij describe the
tions for the first-order HMM can be written as, probability of transitioning from state Zt−1,i to Zt,j in one
N
Y time step where i, j ∈ {1, . . . , K}. This can be written as,
P (Z1:N , X1:N ) = P (Z1 )P (X1 |Z1 ) P (Zt |Zt−1 )P (Xt |Zt )
Aij = P (Zt,j = 1|Zt−1,i = 1). (3)
t=2
(1) P
Each row of A sums to 1, j Aij = 1, and therefore it is
where the notation Z1:N is used as a shorthand for called a stochastic matrix. If any state can reach any other
Z1 , . . . , ZN . Notice that Eq. 1 can be also written as, state in a single step (fully-connected), then Aij > 0 for
N
Y N
Y
P (X1:N , Z1:N ) = P (Z1 ) P (Zt |Zt−1 ) P (Xt |Zt )
t=2 t=1 1-α 1-β A11 A22 A33
(2)
α A12 A23
which is same as the expression given in the lecture notes.
There are five elements that characterize a hidden Markov 1 2 1 2 3
model: β A21 A32
The author is with the School of Engineering and Applied (a) (b)
Sciences at Harvard University, Cambridge, MA 02138 USA.
(adegirmenci@seas.harvard.edu). This document is an excerpt Fig. 2. A state transition diagram for (a) a 2-state, and (b) a 3-state ergodic
from a project report for the MIT 6.867 Machine Learning class taught in Markov chain. For a chain to be ergodic, any state should be reachable from
Fall 2014. any other state in a finite amount of time.

1
c 2014 Alperen Degirmenci
 μΦZ →Zt μZt→ΦZ ,Z μΦZ ,Z →Zt+1
all i, j; otherwise A will have some zero-valued elements. t-1 ,Zt

ΦZt-1,Zt ΦZt,Zt+1
t t+1 t t+1

Fig. 2 shows two state transition diagrams for a 2-state and Zt Zt+1
μZt→ΦZ t-1 ,Zt
μΦZ ,Z →Zt μZt+1→ΦZ t-1 ,Zt

3-state first-order Markov chain. For these diagrams, the state

t t+1

μZt→ΦX ,Z t t
μΦX ,Z →Zt
t t

transition models are,

  ΦXt,Zt
  A11 A12 0
1−α α
A(a) = , A(b) = A21 A22 A23  . (4) μΦX ,Z →Xt μXt→ΦX ,Z
β 1−β t t t t

0 A32 A33
Xt
The conditional probability can be written as
K Y
K Fig. 3. Factor graph for a slice of the HMM at time t.
Z Zt,j
Y
P (Zt |Zt−1 ) = Aijt−1,i . (5)
i=1 j=1
• Problem 2: Given observations X1 , . . . , XN and the
Taking the logarithm, we can write this as model λ, how do we find the “correct” hidden state
K X
X K sequence Z1 , . . . , ZN that best “explains” the observa-
logP (Zt |Zt−1 ) = Zt−1,i Zt,j log Aij (6) tions? This corresponds to finding the most probable
i=1 j=1 sequence of hidden states from the lecture notes, and
= Zt> log (A)Zt−1 . (7) can be solved using the Viterbi algorithm. A related
problem is calculating the probability of being in state
4) Observation model, B: Also called the emission prob- Ztk given the observations, P (Zt = k|X1:N ), which
abilities, B is a Ω × K matrix whose elements Bkj describe can be calculated using the forward-backward algo-
the probability of making observation Xt,k given state Zt,j . rithm.
This can be written as, • Problem 3: How do we adjust the model parameters λ =
Bkj = P (Xt = k|Zt = j). (8) (A, B, π) to maximize P (X1:N |λ)? This corresponds
to the learning problem presented in the lecture notes,
The conditional probability can be written as and can be solved using the Expectation-Maximization
Ω
K Y (EM) algorithm (in the case of HMMs, this is called the
Z Xt,k
Y
P (Xt |Zt ) = Bkjt,j . (9) Baum-Welch algorithm).
j=1 k=1
C. The Forward-Backward Algorithm
Taking the logarithm, we can write this as
K X
Ω
The forward-backward algorithm is a dynamic program-
ming algorithm that makes use of message passing (be-
X
logP (Xt |Zt ) = Zt,j Xt,k log Bkj (10)
j=1 k=1
lief propagation). It allows us to compute the filtered and
smoothed marginals, which can be then used to perform
= Xt> log (B)Zt . (11)
inference, MAP estimation, sequence classification, anomaly
5) Initial state distribution, π: This is a K × 1 vector detection, and model-based clustering. We will follow the
of probabilities πi = P (Z1i=1 ). The conditional probability derivation presented in Murphy [3].
can be written as, 1) The Forward Algorithm: In this part, we compute
K the filtered marginals, P (Zt |X1:T ) using the predict-update
cycle. The prediction step calculates the one-step-ahead
Y
P (Z1 |π) = πiZ1i . (12)
i=1
predictive density,
Given these five parameters presented above, an HMM P (Zt =j|X1:t−1 ) = · · ·
can be completely specified. In literature, this often gets K
X
abbreviated as = P (Zt = j|Zt−1 = i)P (Zt−1 = i|X1:t−1 )
λ = (A, B, π). (13) i=1
(14)
B. Three Problems of Interest
which acts as the new prior for time t. In the update state,
In [2] Rabiner states that for the HMM to be useful in
the observed data from time t is absorbed using Bayes rule:
real-world applications, the following three problems must
be solved: αt (j) , P (Zt = j|X1:t )
• Problem 1: Given observations X1 , . . . , XN and a = P (Zt = j|Xt , X1:t−1 )
model λ = (A, B, π), how do we efficiently compute
P (Xt |Zt = j, 
X1:t−1
)P (Zt = j|X1:t−1 )
P (X1:N |λ), the probability of the observations given =P
j P (Xt |Zt = j, 
X1:t−1
)P (Zt = j|X1:t−1 )
the model? This is a part of the exact inference problem
presented in the lecture notes, and can be solved using 1
= P (Xt |Zt = j)P (Zt = j|X1:t−1 ) (15)
forward filtering. Ct

2
c 2014 Alperen Degirmenci
Algorithm 1 Forward algorithm Algorithm 2 Backward algorithm
1: Input: A, ψ1:N , π 1: Input: A, ψ1:N , α
2: [α1 , C1 ] = normalize(ψ1  π) ; 2: βN = 1;
3: for t = 2 : N do 3: for t = N − 1 : 1 do
4: [αt , Ct ] = normalize(ψt  (A> αt−1 )) ; 4: βt = normalize(A(ψt+1  βt+1 ) ;
P
5: Return α1:N and log P (X1:N ) = t log Ct 5: γ = normalize(α  β, 1)
P 6: Return γ 1:N
6: Sub: [α, C] = normalize(u): C = j uj ; αj = uj /C;

βt−1 (i) = P (Xt:N |Zt−1 = i)

where the observations X1:t−1 cancel out because they are X
d-separated from Xt . Ct is the normalization constant (to = P (Zt = j, Xt , Xt+1:N |Zt−1 = i)
avoid confusion, we used Ct as opposed to Zt from [3]) j
X
given by, = P (Xt+1:N |Zt = j, 
X
t , 
Zt−1
 =
j)
K j
X
Ct , P (Xt |X1:t−1 ) = = P (Xt |Zt = j)P (Zt = j|X1:t−1 ). · · · P (Zt = j, Xt |Zt−1 = i)
j=1
X
(16) = P (Xt+1:N |Zt = j)P (Xt |Zt = j, 
Zt−1
 =i)
j
The K × 1 vector αt = P (Zt |X1:T ) is called the (filtered)
belief state at time t. · · · P (Zt = j|Zt−1 = i)
X
In matrix notation, we can write the recursive update as: = βt (j)ψt (j)A(i, j) (23)
 
j
αt ∝ ψt  A> αt−1 (17)
This can be written as
where ψt = [ψt1 , ψt2 , . . . , ψtK ] = {P (Xt |Zt = i)}1≤i≤K is
the local evidence at time t which can be calculated using βt−1 = A (ψt  βt ) (24)
Eq. 9, A is the transition matrix, and  is the Hadamard The base case for βN is
product, representing elementwise vector multiplication. The
pseudo-code in Algorithm 1 outlines the steps of the com- βN (i) = P (XN +1:N |ZN = i) = P (∅|ZN = i) = 1 (25)
putation.
The log probability of the evidence can be computed as Finally, the smoothed posterior is then
N N αi  βi
X X γi = P (26)
log P (X1:N |λ) = log P (Xt |X1:t−1 ) = log Ct j (α i (j)  β i (j))
t=1 t=1
(18) where the denominator ensures that each column of γ sums
This, in fact, is the solution for Problem 1 stated by Rabiner to 1 to ensure it is a stochastic matrix. The pseudo-code in
[2]. Working in the log domain allows us to avoid numerical Algorithm 2 outlines the steps of the computation.
underflow during computations. D. The Viterbi Algorithm
2) The Forward-Backward Algorithm: Now that we have
the filtered belief states α from the forward messages, we In order to compute the most probable sequence of hidden
can compute the backward messages to get the smoothed states (Problem 2), we will use the Viterbi algorithm. This
marginals: algorithm computes the shortest path through the trellis
diagram of the HMM. The trellis diagram shows how each
P (Zt = j|X1:N ) ∝ P (Zt = j.Xt+1:N |X1:t ) (19) state in the model at one time step connects to the states
∝ P (Zt = j|X1:t )P (Xt+1:N |Zt = j,  ). in the next time step. In this section, we again follow the
X1:t
derivation presented in Murphy [3].
which is the probability of being in state Ztj . Given that the The Viterbi algorithm also has a forward and backward
hidden state at time t is j, define the conditional likelihood pass. In the forward pass, instead of the sum-product algo-
of future evidence as rithm, we utilize the max-product algorithm. The backward
βt (j) , P (Xt+1:N |Zt = j). (20) pass recovers the most probable path through the trellis
diagram using a traceback procedure, propagating the most
Also define the desired smoothed posterior marginal as likely state at time t back in time to recursively find the most
γt (j) , P (Zt = j|X1:N ). (21) likely sequence between times 1 : t. This can be expressed
as,
Then we can rewrite Eq. 19 as δt (j) , max P (Z1:t−1 , Zt = j|X1:t ). (27)
Z1 ,...,Zt−1
γt (j) ∝ αt (j)βt (j) (22)
This probability can be expressed as a combination of the
We can now compute the β’s recursively from right to left: transition from the previous state i at time t−1 and the most

3
c 2014 Alperen Degirmenci
Algorithm 3 Viterbi algorithm Algorithm 4 Baum-Welch algorithm
1: Input: X1:N , K, A, B, π 1: Input: X1:N , A, B, α, β
2: Initialize: δ1 = π  BX1 , a1 = 0; 2: for t = 1 : N do
3: for t = 2 : N do 3: γ(:, t) = (α(:, t)  β(:, t))./sum(α(:, t)  β(:, t));
4: for j = 1 : K do 4: ξ(:, :, t) = ((α(:, t)  A(t + 1)) ∗ (β(:, t + 1) 
5: [at (j), δt (j)] = maxi (log δt−1 (:) + log Aij + B(Xt+1 ))T )./sum(α(:, t)  β(:, t));
log BXt (j)); 5: π̂ = γ(:, 1)./sum(γ(:, 1));
∗
6: ZN = arg max(δN ); 6: for j = 1 : K do
7: for t = N − 1 : 1 do 7: Â(j, :) = sum(ξ(2 : N, j, :), 1)./sum(sum(ξ(2 :
8: Zt∗ = at+1 Zt+1∗
; N, j, :), 1), 2);
∗ 8: ˆ :) = X(:, j)T γ
./sum(γ, 1);
B(j,
9: Return Z1:N
9: Return π̂, Â, B̂

probable path leading to i,

1) E Step:
δt (j) = max δt−1 (i)Aij BXt (j). (28)
1≤i≤K γtk , P (Ztk = 1|X, λold )
Here BXt (j) is the emission probability of observation Xt αk (t)βk (t)
= PN (35)
given state j. We also need to keep track of the most likely
j=1 αj (t)βj (t)
previous state i,
ξtjk , P (Zt−1,j = 1, Ztk = 1|X, λold )
at (j) = arg max δt−1 (i)Aij BXt (j). (29) αj (t)Ajk βk (t + 1)Bk (Xt+1 )
i = (36)
PN
The initial probability is i=1 αi (t)βi (t)
2) M Step: The parameter estimation problem can be
δ1 (j) = πj BX1 (j). (30) turned into a constrained optimization problem where
The most probable final state is P (X1:N |λ) is maximized, subject to the stochastic con-
straints of the HMM parameters [2]. The techniques of
∗
ZN = arg max δN (i). (31) Lagrange multipliers can be then used to find the model
i
parameters, yielding the following expressions:
The most probable sequence can be computing using trace-
back, E[Nk1 ] γ1k
π̂k = = PK (37)
Zt∗ = at+1 Zt+1
∗
. (32) N j=1 γ1j
PN
In order to avoid underflow, we can work in the log domain. E[Njk ] ξtjk
Âjk = P = PK t=2 PN (38)
This is one of the advantages of the Viterbi algorithm, since k0 E[N jk ] l=1 t=2 ξtjl
log max = max log; this is not PN
P possible
P with the forward- E[Mjl ] t=1 γtl Xtj
backward algorithm since log 6= log. Therefore B̂jl = = P N
(39)
E[Nj ] t=1 γtl
log δt (j) , max log δt−1 (i) + log Aij + log BXt (j). (33) λnew = (Â, B̂, λ̂) (40)
i

The pseudo-code in Algorithm 3 outlines the steps of the The pseudo-code in Algorithm 4 outlines the steps of the
computation. computation.

E. The Baum-Welch Algorithm F. Limitations

The Baum-Welch algorithm is in essence the Expectation- A fully-connected transition diagram can lead to severe
Maximization (EM) algorithm for HMMs. Given a sequence overfitting. [1] explains this by giving an example from
of observations X1:N , we would like to find computer vision, where objects are tracked in a sequence of
X images. In problems with large parameter spaces like this,
arg max P (X; λ) = arg max P (X, Z; λ) (34) the transition matrix ends up being very large. Unless there
λ λ
Z are lots of examples in the data set, or unless some a priori
by doing maximum-likelihood estimation. Since summing knowledge about the problem is used, then this leads to
over all possible Z is not possible in terms of computation severe overfitting. A solution to this is to use other types
time, we use EM to estimate the model parameters. of HMMs, such as factorial or hierarchical HMMs.
The algorithm requires us to have the forward and R EFERENCES
backward probabilities α, β calculated using the forward-
[1] Z. Ghahramani, ”An Introduction to Hidden Markov Models and
backward algorithm. In this section we follow the derivation Bayesian Networks,” International Journal of Pattern Recognition and
presented in Murphy [3] and the lecture notes. Artificial Intelligence, vol. 15, no. 1, pp. 9–42, 2001.

4
c 2014 Alperen Degirmenci
[2] L. Rabiner, ”A Tutorial on Hidden Markov Models and Selected
Applications in Speech Recognition,” Proceedings of the IEEE, vol.
77, no. 2, pp. 257–286, 1989.
[3] K.P. Murphy, Machine Learning: A Probabilistic Perspective, Cam-
bridge, MA: MIT Press, 2012.

5
c 2014 Alperen Degirmenci