See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220182032

HMM Based On-Line Handwriting Recognition.

Article  in  IEEE Transactions on Pattern Analysis and Machine Intelligence · November 1996

DOI: 10.1109/34.541414 · Source: DBLP

CITATIONS READS

259 1,115

3 authors, including:

Jianying Hu
IBM
177 PUBLICATIONS   4,062 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Jianying Hu on 17 September 2014.

The user has requested enhancement of the downloaded file.

 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996 1039

HMM Based On-Line (Automatic Evolutional Grammar Interpretation System), is

shown in Fig. 1 (the application generic components of AEGIS
Handwriting Recognition were originally developed for speech applications [8]).The system
may perform both recognition and training, depending on the
Jianying Hu, Member, IEEE, Michael K. Brown, Senior mode of operation and the loaded grammar(s). There are three
Member, I€€€, and William Turin, Senior Member, IEEE application generic components: an HMM scoring module, an
Evolutional Grammar (EG) processing module, and a back-tracing
Abstract-Hidden Markov Model (HMM) based recognition of module. In addition, there are application specific modules that
handwriting is now quite common, but the incorporation of HMM's into perform feature extraction (FX), state scoring, model parameter
a complex stochastic language model for handwriting recognition is still learning, and input flow control.
in its infancy. We have taken advantage of developments in the speech
processing field to build a more sophisticated handwriting recognition Nvdc
system. The pattern elements of the handwriting model are
subcharacter stroke types modeled by HMM's. These HMM's are
concatenated to form letter models, which are further embedded in a K*l"lZ
I I I I
stochastic language model. In addition to better language modeling, SUI"&! Node
we introduce new handwriting recognition features of various kinds. Model Input
State scores
Some of these features have invariance properties, and some are Pa"etCn:
segmental, covering a larger region of the input pattern. We have V

Input Fraluic OG
achieved a writer independent recognition rate of 94.5% on 3,823
unconstrained handwritten word samples from 18 writers covering a 32
word vocabulary.
G r m m a r Index
~

-
Proccablng

stale nurauons
Index Terms-On-line handwriting recognition, hidden Markov
models, subcharacter models, evolutional grammar, invariant features,
segmental features.
Fig. 1. Partial diagram of AEGIS.

1 INTRODUCTION The handwriting data was collected using a newly developed

graphics input tablet [9]. The sampling rate is 200 samples per
COMPUTER recognition of handwritten cursive script has received second; the tablet dimensions are 8.5 in x 11 in; and resolution is
relatively little attention, until recently, when compared to Optical 0.1 mm. Writers were asked to write on a lined sheet of paper with
Character Recognition (OCR), speech recognition, and other image no constraints on speed or style. The lines on the paper suggest an
or scene analysis areas. Interest began to grow significantly in the implied preferable orientation and size.
1980s and 1990s (for a good survey, see [I]) with the introduction Preprocessing consists of two steps: noise reduction and nor-
of small, but sufficiently powerful, portable computers to serve as malization. Input device noise is reduced using a spline smoothing
the support platforms. Hidden Markov Model (HMM) based operator. A spline kernel is convolved with the x and y coordinates
handwriting recognition has now become quite common for both of the input sample points to generate the coordinates of sampled
on-line and off-line systems (e.g., 121, [31, [41, [51, 6[1, [7J).However, points of a local approximating cubic spline [lo]. Cusps are de-
the incorporation of subcharacter HMMs into a complex stochastic tected beforehand and treated as boundary points during the
language model is still in its infancy, and there has been relatively smoothing operation to avoid loosing important shape features.
little effort devoted to feature development. In this paper, we de- The only normalization operation currently applied is deskewing
scribe the application of language and stochastic modeling meth- of word samples. After preprocessing, an equal-arclength resam-
ods developed originally for speech recognition to the problem of pling procedure is applied to remove variations in writing speed.
on-line handwriting recognition, as well as the development of Each word sample is then represented as a time-ordered sequence
new features. We begin with a Hidden Markov Model (HMM) of observations in the form of feature vectors.
based system with subcharacter model units using well known In the next section, we give detailed descriptions of the meth-
point oriented features like stroke tangents, and then introduce ods used in our HMM-based recognition engine, including model
new features for handwriting recognition that are invariant with topology, development of subcharacter model units, stochastic
respect to the three common factors of geometric distortion- language modeling, decoding, and parameter learning. Section 3
translation, rotation, and scaling. These purely stochastic methods introduces invariant features, describes their advantages, and de-
are combined with interleaved segmental features that capture a velops two new invariant features. Segmental features are intro-
larger region of segmented shape information. The stochastic duced in Section 4, where a feature representing holistic letter
processing is repeatedly interrupted to insert these features into shape information is incorporated into the stochastic model de-
partial hypotheses. This new method yields substantially im- coding process. We then present experimental results and con-
proved recognition accuracy. clude in Sections 5 and 6.
A partial diagram of the recognition system, called AEGIS
2 MODELDESIGN AND TRAINING
Consider a hidden Markov chain with N states and transition
J. H u and M.K. Brown are with Lucenf Techriologies Bell Laboratories, 700
Mountain Ave., Murray Hill, NJ 07974. probability matrix A = [u,]~,,. If st is the state index of the Markov
E-mail: {jianhu, mkbl@rbell-labs.com. chain at sample point t, then a,, = Pu(s,+, = j I st = I ) . Suppose the
W. Turin is with ATBT Bell Laboratories, 600 Mountain Ave., Murray Hill, process observations 0, are drawn from a finite observation set
NJ07974. E-mail: wt@researck.att.com. V = [U,,v2, ..., U,], the state conditional distributions are repre-
Manuscript received Feb. 10, 1995; revised May 15, 1996. Recommended for ac- sented by the conditional probabilities: bi ( k ) = PdO, = vUkI sf = j ) .
ceptance by J.J. Hull. Detailed description of HMMs can be found in [lll, [121.
For information on obtaining reprints of this article, please send e-mail to: Among the models describing speech and handwriting the
transpami@computer.org, and reference IEEECS Log Number P96068.

0162-8828/96$05.0001996 IEEE

Authorized licensed use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.
 1040 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996

most popular are the so-called left-to-right HMMs in which ui = 0 figure, there are six, eight, and five strokes in "a," "g," and "j"
for j .:i. The subcharacter and character models we have adopted are (without the dot), respectively. "a" and "g" share the first 4 strokes
even more restrictive disallowing in addition state skipping (uli = 0 SI, ,534, s 5 , s6; "g" and "j" share the last four strokes s18,
for j > i + 1).We have selected this relatively simple topology be- s19, s20, sl;stroke sl (corresponding to upward ligature) is
cause it has been shown to be successful in speech recognition, and shared by all three samples. The training procedure will be dis-
there has not been sufficient proof that more complex topologies cussed in more detail later in Section 2.4.
would necessarily lead to better recognition performance. Fur- Ligatures are attached to letters only during training. At recog-

&T$
thermore, in unconstrained handwriting, skipping of segments nition time they are treated as special, one stroke "connecting"
seem to happen most often for ligatures, which in our system is letters inserted between "core" letters and can be skipped with no
handled by treating ligatures as special "connecting" letters and penalty (see Fig. 3). This treatment insures that our system can
allowing them to be bypassed in the language model. handle mixed style handwriting as opposed to pure cursive only.

2.1 Nebulous Stroke Models

Early HMM based on-line handwriting recognition systems used
word models as basic units, where a separate model is designed
sl
for each word 121. Systems using letter models as basic units have
become popular recently (e.g., [51, 161). By tying models to letters
instead of words, the model set does not increase with the size of
the vocabulary. Furthermore, modification of the vocabulary can 5
be done by simply changing the grammatical constraints of the 1
SI 19
system while leaving the model set intact. However, letter models
are still unduly inefficient. English alphabet, like most western
Fig. 2. Stroke segmentation of three letter samples.
alphabets, contains patterns that are common to more than one
letter. When letter models are used, the constituent patterns of
individual letters are modeled independently regardless of their In English cursive script, crosses (for "t"s and "x"s) and dots
similarities, causing redundancy in storing and comparing the (for "i"s and "j"s) are referred tu as delayed strokes. In previous
models. One way to solve this problem and further improve methods delayed strokes are first detected in preprocessing and
model efficiency is to use subcharacter models as basic units 141. then either discarded or used in postprocessing. There are two
This approach has another advantage: shared patterns can be bet- drawbacks to these approaches. First, the information contained in
ter trained with the same amount of training data, because by delayed strokes is wasted in the first case and inadequately used
sharing models they also share training samples. in the second case because stroke segmentation cannot be influ-
Several difficulties are involved in developing subcharacter enced by this data. Second, it is often difficult to detect delayed
models. First, it is not clear how to break letters into subcharacter strokes reliably during preprocessing. The approach we have
units. Various sets of subcharacter primitives have been proposed taken is to treat delayed strokes as special letters in the alphabet. A
before for non-HMM based on-line systems (e.g., [13]). More re- word with delayed strokes is given alternative spellings to ac-
cently, Bercu and Lorette proposed segmenting an on-line hand- commodate all possible sequences with delayed strokes in differ-
writing sample into a succession of primitives classified as loops, ent positions. During recognition, delayed strokes are considered
humps and cusps and using them as feature observations in an as inherent parts of a script just like normal letters and contribute
HMM based recognition system 141. A common drawback of the directly to the scoring of the hypotheses. Although potentially this
above approaches is that the handwriting sample has to be seg- approach can lead to a large increase in the number of hypotheses,
mented before recognition, which often prematurely limits the the actual increase of searching space can be controlled through
hypothesis space. Furthermore, training of subcharacter models pruning in a beam search mechanism [141. Alternatively, for large
can be difficult, because while training of letter models can be vocabulary tasks one could first use a simpler network with inex-
initiated using samples of isolated letters, it is hard to obtain reli- act delayed stroke modeling (e.g., one optional delayed stroke
able samples of isolated subcharacter patterns. module at the end of the network shared by all paths) to obtain the
We introduce an approach using subcharacter models called top few most likely candidates and then use exact delayed stroke
nebulous stroke models. A letter model is a concatenation of several modeling on the reduced candidate set to obtain the final recogni-
such stroke models. Sharing among letters is enforced by referring tion result.
to the same stroke models. A stroke could be any segment of a
2.2 Grammatical Constraints
handwritten script. This is the major novelty of this approach: we
do not attempt to impose rigidly defined stroke models, instead, The nebulous stroke models described above are concatenated
we make the system learn indefinite stroke models through train- according to a lexicon to form letter models, which are also left to
ing, hence the term nebulous. The only constraints given to the right HMMs without state skipping. A letter could be written in
system are the number of strokes in each letter, and which of the different patterns in which case each pattern is modeled by a sepa-
strokes are shared. No manual segmentation is involved in train- rate HMM. Currently the entire lexicon is specified manually. The
ing: the stroke models are trained first on isolated letters and later letter models are further embedded in a grammar network, which
on whole word samples, but never on isolated or segmented can represent a variety of grammatical constraints, e.g., word dic-
strokes (which would be impossible to obtain since there is no tionaries, statistical N-gram language models and context free
specific definition of a stroke). Using this approach, the system is grammars.
allowed the freedom to decide on the most "natural" way of seg- The three primary components of the evolutional grammar are
menting letters into strokes. Furthermore, this decision can be a grammar arc table, a null arc (unlabeled grammar arc) table, and
adjusted automatically through training when more and more a Recursive Transition Network (RTN) table which records the
new features are taken into consideration. Fig. 2 shows the seg- grammar rules for expanding the nonterminals [SI. Labeled
mentation of several letter samples as a result of the training pro- grammar arcs can have either a nonterminal label representing
cedure when a one-state HMM is used to model each stroke, with another RTN subnetwork or an HMM label representing an HMM
a single tangent slope angle feature. In the samples shown in the defined in another table called the lexicon, which describes the

Authorized licensed use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.
 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. IO, OCTOBER 1996 1041

structure of the HMM for each letter pattern. The EG initially 6,(t)= maxS,_,(t- I), 6,(t - 1) + ~ ~ ( 0 , )
contains a degenerate grammar consisting of only a start node, an
end node and a single arc with a non-terminal label. During the We now explain how scores are propagated through grammar
evolution process, upon encountering a non-terminal arc, the arc is nodes. Suppose g is a grammar node and p(g) and s(g) denote the
first replaced with the subnetwork it represents and is examined sets of preceding and succeeding letter pattern classes corre-
again. This process continues until all nonterminal references on sponding to the incoming and outgoing arcs respectively. For each
the earliest arcs are eliminated. If a resulting label references an letter pattern class I, m(I) denotes the HMM used to model the
HMM, the appropriate model structure is built as indicated by the pattern; h(l) denotes the initial state of the model; and f(I) denotes
lexicon. Once all leading HMM references are built, HMM score the final state of the model. At each sample point t during the
integration proceeds. As a score emerges from an HMM and needs Viterbi search, the maximum of all the accumulated scores at the
to be propagated further in the network, additional evolution of final states of the preceding letter models, also called incoming
the EG may occur. In this way, only those regions of the grammar scores, is found and propagated to the initial state of each of the
touched by the HMM search are expanded. Beam search methods succeeding models, along with the corresponding state sequence.
[14] can be used to limit the amount of grammar expansion. The operation is carried out as follows:
Fig. 3 shows part of the evolutional grammar for a dictionary k = argmaxSf(,)(t- 1); (3)
containing the word "can" and how it evolves during decoding. Wg)
Labels in brackets are nonterminal labels and the others are termi- and for each statej = h(I); E s @:
nal labels. Labels composed of a single letter followed by different
digit subscripts refer to the separate HMMs for different patterns
of the same letter; "lgl" refers to the upward ligature model. Un-
labeled arcs are the null-arcs. For any given dictionary, a grammar
compiler [15] is used to convert a list of word specificationsinto an
optimized network with shared prefixes and suffixes. (5)

<Start>
0 =Q
2.4 Model Training

I
<a>
lg 1
Models are trained using the well known iterative segmental
training method ill]. Given a set of training samples, the HMM
for each sample is instantiated by concatenating HMMs for the
appropriate letters, ligatures and delayed strokes, which are in
turn instantiated by concatenating the composing stroke models.
The training procedure then is carried out through iterations of
segmentation of training samples by Viterbi algorithm using the
current model parameters, followed by parameter reestimation
using the means along the path. The iterative procedure stops
when the difference between the likelihood scores of the current
iteration and those of the previous one is smaller then a preset
threshold.
We have developed a training process composed of three con-
secutive stages. The initial parameters for each stage are the out-
put from the previous stage, while the initial parameters for the
first stage are obtained through equal-length segmentation of the
a
training samples. No manual segmentation is involved at any
Fig. 3. Partial diagrams of a EG network. stage.
The first stage-/etfer training, is carried out on isolated letter
2.3 Decoding samples including ligatures and delayed strokes. This stage essen-
The Viterbi algorithm is used to search for the most likely state tially serves as a model initializer-it is left to the later training
sequence corresponding to the given observation sequence and to stages to fully capture the characteristics of cursive handwriting
give the accumulated likelihood score along this best path [111. and variations among different writers. The model parameters
Suppose that for any state t , q,(t) denotes the selected state se- obtained are then passed on as initial parameters for the second
quence (hypothesis) leading to i at sample point t, and G(t) denotes stage of training-linear word training, which is carried out on
the accumulated log-likelihood score of that hypothesis. 0, repre- whole word samples. We call it linear because during this stage
sents the observation at sample point t, and A,(OJ represents the each word sample is bound to a single sequence of stroke models.
log-likelihood score of 0, in state i. In our current model, for effi- In other words, each sample is labeled not only by the corre-
ciency reasons, we assume that all the state-preserving probabili- sponding word, but also by the exact letter pattern sequence corre-
ties a,, are constant and therefore need not be included in the ac- sponding to the particular style of that sample, which is then con-
cumulated likelihood scores. (We have experimented with variable verted to a unique stroke sequence according to the lexicon. Such
state preserving probabilities and they showed no significant im- highly constrained training is necessary to obtain reliable results
provement in recognition performance over the constant ones.) when the models are not yet adequately trained. The disadvantage
Since each letter model is a left-to-right HMM with no state skip- is that the letter pattern sequence corresponding to each sample
ping, within each letter model the hypothesis and its likelihood are has to be manually composed, which is a demanding and error
updated as: prone process, especially when the training set is large.
This is the reason why we introduce the third training stage-
lattice word training. As the name suggests, during this stage each
word is represented by a lattice, or finite state network, that in-
cludes all possible stroke sequences that can be used to model the

Authorized licensed use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.
 1042 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996

word. The finite state network corresponding to each word is de-

scribed by a subgrammar. Each training sample is labeled only by
the word (or the index to the subgrammar representing the word). where E' = dP / d7 and IC' = d x / ds. Equation (6) defines an in-
The stroke sequence that best matches the sample is chosen by the variant feature which we shall refer to as normalized curvature.
decoding algorithm and the resulting segmentation is used for An other set of invariants that require lower orders of deriva-
parameter re-estimation. This stage of training involves minimal tives can be obtained by using the invariance of distance ratios
amount of human supervision, and therefore can conveniently between corresponding points. Consider again the two equivalent
accommodate a large amount of training data. Fig. 4 shows the
segmentation of a sample "rectangle" at the end of lattice word
curves P(t) and e(F)
defined above. Suppose P, and P, are two
points on curve P(t) whose tangent slope angles differ by 8, P is
training. The solid squares show the boundaries between letters
(including ligatures), and the stars indicate the boundaries be-
the intersection of the two tangents on P(t). Similarly, and F, 4
tween strokes. are two points on curve F(7) whose tangent slope angles also differ
by 0, and P" is the intersection of the two tangents on F(f) (Fig. 5).
n
Since angles and hence turns of the curve are invariant under the
similitude transformation, it can be shown that if point corre- 5
sponds to point P,, then points F, and P" correspond to points P,
w and P, respectively [161, thus lepl=clP,PI and lFF21=clPP,I.
Fig. 4. Segmentation of a training word sample Therefore we have:

3 INVARIANT
FEATURES (7)
In choosing handwriting features, we face the problem of variabil-
ity in handwriting caused by the geometric distortion of letters If we fix the value of 8 to a constant 8,, then (7) defines another
and words by rotation, scaling and translation. In general, transla- invariant feature which can be computed at each sample point. We
tion is not a serious problem because it is easy to chose features call this feature ratio of tangents. In order to enhance the distinctive
that are invariant with respect to translation. Examples include power of the feature, we augment it by the sign of the local cur-
handwriting stroke tangents and curvature. Unfortunately, these vature. The resulted feature is called signed ratio of tangents, and is
features are not invariant with respect to the other two factors. For used instead of ratio of tangents in the experiments described
example, stroke tangents are invariant with respect to scale, but later.
not rotation; curvature is invariant with respect to rotation, but not
scale.

p"'
There are two principal methods for dealing with variability in
pattern recognition. The patterns can be normalized before feature
extraction by some set of preprocessing transformations, or fea-
tures can be chosen to be insensitive to the undesirable variability.
These two methods often need to be combined because neither one
can solve the problem completely by itself. On one hand, excessive
preprocessing is undesirable because it may result in premature, U,
limiting decisions or loss of information. On the other hand, cer-
tain features that are not completely invariant (e.g., tangent slope
angle) prove to be important in distinguishing different symbols.
We have adapted one of the common features for handwriting Fig. 5. Ratio of tangents
recognition-tangent slope angle, which is invariant to translation
and scaling, but not rotation. In this section, we introduce two new To evaluate accurately the invariant features described above,
features for handwriting recognition that are invariant with re- high quality derivative estimates of up to the third order have to
spect to all three factors of geometric distortion. be obtained from the sample points. Obviously simple finite dif-
We define a similitude transformation to be a combination of ference based methods for derivative estimation do not provide
translation, rotation and scaling described by: the needed insensitivity to spatial quantization error or noise. We
have applied a set of smoothing spline operators of up to the fifth
order for this purpose 1101.
With the addition of the two invariant features the observation
where c is a positive scalar. Two curve segments are equivalent if
vector now contains three dimensions. Even though these vectors
they can be obtained from each other through a similitude trans-
are continuous in nature, we chose to use discrete HMMs instead
formation. Invariant features are features that have the same value
of continuous density H M M s to avoid making assumptions on
at corresponding points on different equivalent curve segments.
the form of the underlying distribution. To simplify our models,
Suppose that a smooth planar curve P(t) = (x(t), y(t)) is mapped
we also chose to treat the features as being independent from each
into F(F) = (x"(t),y"(t))
by a reparameterization t(F) and a simili- other. A separate distribution is estimated for each feature in each
tude transformation,i.e., e(F)
= c U P(t(?)) + v . Without loss of gen- state during training. The joint probability of observing symbol
erality, assume that both curves are parameterized by arc length vector S, = {kl,k2,k3]instatejis:
1 2 3
(natural parameter), i.e., t = s and F = 5. Obviously, dS = c d s , thus
the corresponding points on the two curves are related by bI(silir2k3) = f,=I
ibp(ki)r
z,)
F(S) = cUP((S- / c) + v . It can then be shown [161 that curva-
ture (the reciprocal of radius) at the corresponding points of the two where b#,) is the probability of observing symbol k, in state j ac-
curves is scaled by l/c, i.e., z(5) = +IC((?- So) / c). It follows that cording to the probability distribution of the ith feature. In order to
adjust the influence of different features according to their dis-

Authorized licensed use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.
 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996 1043

criminative power, we compute the weighted log-likelihood: where t, = t - d,& - 1) and d,& - 1) is the number of sample
points assigned to letter model m(l) (letter duration) up to sample
point t-1. Using these augmented scores, (3), (41, (5) are replaced
by the following:
derived form the weighted probabilities:

and for each state j = h(0; s(g):

where N,is the state normalization factor such that the weighted
probabilities satisfy the condition:

thus are not biased towards any particular state. The weights wi
are positive scalars and specify the relative dominance of each
feature. By augmenting the incoming scores at each grammar node
with the letter matching scores, the overall shapes of letters are
taken into consideration when the hypotheses leading to the
4 SEGMENTAL
FEATURES
grammar node are ranked and the one with the highest rank is
The HMM system described so far relies on localized features. chosen and propagated to the succeeding letter models. Through
During the decoding process, for each new sample data point this mechanism, letter matching scores computed on dynamically
taken in a time ordered sequence, the HMM hypotheses scores are allocated segments directly affect the decision making at each
discretely integrated and propagated through the HMM network. point during the Viterbi search, so that the system is biased to-
The incremental score at each step depends only on the local fea- wards sequences with better matches at the letter level.
ture at the current point. We will refer to methods of this type as It should be pointed out that in such an augmented HMM sys-
point oriented. The advantage of point oriented methods is that all tem, the state sequence resulted from Viterbi search is no longer
knowledge sources are integrated into a single model. Because of guaranteed to be the optimal sequence, because now the accumu-
this, all possible segmentations and identifications of the input lated score of a sequence leading to state i at sample point t not
pattern are considered in an efficient manner. On the other hand, only depends on the previous state, but also on how the previous
point oriented methods have the disadvantage of using only lo- state was reached (history reflected in the letter duration). This
calized observation measurements. Thus, shape information on dependence violates the basic condition for the Viterbi algorithm
larger scales is missing from the process. One way to remedy this to yield optimum solution. However, as shall be shown later, our
problem is to extract features from a window around each sample experimental results suggest that the gain obtained by incorporating
point, where the window could be of fixed (e.g., [17]) or variable segmental features by far outweighs the loss in the optimality of the
size (e.g., features in [61, the ratio of tangents feature described in algorithm.
the previous section, etc.). The weakness of this method is that it
The segmental matching score a (t,, t,) is computed using a cor-
does not adapt to the varying characteristics of pattern shapes,
relation based metric inspired by the metrics developed by Sinden
sizes, or segmentation boundaries.
There are methods that trade the efficiency advantage of point and Wilfong [18]. Given a segment U with coordinate sequence 17, =
oriented methods for the greater accuracy of measurements over <(q,y,), (x,, y,), ..., (xn,yn)>, the instance vector of a is defined as:
larger regions. These methods fall into the class called segment va = (TI, ij,, T2, ij,, ... ,X , , Y,) , where X, = x, - x, and ijt= y, - y,
oriented 141 where a script is first presegmented into letters or sub-
for 1 2 i 5 n, and (x~, ya) is the centroid of a. The normalized instance
character primitives according to certain predefined boundary con-
ditions (pen-ups, cusps, etc.) and one observation feature vector is vector of a, U, = va/ Inall is a translation and scale independent
computed for each segment. To avoid loosing potentially good hy- representation of segment a in R2" . Through a resampling proce-
potheses, segment oriented methods should first generate all possi- dure, a sample segment of arbitrary length can be mapped to a
ble segmentations because the scoring knowledge is not available at vector in Rm where N is a predetermined number. The difference
the time of segmentation and will be applied later as a post-process, between any two sample segments a and b is defined as: D(a, b) =
but in practices no system actually does this because the number of
possible segmentations makes the problem intractable. 0.5(1 - U, . U J ,whose value ranges from 0 (when a and b are iden-
We propose a new segment oriented method that ameliorates tical) to 1. The segmental matching score U$,, t2), is then defined
the usual tradeoff between efficiency and accuracy. We call this as: a,(t,,t,) = - ~ , D ( a ~ , a , , , ~,~where
) a, is the model segment for
the interleaved segmental method. Point oriented methods are used
to obtain partial segmentation hypotheses which are augmented letter pattern I, a, ,
1' 2
is the segment from sample point t, to t, on
with observation measurements made on the hypothesized seg-
the input sample sequence and w , is a weight factor.
ments. The resulted system is called an augmented HMM system.
In Section 2.3, we explained how hypotheses are propagated In order to compute the above segment matching score, a single
through a grammar node g in a commonly used implementation ((3), model segment needs to be derived for each letter pattern class.
(4), (5)).In order to incorporate seimenta1 shape information (in this Let ul, U,, . , uM be the normalized instance vectors of a set of
case letter shape information) into the search, we augment the in- prototypes for the letter pattern class I (which can be easily ob-
coming scores with letter matching scores, computed using global tained as side products of segmental training). A single model
letter shape models. To be more specific, let a$,, t,) be the likelihood segment representing this class is represented by vector w which
minimizes the sum of distances from individual prototypes. It can
score of the segment from sample point t, to t, on the observation M
sequence being matched to letter pattern class I, the augmented in- be easily shown that w = U"/ I GI, where U" = U, .
coming scores are defined as 6;(,,(t- 1) = 65(1,(t - 1)+ a,(t,,t- l), ,=I
Fig. 6 illustrates the effect of incorporating letter matching

Authorized licensed use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.
 1044 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. IO, OCTOBER 1996

scores in the HMM system by comparing the different letter level training, divided into two groups. Group A is the multiple-writer
segmentations obtained without and with letter matching scores. test group which contains 1,592 samples from the 10 training writ-
Fig. 6a shows the segmentation of a sample of word ”line” when ers, group B is the writer-independent test group composed of
the basic HMM system was used and the sample was falsely 3,823 samples from the eight other writers.
recognized as ”arc.” Fig. 6b shows the segmentation of the same Table 1 summarizes the performance of our recognition system.
sample when letter matching scores are applied and the sample is Error rates from three experiments as well as the settings for each
correctly recognized. The hypothesis shown in Fig. 6a is not se- experiment are listed. The top line of Fig. 7 shows some of the
lected now because the segment corresponding to letter “a” does samples from test set B that were not recognized correctly. In fact
not match the corresponding model segment well and therefore they are so sloppy that even human beings can hardly recognize
yields a poor letter matching score. them correctly. The bottom line of the same figure shows some of
the more legible samples from the same writer that were recog-
nized correctly.
_____
-.-. __...
,_____...
6
-..-.,~

1 - ligature
2 - letter “a.’ TABLE 1
3 - lfgature
-
5‘,

‘,, 4 latter “r” ERRORRATESFROM THREEEXPERIMENTS

n ”,,,.,
5
6
-
-
letter “c”
ligature
1
A (multiple-writer)
10.3%
B (writer-independent)
23.4%
,/- 5 ’,.~ 2 5.6% 10.7%
3 3.1% 5.5%
1. Basic HMM system with single tangent slope angle feature.
2. Basic HMM system with two additional, invaviant features: novmalized
curvature and signed ratio of tangents.
3. Augmented HMM system.

1 - ligature
2 - letter “1“
3 - ligature
4 -
letter “i“
5 - ligature
6 - letter “n”
7 - ligature
8 - letter “ e “
9 - ligature
10 - delayed stroke ”dot”

Fig. 7. Top: examples of scripts recognized incorrectly (“draw,” “circle,”

and “arrow”); bottom: examples of scripts recognized correctly (“size,”
Fig. 6. Different letter segmentations of a sample of word “line” “group,” and “rotate”).

5 EXPERIMENTAL
RESULTS REMARKS
6 CONCLUDING
To test our system, we composed a vocabulary of 32 words, target- We have described experiments in handwriting recognition using
ing an underlying application of a pen driven graphics editor. The hidden Markov modeling and stochastic language modeling meth-
vocabulary covers all 26 lower case letters in the English alphabet, ods originally developed for speech applications. These methods are
contains many groups of easily confused words, such as ”line,” generalized in the AEGIS architecture. Subcharacter models called
”lines” and ”spline,” ”cut” and ”out,” and has both very short words nebulous stroke models are used to model the basic units in hand-
such as “in” and relatively long ones such as ”rectangle.” Samples writing. We introduce two new features for handwriting that are
from 18 writers have been collected, the writer group containing invariant under translation, rotation and scaling. Invariant features
men and women, left-handed and right-handed, of many different have been discussed extensively in computer vision literature. How-
cultural origins: American, European, Asian, and South American. ever, they have been rarely used previously in real applications due
During sample collection, the writers were told to write in their most to the difficulty involved in the estimation of high order derivatives.
natural way with no explicit constraints. Each word was written 15 We have demonstrated that these high order invariant features can
times by each writer. After removing invalid samples (samples with indeed be made useful with careful implementation.
misspelling or parts missing due to hardware problems), the final A method for combining segment oriented features in a sto-
data set consists of 8,595 word samples. chastic pattern recognizer has been developed. In this method,
The isolated letter samples used for letter training were all called interleaved segmental method, partial segmentation hy-
written by one writer, imitating all writing styles known to the potheses obtained using the point oriented features in a conven-
authors. 10-15 samples were written for each unique style of each tional dynamic programming search are combined with scores
letter and a particular stroke sequence is bound to those samples. based on segmental shape measurements made on the hypothe-
There are all together 54 lower-case letter models (as sequences of sized segments. Although certain optimality characteristics of the
stroke models), including delayed strokes and ligatures, composed HMM system are sacrificed in the process, significant reduction in
of a total of 93 stroke models. Each stroke is currently modeled by recognition error was achieved by this method, reducing writer-
a single state. independent error rate by nearly 50%.
Ten writers were chosen (after data collection) to be the Finally, we would like to point out that although we report
”training writers.” The whole word training set is composed of only recognition results on a relatively small vocabulary of 32
about 10 samples of each word from each training writer, a total of words in this paper, none of the techniques presented is inherently
3,180 samples. About 600 of the 3,180 training samples are used for dependent on the size of the vocabulary. The system can be easily
linear word training, and the whole training set is used for lattice adjusted to handle a large or unlimited vocabulary by imposing
word training. The test set is composed of all samples not used for different grammatical constraints. For example, one could use a

Authorized licensed use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.
 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. IO, OCTOBER 1996 1045

statistical N-gram grammar instead of a dictionary to allow an

A New Methodology for Gray-Scale
unlimited vocabulary. Experiments are to be carried out in the
future to test the system’s performance on large and unlimited Character Segmentation and Recognition
vocabularies.
Seong-Whan Lee, Member, IEEE Computer Society,
Dong-June Lee, Member, IEEE, and
REFERENCES Hee-Seon Park, Member, IEEE
I11 C.C. Tappert, C.Y. Suen, and T. Wakahara, ”The State of the Art
in On-Line Handwriting Recognition,” IEEE Trans. Pattern Analy- Abstract-Generally speaking, through the binarization of gray-scale
sis and Machine Intelligence, vol. 12,no. 8, pp. 787-808, Aug. 1990. images, useful information for the segmentation of touched or
[21 R. Nag, K.H. Wong, and F. Fallside, “Script Recognition Using overlapped characters may be lost in many cases. If we analyze gray-
Hidden Markov Models,” Proc. TCASSP ’86, vol. 3, pp. 2,071-2,074, scale images, however, specific topographic features and the variation
Japan, Apr. 1986. of intensities can be observed in the character boundaries. We believe
[31 A. Kundu and P. Bahl, ”Recognition of Handwritten Script: A that such kinds of clues obtained from gray-scale images may work for
Hidden Markov Model Based Approach,” Proc. ICASSP ’88, vol. efficient character segmentation and recognition. In this paper, we
2, pp. 928-931, New York, Apr. 1988. propose a new methodology for character segmentation and
[41 S. Bercu and G. Lorette, ”On-Line Handwritten Word Recogni- recognition which makes the best use of the characteristics of gray-
tion: An Approach Based on Hidden Markov Models,” Proc. Third scale images. In the proposed methodology, the character
IWFHR, pp. 385-390, Buffalo, N.Y., May 1993. segmentation regions are determined by using projection profiles and
151 J. Makhoul, T. Starner, R. Schartz, and G. Lou, ”On-Line Cursive topographic features extracted from the gray-scale images. Then a
Handwriting Recognition Using Speech Recognition Models,” nonlinear character segmentation path in each character segmentation
Proc. ICASSP ’94, pp. v125-v128, Adelaide, Australia, Apr. 1994. region is found by using multi-stage graph search algorithm. Finally, in
[6] K.S. Nathan, H.S. M. Beigi, J. Subrahmonia, G.J. Clary, and H. order to confirm the nonlinear character segmentation paths and
Maruyama, “Real-Time On-Line Unconstrained Handwriting Rec- recognition results, recognition-basedsegmentation method is
ognition Using Statistical Methods,” Proc. ICASSP ’95, pp. 2,619,- adopted. Through the experiments with various kinds of printed
2,622, Detroit, Mich., June 1995. documents, it is convinced that the proposed methodology is very
171 M.Y. chen, A. Kundu, and J. zhou, ”Off-Line Handwritten Word effective for the segmentation and recognition of touched and
Recognition Using a Hidden Markov Model Type Stochastic overlapped characters.
Network,” I E E E Trans. Pattern Analysis and Machine Intelligence,
vol. 16, no. 5, pp. 481-496, May .1994. Index Terms-Character segmentation and recognition, topographic
[81 M.K. Brown and S.C. Glinski, ”Stochastic Context-Free Language feature, gray-scale character recognition, multistage graph search,
Modeling with Evolutional Grammars,” Proc. ICSLP ’94, vol. 2, recognition-basedsegmentation.
pp. 779-782, Yokohama, Japan, Sept. 1994.
191 G.L. Miller and R. Boie, “Capacitive Proximity Sensors,” U.S. 4
Patent #5,337,353,9, Aug. 1994.
I101 J. Hu, M.K. Brown, and W. Turin, “Invariant Features for HMM 1 INTRODUCTION
Based Handwriting Recognition,” Proc. ICIAP ’95, pp. 588-593,
Sanremo, Italy, Sept. 1995. IT is a challengeable issue to develop a practical system which can
1111 L.R. Rabiner and B.H. Juang, Fundamentals of Speech Recognition. maintain a high recognition accuracy, independent of the quality
Prentice Hall, 1993. of the input documents and the character fonts. Very often even in
[121 L.R. Rabiner, “A Tutorial on Hidden Markov Models and Selected printed text, adjacent characters tend to be touched or overlapped.
Applications in Speech Recognition,” Proc. IEEE, vol. 77, no. 2,
Feb. 1989. Therefore, it is essential to segment a given string correctly into its
[13] S.A. Guberman and V.V. Rozentsveig, ”Algorithm for the Recog- character components. Any failure or error in this segmentation
nition of Handwritten Text,” Automation and Remote Control, vol. step produces a negative effect on character recognition Ill.
37, pp. 751-757, May 1976. (Translated from Automatika i Telemek- The complexity of character segmentation stems from the wide
hanika, vol. 37, no. 5, pp. 122-129,May 1976.) variety of fonts, rapidly expanding text styles, and image charac-
1141 B.T. Lowerre and D.R. Reddy, ”The HARPY Speech Understanding teristics such as poor-quality printing and poor binary images.
System,” Trends in Speech Recognition, W.A. Lean, ed., chapter 15,
pp. 340-360. Prentice Hall, 1980. Touched, overlapped, separated, and broken characters are major
[15] M.K. Brown and J.G. Wilpon, “A Grammar Compiler for Con- factors for causing segmentation errors. Moreover, when a docu-
nected Speech Recognition,’’ IEEE Trans. Signal Processing, vol. 39, ment is composed of multiple languages, (e.g., Hangul with al-
no. 1, pp. 17-28, Jan. 1991. phanumeric characters), it is more difficult to segment characters
[16] A.M. Bruckstein, R.J. Holt, AN. Netravali, and T.J. Richardson, due to differences in character sizes and touching types of each
”Invariant Signatures for Planar Shape Recognition Under Partial language.
Occlusion,” CVGIP: Image Understanding, vol. 58, pp. 49-65, July
1993. Previous methods for character segmentation can be roughly
L171 M. Schenkel, 1. Guyon, and D. Henderson, ”On-Line Cursive classified into three categories: straight segmentation method, rec-
Script Recognition Using Time Delay Neural Networks and Hid- ognition-based segmentation method, and cut classification method.
den markov Models,” Special Issue of Machine Vision and Applica-
tion on Cursive Script Recognition, R. Plamondon, ed. Springer
Verlag, 1995.
1181 F. Sinden and G. Wilfong, ”Method of Recognizing Handwritten S.-W. Lee is with the Dept. of Computer Science and Engineering, Korea Uni-
Symbols,” US. Patent #5,333,209, July 26,1994. versity, Anum-dong, Seongbuk-ku, Seoul 236-701, Korea.
E-mail: sw1eeQhuman.korea.ac.kr.
D.-J. Lee is with the TelecommunicationNetwork Research Lab., Korea Telecom,
Woomyun-dong, Suhcho-ku, Seoul 137-792, Korea
E-mail: djlee@tiXermask.kotel.co.kv.
H.-S. Park is with the Multimedia Lab, Samsung Electvonics Co. Ltd., Suwon
P.O. Box 105, Kyungki-do 440-600, Korea.
E-mail: hspark@coda.info.samsung.co.kr.
Manuscript received Mar. 13, 1995; revised Dec. 26, 1995. Recommended for
acceptance by J.J. Hull.
For information on obtaining reprints of this article, please send e-mail to:
transpami@computer.org, and reference I E E E C S Log Number P96081.

0162-8828/96$05.00 0 1996 IEEE

Authorized licensed
View publication stats use limited to: KnowledgeGate from IBM Market Insights. Downloaded on December 11, 2008 at 16:44 from IEEE Xplore. Restrictions apply.