Language Modeling

Mausam
(Based on slides of Michael Collins, Dan Jurafsky, Dan Klein,
Chris Manning, Luke Zettlemoyer)
Outline
• Motivation
• Task Definition
• N-Gram Probability Estimation
• Evaluation
• Hints on Smoothing for N-Gram Models
• Simple
• Interpolation and Back-off
• Advanced Algorithms
2
The Language Modeling Problem
 Setup: Assume a (finite) vocabulary of words

 We can construct an (infinite) set of strings

 Data: given a training set of example sentences

 Problem: estimate a probability distribution
The Noisy-Channel Model
• We want to predict a sentence given acoustics:

• The noisy channel approach:

Acoustic model: Distributions Language model:

over acoustic waves given a Distributions over sequences
sentence of words (sentences)
Acoustically Scored Hypotheses

the station signs are in deep in english -14732

the stations signs are in deep in english -14735
the station signs are in deep into english -14739
the station 's signs are in deep in english -14740
the station signs are in deep in the english -14741
the station signs are indeed in english -14757
the station 's signs are indeed in english -14760
the station signs are indians in english -14790
the station signs are indian in english -14799
the stations signs are indians in english -14807
the stations signs are indians and english -14815
ASR System Components
Language Model Acoustic Model

source channel
w a
P(w) P(a|w)

best observed
decoder
w a

argmax P(w|a) = argmax P(a|w)P(w)

w w
MT System Components
Language Model Translation Model

source channel
e f
P(e) P(f|e)

best observed
decoder
e f

argmax P(e|f) = argmax P(f|e)P(e)

e e
Probabilistic Language Models: Other Applications
• Why assign a probability to a sentence?
• Machine Translation:
• P(high winds tonite) > P(large winds tonite)
• Speech Recognition
• P(I saw a van) >> P(eyes awe of an)
• Spell Correction
• The office is about fifteen minuets from my house
• P(about fifteen minutes from) > P(about fifteen minuets from)
• + Summarization, question-answering, etc., etc.!!
Outline
• Motivation
• Task Definition
• N-Gram Probability Estimation
• Evaluation
• Hints on Smoothing for N-Gram Models
• Simple
• Interpolation and Back-off
• Advanced Algorithms
11
Probabilistic Language Modeling
• Goal: compute the probability of a sentence or
sequence of words:
P(W) = P(w1,w2,w3,w4,w5…wn)

• Related task: probability of an upcoming word:

P(w5|w1,w2,w3,w4)
• A model that computes either of these:
P(W) or P(wn|w1,w2…wn-1) is called a language model.
How to compute P(W)
• How to compute this joint probability:

• P(its, water, is, so, transparent, that)

P(“its water is so transparent”) =

P(its) × P(water|its) × P(is|its water)
× P(so|its water is) × P(transparent|its water is so)
How to estimate these probabilities
• Could we just count and divide?

P(the | its water is so transparent that) =

Count(its water is so transparent that the)
Count(its water is so transparent that)

• No! Too many possible sentences!

• We’ll never see enough data for estimating these
 Markov Assumption

• Simplifying assumption:
Andrei Markov

P(the | its water is so transparent that) » P(the | that)

• Or maybe
P(the | its water is so transparent that) » P(the | transparent that)
 Markov Assumption

… wn )   P ( wi | wi  k …
P( w1w2   wi 1 )
i

• In other words, we approximate each

component in the product
… w )  P( w | w
P( wi | w1w2  … w )

i 1 i i k i 1
Simplest Case: Unigram Models
• Simplest case: unigrams
… wn )   P ( wi )
P( w1w2 
i
• Generative process: pick a word, pick a word, … until you pick </s>
• Graphical model:
w1 w2 …………. wn-1 </s>
• Examples:
• fifth, an, of, futures, the, an, incorporated, a, a, the,
inflation, most, dollars, quarter, in, is, mass
• thrift, did, eighty, said, hard, 'm, july, bullish
• that, or, limited, the

• Big problem with unigrams: P(the the the the) >> P(I like ice cream)!
Bigram Models
• Conditioned on previous single word

… wi 1 )  P ( wi | wi 1 )
P( wi | w1w2 
• Generative process: pick <s>, pick a word conditioned on previous one,
repeat until to pick </s>

• Graphical model: <s> w1 w2 wn-1 </s>

• Examples:
• texaco, rose, one, in, this, issue, is, pursuing, growth, in, a,
boiler, house, said, mr., gurria, mexico, 's, motion, control,
proposal, without, permission, from, five, hundred, fifty, five,
yen
• outside, new, car, parking, lot, of, the, agreement, reached
• this, would, be, a, record, november
N-Gram Models
• We can extend to trigrams, 4-grams, 5-grams
• N-gram models are (weighted) regular languages
• Many linguistic arguments that language isn’t regular.
• Long-distance effects: “The computer which I had just put into the
machine room on the fifth floor ___.”
• Recursive structure
• We often get away with n-gram models

• PCFG LM (later):
• [This, quarter, ‘s, surprisingly, independent, attack, paid, off,
the, risk, involving, IRS, leaders, and, transportation, prices, .]
• [It, could, be, announced, sometime, .]
• [Mr., Toseland, believes, the, average, defense, economy, is,
drafted, from, slightly, more, than, 12, stocks, .]
 An example

<s> I am Sam </s>

c(wi-1,wi )
P(wi | w i-1 ) = <s> Sam I am </s>
c(wi-1) <s> I do not like green eggs and ham </s>
 More examples:
Berkeley Restaurant Project sentences

• can you tell me about any good cantonese restaurants close by

• mid priced thai food is what i’m looking for
• tell me about chez panisse
• can you give me a listing of the kinds of food that are available
• i’m looking for a good place to eat breakfast
• when is caffe venezia open during the day
 Raw bigram counts
• Out of 9222 sentences
Raw bigram probabilities
• Normalize by unigrams:

• Result:
What kinds of knowledge?
• P(english|want) = .0011
World knowledge
• P(chinese|want) = .0065
• P(to|want) = .66
• P(eat | to) = .28
• P(food | to) = 0 Grammatical knowledge

• P(want | spend) = 0
• P (i | <s>) = .25
Practical Issues
• We do everything in log space
• Avoid underflow
• (also adding is faster than multiplying)

log(p1 ´ p2 ´ p3 ´ p4 ) = log p1 + log p2 + log p3 + log p4

Language Modeling Toolkits
• SRILM
• http://www.speech.sri.com/projects/srilm/
 Google N-Gram Release, August 2006

…
 Google N-Gram Release
• serve as the incoming 92
• serve as the incubator 99
• serve as the independent 794
• serve as the index 223
• serve as the indication 72
• serve as the indicator 120
• serve as the indicators 45
• serve as the indispensable 111
• serve as the indispensible 40
• serve as the individual 234

http://googleresearch.blogspot.com/2006/08/all-our-n-gram-are-belong-to-you.html
Outline
• Motivation
• Task Definition
• N-Gram Probability Estimation
• Evaluation
• Hints on Smoothing for N-Gram Models
• Simple
• Interpolation and Back-off
• Advanced Algorithms
42
Evaluation: How good is our model?
• Does our language model prefer good sentences to bad ones?
• Assign higher probability to “real” or “frequently observed” sentences
• Than “ungrammatical” or “rarely observed” sentences?
• We train parameters of our model on a training set.
• We test the model’s performance on data we haven’t seen.
• A test set is an unseen dataset that is different from our training set,
totally unused.
• An evaluation metric tells us how well our model does on the test set.
Extrinsic evaluation of N-gram models
• Best evaluation for comparing models A and B
• Put each model in a task
• spelling corrector, speech recognizer, MT system
• Run the task, get an accuracy for A and for B
• How many misspelled words corrected properly
• How many words translated correctly
• Compare accuracy for A and B
Difficulty of extrinsic (in-vivo) evaluation of N-
gram models
• Extrinsic evaluation
• Time-consuming; requires building applications, new data
• So
• Sometimes use intrinsic evaluation: perplexity
• Bad approximation
• unless the test data looks just like the training data
• So generally only useful in pilot experiments
• But is helpful to think about.
 Intuition of Perplexity
mushrooms 0.1
• The Shannon Game:
• How well can we predict the next word? pepperoni 0.1
anchovies 0.01
I always order pizza with cheese and ____
….
The 33rd President of the US was ____
fried rice 0.0001
I saw a ____ ….
• Unigrams are terrible at this game. (Why?) and 1e-100
• A better model of a text
• is one which assigns a higher probability to the word that actually occurs
 Perplexity
The best language model is one that best predicts an unseen test set
• Gives the highest P(sentence) -
1
PP(W ) = P(w1w2 ...wN ) N
Perplexity is the inverse probability of
the test set, normalized by the number
1
of words: = N
P(w1w2 ...wN )

Chain rule:

For bigrams:

Minimizing perplexity is the same as maximizing probability

The Shannon Game intuition for perplexity
• From Josh Goodman
• How hard is the task of recognizing digits ‘0,1,2,3,4,5,6,7,8,9’
• Perplexity 10
• How hard is recognizing (30,000) names at Microsoft.
• Perplexity = 30,000
• If a system has to recognize
• Operator (1 in 4)
• Sales (1 in 4)
• Technical Support (1 in 4)
• 30,000 names (1 in 120,000 each)
• Perplexity is 53
• Perplexity is weighted equivalent branching factor
Perplexity as branching factor
• Let’s suppose a sentence consisting of random digits
• What is the perplexity of this sentence according to a model
that assign P=1/10 to each digit?
Another form of Perplexity

• Lower is better!
• Example:
• uniform model  perplexity is N
• Interpretation: effective vocabulary size (accounting for statistical regularities)
• Typical values for newspaper text:
• Uniform: 20,000; Unigram: 1000s, Bigram: 700-1000, Trigram: 100-200
• Important note:
• Its easy to get bogus perplexities by having bogus probabilities that sum to
more than one over their event spaces. Be careful!
Lower perplexity = better model

• Training 38 million words, test 1.5 million words, WSJ

N-gram Unigram Bigram Trigram

Order
Perplexity 962 170 109
Outline
• Motivation
• Task Definition
• Probability Estimation
• Evaluation
• Smoothing
• Simple
• Interpolation and Back-off
• Advanced Algorithms
78
The Shannon Visualization Method
• Choose a random bigram
<s> I
(<s>, w) according to its probability I want
• Now choose a random bigram want to
(w, x) according to its probability to eat
• And so on until we choose </s> eat Chinese
• Then string the words together Chinese food
food </s>
I want to eat Chinese food
Approximating Shakespeare
Shakespeare as corpus
• N=884,647 tokens, V=29,066
• Shakespeare produced 300,000 bigram types
out of V2= 844 million possible bigrams.
• So 99.96% of the possible bigrams were never seen
(have zero entries in the table)
• Quadrigrams worse: What's coming out looks
like Shakespeare because it is Shakespeare
The perils of overfitting
• N-grams only work well for word prediction if the test
corpus looks like the training corpus
• In real life, it often doesn’t
• We need to train robust models that generalize!
• One kind of generalization: Zeros!
• Things that don’t ever occur in the training set
• But occur in the test set
Unknown words: Open vs closed vocabulary tasks
• If we know all the words in advanced
• Vocabulary V is fixed
• Closed vocabulary task
• Often we don’t know this
• Out Of Vocabulary = OOV words
• Open vocabulary task
• Instead: create an unknown word token <UNK>
• Training of <UNK> probabilities
• Create a fixed lexicon L of size V
• At text normalization phase, any training word not in L changed to <UNK>
• Now we train its probabilities like a normal word
• At decoding time
• If text input: Use UNK probabilities for any word not in training
 Zeros
• Training set: • Test set
… denied the allegations … denied the offer
… denied the reports … denied the loan
… denied the claims
… denied the request

P(“offer” | denied the) = 0

Zero probability bigrams
• Bigrams with zero probability
• mean that we will assign 0 probability to the test set!
• And hence we cannot compute perplexity (can’t divide by 0)!
 The intuition of smoothing
• When we have sparse statistics:
P(w | denied the)

allegations
3 allegations

outcome
reports
2 reports

attack
…

request
claims
1 claims

man
1 request
7 total
• Steal probability mass to generalize better
P(w | denied the)
2.5 allegations

allegations
allegations
1.5 reports

outcome
0.5 claims

reports

attack
0.5 request
…

man
claims

request
2 other
7 total
 Add-one estimation

• Also called Laplace smoothing

• Pretend we saw each word one more time than we did
• Just add one to all the counts!
c(wi-1, wi )
PMLE (wi | wi-1 ) =
• MLE estimate: c(wi-1 )
c(wi-1, wi ) +1
• Add-1 estimate: PAdd-1 (wi | wi-1 ) =
c(wi-1 ) +V
Berkeley Restaurant Corpus: Laplace
smoothed bigram counts
Laplace-smoothed bigrams
Reconstituted counts
Compare with raw bigram counts
More general formulations: Add-k

c(wi-1, wi ) + k
PAdd-k (wi | wi-1 ) =
c(wi-1 ) + kV
1
c(wi-1, wi ) + m( )
PAdd-k (wi | wi-1 ) = V
c(wi-1 ) + m
What counts do we want?
Count c New count c*
0 .0000270
1 0.446
2 1.26
3 2.24
4 3.24
5 4.22
6 5.19
7 6.21
8 7.24
9 8.25
Absolute Discounting
• Just subtract 0.75 (or some d)!
discounted bigram
c( wi 1 , wi )  d
PAbsoluteDiscounting ( wi | wi 1 ) 
c( wi 1 )
(Maybe keeping a couple extra values of d for counts 1 and 2)

• Problem: all unknown bigrams are equally likely!

96
Outline
• Motivation
• Task Definition
• Probability Estimation
• Evaluation
• Smoothing
• Simple
• Interpolation and Back-off
• Advanced Algorithms
97
Backoff and Interpolation
• Sometimes it helps to use less context
• Condition on less context for contexts you haven’t learned much about
• Backoff:
• use trigram if you have good evidence,
• otherwise bigram, otherwise unigram
• Interpolation:
• mix unigram, bigram, trigram

• Interpolation often works better

Backoff
• Define the words into seen and unseen

• Backoff
 c( wi 1 , wi )
 wi  Α( wi 1 )
PBO ( wi | wi 1 )   c( wi 1 )
 P( wi ) wi  B( wi 1 )

• Problem?
• Not a probability distribution
99
Katz Backoff
c( wi 1 , wi )
PML ( wi | wi 1 )  P * ( wi | wi 1 )  PML ( wi | wi 1 )
c( wi 1 )
• Define the words into seen and unseen
A(v)  {w : c(v, w)  k} B(v)  {w : c(v, w)  k}
• Backoff
 P * ( wi | wi 1 ) wi  Α( wi 1 )
PBO ( wi | wi 1 )  
 ( wi 1 ) P( wi ) wi  B( wi 1 )
1  P * (w | w
wA ( wi 1 )
i 1 )
 ( wi 1 ) 
 P(w)
wB ( wi 1 )
Linear Interpolation
• Simple interpolation

• Lambdas conditional on context:

How to set the lambdas?
• Use a held-out corpus
Held-Out Test
Training Data Data Data
• Choose λs to maximize the probability of held-out data:
• Fix the N-gram probabilities (on the training data)
• Then search for λs that give largest probability to held-out set:

log P(w1...wn | M(l1...lk )) = å log PM ( l1... lk ) (wi | wi-1 )

i
Absolute Discounting Interpolation

discounted bigram Interpolation weight

c( wi 1 , wi )  d
PAbsoluteDiscounting ( wi | wi 1 )    ( wi 1 ) P( wi )
c( wi 1 )
unigram

• But should we really just use the regular unigram P(w)?

117
 Kneser-Ney Smoothing I
• Better estimate for probabilities of lower-order unigrams!
Francisco
glasses
• Shannon game: I can’t see without my reading___________?
• “Francisco” is more common than “glasses”
• … but “Francisco” always follows “San”
• The unigram is useful exactly when we haven’t seen this bigram!
• Instead of P(w): “How likely is w”
• Pcontinuation(w): “How likely is w to appear as a novel continuation?
• For each word, count the number of bigram types it completes
• Every bigram type was a novel continuation the first time it was seen
PCONTINUATION (w)µ {wi-1 : c(wi-1, w) > 0}
 Kneser-Ney Smoothing II
• How many times does w appear as a novel continuation:
PCONTINUATION (w)µ {wi-1 : c(wi-1, w) > 0}

• Normalized by the total number of word bigram types

{(w j-1, w j ) : c(w j-1, w j ) > 0}

{wi-1 : c(wi-1, w) > 0}

PCONTINUATION (w) =
{(w j-1, w j ) : c(w j-1, w j ) > 0}
 Kneser-Ney Smoothing III
• Alternative metaphor: The number of # of word types seen to precede w

| {wi-1 : c(wi-1, w) > 0} |

• normalized by the # of words preceding all words:

{wi-1 : c(wi-1, w) > 0}

PCONTINUATION (w) =
å {w' i-1 : c(w'i-1, w') > 0}
w'

• A frequent word (Francisco) occurring in only one context (San) will have a
low continuation probability
Kneser-Ney Smoothing IV

max(c(wi-1, wi ) - d, 0)
PKN (wi | wi-1 ) = + l (wi-1 )PCONTINUATION (wi )
c(wi-1 )
λ is a normalizing constant; the probability mass we’ve discounted

d
l (wi-1 ) = {w : c(wi-1, w) > 0}
c(wi-1 )
The number of word types that can follow wi-1
the normalized discount = # of word types we discounted
121 = # of times we applied normalized discount
 Kneser-Ney Smoothing: Recursive
formulation

i
max(c (w i-n+1 ) - d, 0)
i-1
PKN (wi | wi-n+1 ) = KN
i-1
+ l (w i-1
)P
i-n+1 KN (wi | w i-1
i-n+2 )
cKN (wi-n+1 )

where
max(c(wi-1, wi ) - d, 0)
PKN (wi | wi-1 ) = + l (wi-1 )PCONTINUATION (wi )
c(wi-1 )
122
What Actually Works?
• Trigrams and beyond:
• Unigrams, bigrams generally
useless
• Trigrams much better (when
there’s enough data)
• 4-, 5-grams really useful in MT,
but not so much for speech

• Discounting
• Absolute discounting, Good-
Turing, held-out estimation,
Witten-Bell, etc…

• See [Chen+Goodman] reading for

[Graphs from
tons of graphs… Joshua Goodman]
 Data vs. Method?

10
9.5 100,000 Katz
• Having more data is 9 100,000 KN
better…
8.5 1,000,000 Katz
• … but so is using a

Entropy
better estimator 8 1,000,000 KN
7.5 10,000,000 Katz

• Another issue: N > 3 has 7 10,000,000 KN

huge costs in speech 6.5 all Katz
recognizers 6 all KN
5.5
1 2 3 4 5 6 7 8 9 10 20
n-gram order