Text Generation

using Markov Chain

Model
By :-
● Abhinav Sagar (2019133)
● Ishaan Jindal (2019246)
● Krishna Nimmagadda (2019250)
● Madhav Rangaiah (2019251)
● Pranay Bhardwaj (2019263)
 Origin Of Markov Chain
● Founded by A. A. Markov, a Russian mathematician in 1913
● Found this new branch of probability theory by applying mathematics to poetry,
namely on Alexander Pushkin’s poem ‘Eugene Onegin’.
● He studied a sequence of 20,000 letters in the poem and discovered probability of
certain events
○ Probability of two consecutive vowels is 0.128
○ Probability of a vowel following a consonant is 0.663
● His finding proved that dependent variables can also converge to predictable
distributions
 Applications Of Markov Chain

● Weather forecasting
● Google’s PageRank Algorithm
● Computer Performance Evaluation by A.L. Scherr
● Wide number of used in finance industry
○ Predicting market trends
○ Credit risk measurement
○ Valuation of Stocks
● Text Generation using NLP
Application - NLP and Text-Generation

Natural Language Processing (NLP) is a popular and fast growing branch of AI. Its aim is to
read, decipher, understand, and make sense of the human languages in a manner that is
valuable. One such application is Text Generation.

The goal of text generation is to create a predictive text or an auto-generated text based on
the previous typed sequence or word. The easiest example of text generation is the
predictive text when you type in the search tab of Google or when you write an email. This
model is trained using markov chain.
 Implementation Using Markov
Chain Model
● In order to generate text using Markov Chain model, we need to define a few things:
○ What are our states going to be?
○ What probabilities will we assign while jumping from each state to a different one?
● The states are decided from the given input of data sample or corpus
○ Depend on the length of given input token
○ Can be based on character length or word length
● The transition matrix is constructed by parsing the data sample and
○ Counting the number of occurrences of events defined by state space
○ Normalizing the matrix to compute the corresponding probabilities
 Text Generation Using Markov Chain

● Mainly two types of models - 1. Character Based 2. Word Based

● Character Based - We define our the state space based on characters of the input
text and then based on the current state, the model tries to predict next character.
● Word Based - Instead of characters the model uses words from the input text as the
state space and then it predicts the next words according to their probabilities.
 Character Based Text Generation

● Training the model with the sample words and then model will learn patterns in the
ordering of characters.
● After the training of the model on the sample data when the model is given some
incomplete words then it will output the next character according to their probabilities.

Fig: Example of Character Based Text

Generation
 Word Based Text Generation
● First the model is provided the input text. It will take k consecutive words as one state
and the next word combined with the previous k - 1 words will works as the next
state.
● Text Generation where the last k consecutive words are used to predict the next word
are called k-gram generators
● Based on the state space, the transition matrix will be formed.
● After training of the data model it will be given a sequence of k-words and then it can
predict the next few words based on the data sample used in training.
● For Example - Input given to the model is: “We few, we happy few, we band of
brothers.” — Henry V by Shakespeare
 Example: Training Markov chain
Let’s imagine our dataset has the following sentence: This words sentence has have seven words.

We will first choose k: the number of words our chain will consider before predicting the next one. For this
example, let’s use k=2.

Now, how many distinct sequences of 2 word does our sentence have? It has 6, one for each pair of words.
We will first initialize a 6(sequences)×6(total distinct words/sequences) matrix of zeros. After that, we will
add 1 to the column corresponding to ‘words sentence’ on the row for ‘this words’. Then another 1 on the row
for ‘words sentence’, on the column for ‘sentence has’. We will continue this process until we’ve gone through
the whole sentence. This would be the resulting matrix:

Since each sequence only appears once, this model would simply generate the same sentence over and over,
but on adding more words/sequences could make this interesting.
Model’s efficiency with varying k

● The efficiency of the model differs for different values of k. Specifically, the efficiency
of the model increases with increasing k.
● For k=1, the number of words in the states is 1. So, the range of states for which Pij >
0 is huge as the corpus is huge. So, even though two consecutive words would make
sense, the entire text generated would have little to no meaning.
● For k=2, the text generated would be more sensible as probability values in Pi vector
would be less distributed.
● However, for higher values of k (k=5, 6, …), the text generated would become
deterministic as most values in Pi vector would be 0. This would make the text
predictable.
Efficiency of the Model

● The major issue with text generation using MC is that the text generated is only
dependent on the previous state. Due to which the text generated isn’t always
grammatically or semantically correct.
● There are several Deep Learning Models which have high efficiency with text
generation. However, most of those models require large amounts of data to train the
model, which requires high computational power.
● MC models in combination with Context Free Grammar: It was observed(Sentence
Generation Using Selective Text Prediction (scielo.org.mx)) that with a combination of
CFG in MC models for text generation, the output generated was semantically and
grammatically accurate.
Conclusion

● Text generation models based on MCs and CFGs had high efficiency while not
requiring large amounts of data.
● These in combination with Deep Learning techniques could further increase the
accuracy of the text generation models.
References

● http://www.scielo.org.mx/scielo.php?pid=S1405-55462019000300991&script=sci_arttext
● https://towardsdatascience.com/markov-chains-how-to-train-text-generation-to-write-like-g
eorge-r-r-martin-cdc42786e4b6
● https://langvillea.people.cofc.edu/MCapps7.pdf
● https://medium.com/future-vision/markov-chains-a-simple-powerful-tool-11e5b733045d
● https://searchengineland.com/what-is-google-pagerank-a-guide-for-searchers-webmasters-
11068
● https://www.educative.io/blog/deep-learning-text-generation-markov-chains