ARTIFICIAL INTELLIGENCE

QUESTION BANK – CLASS 10

CHAPTER 7: NATURAL LANGUAGE PROCESSING

One (01) Mark Questions

1. What is a Chabot?
A chatbot is a computer program that's designed to simulate human conversation
through voice commands or text chats or both. Eg: Mitsuku Bot, Jabberwacky etc.
OR
A chatbot is a computer program that can learn over time how to best interact with
humans. It can answer questions and troubleshoot customer problems, evaluate and
qualify prospects, generate sales leads and increase sales on an ecommerce site.

2. While working with NLP what is the meaning of?

a. Syntax
b. Semantics
Syntax: Syntax refers to the grammatical structure of a sentence.
Semantics: It refers to the meaning of the sentence.

3. What is the difference between stemming and lemmatization?

Stemming is a technique used to extract the base form of the words by removing affixes
from them. It is just like cutting down the branches of a tree to its stems. For example,
the stem of the words eating, eats, eaten is eat.
Lemmatization is the grouping together of different forms of the same word. In search
queries, lemmatization allows end users to query any version of a base word and get
relevant results.
OR
Stemming is the process in which the affixes of words are removed and the words are
converted to their base form.

In lemmatization, the word we get after affix removal (also known as lemma) is a
meaningful one. Lemmatization makes sure that lemma is a word with meaning and
hence it takes a longer time to execute than stemming.

4. What is the full form of TFIDF?

Term Frequency and Inverse Document Frequency

5. What is meant by a dictionary in NLP?

Dictionary in NLP means a list of all the unique words occurring in the corpus. If some
words are repeated in different documents, they are all written just once as while
creating the dictionary.
6. What is term frequency?
Term frequency is the frequency of a word in one document.

7. Which package is used for Natural Language Processing in Python programming?

Natural Language Toolkit (NLTK).

8. What is a document vector table?

Document Vector Table is used while implementing Bag of Words algorithm.
If the document contains a particular word it is represented by 1 and absence of word is
represented by 0 value.
OR
Document Vector Table is a table containing the frequency of each word of the
vocabulary in each document.

9. What do you mean by corpus?

A corpus is a large and structured set of machine-readable texts that have been
produced in a natural communicative setting.
OR
A corpus can be defined as a collection of text documents. It can be thought of as just a
bunch of text files in a directory, often alongside many other directories of text files.

Two (02) Mark Questions

1. Differentiate between a script-bot and a smart-bot. (Any 2 differences)

Script-bot Smart-bot
 A scripted chatbot doesn’t carry  Smart bots are built on NLP and
even a glimpse of A.I ML.
 Script bots are easy to make  Smart –bots are comparatively
difficult to make.
 Script bot functioning is very  Smart-bots are flexible and
limited as they are less powerful.
powerful. ● Smart bots work on bigger
 Script bots work around a script databases and other resources
which is programmed in them directly
● NLP and Machine learning
 No or little language processing skills are required.
skills ● Wide functionality
 Limited functionality

2. What is inverse document frequency?

To understand inverse document frequency, first we need to understand document
frequency.
Document Frequency is the number of documents in which the word occurs irrespective
of how many times it has occurred in those documents.
In case of inverse document frequency, we need to put the document frequency in the
denominator while the total number of documents is the numerator.
 For example, if the document frequency of a word “AMAN” is 2 in a particular document
then its inverse document frequency will be 3/2. (Here no. of documents is 3)

3. Mention some applications of Natural Language Processing.

Natural Language Processing Applications-
● Sentiment Analysis.
● Chatbots & Virtual Assistants.
● Text Classification.
● Text Extraction.
● Machine Translation
● Text Summarization
● Market Intelligence
● Auto-Correct

4. Explain the concept of Bag of Words.

Bag of Words is a Natural Language Processing model which helps in extracting features
out of the text which can be helpful in machine learning algorithms. In bag of words, we
get the occurrences of each word and construct the vocabulary for the corpus.

Bag of Words just creates a set of vectors containing the count of word occurrences in
the document (reviews). Bag of Words vectors are easy to interpret.

5. What are stop words? Explain with the help of examples.

“Stop words” are the most common words in a language like “the”, “a”, “on”, “is”, “all”.
These words do not carry important meaning and are usually removed from texts. It is
possible to remove stop words using Natural Language Toolkit (NLTK), a suite of
libraries and programs for symbolic and statistical natural language processing.

6. Differentiate between Human Language and Computer Language.

Humans communicate through language which we process all the time. Our brain keeps
on processing the sounds that it hears around itself and tries to make sense out of them
all the time.
On the other hand, the computer understands the language of numbers. Everything that
is sent to the machine has to be converted to numbers. And while typing, if a single
mistake is made, the computer throws an error and does not process that part. The
communications made by the machines are very basic and simple.

Four 04 Mark Questions

1. Create a document vector table for the given corpus:

Document 1: We are going to Mumbai
Document 2: Mumbai is a famous place.
Document 3: We are going to a famous place.
Document 4: I am famous in Mumbai.
We Are going to Mumbai is a famous place I am in
1 1 1 1 1 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1 0 0 0
 1 1 1 1 0 0 1 1 1 0 0 0
0 0 0 0 1 0 0 1 0 1 1 1

Perfect Syntax, no Meaning - Sometimes, a statement can have a perfectly correct syntax
but it does not mean anything. In Human language, a perfect balance of syntax and
semantics is important for better understanding.
These are some of the challenges we might have to face if we try to teach
computers how to understand and interact in human language.

2. Through a step-by-step process, calculate TFIDF for the given corpus and mention
the word(s) having highest value.
Document 1: We are going to Mumbai
Document 2: Mumbai is a famous place.
Document 3: We are going to a famous place.
Document 4: I am famous in Mumbai.

Term Frequency
Term frequency is the frequency of a word in one document. Term frequency can easily
be found from the document vector table as in that table we mention the frequency of
each word of the vocabulary in each document.

We Are Going to Mumbai is a famous Place I am in

1 1 1 1 1 0 0 0 0 0 0 0
0 0 0 0 1 1 1 1 1 0 0 0
1 1 1 1 0 0 1 1 1 0 0 0
0 0 0 0 1 0 0 1 0 1 1 1

Inverse Document Frequency

The other half of TFIDF which is Inverse Document Frequency. For this, let us first
understand what does document frequency mean. Document Frequency is the number
of documents in which the word occurs irrespective of how many times it has occurred
in those documents. The document frequency for the exemplar vocabulary would be:

We Are going to Mumbai is a Famous place I am in

2 2 2 2 3 1 2 3 2 1 1 1

Talking about inverse document frequency, we need to put the document frequency in
the denominator while the total number of documents is the numerator. Here, the total
number of documents are 3, hence inverse document frequency becomes:

We Are going to Mumbai is a Famous Place I am in

4/2 4/2 4/2 4/2 4/3 4/1 4/2 4/3 4/2 4/1 4/1 4/1

The formula of TFIDF for any word W becomes:

TFIDF(W) = TF(W) * log (IDF(W))
 The words having highest value are – Mumbai, Famous

3. Normalize the given text and comment on the vocabulary before and after the
normalization:
Raj and Vijay are best friends. They play together with other friends. Raj likes to
play football but Vijay prefers to play online games. Raj wants to be a footballer.
Vijay wants to become an online gamer.

Normalization of the given text:

Sentence Segmentation:
1. Raj and Vijay are best friends.
2. They play together with other friends.
3. Raj likes to play football but Vijay prefers to play online games.
4. Raj wants to be a footballer.
5. Vijay wants to become an online gamer.

Tokenization:

Raj and Vijay Raj and Vijay are best friends .

are best
friends.

They play They play Together with other friends .

together with
other friends

Same will be done for all sentences. Removing Stop words,

Special Characters and Numbers:

In this step, the tokens which are not necessary are removed from the token list.
So, the words and, are, to, an, (Punctuation) will be removed.

Converting text to a common case:

After the stop words removal, we convert the whole text into a similar case, preferably
lower case.
Here we don’t have words in different case so this step is not required for given text.
Stemming:

In this step, the remaining words are reduced to their root words. In other words,
stemming is the process in which the affixes of words are removed and the words are
converted to their base form.
 Word Affixes Stem

Likes -s Like

Prefers
-s Prefer

Wants -s want
In the given text Lemmatization is not required.
Given Text
Raj and Vijay are best friends. They play together with other friends. Raj likes to play
football but Vijay prefers to play online games. Raj wants to be a footballer. Vijay wants to
become an online gamer.
Normalized Text
Raj and Vijay best friends They play together with other friends Raj likes to play football
but Vijay prefers to play online games Raj wants to be a footballer Vijay wants to become
an online gamer