See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/332321070

Movies Reviews Sentiment Analysis and Classification

Conference Paper · April 2019

DOI: 10.1109/JEEIT.2019.8717422

CITATIONS READS

80 9,350

2 authors:

Mais Yasen Sara Tedmori

12 PUBLICATIONS 286 CITATIONS

Princess Sumaya University for Technology
65 PUBLICATIONS 1,078 CITATIONS
SEE PROFILE
SEE PROFILE

All content following this page was uploaded by Mais Yasen on 10 April 2019.

The user has requested enhancement of the downloaded file.

 Movies Reviews Sentiment Analysis and Classification
Mais Yasen, Sara Tedmori
Department of Computer Science
Princess Sumaya University for Technology
Amman, Jordan
mai20130045@std.psut.edu.jo, s.tedmori@psut.edu.jo
ABSTRACT- As humans’ opinions help enhance products (SA), where SA has become the gateway to understanding
efficiency, and since the success or the failure of a movie depends consumer needs [4].
on its reviews, there is an increase in the demand and need to
build a good sentiment analysis model that classifies movies SA, also referred to as opinion mining, is concerned with
reviews. In this research, tokenization is employed to transfer identifying and categorizing opinions -which are subjective
the input string into a word vector, stemming is utilized to impressions not facts- expressed in a text and determining
extract the root of the words, feature selection is conducted to whether the writer's feelings, attitudes or emotions towards a
extract the essential words, and finally classification is particular topic are positive or negative [4]. SA is also defined
performed to label reviews as being either positive or negative. as the process of transferring concrete data to subjective data,
A model that makes use of all of the previously mentioned and it can be performed on different levels (document,
methods is presented. The model is evaluated and compared on sentence, or aspect).
eight different classifiers. The model is evaluated on a real-
world dataset. In order to compare the eight different The process of SA includes tokenization, word filtering,
classifiers, five different evaluation metrics are utilized. The negation handling, stemming, and classification. Tokenization
results show that Random Forest outperforms the other is the identification of the basic units by the process of
classifiers. Furthermore, Ripper Rule Learning performed the segmenting text into sentences and words. Tokenization is
worst on the dataset according to the results attained from the
considered a pre-processing step. Text needs to be segmented
into linguistic units; such as words, numbers and
evaluation metrics.
punctuations, before performing any processing [5]. Words in
English are usually separated by white spaces, and sentences
Keywords- Sentiment Analysis; IMDB Reviews; Tokenization;
are separated by full stops. Errors in tokenization are very
Stemming; Feature Selection; Classification; Random Forest.
dangerous because they will result in more errors in
subsequent steps [6].
I. INTRODUCTION

Humans are subjective creatures and their opinions are Stemming is the process of removing prefixes and affixes
important because they reflect their satisfaction with products, to convert the word into its stem or root form [7].
services and available technologies. Being able to interact
with people on that level has many advantages for information One vital data mining function is classification, which
systems; such as enhancing products quality, adjusting builds a model for labeling testing data based on previous
marketing and business strategies, improving customer training data. Different measures can be used to evaluate this
services, managing crisis, and monitoring performances [1]. model, such as accuracy, Area Under the Curve (AUC), F-
measure, recall, and precision. Assigning classes (negative or
positive) to reviews can be done by such model which predicts
A movie review is an article reflecting its writers’ opinion
about a certain movie and criticizing it positively or the label of new data. Some of the classification algorithms
that has proven their efficiency in previous works are Naïve
negatively, which enables everyone to understand the overall
idea of that movie and make the decision whether to watch it Bayes (NB), K-nearest Neighbors (KNN), Bayes Net (BN),
or not. A movie review can affect the whole crew who worked Ripper Rule Learning (RRL), Support Vector Classifier
on that movie. A study illustrates that in some cases, the (SVM), Random Forest (RF), Stochastic Gradient Descent
success or the failure of a movie depends on its reviews [2]. (SGD), and Decision Tree (DT).
Therefore, a vital challenge is to be able to classify movies
reviews to capture, retrieve, quantify and analyze watchers This research addresses SA of movies reviews as a
more effectively [3]. classification task. Different classification algorithms are
considered and compared to assess their performances for the
task at hand. The reason why NB, BN, DT, KNN, RRL, SVM,
Movie reviews classification into positive or negative
reviews is connected with words occurrences from the RF, and SGD classifiers were chosen to be compared with
reviews text, and weather those words have been used before each other is that these algorithms are supervised classifiers
that have proven their efficiency and reliability in SA based
in a positive or a negative context. These factors help enhance
the review understanding process using Sentiment Analysis
on the previous works studied. Furthermore, these classifiers for sentiment holding words such as adjectives. The authors
are the most popular to be used tackling SA. used Amazon’s German movie reviews and compared their
work to a statistical polarity classifier based on n-grams. The
The contribution of this research can be summarized by: results reflected that their approach was good for multilingual
SA.
1. Using a real reviews dataset from IMDB which
contained almost 43 thousand instances for training In [12], the authors proposed a framework for classifying
and testing. Web forum reviews in multiple languages (English and
2. Using eight different well-known classifiers; NB, BN, Arabic). The authors used entropy weighted genetic algorithm
DT, KNN, RRL, SVM, RF, and SGD for evaluation in (EWGA) to improve the performance. For evaluation a movie
SA for the first time. review dataset was used and results showed that using EWGA
3. Comparing the results using different evaluation with SVM gave higher performance in comparison to other
metrics. feature selection methods, with accuracy of over 91%.

The paper is structured as follows: Section II describes the In [13], the authors presented a feature-based heuristic for
related work in the area of SA. Section III includes the SA of IMDB movie reviews using an aspect-oriented scheme.
methods used in the development. Section IV shows the The authors then combined all aspects scores and generated a
proposed methodology. Section V presents the experiments net sentiment profile. Using a SentiWordNet with two feature
and the results of the proposed approach, and Section VI selection methods, the sentiment analysis of documents
concludes the research and discusses future work. belonging to each movie was found and compared to Alchemy
API. The results showed that their approach gave higher
II. RELATED WORK accuracy in comparison to simple document-level SA.

The authors of [8] addressed SA at the document-level and In [14] the authors used four classifiers; Maximum
proposed the use of a combination of supervised and Entropy (ME), Naive Bayes (NB), Support Vector Machine
unsupervised algorithms and rich sentiment content for (SVM), and Stochastic Gradient Descent (SGD) for SA on the
learning word vectors (also known as words description) IMDB dataset. They used precision, recall, f-measure, and
capturing techniques, which includes continuous, multi- accuracy for evaluation. The results showed that using a
dimensional sentiments and non-sentiment annotations. For combination of unigram, bigram, trigram gave better accuracy
evaluation, the authors used the IMDB movie reviews dataset in all the classifiers used.
and their model got better performance in comparison to other
learn vectors capturing techniques. The authors of [15] proposed using Naive Bayes (NB) and
Support Vector Machines (SVM) classifiers, and the use of a
In [9], the authors presented a model to classify movie modified SVM using NB log-count ratios. The results showed
reviews as “thumbs up” or “thumbs down”. They proposed a that NB was better than SVM in short documents. In longer
text-categorization method using machine learning, and found documents however, SVM was better. Their modified
the difference between subjectivity detection and polarity algorithm gave good results. Also, the use of logistic
classification. The results showed that using subjectivity regression instead of SVM gave the same results, and the use
detection lead to shorter reviews, in Naive Bayes classifier the of word bigram gave a consistent gain on SA.
subjectivity detection was more effective in comparison to the
original document without any subjectivity detection. Also, The authors of [16] proposed using a multi knowledge-
the minimum-cut classification enhanced the accuracy. based approach to produce a feature class-based automatically
in a movie review analysis system. Their approach combined
According to [10], references to the movie in weblog posts statistical analysis, WordNet, and knowledge of movies. The
and the movie financial success are important factors. The results showed that their approach was effective.
results showed that positive sentiment is more efficient for
movies domain with small number of existing reviews, which As mentioned in [17], the authors worked on enhancing
was not a good indication for building a model based on the Naive Bayes accuracy for SA. The proposed approach can
sentiment only, where sentiment could perform better in be specified to a certain number of string categorizations to
conjunction with other factors such as movie genre and enhance accuracy. The results showed that the combination of
season. negation handling, feature selection, and word n-grams
improved the accuracy, using a Naive Bayes classifier which
The authors of [11] tackled SA of text in a multilingual had a linearly increasing time of training and testing. On the
system. They used the lexical resources in the English IMDB dataset the accuracy was 88.80%.
SentiWordNet. With the aid of a translation software, the
authors first translated different languages to English. Then, As discussed in [18] two SA methods were proposed to
they classified them into “positive” or “negative” by searching label reviews as positive, negative, or neutral. The first
method is to label reviews using the number of negative and used [21]. Secondly, unwanted words were removed
positive words, and to extend the term-counting using external manually.
resources. The second method is to use Support Vector
Machine (SVM). The authors applied three valence shifters; C. Stemming
which are negations that invert the label polarity of a text,
intensifiers to increase and decrease positivity or negativity of Stemming is the process of removing prefixes and affixes
a term, and diminishers. The results showed that the term- to convert the word into its stem or root form [7]. Porter
counting method got higher accuracy, and the accuracy of stemming algorithm is used, which is a rule-based algorithm
SVM was very high. introduced by Martin Porter [22]. It defines a word consonant
as any letter other than vowels. Form 1 shows how to
From the related work studied it can be deduced that using calculate the conditions in this algorithm, where the square
feature selection in SA is an open field, with NB and SVM as brackets denote optional content, and (VC)m denote a Vowel
the most commonly used classifiers. (V) followed by a Consonant (C) m times.

This research proposes a model for SA of movie reviews. C (VC)m V (1)

Different evaluation measures were considered such as recall,
accuracy, AUC, precision, and f-measures. The model will be This algorithm follows a list of rules that contain patterns
evaluated using 8 different well-known classifiers in an with their conditions, the rules follow the following form:
inclusive study which enables us to judge on which classifier
results in a better performance more reliably and accurately. 1→ 2 (2)
As far as the authors are aware, this work is the first effort that
aims to compare the use of NB, BN, DT, KNN, RRL, SVM, if the pattern matches, and the word ends with the suffix S1,
RF, and SGD classifiers in SA, using different evaluation the suffix is transformed from S1 to S2 and the algorithm
metrics. restarts from the beginning of the list to find the next
matching pattern. If no pattern matches, then the algorithm
III. METHODS DESCRIPTION outputs the result.
The methods descriptions of this research are included in this
section. D. Attribute Selection

A. Text Tokenization Gain ratio was used as the attribute selection algorithm,
which can be defined as the rate of information gain to the
Text tokenization is segmenting text into sentences and essential information. The attributes with a high count of
words by specifying the basic linguistic units; words, values are the most important in gain ratio. Gain ratio takes
numbers and punctuations [5]. In English language words are the attribute with the highest gain value and uses it to split
usually separated by white spaces. attributes, which reduces the number of features.

Sentence tokenization is dividing a string into sentences. Equation (3) [23] is used to calculate the expected
In English, punctuations especially the full stop character are information.
indications of a sentence ending [19]. However, the full stop
character can be used for abbreviations, which does not ( . ) ( . )
always terminate the sentence. To prevent such problem, a ( )=− × (3)
| | | |
table of abbreviations is used. Sentence tokenization was
applied using NLTK [20], which is trained on many
languages including English. The training includes the Where T is the training data and |T| is the total number of
identification of punctuation and characters that appears in records, ri represents a specific value for each feature and the
the end of a sentence and the beginning of a new sentence. freq is all the possible values in that feature, where i goes
from 1 to n (all possible values of the candidate feature).
In NLTK [20], word tokenization is a wrapper function
that uses the Treebank Word Tokenizer, and splits out Equation (4) [23] calculates the essential information for
punctuations other than periods. a specific value of split (S).

B. Word Filtering | |
( )= × (4)
| |
After tokenization, unexpected words that will not affect
the process of classification were removed. Firstly, regular
expressions which are sequence of characters that illustrates Where |Tj| represents all the possible values of attribute
a search pattern to find and replace unwanted words were number j, and m is the number of possible attributes.
 Equation (5) [23] gives the value of information gain of IV. PROPOSED APPROACH
split (S).
In this research, the researchers present SA model and
( )= ( )− ( ) (5) classification algorithms for the classification of IMDB
movies reviews.
Equation (6) [23] calculates the information gain ratio
between the information gain and the essential information. The execution steps that are shown in Fig. 1 could be
summarized as the following:
( )
( )= (6) 1. Retrieving IMDB reviews datasets.
( )
2. Labeling datasets into POS/NEG classes.
E. Classification 3. Sentence tokenization.
4. Word tokenization: String to Word Vector.
To evaluate the proposed model, eight different well- 5. Remove unwanted words using regular expressions.
known classifiers were run on the same training and testing 6. Remove unwanted words manually.
datasets. The classifiers could be summarized as mentioned 7. Stemming.
below: 8. Attribute Selection using Gain Ratio.
9. Split data into training and testing.
• Naïve Bayes (NB): This classifier has two 10. Classifying the data using 8 different classifiers and
probabilities: P(class) which is the probability an input comparing their results using different metrics.
will produce a certain class, and
P(input_condition|class) is the probability an input
feature has a certain value, given the class. Otherwise,
default probability is 0.
• Decision Tree (DT): A classifier model that gives
labels to tokens based on a tree structure, where tree
branches represent conditions on features, and tree
leaves represent the label.
• Support Vector Classifier (SVM): A classifier that
deals with missing values, normalizes nominal
features to binary features. It formalizes all features by
default. The output coefficients are found using the
normalized form of features.
• Bayes Network (BN): In this classifier learning is
done using search algorithms and quality measures.
BN provides conditional probability distributions.
• K-nearest Neighbors (KNN): This classifier does
distance weighting and is capable choosing the K
value using cross-validation.
• Ripper Rule Learning (RRL): A classifier that uses
Fig. 1 Proposed Approach
RIPPER to gradually prune its propositional rule
learner, to decrease error.
V. EXPERIMENTS AND RESULTS
• Random Forest (RF): The underlying data structure
of the forest classifier is a decision tree, but with
Experiments conducted to evaluate the performance of the
random selection of features to split on.
proposed approach are demonstrated in this section.
• Stochastic Gradient Descent (SGD): A classifier
used with many linear models (SVM, logistic
regression, squared, Huber, and epsilon-insensitive A. Data
losses). It changes missing instances and changes
nominal attributes. Furthermore, it normalizes data For purposes of testing the performance of the proposed
features. A high rate of learning is needed by both approach, the IMDB reviews dataset was used [24]. This
Epsilon-insensitive and Huber loss. dataset represents a group of movie reviews, and it contains
42926 review instances along with their binary classification
(positive or negative). Where in each data file the first line
represents the headers of which the description of the
attributes is in. A “\n” is assigned for missing field.
 B. Experiments settings • RRL: pruning was done using 1 fold, to grow the tree 2
folds were used, optimization executions number equals
First, attribute selection using Gain Ratio was performed 2, and the rule instances weight equals 2.
on the dataset to consider only the attributes that are most • RF: seed was set to 1, number of execution slots was set
relevant to the label feature to train and test our proposed to 1, bag size was set to 100, batch size was also set to
approach. The total count of attributes before and after 100, maximum depth was set to 0.
attribute selection with the label feature is shown in Table 1. • SGD: seed was set to 1, epochs is set to 500, lambda is
set to 0.0001, batch size was 100, loss function was
Next, the data was divided into two sections, training Hinge loss, learning rate was set to 0.01.
included 66% of the total instances in the dataset, and testing
included 34% of the total instances in the dataset; the reason For evaluation five measures were used: Precision, Recall,
behind this split percentage is that it is the most commonly Accuracy, AUC and F-measure. These measures can be
used split in research. Dataset instances distribution after calculated applying the Equations 7 to 11, where TP stands for
dividing it is also shown in Table 1. True Positives, TN stands for True Negatives, FP stands for
False Positives, FN stands for False Negatives.
Table 1 Total Number of Instances and Attributes in the dataset
Training 28331 Positive instances = 14230 = (TP + TN)/ (T + + + ) (7)
dataset instances Negative instances = 14101
14595 Positive instances = 7252 TP
Testing dataset = (8)
instances Negative instances = 7343 TP + FP
Features before
1135 = TP/(TP + FN) (9)
selection
Features after
896 = 2×( × )/( + )(10)
selection

The classifiers used are provided by NLTK [20], and (1 − ) × (1 + ) ×

= + (11)
WEKA wrapper library for Python [25]. The settings of the 2 2
classifiers could be summarized as the following:
Where TPR stands for True Positive Rate (TPR =
• NB: The size of batch equals 100, and default probability TP/(TP+FN)) and FPR stands for False Positive Rate (FPR =
was set to 0. FP/(FP+TN)).
• DT: For pruning 1 fold was set, for tree growing 2 folds
were set, and leaf instances number equals 2 (minimum C. Results
value).
• SVM: The complexity feature equals 1, logistic As conducted from Table 2, RF got the best accuracy
regression was applied as the calibration. (96.01%) in comparison with all of the other classifiers.
• BN: alpha value equals 0.5 which is used for calculating Moreover, it got the highest precision (0.93), f-measure (0.96)
the conditional probability, and hill climbing was and AUC (0.96). Also, RF and KNN got the best recall (1.00)
applied as the search method. in comparison to all of the classifiers in the table where they
• KNN: neighbors count equals one, search of nearest achieved false negatives count of 0. Furthermore, DT got a
neighbor was done applying brute force, and window very competitive recall (0.97), KNN also got a very
size equals 0. competitive f-measure (0.93) and AUC (0.93).

Table 2 Results
Classifier TP FN FP TN Accuracy % Precision Recall Fmeasure AUC
NB 5789 1507 1145 6153 81.83 0.84 0.79 0.81 0.82
DT 7088 208 1064 6234 91.28 0.87 0.97 0.92 0.91
SVM 6412 884 947 6351 87.45 0.87 0.88 0.88 0.88
BN 5697 1599 1106 6192 81.47 0.84 0.78 0.81 0.82
KNN 7296 0 1033 6265 92.92 0.88 1.00 0.93 0.93
RRL 5574 1722 1269 6029 79.51 0.82 0.76 0.79 0.80
RF 7296 0 583 6715 96.01 0.93 1.00 0.96 0.96
SGD 6399 897 1028 6270 86.81 0.86 0.88 0.87 0.87
 for sentiment analysis”, the 49th Annual Meeting of the
To summarize the results, RF has proved its efficiency
Association for Computational Linguistics: Human Language
over 7 other classifiers where it got the best result in all of the Technologies, Vol. 1, PP 142-150.
evaluation measures taken into account, the accuracy of the 8
classifiers ranged from 79.51% to 96.01%. RRL performed [9] Bo Pang, Lillian Lee, (2004), “A Sentimental Education:
Sentiment Analysis Using Subjectivity Summarization Based
the worst. on Minimum Cuts”, the 42nd Annual Meeting on Association
for Computational Linguistics.
VI. CONCLUSION AND FUTURE WORK
[10] Mishne, Gilad and Natalie Glance, (2006), “Predicting Movie
Sales from Blogger Sentiment”, AAAI Spring Symposium:
The research goal of this work is to address SA by Computational Approaches to Analyzing Weblogs.
constructing an approach that can classify movie reviews and
[11] Kerstin Denecke, (2008), "Using SentiWordNet for
then compare the results in an inclusive study of eight well-
multilingual sentiment analysis", IEEE 24th International
known classifiers. To evaluate the proposed model, IMDB Conference on Data Engineering Workshop, Cancun, PP 507-
reviews real dataset was utilized. Tokenization was applied on 512.
the dataset to transfer strings into word vector, then stemming
[12] Ahmed Abbasi, Hsinchun Chen, and Arab Salem, (2008),
was used to extract the root of the words, afterwards gain ratio “Sentiment Analysis in Multiple Languages: Feature Selection
was applied on the dataset as an attribute selection algorithm. for Opinion Classification in Web Forums”, ACM Transactions
Then, the data was split into training and testing datasets using on Information Systems (TOIS), Vol. 26, PP 1-34.
the percentages 66%, 34% respectively. To evaluate the [13] Vivek Singh, R Piryani, Ashraf Uddin and Pranav Waila,
results accuracy, precision, f-measure, recall, and AUC were (2013), "Sentiment analysis of movie reviews: A new feature-
used. based heuristic for aspect-level sentiment classification,"
International Mutli-Conference on Automation, Computing,
The results showed that RF has proved its efficiency over Communication, Control and Compressed Sensing (iMac4s),
7 other classifiers where it got the best result in all of the PP 712-717.
evaluation metrics taken into consideration, KNN also was [14] Abinash Tripathy, Ankit Agrawal, Santanu Kumar Rath,
able to get a recall similar to RF and a very competitive f- (2016), “Classification of sentiment reviews using n-gram
measure and AUC. Furthermore, DT got a very competitive machine learning approach”, Expert Systems with Applications,
recall value. Finally, RRL got the worst result. Vol. 57, PP. 117-126.
[15] Sida Wang and Christopher Manning, (2012), “Baselines and
The authors wish to conduct a similar study on different bigrams: simple, good sentiment and topic classification”, the
languages specifically on Arabic. In addition, the authors wish 50th Annual Meeting of the Association for Computational
to experiment with different SA methods in order to increase Linguistics: Short Papers, Vol. 2, PP 90-94.
the accuracy of the results. [16] Li Zhuang, Feng Jing, Xiao-Yan Zhu, (2006), “Movie Review
Mining and Summarization”, the 15th ACM international
REFERENCES conference on Information and knowledge management, PP.
43-50.
[1] Sampriti Sarkar, “Benefits of Sentiment Analysis for
Businesses”, retrieved on: December 22, 2018, from: [17] Vivek Narayanan, Ishan Arora, Arjun Bhatia, (2013), “Fast and
www.analyticsinsight.net. Accurate Sentiment Classification Using an Enhanced Naive
Bayes Model”, Intelligent Data Engineering and Automated
[2] ACME, “The Significance of a Film Review”, retrieved on: Learning – IDEAL, Springer, Vol. 8206.
December 22, 2018, from: www.revue-acme.com.
[18] Alistair Kennedy and Diana Inkpen, (2006), “Sentiment
[3] Mshne, Gilad and Natalie Glance, (2006), “Predicting Movie Classification of Movie Reviews Using Contextual Valence
Sales from Blogger Sentiment”, AAAI Spring Symposium: Shifters”, Computational Intelligence, Vol. 22, PP 110-125.
Computational Approaches to Analyzing Weblogs, PP 1-4.
[19] Jeffrey Reynar, (1998), “Topic Segmentation: Algorithms and
[4] Bo Pang and Lillian Lee, (2008), “Opinion Mining and Applications”, IRCS Technical Reports Series, Vol. 66, PP 1-
Sentiment Analysis”, Foundations and Trends in Information 189.
Retrieval, Volume 2, PP 1–135.
[20] NLTK, version: 3.3, retrieved on: October 20, 2018, from:
[5] Vinodhini, Chandrasekaran (2012), “Sentiment Analysis and www.nltk.org.
Opinion Mining: A Survey”, International Journal of Advanced
Research in Computer Science and Software Engineering, Vol. [21] Mark Lawson, (2003), “Finite Automata”, CRC Press, PP 98-
2, PP 1-11. 100.
[6] CraigTrim, “The Art of Tokenization”, retrieved on: December [22] M.F. Porter, (1980), “An algorithm for suffix stripping”,
22, 2018, from: www.ibm.com. Program, Vol. 14, PP 130-137.
[7] Lovins, Julie Beth, (1968), “Development of a Stemming [23] SAS, (2015), “Visual Analytics 7.2”, SAS Institute Inc., Ch. 37,
Algorithm”, Mechanical Translation and Computational PP 281.
Linguistics, Vol. 11, PP 22–31. [24] Arunava, “IMDB Movie Reviews Dataset”, retrieved on:
November 1, 2018, from: www.kaggle.com.
[8] Andrew Maas, Raymond Daly, Peter Pham, Dan Huang,
Andrew Ng, Christopher Potts, (2011), “Learning word vectors [25] python-weka-wrapper, version: 3.14, retrieved on: November
30, 2018, from: pypi.org.

View publication stats