Impact of Intervals on the Emotional Effect in

Western Music
Cengiz Kaygusuz Julian Zuluaga
Florida International University Florida International University
ckayg001@fiu.edu jzulu013@fiu.edu

Abstract—Every art form ultimately aims to invoke an emo- TABLE I

tional response over the audience, and music is no different. M USICAL ASPECTS AND THEIR FEELINGS [1].
While the precise perception of music is a highly subjective topic,
there is an agreement in the "feeling" of a piece of music in Aspect Description Feeling
broad terms. Based on this observation, in this study, we aimed Fast: Happiness, excitement, anger.
Tempo Pace of the music .
to determine the emotional feeling associated with short passages Slow: Sadness, serenity
arXiv:1812.04723v1 [q-bio.NC] 10 Dec 2018

of music; specifically by analyzing the melodic aspects. We have Major: happiness, joy.
Mode Type of the scale
used the dataset put together by Eerola et. al. which is comprised Minor: sadness, darkness.
Smooth rhythm: happiness, peace.
of labeled short passages of film music. Our initial survey of the
Rhythm Beat of a song Irregular rhythm: amusement, uneasiness.
dataset indicated that other than "happy" and "sad" labels do Varied rhythm: joy.
not possess a melodic structure. We transcribed the main melody
of the happy and sad tracks, and used the intervals between the
notes to classify them. Our experiments has shown that treating a
melody as a bag-of-intervals do not possess any predictive power II. BACKGROUND
whatsoever, whereas counting intervals with respect to the key
of the melody yielded a classifier with 85% accuracy. Aside from the data mining dimension, this study requires
some knowledge of human psychology and music theory,
I. I NTRODUCTION which we aim to present in this section.
The emotional touch of the music is undoubtedly experi-
A. Emotions, Mood, and Music
enced by many, and it is not a coincidence since every form
of art is usually intended to evoke a certain emotion on the The concepts of emotion and mood are closely related to
listener. To be more specific, recent research indicates that each other as both can be described as feelings. However,
music in particular, serves to regulate mood and increase one’s there are subtle differences between the two, which are not
self-awareness, clearly establishing the relationship between obvious at first sight. The dictionary definition of emotion
music and human emotions [1]. is "a particular feeling that characterizes the state of mind",
Music is not an atomic construct on its own accord however, whereas mood is defined as "a prevailing emotional tone or
and considering its multidimensional essence makes way to a general attitude". As their definitions indicate, we can consider
natural question: what is the relationship between the aspects emotions as concrete feelings, such as joy, anger, and fear,
of music and its emotional content? There is a tremendous whereas mood is the influencer of emotions, such as the state
amount of deliberation put forward for the answer [2], and of being irritable. It is with no doubt that music influences
there seems to be a strong agreement on the emotional effects both emotions and the mood, however we are often aware
of various musical phenomena, as well as the impact of the of our emotions and seldom acknowledge our mood, hence
state of the listener. emotions are much more reliable to reason about.
One aspect of music, namely the melody, constitutes a A basic taxonomy of emotions is given in Figure 1(a),
quantifiable measure. It is also the most significant aspect, as as given by Schaver [4]. In this simple hierarchical model,
listeners mainly relate to musical works by their melodies. In emotions are classified primarily as positive and negative, and
western music, the distance between two notes is measured by get more and more specific as we traverse the tree.
"semitones", which can be thought of as the building blocks Other prevalent taxonomies use a dimensional model to
of a melody. The group of notes in a melody is characterized classify emotions, basically breaking each emotion into their
by the relative distances between them. Western music is "essence", which correspond to a dimension in the model. An
comprised of 12 intervals, each one of them having a unique example two-dimensional model is given in Figure 1(b) taken
feeling associated to it which can be divided in two, as "happy, from Russell et. al. [3]. Other dimensional models vary this
energetic, positive" or as "sad, sentimental, negative". We basic idea by using different or more axes [5]. The dataset we
hypothesize that the positive intervals occur more on music used are labeled in both 6 basic emotions, as well as in a 3D
that induce happiness, and similarly, negative intervals occur model of emotions with the axes named as: valence, activity,
more on music that induce sadness. and intensity.
 (a) (b)

Fig. 1. A hierarchical (a), and a dimensional (b) examples of emotional model [3], [4].

The representability of all kinds of emotion by music is an The relevant aspect of these two scales is that the major scale
open question [5]. However, a few musical aspects are well feels happy and upbeat, while the minor scale feels sad and
known to induce specific feelings, given in Table I. As could sentimental.
be seen in the table, major tones induce happiness, in contrast
of minor tones which feels sad, however, the role of musical III. M ETHODOLOGY
intervals is not clear in this distinction. This section is dedicated to explain both the dataset, and the
necessary details we have undertaken to test our hypothesis.
B. Music Theory
A. Original Dataset
All sound is explained by the phenomenon of vibration. Because the emotional labeling of musical passages are a
The frequency of a vibration that produces sound is named highly subjective process [5], we opted to use an existing
as pitch. Specific values of pitches are named as notes. In dataset. We have used the dataset put together by Eerola et. al.
western music, notes are represented by uppercase alphabet first used in his 2009 study [6]. The passages all taken from
letters from A to G. film scores. The rationale for solely using film scores is given
The difference of pitch between two sound frequencies is as their explicit goal of supporting the feeling at a particular
measured in cents. This logarithmic unit of measure is much moment in a film, thus it is easier to identify and associate
more rich in resolution than it is needed when considering with the emotional content of the passage.
western music, as two notes at worst differ only by 100 cents, There are a grand total of 360 passages, each approximately
which is referred to as a semitone. The interval between a note 15 seconds long, and given in mp3 format. To ensure the la-
and the note double of its frequency is called an octave. An beling is as accurate as possible, the original authors employed
octave is divided into twelve equal parts, which yields us all 12 music experts and 116 university students.
the available notes. Two notes an octave apart are represented Each passage is concretely labeled as one of the 12 labels.
by the same letter, but differentiated by their octave number 6 of these labels are comprised of basic emotions such as
(e.g., A4 and A5). "happy", "sad", "tender", "anger", "fear", and "surprise". Rest
The intervals below an octave are the most significant in
music, and in our study. A list of intervals, alongside its
difference in semitones and its feeling is given in Table II. An TABLE II
important feature of intervals is that each single one of them I NTERVALS IN AN O CTAVE
have a feeling associated with it, mainly differing as positive Note ∆Semitones Interval Name Feeling
and negative. This observation is central to our hypothesis. C 0 Unison Neutral
The concept of "key" in western music refers to the main C] 1 Minor Second Unpleasant, Dark
note that feels like "resolution" or "stability" from the musical D 2 Major Second Neutral, Passing Tone
D] 3 Minor Second Tragedy, Sadness
passage. This main note will usually define the relationship E 4 Major Second Joy, Happiness
with all other notes, creating a hierarchy. A musical "scale" F 5 Perfect Fourth Buoyant, Neutral
can be thought as the "blueprint" of the music, as alongside F] 6 Tritone Danger, Devilishness
G 7 Perfect Fifth Stability
the key, the melody and harmony is arranged around a pre- G] 8 Minor Sixth Anguish, Sadness
determined key and scale. The amount of scales that could A 9 Major Sixth Winsomeness
be used in a western music setting is quite numerous, though A] 10 Minor Seventh Irresolution, Mournfulness
two of them are most commonly used: major and minor scale. B 11 Major Seventh Aspiration
6 of the labels are associated with the 3D space of human 3) First Note of Passage: Consider the first note as the key.
emotions: valence, tension, and energy. A separate label is Western music often, but not always, starts with the key
used for high and low values for each dimension. note.
4) Last Note of Passage: Consider the last note as the key.
B. Preprocessing Western music often, but not always, ends with the key
note.
For the sake of simplicity, we disregarded the 3D emotion
models and focused on basic emotions. The initial survey of The function that calculates the tonal difference for "key
the remainder of the dataset indicated that passages labeled note" and "preceding note" features is slightly different. The
with anything other than happy and sad either did not possess difference mainly lies in that preceding note features counts
a clear melodic structure that could be transcribed, or the main descending intervals (first note’s pitch is higher than the
vector of emotion was not the melody but other factors such second one) separately whereas key note features do not. This
as tempo and loudness (or lack thereof). Filtering the dataset is because the reference point in preceding note features are
down to "happy" and "sad" samples yielded a total of 60 a specific note on an octave (e.g., C4), in contrast of key note
passages. features where the note does not refer to a specific note (e.g.,
As we have mentioned before, the musical passages in the key of C). In other words, the key note exists both under and
dataset are in mp3 format and the music contained other over a given note, as long as there is consistency (i.e., always
elements in addition to the melody, hence we needed to extract choosing the note under or over), the direction is not important.
the melodies first. To do so, we have transcribed the melodies The exact procedures we used for tonal difference is listed in
of the identified passages by ear and converted them to MIDI Algorithm 1. Important things to mention are: we used "the
format. Since the only interest is in the impact of intervals, note under" for calculating absolute tonal diff, and considered
we have paid extra attention to the correctness of the pitch; the octave interval as a unison.
timings and the velocity of each note has been transcribed on
a best-effort basis. As part of this process, we also identified Algorithm 1: Calculating tonal difference.
the key of the melody. input : Two notes, a and b, represented as integers
During the transcription, we identified that some of the according to MIDI specification.
passages, again, did not possess a clear melodic structure, and output: Difference in semitones.
a few of them had multiple melodies. Omitting non-melodic
function tonal_diff(a, b):
passages and adding the extra melodies has yielded a dataset
if a ≥ b then
comprised of 49 records which is the final number of records return (a − b) mod 12
we have conducted our experiments with.
if a < b then
To parse the MIDI files, we used a custom MIDI parser that
return ((a − b) mod 12) − 12
read a MIDI file and returned only the stream of notes. Using
this stream, we have derived several sets of features, which
function abs_tonal_diff(a, b):
we will now explain in the next subsection. return (a − b) mod 12

C. Feature Engineering
Our main approach in deriving features is simply counting The final form of the data which we conducted our classifi-
the intervals and averaging them with the length of the cation experiments are simply produced by iterating all of the
passage. There are two important details: how do we measure notes to count the intervals with respect to the reference point.
the distance the interval between two notes, and what is the The key of the melody were identified by us and was available
reference point? in the meta-data. All other features has been calculated by
MIDI specifies a total of 128 notes, so technically a total solely the notes in the passage.
of 128 intervals can be represented. However, in practice,
intervals find themselves used up to thirteenths, with ones
below octave used most commonly. Taking this into account, D. Classification
we limited the counted intervals up an octave, and considered The accuracy of the features and classification algorithms
intervals higher than an octave to be their counterpart below are given in Table III. All the results have been obtained with
octave. 10-fold cross validation. Preceding-note and last-note-as-key
Since an interval is comprised of two notes, we need a features yielded almost random - or worse than random in case
reference point in the first step to calculate an interval. Four of a decision tree - classifiers. Classifiers using first-note-as-
reference points have been considered: key features did slightly better than random. The highest ac-
1) Key Note: This is the note that the passage resolves to. curacy was obtained by using the identified-key features used
2) Preceding Note: Given a note in a melody, calculate the in conjunction with k-nearest-neighbor algorithm. We used
interval with the next note. WEKA [7] to explore the data and conduct the experiments.
 TABLE III
F EATURE S ETS , A LGORITHMS , AND THE C LASSIFICATION ACCURACIES

Algorithms
Naive Bayes kNN (k=4) Weighted kNN (k=4, 1/distance) SVM Decision Tree (C4.5)
Features
Preceeding Note 50% 58% 56.25% 52% 37.5%
First Note as Key 66.66% 60.41% 64.58% 60.41% 62.5%
Last Note as Key 58.33% 47.91% 47.91% 58.33% 54.16%
Identified Key 77.08% 85.41% 83.33% 83.33% 79.16%

IV. D ISCUSSION TABLE IV

W EIGHTS OF I NDIVIDUAL I NTERVALS IN A SVM CLASSIFIER
The results came as a partial surprise. We initially expected
features other than identified-key-features to be inferior to it Interval Appears in Weight in SVM
indeed, but we expected them to perform much better than Unison Both -0.36
Minor Second None -0.35
a random classifier. This indicates that the intervals between Major Second Both 0.03
consecutive notes in a melody do not associate with the Minor Third Minor Scale -1.27
feeling. Major Third Major Scale 1.07
Perfect Fourth Both -0.18
The first-note-as-key features seem to perform slightly better Tritone None 0.24
than preceeding-note features, though we assert that the reason Perfect Fifth Both 0.08
for this edge is that it is more likely for the first note to be Minor Sixth Minor Scale -1.26
the actual key of the passage. Last-note-as-key features did as Major Sixth Major Scale 2.07
Minor Seventh Minor Scale -0.21
bad as preceeding-note features, yielding random classifiers. Major Seventh Major Scale -0.11
The identified key’s performance can be explained by the
differences in major and minor scale. Recall that music in
major scale is often associated with positive emotions, and mu- notes next to each other in a passage deemed ineffective,
sic in minor scale is often associated with negative emotions. whereas counting intervals with respect to the key of the song
Table IV shows the weights of individual intervals in a SVM yielded a relatively high accuracy classifier (approx. 85%). We
classifier, trained/tested with 66% percentage split and at- reason this indicates that intervals could only be associated
tained similar accuracy (approx. 81%). Negative weights point with emotional dimension of music if they are evaluated in
towards "sad", and positive weights point towards "happy" correctly established musical context.
labelings. From the table, it can be seen that the SVM
model is in agreement with the overall feeling of major and VI. R ELATED W ORK
minor scales, and thirds and sixths in particular are the most Most recent research is mainly focused on extracting emo-
important intervals in emphasizing the feeling of the scale, and tional information directly from audio recordings, instead of
the passage. notes. Much of this effort has been thorough MIREX (Music
It must be mentioned that even with identified-key features, Information Retrieval EXchange) [8], which annually accepts
the best classifier is rather far away from the perfect classifier and evaluates submissions on various musical data mining
(15%). This gap can be explained if we consider even though challenges, including mood classification, instrument recog-
major and minor scales are associated with happy and sad nition, automated drum transcription, and more. Most of the
emotions respectively, composers often use notes outside the prevalent submissions utilize a form of convolutional neural
scale, or even change the scale of the music at an arbitrary network. Features extracted by the usage of Mel Frequency
point. Cepstral Coefficients (MFCC) [9] is also popular among
By taking the failure of preceding-note features and the contestants.
prevalence of identified-key features into account, we reason On MIREX, the best accuracy for mood classification is
that in terms of emotional content, intervals themselves are around 69%. One of the submissions that attained this figure
only meaningful under correctly established musical context. [10] pre-trained their classifiers on a big tagged database
This is in contrast with our initial expectations. for feature extraction, used two distinct deep neural network
architecture outputting to an SVM to make a final decision.
V. C ONCLUSION
A few relatively dated publications regarding the use of
In this study, we have examined the impact of intervals on notes for emotional classification is available. Wang et. al. [11]
the emotional effect specifically in western music. The utilized used statistical features such as average and standard deviation
dataset has been constructed with the help of a large body of of structs such as tempo, interval, pitch, loudness, and other
students and music experts in a 2009 study. We counted the MIDI and audio features to classify 6 emotions, grouped in
intervals with respect to various reference points, averaged three. The classification of final labels has been made in two
them by the length of the passage, and conducted classification steps, in the first step, an SVM decided whether the music
on this derived dataset. Features derived by intervals among was energetic or tranquil, and two separate SVM’s each for
the outcome for the first SVM was used to particularize the
feeling, e.g., classifying between, joyous, robust, or restless
in case the music is energetic. The accuracy for energetic-
tranquil duo was around 95%, while the accuracy for all 6
classes varied around 62% and 85%.
Another study conducted by Lu et. al. [12] used a plethora of
features to classify between four emotions based on Thayer’s
arousal-valence emotion modeling. The features they used can
be analyzed in three subjects: MIDI, audio, and lyrics. They
used AdaBoost with SVM as the weak learning algorithm, and
evaluated feature sets in all possible combinations (standalone,
pairwise combination, and all of them). They found that using
all the feature sets were superior to other combinations, and
attained 72.4% accuracy in that case.
The works mentioned here contains much merit, but none
of them analyzed the musical intervals in context of emotional
recognition as we have done in this study.
R EFERENCES
[1] A. Gabrielsson and E. Lindström, “The influence of musical structure
on emotional expression.” 2001.
[2] K. R. Scherer and M. R. Zentner, “Emotional effects of music: Produc-
tion rules.”
[3] J. A. Russell, “A circumplex model of affect.” Journal of personality
and social psychology, vol. 39, no. 6, p. 1161, 1980.
[4] P. Shaver, J. Schwartz, D. Kirson, and C. O’connor, “Emotion knowl-
edge: Further exploration of a prototype approach.” Journal of person-
ality and social psychology, vol. 52, no. 6, p. 1061, 1987.
[5] T. Li, M. Ogihara, and G. Tzanetakis, Music Data Mining, 1st ed. Boca
Raton, FL, USA: CRC Press, Inc., 2011.
[6] T. Eerola, O. Lartillot, and P. Toiviainen, “Prediction of multidimen-
sional emotional ratings in music from audio using multivariate regres-
sion models.” in ISMIR, 2009, pp. 621–626.
[7] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and
I. H. Witten, “The WEKA data mining software: an update,” SIGKDD
Explorations, vol. 11, no. 1, pp. 10–18, 2009.
[8] “Music information retrieval exchange,” https://www.music-ir.org/mirex/
wiki/MIREX_HOME.
[9] B. Logan et al., “Mel frequency cepstral coefficients for music model-
ing.” in ISMIR, vol. 270, 2000, pp. 1–11.
[10] J. Lee, J. Park, J. Nam, C. Kim, A. Kim, J. Park, and J.-W. Ha,
“Cross-cultural transfer learning using sample-level deep convolutional
neural netowrks,” in Music Information Retrieval Evaluation eXchange
(MIREX) in the 18th International Society for Musical Information
Retrieval Conference (ISMIR). ISMIR, 2017.
[11] W. Muyuan, Z. Naiyao, and Z. Hancheng, “User-adaptive music emotion
recognition,” in Proceedings 7th International Conference on Signal
Processing, 2004. Proceedings. ICSP ’04. 2004., vol. 2, Aug 2004, pp.
1352–1355 vol.2.
[12] Q. Lu, X. Chen, D. Yang, and J. Wang, “Boosting for multi-modal
music emotion,” in 11th International Society for Music Information
and Retrieval Conference, 2010, pp. 105–105.