EEG-based Emotion Recognition Using Nonlinear Feature
EEG-based Emotion Recognition Using Nonlinear Feature
Jingjing Tong, Shuang Liu*, Yufeng Ke, Bin Gu, Feng He, Baikun Wan, Dong Ming
Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University
Tianjin, China
liushuang163163@163.com
Abstract— Emotions are ubiquitous components of everyday 2015, Aayush Bhardwaj [9] et.al classified the emotion elicited
life, as they influence behavior to a large extent. And Emotion by IAPS based on the basic emotion model( basic emotion:
recognition is one of the most important and necessary parts in happy, sad, disgust, neutral, fear, surprised and anger). Also
the field of emotion research. Its accuracy relies heavily on the they extracted EEG signals as features and get a 74.13%
ability to generate representative features. However, this is a very classification accuracy.
challenging problem. In this study, EEG nonlinear features,
power spectrum entropy and correlation dimension, were Usually, the EEG features are extracted from the time
extracted to differentiate emotions. International Affective domain, frequency domain and time-frequency domain. The
Picture System (IAPS) pictures with different valence but similar time domain analysis method mainly analyzes the geometric
arousal level were used to induce the emotions with 8 valence properties of EEGs. Ma Li et.al. [10] proposed an emotion
levels. The results showed that the valence levels were positively recognition system and based on the features they selected in
correlated with these two features, especially in the frontal lobe. time domain (amplitude, average and variance of brain waves)
Based on the two features, SVM gave an average accuracy of SVM could give an average accuracy of 86.25% to classify
82.22%. Analyzing the nonlinear features of EEGs is an efficient happy, relaxed and sad. Feature extraction in frequency domain
way to classify emotions. based on power spectral analysis has a mature process and
widely used in EEG analysis. And there are some pervasive
Keywords—EEG; emotion recognition; power spectrum
features are used in distinguishing emotional EEG such as the
entropy; correlation dimension
ratio of the power spectrum with different frequencies[11], the
I. INTRODUCTION sub band power spectral density[11], the power spectrum
energy[12,13] and so on. Time-frequency characteristic
Affective Computing is proposed by as “computing that provides the information about the energy distribution of the
relates to, arises from or deliberately influences emotions” [1]. time domain and frequency domain, clearly describing the
Domestic scholars hold the idea that affective computing aims relationship between the signal frequency and the changing
at to give computers the ability to recognize, understand and time. Hadjidimitriou S K et.al. [14] employed the Hilbert–
adapt to user’s mental states, and then improving the Huang spectrum (HHS) which was found to be robust against
interactions between digital devices and their users [2]. Thus it noise corruption as the emotional EEG features. Since brain is
is essential to correctly identify and recognize these emotions. a highly complex and nonlinear system, the nonlinear features
Currently, there are numerous studies measuring the should be considered in analyzing emotional EEGs. In 1985,
emotional states by analyzing physiological signals under the Baloyantz et al. firstly applied the nonlinear dynamics and
emotional stimuli, including the Brain Imaging Technology chaos theory to EEG analyses and then revealed that sleep EEG
[3,4] (EEG, fMRI), peripheral physiological signals[5] (Heart was the chaotic signal [15]. In recent years more attentions
Rate, Blood Pressure, EMG, Galvanic Skin Resistance and so have been devoted to nonlinear features, such as entropy [16],
on) and a combination of some abovementioned methods [6]. correlation dimension and fractal dimension of nonlinear. Liu,
Among these, EEG is widely used for its objectivity and et.al [17] extracted the nonlinear characteristics of EEG's
accuracy in recognizing human emotions. Moreover, EEG is fractal dimension and set up the online automatic emotion
the direct response of brain activity with high temporal recognition system.
resolution. Greg Hajcak, et.al[7] used EEGs to study children's This paper introduced nonlinear dynamics principle and
brain patterns when dealing with emotional images. They technology into the emotional research based on EEG. Power
found that a positivity maximal at occipital-parietal recording spectral entropy (PSE) and correlation dimension were elicited
sites was increased from 500 to 1000 ms following pleasant to analyze the corresponding relationship between the
pictures. In 2008, Horlings R [8] classified the emotion elicited emotional levels and EEG nonlinear features in channel AF3,
by IAPS based on the valence-arousal model. Extracting EEG AF4, F3, F4. And three emotions, positive, neutral and
signals as features, they get a 37% classification accuracy in negative emotions, were classified by the SVM classifier.
three classification of valence and 49% of arousal. While in
II. MATERIAL AND METHOD temperature is kept in 23 to 25 °C. During the experiment, 32-
channels EEG signals were recorded continuously using a
A. Emotion model Neuroscan4.5 amplifier system. The electrodes were placed on
In this paper, the valence-arousal emotion model was the scalp according to the extension of the international 10-20
chosen to express the emotions as it will be easier to match the electrode positioning system. [18]. And Fig.2 shows 64-
physiological parameters to dimensional construct of emotions. channels EEG cap layout used in this study. Reference
And the valence scale was ranged from 1 to 9. And the higher potential is near the CZ. The signals were digitized at 1, 000
the score, the more positive the mood. Thus emotion was Hz and stored in a PC for offline analysis.
divided into eight ratings by valence. Tab. 1 shows the score of
each emotion rating.
Entropy refers to the degree of disorder in a system, also reconstructed vector. The distances rij=|Xi-Xj | between any pair
can be understood as the probability of a discrete random of points can be calculated as function (6). For any positive
event. The smaller the probability of the event, the higher the number r, the correlation sum Cm(r) is then defined as the
uncertainty, and the more the information the event provided. fraction of pairs whose distance is smaller than r.
According to the definition of information entropy, the amount
ଶ ଵȀଶ
of information of an uncertain system can be describe as ݎ ൌ ݀൫ܺ ǡ ܺ ൯ ൌ ቂσିଵ
ୀ ൫ݔାൈఛ െ ݔାൈఛ ൯ ቃ ݅ ് ݆
function (1). H is the entropy, and pi refers to the probability of (7)
state i.
ଵ
ܪൌ െ σ ݈݃ כ (1) ܥ ሺݎሻ ൌ σே ே
ୀଵ σୀଵ ߠሺ ݎെ ݎ ሻ (8)
ேሺேିଵሻ
Power spectral entropy (PSE) is regarded as the information ͳ݂ ݎݎ ݎ
entropy in frequency domain. And PSE is computed as follows: ߠሺݎሻ ൜
Ͳ݂ ݎݎ൏ ݎ
ି
మഏ
Due to Nmis much greater than 1, so Cm (r) can also be
ሺ݇ሻ ൌ σேିଵ
ୀ ݔሺ݊ሻ݁ ಿ ൌ σேିଵ
ୀ ݔሺ݊ሻே (2) described as function (9). When r is small enough, the
correlation integral function approaches to function (10).
ଵ
ሺ݇ ൌ Ͳǡͳǡ ǥ ǡ ܰ െ ͳሻ ܥ ሺݎሻ ൌ σே ே
ୀଵ σୀଵ ߠሺ ݎെ ݎ ሻ (9)
ேమ
ଵ
Ƹ ሺ݇ሻ ൌ ȁܺሺ݇ሻȁଶ (3) ݈݊ܥ ሺݎሻ ൌ ݈݊ ܥ ܦሺ݉ሻ݈݊ݎ (10)
ே
ሺሻ D(m) is called the correlation dimension of J(m), can be
ൌ σ σ ൌ ͳ (4) can be calculated by linear regression of lnC_m (r)~lnr curve
ೖ ሺሻ
(function (11)). In this paper, we take the fitted slope of middle
x(n),(n=0,1,…,N-1) is a discrete time sequence of length N, part of the curve as correlation dimension.
and its sample frequency is fs. X(k) is the frequency spectrum
sequence gotten from FFT of x(n). And p (ࡂ k) is effective power
డ ሺሻ
spectral density values on frequency fk=fs/N*k. pk is the ܦሺ݉ሻ ൌ ݈݅݉ (11)
՜ డ
normalization of p (ࡂ k). Regarding the signal between fk and
fk+fs/N as a signal event, then the pk becomes the probability of III. RESULTS
this event. Then we can calculate the entropy of frequency
spectral by using formula (1). A. results in feature extraction
1) results in power spectral entropy
The power spectral entropy is used to characterize the
The PSE of 13 subjects × 8 valence levels are computed for
complexity of a system in frequency domain. A sign wave with
all six brain regions (region F, AT, C, P, PT, O). And Fig. 3
a frequency of 1Hz or 100Hz and a straight line are both a line
described the average PSE in six regions of all participants. In
in power spectrogram. The calculated power spectral entropy
region P, PT and O, the PSE was not changing with the valence
will both be 0. Thus the greater the uncertainty of signals in
rate obviously. It implied that these regions may be not the
time domain, the greater the power spectral entropy, the more
emotion relative brain areas. While for region F, AT and C, the
complex the system.
PSE changed obviously with valence rate. Especially in region
Correlation dimension F, the PSE had a rising trend with the valence level. And the
PSE in region F is positive correlated with the valence level
In chaos theory, the correlation dimension (CD) is a (r=0.894, p=0.003). That was to say, when watching the
measure of the dimensionality of the space occupied by a set of pictures with lower valence rate the brain waved more
random points, often referred to as a type of fractal dimension regularly. And with the increase of valence level, the brain
[20]. It is widely used in analyzing the harmonic component of waved less regularly with the PSE increases accordingly. It was
power system and biomedical signal owe to the advantage of possible that when the subjects were in a highly unpleasant
being straightforwardly and quickly calculated, of being less state, their brain activities were peaceful and regular. While the
noisy when only a small number of points is available. subjects felt pleasant, their brain activities were active.
Grassberger-Proccacia(GP) algorithm is applied to
2) Results in Correlation dimension
calculate correlation dimension. And according to this
Fig. 4 illustrated the CD in six regions of all participants. It
algorithm, the dynamics of a system is reconstructed, in an
was in line with the results in PSE. In region P, PT and O, the
embedding space of dimension M (J(m)), with the time series
CD was not changing with the valence rate obviously while in
of a single variable using delay coordinates scanned at a
region F, AT and C, the increasing trend was obvious which
prescribed time delay, IJ.
suggesting that the higher the valence level, the higher the
ܬሺ݉ሻ ൌ ሾܺଵ ǡ ܺଶ ǡ ǥ ǡ ܺே ሿ (5) chaotic properties. Especially in region F, the CD had a rising
trend with the level. And the CD in region F is positive
ܺ ሺ݉ǡ ܮሻ ൌ ൫ݔ ǡ ݔାఛ ǡ ݔ ڮାሺିଵሻఛ ൯݊ ൌ ͳǡʹǡ͵ǡ ǥ ܰ correlated with the valence level (r=0.885, p=0.003).When in a
(6) state of positive emotion, the complexity of prefrontal cortex
{xk: k=1,2,3,…,N} is a discrete time sequence of length N. EEG was higher than in negative state. The fitting error of
J(m) is the embedding space of dimension M. And Xn is the
region F was minimum, that was to say in region F, we can (the standard scores and the reality scores). Figure 5 described
assume a linear interpolation between valence and CD. the accuracy of all participants.
PSE and CD are not changing with the valence obviously. In [4] S. Erk, A. Kleczar, and H. Walter, "Valence-specific regulation effects
conclusion, our findings, as in previous, might illustrate that in a working memory task with emotional context," Neuroimage, vol.
37, p. 623, 2007.
anterior and central cortex is the emotion related regions, while
[5] L. K. Hildebrandt, C. Mccall, H. G. Engen, and T. Singer, "Cognitive
the posterior area in brain is not fount connected with emotion flexibility, heart rate variability, and resilience predict fine-grained
states. regulation of arousal during prolonged threat," Psychophysiology, vol.
53, pp. 880-890, 2016.
Using the PES and CD, we investigated if the nonlinear
[6] S. Koelstra, A. Yazdani, M. Soleymani, “Single Trial Classification of
features of EEG are distinguishable and got a relative high EEG and Peripheral Physiological Signals for Recognition of Emotions
average classification accuracy (power spectral entropy: Induced by Music Videos”, Brain Informatics. Springer Berlin
79.58%, correlation dimension: 82.22%). PSE and CD can be Heidelberg, 2010.
improved to represent the brain emotional states to a certain [7] G. Hajcak and T. A. Dennis, "Brain potentials during affective picture
extent. While the classification accuracy is relative with the processing in children," Biological Psychology, vol. 80, pp. 333-338,
signal types and feature numbers. In 2010, Sander Koelstra[6] 2009.
et.al. classified the emotion elicited by music videos based on [8] R. Horlings, D. Datcu, and L. J. M. Rothkrantz, "Emotion recognition
using brain activity," in: International Conference on Computer Systems
the valence-arousal model. Extracting only EEG signals as and Technologies and Workshop for Phd Student in Computing ACM,
features, they get a 76% classification accuracy in three 2008, pp. 1-6.
classification of valence and 67% of arousal. While Extracting [9] A. Bhardwaj, A. Gupta, P. Jain, A. Rani, and J. Yadav, "Classification of
the peripheral physiological signals (including galvanic skin human emotions from EEG signals using SVM and LDA Classifiers,"
response, respiration amplitude, skin temperature, in: International Conference on Signal Processing and Integrated
electrocardiogram, blood volume by plethysmograph, Networks, 2015, pp. 180-185.
electromyograms of Zygomaticus and Trapezius muscles and [10] M. Li, Q. Chai, T. Kaixiang, A. Wahab, and H. Abut, "EEG Emotion
Recognition System," Spring US, 2009.
EEG) as features, they get a 78.5% classification accuracy in
two classification of valence and 85.5% of arousal. Here we [11] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, et
al., "EEG-based emotion recognition in music listening," IEEE
only collected EEG, which has a very high time resolution, but transactions on bio-medical engineering, vol. 57, pp. 1798-806, 2010
a low spatial resolution. The classified accuracy might be [12] P. C. Petrantonakis and L. J. Hadjileontiadis, "A novel emotion
improved by combining the EEG signal with some high spatial elicitation index using frontal brain asymmetry for enhanced EEG-based
resolution signals, such as NIRS, etc. NIRS can be used to emotion recognition," IEEE Transactions on Information Technology in
measure the concentration of the oxygenated hemoglobin and Biomedicine, vol. 15, pp. 737-746, 2011.
the deoxidation hemoglobin in the blood, accurately reflecting [13] D. Nie, X. W. Wang, L. C. Shi, and B. L. Lu, "EEG-based emotion
the regions of brain's arousal [23]. recognition during watching movies," in: International IEEE/EMBS
Conference on Neural Engineering, 2011, pp. 667-670.
In addition, this paper has some deficiencies. For example, [14] S. K. Hadjidimitriou and L. J. Hadjileontiadis, "Toward an EEG-Based
the classification to emotion here is rough. We now can only Recognition of Music Liking Using Time-Frequency Analysis," IEEE
distinguish three kinds of emotion represent negative (valence transactions on bio-medical engineering, vol. 59, pp. 3498-3510, 2012.
1), neutral (valence5) and positive (valence 8), while it is hard [15] A. Babloyantz, J. M. Salazar, and C. Nicolis, Evidence of chaotic
dynamics of brain activity during the sleep cycle: Springer Berlin
to identify all eight valence levels. Therefore, we can introduce Heidelberg, 1986.
some new analysis methods to look for new features to
[16] H. Xu and K. N. Plataniotis, "Affect recognition using EEG signal," in:
distinguish the emotions like phase change and correlation IEEE International Workshop on Multimedia Signal Processing, 2012,
analysis. The channel that mainly analyzed in this article is pp. 299-304.
selected by referring articles. In later work, we can use some [17] Y. Liu, O. Sourina, and M. K. Nguyen, Real-Time EEG-Based Emotion
algorithms (e.g. support vector machine, artificial neural Recognition and its application, Springer Berlin Heidelbeg, 2011.
network) to optimize the 64 channels and then obtain the [18] V. Jurcak, D. Tsuzuki, and I. Dan, "10/20, 10/10, and 10/5 systems
unaffected recognition rate. revisited: Their validity as relative head-surface-based positioning
systems," Neuroimage, vol. 34, p. 1600, 20
ACKNOWLEDGMENT [19] A. R. Hidalgo-Muñoz, M. M. López, A. T. Pereira, I. M. Santos, and A.
M. Tomé, "Spectral turbulence measuring as feature extraction method
This research was supported by National Natural Science from EEG on affective computing," Biomedical Signal Processing &
Foundation of China (No. 91520205, 81222021) and National Control, vol. 8, pp. 945-950, 2013.
Key Technology R&D Program of the Ministry of Science and [20] P. Grassberger and I. Procaccia, "Measuring the strangeness of strange
Technology of China (No. 2012BAI34B02). attractors," Physica D Nonlinear Phenomena, vol. 9, pp. 189-208, 1983.
[21] J. A. Coan and J. J. Allen, "Frontal EEG asymmetry as a moderator and
REFERENCES mediator of emotion," Biological Psychology, vol. 67, pp. 7-50, 2004.
[1] R. W. Picard, Affective computing: MIT Press, 1997. [22] R. J. Davidson, "Cerebral asymmetry and emotion: Conceptual and
methodological conundrums," Cognition & Emotion, vol. 7, pp. 115-
[2] L. Li, L. Cheng, K. X. Qian,”An e-learning systerm model based on
138, 1993.
affective computing,” International Conference on Web-based Learning,
vol. 226, pp. 54-56, 2008. [23] H. Ito, H. Yamauchi, H. Kaneko, T. Yoshikawa, K. Nomura, and S.
Honjo, "Prefrontal overactivation, autonomic arousal, and task
[3] K. P. Thomas, C. Guan, C. T. Lau, A. P. Vinod, and K. K. Ang,
performance under evaluative pressure: a near-infrared spectroscopy
"Adaptive tracking of discriminative frequency components in
(NIRS) study," Psychophysiology, vol. 48, pp. 1563-1571, 2011.
electroencephalograms for a robust brain-computer interface," Journal of
Neural Engineering, vol. 8, p. 036007, 2011.