Journal of Medical Engineering & Technology, 2011; 35(6–7): 300–307

Copyright © 2011 Informa UK, Ltd.

ISSN 0309-1902 print/ISSN 1464-522X online
DOI: 10.3109/03091902.2011.601784

Review

Emotion recognition from physiological signals

K. GOUIZI1, F. BEREKSI REGUIG1 and C. MAAOUI2

1
Biomedical Electronic Laboratory, Abou bekr Belkaid University, BP 230–13000 Chetouane Tlemcen, Algeria and 2Laboratory of
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

Automation of Cooperative Systems, Paul Verlaine University, 57070, Metz, France

construct. The former considers emotional characteristics as

Emotion recognition is one of the great challenges in human–
universal and episodic, while the latter proposes to model all
human and human–computer interaction. Accurate emotion
the emotional states with several dimensions [3]. The two
recognition would allow computers to recognize human
dimensions most known are valence (positive, negative) and
emotions and therefore react accordingly. In this paper, an
activation (active/passive).
approach for emotion recognition based on physiological
In our work, we attempt emotional recognition of six emo-
signals is proposed. Six basic emotions: joy, sadness, fear,
tions: joy fear, sadness, amusement, neutrality and disgust.
disgust, neutrality and amusement are analysed using
Researchers at the Cooperative Automatic System Laboratory
physiological signals. These emotions are induced through the
(LASC Laboratory), University of Metz, led the experiment,
presentation of International Affecting Picture System (IAPS)
which consists in collecting physiological signal data using
pictures to the subjects. The physiological signals of interest
For personal use only.

Procomp hardware. In normal conditions, seven subjects test

in this analysis are: electromyogram signal (EMG), respiratory
six affective states using International Affective Picture Sys-
volume (RV), skin temperature (SKT), skin conductance (SKC),
tem (IAPS) images (NIMH Center for Emotion and Attention
blood volume pulse (BVP) and heart rate (HR). These are
(CSEA), University of Florida, FL, USA).
selected to extract characteristic parameters, which will be
The six physiological signals used for emotion recognition
used for classifying the emotions. The SVM (support vector
are skin conductance (SKC), heart rate (HR), blood volume
machine) technique is used for classifying these parameters.
pulse (BVP), respiratory volume (RV), electromyographic
The experimental results show that the proposed methodology
activity (EMG) and skin temperature (SKT).
provides in general a recognition rate of 85% for different
After the acquisition and filtering of signals, relevant pa-
emotional states.
rameters are extracted for each signal. The SVM (support vec-
Keywords:  Physiological signals, Emotion recognition, tor machine) technique is used for classifying the data, using
Characteristic parameters, SVM classifier SVM multiclass programs [4], which are adapted to our ap-
plication. In addition, a comparison between the three SVM
kernels, linear, polynomial and Gaussian, is carried out.
The remainder of this paper is organized as follows. In §2
1. Introduction
the collection of the physiological signals using multimodal
Recently, it was shown that physiological changes occupy an sensors is described. The design of the emotion recognition
important place during emotional experiences [1]. Emotion system using SVM is presented in §3. §4 is concerned with
is a psychophysiological process, produced by the limbic sys- the presentation of the results obtained by the system. Finally,
tem activity in response to a stimulus, which in turn leads to in §5, a conclusion is provided.
activation of the somatosensory system [2]. Furthermore, dif-
ferent peripheral physiological changes lead to different emo-
tions, and a corporal feedback is necessary for the emergence
2. Emotion data collection
of emotion [1]. We can represent this theory as follows: 2.1. Emotion induction
Stimulus → physiological responses → peripheral changes The experiment was carried out in the LASC Laboratory in
sensation → emotion. Metz University, France. Four subjects aged from 25 to 28
Based on psychology studies, some measures consider af- years—two men (subjects 2 and 4) and two women (subjects 1
fective states as categories, and others as a multidimensional and 3)—were asked to carry out the experiment. The different

Corresponding author. Email: hanan_khadi@yahoo.fr

(Accepted: 25 June 2011; revised 25 June 2011)

300
 Emotion recognition from physiological signals  301
physiological signals were detected from these subjects as they frequent breaths. Generally, emotions with negative valence
tested the different emotional states. The tests were carried out cause irregular breathing. The energy of this signal is in the
with an incentive system controlled by computer, scrolling range of 0.1–10 Hz [6].
IAPS images. These images are intended to induce the expres-
sion of the six emotional states (amusement, joy, disgust, fear, 2.2.4. Electromyogram signal (EMG)
neutrality and sadness). Emotional tone is an involuntary, permanent and moderate
contraction of muscle fuelled by nervous energy [7]. The
2.2. Experimental equipment slight tension that affects any muscle at rest to exaggerate the
The equipment used for this experiment is the ProComp In- effort is an expression of emotion changes as a state of mental
finiti (figure 1). This is a multimodal device with eight chan- stress. It is shown that muscle activity increases during stress
nels used for biofeedback data acquisition in real time [5]. and during emotions with negative valence [1].
Physiological signals related to emotional states are de- The EMG signal is filtered by the band pass optimal finite
tected by means of six signals: skin conductance (SKC), blood impulse response filter given by equation (1), with a band pass
volume pulse (BVP), respiratory volume (RV), electromyo- frequency of 20–125 Hz [8].
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

gram signal (EMG), skin temperature (SKT) and heart rate

(HR) estimated from the BVP signal. Each of these detected 2.2.5. Skin temperature (SKT)
signals is initially processed before it is used for extracting Changes in skin temperature are related to vasodilatation of
characteristic parameters in order to recognize the emotions. peripheral blood vessels induced by increased activity of the
sympathetic nervous system [1]. If the person is stressed the
2.2.1. Skin conductance (SKC) temperature of the body extremities decreases because blood
Skin conductance is a measure for determining the electrical is redirected to vital organs as a protection measure. The
level of the skin conductance [1]. This response is the phe- dominant energy of SKT signal is in the band 0–1 Hz [8].
nomenon in which the skin becomes momentarily a better
conductor of electricity when external or internal stimuli oc- 2.2.6. Heart rate (HR)
cur. A typical value should be about 2 μS. The SKC signal is Cardiovascular changes are necessary to prepare for action
filtered by a low-pass optimum finite impulse response filter and likely reflect emotional experiences [1]. In particular, it
given by the equation (1) with a cut-off frequency of 4 Hz. was found that the emotional valence is predicted by the heart
For personal use only.

rate. In this study the heart rate is determined through the

N-1 measurement of the successive peaks in the BVP signal.
H(e jw ) = ∑ bn e − jwn (1) Figures 2–7 show filtered samples of the physiological sig-
n=0
nals described above. These signals are related to subject 2 (a
man) and the six induced emotions.
2.2.2. Blood volume pulse (BVP)
Measurement of blood volume may provide information on 3. Feature extraction
changes in sympathetic activation. This activity, acting on the
diameter of blood vessels, leads to changes in blood volume and Feature extraction is concerned with the extraction from the
blood flow [1]. For this reason, it is taken into account when physiological signals described above of some parameters
studying the influence of emotional context. The BVP signal is which are considered relevant to emotion recognition. These
filtered by a low-pass optimum finite impulse response filter parameters are of two types: temporal parameters and fre-
given by equation (1), with a cut-off frequency of 40 Hz. quency parameters.

2.2.3. Respiratory volume (RV) 3.1. Temporal parameters

Rest and relaxation lead to slower and more superficial Let s(n) be the filtered physiological signal, and T the number
breaths. Emotional excitement and physical activities gen- of samples. The temporal parameters are defined as follows:
erate deeper breaths [3]. A stress state will be detectable by a. Mean of the physiological signal:
T
µ = 1 s(n)
s ∑ T n=1
(2)

 b. Standard deviation of the physiological signal:

1

σ s =  2
1 T
∑
 T − 1 n=1
(s(n) − µs )2 

(3)

 c. Mean of the absolute values of the first differences of the

physiological signal:

1 T−1
δ s = ∑ s(n + 1) − s(n)
T − 1 n=1
(4)
Figure 1.  ProComp Infiniti Materiel.

Copyright © 2010 Informa UK Ltd.

 302  K. Gouizi et al.
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13
For personal use only.

Figure 2.  The 9th sample of BVP signal (relative measure) for the six emotions for subject 2.

Figure 3.  The 9th sample of EMG signal (mV) for the six emotions for subject 2.

Journal of Medical Engineering & Technology

 Emotion recognition from physiological signals  303
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

Figure 4.  The 9th sample of RV signal (relative measure) for the six emotions for subject 2.
For personal use only.

Figure 5.  The 9th sample of SKT signal (°C) for the six emotions for subject 2.

Copyright © 2010 Informa UK Ltd.

 304  K. Gouizi et al.
 d. Mean of the absolute values of the first differences of the  where lk are the labels, xk are input vectors, being in a space
normalized signal: RN and p is the size of the training set.
The technique seeks a separating hyperplane h(x) = wx +
1 T−1 δ w0 which minimizes the number of errors through the intro-
δs = ∑ s(n
T − 1 n=1
 + 1) − s(n)
 = s
σs
(5)
duction of variable spring ξk, which can relax the constraints
on the training vectors [12].
 e. Mean of the absolute values of the second differences of the
physiological signal: l k (w x k + w 0 ) ≥ 1 − ξk , ξk > 0, 1 ≤ k ≤ p (13)

1 T −2  With the previous constraints, the optimization problem is

γ s = ∑ s(n + 2) − s(n)
T − 2 n=1
(6) modified by a penalty term which penalizes high variables,
spring ξk:
 f. Mean of the absolute values of the second differences of the
normalized signal: 1
Minimise w + C∑ ξk ,
2
p
C> 0 (14)
2 K=1
T −2
1
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

γs = ∑ s(n
T − 2 n=1
 + 2) − s(n)
 (7)  where C is a constant that controls the compromise between
the number of classification errors and the margin width (the
 The six parameters from a to f are called the Picard param- margin width being the smallest distance between training set
eters [9]. and separating hyperplane).
g. Ratio max/min of the physiological signal: Different kernel functions can be used. Such functions can
be as cited below [13]:
R = max(s(n))/min(s(n)) (8) a. Linear kernel:
 h. Euclidian distance: Let sN(n) be the signal to the neutral K(x i , x j ) = x i . x j (15)
emotion. The Euclidean distance between the signal to such
an emotion and the signal sN(n) is calculated as follows:  b. Polynomial kernel:
1
K(x i , x j ) = (x i . x j + c)n (16)
Ds =  2 2
1 T
∑
For personal use only.

s(n) − s N (n)  (9)

 T − 1 n=1   c. Gaussian kernel:
 x −x 2 
K(x i , x j ) = exp  
i j

 (17)
 σ
2
3.2. Frequency parameters  
As frequency parameter evaluation considered in this study
the mean and the standard deviation of the spectral coher-  The SVM method is applicable for binary classification tasks,
ence function. Let Sq (ω) and SN (ω) be respectively the signal but there are extensions to the multiclass classification [14].
spectrum to such an emotion and the signal spectrum to the Formally, the training and testing samples can be ordered
neutral emotion. The absolute interspectral density can be here in M classes {C1, C2,..., CM}. The one against all method
given as [10]: consists of constructing M binary classifiers by assigning the
label 1 to samples of one class and the label−1 to all others[14].
|SqN (ω) |
2
≤ Sq(ω) SN (ω) (10) In the test phase, the classifier that gives the highest margin
wins the vote.
 This value can be normalized to give the spectral coherence
function: Class of x = arg maxk (hk(x)), k Є {1,….,M} (18)

SqN (ω)2 4.1. Computational emotion recognition experiments

γ = 2
xy ≤1 (11) The data classification is carried out by programming the
SN (ω)Sq (ω)
 SVM method one against all SVM multiclass. This has been
tested considering the kernel functions linear, polynomial
 Then, we extract the mean meancoh and the standard devia-
and Gaussian. Also in this experiment, and for every emo-
tion σcoh of the spectral coherence function.
tion, eight samples are used for training and two samples for
testing.
4. Emotion recognition After several tests, it was found empirically that for the
polynomial kernel the most appropriate degree is n = 2 and
For the recognition of emotion using the parameters stated
for the Gaussian kernel σ = 1 and C = 1000. The results are
above, the support vector machine (SVM) method is used.
shown in tables 1–4. To study the efficiency of each relevant
SVM is a binary classification method by supervised learning
parameter on the recognition rate relating each kernel type
[11]. The aim is to learn the h(x) function through a training
of kernel function, we extract the latter for several combina-
set given below:
tions of relevant parameters and for the four subjects (see
{(x1, l1), (x2, l2)……(xp, lp)} ЄRN × {–1,1} (12) tables 1–4). From these tables it can be seen that for the four

Journal of Medical Engineering & Technology

 Emotion recognition from physiological signals  305
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

Figure 6.  The 9th sample of SKC signal (μS) for the six emotions for subject 2.
For personal use only.

Figure 7.  The 9th sample of HR signal (bat.min-1) for the six emotions for subject 2.

subjects, linear and polynomial kernels generate improved that pertinent parameters extracted from physiological data
recognition rates compared to the Gaussian kernel. Similarly, are relatively separable.
using the linear kernel, we obtain recognition rates relatively Tables 1–4 show that if we combine the six Picard ­parameters
improved compared to the polynomial kernel. This means with the ratio R or the six Picard parameters with the coher-

Copyright © 2010 Informa UK Ltd.

 306  K. Gouizi et al.

Table 1.  Recognition rate for different combinations of parameters for Table 6.  Recognition rate for different combinations of physiological signals
subject 1. for subject 2.
Linear (%) Poly (%) Gauss (%) Linear (%) Poly (%)
6 Picard parameter 50 41.67 16.67 BVP 58.33 25
6 Picard parameter R 75 66.67 58.33 Fc 50 33.33
6 Picard parameter D 66.6 66.67 25 EMG 25 41.67
6 Picard parameter meancoh, σcoh 75 75 50 SKC 50 25
All parameters 83.3 75 25 SKT 58.33 50
RV 33.33 8.33
BVP,Fc 50 41.67
Table 2.  Recognition rate for different combinations of parameters for
subject 2. BVP,Fc,EMG 58.33 58.33
Linear (%) Poly (%) Gauss (%) BVP,Fc,EMG,RV 75 41.67
6 Picard parameter 66.67 58.33 33.33 BVP,Fc,EMG,RV,SKT 75 50
6 Picard parameter R 75 41.67 25 All signals 83.33 66.67
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

6 Picard parameter D 58.33 50 25

6 Picard parameter meancoh, σcoh 66.67 75 33.33 Table 7.  Recognition rate for different combinations of physiological signals
All parameters 83.33 66.67 16.67 for subject 3.
Linear (%) Poly (%)
BVP 33.33 16.67
Table 3.  Recognition rate for different combinations of parameters for
subject 3. Fc 33.33 25
EMG 50 41.67
Linear (%) Poly (%) Gauss (%)
SKC 50 41.67
6 Picard parameter 50 41.33 16.67
SKT 41.67 33.33
6 Picard parameter R 75 50 33.33
RV 50 41.67
6 Picard parameter D 50 58.33 16.67
BVP,Fc 41.67 25
6 Picard parameter meancoh, σcoh 75 66.67 33.33
BVP,Fc,EMG 66.67 33.33
All parameters 83.33 75 16.67
BVP,Fc,EMG,RV 75 66.67
For personal use only.

BVP,Fc,EMG,RV,SKT 75 66.67
Table 4.  Recognition rate for different combinations of parameters for All signals 83.33 75
subject 4.
Linear (%) Poly (%) Gauss (%)
6 Picard parameter 75% 66.67% 41.67% Table 8.  Recognition rate for different combinations of physiological signals
for subject 4.
6 Picard parameter R 83.33% 75% 50%
6 Picard parameter D 66.67% 50% 25%
Linear (%) Poly (%)
BVP 41.67 16.67
6 Picard parameter meancoh, σcoh 91.67% 91.67% 41.67%
Fc 50 41.67
All parameters 91.67% 83.33% 25%
EMG 58.33 50
SKC 25 33.33
SKT 33.33 50
Table 5.  Recognition rate for different combinations of physiological signals
for subject 1. RV 41.67 50
Linear (%) Poly (%) BVP,Fc 50 25
BVP 50 16.67 BVP,Fc,EMG 58.33 66.67
Fc 33.33 25 BVP,Fc,EMG,RV 66.67 66.67
EMG 58.33 25 BVP,Fc,EMG,RV,SKT 83.33 66.67
SKC 25 33.33 All signals 91.67 83.33
SKT 25 41.67
RV 33.33 25 signal alone cannot give improved recognition rates of emotions.
BVP,Fc 58.33 25 For the four subjects, the tables also show that if we increase the
BVP,Fc,EMG 66.67 58.33 number of physiological signals which are combined, we obtain
BVP,Fc,EMG,RV 66.67 66.67 improved recognition rates. The best recognition rate of 83.33%
BVP,Fc,EMG,RV,SKT 66.67 66.67 is obtained when all physiological signals are combined.
All signals 83.33 75 From these results, we conclude that combination of physi-
ological signals is feasible and efficient to give good recogni-
ence function, the recognition rate is improved compared to tion rates of emotions.
the results found using only six Picard parameters. Thus, we
conclude that the six Picard parameters, coherence function,
5. Conclusion
ratio R and distance D characterize well the six emotions.
Tables 5–8 show the results obtained by combining different In this paper, an approach to emotion recognition has been
physiological signals. These tables show that one physiological proposed, studied and evaluated. This approach is based on

Journal of Medical Engineering & Technology

 Emotion recognition from physiological signals  307
first the analysis and processing of physiological signals such Transactions on Neural Networks. Automatic and Cooperatifs Sys-
tems laboratory, Metz University. France, pp. 1–2.
as SKC, BVP, RV, EMG, SKT, and HR estimated from the BVP 4 Rakotomamonjy, A., 2005, SVM and Kernel Methods (Matlab, Insa de
signal; then emotion recognition using the multiclass SVM Rouen, France).
technique. The results showed high performance in recogniz- 5 Thought Technology, 2003, Guide de l’appareil ProCompInfiniti.
(Montréal, Canada: Thought Technology)), pp. 8, 23–24, 32–35.
ing the six emotions considered in this study. The obtained 6 Webster, J.G., 2006, Encyclopedia of Medical Devices and Instrumenta-
results showed that the recognition rate varies according to tion, Volume 5, 2d edn (Wiley Interscience, University of Wisconsin–
several criteria: Madison, Madison, USA), p. 21. Available online at: http://www.mrw.
interscience.wiley.com/emdi
selection and combination of physiological signals associ- 7 Guiose, M., 2003, Fondements théoriques et techniques de la relax-
ated with the induced emotions; ation. Report, Faculty de medicine, Paris-VI University, November,
selection of characteristic parameters to be extracted from pp. 21–23.
8 Towe, B.C., 2004, Bioelectricity and its measurement. In: Standard
physiological signals; and Handbook of Biomedical Engineering and Design, Part 4, Chapter 17,
selection of the multiclass SVM kernel. pp. 26–43. Digital Engineering Library, McGraw-Hill, Arizona State
University, Tempe, Arizona. Available online at: www.digitalengineer-
inglibrary.com
Declaration of interest: the author declares no conflict of 9 Takahashi, K., 2004, Remarks on emotion recognition from bio-
J Med Eng Technol Downloaded from informahealthcare.com by University of Notre Dame Australia on 05/15/13

interest. potential signals. IEEE Transactions on Neural Networks, Doshisha

University, Kyoto, Japan, pp. 188–189.
10 Gaillard, P., and Lengellé, R., 2006, Analyse et traitement du signal.
References Ellipes Edition, pp. 110, 257, 185.
11 Mohamadally, H., and Boris, F., 2006, SVM: Machines à Vecteurs de
1 Rivière, A., and Godet, B., 2003, L’affective computing: rôle adaptatif Support ou Séparateurs à Vastes Marges. Report, January.
des émotions dans l’interaction homme–machine. Report, Charles de 12 Asselin, J.P., and Kettaf, F.Z., 2005, Bases théoriques pour l’apprentissage
Gaulle University, Lille, France. pp. 9–12, 33–38. et la décision en reconnaissance des formes. Cedadues edition.
2 Lalanne, C., 2005, La cognition: l’approche des Neurosciences Cogni- 13 Cornuéjols, A., 2002, Une nouvelle méthode d’apprentissage: Les sé-
tives. Report, Département d’informatique, René Descartes Univer- parateurs à vaste marge. Report, Orsay University, France, pp. 18–22.
sity, Paris, France. pp. 26–28. 14 Hsu, C., and Lin, C.-J., 2002, A comparison of methods for multi-class
3 Maaoui, C., and Pruski, A., 2008, A comparative study of SVM ker- support vector machines. IEEE Transactions on Neural Networks,
nel applied to emotion recognition from physiological signals. IEEE 415–425.
For personal use only.

Copyright © 2010 Informa UK Ltd.