Electrocardiogram-based Emotion Detection for

Individuals with Cognitive Disorders using Deep Learning

and Machine Learning
Dominic L. Fernandez
Block 9 Lot 60 Phase 1 Phirst Park Homes Calamba
Barangay Palo Alto Calamba Laguna 4027
dominiclfernandez2@gmail.com
2. RELATED WORKS
ABSTRACT This section discussed related studies, works, and literature
gathered that are relevant to the subject of the study.
Nowadays, the use of artificial intelligence in our daily lives has
become normal. It solves a lot of problems and is solving present The study of Hong et.al. created two-phase ECG interpretation,
problems. It also helps researchers research individuals who have consisting of (1) a deep learning-based ECG delineation model and
disorders. (2) AF, PVC, and PAC detection algorithms based on cardiologists'
experiences, has demonstrated its efficacy. It overcomes the
This study focuses on the use of machine learning algorithms and difficulties of P wave discrimination between AF and non-AF
a deep learning algorithm to detect emotions. The datasets provided ECGs, increases the accuracy of ECG interpretation on AF, PVC,
by the author are from individuals with a cognitive disorder that was and PAC, and can assist non-cardiology physicians in diagnosing
retrieved from Mendeley. This includes the ECG signal using ECG. In addition, the proposed model and algorithms can be simply
multimodal and single modals of twenty-five (25) different adapted to the interpretation of other uncommon ECGs. We hope
individuals. that our study can promote artificial intelligence-based solutions for
ECG interpretation in clinical practice and bridge the gap between
The result of the study was not that great compared to the
research and practice. [4]
applications of machine learning and deep learning algorithms
According to Nita et.al. emotion detection using ECG signal is a
since the datasets provided are very limited and some features do
significant area of research with a promising application future.
not correlate to the target vector. To support the claim, other studies
However, existing approaches to detecting emotion using ECG are
provide evidence that the use of electrocardiogram to detect
unable to reliably determine a human's emotion. Consequently, it is
emotions are still limited, hence, still unreliable.
crucial to automate this process and more precisely forecast human
Keywords mood. In addition, training CNN to perform the task of emotion
detection using ECG signals requires an enormous amount of ECG
Electrocardiogram (ECG); Deep Learning; Machine Learning.
data. Due to the delicate nature of these medical data, it is also
1. INTRODUCTION exceedingly difficult to provide such a vast quantity of records. To
address this issue, we proposed a novel ECG data augmentation
Affective computing became a very interesting topic in research as
approach capable of producing generated data that closely mimics
it focuses on the research and development of systems and devices
current data. Then, we created a prediction model employing CNN
capable of recognizing, interpreting, processing, and simulating
as a classifier in comparison to other machine learning models. [5]
emotions, and other emotional phenomena [1] that concerns a very
vast field related to engineering, psychology, physiology, Panahi et.al. (2021) presented a method for identifying emotions
philosophy, and even neuroscience [2]. based on the fractional Fourier transform and retrieved features
from physiological inputs. The FrFT coefficients were recovered by
Some people even gathered datasets as references to use in
applying a one-dimensional FrFT to signals. These coefficients are
affective computing, including datasets from Electrocardiogram
complex, allowing for the extraction of amplitude, phase, real and
(ECG), Electroencephalogram (EEG), Electromyography (EMG),
imaginary values, and even a mixture of these characteristics. In
videos, images, text, etc. that can be used publicly to deepen the
addition, the existence of a parameter (fractional order) in the
study concerning affective computing. fractional Fourier transform provides a degree of study freedom.
In this study, the author utilized the Electrocardiogram data from Any intermediate value results in a signal representation that can be
Mendeley, they provide a young adult's ECG and GSR for emotion thought of as a rotating time-frequency representation of the signal,
recognition, which can aid with a human-computer interface for and each representation has distinct outcomes. [6]
those with cognitive and physical problems. 25 individuals had
Using the proposed DH-AS of Manohar, K., and Logashanmugam.
shimmer3 ECG and GSR sensors worn. Raw data, ECG 2022, the DH-AS has 41.5% enhanced than SVM, 26.8% better
characteristics, and self-annotation labels make up the dataset. than KNN, 18.3% elevated than DNN and 5.4% enriched than
Multimodal and single-modal.dat raw data. Multimodal data RNN. A slight decrease in the performance was observed in the
includes ECG and GSR for 12 subjects and 21 stimulus movies sensitivity value while using dataset 2. In the future, the
during three sessions. Single-modal data includes 13 participants' performance can be further improved by utilizing ensemble
ECGs. These physiological signals correlate with surprise, anger, learning with hybrid optimization algorithms. The suggested
fear, happiness, sadness, disgust, and neutrality. Each of these method regarding four datasets with optimal feature selection and
seven emotional states has five levels of intensity, for a total of 35. HDL-based classification has attained superior performance to the
Participants self-annotated emotional states, valence, arousal, and existing speech emotion recognition methods. Their research has
dominance. Each ECG sample was analyzed for 20 emotion-related implemented a new method for speech emotion recognition with
characteristics [3]. Although the datasets provided include GSR HDL-based classification. [7]
measurements, the author didn’t use the GSR as a feature to predict
the emotions using the datasets. Through a literature review, we attempt to demonstrate the status
and future direction of ECG detection and identification based on
deep learning. This effort, however, appears ineffective considering
the ever-changing objective reality. This work can only objectively
reflect the research problem's evolution over a specific time. [8]
The research of Sinha et.al. 2022, explores a new feature space in
the multilayer aspect to develop a novel classification method for
ECG beats. The hidden characteristics of the ECG waveform due to
arrhythmias are explicitly reflected in the proposed MSC and MTF-
based feature descriptors. Moreover, it is discovered that PS
analysis is an efficient method for examining the phase coupling
between the reference normal and abnormal ECG signals and that
it significantly improves the detection accuracy. [9]
The related works discussed used deep learning with a different
approach. On the other hand, this study focuses only on the ECG
signals to identify emotions. In comparison, if the accuracy of
related works discussed above is below the value of a good
accuracy the accuracy of this study is, unfortunately, lower because
of limited data and approach.
Fig
3. METHODOLOGY 2. Machine Learning and Deep Learning Model
In this section, the author discussed the model of the deep learning Fig. 2 shows how machine learning and the deep learning model
and machine learning algorithm, flowchart, and statistical analyses predict emotions. The features selected were based on the feature
conducted to supply enough information to achieve a good result in engineering conducted in this study. The gender, valence level,
the evaluation section. arousal level, and dominance level, have a good impact on the
emotions that can be used to predict the emotions. The author also
3.1 Data Gathering Procedure analyzes if there are multicollinearities, leaky features, constant
The data gathering procedure used in this study is a very values, and quasi-constant values. On the other hand, the deep
straightforward approach. Specifically, the author started to gather learning framework is a very straightforward approach as it only
data concerning electrocardiograms in emotion detection. This is uses the ECG Signal in a list/array to predict emotions.
one of the challenges encountered since the data needed in this
study should have a target vector to be fitted in the deep learning 3.3 Metrics
and machine learning model which other datasets do not have. The metrics listed in this section are the performance metrics
Succeeding is the preprocessing or the data cleaning, exploratory used in the evaluation to evaluate the model, these are the
data analysis to find the good features for the model, model following:
building, and Evaluation to measure how well the models
performed. See Figure 1, Data Gathering Procedure. R2 Score. R2 Score is used to measure if the MLA or the DLA is
not overfitting and underfitting as it affects the MLA and DLA
model greatly. The R2 Score values should be 0 – 1 as 1 indicates
perfect model fitting.
RMSE. Using their harmonic mean, the F1-score combines the
precision and recall of a classifier into a single metric. Primarily, it
is utilized to evaluate the performance of two classifiers. Consider
that classifier A has a greater recall, while classifier B has greater
precision.
Cross-Validation Score. One parameter for evaluating
classification models is the cross-validation score, just like
prediction accuracy. It enables us to make better use of our data and
provides us with a wealth of information regarding our algorithm's
performance. In complex machine learning models, it can be easy
3.2 Model to not pay sufficient attention and use the same data in multiple
pipeline steps.
There are still a lot of metrics to be used in every regression or
classification task using DLA and MLA, but as the time and
resources are limited, the above-mentioned were the only used
Fig. 1 Data Gathering Procedure performance metrics.
4. EVALUATION Algorithms R2 Cross RMSE
This section shows the result of the performance metrics of the Validation
study. Also, the analysis of the ECG signal was shown to further Score
provide clarification on the predictions created by the deep
learning algorithm.
Neural 0..0594 0.300 .1071
Table 1. Table captions should be placed above the table Network
 Ridge 0.0826 0.048 .303

Lasso -0.008 -0.023 .324

Decision Tree -0.07 0.037 .303

Random Forest -0.004 0.065 .310

Fig.
2e. Angry Emotions Signal
XGBRegressor -0.312 -0.355 .353

Table 1 shows the performance metrics of the machine learning

algorithms and a deep learning algorithm. It shows that the model
performed extremely badly. This means that the data provided do
not have a significant impact on the predictive model. In particular,
the R2 score of all algorithms is all less than 0.1 which shows that
the selected features did not show a correlation to the target vector.
In the cross-validation score, all MLA scores are below 5% which
means that the correct predictions performed by the model are very Fig. 2g. Angry Emotions Signal
low. In the root mean square error, the scores are all around 30%,
which means that the possible prediction error of the model is high.
However, the ANN performs better than the machine learning
algorithms as it scores 30% accuracy and has a particularly low
error rate than the machine learning algorithms.

Fig.
2b. Happy Emotions Signal

Fig.
2a. Sad Emotions Signal

Fig.
2d. Neutral Emotions Signal

Fig.
2c. Angry Emotions Signal

Fig. 2f. Angry Emotions Signal

Fig 2a-f shows the plotted raw data signal of the
Electrocardiogram. It shows that all values from up to 4000 Affective Computing for Late-Life Mood and Cognitive
iterations do not show significant relation to the emotions. This DisordersDing, DOI=
also proves that the existing approaches to detecting emotion https://doi.org/10.3389/fpsyt.2021.782183
using ECG are unable to reliably determine a human's emotion,
[3] Usman, A., Najam, D., Amna R. 2022. Young Adult's
the results do not differ much from the study of Nita et.al.
Affective Data (YAAD) Using ECG and GSR
5. CONCLUSION Signals, Mendeley Data, V4, DOI=
10.17632/g2p7vwxyn2.4
Affective computing is becoming more and more in demand in
the field of research as it helps solve a lot of things even in [4] Panahi, A., Rashidi, S., & Sheikhani, A. 2021. Application
medical related fields. But there are a lot of approaches to of fractional Fourier transform in feature extraction from
conducting predictive analysis. The literature gathered stated ELECTROCARDIOGRAM and GALVANIC SKIN
that approaches in detecting emotions using ECG are not RESPONSE for emotion recognition. DOI=
reliable as of the present this leads to the use of other methods https://doi.org/10.1016/j.bspc.2021.102863
such as Natural Language Processing, with the use of [5] Nita, S., Bitam, S., Heidet, M., & Mellouk, A. 2022. A
sound/audio, or with the use of computer visions, new data augmentation convolutional neural network for
In summary, the evaluation section tells us that all MLA human emotion recognition based on ECG signals. DOI=
presented in this study performs extremely badly as there are a https://doi.org/10.1016/j.bspc.2022.103580
lot of factors to consider in predicting emotions that are not [6] Hong, J., Li, H., Yang, C., Han, C., & Hsieh, J. 2022. A
included in the datasets. In particular, the features selected in clinical study on Atrial Fibrillation, Premature
MLA do not have a significant correlation to the target vector, Ventricular Contraction, and Premature Atrial
this means that the features do not affect the target vector greatly Contraction screening based on an ECG deep learning
as some MLA used are linear models. However, the deep model. DOI =
learning algorithm performs far way better than the machine https://doi.org/10.1016/j.asoc.2022.109213
learning algorithm as it resulted in 30% accuracy with only 10%
possible error. This concludes that deep algorithm learning [7] Manohar, K., & Logashanmugam, E. 2022. Hybrid deep
performs well than the machine learning algorithm. In addition, learning with optimal feature selection for speech
the result of this study does not differ from the study of Nita emotion recognition using improved meta-heuristic
et.al., their study stated that using ECG to detect emotions is still algorithm. DOI=
unreliable as of present. https://doi.org/10.1016/j.knosys.2022.108659
[8] Chen, S. W., Wang S. L., Qi, X. Z., Samuri, S. M., Yang,
6. RECOMMENDATIONS C. 2022. Review of ECG detection and classification
This section discusses the recommendations of the study for the based on deep learning: Coherent taxonomy,
sake of future development of the machine learning model, the motivation, open challenges and recommendations.
following are the author’s recommendations: DOI=
• Add more ECG datasets and features that could greatly https://doi.org/10.1016/j.bspc.2022.103493
affect the emotions of people. [9] Sinha, N., Tripathy, R., K., & Das, A. 2022. ECG beat
• Use other Deep Learning algorithms to try different classification based on discriminative multilevel
approaches. feature analysis and deep learning approach. DOI=
• Try another feature engineering method such as the https://doi.org/10.1016/j.bspc.2022.103943
wrapper method as this study uses filter methods that [10] Shulga, D. 2018. 5 Reasons why you should use Cross
provide lower accuracy than the filter method Validation in your Data Science Projects. Retrieved
• Conduct Hyperparameter Tuning to improve the from https://towardsdatascience.com/5-reasons-why-
performance of the whole model you-should use-cross-validation-in-your-data-science-
• Perform a comparative analysis between the ECG signals project
of a normal person and individuals with cognitive 8163311a1e79
disorders.
ACKNOWLEDGMENTS
The author would like to express his gratitude to his professor, Doc.
Ryan Evangelista, for patiently guiding him in the development phase
of the study. Although this study was made in a rush because of time
constraints, a very busy schedule, and health problems the study still
finishes on time. Also, thank you to all my colleagues and my family
for constantly checking on me and for cheering me up. Lastly, thank
you to the almighty God for giving me the strength to conduct and
finish this study.

REFERENCES
[1] Ho, M., Mantello, P., Nguyes, H, T., Vuong, Q-H. 2021.
Affective computing scholarship and the rise of China:
a view from 25 years of bibliometric data. Humanities
and
[2] Smith, E., Storch, E., A., Vahia I., Wong S., T.C.,
Lavretsky, H., Cummings J., L., & Eyre, H., A. 2021.