Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com

ScienceDirect
ScienceDirect
Available online at www.sciencedirect.com
Procedia Computer Science 00 (2018) 000–000
Procedia Computer Science 00 (2018) 000–000
ScienceDirect www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia

Procedia Computer Science 155 (2019) 698–703

2nd
2nd International Workshop on
International Workshop on Recent
Recent advances
advances onon Internet
Internet of
of Things:
Things: Technology
Technology and Application
and Application
Approaches ( IoT-T&A 2019)
Approaches (IoT-T&A 2019)
August
August 19-21, 2019, Halifax,
19-21, 2019, Halifax, Canada
Canada

Human
Human Activity
Activity Recognition:
Recognition: A
A Survey
Survey
Charmi
Charmi Jobanputra
Jobanputraa ,, Jatna
a
Jatna Bavishi
Bavishib,, Nishant
b
Nishant Doshi
c*
Doshic*
a,b,c
Pandit Deendayal Petroleum University, Raisan,Gandhinagar – 382007, India
a,b,c
Pandit Deendayal Petroleum University, Raisan,Gandhinagar – 382007, India

Abstract
Abstract
Human Activity Recognition (HAR) has been a challenging problem yet it needs to be solved. It will mainly be used for eldercare and
Human Activity Recognition (HAR) has been a challenging problem yet it needs to be solved. It will mainly be used for eldercare and
healthcare as an assistive technology when ensemble with other technologies like Internet of Things(IoT). HAR can be done with the help of
healthcare as an assistive technology when ensemble with other technologies like Internet of Things(IoT). HAR can be done with the help of
sensors, smartphones or images. In this paper, we present various state-of-the-art methods and describe each of them by literature survey.
sensors, smartphones or images. In this paper, we present various state-of-the-art methods and describe each of them by literature survey.
Different datasets are used for each of the methods wherein the data are collected by different means such as sensors, images, accelerometer,
Different datasets are used for each of the methods wherein the data are collected by different means such as sensors, images, accelerometer,
gyroscopes, etc. and the placement of these devices at various locations. The results obtained by each technique and the type of dataset are then
gyroscopes, etc. and the placement of these devices at various locations. The results obtained by each technique and the type of dataset are then
compared. Machine learning techniques like decision trees, K-nearest neighbours, support vector machines, hidden markov models are
compared. Machine learning techniques like decision trees, K-nearest neighbours, support vector machines, hidden markov models are
reviewed for HAR and later the survey for deep neural network techniques like artificial neural networks, convolutional neural networks and
reviewed for HAR and later the survey for deep neural network techniques like artificial neural networks, convolutional neural networks and
recurrent neural networks is also presented.
recurrent neural networks is also presented.

©
© 2019
2019 The Authors. Published by by Elsevier
Elsevier B.V.
B.V.
© 2019 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND
BY-NC-NDlicense
license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility
Peer-review under responsibility of theConference
Conference Program Chairs.
Peer-review under responsibilityofofthe
the ConferenceProgram
Program Chairs.
Chairs.
Keywords: Human Activity Recognition, Machine learning, Neural networks.
Keywords: Human Activity Recognition, Machine learning, Neural networks.

1. Introduction
1. Introduction
The field of Human Activity Recognition (HAR) has become one of the trendiest research topics due to availability of sensors
The field of Human Activity Recognition (HAR) has become one of the trendiest research topics due to availability of sensors
and accelerometers, low cost and less power consumption, live streaming of data and advancement in computer vision, machine
and accelerometers, low cost and less power consumption, live streaming of data and advancement in computer vision, machine
learning, artificial intelligence and IoT.
learning, artificial intelligence and IoT.
In HAR, various human activities such as walking, running, sitting, sleeping, standing, showering, cooking, driving, opening
In HAR, various human activities such as walking, running, sitting, sleeping, standing, showering, cooking, driving, opening
the door, abnormal activities, etc. are recognized. The data can be collected from wearable sensors or accelerometer or through
the door, abnormal activities, etc. are recognized. The data can be collected from wearable sensors or accelerometer or through
video frames or images. HAR can be extensively used in medical diagnosis. For keeping track of elderly people, HAR can be
video frames or images. HAR can be extensively used in medical diagnosis. For keeping track of elderly people, HAR can be
used. Crime rates can be controlled using HAR by monitoring. The smart home environment can be created by the daily activity
used. Crime rates can be controlled using HAR by monitoring. The smart home environment can be created by the daily activity
recognition. Driving activities can be recognized and lead to safe travel. Military actions can be recognized using HAR.
recognition. Driving activities can be recognized and lead to safe travel. Military actions can be recognized using HAR.
The paper is divided into various state-of-the-art methods for human activity recognition and the challenges for activity
The paper is divided into various state-of-the-art methods for human activity recognition and the challenges for activity
recognition. Section 2 describes various state-of-the-art methods. Section 2.1 describes the review and comparison of machine
recognition. Section 2 describes various state-of-the-art methods. Section 2.1 describes the review and comparison of machine
learning methods for HAR such as decision trees, K-nearest neighbours(KNN), support vector machines(SVM) and hidden
learning methods for HAR such as decision trees, K-nearest neighbours(KNN), support vector machines(SVM) and hidden
Markov model(HMM). Section 2.2 describes neural network models such as artificial neural networks(ANN), convolutional
Markov model(HMM). Section 2.2 describes neural network models such as artificial neural networks(ANN), convolutional
neural networks(CNN) and recurrent neural networks(RNN). Fig. 1.(a) summarizes the techniques for HAR. At last, in section 3,
neural networks(CNN) and recurrent neural networks(RNN). Fig. 1.(a) summarizes the techniques for HAR. At last, in section 3,
the open issues and challenges for activity recognition are described. Finally, section 4 presents the conclusion.
the open issues and challenges for activity recognition are described. Finally, section 4 presents the conclusion.
*Corresponding author. Tel.:+917923275458.
*Corresponding
Email author. Tel.:+917923275458.
address: Nishant.Doshi@sot.pdpu.ac.in
Email address: Nishant.Doshi@sot.pdpu.ac.in

1877-0509 © 2019 The Authors. Published by Elsevier B.V.

1877-0509 © 2019 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
© 2019
1877-0509 under
Peer-review The Authors.
responsibility Published
of the byProgram
Conference ElsevierChairs.
B.V.
Peer-review under responsibility of the Conference Program Chairs.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the Conference Program Chairs.
10.1016/j.procs.2019.08.100
2 Charmi Jobanputra et al / Procedia Computer Science 00 (2018) 000–000
Charmi Jobanputra et al. / Procedia Computer Science 155 (2019) 698–703 699

Fig. 1. (a) HAR state-of-the-art methods.

Fig 1. (a) Ref- ML: Machine Learning, NN: Neural Networks, KNN: K-Nearest Neighbour, DT: decision Tree, SVM: support Vector Machine, HMM: Hidden
Markov Model, TBHMM: Threshold Based HMM, HMM-NN: HMM-Neural Networks, NN-HMM: Neural Networks-HMM, ANN: Artificial Neural Network,
CNN: Convolutional NN, RNN: Recurrent NN.

2. State-of-the-art methods

2.1 Machine Learning Techniques:

Table 1. Summary of Decision Trees, K-Nearest Neighbours, and Support Vector Machine techniques

Reference + Dataset Model/Variation Best Features Accuracy

Method
For test on sample set in % For 10 fold cross-
validation in %
[4] + Iterative Through Vector(activity, Mean 73.72 51.82
Dichotomiser accelerometer of position)
3 Decision smartphones
Tree (ID3 DT)
Position (decision Fourier 79.56 61.31
tree for transforms
classification of the coefficients.
position of
smartphone)

Action(classify Maximum 88.32 80.29

activity-position value
independent)
[3] + K-
online activity K-nearest - -
Nearest recognition neighbour(KNN)
Neighbour system working classification
(KNN) on Android algorithm
platforms Clustered K-nearest - 92
neighbour
[9] + Support Weizmann - - Discriminative Task Few Examples
Vector UIUC1
Machine L1AO L1AAO L1SO FE-2 FE-4 FE-8
(SVM) 100 100 100 66.67 70.24 100

99.04 98.04 98.84 45.56 80.65 97.44

2.1.4 Hidden Markov Model

In [1], the author shows the HMM-based approach that uses threshold and voting to automatically segment and recognize
complex activities.
700 Charmi
Charmi Jobanputra et al /Jobanputra et al. /Science
Procedia Computer Procedia Computer
00 (2018) Science 155 (2019) 698–703
000–000 3

Table 2. Summary Hidden Markov Model

Parameters [1] [2]
Dataset Bulling et al. Van Kasteren et al.
Chen et al. Tapia et al.
Anguita et al
Activities
Name total of total of Name House-A House-B House-C
true inferred
labels labels(TI) TT TI TT TI TT TI
(TT)
Breakfast 75.0 76.61 75.0 73.24 - -
Walking 89.11 99.10
W.upstairs 93.21 97.34 Brushing teeth 75.5 79.80 75.5 79.80 72 79.12
W.dowstrais 97.62 81.19
Dinner 78.0 79.75 78.0 75.00 - -
Sitting 96.13 82.37
Standing 90.23 98.56 Drinking 78.0 76.39 78.0 75.36 80 77.67
Lying down 86.59 95.88
Leaving house 85.0 93.61 85.0 94.65 85 96.59
Experiment Accuracy
No filter 80.24 Others 50.0 33.99 60.0 36.23 50 26.67
Filter 79.53
Continuous 81.21 Sleeping 80.5 93.38 80.5 84.55 80.5 75.09
Continuous 88.75
and discrete Showering 77.0 74.90 77.0 79.96 76 78.75
Snack 72.0 86.64 - - 65 74.71
Toileting 77.0 77.00 77.0 79.71 77 78.97
Dressing - - 80.0 89.38 75 90.36
Preparing BF - - 65.0 76.02 65 70.65
Preparing DN - - 66.0 76.74 65 70.65
Using dishwasher - - 76.0 83.51 - -
Eating - - - - 75 72.81

Recognition Continuous Hidden Markov Model (cHMM) Conditional Random Field (CRF)
Models Multiple Class Support Vector Machine Hidden Markov Model (HMM)
(MC-SVM) 2- Layer HMM
Naïve Bayes
Preprocessing A Median filter and a 3rd order lowpass They Clearly not represented any preprocessing required on binary
Butterworth filter with 20Hz frequency. temporal data. But they mentioned about three different feature
representations:
1. Raw. This feature uses the sensor data directly as it was
collected from the sensor network. The value is 1 when the
sensor fires and 0 otherwise.
2. Change Point (CP). This feature indicates when a sensor
changes value. The value is 1 when a sensor state goes from
zero to one or vice versa and 0 otherwise.
3. Last-Fired (LF). This feature indicates which sensor fired last.
The sensor that changed state last continues to value 1 and
changes to 0 when another sensor changes state.

Inputs:
To Record data: InvenSense Data saved in Binary temporal data
MotionFitTM Kit form of
Sensor: MPU-9150 Sensors Reed switch : To measure doors and
Place of the the left hand wrist cupboards are open or closed.
sensors : Pressure mats: To measure sitting on or lying
Daily activity 59 m flat
2
in bed
mostly executed Mercury To detect the movement of
in: contacts: objects
Recording 50Hz passive infrared to detect motion in a specific
sampling (PIR): area,
frequency: float sensors To measure the toilet being
Activities are 3-axis accelerometer, flushed
saved with: 3-axis gyrometer data
 Charmi Jobanputra et al. / Procedia Computer Science 155 (2019) 698–703 701
4 Charmi Jobanputra et al / Procedia Computer Science 00 (2018) 000–000

2.2 Neural Network Techniques

2.2.1 Artificial Neural Network

In [8], the author describes a data acquisition module prototype developed by them, which gathers the data of the patient and
recognizes abnormal status of the patient’s health so that early treatment would be available.
For arm posture recognition,
● Input device: Accelerometer embedded in smart watch.
● Preprocessing: Filtering, normalization, feature extraction.
For body posture recognition.
● Input device: On chest.
A new dataset with different sets of accelerometer data and data from heart rate sensor was used to identify various activities
in [8].

2.2.2 Convolutional Neural Network

In [6], the author describes human activity recognition through a very robust deep neural network technique that is
convolutional neural network which can model the features effectively.
● Input device: Inertial Measurement Unit sensors and triaxial sensors.
● Placement of sensors: Two sensors each on left and right shank, two sensors centred on feet and one on lumbar region
was placed.
● Sensor configuration: Single device, double device and triple device setups which used individual sensor data, in
combination of two sensor data and adding third sensor data in combination of two respectively.
● Shape of window: (6(no. of sensors) ╳ 204).
When observed, the combination of two or three sensors gave better results.

2.2.3 Recurrent Neural Network

Recurrent neural networks(RNN) recognizes the patterns which are separated by some intervals. Long Short Term Memory
(LSTM) is a RNN architecture which models temporal sequences and has the capacity of memorizing the things. In [7], the
author describes HAR using LSTM.
● Input device: Sensors embedded in houses at various locations.
● Data representation:
1) Raw sensor data where the data from the sensor is directly used
2) Last-fired sensor data which are the data received from the sensor that was fired last.

Configuration of LSTM was done as below:

702 Charmi
Charmi Jobanputra et al / Jobanputra et al. /Science
Procedia Computer Procedia Computer
00 (2018) Science 155 (2019) 698–703
000–000 5

Table 3.Summary of Above Methods

Reference + Activities classified Network topology Accuracy

Dataset used +
Implementatio Parameters Description
n method
[8] + Arm downwards Network 2 layer Feedforward- 100%
Self - made Arm upwards Backpropagation
dataset + ANN Arm horizontal
forward Activation Sigmoid
Arm horizontal function
backward Input layer 3 neurons
Arm horizontal Hidden layer 10 neurons
forward rotated
Output layer 6 neurons
upwards
Arm horizontal Training Levenberg-Marquardt
forward rotated algorithm
downwards Performance Mean squared error
evaluation function
[8] + Sitting Network 2 layer Feedforward- 99.96%
Self - made Prone Backpropagation
dataset + ANN Left lateral recumbent Activation Sigmoid
Right lateral function
recumbent Input layer 3 neurons
Supine
Hidden layer 10 neurons
Output layer 5 neurons

Training Levenberg-Marquardt
algorithm
Performance Mean squared error
evaluation function
[8] Standing Network 2 layer Feedforward- 99.08%
+ Different Supine Backpropagation
dataset + ANN Left lateral recumbent Activation Sigmoid
Right lateral function
recumbent Input layer 3 neurons
Prone
Walking (forward) Hidden layer 10 neurons
Walking (backward)
Running (forward) Output layer 10 neurons
Running(backward) Training Levenberg-Marquardt
algorithm
Performance Mean squared error
evaluation function

Performance 5-fold cross validation

evaluation

Three convolution 3 × 5, 2 × 4 and 2 × 2

layers with
kernels of size

Three max 3 × 3, 2 × 2 and 3 × 2

pooling layer with
kernel size
Three dense 500, 250 and 125 units
layers
[6] + Otago
Exercise dataset Activation ReLU function
+ CNN function
Loss calculation Cross entropy
function.
Stochastic Adam optimizer
optimization
method
Output layer m units, m: no. of
Stochastic activities in each
optimization group
method Adam optimizer
Batch size 1024
6 Charmi Jobanputra et al / Procedia Computer Science 00 (2018) 000–000
Charmi Jobanputra et al. / Procedia Computer Science 155 (2019) 698–703 703

3. Research gap and further challenges

● During data collection: If the data is to be collected through sensors, multiple sensors are to be worn by the person and
the placement of sensors is an issue as it affects the results.
● Feature extraction: Extraction of principal features from sensor data is challenging.
● Multiple persons: If sensors are embedded in home environment, there can be multiple residents there and so to map
the activities of multiple residents is a difficult task.
● Time complexity and accuracy: Different classification techniques give different time complexity and accuracy. It is
frequently observed that if computational complexity of any classification model is less, it has somewhat poor accuracy
as compared to the models where the accuracy is too good but computational complexity is less acceptable.
● Real-time data: Many results were calculated on the standard datasets which might vary when real-time dataset is used.
● Multiple activities: If the person performs more than one activity at the same time, recognition is difficult.
● Vision based activity recognition: With the live streaming of data and presence of crowd around, activity recognition
may be difficult.
● Location based activity recognition: Outdoor locations can be traced through Global Positioning System (GPS) but
indoor location is difficult to trace without embedding the sensors inside which creates multiple persons problem.
● Sensor constraints: We do not know if the sensor data is incorrect when faulty sensors are used.
● Overfitting and underfitting: Classification models like decision trees, neural networks can cause overfitting and
SVM can cause underfitting when less training data is available. So, the method of implementation must be in
accordance with the data.

4. Conclusion

In this survey, we carried out the comprehensive study of various tools and techniques which can be used in human activity
recognition which included different machine learning algorithms and neural network techniques. The techniques were
implemented on different datasets and they had varying observations depending upon the environmental conditions, type of data
used such as accelerometer data, other sensor data, placement of sensors, methods of implementation. These techniques are
compared on the basis of those contexts and also on the basis of computational complexities. Finally, challenges to human
activity recognition are also presented. From this survey, we deduce that there is no single method which is best for recognition
of any activity, hence in order to select a particular method for the desired application, one needs to take various factors into
consideration and determine the approach accordingly. So, in spite of having numerous methods, some of the challenges still
remain open and have to be resolved.

References

[1] Sarah Fallmann and Johannes Kropf. (2016) “Human activity recognition of continuous data using Hidden Markov Models and t he aspect of including
discrete data.” Intl IEEE Conferences 121-126.
[2] M. Humayun Kabir, M. Robiul Hoque, Keshav Thapa, and Sung-Hyun Yang. (2016) “Two-Layer Hidden Markov Model for Human Activity Recognition in
Home Environments.” International Journal of Distributed Sensor Networks, Volume: 12 issue: 1
[3] Pinky Paul and Thomas George (2015) “An Effective Approach for Human Activity Recognition on Smartphone” IEEE International Conference on Engineering
and Technology, pp 1-3.
[4] Lin Fan, and Zhongmin Wang. (2014)” Human activity recognition model based on decision tree.” International Conference on Advance Cloud and Big
Data, pp. 64-68.
[5] MS. Kanchan Gaikwad.( 2012) ” HMM Classifier for Human Activity Recognition .” International Journal (CSEIJ), Vol.2,No.4
[6] Antonio Bevilacqua, Kyle MacDonald, Aamina Rangarej, Venessa Widjaya, Brian Caulfield, and Tahar Kechadi.” Human Activity Recognition with
Convolutional Neural Networks”
[7] Deepika Singh, Erinc Merdivan, Ismini Psychoula, Johannes Kropf, Sten Hanke, Matthieu Geist, and Andreas Holzinger.(2018) ” Human Activity
Recognition using Recurrent Neural Networks.” ,267-274.
[8] Stefan Oniga, and József SütĘ. (2014)” Human activity recognition using neural networks.” 15th International Carpathian Control Conference (ICCC)
[9] K. G. Manosha Chathuramali and Ranga Rodrigo (2012) “Faster Human Activity Recognition with SVM” The International Conference on Advances in ICT for
Emerging Regions, 197-203.