1

Driver Drowsiness Recognition System Using Visual Behavior

and Machine Learning Algorithms to Prevent Accidents

Abstract
In current days almost lot of road accidents are occurred due to driver’s fault such as rash
driving, not following rules and regulations, not following safety measures and lack of sleep.
Drowsy driving is one of the major causes of road accidents and death. This drowsiness is
seen in most of the drivers while they drive long distances and hence proper monitoring is
required toidentify such drivers. Till now there was no proper method or mechanism which
can detect the driver’s fatigue and its becoming challenge topic and one of the active research
areas. In primitive days, we try to apply some methods based on vehicle or behavioral or
physiological based. But only few methods are intrusive and distract the driver, some require
expensive sensors and data handling. This motivate me to design this model in which, a low
cost, real time driver’s drowsiness detection system is developed with acceptable accuracy.
In this project by monitoring Visual Behavior of a driver with webcam and machine learning
SVM (support vector machine) algorithm we are detecting Drowsiness of a driver. This model
will use inbuilt webcam to read pictures of a driver and then using OPENCV and also SVM
algorithm extract facial features from the picture and then check whether driver in picture is
blinking his eyes for consecutive 20 frames or yawning mouth then application will alert
driver with Drowsiness messages. Here we try to use One Class SVM and Two Class SVM
Algorithm for pre-trained drowsiness model and then using Euclidean distance function we
are continuously checking or predicting EYES and MOUTH distance closer to drowsiness, if
distance is closer to drowsinessthen application will alert driver. We propose an algorithm
name to locate, track, and analyze both the drivers face and eyes to measure NPERCLOS
(Novel percentage of eye closure), a scientifically supported measure of drowsiness
associated with slow eye closure.

KEYWORDS: Drowsiness Detection, Visual Behavior, Eye Aspect Ratio, Mouth Opening
Ratio, Nose Length Ratio
 2

1. INTRODUCTION
Drowsy driving is one of the major causes of deaths occurring in road accidents. The truck
drivers who drive for continuous long hours (especially at night), bus drivers of long distance
route or overnight buses are more susceptible to this problem. Driver drowsiness is an
overcast nightmare to passengers in every country. Every year, a large number of injuries
and death occur due to fatigue related road accidents. Hence, detection of driver‟s fatigue
and its indication is an active area of research due to its immense practical applicability. The
basic drowsiness detection system has three blocks/modules; acquisition system, processing
system and warning system. Here, the video of the driver‟s frontal face is captured in
acquisition system and transferred to the processing block where it is processed online to
detect drowsiness. If drowsiness is detected, a warning or alarm is send to the driver from the
warning system.
Generally, the methods to detect drowsy drivers are classified in three types; vehicle
based, behavioural based and physiological based. In vehicle based method, a number of
metrics like steering wheel movement, accelerator or brake pattern, vehicle speed, lateral
acceleration, deviations from lane position etc. are monitored continuously. Detection of any
abnormal change in these values is considered as driv drowsiness. This is a nonintrusive
measurement as the sensors are not attached on the driver. In behavioural based method, the
visual behavior of the driver i.e., eye blinking, eye closing, yawn, head bending etc. are
analyzed to detect drowsiness. This is also nonintrusive measurement as simple camera is
used to detect these features. In physiological based method, the physiological signals like
Electrocardiogram (ECG), Electooculogram (EOG), Electroencephalogram (EEG),
heartbeat, pulse rate etc. are monitored and from these metrics, drowsiness or fatigue level is
detected. This is intrusive measurement as the sensors are attached on the driver which will
distract the driver. Depending on the sensors used in the system, system cost as well as size
will increase. However, inclusion of more parameters/features will increase the accuracy of
the system to a certain extent. These factors motivate us to develop a low-cost, real time
driver‟s drowsiness detection system with acceptable accuracy. Hence, we have proposed a
webcam based system to detect driver‟s fatigue from the face image only using image
processing and machine learning techniques to make the system low-cost as well as portable.
.
 3

Figure 1. Represent the Major Cases Recorded by State Wide Police from Drowsy
StateDriving

From the above figure 1, we can clearly identify that X-Axis is represented with year
and Y- Axis is represented with number of accidents or crashes which are recorded by state
wise police department from drivers drowsy state driving. Almost there was very huge
number of cases reported in the year 2019 since we collected the last 6 years of information
and it is gradually increasing day by day. Hence there is utmost importance for creating a
system or model which can identify the driver drowsiness state and immediate inform as an
alert system. If this type of drowsiness detection is performed by the web camera‟s which is
fitted in vehicles, we can able to reduce the number of crashes.
2. LITERATURE SURVEY
Literature survey is that the most vital step in software development process. Before
developing the new application or model, it's necessary to work out the time factor, economy
and company strength. Once all these factors are confirmed and got an approval then we can
start building the application. The literature survey is one which is mainly deal with all the
previous work whichis done by several users and what are the advantages and limitations in
those previous models. This literature survey is mainly used for identifying the list of
resources to construct this proposed application.
 4

MOTIVATION
1. W. B. Horng, C. Y. Chen, Y. Chang, C. H. Fan, “Driver Fatigue Detection based on Eye
Tracking and Dynamic Template Matching”, IEEE International Conference on Networking,,
Sensing and Control, Taipei, Taiwan, March 21-23, 2004.

A vision-based real-time driver fatigue detection system is proposed for driving safely. Here
the authors mainly described the current system initially try to load the driver's face and then
try to gather the color images captured in a car, by using the characteristic of skin colors.
Once color characteristics are found, edge detection is used to locate the regions of eyes.
Along with this wecan also track the eye tracking in the next frame, and those measured values
are used for fatigue detection in order to generate some warning alarms for driving safety.
The authors tested the model on a Pentium III 550 CPU with 128 MB RAM. The experiment
results seem quite encouraging and promising. The system can reach 20 frames per second
for eye tracking, and the average correct rate for eye location and tracking can achieve 99.1%
on four test videos. Thecorrect rate for fatigue detection is l00%, but the average precision
rate is 88.9% on the test videos.

2. B. Alshaqaqi, A. S. Baquhaizel, M. E. A. Ouis, M. Bouumehed, A. Ouamri, M. Keche,

“Driver Drowsiness Detection System”, IEEE International Workshop on Systems, Signal
Processing and their Applications, 2013.
Drowsiness and Fatigue of drivers are amongst the significant causes of road accidents. Every
year, they increase the amounts of deaths and fatalities injuries globally. In this proposed
work, the authors designed a new module for Advanced Driver Assistance System (ADAS)
is presented to reduce the number of accidents due to drivers fatigue and hence increase the
transportation safety. In this current work the authors mainly deal with automatic driver
drowsiness detection based on some visual parameters which are collected from the web
cameraand then try to apply AI on that images in order to track, locate and analyze the driver
face.

3. A. Abas, J. Mellor, and X. Chen, “Non-intrusive drowsiness detection by employing

Support Vector Machine,” 2014 20th International Conference on Automation and
Computing (ICAC), Bedfordshire, UK, 2014, pp. 188- 193.

Monitoring the driver's action while during driving by examining the maneuvered of the
vehicle can be a very prominent task in order to enhance driving safety. Differentiation
betweenunintentional and intentional car steering wheel movements could be a main key
element to detect drowsiness during driving. There is a growth of interests in applying
computerisedautomotive techniques to overcome those safety problems. This paper presents
a new method to detect the drowsiness of drivers non-intrusively, which may trigger
warning to drivers, so as to prevent accidents and to improve safety on the motorways. This
method employs Support Vector Machine (SVM) to train the classifier by using steering
wheel angle and distance to outside lane as input parameters to the SVM. All the parameters
extracted from vehicle parametrical data collected in a driving simulator. With all
considered features, a SVM drowsiness detection model has successfully been constructed.
 5

3. EXISTING SYSTEM AND ITS LIMITATIONS

Generally, the existing methods to detect drowsy drivers are classified in three types;
1. Vehicle Based,
2. Behavioural Based and
3. Physiological Based.
In vehicle-based method, a number of metrics like steering wheel movement, accelerator or
brake pattern, vehicle speed, lateral acceleration, deviations from lane position etc. are
monitored continuously. Detection of any abnormal change in these values is considered as
driver drowsiness. This is a nonintrusive measurement as the sensors are not attached on the
driver. In behavioural based method, the visual behaviour of the driver i.e., eye blinking,
eye closing, yawn, head bending etc. are analysed to detect drowsiness. This is also
nonintrusive measurement as simple camera is used to detect these features,but this type of
application is not present in the current system. In physiological based method, the
physiological signals like Electrocardiogram (ECG), Electrooculogram (EOG),
Electroencephalogram (EEG), heartbeat, pulse rate etc. are monitored and from these
metrics, drowsiness or fatigue level is detected. This is intrusive measurement as the sensors
are attached on the driver which will distract the driver. Depending on the sensors used in
the system, system cost as well as size will increase.

LIMITATION OF EXISTING SYSTEM

1. More Time Delay in finding the drowsiness state of driver
2. It is very costly to deploy sensors physically for the drivers.
3. There is no accurate technique or method which can find out the drowsiness based
on hisvisual pattern.
4. There is always monitored manually and if any abnormal change is present in
drivingstate,then drowsiness is estimated.
5. This may not generate always accurate results.

4. PROPOSED SYSTEM AND ITS ADVANTAGES

In this system at first, the video is recorded using a webcam. The camera will be positioned
in front of the driver to capture the front face image. From the video, the frames are extracted
to obtain 2-D images. Face is detected in the frames using histogram of oriented gradients
(HOG) and linear support vector machine (SVM) for object detection. After detecting the
face, facial landmarks like positions of eye, nose, and mouth are marked on the images.
From the facial landmarks, eye aspect ratio, mouth opening ratio and position of the head
are quantified and using these features and machine learning approach, a decision is obtained
about the drowsiness of the driver. If drowsiness is detected, an alarm will be sent to the
driver to alert him/her.

ADVANTAGES OF THE PROPOSED SYSTEM

1. By using proposed application it takes less time Delay in finding the drowsiness
state ofdriver
2. It is very less cost to deploy the application using software programming.
3. This will generate very accurate results to find out the drowsiness based on his
visualpattern.
4. There is always identified from the user web camera which is attached in the
laptop andif any abnormal change is present in driving state, then drowsiness is estimated.
5. This will generate always accurate results.
 6

Figure 2. Represent the Flow of Proposed Model for Driver Drowsiness Detection

5. PROPOSED ALGORITHMS
In this proposed work, we used two main algorithms such as: SVM and NPERCLOS
algorithm for driver drowsiness detection. Now let us discuss about the both in detail as
follows:

SUPPORT VECTOR MACHINE (SVM)

Support Vector Machine (SVM) is a supervised machine learning algorithm capable of
performing classification, regression and even outlier detection. The linear SVM classifier
worksby drawing a straight line between two classes. All the data points that fall on one side
of the linewill be labelled as one class and all the points that fall on the other side will be
labelled as the second. Sounds simple enough, but there‟s an infinite amount of lines to
choose from. How do we know which line will do the best job of classifying the data?
This is where the LSVM algorithm comes in to play. The LSVM algorithm will select a line
thatnot only separates the two classes but stays as far away from the closest samples as
possible. In fact, the “support vector” in “support vector machine” refers to two position
vectors drawn fromthe origin to the points which dictate the decision boundary.

NPERCLOS (Novel PERcent eyelid CLOSure)

It is a measure of driver alertness, which was identified as the most reliable and valid in a
study by the US Federal Highway Administration; various authors refer NPERCLOS as a
standard for drowsiness detection. The measure is the percentage of eyelid closure over the
pupil over time and reflects slow eyelid closures rather than blinks. The NPERCLOS
drowsiness metric was established in a 1994 driving simulator study as the proportion of
time (%) in a minute when theeyelids are at least 80 percent closed. In the literature three
metrics related to NPERCLOS are mentioned:
 7

P70: the proportion of time when the eyes were closed at least 70 percent.
P80: the proportion of time when the eyes were closed at least 80 percent (the P80 metrics
is usually referred as ”NPERCLOS”) •

EYEMEAS (EM): the mean square percentage of the eyelid closure rating. It has to be
noted that in the study by Wierwille et al. [28] and the related technical brief from the
Federal Highway Administration [29] that in order to detect the eyelid closures the face of
test persons was monitored and recorded, and then trained human scorers viewed the
recordings and ratedthe degree to which the drivers‟ eyes were closed from moment to
moment. The challenge related to the PERCLOS metrics is the automatic measurement of
the eyelid position, however, successful attempts to measure eyelid position (and derive
NPERCLOS from it) are reported by Grace et al. [30], where a CCD camera monitors the
face of the driver. The NPERCLOS metricsis measured directly and estimated with non-
parametric methods for detecting drowsiness in drivers in the paper by Grace et al. [30].

6. IMPLEMENTATION PHASE
Implementation is the stage where the theoretical design is converted into programmatically
manner. In this stage we will divide the application into a number of modules and then coded
fordeployment. The front end of the application takes tkinter as front end GUI for
implementingthe project and as a Back-End Data base we use HAAR Cascade Classifier
and SVM Classifier for identifying the patients drowsiness. The application is divided
mainly into following 5 modules. They are as follows:
1. Import Necessary Libraries
2. Load Input Module
3. Data Pre-Processing
4. Train the Model Using SVM Classifier and NPERCLOS
5. Find the Performance of Proposed Model
Now let us discuss about each and every module in detail as follows:

IMPORT NECESSARY LIBRARIES

In this module initially we need to import all the necessary libraries which are required for
building the model. Here we try to use all the libraries which are used to convert the data
into meaningful manner.Here the data is divided into numerical values which are easily
identified by the system. Also we try to import face libraries which are required to load the
facial part and identify the drowsiness state.

LOAD INPUT MODULE

In this module the we try to load the input from web camera. Once application is started
then theuser can able to load his face from web camera which is attached to the system.
Now we can able to use Face Classifiers to mainly concentrate on Eyes and Mouth regions.
Once eyes and mouth are identified then we need to find out the Euclidean distance for both
 8

eyes and mouth and then check the current status of driver.

DATA PRE-PROCESSING MODULE

Here in this section we try to pre-process the input and try to identify left eye and right
eyealong with mouth. Here we try to get the status of driver based on his eye and mouth
movements.

TRAIN THE MODEL USING SVM CLASSIFIER AND NPERCLOS

Here we try to train the current model on given data using SVM classifier and
NPERCLOS.Here we train the current model using SVM classifier and then identify the
drivers eyes movements whether it is closed on open. Also SVM classifier is used to recognize
mouth normal position or Wide opened. Here NPERCLOS is used to find the distance
between eyes and mouth and then find out that closeness among each other.

FIND THE PERFORMANCE OF PROPOSED MODEL

Here in this module we try to check the performance of current model by taking some
experiments on user who try to open his eyes and open his mouth too widely, close the mouth
and close the eyes and in various ways. All these ways are trained on the current SVM
classifier and finally we can able to find out the drowsiness of driver very easily.
 9

7. EXPERIMENTAL RESULTS

MAIN WINDOW
 10

In above screen click on „Start Behaviour Monitoring Using Webcam‟ button to connect
application with webcam, after clicking button will get below screen with webcam
streaming

USER CHECK HIS INPUT FROM LAPTOP WEB CAMERA

In above screen we can see web cam stream then application monitor all frames to see
personeyes are open or not, if closed then will get below message

USER TRY TO CLOSE HIS EYES AND TH

EN CHECK THE DROWSINESS

 11

SIMILARLY IF MOUTH STARTS YAWN THEN ALSO WILL GET ALERT

MESSAGE

From the above window we can clearly identify if there is mouth opened too large, then it
is detected as drowsiness and an alert is shown clearly and also Yawn Count is also detected

8. CONCLUSION
Drowsy driving is one of the major causes of road accidents and death. Hence, detection of
driver‟s fatigue and its indication is an active research area. Most of the conventional
methods are either vehicle based, or behavioural based or physiological based. Few methods
are intrusive and distract the driver, some require expensive sensors and data handling.
Therefore, in this study, a low cost, real time driver‟s drowsiness detection system is
developed with acceptable accuracy. In the developed system, a webcam records the video
and driver‟s face is detected in each frame employing image processing techniques. Facial
landmarks on the detected face are pointed and subsequently the eye aspect ratio, mouth
opening ratio and nose length ratio are computed and depending on their values, drowsiness
is detected based on developed adaptive thresholding. Machine learning algorithms have
been implemented as well in an offline manner. A sensitivity of 95.58% and specificity of
100% has been achieved in Support Vector Machine based classification.

9. REFERENCES

[1] W. L. Ou, M. H. Shih, C. W. Chang, X. H. Yu, C. P. Fan, "Intelligent Video-Based

Drowsy Driver Detection System under Various Illuminations and Embedded Software
Implementation", 2015 international Conf. on Consumer Electronics - Taiwan, 2015.
[2] W. B. Horng, C. Y. Chen, Y. Chang, C. H. Fan, “Driver Fatigue Detection based on Eye
Tracking and Dynamic Template Matching”, IEEE International Conference on
Networking,, Sensing and Control, Taipei, Taiwan, March 21-23, 2004.
[3] S. Singh, N. P. papanikolopoulos, “Monitoring Driver Fatigue using Facial Analysis
Techniques”, IEEE Conference on Intelligent Transportation System, pp 314-318.
[4] B. Alshaqaqi, A. S. Baquhaizel, M. E. A. Ouis, M. Bouumehed, A. Ouamri, M. Keche,
“Driver Drowsiness Detection System”, IEEE International Workshop on Systems,
Signal Processing and their Applications, 2013.
 12

[5] M. Karchani, A. Mazloumi, G. N. Saraji, A. Nahvi, K. S. Haghighi, B. M. Abadi, A. R.

Foroshani, A. Niknezhad, “The Steps of Proposed Drowsiness Detection System Design
based on Image Processing in Simulator Driving”, International Research Journal of
Appliedand Basic Sciences, vol. 9(6), pp 878-887, 2015.
[6] R. Ahmad, and J. N. Borole, “Drowsy Driver Identification Using Eye Blink Detection,”
IJISET - International Journal of Computer Science and Information Technologies, vol.
6, no. 1, pp. 270-274, Jan. 2015.
[7] A. Abas, J. Mellor, and X. Chen, “Non-intrusive drowsiness detection by employing
Support Vector Machine,” 2014 20th International Conference on Automation and
Computing (ICAC), Bedfordshire, UK, 2014, pp. 188- 193.
[8] A. Sengupta, A. Dasgupta, A. Chaudhuri, A. George, A. Routray, R. Guha; "A
Multimodal System for Assessing Alertness Levels Due to Cognitive Loading", IEEE
Trans. on Neural Systems and Rehabilitation Engg., vol. 25 (7), pp 1037-1046, 2017.
[9] K. T. Chui, K. F. Tsang, H. R. Chi, B. W. K. Ling, and C. K. Wu, “An accurate ECG
based transportation safety drowsiness detection scheme, IEEE Transactions on
Industrial Informatics, vol. 12, no. 4, pp. 1438- 1452, Aug. 2016.
[10] N. Dalal and B. Triggs, “Histograms of Oriented Gradients for Human Detection”,
IEEE conf. on CVPR, 2005.
[11] V. Kazemi and J. Sullivan; "One millisecond face alignment with an ensemble of
regression trees", IEEE Conf. on Computer Vision and Pattern Recognition, 23-28 June,
2014, Columbus, OH, USA.
[12] Richard O. Duda, Peter E. Hart, David G. Stork, “Pattern Classification”, Wiley
student edition.
[13] Dataset: https://sites.google.com/site/invedrifac/
[14] Ingre, M.; Akerstedt, T.; Peters, B.; Anund, A.; Kecklund, G. Subjective sleepiness,
simulated driving performance and blink duration: Examining individual differences. J.
Sleep Res. 2006, 15, 47–53.
[15] Zhang, H.; Wua, C.; Yan, X.; Qiu, T.Z. The effect of fatigue driving on car following
behavior. Transp. Res. Part F 2016, 43, 80–89.