A DRIVER DROWSINESS DETECTION SYSTEM

R. Hari Haran, Dr. P.C. Thirumal , Dr. G.S Nandha kumar

Student, M. Tech Data Science, Kumaraguru College of Technology, Tamil Nadu
Associate professor, M. Tech Information Technology, Kumaraguru College of
Technology, Tamil Nadu
Associate professor, M. Tech Information Technology, Kumaraguru College of
Technology, Tamil Nadu

ABSTRACT:
The main causes of road accidents are the drowsiness and fatigue conditions of the driver.
When the driver is in a state of drowsiness, his facial expression changes such as eyes blinking
and yawning. To avoid such accidents, a detection system known as driver detection can be
done to avoid such accidents and increase road safety. So, the drowsiness detection can be
enchased by detecting the changes in the eyes and mouth regions, duration of eyes closure and
also by capturing video images.
In this paper, we propose a method for driver drowsiness detection using eye blinking rate i.e.,
Eye Aspect Ratio (EAR) and mouth yawning rate i.e., Mouth Aspect Ratio (MAR), and Pupil
Circularity Rate (PUC). The facial components which are the main aspects for drowsiness
detection are extracted and tracked using video sequence frames. The basic algorithm used to
detect the techniques are Long Short-Term Algorithm (LSTM), Recurrent Neural Network
(RNN) and Convolutional Neural Network (CNN). This system can be tested and implemented
in a real-time environment.

Keywords: Driver Detection, Drowsiness, Long Short-Term Algorithm (LSTM), Recurrent

Neural Network (RNN), Convolutional Neural Network (CNN)

I INTRODUCTION
Humans have invented machines and devised techniques to ease and protect their lives, for
mundane activities like traveling to work, or for more interesting purposes like aircraft travel.
With the advancement in technology, modes of transportation kept on advancing and our
dependency on it started increasing exponentially. One kind of carelessness is not confessing
that we are too lazy. Many researchers have written research papers on driver drowsiness
monitoring systems to track and avoid a destructive outcome from such negligence. Detecting
eye blinks and mouth yawns are done by using facial landmarks by extracting images using
webcam, video camera or mobile phone. The main goal of the proposed system drowsiness is
to detect both extreme and softer signal of drowsiness using the data sources collected.

II LITERATURE SURVEY:
In [1] The aim of this study is to investigate the feasibility of classifying drivers' warning status,
in particular slightly drowsy status, on the basis of hybrid sensing of vehicle-based, behavioural
and physiological indications, taking into account the incorporation of these identification into
the detection system. First, using a driving simulator and driver monitoring device, they have
accessed the level of drowsiness, driving efficiency, physiological signals (from
electroencephalogram (EEG) and electrocardiogram results), and behavioural indices of a
driver. Next, machine learning algorithms established driver alarm and drowsy states, and a
dataset was built over a span of 10 seconds from the extracted indices. Lastly, for classification,
ensemble algorithms were used. The results showed that by using hybrid methods to describe
the alarm and slightly drowsy states, the ensemble algorithm would achieve 82.4 percent
classification accuracy and 95.4 percent accuracy classifying the alarm and moderately drowsy.
In [2] The paper presents a report on the possibility of creating a somnolence detection system
based on three types of methods for car drivers: EEG and EOG (Electrooculography) signal
processing and driver image analysis. The research on the first two methods has been described
in previous works by the authors. This device is capable of evaluating the state of the driver
using an IR camera under real day and night conditions. Detection of the face and eyes is carried
out based on symmetry. For the selection of the eye states, Hough Transform for Circles is
used. Moreover, to create an autonomous system working in real time, they have implemented
this algorithm on a DSP (Digital Signal Processor).
In [3] To minimise the number of accidents due to driver fatigue and thus increase transport
safety, a module for Advanced Driver Assistance is presented; this device addresses automated
driver somnolence detection based on visual information and Artificial Intelligence. To
measure PERCLOS, a scientifically supported measure of somnolence associated with slow
eye closure, they have proposed an algorithm to locate, register, and evaluate the drivers face
and eyes. Analyzing the results of applying these networks on the acquisitioned images it can
be concluded that both networks had very good results, with 100% positive classification
results. The small number of neurons used in the hidden layers to successfully classify the
images (10 for the 1 hidden layer network and 15 for the autoencoder network) which allows
using a very limited part of their memory, by implementing the networks on compact
computing devices. For each driver, the training of the network can be performed precisely,
thereby improving the success rate of the classification.
In [4] This paper proposes a sleepy eye's recognition system for drowsiness detection. First, a
cascaded Adaboost classifier with the Haar-like features is utilized to find out the face region.
Second, the search algorithm for Active Shape Models (ASM) locates the eye area. The binary
pattern and edge detection are then adopted to extract the function of the eyes and determine
the state of the eye. Experimental findings show comparative success with other approaches,
even without the training level. This paper presents a sleepy eye's recognition for drowsiness
detection without the training stage. In the beginning, an Adaboost classifier with Haar-like
features is utilized to find out the face area. Then the eyes region is located by ASM. Finally,
the binary and edge detection pattern are adopted to recognize the eye’s state. Experimental
results prove that the proposed method could accurately detect the sleepy eyes.
In [5] In this paper, they have proposed a system called DriCare that detects the drivers' fatigue
status, such as yawning, blinking, and duration of eye closure, using video photos, without
equipping their bodies with devices. They have implemented a new face-tracking algorithm to
improve the accuracy of tracking due to the limitations of previous algorithms. In addition,
based on 68 key points, they have designed a new detection method for facial regions. Then by
using these facial regions the status of the drivers is determined. DriCare will alert the driver
using a fatigue warning by integrating the features of the eyes and mouth. The experimental
findings showed that DriCare achieved an accuracy of about 92%. From the above basis, it is
understood that when the driver is in a circumstance with dull or repetitive tasks that have
increase in frequency of yawning.
In [6] Drowsy driving is a prevalent and serious public health issue that deserves attention.
Recent studies estimate that around 20% of car crashes have been caused by drowsy drivers.
Trustworthy identification of drowsiness is now one of the key objectives in the production of
new advanced driver assistance systems. A somnolence identification technique based on
changes in the respiratory signal is proposed in this paper. In order to identify the driver's state
of alertness as drowsy or awake, the respiratory signal obtained using an inductive
plethysinography belt was processed in real time. These results have been obtained by
comparing the classifications of the quality signal assessment algorithm with the quality signal
rated by the external observers.
In [7] Driver alertness is an important cause of most automobile crash-related incidents. Driver
exhaustion due to sleep loss or sleep disturbances is significant factor in the rising number of
road accidents today. The aim of such a system is to detect the fatigue of drivers. By positioning
the camera inside the vehicle, driver's face and eye movements are tracked. This paper explains
how eyes movement can be predicted and monitored. In order to detect drowsiness, they have
built a device that localizes and monitors the driver's eyes and head movements. In order to
localize the eyes, the system uses a combination of template-based matching and feature-based
matching.
In [8] This paper is a summary report on the research conducted and the computer engineering
project to build a driver somnolence detection system to prevent accidents from occurring due
to driver exhaustion and sleepiness. In the paper, the findings and solutions for the restricted
implementation of the different techniques implemented in the project were suggested.
Whereas the project's execution gives the real world an understanding of how the system
operates and what improvements can be made to enhance the overall system's utility. It fully
meets the system 's goals and specifications. The system recognizes clients who are familiar
with the system and appreciate its focal points and the fact that it takes care of the issue to
warn people with fatigue-related problems about the extent of drowsiness when driving.
In [9] The main cause of accidents in the world as well as in Sri Lanka is driver drowsiness.
Due to lack of sleep and exhaustion, when driving, drowsiness may occur. The best way to
prevent accidents caused by the drowsiness of drivers is to detect the driver's drowsiness and
warn him before falling asleep. Facial feature recognition has been used to identify somnolence
in many methods such as eye retina detection. This paper proposes a way of detecting driver
drowsiness using eye retina detection and driver pulse rate detection. The hybrid approach that
is suggested for the detection of drowsiness is more effective than the detection strategies that
only take into account one type of measurements, such as physiological, behavioral or vehicle-
based approaches. The fuzzy-based model suggested in this study also ensures high precision
in the detection of somnolence and provides excellent results compared to the other models
that only take crisp levels to predict somnolence. This proposed scheme can also be applied in
real time by providing a real-time input mechanism for heart recovery ventilation (HRV)
analysis results.
In [10] Driver fatigue has been one of the world's biggest causes of car accidents in recent
years. Measuring the condition of the driver, i.e. drowsiness, is a direct way of measuring driver
fatigue. So, to save life and property, it is very important to identify the driver's drowsiness.
This project is aimed at creating a test device for detecting drowsiness. This scheme is a real-
time device that continuously captures images and measures the state of the eye according to
the algorithm defined and gives alarm. The value of the eye per closure is taken into account
for the identification of drowsiness. So, if the eye closure reaches a certain number, the driver
is considered to be sleepy. Several OpenCV libraries are used for the implementation of this
framework, including Haar-cascade. Implementation of Raspberry-Pi drowsiness detection
using Raspberry pi involves efficient video capture runtime. The recorded video was broken
into frames and analyzed for each frame and efficient face recognition, accompanied by eye
detection.

IV METHODOLOGY:

This approach mainly focusses on the extraction of driver’s face and prediction of the eye
blinking rate from the eye region and yawning rate from the mouth region. Thus, the proposed
system, estimates the landmarks that are used to determine the drowsiness of the driver. The
major aspects that are used to detect are: Eye Aspect Ratio (EAR), Mouth Aspect Ratio (MAR),
Pupil Circularity (PUC). We collect the input image from the camera and detect the face
frameworks from input image and predict the landmarks of important regions from both eyes
and mouth in the detected face. Once the landmarks are predicted, eyes and mouth aspect ratio
are used to determine to check if a person is drowsy.

Eyes Aspect Ratio (EAR):

When an individual is drowsy, their eyes are likely to get smaller and blink more. So, the
distance between the points (P2 & P6) and (P2 & P5) reduces, which in turn reduces the EAR.
The formula to calculate EAR,
EAR = |P2 – P6| + |P3 – P5|
2|P1 – P4|
Therefore, if the EAR starts to decline, it means that the eyes of the driver are likely to be
closed or they were blinking faster.

Mouth Aspect Ratio (MAR):

Similar to the EAR, the MAR can be used to calculate and detect the drowsiness of the driver.
When an individual becomes drowsy, they are likely to yawn and lose control over their mouth,
making MAR to reach higher. The formula to calculate MAR,
MAR = |EF|
|AB|

Mouth Over Eyes Aspect Ratio (MOE):

 MOE is the ratio of MAR and EAR.
MOE = MAR
EAR
MOE as a measure will be responsive to the changes that is subtle in EAR and MAR and will
exaggerate the changes in the denominator and numerator that will move in opposite directions.
This theory suggests that, when the individual falls drowsy MOE will increase.

Long Short-Term Memory (LSTM):

It is used for classifying, processing & predicting the time series data, for an unknown duration
between important events at any time series. It includes a memory that maintains information
for a longer duration.

Recurrent Neural Networks (RNN):

It uses the internal state memory to process sequence of inputs. These are applicable for
handwriting recognition or speech recognition. In this method, all inputs are independent of
each other.

Random Forest Method (RFM):

It is a supervised learning algorithm which is useful to determine MAR. When we are drowsy,
we try to yawn frequently, which gives an indication of drowsiness by normalising the value
of drowsiness and predicting them. This method constructs an ensemble of decision trees,
typically trained in the method of bagging. The technique of bagging is a mixture of learning
models that improves the overall outcome. This approach constructs and merges several
decision trees to obtain precise and stable forecasting.
Convolutional Neural Networks (CNN):
CNN is very effective in image recognition and classification. Hence, it is used to analyse
image data and output the mapped images into variables.
To analyse the image input and output,1-D CNN model is built and the spatial relationship
between each feature is determined for both states.
The layers between input and output are:
• Convolutional layer
• Flatten layer
• 2 fully-connected dense layers
• Dropout layers

LSTM Method:
It is a special kind of RNN, which is capable of learning long term dependencies on the data.
In this method, RNN memory space is used, because RNN has internal memory that allows
information to persist.

RNN Memory Space:

RNNs consider not only the current input but also the output that it has learned from the previous
inputs. In other neural networks, the inputs are independent of each other. In RNNs, the inputs
are related to each other.

ht = f(ht-1,xt)
xt – input vector
ht – current state
ht-1 – previous state
f -- forget gate
Core Idea:
The key to LSTMs is the cell state, the horizontal line running through the top of the diagram.
The condition of the cell is sort of like a conveyor belt. With just some small linear interactions,
it runs straight down the entire chain. It's really simple for data to only flow unchanged across it.
The LSTM has the capacity, carefully controlled by structures called gates, to remove or add
information to the cell state. Gates are an optional way of allowing data to move through. They
are composed of a layer of sigmoid neural net and a multiplication operation pointwise.
The sigmoid layer outputs numbers between zero and one, specifying how much should be
allowed to pass through each part. A value of zero means "let nothing through," while a value
of one means "let everything through!" To protect and control the cell state, an LSTM has three
of these gates, which makes it easy for the data to flow through the conveyor belt.
There are three gates for each period of time inside the LSTM network: Forget Gate, Input
Gate, and Output Gate.

Forget Gate: This gate attempts to "forget" part of the memory from the previous performance,
as its name implies. This gate determines what data should be thrown away or preserved. The
Sigmoid function transfers information from the previous hidden state and information from the
current input. Values between 0 and 1 are given. The closest you get to 0 means forgetting, and
the closest you get to 1 means holding.

Forget Gate Formula:

ft = σ(Wf[ht-1,xt] + bf)

xt – input vector
ht – hidden layer vector
ft -- forget gate
σ -- sigmoid function
bf – bias function of forget gate
Wf – weight of the forget gate

Input Gate: This gate decides what should be kept from the input in order to modify the
memory.

We have an input gate to change the state of the cell. First, into a sigmoid function, we transfer
the previous hidden state and current input. This specifies which values are modified by the
transformation of values between 0 and 1, 0 means not important, and 1 means important. To
help control the network, you also transfer the secret state and current input into the tanh feature
to squish values between -1 and 1. Then you multiply with the sigmoid output the output of
tanh. The sigmoid output will determine the knowledge that is necessary to retain from the
output of the tanh.

Input Gate Formula:

it = σ(Wi[ht-1,xt] + bi)
Ct = tanh(Wc[ht-1,xt] + bc)

xt – input vector
it – input function
bi – bias function of input gate
bc – bias function of cell state
Ct – cell state
ht – hidden layer vector
ft -- forget gate
σ -- sigmoid function
Wi – weight of the input gate
Wc – weight of the cell state

Cell State

The cell state is first multiplied pointwise by the forgotten vector. This has the potential of falling
values in the cell state if values near 0 are multiplied. We then take the output from the input
gate and add a pointwise addition that updates the cell state to new values that are considered
significant by the neural network. That gives us our new state as a cell.

Output Gate: By integrating the input and memory, the gate determines what the output is. The
gate of production determines what should be the next secret state. The next secret state requires
knowledge about previous inputs. Often used for forecasts is the secret state. First, into a sigmoid
function, we transfer the previous hidden state and the current input. Then to the tanh feature,
we transfer the newly changed cell state. In order to determine what data, the hidden state should
hold, we multiply the tanh output with the sigmoid output. The secret state is the production.
The new cell state and the new secret cell state are then moved to the next stage.

Output Gate Formula:

Ot = σ(Wo[ht-1,xt] + bo)
 Ht = ot * tanh(Ct)
xt – input vector
ht – hidden layer vector
ft -- forget gate
σ -- sigmoid function
ot – output gate
Wo – weight of the output gate
bo – bias function of the output gate
Ct – cell state

V RESULT
Finally, the level of driver drowsiness is determined based on the eye and yawning
detection results at the current time and their history in previous frames. It is important to list
some meanings of true positive, true negative, false positive, and false negative. First, true
positive means the number of times the driver's eye blinking is correctly identified by the
algorithm; second, true negative means success rate of driver alertness detection, false positive
can be defined as the number of times the algorithm counts the eye blink value when the driver
does not blink. Finally, false negative deals with the number of times the algorithm does not
count the eye blink value whenever the driver blinks.

True Positive (TP): Drowsiness can be detected

True Negative (TN): Non-drowsiness is neglected
False positive (FP): Non-drowsiness is shown detected
False negative (FN): Drowsiness is shown neglected
Sensitivity: (TP/(TP+FN)) X100
Specificity: (TN/(TN+FP)) X 100

VI CONCLUSION
Throughout this project, we learned quite a few things. Similar models can be as good at
completing tasks as more complicated models. In our case, the K-Nearest Neighbour model
gave an accuracy similar to the LSTM model. However, because we do not want to misclassify
people who are drowsy as alert, ultimately it is better to use the more complex model with a
lower false-negative rate than a simpler model that may be cheaper to deploy. Normalization
was crucial to our performance. We recognized that everybody has a different baseline for eye
and mouth aspect ratios and normalizing for each participant was necessary. Data pre-processing
and feature extraction / normalisation took up much of our time outside of the runtime for our
models. Updating our project and looking at how we can reduce the false-negative rate for KNN
and other simplified models would be interesting.

REFERENCES:
• [1] Akinari Hinario, jongseong, “An Investigation of Early Detection of Driver
Drowsiness Using Ensemble Machine Learning Based on Hybrid Sensingy”,13,
2018 IEEE International Conference, 22 April 2020.
• [2] T. Vesselenyi’, S. Moca’, “Driver drowsiness detection using ANN image
processing, Materials Science and Engineering”, 2017 IOP Conf.
• [3] Abdulla salem, Mohamed El Amine, “Driver Detection System”, University of
science and Technology.
• [4] Shinfeng D. Lin, Jia-Jen Lin, Chih-Yao Chung, “Sleepy Eye's Recognition for
Drowsiness Detection”, 2013 IEEE, 4, International symposium on biometrics and
security technologies
• [5] Wanghua deng1 , Ruoxue wu1, “Real-Time Driver-Drowsiness Detection System
Using Facial Features”, Preparation of Papers for ieee transactions and journals, vol
4, 2016.
• [6] Mireya Fernandez-Chimeno, Driver Drowsiness Detection Based on Respiratory
Signal Analysis, VOLUME 7, 2019, June 24, 2019
• [7] Singh Hirmani, Parmar mehu jajal, “Driver Drowsiness using Image Processing”,
ISSN .
• [8] Prathyush Agarwal, Rizul Sharma, “Driver Drowsiness Detection System”.
• [9] H.M Chandrasena, M.J Wickramasingh, “Drowsiness Detection and Alarming
System”, vol 4, sep 2016 IEEE.
• [10] Ooran Khunspith, Narit Honoom, “Drowsiness Detection using Eye Closeness
Detection”, 2016 IEEE.