Automatic Control of Driver fatigue and drowsiness using

landmark predictor

Abstract

Safe car driving has become a priority in everyday life. The main reason for designing a real
time system which monitors the state of the driver’s eyes is related to a decrease in car
crashes that will benefit millions of people around the world. This work proposes a method to
detect and to monitor the eyes of the driver, more precisely it analyses the eyes and check if
they are closed or open. The number of frames in which eyes are closed is determined. When
this number of frames is above a certain threshold, the driver will get a visual warning, on the
navigation display, that points out he is drowsy.

Introduction

One of the most frequent causes of road accidents is related to driver’s drowsiness. The
statistics show that drowsiness expose driver to higher crash risks, severe physical injuries, or
even death, while the economic losses are not negligible. A drowsy driver is in a state of
mental and physical flabbiness, which includes decreasing mental alertness and a sensation of
tiredness. Being in this state, he no longer performs competence to safety driving manoeuvre.
Many drivers due to the fatigue do not perceive their own level of drowsiness. Drowsy
driving is a real problem in our society because it affects and puts in danger all traffic
participants - drivers and pedestrians -, not only the drowsy driver. Car driving has become a
common activity for most people, therefore the concern to drive safe has gain maximum
priority. The development of a system, which monitors, in real time, the driver’s level of
drowsiness will decree the number of car accidents and will save millions of lives all over the
world. The use of such an assisting system, able to measure the level of vigilance is critical in
car crash prevention. In order to develop the system is important to know to evaluate the
level of drowsiness. Four types of measurements are commonly used to check the level of
drowsiness:

SOLUTIONS FOR FACE AND EYES DETECTION

A. Face Detection There are several approaches used in face detection. Some of these encode
the knowledge about characteristics of typical face and find structural elements - such as
eyebrows, eyes, nose, mouth and hairline- and use the relationships between them to detect
faces. In a method to identify the face from a cluttered background based on segmentation
was proposed. The Canny edge detector and heuristics are used as an edge map to remove the
group edges. The eclipse was fitted to the boundary between the head region and the
background and the face will be detected. The human skin color and texture faces have also
proven to be good features for face detection. For this method, the most important feature
was the skin color that can be separated from other parts of the background. This method
used maximal varieties variance threshold. Another method used for a face detection was the
histogram intersection in the HSV color space to highlight the skin region . An initial patch of
skin color was used to initiate the iterative algorithm. For the detection of skin color, this
method used a control histogram for control,

the threshold value that was assigned was compared with the result from histogram
comparison to analyse the skin region. The template matching methods store several patterns
of different faces to describe as a whole or the facial features separately, by computing the
correlations between an input image and the stored pattern in order to determine the degree of
similarity of the pattern to a face. The Viola-Jones method is used as a gold standard for faces
and mouth detection. There are several techniques which are used with this method. The first
stage implies the calculation of five Haar-like features and an integral image, which enables a
fast feature evaluation. The second step involves the development of an Adaboost classifier
by selecting a small number of features. The last technique refers to the implementation of an
attentional cascade in order to mitigate the limitations of the inherent Adaboosts’s single
committee of decision. OpenCV offers an implementation of the Viola-Jones method, and is
known as “Haar classifier” since it uses the Haar-like features that contain adding and
subtracting rectangular image areas for feature extraction before thresholding the result. For
the first step of the Viola-Jones technique, OpenCV uses diagonal features as well as the
other type of Haar-like features introduced by Viola-Jones to extract the face features .

B. Eye detection Different researches were done on focusing on changes and movement in
the eyes for driver’s fatigue detection. These researches analyse the changes in driver’s
direction of gaze, eye closure, and blinking rate. When people become drowsy, their blinking
pattern changes. To find the eyes and implement a method of fatigue detection Structural
Similarity Measure was used. In a chromatic algorithm was proposed for a better rate in
closed eyes detection. The algorithm can detect the eyes regions by using Eye Map, is
capable to locate the pupil centre by the centre of mass of the eye region and also achieves
iris boundary localization. In the spectral properties of a pupil under near IR illumination to
 highlight the eyes in the face images is used. In an absolute difference image is computed and
further converted into a bi-level image by an experimental threshold. Papers like use
horizontal projection histogram and

Block Diagram

SMPS 12V DC LM317 5V

230V AC
ADAPTOR SUPPLY DC SUPPLY

LCD

DC
LAPTOP LAPTOP, PYTHON L293D
RENESAS MOTOR
CAMERA FACE
MICROCONTROLLER
RECOGNITION

IMAGE VOICE OUTPUT

EMAIL (laptop speaker)

(laptop wifi)
SYSTEM ARCHITECTURE

Description

R5F100LEA microcontroller from Renesas RL78 series which is a 16-bit

microcontroller is used to implement this project. Microcontroller acts as the heart of this
project, which controls the whole system. It contains of Flash ROM 64KB, RAM 4KB and
Data Flash 4KB, and it has High speed on-chip oscillator, Self-reprogrammable under
software control, 58 GPIO’s, 3 UART’s, Simplified I2C, 10 bit resolution ADC, 28 Interrupt
Sources, ISP programming support etc.

The Renesas microcontroller is the heart of the project it is programmed such that it keeps on
commanding and controlling the complete action through peripherals connected. This smart
system is mainlyused to detect drowsiness in the vehicles. The detection system is able to
detect drowsiness individual by using a web camera to obtain real-time continuous images.
The camera is positioned directly towards the vehicle. whenever the driver feels sleepy then
image will be captured and send to the respective vehicle owners or cab owners so that he can
able to take the actions .in this project PWM motor is used to automatic control of the vehicle
the speed, whenever he feels sleepy automatically vehicle speed will be going to reduce,
meanwhile voice output will be coming from the laptop. The developed system detects the
condition of the eye: opening and closing conditions. The closing eye yielded greater distance
value than the threshold value. Meanwhile, the open eye condition yielded shorter distance
value than the threshold.

Hardware Requirements:

 Computer – Processor, high speed is preferred. 32/64 - bit

 RAM 4GB

 Hard disk – Free space of 5GB

 Laptop built-in Camera

 LCD display for Microcontroller

 Renesas Microcontroller

 DC motor

 L293D driver

 Switch mode power supply

Software’s Used

1. Python 2.7 or 3.7

2. Open CV tool
3. OS – Windows 7 / 8 / 10, 32/64 – bit.
4. Embedded C
5. Cube Suits+ Compiler
6. Renesas Flash Programmer V2.0

Literature survey

Internet of Things (IOT) is a very common term nowadays. It’s not a second internet; rather
it’s a network of devices that are connected to the Internet that is used every day to search
Google, upload images and connect with friends. It’s a network of products that are
connected to the Internet, thus they have their own IP address and can connect to each other
to automate simple tasks. However, IOT is still in its infancy. It has not been completely
developed and is fragmented. For the IOT to be fully realized all devices need to be able to
connect to each other, regardless of what company manufactured the product or which
companies have business relationships with each other. In technical terms, the Internet of
Things (IOT) is a system of interrelated computing devices, mechanical and digital machines,
objects, animals or people that are provided with unique identifiers and the ability to transfer
data over a network without requiring human-to-human or human-to-computer interaction.
IOT is the future of technology that can make our lives more efficient, starting from the most
mundane, everyday events to big, world changing ones.

NUMPY

NumPy is an open source library available in Python that aids in mathematical, scientific,
engineering, and data science programming. It works perfectly well for multi-dimensional
arrays and matrices multiplication. NumPy is a programming language that deals with multi-
dimensional arrays and matrices. On top of the arrays and matrices, NumPy supports a large
number of mathematical operations.

Why use NumPy?

NumPy is memory efficiency, meaning it can handle the vast amount of data more accessible
than any other library. Besides, NumPy is very convenient to work with, especially for matrix
multiplication and reshaping. On top of that, NumPy is fast. In fact, TensorFlow and Scikit
learn to use NumPy array to compute the matrix multiplication in the back end.

OPEN CV

Object Detection using Haar feature-based cascade classifiers is an effective object detection
method proposed by Paul Viola and Michael Jones in their paper, "Rapid Object Detection
using a Boosted Cascade of Simple Features" in 2001. It is a machine learning based
approach where a cascade function is trained from a lot of positive and negative images. It is
then used to detect objects in other images.
Proposed System:

The aim of this project is to avoid vehicle accident as we seen many accident occurs in our
daily life this device is helps to reduce the accident. whenever driver feels sleepy we can
detect by using camera so that we can take care of them and also we can save their life.

Methodology

Block
Diagram

Hardware
testing as per
project

Test Code
preparation for
Peripherals

Logic
Development
as per project

Final Testing of
the project as
per Conditions
  Initially whole block diagram and logical connections of the project is analysed.
Required hardware and software are collected.
 Each hardware is tested manually and conditions that need to be met during
interfacing are noted down.
 Every hardware is interfaced with controller and tested
 Overall logic of the project built and tested
 Project is tested for good number of times to meet its needed accuracy

Demo Points

 A prototype module will be developed for the project. It includes individual PCB
boards for all interfaces according to the block diagram. Every PCB will be inter-
connected with jumper wires.

 In the camera using python scripts to detect the diver conditonwhether the eyes are
open or closed.

 Here the eye detection will exist by using image processing and uses the SIFT
algorithms.

 All the sensors are connected microcontroller. we have used PWM concept to control
the speed of vehicle

 whenever he feels sleepy automatically vehicle speed will be going to reduce,

meanwhile voice output will be coming from the laptop.

 whenever the driver feels sleepy then image will be captured and send to the
respective vehicle owners or cab owners so that he can able to take the actions.

 Here LCD is used to display the all the ongoing information.

Applications

 this system can be used in industrial automotive.

Advantages

 It saves time.

 Manual operation has been reduced to major extent.

 Less man power required.

 Efficient distribution system.

 Easy to use.
 Efficient and reliable.
 My main motive is to eliminate the paper based records using smart phone

Disadvantages

 Carrying power supply.

Future Scope

There is always chance to improve any system as research & development is an endless
process. Our system is no exception to this phenomenon. The following developments can be
done for this project.

For project demo concern, we have developed a prototype module. In future, this project can
be taken to the product level. To make this project as user friendly and durable, we need to
make it compact and cost effective. Going further, most of the units can be embedded along
with the controller on a single board with change in technology, thereby reducing the size of
the system.
CONCLUSION

The project is designed using structured modeling and is able to provide the desired results. It
can be successfully implemented as a Real Time system with certain modifications.

Science is discovering or creating major breakthrough in various fields, and hence technology
keeps changing from time to time. Going further, most of the units can be fabricated on a
single along with microcontroller thus making the system compact thereby making the
existing system more effective. To make the system applicable for real time purposes
components with greater range needs to be implemented.

The suggestions to enhance the system are the camera should be implemented as an actuator
for intrusion scene capture, SMS service should be used for notification when the Internet
connection is not available, product level.

REFERENCES

[1] Åkerstedt, T. and Gillberg, M. (1990). Subjective and objective sleepiness in the active
individual. Int J Neurosci, 52(1-2), pp. 29-37.

[2] Ingre, M., Akerstedt, T., Peters, B., Anund, A., Kecklund, G., and Pickles, A. (2006).
Subjective sleepiness and accident risk avoiding the ecological fallacy. J Sleep Res, 15(2),
142-148.

[3] Gillberg, M., Kecklund, G., and Akerstedt, T. (1994). Relations between performance and
subjective ratings of sleepiness during a night awake. Sleep, 17(3), pp 236-241

[4] Fairclough S.H., Graham R. Impairment of driving performance caused by sleep

deprivation or alcohol: A comparative study. J. Hum. Factors Ergon. 1999;41, pp. 118–128.

[5] Ingre M., ÅKerstedt T., Peters B., Anund A., Kecklund G. Subjective sleepiness,
simulated driving performance and blink duration: Examining individual differences. J. Sleep
Res. 2006;15, pp 47–53 [6] Ruijia F., Guangyuan Z., Bo C. An on-Board System for
Detecting Driver Drowsiness Based on Multi-Sensor Data Fusion Using Dempster-Shafer
Theory. Proceedings of the International Conference on Networking, Sensing and Control;
Okayama, Japan. 26–29 March 2009; pp. 897–902 [7] Xiao F., Bao C.Y., Yan F.S. Yawning
detection based on gabor wavelets and LDA. J. Beijing Univ. Technol. 2009;35:409–413.
[8] Murphy-Chutorian E., Trivedi M.M. Head pose estimation and augmented reality
tracking: An integrated system and evaluation for monitoring driver awareness. IEEE Trans.
Intell. Transp. Syst. 2010;11:300–311.

[9] Xue T.Z., Nan N.Z., Fan M., Yong J.H. Head Pose Estimation Using Isophote Features
for Driver Assistance Systems. Proceedins of the IEEE Intelligent Vehicles Symposium;
Xi’an, China. 3–5 June 2009; pp. 568–572

. [10] S. Hu and R. Bowlds, "Pulse wave sensor for non-intrusive driver's drowsiness
detection," in Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual
International Conference of theIEEE, Minneapolis, MN, 2009