ESHA

ESHA
EMOTIONS STUDY IN HUMANS AND ANALYSIS
(DETECTING THE EMOTIONS IN HUMANS BY USING TECHNOLOGY)
A project report submitted to

Rajiv Gandhi University of Knowledge Technologies

SRIKAKULAM
In partial fulfilment of the requirements for the
Award of the degree of
BACHELOR OF TECHNOLOGY
IN
COMPUTER SCIENCE AND ENGINEERING
Submitted by

B. SHIVANI – S170301
P. SANDHYA – S170331
N. CHINNARI SAI – S170460

3rd year B. Tech 2nd semester

Under the Esteemed Guidance of
Asst.Prof.Sri Sehsa Kumar Sir
 ESHA
Rajiv Gandhi University of Knowledge
Technologies - SKLM

CERTIFICATE

This is to certify that the thesis work titled “ESHA is one of the methods useful for
detecting emotions in humans ” was successfully completed by B SIVANI
(S170301), P SANDHYA (S170331) and N CHINNARI SAI (S170460) In partial
fulfilment of the requirements for the Mini Project in Computer Science and
Engineering of Rajiv Gandhi University of Knowledge Technologies under my
guidance and output of the work carried out is satisfactory.

Prof.Sri.Sesha kumar sir Prof.Sri.Dileep Kumar Sir,M.Tech

Project Guide Project Coordinator
 ESHA

DECLARATION

I declared that B.
thisSIVANI – S170301
thesis work titled “ESHA is the analysis done on different
emotion detection technologies for further extinction of existing projects” is
P.SANDHYA – S170331
carried out by me during the year 2020-21 in partial fulfilment of the
requirements forN.the
CHINNARI
Mini ProjectSAI – S170460
in Computer Science and Engineering.

I further declare that this dissertation has not been submitted elsewhere for
any Degree. The matter embodied in this dissertation report has not been
submitted elsewhere for any other degree. Furthermore, the technical details
furnished in various chapters of this thesis are purely relevant to the above
project and there is no deviation from the theoretical point of view for design,
development and implementation.
 ESHA

ACKNOWLEDGEMENT

I would like to articulate my profound gratitude and indebtedness to my

project guide Prof.Sri Sesha Kumar, Assistant Professor who has always
been a constant motivation and guiding factor throughout the project time. It
has been a great pleasure for me to get an opportunity to work under his
guidance and complete the thesis work successfully.

I wish to extend my sincere thanks to Prof.Smt.S.Laxmi Sri, Head of the

Computer Science and Engineering Department, for her constant
encouragement throughout the project.

I am also grateful to other members of the department without their support

my work would have been carried out so successfully.

I thank one and all who have rendered help to me directly or indirectly in the
completion of my thesis work

Project Associate

B. SIVANI – S170301

P.SANDHYA – S170331

N. CHINNARI SAI – S170460

 ESHA

ABSTRACT

Face detection has been around for ages. Taking a step forward, human
emotion displayed by face and felt by brain, captured in either video, electric
signal (EEG) or image form can be approximated. Human emotion detection
is the need of the hour so that modern artificial intelligent systems can
emulate and determine reactions from face. This can be helpful to make
informed decisions be it regarding identification of intent, promotion of
offers or security related threats. Recognizing emotions from images or video
is a trivial task for human eye, but proves to be very challenging for
machines and requires many image processing techniques for feature
extraction. Several machine learning algorithms are suitable for this job. Any
detection or recognition by machine learning requires training algorithm and
then testing them on a suitable dataset. This paper explores a couple of
machine learning algorithms as well as feature extraction techniques which
would help us in accurate identification of the human emotion.
 ESHA

INDEX
CH.NO CONTENTS PG.NO
1 INTRODUCTION 1
1.1 Introduction 1
1.2 Problem of the Statement 2
1.3 Objectives 2
1.4 Goals 2
1.5 Scope 3
1.6 Applications 3
1.7 Limitations 4

2 LITERATURE SURVEY 5
2.1 Collecting Information 5
2.2 Study 5
2.3 Benefits 5

2.4 Summary 6

3 SYSTEM ANALYSIS 7

3.1 Existing System 7

3.2 Disadvantages 7

3.3 Proposed System 7

3.4 Advantages 8
 ESHA

3.5 System Requirements 8

4 SYSTEM DESIGN 9

Design of the System 10

4.1.1 Class Diagram 10

4.1.2 Use Case Diagram 10

4.1.3 Sequence Diagram 11

4.1.4 Data Flow Diagram 11

5 SYSTEMIMPLEMENTATION 15

6 SYSTEM TESTING 47

Testing Introduction 47

Types Of Testing 47

Levels Of Testing 50

7 CONCLUSION 52

8 FUTURE ENHANCEMENT 53

APPENDIX

REFERENCES 54
 Chapter-1
INTRODUCTION

1.1 Introduction

Human emotion detection is implemented in many areas

requiring additional security or information about the person. It
can be seen as a second step to face detection where we may be
required to set up a second layer of security, where along with
the face, the emotion is also detected. This can be useful to
verify that the person standing in front of the camera is not just
a 2-dimensional representation.
Another important domain where we see the importance of
emotion detection is for business promotions. Most of the
businesses thrive on customer responses to all their products
and offers. If an artificial intelligent system can capture and
identify real time emotions based on user image or video, they
can make a decision on whether the customer liked or disliked
the product or offer.
We have seen that security is the main reason for identifying
any person. It can be based on finger-print matching, voice
recognition, passwords, retina detection etc. Identifying the
intent of the person can also be important to avert threats. This
can be helpful in vulnerable areas like airports, concerts and
major public gatherings which have seen many breaches in
recent years.
Human emotions can be classified as: fear, contempt, disgust,
anger, surprise, sad, happy, and neutral. These emotions are
very subtle. Facial muscle contortions are very minimal and
detecting these differences can be very challenging as even a
small difference results in different expressions. Also,
expressions of different or even the same people might vary for
the same emotion, as emotions are hugely context dependent.
While we can focus on only those areas of the face which
display a maximum of emotions like around the mouth and
eyes, how we extract these gestures and categorize them is still
an important question.
Neural networks and machine learning have been used for
these tasks and have obtained good results. Machine learning
algorithms have proven to be very useful in pattern recognition
and classification. The most important aspects for any machine
learning algorithm are the features.
In this paper we will see how the features are extracted and
modified for algorithms like Support Vector Machines. We
will compare algorithms and the feature extraction techniques
from different papers. The human emotion dataset can be a
very good example to study the robustness and nature of
classification algorithms and how they perform for different
types of dataset. Usually before extraction of features for
emotion detection, face detection algorithms are applied on the
image or the captured frame.
We can generalize the emotion detection steps as follows:
1) Dataset pre-processing
2) Face detection
3) Feature extraction
4) Classification based on the features
In this work, we focus on analysing the feature extraction
technique and emotion detection based on the extracted
features.
 1.2 Statement of the problem
1. There are different project that are developed for detecting emotions in
humans so far by using different feature extraction techniques and classification
algorithms.

2. The main objective of the project is to understanding of those image

processing techniques and machine learning algorithms which gives more
accurate results in predicting the emotion. This analysis is used:
To which find which combination of pre-processing algorithm and
Machine learning algorithm gives best results.
To train model using different datasets to improve it’s accuracy.

1.3 Objective

Understanding of existing projects on human emotion detection and running of

those on our own to find the best trained model

1.4 Goals

To understand different machine learning and pre-processing techniques used

in emotions d detection projects.
1- Finding the model that predicts emotion more accurately.

1.5 Scope
1) Different pre-processing techniques used are explained clearly.
2) Algorithms used in different emotion detection technologies are
explained clearly.
3) Analysis of some existing technologies or codes to determine best
model.
1.6 Applications

1. Video gaming. Using computer vision, the game console/video game detects
emotions via facial expressions during the game and adapts to it.

2. Medical diagnosis. Software can help doctors with the diagnosis of diseases
such as depression and dementia by using voice analysis.

3. Education. Learning software prototypes have been developed to adapt to kids’

emotions. When the child shows frustration because a task is too difficult or too
simple, the program adapts the task so it becomes less or more challenging.
Another learning system helps autistic children recognize other people's
emotions.

4. Employee safety. Based on Gartner client inquiries, demand for employee

safety solutions are on the rise. Emotion AI can help to analyze the stress and
anxiety levels of employees who have very demanding jobs such as first
responders.

5. Patient care. A ‘nurse bot’ not only reminds older patients on long-term
medical programs to take their medication, but also converses with them every
day to monitor the their overall wellbeing.

6. Car safety. Automotive vendors can use computer vision technology to monitor
the driver's emotional state. An extreme emotional state or drowsiness could
trigger an alert for the driver.
7. Autonomous car. In the future, the interior of autonomous cars will have many
sensors, including cameras and microphones, to monitor what is happening and
to understand how users view the driving experience.

8. Fraud detection. Insurance companies use voice analysis to detect whether a

customer is telling the truth when submitting a claim. According to independent
surveys, up to 30% of users have admitted to lying to their car insurance
company in order to gain coverage.
9. Recruiting. Software is used during job interviews to understand the credibility
of a candidate.

10.Call center intelligent routing. An angry customer can be detected from the
beginning and can be routed to a well-trained agent who can also monitor in
real-time how the conversation is going and adjust.
11.Connected home. A VPA-enabled speaker can recognize the mood of the
person interacting with it and respond accordingly.

12.Public service. Partnerships between emotion AI technology vendors and

surveillance camera providers have emerged. Cameras in public places in the
United Arabic Emirates can detect people's facial expressions and, hence,
understand the general mood of the population. This project was initiated by the
country's Ministry of Happiness.

13.Retail. Retailers have started looking into installing computer vision emotion
AI technology in stores to capture demographic information and visitors' mood
and reactions.

14.Reducing suicides: In Reducing the suicide in various

areas(office/school/college,

etc) by detecting the person with unusual behaviour.

1.7 Limitations

1. Accuracies of few algorithms were tested practically.

2. All the algorithms tested using few data sets only.
 Chapter 2
LITERATURE SURVEY

2.1 Collect Information

We have collected different thesis papers, books and journals that are related
to our project from
different sources.
We have taken the information from the other sources like IEEE Papers,
Thesis, different Books and some other journals Published in various
platforms for analysis.

2.2 Study

ESHA key features:

1. Information of various Machine learning models used for emotion
detection
2. Identifying different pre-processing techniques.
3. Listing out the different datasets used for training and testing of a
model

2.3 Benefits

By analysing various sources one can easily identify the best

models for his future work.
Summary
By analysing different sources we are trying to find out the best models that
gives us the accurate results.
 Chapter -3
ANALYSIS

3.1 Existing system

Existing Algorithms and Techniques:

Pre-processing techniques that are used in various emotion detection
technologies:
3.1 Ensemble of regression trees:
This method uses cascaded regression trees and finds the important positions on
the face using images. Pixel intensities are used to distinguish between different
parts of the face, identifying 68 facial landmarks. Based on a current estimate of
shape, parameter estimation is done by transforming the image in the normal
co-ordinate system instead of global. Extracted features are used to re-estimate
the shape parameter vectors and are recalculated until convergence.
The author uses only 19 features as shown in Figure 3 from the 68 extracted
features, focusing only around eyes, mouth and eyebrows.
3.1.1 Displacement ratios:
Once the features are in place, the displacement ratios of these 19 feature points
are calculated using pixel coordinates. Displacement ratios are nothing but the
difference in pixel position in the image from initial expression to final
expression. Twelve types of distances are calculated as shown in following
Table. Instead of using these distances directly, displacement ratios are used as
these pixel distances may vary depending on the distance between the camera
and the person.

Distances calculated to determine displacement ratios between different parts of face

3.1.2Eulerian Motion Magnification (EMM):
Subtle emotions are hard to detect. If we magnify the emotions, there is a
possibility of increasing the accuracy of detection. Motion properties such as
velocity and acceleration can be used for magnification. Image as a whole is
transformed by magnifying changes in properties of amplitude and phase. Based
on the properties, there are A-EMM (Amplitude based) and P-EMM (Phase
based) motion magnification [4].
3.1.2.1 Amplitude- Based Eulerian Motion Magnification:
Suppose I(x,t) is an image profile at location x at time t. If the image undergoes
a translational motion with a displacement function δ(t), the image is given by:
I(x,t) = = f(x + δ(t)) and I(x,0) = f(x)
Pixel intensity I of a magnified motion is computed as follows, where B(x,t) is
differences of intensity and given that I(x,t) = I(x) + B(x,t)
Î(x,t) = I(x) + α * B(x,t) , where α = magnification factor
3.1.2.3 LBP-TOP feature extraction from EMM:
Once the expressions are exaggerated using A-EMM or P-EMM and with
magnification factor α, Local Binary Pattern with Three Orthogonal Plane (LBP
– TOP) is used for feature extraction. For LBP-TOP feature extractions, the
following steps are followed:
1) Spatially resize image to 320 x 240 pixels resolution
2) Partition images into non-overlapping 5 x 5 blocks
3) Blocks are stacked up in 3-D volumes
From these blocks, spatial-temporal features and histograms of binary patterns
are extracted by LBP-TOP4,4,4,1,1,4. Here, 4,4,4 (first 3 numbers) represent 4-
neighbor connections in three orthogonal planes and 1,1,4 (last 3 numbers) are
radii of the respective connections [4]. The dataset used for this experiment was
CASME2 dataset which is a subtle emotion dataset.

3.1.3 Face detection using Viola-Jones face detector:

Images are pre-processed to reduce noise. Also, dataset must have images with
equal level of exposure, illumination and brightness. Image enhancement is
performed on such images using Histogram equalization techniques. The face
detection module is the familiar Viola- Jones technique which uses Ada-boost
algorithm. It combines a lot of weak classifiers to form a strong classifier by
iterations, where a weak classifier reduces the weighted error rate per iteration.
3.1.3.1 LBP technique for feature extraction:
 Local Binary Pattern (LBP) is a very simple and robust technique for feature
extraction and is independent of the illumination differences in the images. A
3x3 matrix is generated and each of the pixels in the matrix is assigned a binary
value depending on the center pixel value. These 8- bit binary values form an 8
bit binary number excluding the center pixel value which is finally converted to
decimal value.
LBP code for a pixel at (xc, yc) is given by
LBPP,R (xc, yc) = ∑(P=0,7) S(gp−gc) 2P , S(x) = {1, x >= 0 and 0, x < 0}
where,
gc = gray value of centre pixel
gp = gray value of neighbouring pixel of gc
P = 8 (maximum 8 neighbours for centre pixel in a 3x3 matrix)
Hence a pixel can have 256 different values as 2P =28 =256.
Figure 6 shows an input image block with centre pixel value as 7. Each value of
the surrounding 8 pixels is reassigned by comparing it to the intensity of the centre
pixel. Pixel values greater than or equal to the centre pixel are assigned 1;
otherwise 0. Figure 7 shows the binary values assigned to all the 8 pixels, which
combined in clockwise direction, gives a binary 8 bit number. Converting the 8-bit
number in the Figure 6 into decimal value (11000010) gives us the number 194.
We can encode all such pixel blocks in an image and use the decimal values
obtained as image feature vectors for classification. LBP is simple to implement as
the binary pattern remains the same in spite of changes in the illumination or
brightness. An increase in brightness or illumination condition will result in all the
pixel intensities increasing by the same value, keeping the relative difference the
same

3.1.4 Using dynamic grid – based HoG features

Histogram of oriented gradient (HoG) is an object detection technique based on
edge information. Each detection window can be defined by edge orientations.
Image is cropped according to the area of interest on the face. This cropped face is
the detection window which is divided into even smaller set of regions called cells.
In each cell, for each pixel, a magnitude of edge gradient is computed for each
orientation bin, thus forming the local histogram of oriented gradients
As the following Figure shows, from the face, required region is cropped and
divided into a matrix of 8 rows x 6 columns. Pixel size may vary in the grid for
each cell.

Cropped image divided into cells of 8 rows x 6 columns

A local histogram is generated for each cell. Further, for each block of 2 x 2 cells,
4 histograms are concatenated. Finally we have 12 normalized histograms
concatenated into one single global histogram which gives us 432 feature vectors.
3.1.5 Geometrical facial features extraction
A set of 19 features are selected empirically by observing the landmark positions
on the face and which are more related to human expressions. These 19 features
are taken as a subset from an existing marker-less system for landmark
identification and localization, which has actual 66 2D features [9] [10]. These 19
features or landmarks on the face are given in Table 3 and Figure 9. Landmark
positions in the image space are used to define two set of features: eccentricity
features and linear features.

Machine Learning Algorithms:

Once the dataset is created using the required features, the next important step is to
use a good classification algorithm. Support Vector Machines (SVM) are used
almost in all cases of multi-class classification of human expressions. They are
combined with one or another feature extraction technique.
Support Vector Machines (SVM):
Support Vector Machines are a set of supervised learning algorithms [18]. These
methods are properly related to problems of classification and regression. Given a
set of training examples (from samples), we can label the classes and train an SVM
to build a model that predicts the class of a new sample. A SVM is a model that
represents the sample points in space, separating the classes into spaces as wide as
possible. This is done by a hyper plane. When the new samples are put in
correspondence with the model, depending on the spaces to which they belong,
they can be classified to one or the other class. More clearly, this model represents
the sample points in the space, separating the classes by a hyper plane in a space of
very high dimensions that can be used on problems of classification or regression.
A clear separation between the classes will allow a correct classification, a scheme
is shown in Figure.

Figure: SVM feature selection

Decision Tree [DT]
Learning based on decision trees is one of the predictive modelling approaches
used in statistics, data mining and machine learning. Tree models, where the target
variable can take a finite set of values are called classification trees. In these tree
structures, the leaves represent class labels and the branches represent the
conjunctions of characteristics that lead to those class labels. Decision trees where
the target variable can take continuous values are called regression trees.
The purpose of this classifier is to create a model that predicts the
value of an objective by learning rules of simple decisions inferred from the
characteristics of the data.
Random Forest [RF]:
Random Forest Classifiers have proven to have an upper hand over SVM. .
Random forests are based on decision trees, but instead of just one classifier, use
more forests or classifiers to decide the class of the target variable .

Figure: Scheme of RF performance

For prediction, a new case is pushed down the tree. Then it is assigned the label
of the terminal node where it ends. This process is iterated by all the trees in the
assembly, and the label that gets the most incidents is reported as the prediction.
So, the estimator fits a series of decision tree classifier in several sub-samples of
the database and uses the average to improve predictive accuracy.
KNN:
For the classification used the KNN method is used to classify objects based
on training data with the distance nearest to the object, where the formula for
calculating the distance is Euclidean distance formula.
Face classification is a stage for the process of matching testing data
and training data from face datasets. KNN is one of the simple algorithms that can
be used for classification. The basic concept of KNN is to have several training
samples and testing samples determined by members. If k=1, the testing sample is
assigned to the nearest single neighbour class.

Convolutional Neural Network [CNN]:

In this kind of deep learning, special neural networks known as
convolutional are used. Like all neural networks they have input layers, hidden
layers, and output layers, but this kind of networks has two types of hidden layers:
convolution layers and pooling layers.
The Output has a 4x4 size because in the 6x6 input image we can put the
3x3 filter in 4x4 positions. The matrix begins to travel through all the neurons in
the upper left and it moves to the right. When you have finished a row, continue
with the row from below.
 Figure: Convolution example
Each output value is the result of the product and sum up of the original image
with the filter, in the first slide.
Ex: 3 + 1 + 2 − 1 − 7 − 5 = -7
Pooling layers:
This type of layers gathers the neurons in groups. This process
obtains the most relevant properties of the input data and obviates superfluous data
that can be misleading. There are several forms of pooling. An "average pooling"
that obtains the average value of the group of neurons or else "max pooling" that
chooses the maximum value of each group.

In this layer, when the neurons are grouped, the entrance is reduced. It is reduced
depending on the size of the groupings. For example, if the groups are a 2x2 size,
at the output we will obtain a quarter of the size of input data. It can be seen in
figure 3.4 that, even if this transformation technique, the spatial relationship is still
maintained.

Figure: Pooling example

Reducing over fitting: Dropout:

These models with a lot of parameters can easily over fit to the training data. To
help avoid this problem, there are some methods that help to reduce it. Dropout
uses a fixed probability to be retained and the choice of units to be dropped is
random.

Figure: Scheme of dropout performance5

Activation functions:
ReLU has been demonstrated to make training faster compared to other activation
functions.The ReLU activation function returns 0 if the output is less than 0 and it
returns the same value of the input if the input is more than 0.

We have thoroughly studied all those thesis and drawn the conclusion as shown
the fallowing the table.

Daraset Preprocessing Feature Classificationmodel Accuracy rate

Extraction
FER2013 -- Ensemble classifier 88.9%
Haar like+ADaboost SVM 89.9%
AAM ADAboost+Dynamic 94.05%
Bayesian Network
Vio;a-Jones Curvelt transform Online sequential Ml 95.17%
with redius bais
function rbf
CK+ Viola-Jones HOG 3 stage SVM 93.29
Viola-Jones HOG SVM 88.70
AAM -- DNN 93.20
interface Pretained CNN CNN 98.47
used as an
extractor
 HOG+PCA DAE(DSAE) 89.84
AAM 95.79
Open cv Haar -- Deep Belif Nets 20%
JAFFFF Viola-Jones Boosted LBP 3 Stage SVM 93.29

-- Patch Based+ SVM 92.30

-- Gobar SVM 94.30
Viola.Jones HOG SVM 91.8
-- LDA Boosted DBN 95.71
Patch-based

3.2 Disadvantages
• So far we have seen many algorithms and techniques related to
Emotion Detection, but we cannot determine which algorithms
with different datasets gives us accurate predictions.
• For today’s world everyone is trying to use best algorithms in their
projects so that we can take it as an opportunity to identify best
suitable algorithms for those projects.
• For suppose someone is developing an emotion detection project
for an application if it does not able to do the predictions
accurately then it is said to be failed.
• These are the disadvantages for us now-a-days.

3.3 Proposed System

• In Proposed work we can find different kinds of algorithms,
Techniques and datasets etc., at one place.
• In our work we are also going to attach codes of some existing
projects that we have executed
3.4 Advantages
Detecting emotions with technology was once a quite challenging task,
yet one where deep learning algorithms have shown great promise. By
using Facial Emotion Recognition, businesses can process images, and
 videos in real-time for monitoring video feeds or automating video
analytics, thus saving costs and making life better for their users.

3.5 System
Requirements
Software
Requirements:
Open CV
Dlib
Scikit-Learn
Python
Jupyter Notebook
Hardware Requirements:
• RAM: 8GB or above
• Hard disk: 500 GB or above
 CHAPTER- 4
SYSTEM DESIGN

DESIGN OF THE SYSTEM:

A UML diagram is a diagram based on the UML(Unified Modeling

Language) with the purpose of visually representing a system along with
its main actors, actions, artifacts or classes.in order to better understand,
alter, maintain, or document information about the system.
Class diagram:
Class diagram in the Unified Modelling Language (UML), is a kind of static
structure diagram hat describes the constitution of a process through showing the
system's classes, their attributes, and the relationships between the class. The
motive of a class diagram is to depict the classes within a model. In an object-
oriented software, classes have attributes (member variables), operations
(member capabilities) and relation.
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3BorthogonalLoop%3D1%3BjettySize%3Dauto%3Bhtml%3D1%3B%22%20edge%3D%221%22%20source%3D%223%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22370%22%20y%3D%2285%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%223%22%20value%3D%22inputting%20image%22%20style%3D%22swimlane%3BfontStyle%3D2%3Balign%3Dcenter%3BverticalAlign%3Dtop%3BchildLayout%3DstackLayout%3Bhorizontal%3D1%3BstartSize%3D26%3BhorizontalStack%3D0%3BresizeParent%3D1%3BresizeLast%3D0%3Bcollapsible%3D1%3BmarginBottom
%3D0%3Brounded%3D0%3Bshadow%3D0%3BstrokeWidth%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2290%22%20y%3D%2250%22%20width%3D%22160%22%20height%3D%2270%22%20as%3D%22geometry%22%3E%3CmxRectangle%20x%3D%22230%22%20y%3D%22140%22%20width%3D%22160%22%20height%3D%2226%22%20as%3D%22alternateBounds%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%224%22%20value%3D%22taking%20photo%2Fimage%22%20style%3D%22text%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingLeft%3D4%3BspacingRight%3D4%3Boverflow%3Dhidden%3Brotatable%3D0%3Bpoints%3D%5B%5B0%2C0.5%5D%2C%5B1%2C0.5%5D%5D%3BportConstraint%3Deastwest%3B%22%20vertex%3D%221%22%20parent%3D%223%22%3E%3CmxGeometry%20y%3D%2226%22%20width%3D%22160%22%20height%3D%2226%22%20as%3D%22geometry%22%2F%3E%3C
%2FmxCell%3E%3CmxCell%20id%3D%225%22%20value%3D%22%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3BorthogonalLoop%3D1%3BjettySize%3Dauto%3Bhtml%3D1%3B%22%20edge%3D%221%22%20source%3D%226%22%20target%3D%2219%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22180%22%20y%3D%22370%22%2F%3E%3CmxPoint%20x%3D%22182%22%20y%3D%22370%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%226%22%20value%3D%22training%22%20style%3D%22swimlane%3BfontStyle%3D0%3Balign%3Dcenter%3BverticalAlign%3Dtop%3BchildLayout%3DstackLayout%3Bhorizontal%3D1%3BstartSize%3D26%3BhorizontalStack%3D0%3BresizeParent%3D1%3BresizeLast%3D0%3Bcollapsible%3D1%3BmarginBottom%3D0%3Brounded
%3D0%3Bshadow%3D0%3BstrokeWidth%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22110%22%20y%3D%22270%22%20width%3D%22160%22%20height%3D%2260%22%20as%3D%22geometry%22%3E%3CmxRectangle%20x%3D%22130%22%20y%3D%22380%22%20width%3D%22160%22%20height%3D%2226%22%20as%3D%22alternateBounds%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%227%22%20value%3D%22output%22%20style%3D%22swimlane%3BfontStyle%3D0%3Balign%3Dcenter%3BverticalAlign%3Dtop%3BchildLayout%3DstackLayout%3Bhorizontal%3D1%3BstartSize%3D26%3BhorizontalStack%3D0%3BresizeParent%3D1%3BresizeLast%3D0%3Bcollapsible%3D1%3BmarginBottom%3D0%3Brounded%3D0%3Bshadow%3D0%3BstrokeWidth%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22660%22%20y%3D%22310%22%20width%3D
%22160%22%20height%3D%2270%22%20as%3D%22geometry%22%3E%3CmxRectangle%20x%3D%22340%22%20y%3D%22380%22%20width%3D%22170%22%20height%3D%2226%22%20as%3D%22alternateBounds%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%228%22%20value%3D%22expression%22%20style%3D%22text%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingLeft%3D4%3BspacingRight%3D4%3Boverflow%3Dhidden%3Brotatable%3D0%3Bpoints%3D%5B%5B0%2C0.5%5D%2C%5B1%2C0.5%5D%5D%3BportConstraint%3Deastwest%3B%22%20vertex%3D%221%22%20parent%3D%227%22%3E%3CmxGeometry%20y%3D%2226%22%20width%3D%22160%22%20height%3D%2226%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%229%22%20value%3D%22%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3BorthogonalLoop%3D1%3BjettySize%3Dauto%3Bhtml
%3D1%3BentryX%3D0.75%3BentryY%3D0%3BentryDx%3D0%3BentryDy%3D0%3B%22%20edge%3D%221%22%20source%3D%2210%22%20target%3D%227%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22660%22%20y%3D%2289%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2210%22%20value%3D%22testing%22%20style%3D%22swimlane%3BfontStyle%3D0%3Balign%3Dcenter%3BverticalAlign%3Dtop%3BchildLayout%3DstackLayout%3Bhorizontal%3D1%3BstartSize%3D26%3BhorizontalStack%3D0%3BresizeParent%3D1%3BresizeLast%3D0%3Bcollapsible%3D1%3BmarginBottom%3D0%3Brounded%3D0%3Bshadow%3D0%3BstrokeWidth%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22380%22%20y%3D%2250%22%20width%3D%22160%22%20height%3D%2278%22%20as%3D
%22geometry%22%3E%3CmxRectangle%20x%3D%22550%22%20y%3D%22140%22%20width%3D%22160%22%20height%3D%2226%22%20as%3D%22alternateBounds%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2211%22%20value%3D%22preproceesing%26%2310%3Benhancing%20image%22%20style%3D%22text%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingLeft%3D4%3BspacingRight%3D4%3Boverflow%3Dhidden%3Brotatable%3D0%3Bpoints%3D%5B%5B0%2C0.5%5D%2C%5B1%2C0.5%5D%5D%3BportConstraint%3Deastwest%3B%22%20vertex%3D%221%22%20parent%3D%2210%22%3E%3CmxGeometry%20y%3D%2226%22%20width%3D%22160%22%20height%3D%2244%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2212%22%20value%3D%22%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3BorthogonalLoop%3D1%3BjettySize%3Dauto%3Bhtml%3D1%3B
%22%20edge%3D%221%22%20source%3D%2213%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22480%22%20y%3D%22130%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2213%22%20value%3D%22feature%20extraction%22%20style%3D%22swimlane%3BfontStyle%3D1%3Balign%3Dcenter%3BverticalAlign%3Dtop%3BchildLayout%3DstackLayout%3Bhorizontal%3D1%3BstartSize%3D26%3BhorizontalStack%3D0%3BresizeParent%3D1%3BresizeParentMax%3D0%3BresizeLast%3D0%3Bcollapsible%3D1%3BmarginBottom%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22400%22%20y%3D%22320%22%20width%3D%22160%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2214%22%20value%3D%22%22%20style%3D
%22endArrow%3Dclassic%3Bhtml%3D1%3Brounded%3D0%3BentryX%3D1%3BentryY%3D0.923%3BentryDx%3D0%3BentryDy%3D0%3BentryPerimeter%3D0%3B%22%20edge%3D%221%22%20parent%3D%2213%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22-10%22%20y%3D%2250%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22-130%22%20y%3D%22-0.0019999999999527063%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2215%22%20value%3D%22hog(histogram%20oriented%20%26%2310%3Bgradients)%22%20style%3D%22text%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingLeft%3D4%3BspacingRight%3D4%3Boverflow%3Dhidden%3Brotatable%3D0%3Bpoints%3D%5B%5B0%2C0.5%5D%2C%5B1%2C0.5%5D%5D
%3BportConstraint%3Deastwest%3B%22%20vertex%3D%221%22%20parent%3D%2213%22%3E%3CmxGeometry%20y%3D%2226%22%20width%3D%22160%22%20height%3D%2244%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2216%22%20value%3D%22%22%20style%3D%22edgeStyle%3DorthogonalEdgeStyle%3Brounded%3D0%3BorthogonalLoop%3D1%3BjettySize%3Dauto%3Bhtml%3D1%3BentryX%3D1%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3B%22%20edge%3D%221%22%20source%3D%2217%22%20target%3D%2211%22%20parent%3D%221%22%3E%3CmxGeometry%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22630%22%20y%3D%22100%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2217%22%20value%3D%22classification%22%20style%3D%22swimlane%3BfontStyle%3D1%3Balign%3Dcenter%3BverticalAlign%3Dtop
%3BchildLayout%3DstackLayout%3Bhorizontal%3D1%3BstartSize%3D26%3BhorizontalStack%3D0%3BresizeParent%3D1%3BresizeParentMax%3D0%3BresizeLast%3D0%3Bcollapsible%3D1%3BmarginBottom%3D0%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22550%22%20y%3D%22180%22%20width%3D%22160%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2218%22%20value%3D%22KNN(k%20nearest%20neighbour%20%26%2310%3B)algorithm%22%20style%3D%22text%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dleft%3BverticalAlign%3Dtop%3BspacingLeft%3D4%3BspacingRight%3D4%3Boverflow%3Dhidden%3Brotatable%3D0%3Bpoints%3D%5B%5B0%2C0.5%5D%2C%5B1%2C0.5%5D%5D%3BportConstraint%3Deastwest%3B%22%20vertex%3D%221%22%20parent%3D%2217%22%3E%3CmxGeometry%20y%3D%2226%22%20width%3D
%22160%22%20height%3D%2234%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2219%22%20value%3D%22store%20the%20classfying%20images%22%20style%3D%22shape%3Dcylinder3%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BboundedLbl%3D1%3BbackgroundOutline%3D1%3Bsize%3D15%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22140%22%20y%3D%22410%22%20width%3D%2284%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2220%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3Brounded%3D0%3BentryX%3D1.031%3BentryY%3D0.115%3BentryDx%3D0%3BentryDy%3D0%3BentryPerimeter%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D
%22540%22%20y%3D%22220%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22274.96000000000004%22%20y%3D%22298.99%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2221%22%20value%3D%22train%20the%20data%22%20style%3D%22text%3Bhtml%3D1%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3Bresizable%3D0%3Bpoints%3D%5B%5D%3Bautosize%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22125%22%20y%3D%22295%22%20width%3D%2290%22%20height%3D%2230%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E

Use Case Diagram:

It is a visually representation what happens when actor interacts with system. A

use case diagram captures the functional aspects of a system. The system is shown
as a rectangle with name of the system inside ,the actor are shown as stick figures,
the use case are shown as solid bordered ovals labeled with name of the use case
and relationships are lines or arrows between actor and use cases.
SEQUENCE DIAGRAM:

A sequence diagram in Unified Modelling Language (UML) is one variety of

interaction diagram that suggests how methods operate with one other and in
what order. It is a construct of a Message Sequence Chart. Sequence diagrams are
quite often referred to as event-hint diagrams, event situations, and timing
diagrams. A sequence diagram suggests, as parallel vertical traces (lifelines),
special systems or objects that are residing at the same time, and, as horizontal
arrows, the messages exchanged between them, within the order the place they
occur.
4.1.4 DFD Diagram:

A data flow diagram or bubble chart (DFD) is a graphical representation

of the "flow" of data .through an information system, modeling its
process aspects. Often they are a preliminary step used to create an
overview of the system which can later be elaborated. DFDs can also be
used for the visualization of data processing (structured design).
A DFD shows what kinds of information will be input to and output from
the system, where the data will come from and go to, and where the data
will be stored. It does not show information about the timing of
processes, or information about whether processes will operate in
sequence or in parallel (which is shown on a flowchart).
The primitive symbols used for constructing DFD’s are:

Symbols used in DFD

A circle represents a process.

A rectangle represents external entity

A square defines a source or destination of the system data.

An arrow identifies dataflow.

Double line with one end closed indicates data store

 CHAPTER -5

SYSTEM IMPLEMENTATION
A static approach using extracted features and emotion recognition using
machine learning is used in this work. The focus is on extracting features using
python and image processing libraries and using machine learning algorithms
for prediction. Our implementation is divided into three parts. The first part is
image pre-processing and face detection. For face detection, inbuilt methods
available in dlib, Open CV library are used. Once the face is detected, the
region of interest and important facial features are extracted from it. There are
various features which can be used for emotion detection. In this work, the
focus is on facial points around the eyes, mouth, eyebrows etc.

We have a multi-class classification problem and not multi-label. There is a

subtle difference as a set of features can belong to many labels but only one
unique class. The extracted facial features along with CNN are used to detect
the multi-class emotions. The papers we have studied different machine
learning, but focused mostly on CNN as it is one of the widely used and
accepted algorithms for emotion classification. Our database has a total of 7
classes to classify. We have compared our results with logistic regression and
random forest to compare the results of different algorithms.

Procedure of emotion recognition

Data Sets we have used are:

1) deepface
2) FER-2013
3) emojify
 Results obtained:
We have run the models used CNN and KNN algorithm.

Using CNN and FER-2013 dataset

When we run the code using CNN algorithm with FER-2013 data set we have
obtained the following results:
The overall accuracy we obtained roughly is – 90%
[2]: EMOTION DETECTION USING CNN WITH EMOTIC DATASET:

When we run the code using CNN algorithm with Emotic data set we have
obtained the following results:
The overall accuracy we obtained roughly is – 75%
EMOTION DETECTION USING DEEP LEARNING WITH DEEPFACE
LIBRARY
When we run the code using CNN algorithm with pre trained dataset which is
available in deepface we have obtained the following results:
This model is giving best prediction result for happy, surprise, neutral emotions.

EMOTION DETECTION USING CNN WITH TRAINING DATASET

When we run the code using CNN algorithm with FER-2013 data set we have
obtained the following results:
This model is giving best prediction result for happy, surprise.
 CHAPTER 7
SYSTEM TESTING

INTRODUCTION

The cause of testing is to detect mistakes. Making an attempt

out is the technique of looking for to realize each viable fault or
weakness in a piece product. It presents a method to determine
the performance of add-ons, sub-assemblies, assemblies and/or
a completed product. It is the method of ex excising g program
with the intent of constructing certain that the application
procedure meets its necessities and client expectations and does
no longer fail in an unacceptable process. There are rather
plenty of forms of scan. Each experiment sort addresses a
special trying out requirement.
TYPES OF TESTS:

Unit testing:

Unit checking out involves the design of scan circumstances

that validate that the internal application good judgment is
functioning safely, and that program inputs produce legitimate
outputs. All decision branches and interior code float must be
validated. It's the checking out of character application items of
the application. It is achieved after the completion of a person
unit earlier than integration. It is a structural checking out, that
relies on competencies of its construction and is invasive. Unit
exams participate in common exams at component level and
scan a distinct business approach, utility, and/or process
configuration. Unit assessments be certain that every specified
course of an industry method performs appropriately to the
documented requisites and involves clearly outlined inputs and
anticipated results.

Integration testing:

Integration Testing are designed to scan built-in program

accessories to determine within the occasion that they evidently
run as one software. Trying out is occasion driven and is
more concerned with the fundamental final result of screens or
fields. Integration assessments reveal that despite the fact that
the accessories had been for my part pleasure, as proven
through effectively unit checking out, the combo of accessories
is correct and regular. Integration checking out is chiefly aimed
at exposing the issues that come up from the performance of
different components.

Functional testing:

Functional Testing checks provide systematic demonstrations

that capabilities established are to be had as particular by means
of the business and technical specifications, method
documentation, and consumer manuals. Functional testing is
working on below mentioned data:

Legitimate input : identified lessons of legitimate input ought to be

accredited.
Invalid enter: recognized lessons of unacceptable effort must be
rejected.
Capabilities: recognized features ought to be exercised.
Output : recognized courses of software outputs have got to be
exercised.
Systems/Procedures : performance of the system here was invoked

Individual and team work of useful checks is fascinated by

specifications, key capabilities, or special scan instances.
Moreover, systematic insurance plan concerning establish
business method flows; data fields, predefined processes, and
successive strategies have to be regarded for trying out. Before
 useful trying out is whole, extra checks are recognized and the
strong price of present checks be strong minded.

System testing:
Scheme difficult ensure so as to the whole included agenda process meets
principles. It exams a pattern to make sure identified and predictable
outcome. An illustration of procedure testing is the configuration oriented
approach integration scan. System testing is based on approach
descriptions and flows, emphasizing pre-driven system links and
integration aspects.

White Box Testing:

This testing is a trying out wherein where the application tester has
competencies of the interior workings, constitution and software
language, or at least its cause.
It's rationale. It's used to test areas that can't be reached from a black
box stage.

Black Box Testing:

This is testing the software with none advantage of the inside

workings, establishment or words of the unit life form veteran.
Black field checks, as most other sorts of

LEVELS OF TESTING

Unit testing strategy

 Unit checking out is most commonly performed as a part of a
mixed code and unit experiment part of the software lifecycle,
though it be not exceptional for coding and unit checking out
to be performed as two targeted phases.
Test strategy and approach:
Field testing out can be carried out manually and sensible
assessments shall be written in element.
Test objectives
each field must be work correctly.
 each page must be activated through the specified link.
 Features to be tested Verify that the entries are of the correct
format No duplicate entries should be allowed

Integration testing strategy

Software integration testing is the incremental integration

checking out of two otherwise further included software gears
on top of a solo stage to fabricate failure induced with the aid of
interface defects. The project of the mixing scan is to check that
components or program applications,
e.g. Components in a program approach or œ one step up œ software
purposes at the
company degree œ interact without error.
Test Results:
All of the scan circumstances recounted above passed efficiently. No
defects encountered.

Acceptance Testing
 User Acceptance testing trying out is a crucial section of any
mission and requires enormous participation by the tip user. It
additionally ensures that the procedure meets the functional
specifications.
Test Results:
The entire test cases recounted above passed effectually. No defects
Encountered

Conclusion

• We have run different three codes and we get different accuracy

• when we run the code using cnn algorithm with emotic dataset we have
obtained the overall accuracy we obtain roughly as 75%
• When we run the code using CNN algorithm with FER-2013 dataset we
have obtained the overall accuracy we obtain roughly as 90%
• When we run the code using CNN algorithm with pre trained dataset which
is available in deepface we got best predictions results for happy, surprise,
neutral emotions.
• When we run the code using CNN algorithm with FER-2013 dataset we got
best predictions results for happy, surprise, neutral emotions.
• Hence overall we conclude that CNN algorithm with FER-2013 Dataset
gives best accurate results

Future Enhancement
Based on the research we have done we concluded CNN with FER-Data Set
gives best prediction results.

This can further help to check the algorithms even with more datasets and
algorithms.

REFERENCES
 https://drive.google.com/drive/u/0/folders/
1_0ct6vTPED6oHis4mc8A0BvId5UwQcmZ
https://drive.google.com/drive/u/0/folders/
1_0ct6vTPED6oHis4mc8A0BvId5UwQcmZ
https://drive.google.com/drive/u/0/folders/
1_0ct6vTPED6oHis4mc8A0BvId5UwQcmZ
https://drive.google.com/drive/u/0/folders/
1_0ct6vTPED6oHis4mc8A0BvId5UwQcmZ

APPENDIX

[1]:Model Trained using CNN and FER-2013

# -*- coding: utf-8 -*-

"""Facial_Expression.ipynb
Automatically generated by Colaboratory.

Original file is located at

https://colab.research.google.com/drive/1gD6yKGgsfYKC4YlxS_gruxpyn8P_uecZ
"""

from google.colab import drive

drive.mount('/content/drive')

# Commented out IPython magic to ensure Python compatibility.

# %cd drive/MyDrive/Facial_Expression/

!ls

import tarfile

fname = 'archive.zip'
if fname.endswith("tar.gz"):
tar = tarfile.open(fname, "r:gz")
tar.extractall()
tar.close()
elif fname.endswith("tar"):
tar = tarfile.open(fname, "r:")
tar.extractall()
tar.close()

import os
import numpy as np
import pandas as pd
import tensorflow
from keras import optimizers
from matplotlib import pyplot
from sklearn.model_selection import train_test_split

df = pd.read_csv('fer2013/fer2013.csv')
df.head()

df.emotion.unique()

label_to_text =
{0:'anger',1:'disgust',2:'fear',3:'happiness',4:'sadness',5:'surprise',6:'
neutral'}

np.array(df.pixels.loc[0].split(' ')).reshape(48,48)

pyplot.imshow(np.array(df.pixels.loc[0].split('
')).reshape(48,48).astype('float'))

fig = pyplot.figure(1, (14, 14))

k = 0
for label in sorted(df.emotion.unique()):
 for j in range(3):
px = df[df.emotion==label].pixels.iloc[k]
px = np.array(px.split(' ')).reshape(48, 48).astype('float32')
k += 1
ax = pyplot.subplot(7, 7, k)
ax.imshow(px)
ax.set_xticks([])
ax.set_yticks([])
ax.set_title(label_to_text[label])
pyplot.tight_layout()

img_array = df.pixels.apply(lambda x : np.array(x.split(' ')).reshape(48,

48,1).astype('float32'))

img_array = np.stack(img_array,axis=0)

labels = df.emotion.values

X_train,X_test,y_train,y_test =
train_test_split(img_array,labels,test_size = .2)

X_train.shape,y_train.shape,X_test.shape,y_test.shape

X_train = X_train/255
X_test = X_test/255

basemodel =
tensorflow.keras.models.Sequential([tensorflow.keras.layers.Conv2D(32,
(3,3),activation='relu',input_shape = (48,48,1)),
tensorflow.keras.layers.MaxPool2D(2,2),

tensorflow.keras.layers.BatchNormalization(),

tensorflow.keras.layers.Conv2D(64,
(3,3),activation='relu',input_shape = (48,48,1)),

tensorflow.keras.layers.MaxPool2D(2,2),

tensorflow.keras.layers.BatchNormalization(),

tensorflow.keras.layers.Conv2D(128,(3,3),activation='relu',input_shape =
(48,48,1)),

tensorflow.keras.layers.MaxPool2D(2,2),

tensorflow.keras.layers.BatchNormalization(),
#

tensorflow.keras.layers.Flatten(),
tensorflow.keras.layers.Dense(128,activation='relu'),

tensorflow.keras.layers.Dense(7,activation='softmax')
basemodel.compile(optimizer=tensorflow.keras.optimizers.RMSprop(learning_r
ate=.0001),
loss = 'sparse_categorical_crossentropy',
metrics =['accuracy'])

import os
try:
os.mkdir('checkpoint')
except:
pass

file_name = 'best_model.h5'

checkpoint_path= os.path.join('checkpoint',file_name)

call_back =
tensorflow.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
monitor='val_accuracy',
verbose=1,
save_freq='epoch',
save_best_only=True,
save_weights_only=False,
mode='max')

checkpoint_path

basemodel.fit(X_train,y_train,epochs=20,validation_split=.1,callbacks=call
_back)

final_model = tensorflow.keras.models.load_model(checkpoint_path)
from IPython.display import clear_output
import time
for k in range(40):
print(f'actual label is {label_to_text[y_test[k]]}')
predicted_class =
final_model.predict(tensorflow.expand_dims(X_test[k],0)).argmax()
print(f'predict label is {label_to_text[predicted_class]}')
pyplot.imshow(X_test[k].reshape((48,48)))
pyplot.show()
time.sleep(5)
clear_output(wait=True)
s
[2]: EMOTION DETECTION USING CNN WITH EMOTIFY DATASET

!pip install google-colab

from google.colab import files
uploaded = files.upload()
for fn in uploaded.keys():
print('User upload file "{name}" with length {length} bytes'.format(
name = fn, length = len(uploaded[fn]))
)
from zipfile import ZipFile
file_name = "emojify.zip"

with ZipFile(file_name, 'r') as zip:

zip.extractall()
print("Done")

import numpy as np
import cv2
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D
from tensorflow.keras.optimizers import Adam
from keras.layers import MaxPooling2D
from keras.preprocessing.image import ImageDataGenerator

train_dir = 'train'
val_dir = 'test'
train_datagen = ImageDataGenerator(rescale=1./255)
val_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
train_dir,
target_size=(48,48),
batch_size=64,
color_mode="grayscale",
class_mode='categorical')

validation_generator = val_datagen.flow_from_directory(
val_dir,
target_size=(48,48),
batch_size=64,
color_mode="grayscale",
class_mode='categorical')
emotion_model = Sequential()
emotion_model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_
shape=(48,48,1)))
emotion_model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
emotion_model.add(Dropout(0.25))
emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
emotion_model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
emotion_model.add(MaxPooling2D(pool_size=(2, 2)))
emotion_model.add(Dropout(0.25))
emotion_model.add(Flatten())
emotion_model.add(Dense(1024, activation='relu'))
emotion_model.add(Dropout(0.5))
emotion_model.add(Dense(7, activation='softmax'))

emotion_model.compile(loss='categorical_crossentropy',optimizer=Adam(lr=0.
0001, decay=1e-6),metrics=['accuracy'])
emotion_model_info = emotion_model.fit_generator(
train_generator,
steps_per_epoch=28709 // 64,
epochs=50,
validation_data=validation_generator,
validation_steps=7178 // 64)
#Saving the model
emotion_model.save('model.h5')

from keras.models import load_model

emotion_model = load_model('model.h5')

def emotion_analysis(emotions):
objects = ('angry', 'disgust', 'fear', 'happy', 'sad', 'surprise', 'ne
utral')
y_pos = np.arange(len(objects))

plt.bar(y_pos, emotions, align='center', alpha=0.5)

plt.xticks(y_pos, objects)
plt.ylabel('percentage')
plt.title('emotion')

plt.show()
from IPython.display import display, Javascript
from google.colab.output import eval_js
from base64 import b64decode
def take_photo(filename='photo.jpg', quality=0.8):
js = Javascript('''
async function takePhoto(quality) {
const div = document.createElement('div');
const capture = document.createElement('button');
capture.textContent = 'Capture';
div.appendChild(capture);

const video = document.createElement('video');

video.style.display = 'block';
const stream = await navigator.mediaDevices.getUserMedia({video: tru
e});

document.body.appendChild(div);
div.appendChild(video);
video.srcObject = stream;
await video.play();

// Resize the output to fit the video element.

google.colab.output.setIframeHeight(document.documentElement.scrollH
eight, true);

// Wait for Capture to be clicked.

await new Promise((resolve) => capture.onclick = resolve);

const canvas = document.createElement('canvas');

canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
canvas.getContext('2d').drawImage(video, 0, 0);
stream.getVideoTracks()[0].stop();
div.remove();
return canvas.toDataURL('image/jpeg', quality);
}
''')
display(js)
data = eval_js('takePhoto({})'.format(quality))
binary = b64decode(data.split(',')[1])
with open(filename, 'wb') as f:
f.write(binary)
return filename

take_photo()

import cv2
def facecrop(image):
facedata = '/content/haarcascade_frontalface_alt.xml'
cascade = cv2.CascadeClassifier(facedata)

img = cv2.imread(image)

try:

minisize = (img.shape[1],img.shape[0])
miniframe = cv2.resize(img, minisize)

faces = cascade.detectMultiScale(miniframe)

for f in faces:
x, y, w, h = [ v for v in f ]
cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0), 2)

sub_face = img[y:y+h, x:x+w]

cv2.imwrite('capture.jpg', sub_face)
#print ("Writing: " + image)

except Exception as e:
print (e)

if __name__ == '__main__':
facecrop('/content/photo.jpg')
#Testing a file.

from keras.preprocessing import image

from keras.preprocessing.image import ImageDataGenerator

import numpy as np
import matplotlib.pyplot as plt

file = '/content/capture.jpg'
true_image = image.load_img(file)
img = image.load_img(file, color_mode="grayscale", target_size=(48, 48))

x = image.img_to_array(img)
x = np.expand_dims(x, axis = 0)

x /= 255

custom = emotion_model.predict(x)
emotion_analysis(custom[0])

x = np.array(x, 'float32')
x = x.reshape([48, 48]);

plt.imshow(true_image)
plt.show()

[3]: EMOTION DETECTION USING DEEP LEARNING WITH DEEPFACE LIBRARY

pip install deepface

pip install opencv-python
from deepface import DeepFace
import cv2
import matplotlib.pyplot as plt
img1 = cv2.imread("/content/happyboy.jpg")
plt.imshow(img1[:,:,::-1])
plt.show()
result = DeepFace.analyze(img1,actions = ['emotion'])
print(result)
pip install numpy

import imutils
import numpy as np
import cv2
from google.colab.patches import cv2_imshow
from IPython.display import display, Javascript
from google.colab.output import eval_js
from base64 import b64decode
def take_photo(filename ='photo.jpg',quality = 0.8):
js = Javascript('''
async function takePhoto(quality){
const div = document.createElement('div');
const capture = document.createElement('button');
capture.textContent = 'Capture';
div.appendChild(capture);

const video = document.createElement('video');

video.style.display = 'block';
const stream = await navigator.mediaDevices.getUserMedia({video: tru
e});

document.body.appendChild(div);
div.appendChild(video);
video.srcObject = stream;
await video.play();

google.colab.output.setIframeHeight(document.documentElement.scrollH
eight,true);

await new Promise((resolve) => capture.onclick = resolve);

const canvas = document.createElement('canvas');

canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
canvas.getContext('2d').drawImage(video,0,0);
stream.getVideoTracks()[0].stop();
div.remove();
return canvas.toDataURL('image/jpeg',quality);
}
''')
display(js)
data = eval_js('takePhoto({})'.format(quality))
binary = b64decode(data.split(',')[1])
with open(filename, 'wb') as f:
f.write(binary)
return filename
image_file = take_photo()

image = cv2.imread(image_file)

image = imutils.resize(image, width = 400)

(h, w) = image.shape[:2]
print(w,h)
cv2_imshow(image)

[4]: EMOTION DETECTION USING CNN WITH TRAINING DATASET

from google.colab import files

uploaded = files.upload()
for fn in uploaded.keys():
print('User upload file "{name}" with length {length} bytes'.format(
name = fn, length = len(uploaded[fn]))
)

pip install numpy

from zipfile import ZipFile

file_name = "imp.zip"

with ZipFile(file_name, 'r') as zip:

zip.extractall()
print("Done")

from keras.models import load_model

from keras.layers import Lambda
import tensorflow as tf
import cv2
import os
import matplotlib.pyplot as plt
import numpy as np

#reading the image

img_array =
cv2.imread('/content/imp/Training1/1/Training_10371709.jpg')

img_array.shape #size of the image rgb 3

print(img_array)

plt.imshow(img_array) # show image

Datadirectory = "/content/imp/Training1" #training dataset

Classes = ["0","1","2","3","4","5","6"] #list of classes

for category in Classes:

path = os.path.join(Datadirectory,category) # the path for all the
datasets of all folders
 for img in os.listdir(path):
img_array = cv2.imread(os.path.join(path,img))
plt.imshow(cv2.cvtColor(img_array, cv2.COLOR_BGR2RGB))
plt.show()
break
break

img_size = 224 #imagenet => 224*224

new_array = cv2.resize(img_array,(img_size,img_size)) # change the
image size
plt.imshow(cv2.cvtColor(new_array,cv2.COLOR_BGR2RGB))
plt.show()

new_array.shape #check the array size

# Read all the image in the dataset and convert into array
training_Data = [] # data array

def create_training_Data():
for category in Classes:
path = os.path.join(Datadirectory, category)
class_num = Classes.index(category)
for img in os.listdir(path):
try:
img_array = cv2.imread(os.path.join(path,img))
new_array = cv2.resize(img_array,(img_size, img_size))
training_Data.append([new_array,class_num])
except Exception as e:
pass

create_training_Data() # function call

print(len(training_Data))

temp = np.array((training_Data),dtype = object)

temp.shape

import random
random.shuffle(training_Data) # shuffle the features

x = [] #data
y = [] #label

for features,label in training_Data:

x.append(features)
y.append(label)

x = np.array(x).reshape(-1, img_size, img_size, 3) #convert into 4

dimensions

x.shape
x = x/255.0; #normalize the data

y = np.array()

y.shape

#Deep learning ,model for training _ Transfer Learning

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

model = tf.keras.applications.MobileNetV2() #pretrained model

model.summary() # what inside it.

#Transfer Learning- tuning,weights will start from last check point

base_input = model.layers[0].input #input

base_output = model.layers[-2].output

base_output

final_output = layers.Dense(128)(base_output) #add new layer,after

the output of global pooling layer
final_output = layers.Activation('relu')(final_output) # activation
function
final_output = layers.Dense(64)(final_output)
final_output = layers.Activation('relu')(final_output)
final_output = layers.Dense(7,activation = 'softmax')(final_output)
#classes are 7

final_output #output

new_model = keras.Model(inputs = base_input, outputs = final_output)

new_model.summary()

new_model.compile(loss = "sparse_categorical_crossentropy", optimizer

= "adam", metrics = ["accuracy"])

new_model.fit( x, y,epochs = 15)

new_model.save('/content/imp/final_model.h5') #pre trained model

new_model = tf.keras.models.load_model('/content/imp/final_model.h5')

frame = cv2.imread("/content/imp/image4.jpg")

frame.shape
plt.imshow(cv2.cvtColor(frame,cv2.COLOR_BGR2RGB))

# we need face detection algorethms

faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades +
'haarcascade_frontalface_default.xml')

gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)

gray.shape

faces = faceCascade.detectMultiScale(gray,1.1,4)
for x,y,w,h in faces:
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]
cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,0),2)
faces = faceCascade.detectMultiScale(roi_gray)
if len(faces) == 0:
print("Face not detected")
else:
for(ex,ey,ew,eh) in faces:
face_roi = roi_color[ey: ey+eh, ex:ex + ew]

plt.imshow(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))

plt.imshow(cv2.cvtColor(face_roi, cv2.COLOR_BGR2RGB))

final_image = cv2.resize(face_roi, (224,224))

final_image = np.expand_dims(final_image,axis = 0)
final_image = final_image/255.0 # normalize

Predictions = new_model.predict(final_image)

Predictions[0]

np.argmax(Predictions) #show predictions

[5]: EMOTION DETECTION USING KNN WITH WEBCAM ACCESS

!pip install streamlit

pip install -U hydralit

pip install hydralit

pip install streamlit_webrtc

import numpy as np
import cv2
import streamlit as st
from tensorflow import keras
from keras.models import model_from_json
from keras.preprocessing.image import img_to_array
from streamlit_webrtc import webrtc_streamer, VideoTransformerBase,
RTCConfiguration, VideoProcessorBase, WebRtcMode

# load model
emotion_dict = {0:'angry', 1 :'happy', 2: 'neutral', 3:'sad', 4:
'surprise'}
# load json and create model
json_file = open('/content/emotion_model1.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
classifier = model_from_json(loaded_model_json)

# load weights into new model

classifier.load_weights("/content/emotion_model1.h5")

#load face
try:
face_cascade =
cv2.CascadeClassifier('/content/haarcascade_frontalface_default.xml')
except Exception:
st.write("Error loading cascade classifiers")

RTC_CONFIGURATION = RTCConfiguration({"iceServers": [{"urls":

["stun:stun.l.google.com:19302"]}]})

class Faceemotion(VideoTransformerBase):
def transform(self, frame):
img = frame.to_ndarray(format="bgr24")

#image gray
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(
image=img_gray, scaleFactor=1.3, minNeighbors=5)
for (x, y, w, h) in faces:
cv2.rectangle(img=img, pt1=(x, y), pt2=(
x + w, y + h), color=(255, 0, 0), thickness=2)
roi_gray = img_gray[y:y + h, x:x + w]
roi_gray = cv2.resize(roi_gray, (48, 48),
interpolation=cv2.INTER_AREA)
if np.sum([roi_gray]) != 0:
roi = roi_gray.astype('float') / 255.0
roi = img_to_array(roi)
roi = np.expand_dims(roi, axis=0)
prediction = classifier.predict(roi)[0]
maxindex = int(np.argmax(prediction))
finalout = emotion_dict[maxindex]
output = str(finalout)
label_position = (x, y)
 cv2.putText(img, output, label_position,
cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)

return img

def main():
# Face Analysis Application #
st.title("Real Time Face Emotion Detection Application")
activiteis = ["Home", "Webcam Face Detection", "About"]
choice = st.sidebar.selectbox("Select Activity", activiteis)
st.sidebar.markdown(
""" Developed by Mohammad Juned Khan
Email : [email protected]
[LinkedIn] (https://www.linkedin.com/in/md-juned-khan)""")
if choice == "Home":
html_temp_home1 = """<div style="background-
color:#6D7B8D;padding:10px">
<h4
style="color:white;text-align:center;">
Face Emotion detection
application using OpenCV, Custom CNN model and Streamlit.</h4>
</div>
</br>"""
st.markdown(html_temp_home1, unsafe_allow_html=True)
st.write("""
The application has two functionalities.
1. Real time face detection using web cam feed.
2. Real time face emotion recognization.
""")
elif choice == "Webcam Face Detection":
st.header("Webcam Live Feed")
st.write("Click on start to use webcam and detect your face
emotion")
webrtc_streamer(key="example", mode=WebRtcMode.SENDRECV,
rtc_configuration=RTC_CONFIGURATION,
video_processor_factory=Faceemotion)

elif choice == "About":

st.subheader("About this app")
html_temp_about1= """<div style="background-
color:#6D7B8D;padding:10px">
<h4 style="color:white;text-
align:center;">
Real time face emotion detection
application using OpenCV, Custom Trained CNN model and Streamlit.</h4>
</div>
</br>"""
st.markdown(html_temp_about1, unsafe_allow_html=True)

html_temp4 = """
<div style="background-
color:#98AFC7;padding:10px">
 <h4 style="color:white;text-
align:center;">This Application is developed by Mohammad Juned Khan
using Streamlit Framework, Opencv, Tensorflow and Keras library for
demonstration purpose. If you're on LinkedIn and want to connect, just
click on the link in sidebar and shoot me a request. If you have any
suggestion or wnat to comment just write a mail at
[email protected]. </h4>
<h4 style="color:white;text-
align:center;">Thanks for Visiting</h4>
</div>
<br></br>
<br></br>"""

st.markdown(html_temp4, unsafe_allow_html=True)

else:
pass

if __name__ == "__main__":
main()