International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072

DETECTION OF HUMAN EMOTIONS IN AN IMAGE USING CNN

Kuppa Srivalli1, Inkollu Chandana2, Guddanti RamyaSri3, Kothapalli Swathi4,
Dr. T. Kameswara Rao5
1,2,3,4Graduation students, Dept. of Comp. Sc. & Engg., Vasireddy Venkatadri Inst. of Tech., Guntur, AP, India
5Professor, Dept. of Comp. Sc. & Engg., Vasireddy Venkatadri Inst. of Tech., Guntur, AP, India
---------------------------------------------------------------------***-------------------------------------------------------------------
Abstract - Emotions possessed by humans can be improving sales, these are the various areas where emotion
recognized and has a vast scope of study in the computer recognition technique is used. Music therapy helps patients
vision industry. Emotions can be recognized by various means. deal with stress, anxiety, depression and a positive effect to
Facial expression, body language, voice level, etc. can be used Alzheimer. In learning phase staff understands the student
for detection. In this paper we used a model to recognize situation and try to give more information effectively and
emotion by facial expressions which play very important role interactive. In monitoring phase if the car driver is sleepy,
in human communication. Since human face is the richest emotion is detected and alert others. This leads to less
source of emotion, in which conversations depend much on number of accidents.
emotions. There are numerous human-machine interactions.
Primary objective of this paper is to develop a system which Human emotion detection has been implemented in
can analyze the image and recognize the expressions of the many areas requiring additional security or information
person by using Deep learning techniques. These expressions about the person and also used for business promotions [3],
can be derived from System’s camera or any existing images [4]. Humans share a rich set of emotions which are
available. Limitation of this paper is to recognize only seven expressed through consistent facial expressions. Human
emotions viz happiness, sadness, anger, disgust, fear, surprise Emotions can be classified as happy, sad, neutral, angry,
and neutral. disgust, fear, anxiety etc. [5], [6]. These Emotions are very
subtle. Facial muscle contraction for different emotions is
Key words: Convolution Neural Networks, Emotion different. Detecting these small changes is difficult as even a
detection, Expression detection, Deep learning. small contrast results in Different Expressions [28]. Machine
Learning have been used for these tasks to obtain good
results. There are many methods for facial expression
1. INTRODUCTION
classification like Hidden Markov model, SVM, Adaboost,
Artificial Neural Networks [8], [9].
The Study of emotions in human-machine interaction has
increased in recent years [1], [2]. Computer Systems used Neural Networks consumes a lot of computational
mouse and keyboard and the trend was shifted to automatic power and they are very slow because of high quality and
speech recognition and now human machine interfaces are more input parameters.
quite trending all over. The communication between humans
and computers will be natural if computers are able to Convolution Neural Networks (CNN) uses less
perceive and respond to human nonverbal communication preprocessing when compared with other algorithms. The
such as emotion [18], [19]. proposed system has different phases such as face detection,
emotion recognition, generation of output image with
If computers are able to recognize these emotions recognized emotions. In this paper we focus on the Facial
they could give specific and appropriate help to the users in expression recognition using convolutional neural networks.
order to satisfy user’s needs and preferences. If systems are Data preprocessing and emotion recognition model
successful in classification of emotions, then it gains better developed based on CNN are two major phases of the
understanding of human behaviour by using the information developed system.
technology. This makes the systems and user interfaces
more emphatic and intelligent. Facial Expression recognition 2. DATA PREPROCESSING
problem attracted computer vision community [20]-[27].

Facial emotion detection applications are spread Original data if directly used for emotion detection, it takes
across different fields like medicine, e-learning, marketing, lot of computational power due to more number of input
monitoring, entertainment, and law. Counselling. parameters. Model is said to be robust only when it uses less
Determination of medical state of person, determining computational power.
feelings, comfort level to treatment [7], adjusting the
To overcome these problems the data is pre-processed.
learning technique by determining emotion, ATM not
Data preprocessing includes:
dispensing money when person is scared while withdrawing
money, prioritizing angry calls in call centres, recognizing a) Face detection
mood and satisfying needs, purchasing decisions and b) Normalization

© 2020, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 6253
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072

C) Gray level equalization C. GRAY LEVEL EQUALIZATION

A. FACE DETECTION Images will show state of uneven distribution due to

shadows. Factors like illumination, shade effects the feature
Open CV by default comprise of a trainer as well as a extraction process, to overcome this problem gray scale
detector. Classifier of any kind can be generated to detect values are averaged to get even distribution. In this paper
fire, cars, planes and other objects. Open CV has many pre- Histogram Equalization (HE) technique is used [11]. The
defined classifiers for face detection. The XML file is stowed technique used converts the histogram of original image to
in Open CV/data/Haar cascades folder. even distribution.

Haar cascades classifier is trained by using the positive The results are showed in the fig.3. Gray level
and negative images. Edge, line, four rectangle, diagonal are Equalization increases the contrast of image. It makes details
its main features and are shown in Fig.1. Important facial clearer. The obtained image is conducive for facial features
features are extracted from large number of Haar-like extraction.
features. Haar Classifier is highly efficient and so it is used
widely.

XML classifiers that are required in detecting a face are

loaded, after which input image is loaded.

Fig -3: Image before and after gray scale normalization

The image which is passed in the above steps is pre-
Fig -1: Haar features for face detection processed and is now ready to enter into the emotion
recognition phase.
B. NORMALIZATION
3. EMOTION RECOGNITION MODEL BASED ON
The input layer of the network accepts image of specific size, CNN
so the original picture is normalized to that size. Scaling ratio
of x-axis and y-axis are used to scale the image. Many other Artificial neural networks inspired by human brain learn
normalization techniques can be used but scale from large amounts of data. Convolution neural networks
normalization is one of the efficient technique and hence assign importance to objects and differentiate one from
used in this model [10]. other. Convolution neural network is feed forward neural
network. It extracts features from image which are
The points in the original picture are mapped to other
important for emotion detection and classification. Back
points by using scale normalization. In both the directions
propagation algorithm is used to optimize network
image is scaled by using the scaling ratio’s sx and sy.
parameters. The model will be trained to recognize seven
emotions as shown in the figure 4

a) before normalization b) after normalization

Fig -2: Distinction of before and after normalization

Fig -4: Training Process
© 2020, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 6254
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072

The nature of deep learning method is to build neural Input layer is two-dimensional matrix. It is composed of
networks which learns features. The rules are developed at image pixels. The model presented in this paper used four
the end of training as shown in the figure 5. Rules are convolution layers. The gray scale image with 48 x 48 pixel
nothing but the weights. In the training phase, the network is matrix is the input to this layer. Every feature map is
initialized with random weights. Training pattern is feed in connected to its previous map. In every layer there are
to get output. The obtained output is compared to target several feature maps. The convolution layer C1 uses 64
output. Adjust weights based on error. This process is convolutional nuclei. The size of convolution nuclei is 3 x 3.
repeated until all patterns are passed one time which is Layer C2 and C3 uses 128 nuclei. Softmax layer contains 7
called as epoch. Like this the process is carried out for every neurons. The feature of the output layer is classified among
epoch. the seven emotions. .Layers as shown in figure 9.

A. CONVOLUTION LAYER

Convolution layer is the basic building block of a

Convolutional Network. It does most of the computational
work to the model. Convolution is a mathematical operation
on two functions to produce a third function. The
convolution operation is associated with local connectivity,
which learns correlations among neighboring pixels and
process. Image is matrix of numbers as seen by computer.
These numbers represents pixel values.

In Convolution layer every node is not connected to

Fig -5: Detailed training view every other neuron. Connections are localized. The
Initially the pre-processed image is passed to the convolution is applied on the input image using a
network’s input layer. Layer by layer processing of the image convolution filter. It is also called as kernel is used to
is done. Every layer has two-dimension planes. They are produce a feature map. A filter passes over the input image,
nothing but feature maps, it has several neurons. Computer [12] scanning a few pixels at a time and creating a feature
understands the image by its pixel values. Neural networks map that predicts the class to which each feature belongs
interpret the image from pixel to the lines, curves, edges, [13]. Padding and stride are two main properties of every
finally objects understood by human brain, then the emotion
filter. Activation Function is used.
is recognized. This paper designs CNN structure with the
following layers:
In the proposed model a kernel of 3x3 is used to
a) Convolution layer detect the features like edge detection, curved and sharpen
b) Pooling layer features by applying different filters. Relu is an Activation
c) Fully connected layer Function which performs non-linearity. In the past,
d) SoftMax layer nonlinear functions like tanh, sigmoid are used now Relu is
Layers involved in this model are: Input->Conv(C1)- used as it is far better because the network is able to train a
>Pool(S1)->Conv(C2)->Pool(S2)->Conv(C3)- lot faster without making a significant difference in accuracy.
>Pool(S3)->FC->Softmax
Relu function operates as below:

f(x) = max (0, x).

This layer changes the negative values to 0.

It was already proved that the network using Relu

activation function has moderate sparsity. It solves the
problem of gradient disappearance. This may occur in the
process of modifying back propagation parameters. This
accelerates convergence of the network. The features
extracted by convolution operation can be directly used to
train classifiers, but it has lot of computational challenges. To
Fig -6: CNN Structure for emotion recognition reduce the parameters, down sampling operation is
proposed after the convolution. The average or maximum
value of a specific feature in an image are computed. This

© 2020, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 6255
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072

statistical dimensionality reduction method reduces the C. FULLY CONNECTED LAYER

number of parameters, prevents fitting.
Fully connected layers are an essential component of
B. POOLING LAYER Convolutional Neural Networks (CNN), which have been
proven very successful in recognizing and classifying images
After Convolutional layer, the process undergoes to pooling for computer vision. Convolutional layer and pooling layer
layer [14], [15]. Pooling reduces the amount of information do feature extraction whereas fully connected layers do
in each feature obtained in the above layer and also classification based on the features by previous layer.
maintains the most important information. Pooling reduces
the dimensions and so it is also called as dimensionality Fully Connected input layer(flatten): This layer
reduction. In pooling layer there are different types. They are takes the output of the previous layers i.e., (Convolutional
MaxPooling, MinPooling and AveragePooling. Mostly used layers and pooling layers) flattens them and returns them
pooling is MaxPooling. This layer normally takes a filter into a single vector that can be input to the next stage.
(normally size of 2x2) and a stride of same length. It then Flatten means the matrix is reduced to a single dimension
applies it to the values and place the maximum number in vector.
every region that the filter moved around into the output
matrix. First Fully Connected layer: This layer takes the
input from the feature analysis. It applies weights to predict
The addition of a pooling layer after the the correct label.
convolutional layer is a common pattern used for ordering
the layers within a convolutional neural network that may be Fully Connected output layer: This layer gives the
repeated one or more times in the give model. The pooling final probabilities for each label. The 1D data acts as the
layer operates on each feature map separately. It creates a input to the neurons of this layer which performs a dot
new set of the same number of pooled feature maps. Pooling product of this input data and the neurons weights to
involves selecting a pooling operation. It is like a filter to be produce a single number as output (probability values)
applied to feature maps. The size of the pooling operation or single value per neuron.
filter is smaller than the size of the feature map, specifically,
In fully connected layer every neuron in one layer is
it is almost 2x2 pixels applied with a stride of 2 pixels. For
connected to every other neuron in another layer. In general
example if we take feature map of size (224x224) then after
neural networks the input image is directly given to the
pooling is applied the output matrix will be of size
network in which every neuron is connected to every other
(112x112) as shown in figure 7.
neuron, there the computations are high, the speed becomes
It halves the dimensions of the next feature map. low and the results will be accurate as every parameter is
Computational complexity is for the next convolution considered which improves the quality.
operation is reduced. Training speed is increased. If the
In CNN before coming to this layer, dimensions are
softmax layer is trained directly with the learnt features with
reduced and the actual training is done in this layer. The
reducing dimensions, then dimension problem occurs and
main difference of NN and CNN is that all the layers before
computations will be high. So to avoid this pooling layer is
this layers handle preprocessing in CNN, but for traditional
used after convolution layer.
NN separate preprocessing is done.

Fig -8: Fully Connected layer functionality

Fig -7: Pooling Procedure D. SOFTMAX LAYER

This is the last layer in the process to detect the emotion. It

takes the output from the previous layer i.e., Fully Connected
layer, the output is probability values, after applying softmax
function the values are in between (0, 1). For the given image

© 2020, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 6256
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072

the probability for every emotion is calculated. All Happiness is the most desired expression of a
probability values are compared the one which is having human. The path of the image is set to trained data model to
highest is declared as the emotion state for the given input. recognize the emotions. Model discussed in this paper can
detect a maximum number of images in a group as shown in
fig 11.

Fig -9: Softmax layer with probabilities detected at each

neuron
Fig -11: Happy emotion detection
E. DATASET STUDY
A group of expressions in an Image showing different
The data used for training and testing is ICML 2013 dataset. kind of emotions is shown in fig12.
It comprises of 35, 917 images. This dataset has images with
emotions like happy, sad, disgust, angry, fear, surprise,
neutral. 80 percent of images in the dataset are used for
training and 20 percent of images are used for testing. The
dataset contains 48x48 pixel gray scale images of faces.
Neural networks performs well when more data is used for
training. The images have front viewed faces. This data set
when compared to FER 2013 has more number of disgust Fig -12: Model output detecting different emotions
images. So this dataset is chosen. Figure 10 shows the
dataset images. In the below fig13 group of 4 images, the side faces are
not recognized because the model detects only images
happy angry disgust fear capturing face at least 70 Percentage as in second left image.

sad surprise neutral

Fig -13: Face detection

Image showing different kind of emotions. Neutral is not
Fig -10: Dataset Images satisfactory result of our model as the edge detection of this
emotion is not perfect.
For the training and testing the network. Dataset plays a
very important role. Even though the model developed is 5. CONCLUSION
robust if the dataset is not good, it leads to wrong outcomes.
The system proposed in this paper detects emotion of a
4. RESULT ANALYSIS group of persons in a given image by their facial expression
using convolutional neural network. It primarily comprises
The model developed takes the image as input and detects of three parts i.e., face representation, emotion pattern
the emotion of every person in the given image and produces recognition and classification. CNN detects the patterns of
output image with emotions. To get how the model performs emotion as we trained the model with large amount of data.
on each emotion it is depicted as follows. It is found that The proposed system detects seven major emotions they are
happiness emotion is obtained with high accuracy and happy, sad, angry, fear, disgust, surprise and neutral.
neutral with low accuracy as it is matched with sad
expression until carefully examined. The data with images of even size collected are fed
to model. Compared to past literature this model is able to
recognize disgust emotion accurately and this model is

© 2020, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 6257
 International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056
Volume: 07 Issue: 04 | Apr 2020 www.irjet.net p-ISSN: 2395-0072

robust to noise. We achieved a performance of detecting Comput., Commun., Appl., vol. 14, no. 1s, Apr. 2018, Art.
expression with 80 percent accuracy. The proposed model no. 27.
detects the emotion of a person if face is recognized more [15] H. Ma and T. Celik, ``FER-Net: Facial expression
than 70% in an image. recognition using densely connected convolutional
network,'' Electron. Lett., vol. 55, no. 4, pp. 184_186, Feb.
Future works should attempt to the development of 2019.
model to detect emotion of a person even with half faces, [16] L. Wei, C. Tsangouri, F. Abtahi, and Z. Zhu, ``A recursive
detecting emotion in video’s and dynamic recognition of framework for expression recognition: From Web
emotion with 3D technology. images to deep models to game dataset,'' Mach. Vis.
Appl., vol. 29, no. 3, pp. 489_502, 2018.
REFERENCES [17] S. Li andW. Deng, ``Reliable crowdsourcing and deep
locality-preserving learning for unconstrained facial
[1] R. M. Mehmood, R. Du, and H. J. Lee, ``Optimal feature expression recognition,'' IEEE Trans.Image Process., vol.
selection and deep learning ensembles method for 28, no. 1, pp. 356_370, Jan. 2018
emotion recognition from human brain [18] A.Mehrabian, “Communication without words,”
[2] T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and Z. Cui, Psychology today, vol.2, no.4, pp.53-56, 1968.
``MPED: A multi-modal physiological emotion database [19] R.W. Picard, Affective Computing. Cambridge.MA : MIT
for discrete emotion recognition,''IEEE Access, vol. 7, pp. Press, 1997.
12177_12191, 2019 [20] D. Beymer, A. Shashua, and T. Poggio, Example Based
[3] E. Batbaatar, M. Li, and K. H. Ryu, ``Semantic-emotion Image Analysis and Synthesis, M.I.T. A.I. Memo No. 1431,
neural network for emotion recognition from text,'' IEEE 1993.
Access, vol. 7, pp. 111866_111878, 2019. [21] I.A. Essa and A. Pentland, “A Vision System for Observing
[4] H. Meng, N. Bianchi-Berthouze, Y. Deng, J. Cheng, and J. P. and Extracting Facial Action Parameters”, Proc. IEEE
Cosmas,``Time-delay neural network for continuous CVPR, pp.76-83, 1994.
emotional dimension prediction from facial expression [22] H. Li, P. Roivainen, and R. Forcheimer,“3D Motion
[5] sequences,'' IEEE Trans. Cybern., vol. 46, no. 4, pp. Estimation in Model-Based Facial Image Coding”, IEEE
916_929, Apr. 2016. Trans. Pattern Analysis and Machine intelligence, vol.
[6] X. U. Feng and J.-P. Zhang, ``Facial microexpression 15, pp. 545-555,1993.
recognition: A survey,''Acta Automatica Sinica, vol. 43, [23] K. Mase, “Recognition of Facial Expression from Optical
no. 3, pp. 333_348, 2017 Flow”, IEICE Trans., vol. E 74, pp. 3474-3483, 1991.
[7] M. S. Özerdem and H. Polat, ``Emotion recognition based [24] K. Matsuno, C. Lee, and S.Tsuji, “Recognition of Human
on EEG features in movie clips with channel selection,'' Facial Expressions without Feature Extraction”, Proc.
Brain Inf., vol. 4, no. 4, pp. 241_252, 2017. ECCV, pp. 513-520, 1994.
[8] F. Vella, I. Infantino, and G. Scardino, ``Person [25] M. Rosenblum, Y. Yacoob, and L.S. Davis, “Human
identi_cation through entropy oriented mean shift Emotion Recognition from Motion Using a Radial Basis
clustering of human gaze patterns,'' Multime-dia Tools Function Network Architecture”, IEEE Workshop Motion
Appl., vol. 76, no. 2, pp. 2289_2313, Jan. 2017. of Non-Rigid and Articulated Objects, Austin, Texas, pp.
[9] S. K. A. Kamarol, M. H. Jaward, H. Kälviäinen, J. 43-49, Nov. 1994.
Parkkinen, and R. Parthiban, ``Joint facial expression [26] D. Terzopoulos and K. Waters, “Analysis and Synthesis of
recognition and intensity estimation based on weighted Facial Image Sequences Using Physical and Anatomical
votes of image sequences,'' Pattern Recognit. Lett., vol. Models”, IEEE Trans. Pattern Analysis and Machine
92, pp. 25_32, Jun. 2017. Intelligence, vol. 15, pp. 569-579, 1993.
[10] J. Cai, Q. Chang, X.-L. Tang, C. Xue, and C. Wei, ``Facial [27] Y.Yacob and L Devis, “Recognizing Human facial
expressionrecognition method based on sparse batch expression from long image sequences using optical
normalization CNN,'' in Proc.37th Chin. Control Conf. flow”, IEEE transaction on Pattern Analysis and Machine
(CCC), Jul. 2018, pp. 9608_9613. Intelligence [PAMI], 18{6}: 636-642, 1996.
[11] M. Takalkar, M. Xu, Q. Wu, and Z. Chaczko, ``A survey: [28] J. N. Bassili, Emotion recognition: The role of facial
Facial micro-expression recognition,'' Multimedia Tools movement and the relative importance of upper and
Appl., vol. 77, no. 15, pp. 19301_19325, 2018. lower areas of the face, J. Personality and Social Psych.,
[12] Magudeeswaran and J. F. Singh, ``Contrast limited fuzzy vol. 37, pp. 204
adaptive histogram equalization for enhancement of
brain
[13] images,'' Int. J. Imag. Syst. Technol., vol. 27, no. 1, pp.
98_103, 2017.
[14] F. Zhang, Q. Mao, X. Shen, Y. Zhan, and M. Dong,
``Spatially coherent feature learning for pose-invariant
facial expression recognition,'' ACM Trans. Multimedia

© 2020, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page 6258