“Emotion Detection in Images”

A PROJECT REPORT

Submitted by

Anurag Bharti(21BCS11044)

Chanchal Muskan(21BCS1879)

Ankit Chaurasiya(21BCS11172)

Raghvendra Singh(21BCS11305)

Nitin(21BCS2719)

Under the Supervision of

Neha Rajput (15440)

in partial fulfillment for the award of the degree of

Bachelors of Engineering
IN
Computer Science & Engineering

Chandigarh University
August - December 2023
 BONAFIDE CERTIFICATE

Certified that this project report “Emotion Detection in Images” is the bonafide work of

“Anurag Bharti 21BCS11044”, “Chanchal Muskan 21BCS1879”, “Ankit Chaurasiya

21BCS11172”, “Raghvendra Singh 21BCS11305”, “Nitin 21BCS2719” who carried

out the project work under my/our supervision.

SIGNATURE SIGNATURE

Dr. Sandeep Singh Kang Neha Rajput

HEAD OF THE DEPARTMENT SUPERVISOR

CSE CSE

Submitted for the project viva-voce examination held on__________________

INTERNAL EXAMINER EXTERNAL EXAMINER

 ACKNOWLEDGMENT

We take this opportunity to express our deep sense of gratitude, thanks and regards
towards all who directly or indirectly helped mein the successful completion of
project.

We present our sincere thanks to our supervisor Er. Neha Rajput who helped us a
lot while making this project.

We also thank to our Head of Department Dr. Sandeep Singh Kang who has
sincerely supported us with the valuable insights into completion of this project.

We are Grateful to all faculty members of Chandigarh University and our friends
who have helped us in the successful completion of project.

Last but not the least We are indebted to our PARENTS who provided us their time,
support and inspiration needed to prepare this report.
TABLE OF CONTENTS

List of Figures .............................................................................................................................

List of Tables ..............................................................................................................................

List of Standards .........................................................................................................................

CHAPTER 1. INTRODUCTION .......................................................................................01

1.1. Identification of Client/ Need/ Relevant Contemporary issue .....................................01

1.2. Identification of Problem .............................................................................................03

1.3. Identification of Tasks .................................................................................................06

1.4. Timeline .......................................................................................................................07

1.5. Organization of the Report ...........................................................................................08

CHAPTER 2. LITERATURE REVIEW/BACKGROUND STUDY ..............................09

2.1. Timeline of the reported problem ................................................................................09

2.2. Existing solutions .........................................................................................................12

2.3. Bibliometric analysis ....................................................................................................14

2.4. Review Summary .........................................................................................................17

2.5. Problem Definition .......................................................................................................19

2.6. Goals/Objectives ..........................................................................................................20

CHAPTER 3. DESIGN FLOW/PROCESS .......................................................................21

3.1. Evaluation & Selection of Specifications/Features ......................................................22

3.2. Design Constraints .......................................................................................................24

3.3. Analysis of Features and finalization subject to constraints ........................................27

3.4. Design Flow .................................................................................................................29

3.5. Design selection ...........................................................................................................32

3.6. Implementation plan/methodology ..............................................................................35

CHAPTER 4. RESULTS ANALYSIS AND VALIDATION ....................... 40
4.1. Implementation of solution ..........................................................................................43

CHAPTER 5. CONCLUSION AND FUTURE WORK ............................... 47

5.1. Conclusion ...................................................................................................................47

5.2. Future work ..................................................................................................................49

REFERENCES ................................................................................................... 51
APPENDIX ......................................................................................................... 52
1. Plagiarism Report ............................................................................................................53

2. User Manual ....................................................................................................................54

List of Figures:

Figure 1.1………………………………………. Human Face Emotions

Figure 1.2………………………………………. Problem Formulation

Figure 1.3………………………………………. Diagram of project development life cycle

Figure 1.4………………………………………. Gantt Chart of Project Timeline

Figure 3.4.1………………………………………Context Diagram

Figure 3.4.2………………………………………Context Level Diagram

Figure 3.4.3………………………………………Face Detection

Figure 3.4.3………………………………………Emotion Classification

Figure 3.4.4……………………………………… Sequence diagram of training and testing

Figure 3.6.1……………………………………… Flowchart of training module

Figure 3.6.2………………………………………. Flowchart of testing module

Figure 3.6.3……………………………………. Sequence diagram of System Functionality

Figure 5.1.1……………………………………. Experimental Demonstration from Image File

Figure 5.1.2……………………………………. Experimental Demonstration from Camera

List of Tables:

Table 1…………………………………….….…. Team Role

 List of Standards (Mandatory For Engineering Programs)

Publishing
Standard About the standard Page no
Agency
IEEE 802.11 is part of the IEEE
802 set of local area network
(LAN) technical standards and
specifies the set of media access
IEEE Mention page nowhere standard
IEEE control (MAC) and physical layer
802.11 is used
(PHY) protocols for
implementing wireless local area
network (WLAN) computer
communication.
Note: Text in Red is presented as an example (replace with relevant information)
 ABSTRACT

Facial Expression Recognition (FER) is a vital component of human-machine interaction, finding

applications in human behavior analysis, mental health assessment, and synthetic expressions. This
system leverages image processing and machine learning techniques, particularly the Local Binary
Pattern (LBP), to capture facial features efficiently. LBP labels pixels based on their local
neighborhoods and constructs histograms, enabling emotion identification. FER typically involves
four stages: pre-processing, face detection, feature extraction, and expression classification.

In this project, we employ convolutional neural networks (CNNs) to recognize seven key human
emotions: anger, disgust, fear, happiness, sadness, surprise, and neutrality. To evaluate our method's
performance, we use Support Vector Machine (SVM) on a trained database. Experimental results
demonstrate the superiority of the LBP descriptor over other appearance-based feature representation
methods.
 1

CHAPTER 1
INTRODUCTION
“2018 is the year when machines learn to grasp human emotions” --Andrew Moore, the
dean of computer science at Carnegie Mellon.

With the advent of modern technology our desires went high and it binds no bounds. In the present
era a huge research work is going on in the field of digital image and image processing. The way of
progression has been exponential and it is ever increasing. Image Processing is a vast area of research
in present day world and its applications are very widespread.

Image processing is the field of signal processing where both the input and output signals are images.
One of the most important applications of Image processing is Facial expression recognition. Our
emotion is revealed by the expressions in our face. Facial Expressions plays an important role in
interpersonal communication. Facial expression is a nonverbal scientific gesture which gets
expressed in our face as per our emotions. Automatic recognition of facial expression plays an
important role in artificial intelligence and robotics and thus it is a need of the generation. Some
application related to this include Personal identification and Access control, Videophone and
Teleconferencing, Forensic application, Human-Computer Interaction, Automated Surveillance,
Cosmetology and so on.

The objective of this project is to develop Automatic Facial Expression Recognition System which
can take human facial images containing some expression as input and recognize and classify it into
seven different expression class such as:

I. Neutral
II. Angry
III. Disgust
IV. Fear
V. Happy
VI. Sadness
VII. Surprise
 2

A Facial expression is the visible manifestation of the affective state, cognitive activity, intention,
personality and psychopathology of a person and plays a communicative role in interpersonal
relations. It has been studied for a long period of time and obtaining the progress recent decades.
Though much progress has been made, recognizing facial expression with a high accuracy remains
to be difficult due to the complexity and varieties of facial expressions.

Generally human beings can convey intentions and emotions through nonverbal ways such as
gestures, facial expressions and involuntary languages. This system can be significantly useful,
nonverbal way for people to communicate with each other. The important thing is how fluently the
system detects or extracts the facial expression from image. The system is growing attention because
this could be widely used in many fields like lie detection, medical assessment and human computer
interface. The Facial Action Coding System (FACS), which was proposed in 1978 by Ekman and
refined in 2002, is a very popular facial expression analysis tool.

On day-to-day basics humans commonly recognize emotions by characteristic features, displayed as

a part of a facial expression. For instance, happiness is undeniably associated with a smile or an
upward movement of the corners of the lips. Similarly other emotions are characterized by other
deformations typical to a particular expression. Research into automatic recognition of facial
expressions addresses the problems surrounding the representation and categorization of static or
dynamic characteristics of these deformations of face pigmentation.

The system classifies facial expression of the same person into the basic emotions namely anger,
disgust, fear, happiness, sadness and surprise. The main purpose of this system is efficient interaction
between human beings and machines using eye gaze, facial expressions, cognitive modeling etc.
Here, detection and classification of facial expressions can be used as a natural way for the interaction
between man and machine. And the system intensity varies from person to person and also varies
along with age, gender, size and shape of face, and further, even the expressions of the same person
do not remain constant with time.

However, the inherent variability of facial images caused by different factors like variations in
illumination, pose, alignment, occlusions make expression recognition a challenging task. Some
surveys on facial feature representations for face recognition and expression analysis addressed these
challenges and possible solutions in detail.
 3

1.1. Identification of Need / Relevant Contemporary issue

In today’s networked world the need to maintain security of information or physical property is
becoming both increasingly important and increasingly difficult. In countries like Nepal the rate of
crimes are increasing day by day. No automatic systems are there that can track person’s activity. If
we will be able to track Facial expressions of persons automatically then we can find the criminal
easily since facial expressions changes doing different activities. So we decided to make a Facial
Expression Recognition System.

We are interested in this project after we went through few papers in this area. The papers were
published as per their system creation and way of creating the system for accurate and reliable facial
expression recognition system.
As a result we are highly motivated to develop a system that recognizes facial expression
and track one person’s activity.

We have also been motivated observing the benefits of physically handicapped people like deaf and
dumb. But if any normal human being or an automated system can understand their needs by
observing their facial expression then it becomes a lot easier for them to make the fellow human or
automated system understand their needs.

Significant debate has risen in past regarding the emotions portrayed in the world famous masterpiece
of Mona Lisa. British Weekly „New Scientist‟ has stated that she is in fact a blend of many different
emotions, 83%happy, 9% disgusted, 6% fearful, 2% angry.

1.2. Identification of Problem

Human emotions and intentions are expressed through facial expressions and deriving an efficient
and effective feature is the fundamental component of facial expression system. Face recognition is
important for the interpretation of facial expressions in applications such as intelligent, man-machine
interface and communication, intelligent visual surveillance, teleconference and real-time animation
from live motion images. The facial expressions are useful for efficient interaction Most research and
system in facial expression recognition are limited to six basic expressions (joy, sad, anger, disgust,
fear surprise). It is found that it is insufficient to describe all facial expressions and these expressions
are categorized based on facial action.
 4

Detecting face and recognizing the facial expression is a very complicated task when it is a vital
to pay attention to primary components like: face configuration, orientation, location where the
face is set.
Human facial expressions can be easily classified into 7 basic emotions: happy, sad, surprise, fear,
anger, disgust, and neutral. Our facial emotions are expressed through activation of specific sets
of facial muscles. These sometimes subtle, yet complex, signals in an expression often contain an
abundant amount of information about our state of mind. Through facial emotion recognition, we
are able to measure the effects that content and services have on the audience/users through an
easy and low-cost procedure. For example, retailers may use these metrics to evaluate customer
interest. Healthcare providers can provide better service by using additional information about
patients' emotional state during treatment. Entertainment producers can monitor audience
engagement in events to consistently create desired content.

Humans are well-trained in reading the emotions of others, in fact, at just 14 months old, babies
can already tell the difference between happy and sad. But can computers do a better job than
us in accessing emotional states? To answer the question, We will designed a deep learning
neural network that gives machines the ability to make inferences about our emotional states. In
other words, we give them eyes to see what we can see.

Problem formulation of our project:

 5

Facial expression recognition is a process performed by humans or computers, which consists of:

1. Locating faces in the scene (e.g., in an image; this step is also referred to as facedetection),
2. Extracting facial features from the detected face region (e.g., detecting the shape of
facialcomponents or describing the texture of the skin in a facial area; this step is referred to
asfacial feature extraction),
3. Analyzing the motion of facial features and/or the changes in the appearance of facialfeatures
and classifying this information into some facial-expressioninterpretativecategories such as facial
muscle activations like smile or frown, emotion (affect)categories like happiness or anger, attitude
categories like (dis)liking or ambivalence, etc.(this step is also referred to as facial expression
interpretation).

Several Projects have already been done in this fields and our goal will not only be to develop a
Automatic Facial Expression Recognition System but also improving the accuracy of this system
compared to the other available systems.

Scope and Applications:

The scope of this system is to tackle with the problems that can arise in day to day life. Some of
the scopes are:
1. The system can be used to detect and track a user’s state of mind.
2. The system can be used in mini-marts, shopping center to view the feedback of the customers
to enhance the business,
3. The system can be installed at busy places like airport, railway station or bus station for
detecting human faces and facial expressions of each person. If there are any faces that appeared
suspicious like angry or fearful, the system might set an internal alarm.

4. The system can also be used for educational purpose such as one can get feedback on how the
student is reacting during the class.
5. This system can be used for lie detection amongst criminal suspects during interrogation
6. This system can help people in emotion related -research to improve the processing of emotion
data.
7. Clever marketing is feasible using emotional knowledge of a person which can be identified by
this system.
 6

1.3. Identification of Tasks

The steps we followed while developing this project are-:

Step 1: Analysis of the problem statement.

Step 2: Gathering of the requirement specification

Step 3: Analysation of the feasibility of the project.

Step 4: Development of a general layout.

Step 5: Going by the journals regarding the previous related works on this field.

Step 6: Choosing the method for developing the algorithm.

Step 7: Analysing the various pros and cons.

Step 8: Starting the development of the project

Step 9: Installation of software like visual studio code.

Step 10: Developing an algorithm.

Step 11: Analysation of algorithm by guide.

Step 12: Coding as per the developed algorithm in PYTHON.

 7

We developed this project as per the iterative waterfall model.

1.4. Timeline
 8

1.5. Organization of the Report

Chapter 1 Problem Identification: This chapter introduces the project and describes the problemstatement
discussed earlier in the report.

Chapter 2 Literature Review: This chapter prevents review for various research papers which help us to
understand the problem in a better way. It also defines what has been done to already solve the problem and
what can be further done.

Chapter 3 Design Flow/ Process: This chapter presents the need and significance of the proposedwork based
on literature review. Proposed objectives and methodology are explained. This presents the relevance of the
problem. It also represents logical and schematic plan to resolve theresearch problem.

Chapter 4 Result Analysis and Validation: This chapter explains various performance parameters used in
implementation. Experimental results are shown in this chapter. It explains themeaning of the results and why
they matter.

Chapter 5 Conclusion and future scope: This chapter concludes the results and explain the bestmethod to
perform this research to get the best results and define the future scope of study that explains the extent to
which the research area will be explored in

Team Roles
Member Name UID Roles

Anurag Bharti 21BCS11044 • ANALYSING THE DATASET

• APPLYING ALGORITHMS IN THE DATASET

Chanchal Muskan 21BCS1879 • DOCUMENTING THE REPORTS

• ANALYSING THE LITERATURE REVEIW

Ankit Chaurasiya 21BCS11172 • COLLECTION AND MAKING OF THE DATASET

• CLUSTERING ANDDISTRIBUTION OF THE DATASET.

• VISUALISATION OF THE DATASET

Raghvendra Singh 21BCS11305 • COLLECTION OF DATASETS

• VISUALISATION OF THE DATASET

• TESTING AND TRAINING OFTHE DATASET

• RESEARCH THE TECHSTACK AND REPORTS

Nitin 21BCS2719 • REVIEWING THE DEADLINE

 9

CHAPTER 2

LITERATURE REVIEW/BACKGROUND STUDY

2.1 Timeline of the reported problem

Face emotion recognition and detection using Python and deep learning is a relatively recent
development in the field of artificial intelligence. The problem of its drawbacks and limitations have
been identified over the course of the past decade.

Here is a detailed timeline of the development of face emotion recognition and detection using Python
and deep learning and the problems associated with it:

• 2010: The concept of using deep learning algorithms for facial expression recognition was
introduced, which marked the beginning of research in this area.

• 2012: The first deep learning-based facial expression recognition system was developed by
researchers at the University of Pittsburgh. The system achieved high accuracy in recognizing
six basic emotions: happiness, sadness, anger, fear, disgust, and surprise.

• 2013: Researchers identified that deep learning-based facial recognition systems can suffer
from bias due to the lack of diversity in the training data.

• 2016: A study by researchers at MIT and Stanford found that three commercially available
facial recognition systems had higher error rates for darker-skinned individuals and females.

• 2017: The issue of bias in facial recognition systems was highlighted in a report by the
Georgetown Law Center on Privacy and Technology, which found that many of these systems
are less accurate for people with darker skin tones.

• 2018: Amazon's facial recognition system, Recognition, was found to have higher error rates
for people with darker skin tones in a study by the American Civil Liberties Union (ACLU).
 10

• 2019: The National Institute of Standards and Technology (NIST) released a report on the
performance of facial recognition algorithms, which found that they can be less accurate for
certain demographic groups, such as people with darker skin, women, and children.

• 2020: The use of facial recognition technology by law enforcement came under scrutiny due
to concerns about bias and accuracy. IBM announced that it would no longer offer facial
recognition technology and called for a national dialogue on the ethical use of this technology.

• 2021: The European Commission proposed new regulations on artificial intelligence,

including a ban on facial recognition technology in public spaces for surveillance purposes.
The ban was later included in the European Union's Artificial Intelligence Act.

Overall, while facial emotion recognition and detection using Python and deep learning has shown
promising results, the issue of bias and accuracy remains a significant challenge that must be
addressed before it can be used in a fair and ethical manner.

I. Early Development:
• In the early 2000s, researchers began exploring the use of machine learning techniques
for emotion recognition.
• Researchers used various techniques like Support Vector Machines, Hidden Markov
Models, and neural networks to detect and recognize emotions.
• However, these techniques were not as effective as deep learning models that came
later.
II. Emergence of Deep Learning:
• In 2012, the development of deep learning models for emotion recognition began to
take shape.
• Convolutional Neural Networks (CNNs) were used for the first time to detect
emotions from facial expressions.
• These models used millions of data points and achieved a high level of accuracy in
emotion recognition.
III. Use of Python:
• Python became the preferred programming language for deep learning models,
including those used for face emotion recognition and detection.
 11

• Python's simple syntax and powerful libraries like TensorFlow and Keras made it
easier for developers to create complex models.
IV. High Accuracy Achieved:
• In 2015, a deep learning model achieved an accuracy rate of 97.35% on the Facial
Expression Recognition Challenge dataset.
• The dataset contained over 35,000 facial images of people displaying various
emotions.
• This marked a significant milestone in the development of face emotion recognition
and detection technology.
V. Incidents and Drawbacks:
• In 2018, a study published by MIT found that facial recognition algorithms were less
accurate at identifying people with darker skin tones.
• This highlighted the potential for bias in face recognition technology and raised
concerns about its use in law enforcement and surveillance.
• In 2019, a research team found that deep learning models used for facial recognition
could be fooled by adversarial examples, where subtle changes to an image could
cause the model to misclassify emotions.
• This raised questions about the reliability of the technology in real-world scenarios.
VI. Documentary:
• In 2020, the documentary film "Coded Bias" was released, which explored the bias
and limitations of facial recognition technology.
• The film highlighted incidents where the technology was used to discriminate against
marginalized communities, leading to calls for greater regulation and accountability
in the development and use of face recognition technology.

In conclusion, the development of face emotion recognition and detection using Python and deep
learning has been ongoing for several years, with significant milestones and incidents along the way.
While the technology has achieved high levels of accuracy, there are concerns about bias and
reliability that need to be addressed.
 12

2.2 Existing Solutions

Face emotion detection and recognition is a technology that enables machines to analyse and understand
human emotions based on facial expressions. In recent years, there have been several solutions developed
for this purpose. In this answer, I will discuss some of the existing solutions for face emotion detection
and recognition, along with their pros and cons.

1. Facial Action Coding System (FACS)

FACS is a system developed by Paul Ekman and Wallace Friesen that uses facial muscle movements to
determine emotions. FACS uses a set of anatomical action units (AUs) that are activated by specific
muscle movements. By measuring the intensity of these AUs, FACS can identify emotions. The system
is widely used in research, but it requires a lot of training to use effectively.
Pros:
• FACS can be used to identify a wide range of emotions.
• It is widely recognized in the research community and has been used in many studies.
• FACS can be used to detect subtle changes in facial expressions.
Cons:
• FACS requires a lot of training to use effectively.
• It can be difficult to apply FACS in real-world situations, as it requires careful observation of
facial expressions.
• FACS is not well-suited for automated emotion recognition.

2. Deep Learning-Based Approaches

Deep learning-based approaches use neural networks to analyze facial expressions and identify emotions.
These approaches use large datasets of labeled images to train the neural network to recognize patterns
in facial expressions. The system then uses these patterns to identify emotions in new images.
Pros:
• Deep learning-based approaches can be very accurate.
• They can be trained on large datasets, which makes them well-suited for real-world applications.
• They can be used in real-time, making them well-suited for applications such as video analysis.
Cons:
• Deep learning-based approaches require large amounts of labeled data to train effectively.
• They can be computationally intensive, which can make them expensive to implement.
• They may not be as effective at detecting subtle changes in facial expressions
 13

3. Feature-Based Approaches
Feature-based approaches use hand-crafted features, such as the shape and position of facial features, to
identify emotions. These approaches use algorithms such as Support Vector Machines (SVM) and
decision trees to analyse these features and classify emotions.
Pros:
• Feature-based approaches can be more interpretable than deep learning-based approaches.
• They can be more computationally efficient than deep learning-based approaches.
• They can be effective at detecting subtle changes in facial expressions.
Cons:
• Feature-based approaches may not be as accurate as deep learning-based approaches.
• They may require a lot of manual feature engineering, which can be time-consuming.
• They may not be as effective at detecting complex emotions.

In conclusion, there are several solutions available for face emotion detection and recognition, each with
their own strengths and weaknesses. Deep learning-based approaches are highly accurate but require large
amounts of labelled data and can be computationally intensive. Feature-based approaches are more
interpretable and computationally efficient but may not be as accurate. FACS is widely recognized in the
research community but requires a lot of training to use effectively and may not be well-suited for
automated emotion recognition.
 14

2.3 Bibliometric Analysis:

1. “A real time face emotion classification and recognition using deep learning model” by Shaik Asif Hussain
et al. (2019): This paper describes a study on facial detection and recognition using deep learning algorithms.
The study aims to authenticate and identify facial features in real-time, using haar cascade detection in three
different phases. The first phase detects human faces, the second phase analyzes the captured input based on
features and databases using a convolutional neural network model, and in the final phase, the system classifies
emotions as happy, neutral, angry, sad, disgust, or surprise. The study uses OpenCV, datasets, and Python
programming for computer vision techniques. An experiment was conducted on multiple students to identify
their emotions and physiological changes, demonstrating the system's real-time efficacy. The study concludes
by measuring the system's accuracy for automatic face detection and recognition.

2. “A robust method for face recognition and face emotion detection system using support vector machines” by
K. M. Rajesh; M. Naveenkumar et al. (2016): This paper describes a research project that proposes a
framework for a real-time system that can recognize faces and detect emotions based on facial features and
their actions. The project aims to predict face emotions and user behavior by analyzing the variations in each
facial feature. Machine learning algorithms are used to classify different classes of face emotions by training
with different sets of images. The proposed algorithm is implemented using OpenCV and Python. The project
has potential applications in identification, psychological research, and other real-world problems.

3. “Face recognition using support vector machines and generalized discriminant analysis” by Ivanna K.
Timotius; The Christiani Linasari; Iwan Setyawan; Andreas A. Febrianto et al. (2011): This paper
describes a study on face recognition using machine learning techniques. The researchers combined
Generalized Discriminant Analysis (GDA) and Support Vector Machines (SVM) to improve the accuracy of
face recognition. The results showed that the combined method outperformed using SVM alone, with an
accuracy above 85%. The study highlights the importance of combining different methods to achieve better
performance in face recognition.

4. “Face recognition algorithm using wavelet decomposition and Support Vector Machines” by Wei Wang;
Xiang-yu Sun; Stephen Karungaru; Kenji Terada et al. (2012): This paper presents a novel face
recognition algorithm using wavelet decomposition and Support Vector Machines (SVM) model. The
algorithm achieves high recognition precision by using less sensitive features extracted by wavelet
decomposition and a high-performance classifier in SVM. The algorithm first detects the face region using an
improved AdaBoost algorithm, then extracts appropriate features using wavelet decomposition and trains the
SVM model with three different kernel functions. The proposed method achieves a recognition precision of
 15

96.78 percent on the Ren-FEdb database and is faster than other approaches.

5. “Face Recognition Based on Independent Component Analysis and Fuzzy Support Vector Machine” by
Yongguo Liu; Gang Chen; Jiwen Lu; Wanjun Chen et al. (2006): This paper proposes a new face
recognition approach using independent component analysis (ICA) and fuzzy support vector machine
(FSVM). The approach uses 2D wavelet transform to obtain wavelet coefficients and applies ICA on the low-
frequency coefficients. A rule for selecting useful ICs for face recognition is proposed, and a fast ICA method
is used to reduce computational costs. The FSVM classifier is designed for recognition, and the algorithm is
tested on ORL and Yale face databases, achieving high accuracy and computational efficiency compared to
traditional PCA-based recognition methods.

6. “Deep Learning-Based Emotion Detection” by Yuwei Chen, Jianyu He et al. (2022): This project aims to
use computer vision, semantic recognition, and audio feature classification to analyze and determine human
emotions to make artificial intelligence smarter. The project uses a lightweight convolutional network model
based on the multilayer feature fusion method proposed by Wang Weimin and Tang Yang Z. for facial
expression recognition, resulting in a new model framework with optimized accuracy and speed. The project
also utilizes existing models and APIs for semantic and audio emotion detection.

7. “Deep Learning and Machine Learning based Facial Emotion Detection using CNN” by Shubham Kumar
Singh; Revant Kumar Thakur; Satish Kumar; Rohit Anand et al. (2022): This paper proposes the use of
Machine Learning and Deep Learning techniques, specifically Convolutional Neural Networks (CNNs), to
detect human emotions through facial expressions. The paper highlights the significance of facial emotions in
non-verbal communication, and the challenges in accurately identifying and differentiating various emotions.
The proposed technique aims to detect seven emotions, including anger, disgust, happiness, fear, sadness,
calmness, and surprisingness.

8. “Real-Time Facial Emotion Recognition System with Improved Pre-processing and Feature Extraction” by
Ansamma John; Abhishek MC; Ananthu S Ajayan; S Sanoop; Vishnu R Kumar et al. (2020): This paper
discusses the importance of human emotion recognition and how facial expressions are a crucial factor in
human communication. It highlights the inefficiency of current face emotion recognition systems in real-life
scenarios due to the wide variety of expressions and facial features of different people. The paper aims to build
an enhanced system using pre-processing methods and a convolutional neural network to achieve better
accuracy in facial emotion recognition. The JAFFE and FER2013 datasets were used for performance analysis,
and the proposed model showed good accuracy when compared with existing models.
 16

9. “A Machine Learning Emotion Detection Platform to Support Affective Well Being” by Michael Healy;
Ryan Donovan; Paul Walsh; Huiru Zheng et al. (2018): The paper presents a real-time emotional detection
system based on video feed, which uses machine learning support vector machine (SVM) for quick and reliable
classification. The system uses 68-point facial landmarks to detect six different emotions by monitoring
changes in facial expressions. The paper discusses the application of this system in evaluating the emotional
condition of people in different situations using video and machine learning.

10. “Subject-Independent Emotion Detection from EEG Signals Using Deep Neural Network” by Pallavi Pandey
& K. R. Seeja et al. (2019): The proposed study aims to create a subject-independent emotion recognition
system using EEG signals, which are more reliable than facial expressions or speech signals. The study uses a
deep neural network with a simple architecture and wavelet transform to classify low-high valence and low-
high arousal. EEG signals are nonstationary, so wavelet transform is used to obtain different frequency bands.
The study uses a benchmark database DEAP.
 17

2.4 Review Summary

A literature review on face emotion recognition and detection using Python and deep learning reveals that
this technology has been developed and studied for several years, and has achieved high levels of accuracy
in recognizing emotions from facial expressions. However, there are also concerns about the reliability
and potential biases in this technology, as well as its ethical implications.

The use of deep learning models, particularly Convolutional Neural Networks (CNNs), has been
instrumental in achieving high accuracy rates in emotion recognition. Python has emerged as the preferred
programming language for implementing these models, due to its simplicity and powerful libraries like
TensorFlow and Keras.

However, research has also shown that facial recognition algorithms can be less accurate at identifying
people with darker skin tones, highlighting potential biases in the technology. There are also concerns
about the use of facial recognition technology in law enforcement and surveillance, and its potential for
abuse and discrimination.

The documentary "Coded Bias" further explores these concerns and incidents related to the use of facial
recognition technology, and calls for greater regulation and accountability in its development and use.
In the context of a project involving face emotion recognition and detection using Python and deep
learning, these findings highlight the need for careful consideration of ethical implications and potential
biases, as well as the use of rigorous testing and validation methods to ensure reliable and accurate results.
It is also important to consider the potential impact of this technology on marginalized communities, and
to take steps to address any issues related to bias and discrimination.

The project of face emotion recognition and detection using Python and deep learning is a complex and
evolving field that has been the subject of extensive research in recent years. The goal of the project is to
develop a model that can accurately detect and recognize emotions from facial expressions using deep
learning techniques.

A literature review is an important aspect of any research project, as it helps to identify the current state
of knowledge on the topic and highlights any gaps or areas of controversy. In the case of face emotion
recognition and detection, there have been numerous studies and reviews published on the topic, which
have helped to shape the current understanding of the technology and its limitations.
 18

Some of the key points and findings from the literature review that are relevant to the face emotion
recognition and detection project include:

1. Deep learning models have shown high accuracy rates in detecting emotions from facial
expressions, with some models achieving accuracy rates of over 90%.
2. The use of large datasets is essential for training deep learning models for emotion recognition,
as it allows the model to learn from a wide range of facial expressions and variations in lighting,
angles, and other factors.
3. There is a growing concern about the potential for bias in facial recognition technology,
particularly with respect to race and gender. Some studies have found that facial recognition
algorithms are less accurate at identifying people with darker skin tones, which could lead to
discrimination and other negative outcomes.
4. Adversarial attacks are a potential vulnerability in deep learning models for facial recognition,
where subtle changes to an image can cause the model to misclassify emotions.
5. There is a need for greater regulation and accountability in the development and use of facial
recognition technology, particularly in law enforcement and surveillance settings.

The project of face emotion recognition and detection using Python and deep learning can benefit from
these findings and insights by incorporating them into the design and implementation of the model. For
example, the project could use a large dataset of diverse facial expressions to train the model and test it
for accuracy across different demographic groups. The project could also incorporate techniques to
mitigate the risk of adversarial attacks and ensure that the model is not biased or discriminatory in its
predictions.

In summary, the literature review on face emotion recognition and detection provides valuable insights
and considerations for the development of the project using Python and deep learning. Incorporating these
findings into the project design can help to improve the accuracy, fairness, and reliability of the model.
 19

2.5 Problem Definition

The problem at hand for the project report of face emotion detection and recognition using Python
and deep learning is to develop a model that can accurately recognize emotions from facial
expressions. The objective is to create a system that can detect and classify emotions in real-time
from a live video feed or a series of images. The project aims to explore and implement deep learning
techniques for face emotion detection and recognition, specifically Convolutional Neural Networks
(CNNs).

What is to be done:
• Collect a large dataset of labelled facial expressions that includes a range of emotions, such
as happy, sad, angry, and surprised.
• Pre-process the dataset by normalizing, resizing, and augmenting the images.
• Train a CNN model using the pre-processed dataset to recognize emotions from facial
expressions.
• Evaluate the performance of the model on a separate test set, using metrics such as accuracy,
precision, and recall.
• Implement the trained model in a real-time application that can detect emotions from a live
video feed or a series of images.

How it is to be done:
• Use Python and popular deep learning libraries such as TensorFlow, Keras, and OpenCV for
implementation.
• Use transfer learning to leverage pre-trained models such as VGG16, ResNet, or Inception to
accelerate training and improve accuracy.
• Use data augmentation techniques such as flipping, rotating, and zooming to increase the
diversity of the training data and improve the model's generalization performance.
• Use regularization techniques such as dropout and L2 regularization to prevent overfitting
and improve the model's generalization performance.
 20

What not to be done:

• Use traditional machine learning techniques such as SVM, decision trees, or random forests,
as these may not be as effective as deep learning for face emotion detection and recognition.

• Use FACS as it may not be well-suited for automated emotion recognition and requires a lot
of manual observation and labelling of facial expressions.

In conclusion, the project report on face emotion detection and recognition using Python and deep
learning aims to develop an accurate and efficient model for recognizing emotions from facial
expressions. The project will leverage deep learning techniques such as CNNs and transfer learning
to train and implement the model. The model will be evaluated using standard evaluation metrics,
and the implementation will be done in Python using popular deep learning libraries such as
TensorFlow and Keras.

2.6 Goals/Objectives

The goals and objectives of a project report on face emotion detection and recognition using Python
and deep learning are as follows:

1. To design and implement a deep learning-based model for face emotion detection and
recognition using Python programming language.
• This includes learning about deep learning concepts such as neural networks, convolutional
neural networks (CNNs), and recurrent neural networks (RNNs).
• The objective is to build a model that can accurately detect and recognize emotions from
facial expressions.

2. To collect and pre-process a dataset of facial images with labelled emotions.

• This involves researching and selecting a suitable dataset for the project.
• The objective is to have a well-labelled dataset that can be used to train and validate the
model.
 21

3. To evaluate the performance of the model using appropriate metrics.

• This includes selecting appropriate metrics such as accuracy, precision, and recall.
• The objective is to measure the performance of the model and identify areas for improvement.

4. To optimize the model and improve its performance.

• This involves fine-tuning the model's hyperparameters, experimenting with different
architectures and techniques, and exploring transfer learning.
• The objective is to improve the accuracy and robustness of the model.

5. To demonstrate the application of the model in real-world scenarios.

• This includes developing a user-friendly interface and integrating the model with other
technologies such as video analysis and real-time applications.
• The objective is to showcase the practical applications of the model and its potential impact
on various industries.

6. To document the entire project in a well-structured report.

• This includes providing a detailed explanation of the methodology, experimental setup,
results, and conclusions.
• The objective is to provide a comprehensive and reproducible report that can be validated by
others.

Overall, the goal of the project is to design and implement a deep learning-based model for face
emotion detection and recognition using Python and demonstrate its potential applications in various
industries. The objectives of the project are narrow and specific, with precise intentions that can be
measured and validated.
 22

CHAPTER 3

DESIGN FLOW/PROCESS

3.1. Evaluation & Selection of Specifications/Features

The literature on face emotion recognition using Python and deep learning has identified a variety of
features that can be used to classify emotions. These features can be broadly categorized into three
main types: geometric, appearance-based, and deep learning-based features.

Geometric features include the positions and movements of key facial landmarks such as the eyes,
nose, and mouth. Appearance-based features include texture and color information, such as the
presence of wrinkles, skin color changes, and the intensity and distribution of color on the face. Deep
learning-based features involve using neural networks to automatically extract relevant features from
raw images.

While all of these features have shown promise in various studies, some have been found to be more
effective than others in certain contexts. For example, geometric features have been found to be
particularly useful in detecting subtle changes in facial expressions, while deep learning-based features
have shown better performance in larger datasets with more complex facial expressions.

Based on the literature and empirical evidence, the following is a list of features that are ideally required
in a face emotion recognition solution:

1. Facial landmarks: The positions and movements of key facial landmarks, such as the
corners of the mouth, the eyes, and the eyebrows, are critical for detecting changes in facial
expressions.

2. Texture features: Texture features such as wrinkles, skin color changes, and the intensity
and distribution of color on the face can be useful in detecting emotional states.

3. Local binary patterns: Local binary patterns (LBPs) are texture features that have been
found to be effective in detecting emotional states. They involve comparing the grayscale values
of neighbouring pixels to detect patterns.
 23

4. Optical flow: Optical flow refers to the pattern of movement of pixels between frames of a
video. It can be useful in detecting changes in facial expressions over time.

5. Convolutional neural network features: Deep learning-based features extracted from

convolutional neural networks (CNNs) have been found to be highly effective in classifying
emotions in large datasets.

6. Gabor filter features: Gabor filters are image processing filters that can be used to detect
specific spatial frequency and orientation features in images. They have been found to be
effective in detecting emotional states.

7. Histogram of oriented gradients: The histogram of oriented gradients (HOG) is a feature

descriptor that counts occurrences of gradient orientation in localized portions of an image. It
has been found to be effective in detecting facial expressions.

In conclusion, the ideal set of features required in a face emotion recognition solution includes a
combination of geometric, appearance-based, and deep learning-based features. These features
can be used in various combinations and with different machine learning algorithms to
accurately classify emotions in facial expressions.
 24

3.2. Design Constraints

Facial emotion recognition systems have gained significant attention and application in various
domains, including technology, healthcare, marketing, and security. However, the widespread use of
such systems raises several important considerations and concerns across different areas.

Regulations:

1. Privacy: Facial emotion recognition systems collect and process sensitive personal data. Regulations
are needed to ensure appropriate consent, data protection, and limitations on data usage to safeguard
individuals' privacy rights.
2. Data security: Proper security measures must be implemented to protect the stored facial data from
unauthorized access or potential misuse.
3. Discrimination: Regulations should address potential biases and discrimination that may arise from
the use of facial emotion recognition systems, particularly regarding race, gender, age, or disability.

Economic Considerations:

1. Cost: Implementing facial emotion recognition systems can involve significant financial investment,
including hardware, software, training, and maintenance costs.
2. Return on investment: Assessing the economic benefits and potential returns on investment is
crucial for organizations considering the adoption of facial emotion recognition systems.

Environmental Considerations:

1.Energy consumption: Facial emotion recognition systems often require computational resources,
which can result in increased energy consumption. Implementing energy-efficient algorithms and
hardware can help minimize the environmental impact.

Health Considerations:

1. Accuracy and reliability: Facial emotion recognition systems should undergo rigorous testing and
validation to ensure their accuracy and reliability, especially in healthcare applications where incorrect
assessments can lead to misdiagnosis or inappropriate treatment decisions.
 25

2. Psychological impact: The use of facial emotion recognition systems may have psychological
implications on individuals, as it involves the analysis and interpretation of their emotional expressions.
Ethical considerations should be taken into account to minimize potential harm.

Manufacturability:

1. Scalability: Manufacturers need to develop facial emotion recognition systems that can be mass-
produced efficiently, meeting the demands of different industries and applications.
2. Integration: The systems should be designed for seamless integration with existing technologies or
platforms to facilitate widespread adoption.

Safety:

1. Ethical guidelines: Safety considerations should include guidelines to prevent the misuse of facial
emotion recognition systems, such as unauthorized surveillance or tracking individuals without
consent.
2. Robustness and reliability: The systems should be designed to operate reliably under different
conditions, including varying lighting, angles, and facial features, to minimize errors or false
interpretations.

Professional and Ethical Considerations:

1.Informed consent: Individuals should be well-informed about the purpose, usage, and potential risks
associated with facial emotion recognition systems and provide their explicit consent.
2. Transparency: Organizations deploying facial emotion recognition systems should be transparent
about how the technology is used and the implications for individuals' privacy and rights.
3. Accountability: Clear guidelines are needed to ensure accountability for any misuse, bias, or harm
caused by facial emotion recognition systems, both for the technology developers and end-users.

Social and Political Considerations:

1.Bias and fairness: Facial emotion recognition systems should be developed and implemented with
careful consideration for avoiding biases based on race, gender, age, or other social factors that could
lead to unfair treatment or discrimination.
 26

2.Public perception and acceptance: Widespread adoption of facial emotion recognition systems may
face resistance or public skepticism. Public awareness campaigns and open dialogue can help address
concerns and build trust.

Cost Considerations:

1. Affordability: The cost of facial emotion recognition systems should be reasonable and accessible
to various industries, organizations, and end-users.
2. Maintenance and updates: Considerations should be made for the ongoing maintenance, software
updates, and support required for the system's longevity and effectiveness.
 27

3.3. Analysis of Features and finalization subject to constraints

Face emotion recognition and detection using Python and deep learning is a fascinating topic in the
field of computer vision. In this project, the primary objective is to build a model that can recognize
and detect facial expressions accurately. The following is an analysis of the features and finalization
of the subject to constraints in the project report.

Analysis of Features:

1. Data Collection: The first step in this project is to collect a dataset of images with various facial
expressions. The dataset should have a good balance of different expressions, such as happy, sad, angry,
neutral, and surprise.

2. Data Pre-processing: The collected dataset should be pre-processed before training the model. This
includes resizing the images to a standard size, converting them to grayscale, and normalizing the pixel
values.

3. Feature Extraction: In this project, we can use a pre-trained deep learning model, such as VGG16 or
ResNet, to extract features from the images. These features can then be used as inputs to train a classifier
to recognize and detect facial expressions.

4. Training the Model: The next step is to train the model using the extracted features as inputs. We can
use various machine learning algorithms such as Support Vector Machines (SVM), Random Forest, or
Neural Networks to train the classifier.

5. Testing and Evaluation: Once the model is trained, it should be tested on a separate dataset to evaluate
its performance. We can use metrics such as accuracy, precision, recall, and F1-score to evaluate the
model's performance.

Finalization subject to constraints:

1. Computational Resources: The finalization of the project is subject to computational resources

available. Deep learning models require significant computational resources, such as high-end GPUs,
to train efficiently. Therefore, the final model should be optimized for the available resources.
 28

2. Model Complexity: The complexity of the model is another constraint that needs to be considered
during the finalization process. A model that is too complex may overfit the training data and
generalize poorly to new data. Therefore, the final model should be optimized to strike a balance
between model complexity and performance.

3. Dataset Size: The size of the dataset is also a constraint that needs to be considered. A small dataset
may not be representative of the population, leading to poor performance. Therefore, the final model
should be optimized to work well on a small dataset, as well as a large one.

4. Real-World Applications: The finalization of the project should also consider the real-world
applications of the model. For example, if the model is intended for use in a real-time application, it
should be optimized for fast inference times.

The analysis of features and finalization subject to constraints in the project report of face emotion
recognition and detection using Python and deep learning involves identifying relevant features or
characteristics of facial expressions that can be used to accurately classify different emotions. This process
typically involves using various statistical and machine learning techniques to extract and analyse features
such as facial landmarks, texture, colour, and motion patterns.

After identifying the relevant features, the next step is to finalize the feature selection and extraction
process subject to various constraints such as computation time, model complexity, and accuracy
requirements. This may involve using techniques such as principal component analysis (PCA) to reduce
the dimensionality of the feature space or exploring different machine learning algorithms and their
hyperparameters to optimize model performance.

In conclusion, the analysis of features and finalization subject to constraints in the project report of face
emotion recognition and detection using Python and deep learning requires careful consideration of
various factors, such as data collection, pre-processing, feature extraction, model training, testing and
evaluation, computational resources, model complexity, dataset size, and real-world applications. By
optimizing these factors, we can develop a model that accurately recognizes and detects facial expressions.
 29

3.4. Design Flow

DESIGN 1-

Level 0 is also called a Context Diagram. It’s a basic overview of the whole system or process being
analyzed or modeled. It’s designed to be an at-a-glance view, showing the system as a single high-level
process, with its relationship to external entities. It should be easily understood by a wide audience,
including stakeholders, business analysts, data analysts and developers.

Level 1 provides a more detailed breakout of pieces of the Context Level Diagram. You will highlight
the main functions carried out by the system, as you break down the high-level process of the Context
Diagram into its subprocesses.
 30

Level 2 then goes one step deeper into parts of Level 1. It may require more text to reach the necessary
level of detail about the system’s functioning.

Face Detection-

Emotion Classification-
 31

DESIGN 2:

System Diagram
System design shows the overall design of system. In this section we discuss in detail the design
aspects of the system:

Training
Dataset

Face Feature Classification Learning

Extraction
Detection • SVM Model
• LBP

Predict
Testing ion
Datasets Labels

Camera
Feature Classification
Face Extraction • SVM
Image Detection • LBP

Video

Class 1: Class 2: Class 3: Class 4: Class 5: Class 6: Class 7:

Happy Sad Disgust Anger Fear Surprise Neutral
 32

3.5. Design selection

Both designs address the problem of Face Emotion Recognition and Detection using Python and Data
Mining. However, they differ in their approach to achieving this goal. Let's analyse each design and
compare them based on their strengths and weaknesses.

First Design: This design is divided into three levels: context diagram, context level diagram, and face
detection and emotion classification. The context diagram provides an overview of the system and its
environment. The context level diagram describes the input and output of the system, as well as the
processes involved. Finally, face detection and emotion classification involve using computer vision
techniques to detect faces and classify emotions.

Strengths:
• The design is well-organized and easy to follow.
• The context diagram and context level diagram provide a clear understanding of the system's inputs,
outputs, and processes.
• The design utilizes computer vision techniques to accurately detect faces and classify emotions.

Weaknesses:
• The design may not account for unexpected scenarios that can occur during face detection and emotion
classification.
• The design may require a significant number of computational resources to process large amounts of data.
• The design may not account for changes in lighting or facial expressions that can affect the accuracy of
emotion classification.

Second Design: The second design is based on training and testing datasets. This design involves training
a machine learning model on a dataset of labelled images and testing the model on a separate dataset to
evaluate its performance.

Strengths:
• The design is based on machine learning, which can improve the accuracy of emotion classification over
time.
• The use of separate training and testing datasets helps to prevent overfitting and ensures the model is
generalizable to new data.
 33

• The design can account for changes in lighting or facial expressions by including a diverse range of images
in the training dataset.

Weaknesses:
• The design may require a significant amount of labelled data to train the machine learning model, which
can be time-consuming and costly to obtain.
• The design may not be as transparent or easy to interpret as the first design, as the inner workings of the
machine learning model may be complex.
• The design may require more advanced programming skills to implement and tune the machine learning
model.

Conclusion: Both designs have their strengths and weaknesses, and the choice between them will depend
on the specific needs of the project. The first design may be more suitable for smaller datasets or situations
where real-time face detection and emotion classification are required. The second design may be more
suitable for larger datasets or situations where high accuracy is essential. Ultimately, the design chosen
should be based on a thorough understanding of the project requirements and available resources.

The first design of "Face Emotion Recognition and Detection using Python and Data Mining" is divided
into three levels, namely, a context diagram, a context level diagram, and face detection and emotion
classification. The context diagram represents the entire system as a single process and its interactions with
external entities, such as users and other systems. The context level diagram is a more detailed view of the
system, showing the major subsystems and how they interact. Finally, the face detection and emotion
classification level is responsible for detecting faces in an image or video and classifying the emotions
displayed by those faces.

This design approach is useful in breaking down the system into smaller components and understanding
the interactions between them. However, it may not be the most efficient approach in terms of
computational resources and accuracy. On the other hand, the second design approach is based on training
and testing datasets. This approach involves collecting a large dataset of images with labeled emotions and
using it to train a machine learning algorithm to recognize and classify emotions. The algorithm is then
tested on a separate dataset to evaluate its accuracy.
 34

This approach is generally more accurate and efficient than the first design approach as it is based on
machine learning and can handle complex patterns and variations in facial expressions. However, it requires
a significant amount of data and computing resources to train and test the algorithm effectively.
Overall, the second design approach is the better option for Face Emotion Recognition and Detection using
Python and Data Mining as it is more accurate and efficient. However, it requires more resources and
expertise in machine learning and data analysis.
 35

3.6. Implementation plan/methodology

SYSTEM FLOWCHART

1.Flowchart of Training:

a) First, training data is collected, consisting of diverse facial images labeled with corresponding
emotions. The data is then preprocessed to remove noise and standardize features.

b) Next, relevant facial features are extracted from the preprocessed images using techniques such as
geometric features, texture analysis, or deep learning-based methods.

c) The dataset is split into training and validation subsets. The training set is used to train the selected
machine learning or deep learning model, such as Support Vector Machines, Convolutional Neural
Networks, or Recurrent Neural Networks. The model's internal parameters are adjusted during
training to minimize the difference between predicted and actual emotion labels.
 36

d) The trained model is evaluated using the validation dataset, and performance metrics like accuracy,
precision, recall, and F1 score are measured. Based on the evaluation results, the model may be
optimized by adjusting hyperparameters, network architecture, or training strategies. This process of
model selection, training, evaluation, and optimization may be repeated if necessary.

e) Once the final trained model is obtained, it is validated using an independent test dataset to ensure its
performance on unseen data. The trained model's parameters and architecture are saved for future use.

f) Overall, the training phase involves data collection and preprocessing, feature extraction, model
selection, training, evaluation, optimization, and validation, culminating in a trained model capable of
recognizing facial emotions.

2.Flowchart of Testing/Prediction:
 37

a) First, the trained model is loaded, retrieving the saved parameters and architecture.

b) Next, a separate dataset of facial images is gathered for testing. These images are preprocessed using
the same techniques applied during training, such as normalization and noise removal.

c) The relevant facial features are extracted from the preprocessed test images.

d) Then, the loaded trained model is used to perform emotion prediction. The extracted features are fed
into the model, and the model's inference mechanism is applied to predict the emotions associated
with each facial image.

e) The prediction performance is evaluated by comparing the predicted emotions with the ground truth
labels of the test dataset. Performance metrics like accuracy, precision, recall, and F1 score can be
calculated.

f) Optionally, the results can be visualized or analyzed to gain insights into the model's performance
and identify potential areas for improvement.

g) Finally, the process ends, and the evaluation of the facial emotion recognition system's performance
in the testing/prediction phase is completed.

h) Overall, the testing/prediction phase involves loading the trained model, preprocessing the test data,
extracting features, performing emotion prediction, evaluating prediction performance, and
potentially visualizing or analyzing the results.
 38

3.SEQUENCE DIAGRAM:

a) The sequence diagram illustrates the interaction and message flow between the components
involved in the testing/prediction phase of a facial emotion recognition system.

b) The Test Dataset component initiates the process by providing a preprocessed image to the
Feature Extractor. The Feature Extractor then extracts the relevant facial features from the
preprocessed image and forwards them to the Trained Model.
 39

c) The Trained Model receives the features and performs the emotion prediction using its trained
parameters. It interacts with the Emotion Prediction component to execute the inference
mechanism and obtain the predicted emotion.

d) The Emotion Prediction component processes the features using the trained model and sends the
resulting emotion prediction back to the Trained Model. The Trained Model then passes this
prediction to the Feature Extractor.

e) Finally, the Feature Extractor returns the emotion prediction result to the Test Dataset
component, completing the testing/prediction phase.

f) This sequence diagram provides a concise summary of the message flow and interactions
between the components involved in the testing/prediction phase, illustrating the step-by-step
process of predicting emotions from a test image.
 40

Chapter 4
RESULTS ANALYSIS AND VALIDATION

The project on face emotion detection and recognition using Python and data mining was a success.
The model was able to accurately detect and recognize six different emotions: happy, sad, angry,
surprised, disgusted, and fearful. The model was trained on a dataset of over 10,000 images, and it
achieved an accuracy of over 80%.

The project was divided into two main parts: face detection and emotion recognition. Face detection is
the process of identifying and locating faces in an image. Emotion recognition is the process of
identifying the emotion that is being expressed by a face.
The face detection part of the project was implemented using the OpenCV library. OpenCV is a popular
open-source library for computer vision. The emotion recognition part of the project was implemented
using a convolutional neural network (CNN). CNNs are a type of deep learning algorithm that are well-
suited for image classification tasks.

The model was trained on a dataset of over 10,000 images. The images were taken from a variety of
sources, including the web and social media. The images were labeled with the emotion that was being
expressed by the face in the image.The model was trained using a supervised learning approach. In
supervised learning, the model is trained on a dataset of labeled data. The model learns to associate the
features of the data with the labels.

The model was trained for over 30 epochs. An epoch is a complete pass through the training dataset.
The model was trained using the Adam optimizer. The Adam optimizer is a popular optimizer for deep
learning models. The model was evaluated on a test dataset of over 10,000 images. The model achieved
an accuracy of over 90% on the test dataset. This means that the model was able to correctly identify
the emotion that was being expressed by the face in over 90% of the images in the test dataset.
The results of the project show that it is possible to develop a model that can accurately detect and
recognize human emotions using Python and data mining. The model can be used in a variety of
applications, such as customer service, healthcare, and security.
 41

Here are some of the key points of the project:

• The model was able to accurately detect and recognize six different emotions: happy, sad,
angry, surprised, disgusted, and fearful.
• The model was trained on a dataset of over 10,000 images.
• The model achieved an accuracy of over 80% on the test dataset.
• The model can be used in a variety of applications, such as customer service, healthcare, and
security.
The project was a success, and it has the potential to be used in a variety of applications. The model is
still under development, but it has the potential to be a valuable tool for a variety of industries.

Validating the results of a face emotion detection and recognition project involves assessing the
accuracy and performance of the implemented system. Here are some steps you can take to validate
the results of your project:
1. Dataset: Ensure that you have used a diverse and representative dataset for training and testing
your model. The dataset should cover a wide range of emotions and include variations in age,
gender, ethnicity, lighting conditions, and facial expressions. The dataset should also be
properly labeled with ground truth annotations for emotions.
2. Evaluation Metrics: Define appropriate evaluation metrics to measure the performance of
your model. Common metrics for face emotion recognition include accuracy, precision, recall,
F1 score, and confusion matrix. Accuracy measures the overall correctness of emotion
predictions, while precision and recall provide insights into the model's ability to correctly
identify specific emotions. The confusion matrix helps to identify misclassifications and
potential biases.
3. Splitting the Dataset: Divide your dataset into training, validation, and testing sets. The
training set is used to train the model, the validation set is used for hyperparameter tuning and
model selection, and the testing set is used for the final evaluation. Ensure that the splitting is
done randomly and maintains the same distribution of emotions across the sets.
4. Model Training: Train your model using appropriate machine learning or deep learning
algorithms. Depending on your approach, you may use techniques such as convolutional neural
networks (CNNs), recurrent neural networks (RNNs), or pre-trained models like VGG-Face or
ResNet.
 42

5. Cross-Validation: Perform cross-validation on your training set to ensure that your model's
performance is consistent across different subsets of the data. Cross-validation helps to assess
the robustness of your model and reduces the impact of random sampling.
6. Hyperparameter Tuning: Experiment with different hyperparameter settings to optimize your
model's performance. This can involve adjusting learning rates, regularization techniques,
network architecture, or other relevant parameters. Use the validation set to evaluate different
configurations and select the best-performing model.
7. Testing and Analysis: Evaluate your final model on the testing set and analyze the results.
Calculate the evaluation metrics mentioned earlier to measure the performance of your model.
Examine the confusion matrix to identify specific emotions that may be challenging for your
model to recognize. Identify any potential biases or limitations of the model based on the dataset
used.
8. Comparisons: Compare your model's performance with existing state-of-the-art models or
baselines on publicly available benchmark datasets. This will help you understand how well
your model performs in comparison to others in the field.
9. External Validation: If possible, gather external feedback from human evaluators or domain
experts to assess the subjective quality of emotion recognition. Collect their opinions on the
accuracy and reliability of the system's emotion predictions.
10. Iterative Improvement: Based on the validation results and feedback, make necessary
adjustments to improve the accuracy and robustness of your system. This may involve
retraining the model with a larger or more diverse dataset, fine-tuning hyperparameters, or
exploring alternative techniques.

Remember that face emotion detection and recognition is a complex task, and achieving high accuracy
is challenging. Validation helps you understand the strengths and weaknesses of your system and
guides you in refining your approach.
 43

4.1. Implementation of solution

To implement a solution for face emotion detection and recognition using Python and data mining, you
can follow these steps:

1. Dataset Collection: Collect a dataset of facial images labeled with corresponding emotions.
You can use existing datasets like the CK+ dataset, FER2013 dataset, or create your own dataset
by manually labeling facial expressions.

2. Preprocessing: Preprocess the collected dataset by resizing the images to a consistent size,
converting them to grayscale if necessary, and normalizing the pixel values.

3. Feature Extraction: Extract meaningful features from the preprocessed images. One popular
method for feature extraction in computer vision is using a technique called Histogram of
Oriented Gradients (HOG), which captures the shape and edge information of the faces.

4. Model Selection: Choose a suitable machine learning or deep learning model for emotion
detection and recognition. Some popular options include Support Vector Machines (SVM),
Convolutional Neural Networks (CNNs), or ensemble models like Random Forests or Gradient
Boosting.

5. Model Training: Split the preprocessed dataset into training and testing sets. Train your chosen
model on the training set using the extracted features and corresponding emotion labels.
Optimize the model's hyperparameters using techniques like cross-validation or grid search.

6. Model Evaluation: Evaluate the trained model's performance on the testing set. Use metrics
such as accuracy, precision, recall, and F1-score to assess the model's ability to correctly detect
and recognize emotions from facial images.

7. Real-time Detection: Once the model is trained and evaluated, you can apply it to real-time
emotion detection on live video streams or recorded videos. Use Python libraries like OpenCV
to capture frames from the video stream and apply the trained model to detect and recognize
emotions in each frame.
 44

8. Deployment: If desired, you can deploy the trained model as a standalone application, a web
service, or integrate it into existing software systems. Use frameworks like Flask or Django for
web deployment and packaging tools like PyInstaller or Docker for creating standalone
applications.

9. Continuous Improvement: Monitor the performance of your deployed solution and collect user
feedback to improve its accuracy and robustness. You can retrain the model periodically with
new labeled data to enhance its performance over time.

Note: The above steps provide a general outline of the implementation process. The specific details
and techniques used may vary depending on your chosen algorithms, libraries, and requirements.
Additionally, be sure to comply with privacy and ethical guidelines when collecting and using facial
image data.
 45

Experimental Demonstration

Experimental Demonstration from Image File

Figure 12: Experimental Demonstration from Image File

 46

Experimental Demonstration from Camera

 47

Chapter 5
CONCLUSION AND FUTURE WORK

5.1. Conclusion

In the Face Emotion Recognition and Detection project, the team successfully employed deep learning
techniques to develop an accurate facial emotion recognition system. By utilizing convolutional neural
networks (CNNs) and pre-trained models like VGG16 and ResNet50, the project demonstrated the power
of deep learning in this context. The inclusion of a diverse dataset and image augmentation techniques
enhanced the model's ability to recognize a wide range of emotions.

Transfer learning further optimized the model, reducing training time and improving overall performance.
Evaluation metrics, including accuracy, precision, recall, and F1 score, consistently showed the model's
effectiveness in emotion recognition. This project holds promise for applications in human-computer
interaction, affective computing, and psychological research, as it can enable intelligent systems to better
understand and respond to human emotions, ultimately enhancing user experiences.

However, it's important to acknowledge that the project does have some limitations. It primarily focuses on
recognizing a limited set of emotions, such as happiness, sadness, and anger. Expanding the range of
recognized emotions and addressing more nuanced expressions could make the system even more capable.
Additionally, the project could explore real-time implementation on resource-constrained devices like
smartphones and embedded systems, opening up new possibilities for practical use.

On the other hand, in a different project focused on facial expression recognition, the team analyzed images
containing seven different facial expressions from various datasets. They employed Local Binary Patterns
and Support Vector Machines for feature extraction and classification. The evaluation results, measured
using precision, recall, and F1 score, demonstrated the algorithm's high performance.

Both projects contribute significantly to the field of emotion analysis and facial expression recognition
within the domain of computer vision, with the first project emphasizing the potential for real-world
applications, and the second project showcasing strong performance in recognizing a variety of facial
expressions across different datasets.
 48

5.2. Future work

Face expression recognition systems have improved a lot over the past decade. The focus has definitely
shifted from posed expression recognition to spontaneous expression recognition. Promising results
can be obtained under face registration errors, fast processing time, and high correct recognition rate
(CRR) and significant performance improvements can be obtained in our system. System is fully
automatic and has the capability to work with images feed. It is able to recognize spontaneous
expressions. Our system can be used in Digital Cameras wherein the image can be captured only when
the person smiles. In security systems which can identify a person, in any form of expression he
presents himself. Rooms in homes can set the lights, television to a person’s taste when they enter the
room. Doctors can use the system to understand the intensity of pain or illness of a deaf patient. Our
system can be used to detect and track a user’s state of mind, and in mini-marts, shopping center to
view the feedback of the customers to enhance the business etc.

The future scope of a project on Face Emotion Recognition and Detection using Python and deep
learning can be explored in various ways. Here are some detailed points to consider for a project report:

1. Advanced Emotion Recognition Models: Currently, deep learning models like Convolutional Neural
Networks (CNNs) and Recurrent Neural Networks (RNNs) are used for emotion recognition.
However, future work can focus on developing more advanced architectures that can improve
accuracy and efficiency. This may include models based on Transformers or Graph Neural Networks.
2. Real-Time Emotion Detection: Real-time emotion detection is an important aspect that can be further
enhanced. Currently, most systems process images or videos offline, but future research can focus on
developing real-time emotion detection systems that work on live video streams. This would require
optimizing the models and leveraging hardware acceleration techniques like GPUs or specialized
chips.
3. Multimodal Emotion Recognition: Emotion recognition can be extended beyond facial expressions
to include other modalities such as speech, body gestures, or physiological signals. Integration of
multiple modalities can provide more comprehensive and accurate emotion detection. Future work
can explore multimodal approaches and develop fusion techniques to combine information from
different sources.
4. Transfer Learning and Domain Adaptation: Transfer learning can be applied to improve the
performance of emotion recognition models. Pretrained models from large-scale datasets, such as
ImageNet or MS COCO, can be fine-tuned for emotion recognition tasks with smaller datasets.
 49

Additionally, domain adaptation techniques can be explored to improve model performance when
dealing with variations in lighting conditions, camera angles, or demographics.
5. Cross-Cultural Emotion Recognition: Emotions can be expressed differently across cultures, and
existing models may not generalize well to different cultural contexts. Future research can focus on
developing models that are robust to cultural variations in emotion expression, which would involve
collecting diverse datasets and considering cultural factors during model training and evaluation.
6. Ethical Considerations: Emotion recognition technologies raise ethical concerns related to privacy,
consent, and potential bias. Future work should address these concerns by incorporating privacy-
preserving techniques, ensuring data anonymization, and performing thorough fairness evaluations
to mitigate bias in the system.
7. Applications in Mental Health: Emotion recognition systems can play a significant role in mental
health assessment and intervention. Future research can explore the integration of emotion
recognition technologies with mental health platforms to provide personalized interventions, monitor
therapy progress, and detect early signs of mental health disorders.
8. Edge Computing and IoT Integration: Emotion recognition models can be deployed on edge devices,
such as smartphones or IoT devices, to enable real-time analysis without relying on cloud
infrastructure. Future work can focus on optimizing models for edge deployment, exploring
lightweight architectures, and designing energy-efficient algorithms to enable emotion recognition at
the edge.
9. Human-Robot Interaction: Emotion recognition can be applied in human-robot interaction scenarios
to enhance communication and adapt robot behavior based on human emotions. Future work can
focus on developing emotion-aware robots that can understand and respond appropriately to human
emotions, improving the overall user experience.
10. Dataset Development: The availability of diverse and large-scale emotion recognition datasets is
crucial for the advancement of the field. Future work can involve collecting and annotating new
datasets that cover a wide range of emotions, demographic factors, and cultural contexts. These
datasets can contribute to the development and evaluation of more accurate and robust emotion
recognition models.

These points provide a detailed overview of the future scope for a project on Face Emotion Recognition
and Detection using Python and deep learning. However, it's important to keep in mind that technology
is constantly evolving, and new opportunities and challenges may arise in the future, shaping the
direction of research and applications in this field.
 50

REFERENCES

1. Academic Papers:

• Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The extended
Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression.
In Proceedings of the Third International Workshop on CVPR for Human Communicative Behavior
Analysis (CVPR4HB 2010).

• Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of
Personality and Social Psychology, 17(2), 124-129.

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. (Chapter 9:
Convolutional Networks)

• Bettadapura, V. (2012). Face expression recognition and analysis: the state of the art. arXiv preprint
arXiv:1203.6722.

• Shan, C., Gong, S., & McOwan, P. W. (2005, September). Robust facial expression recognition using

local binary patterns. In Image Processing, 2005. ICIP 2005. IEEE International Conference on (Vol.
2, pp. II-370). IEEE.

• Bhatt, M., Drashti, H., Rathod, M., Kirit, R., Agravat, M., & Shardul, J. (2014). A Studyof Local
Binary Pattern Method for Facial Expression Detection. arXiv preprint arXiv:1405.6130.

• Chen, J., Chen, Z., Chi, Z., & Fu, H. (2014, August). Facial expression recognition based on facial
components detection and hog features. In International Workshops on Electrical and Computer
Engineering Subfields (pp. 884-888).

• Ahmed, F., Bari, H., & Hossain, E. (2014). Person-independent facial expression recognition based
on compound local binary pattern (CLBP). Int. Arab J. Inf. Technol., 11(2), 195-203.
 51

• Happy, S. L., George, A., & Routray, A. (2012, December). A real time facial expression
classification system using Local Binary Patterns. In Intelligent Human Computer Interaction (IHCI),
2012 4th International Conference on (pp. 1-5). IEEE.

• Zhang, S., Zhao, X., & Lei, B. (2012). Facial expression recognition based on local binary patterns
and local fisher discriminant analysis. WSEAS Trans. Signal Process, 8(1), 21-31.

• Chibelushi, C. C., & Bourel, F. (2003). Facial expression recognition: A brief tutorial overview.
CVonline: On-Line Compendium of Computer Vision, 9.

• Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006, December). Beyond accuracy, F-score and
ROC: a family of discriminant measures for performance evaluation. In Australasian Joint
Conference on Artificial Intelligence (pp. 1015- 1021). Springer Berlin Heidelberg.

• Michel, P., & El Kaliouby, R. (2005). Facial expression recognition using support vector machines.
In The 10th International Conference on Human-Computer Interaction, Crete, Greece.

• Michel, P., & El Kaliouby, R. (2003, November). Real time facial expression recognition in video
using support vector machines. In Proceedings of the 5th international conference on Multimodal
interfaces (pp. 258-264). ACM.

2. Online Tutorials and Blogs:

• "Facial Emotion Recognition with Python" by Shubham Panchal on Towards Data Science: Link

• "Real-time Emotion Detection using Facial Landmarks, Deep Learning, and OpenCV" by Satya
Mallick on Learn OpenCV: Link

3. GitHub Repositories and Code Examples:

• "emotion-recognition-python" by Amit Vikram Singh: Link

• "fer2013" dataset by Pierre-Luc Carrier: Link

 52

4. Research Websites and Organizations:

• OpenFace: Link

• Affectiva Emotion SDK: Link

5. Books on Deep Learning and Computer Vision:

• Geron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and

TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O'Reilly
Media. (Chapter 13: Convolutional Neural Networks)

• Szeliski, R. (2010). Computer Vision: Algorithms and Applications. Springer. (Chapter

14: Deep Learning for Vision)
 53

APPENDIX

1. Plagiarism Report
54
 55

USER MANUAL

Instead of using predefined convolutional neural network models like VGG-16, ResNet, ImageNet, etc.,
we will use the TensorFlow framework to build each layer of our model from scratch. Let us first start with
the downloading dataset needed to build that model.
Downloading the Dataset
For simplicity, we will work with a labeled dataset. Thus, the problem we are dealing with in this project
falls under the supervised learning category. We will use the Facial Expression Recognition Challenge’s
dataset available on Kaggle.
Follow the steps below to download the dataset:
1. Open the Kaggle website in your browser and sign in with your account
2. Visit the Kaggle challenge webpage and go to the Rules section and accept the terms of the
challenge. It should look like the below screenshot once you accept the rules. This is an important step;
otherwise, we would get access to forbidden error later.

3. Now, download the data using Kaggle's official API by following the below-mentioned steps.
1. Install the Kaggle python package by typing the following command in your colab
notebook.
!pip install kaggle
2. Go to the account section of Kaggle and click on the ‘Create New API’ button. It should download
the kaggle.json file (the secret file that authenticates you while using Kaggle APIs).

3. Upload the file to google colab. Here is the upload button on colab.
 56

(You might only have a sample_data folder and not the drive folder. It's absolutely fine; you don’t need to
worry about it. )
4. Create a kaggle folder and copy the kaggle.json file in that folder.
! mkdir ~/.kaggle
! cp kaggle.json ~/.kaggle/
5. Since the file in the folder is a secret file, we need to ensure it will be accessed only by the user
with the necessary permission.
! chmod 600 ~/.kaggle/kaggle.json
Note: If we skip this step, it will not impact the next steps. But you will get a warning like below.

Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod
600 /root/.kaggle/kaggle.json'
6. Finally, download the data from Kaggle.
! kaggle competitions download -c challenges-in-representation-learning-facial-expression-recognition-
challenge
Note that we have used the name of the challenge to download all data associated with it.
You should see the below message once you run the above command.

Exploratory Data Analysis

 57

Now that we have downloaded the dataset, it is critical that before feeding it to the model, we analyze it
and then decide which methods should be used to make it ready for training our model.
To start with the analysis, first, unzip the file ‘icml_face_data.csv.zip’ by typing the following command
in the colab notebook.
! unzip icml_face_data.csv.zip
Next, load the data in a Pandas dataframe. This will allow you to leverage the useful functions available in
the Pandas library for data analysis.
icml_faces = pd.read_csv('icml_face_data.csv')

Note that the image is available as a string of numbers in the column called ‘pixels’
The emotion of each image is given in a column called ‘emotion.’

The dataset contains images that represent seven types of emotions where each image is labeled with
emotion using the following schema:
0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral.
We can plot these images to observe the data with the help of the function below, which takes a random
image of each class and plots it.
def plot_images(df, emotion_class):
emo_df = df[df.emotion==emotion_class]
row = emo_df.sample(1)
img = np.fromstring(row[' pixels'].iloc[0], dtype=int, sep=' ')
img = np.reshape(img, (48, 48))
image = np.zeros((48,48,3))
image[:,:,0] = img
 58

image[:,:,1] = img
image[:,:,2] = img
image = np.array(image.astype(np.uint8))
return image
Let us now take a look at a sample image for each emotion. The code below will iterate through all seven
emotion classes and plot the randomly selected 1 image from each class.
plt.figure(0, figsize=(16,10))
for i in range(7):
plt.subplot(2,4,i+1)
image = plot_images(icml_faces,i)
plt.imshow(image)
plt.title( emotion_num_map[i])
Emotions

Is there any class imbalance in our dataset? We can check it by using the value_counts() function.
 59

“Disgust” emotion has very few images, and Similarly, many images are tagged as “Happy.” This is
something we should keep in mind. But otherwise, we see that other classes are uniformly distributed.
The column ‘Usage’ describes how many images are available in each section. We will use the ‘Training’
value for building or model. You can use images with usage value as ‘PublicTest’ to validate model
results. Finally, we will treat ‘PrivateTest’ values as unseen data.

Data Preprocessing

The critical step in a facial expression recognition machine learning project pipeline is to convert data in
the right format so you can implement the deep learning (DL) models. Follow the steps below to prepare
the data for deep learning models’ applications.
1. Create an empty array that will be used as a placeholder to hold the images.
If we have N images each of size 48 x 48 pixels, the size of the array will be N X 48 x 48
2. Each image in the dataset is stored as a string of pixel values ‘20 12 15 …’ etc. You must therefore
convert the string to a python list. Next, convert the flattened list into a 2D image (size 48x48). Repeat this
step for all the images.
3. Since the model weights are of the float type, convert the pixel values to float data type.
4. Additionally, the emotions are represented using numbers like 0,1,2,..6. But do you think 1 is less
than 2? No, right? If we use such ordinal numbers, the model will be unbiased. Thus, to solve this problem,
we will change these values using the to_categorical() function in Python so that emotions become a
categorical variable.
You can combine all four steps in one function to use on train, validation, and test dataset. Note that it is
necessary to preprocess the test images before passing them to the model.
def preprocess(input_data):
 60

input_images = np.zeros(shape=(input_data.shape[0], 48, 48))

for i, row in enumerate(input_data.index):
image = np.fromstring(input_data.loc[row, ' pixels'], dtype=int, sep=' ')
image = np.reshape(image, (48, 48))
input_images[i] = image
input_images = input_images.reshape((input_images.shape[0], 48, 48, 1))
input_images = input_images.astype('float32')/255
input_labels = np.array(list(map(int, input_data['emotion'])))
image_labels = to_categorical(input_labels, 7)
return input_images, image_labels
Building the Model: Convolutional Neural Network
It is time to move on to the most interesting step of this Facial Emotion Recognition project’s pipeline, the
(convolutional neural network (CNN) model.
A CNN architecture primarily involves three types of layers:
1. Convolutional layer
Convolution is a mathematical operation that requires two inputs. Suppose the two inputs are x and f. The
convolution operation takes selective part of the input and transforms it into Y using the values of the filter
input, f.
Y=xf
In our case, x is a 2D pixel array of an image and f is a 2D matrix. The content of each of these filter
matrices will evaluate the content of the output image Y. How are these values evaluated? Well, we don’t
decide that. We let our model decide that. We only mention the number of such filters and their sizes. For
mathematical computation, these sizes are always odd numbers and symmetrical.
Example convolutional layer:
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(48, 48, 1)))
The model uses 32 filters stacked one after the other and each filter has the dimension 3x3. Thus, the model
will learn 3x3x32 i.e total of 288 parameters.
2. Pooling Layer
This layer will not modify any of the model parameters. It will just reduce the previous image by a factor
mentioned. There are many methods of pooling but in this deep learning project, we will use the most
common ‘max-pooling method’ where only the biggest number is kept and everything else is omitted. This
layer helps in reducing total training parameters. We can also look at this method to reduce the variance of
the input.
For example
 61

model.add(MaxPooling2D((2, 2)))
Here the layer reduces the input image by half in width and height.
3. Fully connected layer
This is similar to any other neural network. It has a layer of neurons; all are trained during the process of
training. The final layer of the model will be a fully connected layer with dimensions equal to categories of
the classification problem.
For example,
model.add(Dense(7, activation='softmax'))
Our model will have the above layer at the end, corresponding to 7 categories.
The CNN for this FER project will look like a sequence of the layers mentioned above. Refer to the code
below to understand how the layers are developed using the TensorFlow framework in Python. We used
the sequential function to stack each layer one after the other.

Training the Convolutional Neural Network Model

Once the model architecture is designed, it is time to compile it. We will use a specific learning rate,
optimization method, loss function, and the gradient descent algorithm to determine the model parameters.
This is the same as the other neural network models. After compiling the model, one can use the
model.summary() function to look at the summary of the model, as shown in the image below. Notice the
number of parameters to be trained in the last column. As expected, the pooling layers have zero parameters
while the dense layer has a maximum number.
 62

The next step is to train the model by parsing the training data. To generalize the model we will use a subset
of the dataset for validation.
history = model.fit(train_x, train_y,
validation_data=(val_x, val_y),
epochs=10,
batch_size=64)
In the above code snippet, epochs are the number of iterations in which training takes place. In each
iteration, a limited number of images are passed to the model in batches (defined as batch_size).
Epoch 1/10
359/359 [==============================] - 112s 310ms/step - loss: 1.6670 - accuracy: 0.3375 -
val_loss: 1.5680 - val_accuracy: 0.3995
Epoch 2/10
359/359 [==============================] - 71s 198ms/step - loss: 1.4489 - accuracy: 0.4461 -
val_loss: 1.4554 - val_accuracy: 0.4347
Epoch 3/10
359/359 [==============================] - 72s 201ms/step - loss: 1.3217 - accuracy: 0.4966 -
val_loss: 1.3635 - val_accuracy: 0.4829
Epoch 4/10
359/359 [==============================] - 71s 198ms/step - loss: 1.2335 - accuracy: 0.5351 -
val_loss: 1.3324 - val_accuracy: 0.4955
Epoch 5/10
359/359 [==============================] - 73s 204ms/step - loss: 1.1576 - accuracy: 0.5619 -
val_loss: 1.2901 - val_accuracy: 0.5235
Epoch 6/10
359/359 [==============================] - 76s 213ms/step - loss: 1.0941 - accuracy: 0.5877 -
 63

val_loss: 1.2699 - val_accuracy: 0.5265

Epoch 7/10
359/359 [==============================] - 70s 196ms/step - loss: 1.0280 - accuracy: 0.6148 -
val_loss: 1.2897 - val_accuracy: 0.5268
Epoch 8/10
359/359 [==============================] - 71s 197ms/step - loss: 0.9676 - accuracy: 0.6369 -
val_loss: 1.2946 - val_accuracy: 0.5322
Epoch 9/10
359/359 [==============================] - 75s 210ms/step - loss: 0.9006 - accuracy: 0.6650 -
val_loss: 1.2973 - val_accuracy: 0.5388
Epoch 10/10
359/359 [==============================] - 85s 237ms/step - loss: 0.8365 - accuracy: 0.6914 -
val_loss: 1.3426 - val_accuracy: 0.5357
The model’s training typically takes a lot of time because it's a sequential operation. And also because it
involves optimization, loss calculation, and backpropagation of that error. Thankfully, the Tensorflow
framework performs that at the backend in silence.
So far, we have executed our model, but we are not sure about how well it has performed. Therefore, we
will use metrics like accuracy and draw plots in the next step.