Thesis Report

Emotion Detection from

Frontal Facial Image

Submitted By: Sakib Hussain

ID: 09101004

Supervised By: Abu Mohammad Hammad Ali

1|Page
 Table of Content
Page No .
Abstract -------------------------- 3
1. Introduction -------------------------- 4
2. Field of Usage -------------------------- 5
3. Existing Techniques ------------------- 5
4. Color Models ------------------------- 9
5. My Approach -------------------------- 12
6. Face Feature Extraction -------------- 14
7. Use of Neural Network ---------------- 21
8. Drawbacks of this Approach ------- 26
9. Future Work ------------------------ 27
10. References -------------------------- 28

2|Page
Abstract:

Emotion recognition from facial images is a very active re-search topic

in human computer interaction (HCI). In order to detect Emotion from an image I
have used frontal view facial images. If computers can understand more of human
emotion, we can make better systems to reduce the gap of human computer
interaction .To handle the emotion recognition problem from arbitrary view facial
images. The facial region and others part of the body have been segmented
from the complex environment based on skin color model. Thus, in this paper
I showed some differences between different color models that are used to
implement the system and which color model can be used where. Another aspect is
to extract facial parts from the face. And for that I have used Viola - Jones
algorithm to detect the eye and lips region from a face and then by the help of
neural network I have detected emotion from those features. From the positioning
of mouth and eyes I tried to detect emotion of a face. In this research I will propose
an effective way to detect neutral, happy, sad and surprise these four emotions
from frontal facial image of Human Being.

Keywords: Emotions, Color Model, Feature Extraction, Viola- jones, Neural

Network, Emotion recognition

3|Page
1.Introduction:

Facial expressions play a key role for understanding and detecting emotion. Even
the term “interface” suggests how important face plays in communication between
two entities. Studies have shown that reading of facial expressions can
significantly alter the interpretation of what is spoken as well as control the flow of
a conversation. The ability for humans to interpret emotions is very important to
effective communication; accounting for up to 93% of communication used in a
normal conversation depends on emotion of an entity. For ideal human-computer
interfaces (HCI), we would desire that machines have the capability to read human
emotion. For that this research is all about how computers can detect emotion
properly from its various sensors. For this experiment I have used facial image as a
medium to read human emotion. The research on human’s emotion can be traced
back to the Darwin’s pioneer working and since then has attracted a lot of
researchers to this area. There are seven basic emotions that are universal to human
beings. Namely neutral, angry, disgust, fear, happy, sad, and surprise, and these
basic emotions can be recognized from human’s facial expression. In this research
I will propose an effective way to detect neutral, happy, sad and surprise these four
emotions from frontal facial emotion. During the past decades, various methods
have been proposed for emotion recognition. Many algorithms were suggested to
develop systems/applications that can detect emotions very well. Computer
applications could better communicate by changing responses according to the
emotional state of human users in various interactions. Emotion of a person can be
determined by speech or his face or even one’s gesture. The work presented in this
paper explores the recognition of expressions from face.

4|Page
2.Field of Usage:
With the Enhancement of communication between human and computer, this
would open a variety of possibilities in robotics and human-computer interfaces
such as devices that warn a drowsy driver, attempt to soothe an angry customer, or
even in the field of Investigation to detect psychological condition of a criminal.
The most important use it can provide that , in case of emergency for medical
patient it can determine patience agony and alarm doctor or nurse to help that
patience quickly. Field of psychology has played an important role in
understanding human emotion and in developing concepts that may aid these HCI
technologies.

3.Existing Techniques:
So far, numerous research projects have been done on recognizing emotion from
Face. Many used many methods to implement their systems. Facial expressions
provide the building blocks with which to understand emotion. In order to
effectively use facial expressions, it is necessary to understand how to interpret
expressions, and it is also important to study what others have done in the past.

Facial Action Coding System (FACS) is a system to determine human facial

expressions, originally developed by Paul Ekman and Wallace V. Friesen, and
published in 1978. Ekman, Friesen, and Joseph C. Hager published a significant
update to FACS in 2002. Movements of individual facial muscles are encoded by
FACS from slight different instant changes in facial appearance. It is a common
standard to systematically categorize the physical expression of emotions.
Recently, FACS has been established as a computed automated system that detects
faces in videos, extracts the geometrical features of the faces, and then produces

5|Page
temporal profiles of each facial movement. FACS is a key system to determine
Facial Feature Extraction.

Fasel and Luettin performed an in depth study in an attempt to understand the

sources that drive expressions. Much of expression research to date has focused on
understanding how underlying muscles move to create expressions. For example,
studies have shown that movement of the eyes and eyebrows and mouth of a
person are primary indicators of almost all the emotions.
Much of previous work has used FACS as a frame-work for classification. In
addition to this, previous studies have traditionally taken two approaches to
emotion classification according to Fasel and Luettin.

 a judgment based approach

 a sign-based approach.

The judgment approach develops the categories of emotion in advance such as the
traditional six universal emotions.
The sign-based approach uses a FACS system, encoding action
Units in order to categorize an expression based on its characteristics.

This approach assigns an emotional value to a face using a combination of the key
action units that create the expression. A neural-network is used to recognize
action units from the coordinates of facial features like lip corners or the curve of
eye brows.

6|Page
For self-organized learning method, principle component analysis
(PCA) is widely used in the field of data compression and feature extraction.
There are two basic approaches to the computation of principal components: batch
and adaptive methods. The batch methods include the method of Eigen
decomposition and the method of singular value decomposition (SVD), while the
adaptive methods are mainly done by neural networks. The main target of PCA is
to explain the variance–covariance structure of the data through a few linear
combinations of the original variables.

The main concerning thing about PCA is that it utilize only the global information
of face images, this method is not very effective for different facial
expressions.

Aw-SpPCA algorithm is another promising mechanism that has been used to

detect emotion from image. This algorithm is based on the concepts of impound
illumination conditions, facial expressions variations to local areas dividing a face
image into several sub-images, and carry out PCA computation on each local area
independently.
Another Approach is to make Appearance based Models. These rely on techniques
from statistical analysis and machine learning to find the relevant characteristics of
face images. Some appearance-based methods work in a probabilistic network. An
image or feature vector is a random variable with some probability of belonging to
a face or not.

Neural Networks is another key tool of extracting feature from an image. Many
pattern recognition problems like object recognition, character recognition, etc.

7|Page
have been faced successfully by neural networks. These systems can be used in
face detection in different ways.

Support Vector Machines. SVMs are linear classifiers that maximize the margin
between the decision hyperplane and the examples in the training set. So, an
optimal hyperplane should minimize the classification error of the unseen test
patterns. This classifier was first applied to face detection by Osuna et al.

Hidden Markov Model is another statistical model that has been used for face
detection. The challenge is to build a proper HMM, so that the output
Probability can be trusted. The states of the model would be the facial features
which are often defined as strips of pixels. The probabilistic transition between
states is usually the boundaries between these pixel strips. As in the case of
Bayesians, HMMs are commonly used along with other methods to build detection
algorithms.

Linear discriminant analysis (LDA) method is used in statistics and pattern

recognition to find a linear combination of features. The resulting combination
may be used as a linear classifier or, more commonly, for dimensionality reduction
before later classification. LDA explicitly attempts to model the difference
between the classes of data. PCA on the other hand does not take into account any
difference in class, and factor analysis builds the feature combinations based on
differences rather than similarities.

8|Page
4.Color Models:
Color models Plays a very important role in Detecting face form an image and
Feature Recognition. Research in color modeling for computational purposes has
concentrated on finding numerical representations of colors which are convenient
for use in image processing. Among many models of colors, these representation
schemes use a three dimensional color space (RGB, CMY, YIQ, HSV, and HSL).

Some Details about Different color models:

RGB: The RGB color model is an additive color model in which red, green, and
blue light are added together in various ways to reproduce a broad array of colors.
RGB is a device-dependent color model. Different devices detect or reproduce a
given RGB value differently; it depends on manufacture to manufacture how the
color will be taken. Thus an RGB value does not define the same color across
devices without some kind of color management.

HSV and HSL: To make the RGB Color model more intuitive HSV and HSL
color model was developed. HSL and HSV are used today in color pickers, in
image editing software and in image analysis. HSL stands for hue, saturation, and
lightness, and HSV stands for hue, saturation, and value.

CMYK: This Color Model is mainly used for printing process. CMYK refers to
the four inks used in some color printing: cyan, magenta, yellow, and key (black).
This Model is Rarely Used for Image Processing.

9|Page
YCbCr: YCbCr is not an absolute color space; rather, it is a way of encoding
RGB information. The actual color displayed depends on the actual RGB primaries
used to display the signal. The YCbCr color domain was developed for efficient
image compression by separating luminance (Y) and chrominance (Cb, Cr)
components. This model is also known as an effective model for skin color
segmentation.

The RGB displays are made to match our color-perception system. They have a
computer counter-part, the (R,G,B) "coordinates". Since these coordinates map to
implementation rather to the physical, "real nature" properties of color, they are not
suited to perform some mathematical processing, e.g. perceptually linear gradient
interpolation, color correction, brightness and saturation ops, etc.

Even though most of digital image acquisition devices produce R, G, and B

components, the RGB color space is converted into different color spaces
depending on applications. For face recognition, the eigenface analysis in the RGB
domain may not be effective, because R, G, and B components are largely
correlated with each other. Therefore, we need to find the color space that is less
correlated between its components for improvement of face classification
performance. Here, we will look into effective color spaces for face recognition.
Diverse color spaces including RGB, HSV, YCbCr, and YCgCr were investigated
in the PCA-based color face recognition system.

10 | P a g e
 As we can see from here that YCbCr and HSV color model are very effective for
face recognition and facial expression variation.

But HSV is also very popular model that is used in Face Recognition and Feature
Extraction. In terms of Shadow in a given Image, HSV works Better than other
models because of its handling of saturation of that shadowed portion.

11 | P a g e
One of the Main Reason of HSV being so popular is the Ease of implementation.
And various Researches showed that in terms of Extraction of features HSV gives
a Constant High Performance where YCbCr is not that Constant in Case of Small
size images.

My Approach :
Given Image
To make an efficient system that will be able to
determine emotion from frontal face image is
the main goal here and to reach that I have Select adequate
Color Model
followed a certain workflow.

Face region
For a given Image I have selected to use HSV Detection
Image of the actual Image to Determine the
Facial Region. When the image is converted to Feature Extration
(lips and Eyes)
HSV image, detecting the larger portion of the
image with a certain HSV Value for Human
Face can Emotion
Classification
provide the Required Face Region from the
image.
HSV are used today in color pickers, in image Fig: General Workflow

editing software and in image analysis. HSL stands for hue, saturation, and
lightness, and HSV stands for hue, saturation, and value.

The Image below Shows the Conversion of image from RGB to HSV image of it.
This detects the face from an image. As we can see that there are many white or 0

12 | P a g e
portions in image but the Face on the image is the only part where the white is
connected and it is the biggest region of the image.

In such Scenario this binary image is giving us face region very accurately. But we
face some problem when the background is very bright such as White Background.
This system works for only a certain color region.

So in this case if we convert the given image to HSV first then try to detect it we
can have more accurate result. From a HSV Image using RGB Range from R=
0.0488, G= 0.364, B= 0.882 to RGB value of R= 0.468, G = 0.546, B = 0.639
taken as accepting Range we can Detect Face Region From a Given Image as
Input.

13 | P a g e
Next Step is to Crop that Portion and process of Feature Extraction. After
Cropping we can see we have,

Fig : Face Region Detection and Cropped Version in YCbCR and HSV Color model.

Face Feature Extraction:

Now for Feature extraction there are many methods to apply. They can be
categorized according to whether they focus on motion or deformation of faces and
facial features, respectively, whether they act locally or holistically. In our case we

14 | P a g e
need deformation of faces and facial features. There are many techniques of feature
extraction from an image. Here is a list of the methods:

Among all these Algorithms I have tried to Work with Neural Network based
methods, Gavor Wavelet Transforms, and Viola Jones Algorithms to find out the
one that provide the most accurate face feature extraction and implement the
System which will detect the emotion by reading those features. Here are some of
the Performance evaluations of these Methods of Feature Extraction and Emotion
Recognition using these.
15 | P a g e
 Fig: performance analysis for Weighted PCA approach

Fig: Performance analysis of gavor Wavelet Method

Despite there are these approaches which provides quite accurate output for the
system, But I have used Viola Jones Algorithm to Extract Feature from the Image.

Viola Jones object detection Framework is one of the modern algorithm which is
based on Haar Cashcades which is an appearance based model. The Viola–Jones
object detection framework is the first object detection framework to provide
competitive object detection rates in real-time proposed in 2001 by Paul Viola and

16 | P a g e
Michael Jones. Although it can be trained to detect a variety of object classes, it
was motivated primarily by the problem of face detection.

This approach to detecting objects in images combines four key concepts:

 Simple rectangular features, called Haar features

 An Integral Image for rapid feature detection
 The AdaBoost machine-learning method
 A cascaded classifier to combine many features efficiently

The features that Viola and Jones used are based on Haar wavelets. Haar wavelets
are single wavelength square waves (one high interval and one low interval). In
two dimensions, a square wave is a pair of adjacent rectangles - one light and one
dark.

Fig: Haar-Like Feature

These are few types of Haar Classifier that were developed for detecting different

17 | P a g e
type of features and for different environment.
The actual rectangle combinations used for visual object detection are not true
Haar wavlets. Instead, they contain rectangle combinations better suited to visual
recognition tasks. Because of that difference, these features are called Haar
features, or Haarlike features, rather than Haar wavelets. Figure above shows the
features that OpenCV uses.

The simple rectangular features of an image are calculated using an intermediate

representation of an image, called the integral image. The integral image is an
array containing the sums of the pixels’ intensity values located directly to the left
of a pixel and directly above the pixel at location (x, y) inclusive. So if A[x,y] is
the original image and AI[x,y] is the integral image then the integral image is
computed as shown in equation 1 and illustrated in Figure below.

As above Figure shows, after integration, the value at each pixel location, (x,y),

18 | P a g e
contains the sum of all pixel values within a rectangular region that has one corner
at the top left of the image and the other at location (x,y). To find the average pixel
value in this rectangle, you'd only need to divide the value at (x,y) by the
rectangle's area.

But what if you want to know the summed values for some other rectangle, one
that doesn't have one corner at the upper left of the image? Figure on the right
shows the solution to that problem. Suppose you want the summed values in D.
You can think of that as being the sum of pixel values in the combined rectangle,
A+B+C+D, minus the sums in rectangles A+B and A+C, plus the sum of pixel
values in A. In other words,
D = A+B+C+D - (A+B) - (A+C) + A.

It only takes two passes to compute both integral image arrays, one for each array.
using the appropriate integral image and taking the difference between six to eight
array elements forming two or three connected rectangles, a feature of any scale
can be computed. Thus calculating a feature is extremely fast and efficient. It also
means calculating features of various sizes requires the same effort as a feature of
only two or three pixels. The detection of various sizesof the same object requires
the same amount of effort and time as objects of similar sizes since scaling requires
no additional effort.

To select the specific Haar features to use, and to set threshold levels, Viola and
Jones use a machine-learning method called AdaBoost. AdaBoost combines many
"weak" classifiers to create one "strong" classifier. "Weak" here means the
classifier only gets the right answer a little more often than random guessing
would. That's not very good. But if you had a whole lot of these weak classifiers,
and each one "pushed" the final answer a little bit in the right direction, you'd have
19 | P a g e
a strong, combined force for arriving at the correct solution. AdaBoost selects a set
of weak classifiers to combine and assigns a weight to each. This weighted
combination is the strong classifier.

Viola and Jones combined a series of AdaBoost classifiers as a filter chain, shown
in Figure below, that's especially efficient for classifying image regions. Each filter
is a separate AdaBoost classifier with a fairly small number of weak classifiers.

Here are some of the images where face features were detected by the System.

20 | P a g e
 Fig: Detection of face Features Such as Eyes Nose and Lips.

Use of Neural Network to Detect Emotion from the

Extracted Feature:
From the Extracted Lips and Eyes Region and from reading them we can detect the
emotion of a Human Face. I have used neural network to detect the emotion from
the lips and eye region. And for that knowing neural network is a vital step.

Neural Network is a powerful and efficient way of getting a solution using pattern
recognition. It is a widely used method to recognize pattern where in my research I
have used it to determine emotion by remembering the pattern of lips region.
Neural Network is an efficient way of solving complex problems is following the
lemma “divide and conquer”. A complex system may be divided into simpler and
smaller systems, in order to be able to understand it. Networks are one approach
for achieving this. There are a large number of different types of networks, but
they all are characterized by the following components: a set of nodes, and
connections between nodes. Nodes can be seen as computational units. They

21 | P a g e
receive inputs, and process them to obtain an output. This processing might be very
simple (such as summing the inputs), or quite complex.

The connections determine the information flow between nodes. They can be
unidirectional, when the information flows only in one direction, and bidirectional,
when the information flows in either direction.

Similarly we are also using a kind network known as “Artificial Neural Network”
in short (ANN). Biological human brain neural structure is more similar to this
network. They are powerful tools for modeling, especially when the underlying
data relationship is unknown. ANN can identify and learn similar patterns between
input data sets and corresponding target values. After training ANN can be used to
predict the outcome of new independent input data. As we mentioned ANN
follows the learning process of the human brain and can process problems
involving non-linear and complex data even if the data are imprecise and noisy. It
consists of simple computational units called neurons, which are highly
interconnected. A very important feature of these networks is there adaptive nature
where, “learning by example” replaces “programming” in solving problems. This
feature makes such computational models very appealing in application domains.

Structure of the Network:

The network we have talked about so far must have a structure or model that would
help all the learning and computational processes. The kind of model we are
implementing is called Multilayer Perceptron (MLP) or sometimes called
Multilayer Feed Forward neural network. The name itself defines the structure that
is a model that has multiple layers of neurons or nodes. MLP neural network
learned by back propagation algorithm is based on supervised procedure, i.e. the
network constructs a model based on examples of data with known output. As like

22 | P a g e
any network models this network also contains two basic layers called input layer
and output layer. The number of neurons in the input layer depends on the input
image size that is each pixel represents each node respectively. The output layer
may contain one or more neurons depending on the developer and the system. In
our system we kept 4 output neurons which will resembles emotions such as
HAPPY, NEUTRAL, SAD and SURPRISED.

In between these two layers there may be one or more additional layers called the
hidden layers. We have added one hidden layer in our system because it increases
accuracy while the learning process proceeds. There is no fixed rule for the number
of neurons that would be in the hidden layer. These bunches of layers are
interconnected and as part of MLP the information flows unidirectional from input
layer to output layer through hidden layer.

In each node there are assigned weight depending on the nodes type. And
idealization of a NEURON processes activities or signals. Each input activity is
multiplied by a number called weight. The unit adds together the weighted inputs.
Then it computes the output activity using and input- output function. The process
23 | P a g e
can be viewed in detail from the following figure.

Training the Network:

To train a neural network to perform some task, we must adjust the weights of each
unit in such a way that the error between the desired output and the actual output is
reduced. This process requires that the neural network compute the error derivative
of the weights (EW). In other words, it must calculate how the error changes as
each weight is increased or decreased slightly. The back-propagation algorithm is
the most widely used method for determining EW.

Also in term of implementation, I had to Tell Matlab what optimization (or

training) routine to use. I have used use Sigmoid Activation Function (transig),
which is gradient descent, Secondary as trainlm (for Levenburg-Marquardt, which

24 | P a g e
is a combination of gradient descent and Newton’s Method).

Here is a Portion of Training set of input that was used to train the Network.
There were 100 input data used to train the network.

The Back Propagation Algorithm works the Following way.

25 | P a g e
After our training is successful then we will see if out network able to recognize
any given pattern.

This shows that the System is perfectly recognizing the Emotion of the given input.

By this Technique I have tested 50 times with different Images with Different
Emotion and the Performance that the method showed are given below.

Emotion % of Detection
HAPPY 100%
NEUTRAL 98%
SAD 96%
SURPRISED 100%

Fig: Accuracy measurement

The Drawback of this approach:

This work has presented the face recognition area, explaining different approaches,
methods, tools and algorithms used to determine Emotion from Images. Some
algorithms are better, some are less accurate, some of them are more versatile and
others are too computationally costly. Despite this variety, face recognition faces
some issues inherent to the problem definition, environmental conditions and
hardware constraints. In terms of side Face or where the image is not providing

26 | P a g e
proper frontal face view these algorithms and methods give very less accurate
recognition of emotion.

Some of the Drawback of this System:

 In Case of a Dark person with a Bright Background, this System Won’t be

able to detect face.
 Side Faced Images will fail to detect emotion from them as it will not be
able to extract face feature.
 I have only used Lips as Neuron to detect emotion. Some of the mixed
emotion will be failed to detect;

Future Work:
The System that I have implemented works quite well with normal background but
if the background is too bright or matches with skin color of the person or people
with dark skin, the system fails to deliver properly, so I will work on how to make
the system more efficient so that it works on bright background or for people with
darker skin. Eyes also help to detect emotions. So I will use eyes to detect emotion
more accurately. Adding more emotions to detect is one of my future plans as I
have only detected four emotions for this system. Another plan is to detect
emotions from live video footage as it can help on many sectors of daily usage
such as medical or investigation or security purpose and many more.

I will try to make a better system in future as Emotion Detection can help us to
decide many problems that we face in our daily life and it will help us to make the
society a better place to live in.

27 | P a g e
References:

 http://en.wikipedia.org/wiki/RGB_color_model#Color_management
 http://en.wikipedia.org/wiki/HSL_and_HSV#Use_in_image_analysis
 www.stanford.edu/class/ee368/Project_03/.../ee368group16.ppt
 http://en.wikipedia.org/wiki/Facial_Action_Coding_System
 http://face-and-emotion.com/dataface/facs/manual/TitlePage.html
 B. Fasel and J. Luettin. Automatic facial expression
 analysis: A survey. Pattern Recognition, 2003.
 www.ehu.es/ccwintco/uploads/e/eb/PFC-IonMarques.pdf
 isl.ira.uka.de/downloads/icarcv2000_jie1.pdf
 http://face-and-emotion.com/dataface/facs/manual/TOC.html
 www.ijcte.org/papers/213-H282.pdf
 Face Recognition Algorithms - Proyecto Fin de Carrera (June 16, 2010)
 Principal Component Analysis Based Feature Extraction,
Morphological Edge Detection and Localization for Fast Iris Recognition
Suganthy, M. and P. Ramamoorthy
 Face Detection Using Gabor Feature Extraction and Artificial Neural
Network -Bhaskar Gupta , Sushant Gupta , Arun Kumar Tiwari(ABES
Engineering College, Ghaziabad)
 Speed limit sign board detection & extraction of digits in different weather
conditions- Khan Hafizur Rahman & MD. Zubair Alam
 ArtificialNeural Networksfor Beginners-Carlos Gershenson
([email protected])

 Artificial neural networks:Training with backpropagation-Juan Rojo (INFN,

Sezione di Milano)
 http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html
 http://www.dacs.dtic.mil/techs/neural/neural_ToC.html
 http://www.avaye.com/files/articles/nnintro/nn_intro.pdf
 http://www.cognotics.com/opencv/servo_2007_series/part_2/sidebar.html
 FACIAL FEATURE DETECTION USING HAAR CLASSIFIERS - Phillip
Ian Wilson & Dr. John Fernandez

28 | P a g e