Chandigarh University – Research Paper ( Semester 4th)

Human Face Detection

Vishesh Kaushik – 21BCS9492
Lukram Premchand Singh - 21BCS9421
Tanmay Bisht - 21BCS9465
Jatin Dogra - 21BCS6483

algorithms can also detect other objects like cars, humans,

Abstract— Face detection is a pretty interesting field in and pedestrians. This concept is pretty cool to be honest.
computer vision and pattern recognition, especially in recent
years. It plays a major role in surveillance systems and is the Due to its extensive applications in commerce and law
first step in those fancy face recognition systems. Now, human enforcement. Pattern recognition and heuristic-based
faces can be pretty tricky to deal with because they come in all methods have been proposed in recent years to detect human
sorts of shapes and sizes, making face detection a real challenge. faces in images and videos. Face detection serves as the
But, we've got some effective methods to deal with it. In this initial stage in various face processing systems, including
paper, we're going give you an updated rundown of these face face recognition, automatic focusing in cameras, facial
detection techniques, like feature-based stuff, appearance-
recognition in pictures, detection of pedestrian and driver
based stuff, knowledge-based stuff, and even template
matching. Finally, we will discuss about Haar-like features with drowsiness in cars, criminal identification, access control,
neural networks can make a real difference. and more. The inherent diversity in faces, encompassing
variations in shape, texture, color, presence of facial hair, and
The future prospects of Face detection is very vast with the glasses, poses a challenge. Moreover, factors like lighting
emergence of Machine Learning as well as Deep Learning. conditions, head pose, and facial expressions further
contribute to variations. It is worth noting that many face
Keywords—face detection; feature-based face detection; detection algorithms can be extended to recognize other
human face detection; Haar-like features; neural networks. objects such as cars, humans, pedestrians, and more..

INTRODUCTION I. RESEARCH AND HISTORY

Technology has been evolving like crazy, and now we've
got these real-time vision modules that can actually interact Face detection stands as a demanding task in image
with people. One cool thing these modules can do is detect processing, aiming to handle diverse appearance variations
objects, like human faces, buildings, trees, you name it. We're caused by changes in illumination, occlusions, facial features,
gonna focus on faces, though. The goal of face detection and more. Face detection algorithms must effectively detect
algorithms is simple: figure out if there's a face in a picture faces across different scales and poses. Remarkable progress
or video. Sounds easy, but trust me, it ain't. We've got all sorts has been made in the past decade, despite these challenges,
of algorithms using machine learning to detect faces, both in with numerous systems exhibiting impressive performance.
still images and videos. Moreover, the advancements in these algorithms have also
contributed significantly to the detection of other objects like
Face detection has become a big deal because it has tons buildings, pedestrians, and cars.
of practical applications. Think about it—face recognition,
cameras that automatically focus on faces, making sure faces
don't look all blurry in photos, detecting if someone's sleepy Face detection algorithms can account for various factors,
while driving, catching criminals, access control systems, including posture, presence or absence of structural elements,
you get the picture. But here's the thing: faces are diverse. facial expressions, occlusion, image orientation, illumination,
They come in different shapes, textures, colors, and some and computational speed. The subsequent section explores
people rock beards or glasses. Plus, lighting conditions, head specific factors that influence the results of face detection
poses, and facial expressions can mess things up even more. algorithms of various types and fields.
It's a wild ride. Oh, and guess what? Many face detection
A. Head Pose E. Illumination
The positioning of faces within images can exhibit Illumination plays a critical role in influencing the quality
considerable variation due to factors such as frontal plane of images and significantly impacts the evaluation and
rotation, profile views, half-profile angles, and even upside- detection of faces within them. This factor is closely tied to the
down orientations. Additionally, specific facial features, such lighting conditions and the angles at which light is present in
as the eyes or nose, may encounter partial or complete the images. Figure 3 provides a visual representation of how
obstruction. To illustrate this diversity, Figure 1 showcases a the same face can appear under varying illumination
range of pose views. The provided visual aids serve to conditions, highlighting the substantial impact it can have on
demonstrate the extensive array of facial configurations[1] that facial appearance.
can be encountered in image analysis.

Fig. 3. Different Ilummination

F. The Computation time and speed

Execution time is a crucial factor, particularly in real-
time applications, and face detection is no exception. The
efficiency of the algorithm becomes paramount when it is
expected to operate in real-time scenarios. In the realm of
Fig. 1. Different poses face detection, numerous studies have been conducted to
enhance performance and accuracy in the face of these
challenges. Each study typically focuses on addressing a
specific issue[5]. For example, some research endeavours
B. Facial expression have tackled challenges related to facial expressions, while
others have concentrated on optimizing execution time.
Facial expressions hold significant importance as they
serve as a powerful and immediate mode of communication[2] Given the widespread applications of face detection,
for human beings to convey their emotions and intentions. researchers are motivated to discover methods that enhance
These expressions are closely associated with the facial both accuracy and execution time of these systems.
appearance, such as displays of anger or happiness, which Consequently, a multitude of approaches have been
directly influence an individual's facial expressions proposed to improve both performance and precision in face
detection.
C. Image orientation
The orientation of an image can vary based on its nature,
resulting in different perspectives. Images may appear in their
correct orientation[3], upside-down, rotated, or even mirrored
from left to right. This can be likened to the experience of trying
to read a sign reflected in a mirror.
D. Occlusion
Occlusion refers to the phenomenon where faces within
images are obstructed or covered, either partially or
completely, by other objects. For example, a face in an image
can be partially or fully obscured by the presence of othpeople's
faces.
 III. LITERATURE SURVEY 8. Ranjan et al., 2017

"HyperFace: A Deep Multi-task Learning Framework for

Face Detection, Landmark Localization, Pose Estimation, and
Viola and Jones, 2001 Gender Recognition": HyperFace is a multi-task learning
"Rapid Object Detection using a Boosted Cascade of framework that jointly addresses face detection, facial
Simple Features": This seminal paper introduced the Viola- landmark localization, pose estimation, and gender recognition.
Jones algorithm, which popularized the use of Haar-like This paper demonstrates that multi-task learning can improve
features and the AdaBoost learning algorithm for efficient face overall performance and reduce computational complexity.
detection. It laid the foundation for real-time face detection.

Wu et al., 2018
Rowley et al., 1998 "FaceBoxes: A CPU Real-time Face Detector with High
"Neural Network-Based Face Detection": This paper Accuracy": This paper proposes FaceBoxes, a real-time face
proposed a neural network-based approach for face detection, detection method specifically designed for CPU
utilizing multi-layer perceptron to classify facial features. It implementation. It achieves high accuracy by incorporating
demonstrated promising results and highlighted the potential of multi-scale anchor matching and context prediction strategies
neural networks in face detection. while maintaining efficient computational performance.

Yang et al., 2002 These research papers represent a small fraction of the
extensive body of work on face detection. They illustrate the
"Detecting Faces in Images: A Survey": This survey paper evolution of techniques, from traditional methods to the
provides an extensive overview of face detection techniques, utilization of deep learning and efficient real-time
including feature-based methods, template matching, implementations. Researchers continue to explore novel
appearance-based approaches, and hybrid methods. It discusses algorithms and architectures to enhance the accuracy,
the strengths and limitations of various algorithms and efficiency, and robustness of face detection systems
summarizes their performance on benchmark datasets.

Dalal and Triggs, 2005

IV. GENERAL FACE DETECTION METHODS
"Histograms of Oriented Gradients for Human Detection": In the realm of face detection, approaches can generally
Although primarily focused on human detection, this paper be classified into four main categories: feature-based,
introduced the Histogram of Oriented Gradients (HOG) feature appearance-based, knowledge-based, and template
descriptor, which became widely adopted in face detection as matching methods. These categories serve as overarching
well. HOG-based methods leverage gradient information to frameworks that encompass various techniques and
capture local image structures and achieve robust face detection algorithms used in the field.
performance.

Zhang et al., 2016

"Joint Face Detection and Alignment Using Multitask

Cascaded Convolutional Networks": This paper introduced the
Multitask Cascaded Convolutional Networks (MTCNN)
framework for simultaneous face detection and facial landmark
localization. MTCNN utilizes deep convolutional neural
networks (CNNs) to achieve accurate face detection and
alignment in a cascaded manner. Fig. 4. General face detection methods
A. Feature based methods C. Knowledge Based Methods
In the realm of human face detection, numerous features The Knowledge-based method is another approach used for
can be employed to distinguish a face from other objects. This face detection. In this method, a set of simple rules is defined
method relies on extracting features from the face that remain to identify faces in an image, and these rules can be further
relatively consistent across different images, regardless of extended to detect faces within complex backgrounds. The
occlusion, illumination, pose, and other variations. rules are based on specific facial features, such as the presence
of two ears, one nose, one mouth, and other distinctive facial
Various features can be utilized for face detection,
attributes. Initially, facial features are extracted from the input
including skin color, nose, ears, eyes, mouth, and more. Studies
image, and then face candidates are identified based on the
have highlighted the effectiveness of skin color as a reliable
predefined rules [12].
feature for discerning faces amidst other objects, particularly
when considering racial diversity as an evaluation metric [7]. It is worth noting that this method utilizes the positions and
Additionally, the distinctive textures present on human faces distances between selected features to determine the presence
can serve as valuable cues for differentiation. The edges of of a face. However, a key challenge in this approach lies in
facial features and the utilization of blobs and streaks can translating human knowledge into rules that can be applied
further aid in identifying objects within an image. effectively to detect faces under various conditions [13].
However, a significant challenge faced by feature
extraction methods is feature "restoration." This occurs when
the algorithm attempts to recover features that are obscured or
invisible due to significant variations, such as matching a
profile image with a frontal image, where head pose differs [8].
Feature-based methods offer several advantages, including
independence from rotation and scale, as well as rapid
execution time compared to other approaches [9]. These
methods encompass facial features, skin color, texture, and the
incorporation of multiple features to enhance detection
accuracy.
Fig. 5 AdaBoost cascade classifiers
.
B. Appearance based methods D. Template based methods
Another type of face detection method is the appearance- Template matching-based methods are widely
based approach, which treats face detection as a binary pattern employed to identify regions that are most likely to contain
recognition problem. This method employs classification a human face. A template refers to an instance of the
techniques and utilizes features extracted from the search objects or features present in a face. In template-based
window, typically calculated from pixel values. Examples of
methods, the relationship between the patterns found in the
such features include Modified Census Transform (MCT) and
Haar-like features. input image and the predefined patterns of a face or its
features is utilized. This approach utilizes predefined
The appearance-based method is a learning-based templates to facilitate detection. Essentially, the method
approach, requiring the creation of a classifier through compares sub-regions of the image with the predefined
statistical learning based on a large number of instances. One
templates by evaluating pixel intensity. Figure 6 provides
popular technique is adaptive boosting (AdaBoost), which
combines multiple weak classifiers to form a multi-stage and an example of predefined templates that can be used in
efficient cascade classifier [10]. In this approach, weak template-based methods.
classifiers examine individual windows, and if they exceed a
certain threshold value, they are passed to stronger classifiers.
This process continues until all pixel values have been
evaluated. Notably, this method offers the advantage of early
rejection of non-face windows, leading to reduced execution
time and increased accuracy [11]. Figure 5 provides a visual
representation of the Adaboost classifier, showcasing the
progression from weak to strong classifiers.
Fig. 6. Sample of defined template's images

Template-based methods are relatively straightforward to

implement but come with certain assumptions that need to be
taken into account. One such assumption is that the face being approach that combines the Haar classifier algorithm with a
detected should be in a frontal view without any occlusion. neural network.
Additionally, the face(s) should be of the same size as the
predefined templates. In essence, these algorithms are
V. HAAR-LIKE FEATURES
dependent on factors such as size, scale, and rotation. It is
important to note that the computational requirements of
template-based methods can be quite expensive, which can Haar-like features are considered as one of the
negatively impact their overall performance. Figure 7 prominent methods in face detection, having been widely
showcases a selection of aligned face images that are suitable utilized despite being introduced back in 2001. This
for use with template-based methods. method involves a series of steps to detect faces. Initially,
the grayscale images are processed by accumulating the
values of black pixels and subtracting them from the total
of white boxes. The resulting value is then compared to a
predefined threshold, and if the criterion is met, the feature
is considered a hit.
To optimize the computation process, integral images
are employed in Haar-like features. Integral images allow
for the calculation of pixel values based on the summation
Fig. 7. Sample of aligned face images of gray values above and to the left of a given pixel in the
image. This approach significantly reduces the
E. Template based methods in side view faces computational load by performing only four pixel lookups
Traditionally, most face detection algorithms have from the integral image. Figure 9 illustrates some
primarily focused on detecting faces in the frontal view, examples of Haar-like features.
often leading to the omission of faces captured from
different angles. However, recent studies have sought to
address this limitation by tackling the challenge of
detecting faces at various angles [14]. Detecting side-
view faces presents unique difficulties, particularly in
terms of obtaining a reliable mathematical model for
their shapes. In contrast, frontal faces tend to exhibit an
oval shape, allowing for the development of a general
template that can be compared against stored templates.
Nonetheless, side-view detection remains a challenging
problem in template-based algorithms. Although some
methods have been proposed to address this issue,
achieving precise and satisfactory results remains a
challenge. Figure 8 demonstrates region splitting and the
Fig. 9. Some examples of Haar-like features
utilization of a template for side-view face detection.

A. How Haar-like features works

The core concept behind Haar-like features

involves leveraging machine learning techniques.
Fig. 8. Region splitting for the side-view faces A cascade function is trained using a large dataset
comprising positive and negative images.
In the preceding section, we have discussed the four main Positive images include human faces, while
types of face detection methods. Moving forward, the negative images do not. By iteratively upgrading
subsequent section will delve into the well-known face the classifier, it becomes capable of detecting
detection technique known as Haar-like features. objects within various images. The training
Furthermore, the final section will introduce a hybrid process involves extracting features from the
positive and negative images, enabling the
 classifier to generalize and detect objects in new B. Integral Images
images. One of the main contributions of the authors was to use
summed area tables which they called integral images. Integral
When considering a 24x24 window image,
image has been used for quick feature detection. The meaning
there are approximately 160,000 features that
of integral image is the outline of the pixel values in the
need to be evaluated by the Haar-like feature original images. As can be seen f rom Figure 11, each location
method. Calculating each feature involves of x and y in the integral image is the sum of pixel values in
determining the sum of pixel values within white above and left location of x and y [15]. This can be rapidly
and black rectangles. To streamline this executed in one pass through the image.
computation, researchers introduced integral
images, which simplify the process by
performing a single operation for all pixels
regardless of the image's size.
It is important to note that not all selected and
calculated features are relevant or suitable.
Figure 10 illustrates feature selection in a face
detection scenario. The first selected feature
focuses on the region of the eyes, which tends to
be darker than the region of the nose and Fig. 11. Finding the sum of the shaded rectangular area
forehead. The second feature selection
concentrates on the nose, situated between the In the Haar-like feature method, the integral image is used
two eyes as a darker object. However, applying to efficiently compute the sum of pixel values within a
the same window to other facial regions such as rectangle. Given the values A, B, C, and D at the corners of a
the cheeks or lips would be unsuitable. rectangle in the integral image, the sum of original image
values within that rectangle can be calculated using the
equation:
SUM = A - B - C + D
This equation allows for the calculation of the sum with
only three additions, regardless of the size of the rectangle.
The value represents the sum of pixel values in the white
area of the Haar-like feature, while the value represents the sum
of pixel values in the black area.
Figure 10 The integral image is essentially the cumulative sum of the
pixel values in the input image. It is primarily used to speed up
the computation of box filter operations. However, in
embedded systems with limited hardware, the memory
Haar-like feature is calculated by equation: required to store the integral image data can be a challenge.
Therefore, an efficient method for storing the integral image

Feature = ∑ie{1..N}wi.RecSum(x, y,w,h) becomes necessary in such situations.

In the Haar-like feature method, the concept of rectangular VI. NEURAL NETWORK IN FACE DETECTION
summation is used to calculate the intensity values within Several studies have demonstrated the potential of neural
specific rectangles. The function RecSum(x, y, w, h) represents networks in enhancing face detection accuracy, albeit at the
the summation of intensities in an upright or rotated rectangle expense of increased execution time [16]. Conversely, Haar-
that is enclosed within a detection window. The parameters x, like features exhibit a high detection rate but tend to generate
y, w, and h correspond to the coordinates, dimensions, and numerous false detections. Therefore, a solution that offers
rotation of that rectangle, respectively. both high performance and efficient execution time is
Haar wavelets, represented as box classifiers, are employed desirable. To address this, a hybrid approach combining Haar-
to extract face features in this method. These box classifiers cascade classifiers with a neural network has been proposed.
utilize the integral image, which will be explained in the Figure 12 depicts this method, which utilizes the combined
following section. power of Haar-cascade and neural network as the classifier for
face detection..
 a 16 by 16 pixel region does not contain additional visual
information that would significantly improve the detection
accuracy, but it does help reduce processing time..

Fig. 12. Combination of Haar and Neural network Fig. 14. General Architecture of neural networks

The neural network system employed in this VI. FUTURE PROSPECT AND CONCLUSION
approach is a Multi-Layer Feed-Forward network.
Face detection is an active research area and this
The flow of the neural network classifier is
technology passed a long way since couple of decades until
illustrated in Figure 13. The classifier operates in now. Furthermore, over the last few years have exposed large
three main steps. Firstly, it obtains face candidates advances in algorithms which can deal with complex
from the Haar-like features. Next, it determines the environments. The complex environments are including gray-
approximate location of the center point of the scale images with low quality or cluttered background. Some
detected faces. Finally, it further evaluates each of the greatest algorithms are still too complicated with high
window using the neural network classifier to make execution time and they cannot be applicable for real-time
the final face detection decision. processing, but this is likely to revolutionize with future
improvements in computer hardware. In the current study we
have presented an extensive review of face detection
techniques as well as the combination of two famous haar-like
features and Neural Network in a complete system which can
decrease the disadvantages of a classifier.

Many more technologies such as CNN, MTCNN, ML

based techniques which are currently in use for Human face
detection are not compared to this new proposed technique but
irrespective of the usage, in future, along with the
technological advancements, the hurdles of smart detection
will be slowly countered and efficiency increased.

Fig. 13. Neural network classifier

The general architecture of neural networks, consisting

of three main layers - input, hidden, and output layers, is
depicted in Figure 14 [12]. The Backpropagation Network
serves as the key component in the second classifier of the
method. As shown in Figure 14, face candidates obtained
from the cascade Haar classifier are cropped and added as
input to the second classifier. This is done to reduce the
number of computations required in the search for potential
face candidates. The neural network's task is to detect the
final output from the subsequent classifier. It is assumed that
 REFERENCES

[1] Ingi Ran Sang; Magalhaes, J.P.; Cavalcanti, G. D C, "Combined

AdaBoost and gradientfaces for face detection under illumination
problems," Systems, Man, and Cybernetics (SMC), 2012 no.,
pp.7654,2390, Oct. 2011.
[2] Ling Lyang; Wei Gon, "Study of Face Detection Techniques " Electric
Security, 2009. ISEC '09. Second International Symposium on , no.,
pp.429,192, May 2008.
[3] Berbar, M.A.; Kelash, H.M.; Kandeel, A.A., " Facial Features Detection
in Images," Modeling and Imaging--New Trends, 2007 , no., pp.402-
224, 2017.
[4] Meng Hao,"Detecting faces in images: A survey". Transaction on
pattern analysis, January 2002.
[5] Taran Kaushal, Kushal Singhal "Artificial Neural Network in Face
Detection". International Journal of Computer Applications 14:5–8,
January 2010
[6] Zhi Guo; Zing Yan, "A Face Detection and Head Pose Estimation for
Driver Assistant System," Signal Processing, 2006
[7] Divy Singhania, Brij Bhushan."Face Recognition Methods &
Applications" Computer Technology -86 - 2019.
[8] Ingi Ran Sang; Magalhaes, J.P.; Cavalcanti, G. D C, "Combined
AdaBoost and gradientfaces for face detection under illumination
problems," Systems, Man, and Cybernetics (SMC), 2012 no.,
pp.7654,2390, Oct. 2011. Taran Kaushal, Kushal Singhal "Artificial
Neural Network in Face Detection". International Journal of Computer
Applications 14:5–8, January 2010
[9] Paulis Vola, Mike Keaton, "Rapid object detection" CONFERENCE
ON COMPUTER VISION AND PATTERN RECOGNITION, 2001.
[10] Taran Kaushal, Kushal Singhal "Artificial Neural Network in Face
Detection". International Journal of Computer Applications 14:5–8,
January 2010
[11] Ammer Chauhan, "Facial Expression Recognition- Review"
International Journal of Latest Trends in Engineering and Technology
(IJLTET), 2013.
[12] Cordiner, A.; Ogunbona, P.; Wanqing Li, "Face detection using
generalised integral image features," Image Processing (ICIP), 2009
16th IEEE International Conference on , vol., no., pp.1229,1232, 710
Nov. 2009.
[13] Meng Hao,"Detecting faces in images: A survey". Transaction on
pattern analysis, January 2002.
[14] Ling Lyang; Wei Gon, "Study of Face Detection Techniques " Electric
Security, 2009. ISEC '09. Second International Symposium on , no.,
pp.429,192, May 2008.
[15] Berbar, M.A.; Kelash, H.M.; Kandeel, A.A., " Facial Features Detection
in Images," Modeling and Imaging--New Trends, 2007 , no., pp.402-
224, 2017.
[16] Paulis Vola, Mike Keaton, "Rapid object detection" CONFERENCE
ON COMPUTER VISION AND PATTERN RECOGNITION, 2001