2023 International Conference on Computer Science, Information Technology and Engineering (ICCoSITE)

Development of Real-Time Face Recognition for

Smart Door Lock Security System using Haar
Cascade and OpenCV LBPH Face Recognizer
2023 International Conference on Computer Science, Information Technology and Engineering (ICCoSITE) | 979-8-3503-2095-4/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICCoSITE57641.2023.10127753

1st Daniel Anando Wangean 2nd Sinjiru Setyawan 3rd Fairuz Iqbal Maulana
Computer Science Department, Computer Science Department, Computer Science Department,
School of Computer Science School of Computer Science School of Computer Science
Bina Nusantara University Bina Nusantara University Bina Nusantara University
Jakarta, Indonesia 11480 Jakarta, Indonesia 11480 Jakarta, Indonesia 11480
[email protected] [email protected] [email protected]

4th Gusti Pangestu 5th Choirul Huda

Computer Science Department, Computer Science Department,
School of Computer Science School of Computer Science
Bina Nusantara University Bina Nusantara University
Jakarta, Indonesia 11480 Jakarta, Indonesia 11480
[email protected] [email protected]

Abstract— Face recognition is a technology that is widely Android application, PIR sensor that connected to Wemos
used in security systems. In a door security system, facial ESP8266, and TTGO camera to recognize faces that has
recognition can be used to open the door simply by recognizing been registered to open a door that locked by solenoid.
the face of the door owner. This study aims to develop a real- Simply, they make door that can be opened by button in
time facial recognition system for smart locking doors using Android application and movement in front of the door.
Haar Cascade and OpenCV LBPH Face Recognizer. The There is not much information about TTGO uses in their
purpose of this project is creating security system to limit project, it just said that TTGO camera can recognized
people who can access a room. The Haar Cascade method is registered user faces but still not integrated with the door.
used to detect faces in images, while the OpenCV LBPH Face
Recognizer is used to recognize detected faces. This system was Based on Andreies’ project, it still need a system that not
developed using the Python programming language and the just unlock door automatically, but also a system or program
OpenCV library. The test results show that this system can that can recognize people who want to access the room and
detect and recognize faces with an accuracy of 62.7% with our make decision to unlock the door or not. So, Author want to
dataset and can be improved by adding more datasets and make a system that can open a door with face recognition
using deep learning algorithms to train the recognizer model. concept using Open Source Computer Vision (OpenCV)
Thus, the developed real-time facial recognition system can be library from Python and Haar cascade to pre-processing
used as a smart locking door security solution with high
image and Local Binary Patterns Histograms (LBPH) Face
accuracy.
Recognizer algorithm to train image dataset. The purposes
Keywords—face recognition, Haar cascade, Smart Gate, are to improve security of the room by limiting people who
LBPH Face Recognizer, room security can access the room using face recognition and bring up high
technology environment so people can be encouraged to start
their innovation. For comparison, this project will be more
I. INTRODUCTION secure than previous project from Anderies since this project
A real time face recognition system is already used in implement face recognition system to limit people accessing
many aspects and purposes. Including security purpose in a room.
devices such as smartphone, tablets, laptop, and many other
electronic gadgets. Not just for security, face recognition II. RELATED WORK
concept is one of potential concepts that influence our
lifestyle in industrial revolution 4.0. In principle, face Many researcher [13] explored many face detection
recognition can help improving security problems and bring techniques in OpenCV that may be used in a variety of
up high technology environment simultaneously. situations like in table 1. Algorithms for static pictures were
summarized, such as face extraction from photographs with
Technologies are created to make human life easier. noncolored and colored backgrounds. Furthermore, moving
Including face recognition technology, it can be used for face processing methods such as face identification by
unlocking smartphone [1], tablets and laptops [2], unlocking detection of flashing eyes and Haar Cascade Algorithms
house door [3] in IoT [4], [5], smart home security system were studied. The Haar detection approach was touted as the
[6], [7], person identification using smart glasses [8], most successful at the time. Some research [14] created one
surveillance system [9], basketball players’ movement of the works that used the Haar face detection approach.
prediction [10], criminal detection [11], and many more. It They used computer vision and a neural network-based
concludes that face recognition technology can help humans’ Gaussian Mixture Model to construct the face mask
life easier and need to develop into other aspects including recognition system. This AI-based model was developed to
public areas’ facility. predict if a face picture is normal (just the face) or abnormal
This paper is inspired by smart door decision system (in case if human wears glasses or facemasks). Based on the
from Anderies’ team [12]. In their architecture. They use an threshold, the model determines if a human face picture is

Authorized licensed use limited to: Modern Education Society's Wadia College of Engineering. Downloaded on July 22,2024 at 10:20:38 UTC from IEEE Xplore. Restrictions apply.
979-8-3503-2095-4/23/$31.00 ©2023 IEEE 506
 normal or abnormal. In the investigation by Cabani et al., For the source code, it’s divided to 2 programs. First, the
Haar classifiers were employed to identify correct mask- face recognizer generator. Author uses
wearing [15]. cv2.face.LBPHFaceRecognizer function from OpenCV [12].
With preparing some picture of 2 or more people, a Python
As a result, this study will result in a system that is not program will read all the image and use them as data
only secure due to its real-time facial recognition training. Then the recognizer will be saved as .yml file. This
authentication techniques, but also simple and file will be used in detection program to classify person that
straightforward to use since it does not need extra devices, want to entry the door. Here some image dataset used in
manual input instructions from users, or any user interface recognizer generator.
expertise. The whole smart door lock mechanism is built in
the door and operates completely autonomously. This is a
revolutionary smart door lock system design that will be
studied further in this research.

TABLE I. RELATED WORKS

Ref No Title Authentication method

“Password Protected Home
[16] Automation System with Password matching
Automatic Door Lock”
“Design and implementation
of a door lock control based NFC authentication
[17] on a near field and password
communication of a matching
smartphone”
“Smart and real-time door
lock system for an elderly Real-time face Fig. 2. Face dataset for training
[18]
user based on face recognition
recognition”
“Real time face recognition
For the smile detection program, Python programming
Real-time face language is used with OpenCV library to help image pre-
[19] of video surveillance system
recognition processing. Using a web cam, an image, saved as 3
using haar cascade classifier”
“Smart Door for COVID
Mobile phone and dimensions array (3 RGB color intensity for every pixel),
[20] Real-time face will be transformed to grayscale and used as face detection
Restricted Areas”
recognition
object. With help from
“Intelligent Secure Smart
[21] Locking System using Face
Real-time face cv2.CascadeClassifier().detectMultiScale() function and
recognition Haar cascade model [12], [22] the program can find the
Biometrics”
location of a face or more in x, y, w, h format. The x and y
represent coordinate of top-left point of the face, the w and h
represent width and height of the face starting (x,y)
III. PROJECT DEVELOPMENT
coordinate.
A. Construct Source Code and Hardware When a face detected, using same function but another
For the hardware, Author use Arduino UNO, 12V Power Haar cascade model that specifically have purpose to detect
Supply, and Mini PC with Windows 10 Pro Operating smile, the program will find any smile within the face.
System. The target is every time someone smile, the program Besides, there are still 2 more parameters [13], scaleFactor
recognizes the person, detect the smile and, unlock the door and minNeighbors, that can affect smile detection accuracy.
for 15 seconds. Those devices will be connected into one scaleFactor is a parameter specifying how much the image
system and control magnetic door lock. More detail is shown size is reduced at each image scale and minNeighbors is a
in Fig. 2 below. parameter specifying how many neighbors each candidate
rectangle should have to retain it. The best value for these
parameters will be explained in the third section. Finally,
when a smile detected, the program will send signal to the
Arduino using Serial library.

B. Architecture of the Project

Referring from the previous statement, it’s needed to
make a system that can recognize face and detect smile.
Using Python programming, OpenCV library, and Haar
Cascade model, images that saved in subfolder that named
by the faces’ owner’s name will be processed into grayscale
image, labelled by the name of the subfolder. Then, a
recognizer will be generated and used in smile detector to
predict faces from webcam image. For more details, it can be
seen in Fig. 1.
Fig. 1. Hardware diagram

Authorized licensed use limited to: Modern Education Society's Wadia College of Engineering. Downloaded on July 22,2024 at 10:20:38 UTC from IEEE Xplore. Restrictions apply.
507
 than 100 it can be concluded as ‘not recognized’ [12], [14].
Then, the percentage value can be obtained by subtract 100
with confidences value. Here the result of recognizing:

(a) (b)
Fig. 4. Face detection testing using Haar Cascade (a), (b)

As shown in Fig. 4, Daniel’s and Sinjiru’s face also can

be recognized with the recognizer. Unfortunately, Sinjiru’s
face has low confidences percentage (below 40%). It’s
happened because the images in dataset of Sinjiru’s face are
not too variative. In the fact, to get high confidences
percentage, it needs more than 50 images per person
according to Marcelo project [12]. So, it must be better if
more datasets are available.
In this project, accuracy of the detector is obtained by the
highest confidences percentage. From 7 people tested, the
Fig. 3. Smart Gate Architechture average highest accuracy achieved is 62.7%. This result
excluded from unrecognized result (below 40%).
Based on Sarwar’s project [11], the LBPH Face
Recognizer is a function that count Euclidean distance B. Smile Detector Result
between two histograms in a face. The Euclidean distance For smile detector and door unlock, some tests were done to
formula is shown in (1): find out the best cascade.detectMultiScale() parameters. This
table will show the parameters value, how far and how smile
is the person, and the result:
(1)
TABLE II. DETECTMULTISCALE PARAMETERS’ COMPARISON
Here, and are the histograms of the image Distance
Min-
in dataset and target image from webcam and D is the value Scale-
Neighb
face to
Smile Type
Smile Descripti
Factor camera detected? on
that will return as confidence value. ors
(cm)
False
2.7 60 10 Little smile No
Negative
Smile with True
2.7 60 10 Yes
IV. RESULT AND DISCUSSION teeth Positive
Smile with False
The development of this project consists of three main 2.7 60 50 No
teeth Negative
parts: data collection for dataset requirements, training, and False
1.3 25 10 Poker face Yes
face recognition. The proposed system works after a user's Positive
face is detected, automatically recognising a person's identity True
1.3 60 10 Poker face No
based on information from the intended person recorded in Negative
Smile with False
the dataset. Furthermore, it will detect whether the user is 1.3 60 50
teeth
No
Negative
smiling widely or not to be able to process the locking Smile with True
system on the door. The recognised result shows a person 1.1 60 50 Yes
teeth Positive
who matches in the database and smiles broadly with a 1.1 60 10 Little smile Yes
True
percentage display, while the unrecognised result shows an Positive
unknown. This section will be divided into 3 subsections. True
1.1 60 10 Poker Face No
Negative
The result of face recognition and the result of smile
detector. Then, the final result and problems of the Smart As shown in Table 2, too high scaleFactor make the
lock door. smile detector can’t detect smile from 50 cm distance. It’s
caused by far object has less resolution than near object. So,
A. Face Recognition Result when it’s stretched too much, the image will so blur and hard
The result will be shown in face, message containing the to determine is it a smile or not. In fact, some Python
face’s name, and confidences percentage. This percentage programmer conclude that the best scaleFactor parameter of
obtained from cv2.face.FaceRecognizer.predict() return detectMultiScale() is around 1.05 to 1.4 [15]. For the
value [12]. Besides image’s label, predict() function also minNeighbors parameter, lower value the minNeighbors,
return confidences value to show how far is the nearest more sensitive the program with the smile. This project is
neighbour. If the confidences value 0, it means the image is purposed to unlock the door using smile and recognized face.
perfectly match with the recognizer’s dataset and if it’s more So, it will be good using 60 as minNeighbors value so that

Authorized licensed use limited to: Modern Education Society's Wadia College of Engineering. Downloaded on July 22,2024 at 10:20:38 UTC from IEEE Xplore. Restrictions apply.
508
 person who want to entry need to smile with him/herself There are some problems occur when experimenting this
teeth to unlock the door. project. First, People who wearing glasses will be detected as
smiling even he/she isn’t smiling. It’s caused by the curve of
C. Smart Gate Final Result, Problems, and Solutions the glasses frame is similar with curve of smiling lips. So, for
Smart Gate system is installed on one of office room of the solution, after face detected using Haar cascade frontal
Author institution. This system improves security of the face, the rectangle of the face is cut into bottom half so that
room, so only eligible person (recognized face) that can entry only chick, mouth, and chin that the program finds out smile.
the room. Here some picture of the result. Figure 5 shows a Then, Program crash if there 2 or more smiles detected. It’s
test of a person trying to open a glass door that has been caused by Serial library that sending signal every time a
installed with this project. However, the user cannot open the smile detected. Two or more smiles detected means 2 or
glass door because it is still in locked status. more signal send to Arduino in 0.033 second. It’s causing
crash because Arduino need time to execute the first signal
before the second. For the solution, face detection limited to
1 face for each frame. Then, if one smile detected, the other
smiles ignored, and the program will stop detect any more
face for 90 frames (3 seconds).

V. CONCLUSIONS
A face-recognition-based Smart lock door is a solution to
improve room security by limiting person that can access the
room and simultaneously bring up high tech environment
that can encourage people to be innovative. This system can
be realized using Python programming, OpenCV library, and
Fig. 5. Door locked when no smile detected
Haar Cascade model to create face recognizer and smile
detector program. People who wearing glasses will be
detected as smiling even he/she isn’t smiling. It’s caused by
To be able to open the door, the person must face the the curve of the glasses frame is similar with curve of
monitor screen that has a webcam camera installed to detect smiling lips. So, for the solution, after face detected using
the person's face and a big smile. In Figure 6, a display Haar cascade frontal face, the rectangle of the face is cut into
appears on the monitor screen indicating that the user has bottom half so that only chick, mouth, and chin that the
smiled widely and the lock status on the door is open. program find out smile. It still tested with one single face in
webcam, face detection limited to 1 face for each frame.
Then, if one smile detected, the other smiles ignored, and the
program will stop detect any more face for 90 frames (3
seconds). Although the accuracy of the face recognition is
62.7% with our dataset, it can be improved by adding more
datasets or using deep learning to train recognizer model. In
the future research, we hope can make more better accuracy
with bigger training dataset and more better algorithms.

ACKNOWLEDGMENT
We would like to thank our fellow researchers at Bina
Nusantara University for providing the opportunity and time
Fig. 6. Smile detected and “Door Unloked” message to complete this Research Project with the topic Face-
Recognition-Based Smart Gate using Haar Cascade and
OpenCV LBPH Face Recognizer, which also helped us to
In Figure 7, the user can enter after the program reads the learn many things.
smile on the user's face and processes the unlock on the
door. We limit the open door time to be locked
automatically after 15 seconds the door is unlocked. REFERENCES

[1] S. Stoimenov, G. T. Tsenov, and V. M. Mladenov,

“Face recognition system in Android using neural
networks,” in 2016 13th Symposium on Neural
Networks and Applications (NEUREL), Nov. 2016,
pp. 1–4, doi: 10.1109/NEUREL.2016.7800138.
[2] Y. Li, Y. Li, K. Xu, Q. Yan, and R. H. Deng,
“Empirical Study of Face Authentication Systems
Under OSNFD Attacks,” IEEE Trans. Dependable
Secur. Comput., vol. 15, no. 2, pp. 231–245, Mar.
2018, doi: 10.1109/TDSC.2016.2550459.
[3] K. Adalan and B. Erkmen, “Face recognition, NFC
and voice controlled door lock system,” in 2016
Fig. 7. Door unlocked for 15 seconds

Authorized licensed use limited to: Modern Education Society's Wadia College of Engineering. Downloaded on July 22,2024 at 10:20:38 UTC from IEEE Xplore. Restrictions apply.
509
 National Conference on Electrical, Electronics and [13] K. Goyal, K. Agarwal, and R. Kumar, “Face
Biomedical Engineering (ELECO), 2016, pp. 696– detection and tracking: Using OpenCV,” in 2017
700. International conference of Electronics,
[4] V. Babu D, V. Reddy, G. Devi, A. Prakash, T. Communication and Aerospace Technology
Coumaressin, and S. Begum, “Facial Recognition (ICECA), Apr. 2017, vol. 1, pp. 474–478, doi:
Based Upon the Domestic Security System Using 10.1109/ICECA.2017.8203730.
IOT,” J. Nucl. Energy Sci. Power Gener. Technol., [14] Q. Chen and L. Sang, “Face-mask recognition for
vol. 10, pp. 1–4, 2021. fraud prevention using Gaussian mixture model,” J.
[5] N. R. S, R. Venkatasamy, J. A. Dhanraj, S. Vis. Commun. Image Represent., vol. 55, pp. 795–
Aravinth, K. Balachandar, and D. N, “Design and 801, Aug. 2018, doi: 10.1016/j.jvcir.2018.08.016.
Development of IOT based Smart Door Lock [15] A. Cabani, K. Hammoudi, H. Benhabiles, and M.
System,” in 2022 Third International Conference on Melkemi, “MaskedFace-Net – A dataset of
Intelligent Computing Instrumentation and Control correctly/incorrectly masked face images in the
Technologies (ICICICT), Aug. 2022, pp. 1525– context of COVID-19,” Smart Heal., vol. 19, p.
1528, doi: 10.1109/ICICICT54557.2022.9917767. 100144, Mar. 2021, doi:
[6] K. Sethi, S. Kaul, I. Patel, and S. R., “FaceLock 10.1016/j.smhl.2020.100144.
Homes: A Contactless Smart Home Security System [16] A. Jain and A. Shukla, “Password Protected Home
to Prevent COVID Transmission,” 2021 Sixth Int. Automation System with Automatic Door Lock,”
Conf. Wirel. Commun. Signal Process. Netw., pp. MIT Int. J. Electr. Instrum. Eng., vol. 6, no. 1, pp.
75–79, 2021, doi: 10.24507/icicelb.12.10.965. 28–31, 2016.
[7] O. B. Doshi, H. N. Bendale, A. M. Chavan, and S. [17] C.-H. Hung, Y.-W. Bai, and J.-H. Ren, “Design and
S. More, “A Smart Door Lock Security System implementation of a door lock control based on a
using Internet of Things,” in 2022 International near field communication of a smartphone,” in 2015
Conference on Applied Artificial Intelligence and IEEE International Conference on Consumer
Computing (ICAAIC), 2022, pp. 1457–1463, doi: Electronics - Taiwan, Jun. 2015, pp. 45–46, doi:
10.1109/ICAAIC53929.2022.9792831. 10.1109/ICCE-TW.2015.7216992.
[8] W. Xie, Q. Zhang, and J. Zhang, “Acoustic-Based [18] R. Saputra and N. Surantha, “Smart and real-time
Upper Facial Action Recognition for Smart door lock system for an elderly user based on face
Eyewear,” Proc. ACM Interact. Mob. Wearable recognition,” Bull. Electr. Eng. Informatics, vol. 10,
Ubiquitous Technol., vol. 5, no. 2, Jun. 2021, doi: no. 3, pp. 1345–1355, Jun. 2021, doi:
10.1145/3448105. 10.11591/eei.v10i3.2955.
[9] N. K. Mishra, M. Dutta, and S. K. Singh, [19] A. H. Ahmad et al., “Real time face recognition of
“Multiscale parallel deep CNN (mpdCNN) video surveillance system using haar cascade
architecture for the real low-resolution face classifier,” Indones. J. Electr. Eng. Comput. Sci.,
recognition for surveillance,” Image Vis. Comput., vol. 21, no. 3, pp. 1389–1399, 2021, doi:
vol. 115, p. 104290, 2021, doi: 10.11591/ijeecs.v21.i3.pp1389-1399.
https://doi.org/10.1016/j.imavis.2021.104290. [20] D. Syrlybayev, N. Nauryz, A. Seisekulova, K.
[10] D. K. Stephanos, G. Husari, B. T. Bennett, and E. Yerzhanov, and M. H. Ali, “Smart Door for COVID
Stephanos, “Machine Learning Predictive Analytics Restricted Areas,” Procedia Comput. Sci., vol. 201,
for Player Movement Prediction in NBA: no. C, pp. 478–486, 2022, doi:
Applications, Opportunities, and Challenges,” in 10.1016/j.procs.2022.03.062.
Proceedings of the 2021 ACM Southeast [21] N. Y. L. Venkata, C. Rupa, B. Dharmika, T. G.
Conference, 2021, pp. 2–8, doi: Nithin, and N. Vineela, “Intelligent Secure Smart
10.1145/3409334.3452064. Locking System using Face Biometrics,” in 2021
[11] A. K. Chaitanya, C. H. Kartheek, and D. Nandan, International Conference on Recent Trends on
“Study on Real-Time Face Recognition and Electronics, Information, Communication &
Tracking for Criminal Revealing,” in Soft Technology (RTEICT), Aug. 2021, pp. 268–273,
Computing: Theories and Applications, 2020, pp. doi: 10.1109/RTEICT52294.2021.9573869.
849–857. [22] M. G. Sarwar, A. Dey, and A. Das, “Developing a
[12] B. A. JABAR, R. YUNANDA, A. A. S. LBPH-based Face Recognition System for Visually
GUNAWAN, and others, “The development of a Impaired People,” in 2021 1st International
smart door decision system, based on pir sensor, Conference on Artificial Intelligence and Data
embedded face recognition and server request using Analytics (CAIDA), Apr. 2021, pp. 286–289, doi:
ttgo esp 32,” ICIC express Lett. Part B, Appl. an Int. 10.1109/CAIDA51941.2021.9425275.
J. Res. Surv., vol. 12, no. 10, pp. 965–970, 2021.

Authorized licensed use limited to: Modern Education Society's Wadia College of Engineering. Downloaded on July 22,2024 at 10:20:38 UTC from IEEE Xplore. Restrictions apply.
510