Eye blink detection with OpenCV, Python, and dlib
Last Updated :
03 Jan, 2023
In this article, we are going to see how to detect eye blink using OpenCV, Python, and dlib. This is a fairly simple task and it requires you to have a basic understanding of OpenCV and how to implement face landmark detection programs using OpenCV and dlib, since we'll be using that as the base for today's project.
Stepwise Implementation
Step 1: Installing all required packages
So we'll install all our dependencies in this step. We're going to use OpenCV for computer vision, dlib library for facial recognition, and also the imutils package to use some functions that will help us convert the landmarks to NumPy array and make it easy for us to use, so let's install these first.
pip install opencv-python numpy dlib imutils
Step 2: Initialize and read from the webcam
Python3
import cv2
cam = cv2.VideoCapture(0)
while True:
_, frame = cam.read()
cv2.imshow('Camera Feed', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cam.release()
Step 3: Facial Landmark Detection using dlib
Note: The facial landmark detector included in the dlib library is an implementation of the One Millisecond Face Alignment with an Ensemble of Regression Trees paper by Kazemi and Sullivan (2014).
Facial landmarks are the key attributes of a face in an image like eyes, eyebrows, nose, mouth, and Jaw. Since Steps 1 - 3 is not the primary focus of this article so we won't go in-depth, but instead, I'll write comments on the code for easy understanding.
Here is the code basic code for facial landmark detection, that we'll be using later for eye blink detection.
Python3
# Importing the required dependencies
import cv2 # for video rendering
import dlib # for face and landmark detection
import imutils
# for calculating dist b/w the eye landmarks
from scipy.spatial import distance as dist
# to get the landmark ids of the left
# and right eyes ----you can do this
# manually too
from imutils import face_utils
cam = cv2.VideoCapture('assets/Video.mp4')
# Initializing the Models for Landmark and
# face Detection
detector = dlib.get_frontal_face_detector()
landmark_predict = dlib.shape_predictor(
'Model/shape_predictor_68_face_landmarks.dat')
while 1:
# If the video is finished then reset it
# to the start
if cam.get(cv2.CAP_PROP_POS_FRAMES) == cam.get(
cv2.CAP_PROP_FRAME_COUNT):
cam.set(cv2.CAP_PROP_POS_FRAMES, 0)
else:
_, frame = cam.read()
frame = imutils.resize(frame, width=640)
# converting frame to gray scale to pass
# to detector
img_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# detecting the faces---#
faces = detector(img_gray)
for face in faces:
cv2.rectangle(frame, face[0], face[1],
(200, 0, 0), 1)
cv2.imshow("Video", frame)
if cv2.waitKey(5) & 0xFF == ord('q'):
break
cam.release()
cv2.destroyAllWindows()
Now the question arises, how are we going to use these landmarks for eye detection.
Eye Landmarks
We saw that we can extract any facial structure from the 68 Facial Landmarks that we detected. So, we'll extract the landmarks of the eyes i.e 6 (x,y) coordinates for each eye, for any given face in an image. And then we'll calculate the EAR for these landmarks.
Eye Aspect Ratio (EAR)
This method is very simple, efficient, and doesn't require anything like image processing. Basically, this ratio gives us a certain relation between the horizontal and vertical measurements of the eye. This is the equation to calculate the EAR using the six parameters of the eye :
image created by the author using canvas
We can use the given function to calculate the EAR :
Python3
def calculate_EAR(eye):
# calculate the vertical distances
# euclidean distance is basically
# the same when you calculate the
# hypotenuse in a right triangle
y1 = dist.euclidean(eye[1], eye[5])
y2 = dist.euclidean(eye[2], eye[4])
# calculate the horizontal distance
x1 = dist.euclidean(eye[0], eye[3])
# calculate the EAR
EAR = (y1+y2) / x1
return EAR
What's so magical about this EAR?
This is the most important part, when you calculate the EAR of an eye, it remains constant when the eye is open but it suddenly drops when the eye is blinked. Below, I have shown a graph to show it's working:
. ~image by the author using canvas
As you can see in the image the overall value of EAR was constant throughout except at one point i.e when the eye is blinked, making it one of the most simple and most efficient ways of detecting an eye blink.
Since we have two EAR for each eye respectively we'll take the average of both the EAR for the right eye and the EAR for the left eye and then check if it is lower than a certain threshold ( we'll create a variable to set its value) and this threshold might vary a bit, for me it worked with 0.4 or 0.5 but in some cases, it works with 0.25 or 0.3 as well. It depends on the FPS of your video or webcam.
Next: We'll keep the count of the frames when the EAR is lower than the threshold and if the count is 3 (or 5 depending on the fps) frames then we'll consider a blink detected.
Below is the full implementation
Python3
# Importing the required dependencies
import cv2 # for video rendering
import dlib # for face and landmark detection
import imutils
# for calculating dist b/w the eye landmarks
from scipy.spatial import distance as dist
# to get the landmark ids of the left and right eyes
# you can do this manually too
from imutils import face_utils
# from imutils import
cam = cv2.VideoCapture('assets/my_blink.mp4')
# defining a function to calculate the EAR
def calculate_EAR(eye):
# calculate the vertical distances
y1 = dist.euclidean(eye[1], eye[5])
y2 = dist.euclidean(eye[2], eye[4])
# calculate the horizontal distance
x1 = dist.euclidean(eye[0], eye[3])
# calculate the EAR
EAR = (y1+y2) / x1
return EAR
# Variables
blink_thresh = 0.45
succ_frame = 2
count_frame = 0
# Eye landmarks
(L_start, L_end) = face_utils.FACIAL_LANDMARKS_IDXS["left_eye"]
(R_start, R_end) = face_utils.FACIAL_LANDMARKS_IDXS['right_eye']
# Initializing the Models for Landmark and
# face Detection
detector = dlib.get_frontal_face_detector()
landmark_predict = dlib.shape_predictor(
'Model/shape_predictor_68_face_landmarks.dat')
while 1:
# If the video is finished then reset it
# to the start
if cam.get(cv2.CAP_PROP_POS_FRAMES) == cam.get(
cv2.CAP_PROP_FRAME_COUNT):
cam.set(cv2.CAP_PROP_POS_FRAMES, 0)
else:
_, frame = cam.read()
frame = imutils.resize(frame, width=640)
# converting frame to gray scale to
# pass to detector
img_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# detecting the faces
faces = detector(img_gray)
for face in faces:
# landmark detection
shape = landmark_predict(img_gray, face)
# converting the shape class directly
# to a list of (x,y) coordinates
shape = face_utils.shape_to_np(shape)
# parsing the landmarks list to extract
# lefteye and righteye landmarks--#
lefteye = shape[L_start: L_end]
righteye = shape[R_start:R_end]
# Calculate the EAR
left_EAR = calculate_EAR(lefteye)
right_EAR = calculate_EAR(righteye)
# Avg of left and right eye EAR
avg = (left_EAR+right_EAR)/2
if avg < blink_thresh:
count_frame += 1 # incrementing the frame count
else:
if count_frame >= succ_frame:
cv2.putText(frame, 'Blink Detected', (30, 30),
cv2.FONT_HERSHEY_DUPLEX, 1, (0, 200, 0), 1)
else:
count_frame = 0
cv2.imshow("Video", frame)
if cv2.waitKey(5) & 0xFF == ord('q'):
break
cam.release()
cv2.destroyAllWindows()
Output:
If you're using a different video or if you're using a webcam you're FPS is going to be different, so might wanna trying changing the values of the variables we defined, although they work fine in most cases.
Similar Reads
White and black dot detection using OpenCV | Python
Image processing using Python is one of the hottest topics in today's world. But image processing is a bit complex and beginners get bored in their first approach. So in this article, we have a very basic image processing python program to count black dots in white surface and white dots in the blac
4 min read
Detect an object with OpenCV-Python
Object detection refers to identifying and locating objects within images or videos. OpenCV provides a simple way to implement object detection using Haar Cascades a classifier trained to detect objects based on positive and negative images. In this article we will focus on detecting objects using i
4 min read
Feature detection and matching with OpenCV-Python
In this article, we are going to see about feature detection in computer vision with OpenCV in Python. Feature detection is the process of checking the important features of the image in this case features of the image can be edges, corners, ridges, and blobs in the images. In OpenCV, there are a nu
5 min read
Detecting ArUco markers with OpenCV and Python
ArUco markers are widely used in computer vision applications for tasks such as camera calibration, pose estimation, and augmented reality. These markers are square fiducial markers with a unique binary pattern that can be easily detected by computer vision algorithms. In this article, we will explo
6 min read
Detect and Read Barcodes with OpenCV in Python
A barcode is a graphical representation of data that is machine-readable. It consists of parallel lines or rectangles of varying widths and spacings, along with specific patterns, that encode information. Barcodes are widely used for automatic identification and tracking of products, assets, invento
3 min read
Yawn Detection using OpenCV and Dlib
In this article, we'll cover all the steps required to build a Yawn detection program using OpenCV and dlib packages. But before doing this project should be familiar with the basics of OpenCV and you should also know how to use face detection and landmark detection using dlib module. Requirements:
2 min read
Face Detection using Python and OpenCV with webcam
Face detection is a important application of computer vision that involves identifying human faces in images or videos. In this Article, we will see how to build a simple real-time face detection application using Python and OpenCV where webcam will be used as the input source.Step 1: Installing Ope
3 min read
Face Alignment with OpenCV and Python
Face Alignment is the technique in which the image of the person is rotated according to the angle of the eyes. Â This technique is actually used as a part of the pipeline process in which facial detection is done using the image. This implementation of face alignment can be easily done with the help
6 min read
Convert BGR and RGB with Python - OpenCV
Prerequisites: OpenCV OpenCV is a huge open-source library for computer vision, machine learning, and image processing. OpenCV supports a wide variety of programming languages like Python, C++, Java, etc. It can process images and videos to identify objects, faces, or even the handwriting of a human
2 min read
Line detection in python with OpenCV | Houghline method
The Hough Transform is a method that is used in image processing to detect any shape, if that shape can be represented in mathematical form. It can detect the shape even if it is broken or distorted a little bit.We will see how Hough transform works for line detection using the HoughLine transform m
6 min read