Android - Detect Open or Closed Eyes Using ML Kit and CameraX
Last Updated :
21 Mar, 2024
Google's ML kit is one of the best-trained models for face detection and its characteristics. Integrating with your own CameraX library can be quite a challenging task. so we are going to build an Android app that will detect whether a person's eyes are open or closed in real time. This process going to be long so without delay let's deep dive into the project. A sample video is given below to get an idea about what we are going to do in this article.
Note: Before starting the project please read about how the ML Kit detects faces and how cameraX works.
Project Setup
- Start a project with an empty activity and name your project whatever you want we are naming it EyeDetection.
- Language using Kotlin
- Minimum SDK set to Android 7.0(Nougat)
Step by Step Implementation
Adding Dependencies and Permissions
Open project-level build.gradle file, make sure to include Google's Maven repository in both your buildscript and all projects sections. Add the dependencies for the ML Kit Android libraries to the module's app-level gradle file, which is usually app/build.gradle.
dependencies {
// This dependency will dynamically download the model in Google Play Services
implementation 'com.google.android.gms:play-services-mlkit-face-detection:17.1.0'
// camera dependencies
def camerax_version = "1.2.2"
implementation "androidx.camera:camera-core:${camerax_version}"
implementation "androidx.camera:camera-camera2:${camerax_version}"
implementation "androidx.camera:camera-lifecycle:${camerax_version}"
implementation "androidx.camera:camera-view:${camerax_version}"
}Add this code in your manifest file
<uses-feature android:name="android.hardware.camera.any" />
<uses-permission android:name="android.permission.CAMERA" />
Add the following declaration to your app's AndroidManifest.xml file. This will automatically download the model to the device if your app is installed from the Play Store.
<application ...>
...
<meta-data
android:name="com.google.mlkit.vision.DEPENDENCIES"
android:value="face" >
<!-- To use multiple models: android:value="face,model2,model3" -->
</application>
We are going to use viewbinding so don’t forget to enable it
buildFeatures {
viewBinding true
}Configure the Layout
Open the activity_main layout file at res/layout/activity_main.xml, and replace it with the following code.
XML
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<FrameLayout
android:id="@+id/frameLayout"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toTopOf="parent">
<androidx.camera.view.PreviewView
android:id="@+id/previewView_finder"
android:layout_width="match_parent"
android:layout_height="match_parent"
app:scaleType="fillCenter">
</androidx.camera.view.PreviewView>
<com.example.eyedetection.GraphicOverlay
android:id="@+id/graphicOverlay_finder"
android:layout_width="match_parent"
android:layout_height="match_parent" />
</FrameLayout>
<TextView
android:id="@+id/tvWarningText"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_marginBottom="150dp"
android:background="@color/white"
android:focusableInTouchMode="false"
android:gravity="center"
android:padding="20dp"
android:text="No Face detected"
android:textColor="@android:color/holo_red_dark"
android:textSize="18sp"
android:visibility="visible"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>
This code includes the PreviewView for preview of cameraX that will let the user to preview the photo they will be taking. A textview for the indication of face detection and about eyes, whether they are open or closed and GraphicOverlay to draw the box on detected faces.
Note: This GrapichOverlay view is a custom view, that you have to create a first.
Making Classes
Create a GraphicOverlay class and make It open. We have to define some methods and logics to draw over the screen. Below is the code.
Kotlin
import android.content.Context
import android.content.res.Configuration
import android.graphics.Canvas
import android.graphics.Rect
import android.graphics.RectF
import android.util.AttributeSet
import android.view.View
import kotlin.math.ceil
open class GraphicOverlay(context: Context?, attrs: AttributeSet?) :
View(context, attrs) {
private val lock = Any()
private val faceBoxes: MutableList<FaceBox> = ArrayList()
var mScale: Float? = null
var mOffsetX: Float? = null
var mOffsetY: Float? = null
abstract class FaceBox(private val overlay: GraphicOverlay) {
abstract fun draw(canvas: Canvas?)
fun calculateRect(height: Float, width: Float, boundingBoxT: Rect): RectF {
// for land scape
fun isLandScapeMode(): Boolean {
return overlay.context.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE
}
fun whenLandScapeModeWidth(): Float {
return when(isLandScapeMode()) {
true -> width
false -> height
}
}
fun whenLandScapeModeHeight(): Float {
return when(isLandScapeMode()) {
true -> height
false -> width
}
}
val scaleX = overlay.width.toFloat() / whenLandScapeModeWidth()
val scaleY = overlay.height.toFloat() / whenLandScapeModeHeight()
val scale = scaleX.coerceAtLeast(scaleY)
overlay.mScale = scale
// Calculate offset (we need to center the overlay on the target)
val offsetX = (overlay.width.toFloat() - ceil(whenLandScapeModeWidth() * scale)) / 2.0f
val offsetY = (overlay.height.toFloat() - ceil(whenLandScapeModeHeight() * scale)) / 2.0f
overlay.mOffsetX = offsetX
overlay.mOffsetY = offsetY
val mappedBox = RectF().apply {
left = boundingBoxT.right * scale + offsetX
top = boundingBoxT.top * scale + offsetY
right = boundingBoxT.left * scale + offsetX
bottom = boundingBoxT.bottom * scale + offsetY
}
return mappedBox
}
}
fun clear() {
synchronized(lock) { faceBoxes.clear() }
postInvalidate()
}
fun add(faceBox: FaceBox) {
synchronized(lock) { faceBoxes.add(faceBox) }
}
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
synchronized(lock) {
for (graphic in faceBoxes) {
graphic.draw(canvas)
}
}
}
}
To handle the camera we will create a cameraManager class to start the camera. Here we have not implemented the Picture taking ability as we are only doing real time detection so only preview will enough for us. CameraManager class takes five parameters context, previewView for the Preview, lifecycleOwner for the life cycle of camera. Listener to get the Status(which is an another class to track the status and change the textview). And graphicOverlay. For the analyzer we are using custom analyzer for face detection and draw boxes. Here is the code of cameraManager class.
Kotlin
import android.content.Context
import android.util.Log
import androidx.camera.core.Camera
import androidx.camera.core.CameraSelector
import androidx.camera.core.ImageAnalysis
import androidx.camera.core.ImageCapture
import androidx.camera.core.Preview
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.camera.view.PreviewView
import androidx.core.content.ContextCompat
import androidx.lifecycle.LifecycleOwner
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors
class CameraManager(
private val context: Context,
private val previewView: PreviewView,
private val lifecycleOwner: LifecycleOwner,
private val graphicOverlay: GraphicOverlay,
private val listener: ((Status) -> Unit),
) {
private var preview: Preview? = null
private var imageCapture: ImageCapture? = null
private var camera: Camera? = null
private lateinit var cameraExecutor: ExecutorService
private var cameraSelectorOption = CameraSelector.LENS_FACING_BACK
private var cameraProvider: ProcessCameraProvider? = null
private var imageAnalyzer: ImageAnalysis? = null
init {
createNewExecutor()
}
private fun createNewExecutor() {
cameraExecutor = Executors.newSingleThreadExecutor()
}
fun startCamera() {
val cameraProviderFuture = ProcessCameraProvider.getInstance(context)
cameraProviderFuture.addListener(
{
cameraProvider = cameraProviderFuture.get()
preview = Preview.Builder().build()
imageCapture = ImageCapture.Builder().build()
imageAnalyzer = ImageAnalysis.Builder()
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build().also {
it.setAnalyzer(cameraExecutor, selectAnalyzer())
}
val cameraSelector =
CameraSelector.Builder().requireLensFacing(cameraSelectorOption).build()
setCameraConfig(cameraProvider, cameraSelector)
}, ContextCompat.getMainExecutor(context)
)
}
// Custom analyzer
private fun selectAnalyzer(): ImageAnalysis.Analyzer {
return FaceDetection(graphicOverlay, listener)
}
private fun setCameraConfig(
cameraProvider: ProcessCameraProvider?, cameraSelector: CameraSelector
) {
try {
cameraProvider?.unbindAll()
camera = cameraProvider?.bindToLifecycle(
lifecycleOwner, cameraSelector, preview, imageCapture, imageAnalyzer
)
preview?.setSurfaceProvider(
previewView.surfaceProvider
)
} catch (e: Exception) {
Log.e("Error", "Use case binding failed", e)
}
}
}
This is the Status class to get the indication call back directly in our activity. that will be helpful to get the status and change the textview according to analyzer.
Kotlin
enum class Status { NO_FACE, MULTIPLE_FACES, LEFT_EYE_CLOSED,
RIGHT_EYE_CLOSED, BOTH_EYES_CLOSED,VALID_FACE
}
This is the face detection class that will be used by our custom analyzer. Which is inheriting the Analyzer class. And overriding its methods. The code doesn’t draw the box if multiple faces detected but will notifiy the main activity using the listener. And if it is a a single face , we will pass its properties to FaceBox class to draw the box and for eye detection.
Kotlin
import android.graphics.Rect
import android.util.Log
import androidx.camera.core.ImageProxy
import com.google.android.gms.tasks.Task
import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.face.Face
import com.google.mlkit.vision.face.FaceDetection
import com.google.mlkit.vision.face.FaceDetectorOptions
import java.io.IOException
class FaceDetection(
private val graphicOverlayView: GraphicOverlay,
private val listener: (Status) -> Unit
) : Analyzer<List<Face>>() {
private val realTimeOpts =
FaceDetectorOptions.Builder().setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
.setContourMode(FaceDetectorOptions.CONTOUR_MODE_ALL)
.setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL).build()
private val detector = FaceDetection.getClient(realTimeOpts)
override val graphicOverlay: GraphicOverlay
get() = graphicOverlayView
override fun detectInImage(image: InputImage): Task<List<Face>> {
return detector.process(image)
}
override fun stop() {
try {
detector.close()
} catch (e: IOException) {
Log.e("Error", "Exception thrown while trying to close Face Detector: $e")
}
}
override fun onSuccess(
results: List<Face>, graphicOverlay: GraphicOverlay, rect: Rect, imageProxy: ImageProxy
) {
graphicOverlay.clear()
// If multiple faces then don't draw
if (results.isNotEmpty()) {
if (results.size > 1) {
listener(Status.MULTIPLE_FACES)
} else {
for (face in results) {
val faceGraphic =
FaceBox(graphicOverlay, face, rect, listener)
graphicOverlay.add(faceGraphic)
}
}
graphicOverlay.postInvalidate()
} else {
listener(Status.NO_FACE)
Log.e("Error", "Face Detector failed.")
}
}
override fun onFailure(e: Exception) {
Log.e("Error", "Face Detector failed. $e")
listener(Status.NO_FACE)
}
}
The faceBox class inherting the Graphic overlay class. This class will calculate the probability of eyes of this face and also put the green color box to the detected face. We have set the probability to 0.6. if any eye probability is less than or equal to 0.6 we will consider that eye as closed eye and set status accordingly. Here is code of FaceBox class.
Kotlin
import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Paint
import android.graphics.Rect
import com.google.mlkit.vision.face.Face
class FaceBox(
overlay: GraphicOverlay,
private val face: Face,
private val imageRect: Rect,
private val listener: (Status) -> Unit
) : GraphicOverlay.FaceBox(overlay) {
private val facePositionPaint: Paint
private val idPaint: Paint
private val boxPaint: Paint
init {
val selectedColor = Color.WHITE
facePositionPaint = Paint()
facePositionPaint.color = selectedColor
idPaint = Paint()
idPaint.color = selectedColor
boxPaint = Paint()
boxPaint.color = selectedColor
boxPaint.style = Paint.Style.STROKE
boxPaint.strokeWidth = 5.0f
}
private val greenBoxPaint = Paint().apply {
color = Color.GREEN
style = Paint.Style.STROKE
strokeWidth = 5.0f
}
override fun draw(canvas: Canvas?) {
val rect = calculateRect(
imageRect.height().toFloat(), imageRect.width().toFloat(), face.boundingBox
)
val leftEyeProbability = leftEyeProbability()
val rightEyeProbability = rightEyeProbability()
when {
// both eyes are closed
leftEyeProbability <= 0.6 && rightEyeProbability() <= 0.6 -> {
listener(Status.BOTH_EYES_CLOSED)
}
// left eye is closed
leftEyeProbability <= 0.6 -> {
listener(Status.LEFT_EYE_CLOSED)
}
// right is closed
rightEyeProbability <= 0.6 -> {
listener(Status.RIGHT_EYE_CLOSED)
}
// valid face, set face box color green
else -> {
listener(Status.VALID_FACE)
canvas?.drawRect(rect, greenBoxPaint)
}
}
}
private fun leftEyeProbability(): Float {
var probability = 0.0F
if (face.leftEyeOpenProbability != null) {
val leftEyeOpenProb = face.leftEyeOpenProbability
probability = leftEyeOpenProb!!
}
return probability
}
private fun rightEyeProbability(): Float {
var probability = 0.0F
if (face.rightEyeOpenProbability != null) {
val rightEyeOpenProb = face.rightEyeOpenProbability
probability = rightEyeOpenProb!!
}
return probability
}
}
We are almost ready. Before starting the app we have to add the camera permission and handle it accordingly. And after handling the permission in our activity out code will look like this.
Kotlin
import android.Manifest
import android.content.ActivityNotFoundException
import android.content.Intent
import android.content.pm.PackageManager
import android.net.Uri
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import android.provider.Settings
import android.util.Log
import androidx.activity.result.contract.ActivityResultContracts
import androidx.appcompat.app.AlertDialog
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import com.example.eyedetection.databinding.ActivityMainBinding
class MainActivity : AppCompatActivity() {
private lateinit var binding: ActivityMainBinding
private lateinit var cameraManager: CameraManager
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
binding = ActivityMainBinding.inflate(layoutInflater)
setContentView(binding.root)
createCameraManager()
// to handle the permission
cameraPermission()
}
private fun openCamera() {
// this will start the camera if permission is enabled
cameraManager.startCamera()
}
private fun cameraPermission() {
val cameraPermission = Manifest.permission.CAMERA
if (ContextCompat.checkSelfPermission(
this, cameraPermission
) == PackageManager.PERMISSION_GRANTED
) {
openCamera()
} else if (ActivityCompat.shouldShowRequestPermissionRationale(
this, cameraPermission
)
) {
val title = "Permission Required"
val message = "App needs Camera Permission to detect faces"
val builder: AlertDialog.Builder = AlertDialog.Builder(this)
builder.setTitle(title).setMessage(message).setCancelable(false)
.setPositiveButton("OK") { dialog, _ ->
requestCameraPermissionLauncher.launch(cameraPermission)
dialog.dismiss()
}.setNegativeButton("Cancel") { dialog, _ ->
dialog.dismiss()
}
builder.create().show()
} else {
requestCameraPermissionLauncher.launch(
cameraPermission
)
}
}
private val requestCameraPermissionLauncher = registerForActivityResult(
ActivityResultContracts.RequestPermission()
) { isGranted: Boolean ->
if (isGranted) {
openCamera()
} else if (!ActivityCompat.shouldShowRequestPermissionRationale(
this, Manifest.permission.CAMERA
)
) {
val title = "Permission required"
val message =
"Please allow camera permission to detect faces"
val builder: AlertDialog.Builder = AlertDialog.Builder(this)
builder.setTitle(title).setMessage(message).setCancelable(false)
.setPositiveButton("Change Settings") { _, _ ->
try {
val intent = Intent(Settings.ACTION_APPLICATION_DETAILS_SETTINGS)
val uri = Uri.fromParts("package", this.packageName, null)
intent.data = uri
startActivity(intent)
} catch (e: ActivityNotFoundException) {
e.printStackTrace()
}
}.setNegativeButton("Cancel") { dialog, _ ->
dialog.dismiss()
}
builder.create().show()
} else {
cameraPermission()
}
}
private fun createCameraManager() {
cameraManager = CameraManager(
this,
binding.previewViewFinder,
this,
binding.graphicOverlayFinder,
::checkStatus
)
}
private fun checkStatus(status: Status) {
Log.e("status","$status")
when (status) {
Status.MULTIPLE_FACES -> {
binding.tvWarningText.text = "Multiple Faces detected"
}
Status.NO_FACE -> {
binding.tvWarningText.text = "No Face detected"
}
Status.LEFT_EYE_CLOSED -> {
binding.tvWarningText.text = "Left eye is closed"
}
Status.RIGHT_EYE_CLOSED -> {
binding.tvWarningText.text ="Right eye is closed"
}
Status.BOTH_EYES_CLOSED->{
binding.tvWarningText.text = "Both Eyes are closed"
}
Status.VALID_FACE ->{
binding.tvWarningText.text ="Correct Face"
}
}
}
}
checkStatus function will listen to the listener and set the text of textview as soon as any face or any changes in face is detected. And now we are finally ready. Build the app and run in your device. Our final result will be look like this.
Output:
Similar Reads
How to Create Custom Camera using CameraX in Android?
CameraX is used to create a custom camera in the app. CameraX is a Jetpack support library, built to help you make camera app development easier. A sample video is given below to get an idea about what we are going to do in this article. Note that we are going to implement this project using Java/Ko
6 min read
Text Detector in Android using Firebase ML Kit
Nowadays many apps using Machine Learning inside their apps to make most of the tasks easier. We have seen many apps that detect text from any image. This image may include number plates, images, and many more. In this article, we will take a look at the implementation of Text Detector in Android us
6 min read
How to create a Face Detection Android App using Machine Learning KIT on Firebase
Pre-requisites:Firebase Machine Learning kitAdding Firebase to Android AppFirebase ML KIT aims to make machine learning more accessible, by providing a range of pre-trained models that can use in the iOS and Android apps. Let's use ML Kit’s Face Detection API which will identify faces in photos. By
8 min read
How to Load an Image using OpenCV in Android?
OpenCV (Open Source Computer Vision Library) is an open-source computer vision and machine learning software library which is used for image and video processing. In this article, we are going to build an application that shows the demonstration of how we can load images in OpenCV on Android. Mat Cl
4 min read
Detect Screen Orientation in Android using Jetpack Compose
In Android, Screen Orientation is an important aspect where the user would like to run an activity in Fullscreen or a Wide Landscaped mode. Most commonly running applications that can switch between portrait and landscape mode can be Image Viewers, Video Players, Web Browsers, etc. In such applicati
2 min read
How to Get Screen Width and Height in Android using Jetpack Compose?
Android applications are being developed for different device orientations to support a huge range of devices. So that users with different device size configurations can use the application. Many applications need to get the height and width of the device screen to create UI. In this article, we wi
5 min read
Generate QR Code in Android using Kotlin
Many applications nowadays use QR codes within their application to hide some information. QR codes are seen within many payment applications which are used by the users to scan and make payments. The QR codes which are seen within these applications are generated inside the application itself. In t
4 min read
How to Open Camera Through Intent and Display Captured Image in Android?
The purpose of this article is to show how to open a Camera from inside an App and click the image and then display this image inside the same app. An android application has been developed in this article to achieve this. The opening of the Camera from inside our app is achieved with the help of th
6 min read
How to Read QR Code using CAMView Library in Android?
CAMView Library is a simple solution for accessing users' device camera. By using this library we can access users' cameras and use to perform so many functions of the camera such as scanning barcodes that are done by using a built-in ZXing decoding engine. This library contains a set of components
5 min read
Detect Different Types of Touch Gestures in Android using Jetpack Compose
Most of the smartphones of this generation have a touchscreen which serves as a medium for input as well as output. Users can click the screen for giving inputs to the device and the same screen is used for displaying the output for the given action. A user may touch different points on the screen a
3 min read