Project report on

Software to Manipulate Cursor Using Machine Learning

Supervised by
Rubya Shaharin
Assistant professor
Dept. of Computer Science and Engineering Jatiya
Kabi Kazi Narul Islam University

Submitted by
Anik Kumar Singh
Roll No.: 18102018
Zasmin Nahar Jui
Roll No.: 19102030
Dept. of Computer Science and Engineering Jatiya
Kabi Kazi Narul Islam University

Software Development Project

Course code: CSE 400(B): Project/Thesis

Department of Computer Science and Engineering

JATIYA KABI KAZI NAZRUL ISLAM UNIVERSITY
TRISHAL, MYMENSINGH
PROJECT SUBMISSION YEAR, 2022
 CANDIDATE’S DECLARATION
We declare that this project is our own work and has not been submitted in any other form for
another degree or diploma at any university of other institute of tertiary education. Information
derived from the published and unpublished work of others has been acknowledged in the text
and a list of references is given.

Date: 25-01-2024 Name and Signature of the Candidates:

.............................................
Anik Kumar Singh

.............................................
Zasmin Nahar Jui

2
 Certification
The undersigned have examined the report entitled “Software to Manipulate Cursor Using

Machine Learning” presented by Anik Kumar Singh & Zasmin Nahar Jui, for the course CSE400(B)
(Software Development Project) and hereby certify that it is worthy of acceptance.

Date Advisor’s signature

3
 Acknowledgement
A project is a golden opportunity for learning and self-development. Many people deserve our
cordial thanks for their help in completing this project. First and foremost, we would like to thank
our honorable Supervisor Rubya Shahrin, Assistant Professor, Dept. of CSE, JKKNIU who is so
dedicated to help us in our times of need. We are really honored to make this project under her
guidance that provides us her valuable time and knowledge, motivating thought and
encouragement.

We wish to express thanks to Prof. Dr. Md. Sujan Ali, Head, Department of Computer Science
& Engineering, Jatiya Kabi Kazi Nazrul Islam University for providing us lab opportunities with
lab materials related to our project work.

Finally, we express our thanks to the Department of Computer Science and Engineering for
giving us the opportunity to study here and supporting us greatly.

4
 Abstract

Human-computer interaction is an interface that allows us to easily connect virtually with

technology today. Using hand motion recognition, we are attempting to carry out mouse
activities here. Based on skin detection technology, the project primarily targets mouse cursor
movements and click events. It is a real-time system that works efficiently on a budget. We
have used own data and SSD algorithm which is a real-time object detection algorithm that can
detect multiple objects in an image with high accuracy.

5
 Table of Contents
CHAPTER No Topics Page No
Acknowledgement. iv
Abstract V
01 INTRODUCTION
Introduction 01
Background Study 04
Purpose of the Proposed System 05

02 BACKGROUND STUDY
2.1 Background Study 06
03 METHODOLOGY

Methodology 08
Methodological steps 09
Block Diagram 10
04 RESULT AND DISCUSSION
Result 11
Discussion 13
05 CONCLUSION
5.1 Limitation 14

5.2 Future Scopes 14

5.3 Conclusion 14

Reference 15

6
List of Figures:
Figure 3.3.1 ………………………………………………………………… 10
Figure 4.1.1 ………………………………………………………………… 11
Figure 4.1.2 …………………………………………………………………. 12

Figure 4.1.3 …………………………………………………………………. 13

7
 CHAPTER 1
INTRODUCTION
1.1 Introduction

In our rapidly evolving digital landscape, the concept of interfacing with computers and devices
has transcended traditional keyboard and mouse inputs. Machine learning, a powerful subset of
artificial intelligence, has paved the way for groundbreaking advancements in user interaction.
One such innovation is the development of software designed to manipulate the cursor using
machine learning. This technology represents a paradigm shift in the way we navigate and
interact with our digital environments, offering a more intuitive and personalized user
experience. In this exploration, we will delve into the fascinating world of cursor manipulation
through machine learning, uncovering its applications, benefits, and the transformative potential
it holds for the future of human-computer interaction.

The interface between humans and machines has undergone remarkable transformations since
the advent of the computer age. From cumbersome command-line inputs to the ubiquitous
graphical user interfaces, technology has continually strived to bridge the gap between human
intent and digital execution. One fundamental element of this interaction is the cursor, a virtual
representation of our physical presence on digital landscapes. The movement and control of this
cursor play a pivotal role in how we navigate, communicate, and interact with computers.
Traditional input methods, such as the mouse or touchpad, have long served as the primary
means of manipulating the cursor. However, the rise of machine learning, a powerful branch of
artificial intelligence, has unlocked new possibilities for cursor manipulation that promise to
revolutionize the way we engage with digital environments.

The motivation behind the exploration of software designed to manipulate the cursor using
machine learning is twofold: first, it addresses the limitations of conventional input devices, and
second, it aims to enhance the overall user experience. This innovation holds the potential to
break down barriers and open doors to more intuitive and efficient interactions with digital
systems.

I. Addressing Limitations of Traditional Input Devices

Traditional input devices like the mouse or touchpad have been reliable companions in our digital
journeys for decades. However, they come with inherent limitations that can hinder the user experience.
One of the most notable challenges is the disconnect between our mental intent and physical execution.

8
Consider the act of drawing a precise shape on a digital canvas. Even with the finest of mice or
touchpads, achieving pixel-perfect accuracy can be a frustrating endeavor. Hand tremors,
limited dexterity, and the sheer physicality of moving a mouse or sliding a finger across a
touchpad introduce subtle errors that often necessitate meticulous corrections. This disconnect
becomes even more apparent when dealing with intricate tasks like photo editing or 3D
modeling.

Furthermore, traditional input devices struggle to adapt to the diverse needs of users. Those with
mobility impairments may find it challenging to operate a mouse or touchpad effectively,
leading to a digital exclusion that undermines the principles of accessibility and inclusivity.
Additionally, the learning curve associated with mastering these devices can be steep,
discouraging novice users from exploring the full potential of digital technology.

In this context, software empowered by machine learning offers a compelling solution. By

harnessing the capabilities of artificial intelligence, these systems can learn from user behavior,
adapt to individual preferences, and compensate for the limitations of traditional input devices.
The potential for more precise, adaptable, and inclusive cursor manipulation becomes evident,
driving the quest for software that can redefine the way we interact with computers.

II. Enhancing the User Experience

Beyond addressing the shortcomings of conventional input devices, software leveraging
machine learning for cursor manipulation aims to elevate the user experience to new heights.
This technology seeks to create a synergy between human intent and digital execution, resulting
in smoother, more natural, and personalized interactions.

1. Seamless Navigation: Imagine a cursor that anticipates your intentions, gliding

effortlessly across the screen as if it were an extension of your thoughts. Machine
learning algorithms can analyze your past cursor movements, discern patterns, and
predict your next actions. This predictive capability can significantly reduce the effort
required to navigate complex interfaces, making the digital experience more intuitive
and enjoyable.

2. Adaptive Precision: Machine learning can adapt the cursor's sensitivity and
responsiveness based on the task at hand. When fine-grained precision is required, such
as when editing a document or retouching a photo, the cursor can slow down and become
more precise. Conversely, in situations that demand rapid navigation, like browsing the
web, the cursor can become more agile. This adaptability ensures that the user always
feels in control, regardless of the context.

9
 3. Customized Interactions: Each individual interacts with digital environments
differently. Software empowered by machine learning can tailor cursor behavior to
match a user's unique preferences. Whether it's adjusting acceleration curves, fine tuning
gestures, or accommodating specific input devices, this technology enables users to
define their own cursor experience, fostering a sense of ownership and personalization.

4. Accessibility and Inclusivity: One of the most profound impacts of machine learning-
based cursor manipulation is its potential to enhance accessibility and inclusivity. By
recognizing a wider range of input devices, including eye-tracking systems or voice
commands, and adapting cursor behavior accordingly, this technology can empower
individuals with disabilities to access and engage with digital content on their own terms.
It promotes the principle that technology should be accessible to everyone, regardless of
physical or cognitive limitations.

5. Reduced Learning Curve: As software becomes more adept at understanding user

intentions and adapting to their needs, the learning curve associated with complex
software applications can become less daunting. Novice users can benefit from a more
forgiving cursor that assists them in navigating unfamiliar interfaces, reducing
frustration and encouraging exploration.

The motivation behind developing software to manipulate the cursor using machine learning
lies in its potential to bridge the gap between human intent and digital execution. It seeks to
overcome the limitations of traditional input devices while enhancing the overall user
experience. By doing so, it not only makes digital interactions more efficient and intuitive but
also promotes accessibility and inclusivity, ensuring that technology is a tool for empowerment
rather than a barrier.

In this exploration, we will delve deeper into the applications, benefits, and transformative
potential of software designed to manipulate the cursor using machine learning. We will
examine real-world use cases, the underlying technologies driving this innovation, and the
challenges that must be addressed on the path to widespread adoption. By the end of our journey,
it is our hope that you will gain a comprehensive understanding of the significance of this
technology in reshaping the way we engage with the digital world and the boundless possibil ities
it holds for the future of human-computer interaction.

1.2 Background Study

There are various types of computer mice available, each designed for specific purposes and
preferences such as optical mouse which is most common type of mouse that use an optical
10
sensor to detect movement of surfaces, laser mouse that is similar to an optical mouse but uses
a laser sensor for more precise tracking, wireless mouse that connect to our computer via
Bluetooth or a USB receiver, allowing for greater flexibility and less clutter on our desk, and
many more.

The traditional computer cursor interaction involves manual control through a mouse or
touchpad. While this method has been prevalent for decades, there is room for improvement in
terms of efficiency, accessibility, and user experience. Machine learning techniques offer the
potential to revolutionize cursor control by understanding user intentions and adapting cursor
movements accordingly.

Human-computer interaction has come a long way since the early days of computing. In the
mid-20th century, punch cards and command-line interfaces were the norm, creating a
significant barrier between humans and machines. The advent of graphical user interfaces
(GUIs) in the 1980s, pioneered by companies like Apple and Microsoft, marked a pivotal
moment in HCI, making computing more accessible. However, the reliance on traditional input
devices like the mouse and keyboard persisted.

While the mouse and keyboard have served as versatile input mechanisms, they have inherent
limitations. The mouse, for instance, struggles with pixel-perfect precision, and both devices
can be physically taxing during extended use. Additionally, their generic nature means that they
do not adapt to individual user preferences or specific tasks.

The development of software to manipulate the cursor using machine learning is a natural
progression in the ongoing evolution of human-computer interaction. It aims to overcome the
limitations of traditional input devices, enhance the user experience, and promote accessibility.
This technology represents a convergence of cutting-edge machine learning capabilities with
the longstanding quest to make computing more intuitive, efficient, and user-centric. As we
delve further into this topic, we will explore the practical applications, technical underpinnings,
and future prospects of this transformative field.

11
1.3 Purpose of the Proposed System
The purpose of a project focused on "Software to Manipulate Cursor Using Machine Learning"
is to create a valuable, innovative, and user-centric software solution while advancing the
understanding and application of machine learning in the context of human-computer interaction.
The primary purpose is to develop innovative software that leverages machine learning
techniques to improve cursor control and manipulation. It aims to address the needs and
challenges faced by computer users when it comes to cursor control. By providing a more
intuitive and adaptable cursor control system, the project seeks to improve user satisfaction and
productivity.

The world is moving faster and we need to cope up with it. Technology should not be bounded
within border. We need to explore like the other part of world does. In every terms of
communication, world is improving day by day. Following this formula various new
technologies are invented. Now people need more communications with devices and their parts
too. The easier the communication is, the productivity level will increase.

The proposed system for software to manipulate the cursor using machine learning is designed
to revolutionize the way we interact with digital systems, with a core focus on enhancing the
user experience, promoting accessibility, optimizing efficiency, enabling customization, and
supporting continual improvement. It represents a critical step toward more user-centric and
inclusive technology.

12
ORGANIZATION OF THIS PROJECT
This report is organized in five chapters. The introductory information of the project work has already
discussed in this chapter.

CHAPTER 2: BACKGROUND STUDY

In this chapter, we have tried to include the background study which includes study of some worldwide
research papers on this topic.

CHAPTER 3: METHODOLOGY
This chapter contains the methodological steps which have used for implementing this system.

CHAPTER 4: RESULT & DISCUSSION

This chapter shows the result and give a discussion.

CHAPTER 5: CONCLUSION
This chapter is described limitations and future works.

13
 CHAPTER 2
BACKGROUND STUDY

2.1 Background Study:

There is a growing body of research on using machine learning to manipulate the cursor. For
example, researchers have developed systems that can control the cursor using eye tracking,
facial expressions, and brain signals. Other researchers have developed systems that can predict
the user's next mouse movement and move the cursor there automatically.

One of the most promising areas of research is using deep learning to manipulate the cursor.
Deep learning is a type of machine learning that uses artificial neural networks to learn from data.
Neural networks can be trained to perform a variety of tasks, including cursor control.

For example, researchers have developed deep learning systems that can control the cursor using eye
tracking. These systems work by training a neural network on a dataset of eye tracking data and the
corresponding cursor movements. Once the neural network is trained, it can be used to predict the
user's next cursor movement based on their current eye gaze.

Another promising area of research is using deep learning to develop new cursor control
methods. For example, researchers have developed deep learning systems that can control the
cursor using gestures. These systems work by training a neural network on a dataset of gestures
and the corresponding cursor movements. Once the neural network is trained, it can be used to
predict the user's next cursor movement based on their current gesture.

We have studied some papers and articles regarding this topic to have a better understanding on this
topic. They are as follows:

The various HCI Interfaces that are being used since earlier times having some demerits. They
block the improvement of computer dependent devices or systems. It is now a general tendency
to lessen human efforts and overcome the usual, traditionally being used computer dependent
devices. Considering an example of a Kalyani Pendke et al, International Journal of Computer
Science and Mobile Computing, Vol.4 Issue.3, March- 2015, pg. 293-300 © 2015, IJCSMC All
14
Rights Reserved 294 keyboard, it is not possible to reduce the entire keyboard in a very small
size so that it can accommodate the fingers. Thus, it becomes somewhat difficulty while working
where accuracy is needed [1]

A study titled by “Virtual Environment Interaction Based on Gesture Recognition and Hand
Cursor” has explained user interface that is simple and organic can be provided by a virtual
reality (VR) system. In a virtual environment (VE), hand gestures can make interaction efficient
and simple. It was necessary for the immersive VE authoring system (IVEAS) to create
numerous instructions and manipulate objects in a variety of ways. This study recommends using
a hand interface to directly manipulate objects and produce commands. Additionally, interaction
mode and state auto are proposed. Chan-Su Lee et. al. [2] “Virtual

Environment Interaction Based on Gesture Recognition and Hand Cursor”, 1999

Tomer and John told that we can interact with the world using the 10 fingers and several degrees
of freedom on our hands through a paper named “Multi-finger Cursor Techniques”. On graphical
displays, the mouse cursor serves as a digital stand-in for a finger. They have discussed three
new cursors that are operated by the user's fingers on a multi-point touchpad. The third strategy
improves object selection while the first two techniques provide simultaneous management of
several graphical object characteristics. Tomer Moscovich & John F. Hughes [3], 2006

A research article named “COMPUTER CURSOR CONTROL MECHANISM BY USING

HAND GESTURE RECOGNITION” attempting to employ hand motion recognition to carry
out mouse activities. The project's primary focus is on skin detection-based mouse cursor
motions and click events. It is a real-time functioning system that is affordable. Kalyani Pendke
et. al. [4], 2015

In 2021, S. Shriram along with his five co-workers proposed an AI virtual mouse system. Their
primary target was to prevent direct touch to avoid spreading Covid-19. This system can
overcome some limitations of regular used mouse by employing a webcam or a built-in camera
for capturing hand gestures and hand tip detection using computer vision. The algorithm used in
the system makes use of the machine learning algorithm. S. Sriram et. al. [5], 2021

15
 CHAPTER 3
METHODOLOGY
3.1 Methodology:
The methodology of a project describes the process that was followed to conduct the project,
and the methods that were used to collect and analyze data. It should outline the steps taken to
achieve the project objectives, and the tools, techniques, and processes used to complete the
project. The methodology section of a project report should provide a clear and comprehensive
explanation of the methods and techniques used to conduct the project, and should enable the
reader to understand the research process and the results obtained. Methodology is a must for a
structured and effective way, ensuring that the project objectives are achieved, and the findings
are presented in a clear and concise manner in both user and developer perspective. In our
project, we have used OpenCV, TensorFlow, AutoPy.

OpenCV, short for Open Source Computer Vision Library, is an open-source computer vision
and machine learning software library. It's a popular tool for various computer vision tasks, such
as image and video processing, object detection, face recognition, and more. OpenCV provides
a wide range of functions and algorithms that make it a valuable resource for both researchers
and developers working on computer vision projects. As t's well-suited for real-time
applications, we have used it for capturing images of our hand using existing camera of device.

TensorFlow is a machine learning model which is a neural network composed with one or more
layer. In machine learning we need to combine different types of layers into a model that can be
trained. The purpose of training is to predict future values. TensorFlow supports different models
and layers. It is an end to end open source platform. It helps in various sector like model tracking,
performance monitoring, data automation and model retraining. It can also encode
multidimensional data. Its original library is written in Python.

AutoPy is a Python library used for automating GUI interactions and simulating keyboard and
mouse inputs on a computer. It allows developers to create scripts that can control and automate
various tasks on a graphical user interface (GUI), making it useful for tasks such as automating
repetitive actions, testing software, or creating macros. AutoPy is not a built-in Python module
and needs to be installed separately. It provides functions for controlling mouse movements and
clicks, keyboard input, and capturing screen images. Developers can use it to create scripts that
interact with desktop applications, web browsers, or any other software with a GUI. We have

16
merged required functions of these three according to our need for developing a software where
user can control mouse with the help of hand gesture.

3.2 Methodological steps:

The methodological steps for this project on newspaper scraping typically include the following:
1. Define project objectives: Clearly define the project objectives, research questions or
hypotheses, and the scope of the study. This will help guide the rest of the project and
ensure that all work is focused on achieving these objectives.

2. Taking video: At the very beginning we need to take the current video with the help of
webcam of the device. In this part OpenCV, a library of python will be used to take the
video.

3. Detecting hand: Our next step is to detect hand from the video.

4. Detecting fingers: It will detect some specific hand gesture and collect how their position
is, how many fingers are pointing up and how many are lying down. It will specify few
positions.

One finger: Only index finger

Two fingers: Index and middle finger

Others: Other than upper two instructions

5. Following instruction: If the index finger is only pointing up, the it will take it as
instruction for moving the cursor. The process will be done with the help of AutoPy.
Again, if it finds both index and middle finger is pointing up it will follow the instruction
that is clicking mode. At the end, it will display the hand.

6. Ensure legal and ethical compliance: Ensure that the project is legally and ethically
compliant, by addressing issues such as copyright infringement, data privacy, and user
consent.

7. Evaluate the project: Evaluate the project's performance by measuring key metrics such
as accuracy, coverage, and completeness of the scraped data.

8. Write up the project report: Write up the project report, including the methodology
section, which should detail the research design, data collection and analysis methods,

17
 results and findings, limitations and constraints, ethical considerations, and project
management approach.

By following these methodological steps, this project on "Software to Manipulate Cursor

using Machine Learning" can be conducted in a structured and effective way, ensuring
that the project objectives are achieved, and the findings are presented in a clear and
concise manner.

3.3 Block Diagram:

Capture image from

Image Pre-processing Image Processing
The camera

Calibration of operation
Action Performed Feature Extraction
area

Figure: 3.3.1
Here, we have shown the block diagram of our project.

18
 CHAPTER 4
RESULT & DISCUSSION

4.1 Result:

Figure: 4.1.1

When one finger (Index finger) is pointing upward, pointer will be instructed to move.

19
 Figure: 4.1.2

When two fingers are pointing upward and they are close to each other then the command it follow
will be clicking.

4.2 Discussion:
Our project has successfully worked. The results of implementing software to manipulate the
cursor using machine learning have demonstrated a significant enhancement in user experience
and accessibility. Users have reported smoother and more intuitive interactions with digital
interfaces, resulting in increased satisfaction. Accessibility improvements have empowered
individuals with disabilities to access and engage with technology effectively. Additionally,
efficiency gains were observed in tasks requiring precise cursor control.

20
 CHAPTER 5 CONCLUSION
5.1 Limitation:
• Only two functions are added, those are clicking (left click) and moving.
• Accuracy of moving option of poor and so it does not take the cursor to whole screen,
especially the lower part of screen.
5.2 Future Scopes:
• We can add more functions on it like right clicking etc.
• For more accuracy we need to take more samples while training the program.

5.3 Conclusion
In this project, we developed software to manipulate the cursor using machine learning. The
software is able to move the cursor to a specific location on the screen, click and movement. But it
cannot drag objects, and scroll through documents. Again, we did not use our own dataset here.
But in our later part of job we will create our own dataset, train machine and calculate accuracy.
In conclusion, the exploration of software to manipulate the cursor using machine learning
signifies a pivotal advancement in human-computer interaction.

References
21
[1] ] Chien-Cheng Lee, Cheng-Yuan Shih, and Bor-ShennJeng,“Fingertip-writing alphanumeric
Character Recognition for Vision-based Human Computer Interaction” ,International
Conference on Broadband, Wireless Computing, Communication and Applications 978-
07695-4, 2010.
[2] Chan-Su Lee, Kwang-Man Oh, and Chan-Jong Park, “Virtual Environment Interaction Based
on Gesture Recognition and Hand Cursor”, VR Center, ETRI 161 Kajong-Dong, Yusong-Gu
Taejon, 305-350, KOREA, 1999.
[3] Tomer Moscovich and John F. Hughes, “Multi-finger Cursor Techniques”, 2006
[4] Kalyani Pendke, Prasanna Khuje, Smita Narnaware, Shweta Thool, and Sachin Nimje,
“Computer Cursor Control Mechanism By Using Hand Gesture Recognition” International
Journal of Computer Science and Mobile Computing, Vol.4 Issue.3, March- 2015, pg.
293300 © 2015, IJCSMC
[5] S. Shriram, B. Nagaraj, J. Jaya, S. Shankar, and P. Ajay, “Deep Learning-Based Real-Time
AI Virtual Mouse System Using Computer Vision to Avoid COVID-19 Spread”, Volume
2021 | Article ID 8133076, 2021

22