A Mini Project Report on

VIRTUAL MOUSE
submitted in partial fulfillment of the requirements for the award of
degree of

BACHELOR OF TECHNOLOGY
in

COMPUTER SCIENCE & ENGINEERING

by
Vedunuri Sujith (18211A05V5)
U. Bhuvana Teja (18211A05U5)
Thatipelli Ashish (18211A05T6)

Under the guidance of

Mrs. P. Jhansi Devi, MTech,

Assistant Professor

DEPARTMENT OF COMPUTER SCIENCE &ENGINEERING

B.V. RAJU INSTITUTE OF TECHNOLOGY

(UGC Autonomous, Accredited by NBA & NAAC)

Vishnupur, Narspur, Medak (Dist.), Telangana State, India - 502313

2021 - 2022
 B. V. Raju Institute of Technology
(UGC Autonomous, Accredited by NBA & NAAC)
Vishnupur, Narspur, Medak (Dist.) Telangana State, India –
502313

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

CERTIFICATE

This is to certify that the Mini Project entitle “VIRTUAL MOUSE”, being
submitted by

Vedunuri Sujith (18211A05V5)

U. Bhuvana Teja (18211A05U5)
Thatipelli Ashish(18211A05T6)

In partial fulfillment of the requirements for the award of degree of BACHELOR

OF TECHNOLOGY in COMPUTER SCIENCE AND ENGINEERING to B.V. RAJU
INSTITUTE OF TECHNOLOGY is a record of Bonafede work carried out during a
period from May 2021 to July 2021 by them under the guidance of Mrs. P. Jhansi
Devi, Assistant Professor, CSE Department.

This is to certify that the above statement made by the students is/are
correct to the best of my knowledge.

Ms. P. Jhansi Devi Dr. Ch. Madhu Babu

Assistant Professor Professor & HOD-CSE

External Examiner
 CONTENTS

Candidate’s Declaration i
Acknowledgement ii
Abstract iii
Contents
List of Figures
UseCase Diagram 11
Class Diagram 12
Sequence Diagram 12
Architecture 18
Output Screen 19
Final Output 22

1. INTRODUCTION
9
1.1 Motivation
1.2 Problem Definition 9
1.3 Objective of Project 9

2. LITERATURE SURVEY
2.1 Introduction 12

2.2 Existing System 12

2.3 Disadvantages of Existing system 13
2.4 Proposed System 13
3. ANALYSIS
3.1 Introduction 15
3.2 Software Requirement Specification 15
3.2.1 User requirements 15
 3.2.2 Software requirements 15
3.2.3 Hardware requirements 15
3.3 Architecture Diagram of Project 16
3.4 Algorithms and Flowcharts 16
4. DESIGN
4.1 Introduction 19
4.2 DFD / ER / UML diagram 19
(any other project diagrams)
4.3 Module design and organization 20
5. IMPLEMENTATION & RESULTS
5.1 Introduction 24
5.2 Explanation of Key functions 24
5.3 Method of Implementation 25
5.3.1 Forms 25
5.3.2 Output Screens 25
5.3.3 Result Analysis 26
6. TESTING & VALIDATION
6.1 Introduction 29
6.2 Design of test cases and scenarios 29
6.3 Validation 30
7. CONCLUSION& FUTURE WORK 32
8. REFERENCES 34
 B. V. Raju Institute of Technology
(UGC Autonomous, Accredited by NBA & AAC)
Vishnupur, Narspur, Medak (Dist.), Telangana
State, India – 502313

CANDIDATE’S DECLARATION

We hereby certify that the work which is being presented in the project
entitled “VIRTUAL MOUSE” in partial fulfillment of the requirements for the
award of Degree of Bachelor of Technology and submitted in the Department
of Computer Science and Engineering, B. V. Raju Institute of Technology,
Narsapur is an authentic record of my own work carried out during a period
from May 2021 to July 2021 under the guidance of Mrs. P. Jhansi Devi,
Assistant Professor. The work presented in this project report has not been
submitted by us for the award of any other degree of this or any other
Institute/University.

Vedunuri Sujith(18211A05V5)
U. Bhuvana Teja(18211A05U5)
Thatipelli Ashish(18211A05T6)
 ACKNOWLEDGEMENT

The success and final outcome of this project required a lot of guidance
and assistance from many people and we are extremely fortunate to have got
this all along the completion. Whatever we have done is due to such guidance
and assistance. We would not forget to thank them.

We thank Mrs. P. Jhansi Devi for guiding us and providing all the
support in completing this project. We are thankful to Mrs. T. Shilpa, our
section project coordinator for supporting us in doing this project. We are
thankful to Dr. Pradeep Kumar V, project coordinator for helping us in
completing the project in time. We thank the person who has our utmost
gratitude is Dr. Ch. Madhu Babu, Head of CSE Department.

We are thankful to and fortunate enough to get constant

encouragement, support and guidance from all the staff members of CSE
Department.

Vedunuri Sujith (18211A05V5)

U.Bhuvana Teja (18211A05U5)
Thatipelli Ashish (18211A05T6)
 ABSTRACT
This paper proposes a novel camera vision-based cursor control system,
using hand gestures captured from a webcam through a color detection
technique. The system will allow the user to navigate the computer cursor
using their hand bearing color caps or tapes and left click and dragging will
be performed using different hand gestures and also, it performs file transfer
between two systems in a single same network. The proposed system uses
nothing more than a low- resolution webcam that acts as a sensor and it is
able to track the users hand bearing color caps in two dimensions. The
system will be implemented using the python and OpenCV. The hand gesture
is the most effortless and natural way of communication. The output of the
camera will be displayed on the monitor. Shape and position information
about the gesture will be gathered using detection of color. The file
transferring scheme is implemented by using the python server
programming

Key Words: OpenCV, NumPy, Media pipe, Python

 CHAPTER 1
INTRODUCTION
1. INTRODUCTION
1.1 Motivation
The most efficient and expressive way of human communication is
through hand gesture, which is a universally accepted language. It is pretty
much expressive such that the dumb and deaf people could understand it.
In this work, real-time hand gesture system is proposed.
1.2 Problem Definition
Experimental setup of the system uses fixed position low-cost web camera
high-definition recording feature mounted on the top of monitor of
computer or a fixed camera on a laptop, which captures snapshot using Red
Green Blue [RGB] color space from fixed distance. This work is divided into
four stages such as image preprocessing, region extraction, feature
extraction, feature matching. Recognition and the interpretation of sign
language is one of the major issues for the communication with dump and
deaf people
1.3 Objective of Project
In this project an effective hand gesture segmentation technique has been
proposed based on the preprocessing, background subtraction and edge
detection techniques [1]. Pre-processing is defined as procedure of
formulating data for another process. The main objective of the
preprocessing process is to transform the data into a form that can be more
effectively and effortlessly processed. In the proposed work, the pre-
processing techniques are created on the basis of different types of
combinations from the subsequent hand gesture image processing
operations such as capturing image, removing noise, background
subtraction, and edge detection and these image processing methods are
discussed is as follows. Initially, the hand gesture images are captured from
the vision-based camera, The hand gestures can be observed with the
different kind of interfaces like “data gloves” that accurately records every
abduction angle and digits and position sensors for wrists and optical
orientation or electromagnetic, requiring the user to wear trackers or
gloves. Habitually, the glove-based interfaces even need the user to be
hitched to the computer, all minimizing time to user comfort and interface,
conversely, vision-based interfaces offer unencumbered
interaction with human. By this paper we are aiming in creating a cost-free
hand recognition software for laptops and PCs with a web- cam support.The
project covers as a hand recognition tool which could be used to move the
mouse pointer, perform simple operations like clicking and other hand gesture
operations like moving file from computer to computer through delicate socket
programming and performing simple but fascinating operations that could be
covered with the hand recognition.
 CHAPTER 2
Literature Survey
2. Literature Survey
2.1 Introduction
As modern technology of human computer interactions become important
in our everyday lives, varieties of mouse with all kind of shapes and sizes were
invented, from a casual office mouse to a hard-core gaming mouse. However,
there are some limitations to these hardware as they are not as environmentally
friendly as it seems. For example, the physical mouse requires a flat surface to
operate, not to mention that it requires a certain areato fully utilize the functions
offered. Furthermore, some of this hardware are completely useless when it
comes to interact with the computers remotely due to the cable lengths
limitations, rendering it inaccessible.
2.2 Existing System
Erden et al have used a camera and computer vision technology, such as
image segmentation and gesture recognition, to control mouse tasks. Our
project was inspired by a paper of Hojoon Park [2] where he used Computer
vision technology and Web camera to control mouse movements. However,
he used finger-tips to control the mouse cursor and the angle between the
thumb and index finger was used to perform clicking actions.

Chu-Feng Lien [3] had used an intuitive method to detect hand motion by its
Motion History Images (MHI). In this approach only fingertip was used to
control both the cursor and mouse click. In his approach the user needs to
hold the mouse cursor on the desired spot for a specific period of time for
clicking operation.

Kamran Niyazi[4] et al used Web camera to detect color tapes for cursor
movement. The clicking actions were performed by calculating the distance
between two colored tapes in the fingers. In their paper K N Shah et al have
represented some of the innovative methods of the finger tracking used to
interact with a computer system using computer vision. They have dividedthe
approaches used in Human Computer Interaction (HCI) in two categories:
1. HCI without using interface and
2. HCI using interface.
2.3 Disadvantages of Existing System
Fingertip detection with multiple people simultaneously poses a great
difficulty that currents systems have not yet overcome. In addition,
choosing a target person when multi-people directly stand to face the
camera is a challenging case because it is difficult to determine
accurately who will be the target. Therefore, long-term fingertip tracking
remains a challenging task.

2.4 Proposed System

Any new product should either make human life more comfortable more
productive or more fun. Provides greater flexibility than the existing
system. Can provide more functions depending on the choice of object.
Easy to modify and adapt Less prone to physical damage due to absence
of fixed physical device. Also, there is a certain degree of fun &
entertainment associated with the whole idea.

It is divided into following modules:

Image acquisition.

Object recognition

Object tracking and information Retrieval

Point coordinate calculation and Motion

Analysis Setting Cursor Position

Event generation
Chapter 3
Analysis
3.1 Introduction
The A Computer Mouse is an input device that helps to point and to interact
with whatever that is being pointed. There are so many types of mouse in
the current trend, there’s the mechanical mouse that consists of a single
rubber ball which can rotate in any direction and the movement of the
pointer is determined by the motion of that rubber ball. Later the mechanical
mouse is replaced by the Optical Mouse. Optical Mouse consists of a led
sensor to detect the movement of the pointer. Years Later the laser mouse
was introduced to improve the accuracy and to overcome the drawbacks of
the Optical Mouse. Later as the Technology has been increased drastically
wireless mouse was introduced so as to enable hassle free movement of the
mouse and to improve the accuracy. No Matter how much the accuracy of the
mouse increases but there will always be limitations of the mouse as the
mouse is a hardware input device and there can be some problems like
mouse click not functioning properly and etc., as the mouse is a hardware
device like any other physical object even the mouse will have a durability
time within which is functional and after its durability time, we have to
change the mouse

3.2 Software Requirement Specification

3.2.1 User requirements
Tool: Vscode
Language: python 3.7
Modules: OpenCV, Media Pipe
Other: webcam
3.2.2 Software Requirements
Operating System: Windows 7 64 bit
Technology: python 3.7
IDE: Vscode
 3.3 Hardware Requirements
Processor: min core i3(intel)

Ram: min 4gb

Hard Disk: min 100Gb

3.4 Architecture Diagram

3.5 Algorithms and Flow charts

Hand Detection using Media Pipe: It is a cross-platform for

building multimodal applied machine learning pipeline. This module helps
us in detecting hands and using their coordinates we detect exact location
of finger and control mouse operation.

Flow Chart
CHAPTER 4
DESIGN
4.1 Introduction
This system is design in a way where whole display is resembled in into a
small screen box, this box now represents whole screen. The translations
are resulted based on coordinates.

4.2 DFD / ER / UML diagram

 Use Case Diagram

4.3 Module Design and Organization

In the Methodology, the method used in each component of the
system will be explained separately. They are following subsections
are following subsections:

4.3.1 Camera Settings

The runtime operations are managed by the webcam of the connected laptop or
desktop. To capture a video, we need to create a Video Capture object. Its
argument can be either the device index or the name of a video file. Device
index is just the number to specify which camera. Since we only use a single
camera, we pass it as ‘0’. We can add additional camera to the system and
pass it as 1,2 and so on. After that, you can capture frame-by-frame. But at
the end, don’t forget to release the capture. We could also apply color
detection techniques to any image by doing simple modifications in the code.
4.3.2 Capturing frames
The infinite loop is used so that the web camera captures the frames in every
instance and is open during the entire course of the program. We capture
the live feed stream, frame by frame. Then we process each captured frame
which is in RGB (default) color space to HSV color space. There are more than
150 color- space conversion methods available in OpenCV. But we will look into
only two which are most widely used ones, BGR to Gray and BGRto HSV.

4.3.3 Masking technique

The mask is basically creating some specific region of the image following
certain rules. Here we are creating a mask that comprises of an object in red
color. After that we perform a bitwise AND operation on the Input image and
the Threshold image, which result in only the red colored objects are
highlighted. This result of the AND operation is stored in res. We then display
the frame, res and mask on 3 separate windows using imshow() function.

4.3.4 Display the frame

The imShow() is a function of High GUI and it is required to call the waitKey
regularly. The processing of the event loop of the imshow() function is done
by calling waitKey. The function waitKey() waits for key event for a “delay”
(here, 5 milliseconds). Windows events like redraw, resizing, input event etc.
are processed by High GUI. So we call the waitKey function, even with a 1ms
delay.

4.3.5 Mouse Movement

We have to first calculate the center of both detected red object which we can
easily do by taking the average of the bounding boxes maximum and minimum
points. now we got 2 co- ordinates from the center of the 2 objects we will find
the average of that and we will get the red point shown in the image. We are
converting the detected coordinate from camera resolution to
 the actual screen resolution. After that we set the location as the mouse
position. but to move the mouse pointer it will take time. So, we have to wait
till the mouse pointer reaches that point. So, we started a loop and we are
not doing anything there we are just waiting will the current mouse location
is same as assigned mouse location. That is for the open gesture.

4.3.6 Clicking
the next step is to implement the close gesture. The operation is performed
by clicking the object and dragging it. It is similar to the open gesture, but
the difference is we only have one object here so we only need to calculate
the center of it. And that will be placed on the location where we will position
our mouse pointer. Instead of mouse release operation we will be performing
a mouse press operation.

4.3.7 Drag
In order to implement the dragging we introduce a variable ‘pinchflag’. It will be
set to 1 if it was clicked earlier. So after clicking whenever we find theopen
gesture we check if the pinchflag is set to 1. If it is set to one then Drag operation
is performed otherwise the mouse move operation is performed.

4.3.8 DnD Frame

First, we create the MyFileDropTarget class. Inside that we have one overridden
method, OnDropFiles. This method accepts the x/y position of the mouse
along with the file paths that are dropped.

4.3.9 File Transfer

Using the file path obtained from MyFileDropTarget class we send the file
to the other system using the python socket module. And there it receives and
creates the file.
 CHAPTER 5
IMPLEMENTATION AND RESULTS
5.1 Introduction
This project is implemented in python and the program outputs a window in
which the window screen is scaled down. This program uses webcam and
detects our hand and process mouse operation.

5.2 Key Functions

 Hand detection
 Calculate coordinates
 Translate coordinates
 Mouse operations

This project is mainly focused to help patients who don’t have control of their
limbs and also improvise the interaction between the machine and humans.
Our motive was to create this technology in the cheapest possible way and
also to create it under a standardized operating system. The proposed system
controls the functions of mouse pointer by detecting point on finger, and
perform the mouse functions such as left click, dragging, cursor movement,
and the file transfer between two systems in a same network.

This method detects the color point objects for the mouse control. The user
uses the color point objects on their finger-tip for the better performance.
When the number of contours is two, then it performs the simple mouse
movement action. Otherwise, when the number of contours is one then it
performs the left click. This system also supports the simple file transfer
between two or more systems in the same network connection.

The left side of the computer’s screen act as a communication channel

between the systems. i.e., the file which is to be copy should drag and drop
on the left side of the computer’s screen. Then the dropped file will be copied
to the destination or the receiver system. This system is mainly aimed to
reduce the use of hardware components attached with the computer.
Although the application can be run in an ordinary computer having a web
camera, but ideally it requires having at least 2MP frontal camera with at
least Pentium processor and at least 256 MB RAM.
We can see the operations in the following images.
29
 CHAPTER 6
TESTING AND VALIDATION
6.1 Introduction
While running the application/program, this program is tested under
various circumstances like in low light conditions and blurry webcam. This
shows the experimental test results of our virtual mouse system. The average
accuracy is 96.13%. On average conditions, it shows 82% accuracy under
low light and other situations. This is exceptionally high performance for a
fingertip gesture-based interface. As expected, the highest accuracy
occurred in the easier gesture ‘mouse movement’ and the lowest in the
harder gesture ‘right-click’. The accuracy was reduced in the ‘right-click’
gesture because, with fast fingertip tracking, the gesture was sometimes
confused with others. The experiment also showed that the results did not
change significantly through several resolutions.

6.2 Design of test cases and scenarios

6.1 Validation
 Chapter 7
Conclusion and Future Work
Conclusion
Gesture recognition gives the best interaction between human and machine.
Gesture recognition is also important for developing alternative human
computer interaction modalities. It enables human to interface with machine
in a more natural way. Gesture recognition can be used for many
applications like sign language recognition for deaf and dumb people, robot
control etc. This technology has wide applications in the fields of augmented
reality, computer graphics, computer gaming, prosthetics, and biomedical
instrumentation. Digital Canvas is an extension of our system which is
gaining popularity among artists, by which the artist could create 2D or 3D
images using the Virtual Mouse technology using the hand as brush and a
Virtual Reality kit or a monitor as display set. This technology can be used to
help patients who don’t have control of their limbs. In case of computer
graphics and gaming this technology has been applied in modern gaming
consoles to create interactive games where a person’s motions are tracked
and interpreted as commands. The major extension to this work can be done
to make system able to work at much complex background and compatible
with different light conditions. It can be made as an effective user interface
and which can include all mouse functionalities. It would be ideal to research
into advanced mathematical materials for image processing and investigate
on different hardware solutions that would result in more accurate hand
detections. Not only did this project show the different gesture operations
that could be done by the users but it also demonstrated the potential in
simplifying user interactions with personal computers and hardware
systems.

Future Work

We can further enhance this project to make it more useful by providing

scroll down, scroll up as well as taking screen shots copying and pasting
content. Lifting our hand towards camera continuously can result in pain so
instead of lifting hand we could use a sensor on our hands to make it more
effortless.
CHAPTER 8
REFERENCES
 References
 Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E. “Computer vision-based
mouse”, Acoustics, Speech, and Signal Processing, Proceedings. (ICASS). IEEE
International Conference, 2002.

 Hojoon Park, “A Method for Controlling the Mouse Movement using a Real
Time Camera”, Brown University, Providence, RI, USA, Department of computer
science, 2008.

 Chu-Feng Lien, “Portable Vision-Based HCI – A Real-time Hand Mouse

System on Handheld Devices”, National Taiwan University, Computer
Science and Information Engineering Department

 Kamran Niyazi, Vikram Kumar, Swapnil Mahe, Swapnil Vyawahare, “Mouse

Simulation Using Two Coloured Tapes”, Department of Computer Science,
University of Pune, India, International Journal of Information Sciences and
Techniques (IJIST) Vol.2, No.2, March 2012

 K N. Shah, K R. Rathod and S. J. Agravat, “A survey on Human Computer

Interaction Mechanism Using Finger Tracking” International Journal of
Computer Trends and Technology, 7(3), 2014, 174- 177

 Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, 2nd

edition, Prentice Hall, Upper Saddle River, New Jersey, 07458

 Shahzad Malik, “Real-time Hand Tracking and Finger Tracking for

 Interaction”, CSC2503F Project Report, December 18, 2003