1

COURSE REPORT
submitted in partial fulfilment of the requirements for the Award of Degree of
BACHELOR OF TECHNOLOGY
in
COMPUTER SCIENCE AND ENGINEERING
on
VIRTUAL MOUSE

SUBMITTED BY: GOVIND AGARWAL

(ROLL NO.: 2000910100069)

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

Dr. Deepti Aggarwal

JSS ACADEMY OF TECHNICAL EDUCATION

C-20/1 SECTOR-62, NOIDA
2021- 2022

ACKNOWLEDGEMENT
2

I have taken efforts in this Course. However, it would not have been possible without the kind
support and help of many individuals. I would like to extend my sincere thanks to all of them. I
am highly indebted to Maven Silicon for their guidance and constant supervision as well as for
providing necessary content regarding the course & also for their support in completing the
course. I would like to express my gratitude towards my parents & member of Maven Silicon
for their kind co-operation and encouragement which help me in completion of this Course. I
am highly indebted to Dr. Sampath Kumar V, for the facilities provided to accomplish this
course. I would like to thank my Head of the Department Mr. Arun Kumar G for his
constructive criticism throughout my internship. I would like to thank Dr. Bajrang Mishra,
College internship coordinator Department of ECE for their support and advices to get and
complete course from Maven Silicon. I am extremely great full to my department staff members
and friends who helped me in successful completion of this internship.

GOVIND AGARWAL
(200910100069)

ABSTRACT

This project represents a concept of controlling our video display with the
assistance of hand gestures, which relies on one in each of the studies of Human
Computer Interaction. This project shows that we are able to control our screen by
moving our fingers which will work as cursor. To make this happen, all we need a
3

working webcam and three main algorithms that are, mediapipe, OpenCV and
autopy. Mediapipe is employed for hand tracking, OpenCV for image processing
and drawing and at last autopy for controlling the mouse movement and its
functioning.
4

TABLE OF CONTENT

1. Abstract
2. Introduction
3. Related Work
4. Algorithms used for Hand tracking
5. Experimental Results and Evaluation
6. Future Scope
7. Applications
8. Conclusions
9. Reference
5

Deep Learning-Based Real-Time AI Virtual Mouse System Using

Computer Vision

The mouse is one of the wonderful inventions of Human-Computer

Interaction (HCI) technology. Currently, wireless mouse or a Bluetooth
mouse still uses devices and is not free of devices completely since it
uses a battery for power and a dongle to connect it to the PC. In the
proposed AI virtual mouse system, this limitation can be overcome by
employing webcam or a built-in camera for capturing of hand gestures and
hand tip detection using computer vision. The algorithm used in the
system makes use of the machine learning algorithm. Based on the hand
gestures, the computer can be controlled virtually and can perform left
click, right click, scrolling functions, and computer cursor function without
the use of the physical mouse. The algorithm is based on deep learning
for detecting the hands. Hence, the proposed system will avoid COVID-19
spread by eliminating the human intervention and dependency of devices
to control the computer.
6

1. Introduction

With the development technologies in the areas of augmented reality and devices that we use
in our daily life, these devices are becoming compact in the form of Bluetooth or wireless
technologies. This paper proposes an AI virtual mouse system that makes use of the hand
gestures and hand tip detection for performing mouse functions in the computer using
computer vision. The main objective of the proposed system is to perform computer mouse
cursor functions and scroll function using a web camera or a built-in camera in the computer
instead of using a traditional mouse device. Hand gesture and hand tip detection by using
computer vision is used as a HCI with the computer. With the use of the AI virtual mouse
system, we can track the fingertip of the hand gesture by using a built-in camera or web
camera and perform the mouse cursor operations and scrolling function and also move the
cursor with it.

While using a wireless or a Bluetooth mouse, some devices such as the mouse, the dongle to
connect to the PC, and also, a battery to power the mouse to operate are used, but in this
paper, the user uses his/her built-in camera or a webcam and uses his/her hand gestures to
control the computer mouse operations. In the proposed system, the web camera captures
and then processes the frames that have been captured and then recognizes the various
hand gestures and hand tip gestures and then performs the particular mouse function.

Python programming language is used for developing the AI virtual mouse system, and also,
OpenCV which is the library for computer vision is used in the AI virtual mouse system. In the
proposed AI virtual mouse system, the model makes use of the MediaPipe package for the
tracking of the hands and for tracking of the tip of the hands, and also, Pyautogui, Autopy,
and PyAutoGUI packages were used for moving around the window screen of the computer
for performing functions such as left click, right click, and
scrolling functions. The results of the proposed model showed very high accuracy level, and
the proposed model can work very well in real-world application with the use of a CPU without
the use of a GPU.
7

1.1. Problem Description and Overview.

The proposed AI virtual mouse system can be used to overcome problems in the real world
such as situations where there is no space to use a physical mouse and also for the persons
who have problems in their hands and are not able to control a physical mouse. Also, amidst
of the COVID-19 situation, it is not safe to use the devices by touching them because it may
result in a possible situation of spread of the virus by touching the devices, so the proposed
AI virtual mouse can be used to overcome these problems since hand gesture and hand Tip
detection is used to control the PC mouse functions by using a webcam or a built-in camera

1.2. Objective.

The main objective of the proposed AI virtual mouse system is to develop an alternative to
the regular and traditional mouse system to perform and control the mouse functions, and this
can be achieved with the help of a web camera that captures the hand gestures and hand tip
and then processes these frames to perform the particular mouse function such as left click,
right click, and scrolling function. The main objective of this project paper is to cut back the price
and to own accuracy at a high level besides of this we trying to help those people who are physically
challenged and unable to use their hands on physical mouse so we try to give them an easy
method to go their task easily and effortlessly method .

2. Related Work
8

There are some related works carried out on virtual mouse using hand gesture detection by
wearing a glove in the hand and also using color tips in the hands for gesture recognition, but
they are no more accurate in mouse functions. The recognition is not so accurate because of
wearing gloves; also, the gloves are also not suited for some users, and in some cases, the
recognition is not so accurate because of the failure of detection of color tips. Some e fforts
have been made for camera-based detection of the hand gesture interface.

In 1990, Quam introduced an early hardware-based system; in this system, the user
should wear a DataGlove . The proposed system by Quam although gives results of higher
accuracy, but it is difficult to perform some of the gesture controls using the system.

Dung-Hua Liou, ChenChiung Hsieh, and David Lee in 2010 proposed a study on “A Real-
Time Hand Gesture Recognition System Using Motion History Image.” The main limitation of
this model is more complicated hand gestures.

Monika B. Gandhi, Sneha U. Dudhane, and Ashwini M. Patil in 2013 proposed a study on
“Cursor Control System Using Hand Gesture Recognition.” In this work, the limitation is
stored frames are needed to be processed for hand segmentation and skin pixel detection.

Vinay Kr. Pasi, Saurabh Singh, and Pooja Kumari in 2016 proposed “Cursor Control using
Hand Gestures” in the IJCA Journal. The system proposes the different bands to perform
different functions of the mouse. The limitation is it depends on various colors to perform
mouse functions.

Chaithanya C, Lisho Thomas, Naveen Wilson, and Abhilash SS in 2018 proposed “Virtual
Mouse Using Hand Gesture” where the model detection is based on colors. But, only few
mouse functions are performed.

3. Algorithm Used for Hand Tracking

9

For the purpose of detection of hand gestures and hand tracking, the MediaPipe framework is
used, and OpenCV library is used for computer vision . The algorithm makes use of the
machine learning concepts to track and recognize the hand gestures and hand tip.

3.1. MediaPipe:

 MediaPipe is a framework which is used for applying in a machine learning pipeline,

and it is an opensource framework of Google. The MediaPipe frame-work is useful for
cross platform development since the framework is built using the time series data.
The MediaPipe framework is multimodal, where this framework can be applied to
various audios and videos. The MediaPipe framework is used by the developer for
building and analyzing the systems through graphs, and it also been used for
developing the systems for the application purpose.

 The MediaPipe framework is based on three fundamental parts; they are performance
evaluation, framework for retrieving sensor data, and a collection of components which
are called calculators , and they are reusable. A pipeline is a graph which consists of
components called calculators, where each calculator is connected by streams in
which the packets of data flow through. Developers are able to replace or define
custom calculators anywhere in the graph creating their own application.

 Single-shot detector model is used for detecting and recognizing a hand or palm in real
time. The single-shot detector model is used by the MediaPipe. First, in the hand
detection module, it is first trained for a palm detection model because it is easier to
train palms. Furthermore, the non-maximum suppression works significantly better on
small objects such as palms or fists.

 A model of hand landmark consists of locating 21 joint or knuckle co-ordinates in the

hand region, as shown in Figure 2.

3.2. OpenCV:

 OpenCV is a computer vision library which contains image-processing algorithms for

object detection.

 OpenCV is a library of python programming language, and real-time computer vision

applications can be developed by using the computer vision library. The OpenCV
library is used in image and video processing and also analysis such as face detection
and object detection .

4. Methodology
10

The various functions and conditions used in the system are explained in the flowchart of the
real-time AI virtual mouse system in Figure 3.

4.1. -e Camera Used in the AI Virtual Mouse System. The proposed AI virtual mouse
system is based on the frames that have been captured by the webcam in a laptop or PC. By
using the Python computer vision library OpenCV, the video capture object is created and the
web camera will start capturing video, as shown in Figure 4. The web camera captures and
passes the frames to the AI virtual system.

4.2. Capturing the Video and Processing. The AI virtual mouse system uses the webcam
where each frame is captured till the termination of the program. The video frames are
processed from BGR to RGB color space to find the hands in the video frame by frame as
shown in the following code:
11

FIGURE 2: Co-ordinates or land marks in the hand .

4.3. (Virtual Screen Matching) Rectangular Region for Moving through the Window. The AI
virtual mouse system makes use of the transformational algorithm, and it converts the co-
ordinates of fingertip from the webcam screen to the computer window full screen for
controlling the mouse. When the hands are detected and when we find which finger is up for
performing the specific mouse function, a rect-angular box is drawn with respect to the
computer window in the webcam region where we move throughout the window using the
mouse cursor, as shown in Figure 5.

4.4. Detecting Which Finger Is Up and Performing the Par-ticular Mouse Function. In this
stage, we are detecting which finger is up using the tip Id of the respective finger that we
found using the MediaPipe and the respective co-ordinates of the fingers that are up, as
shown in Figure 6, and according to that, the particular mouse function is performed.

4.5. Mouse Functions Depending on the Hand Gestures and Hand Tip Detection Using
Computer Vision
4.5.1. For the Mouse Cursor Moving around the Computer Window. If the index finger is up
with tip Id = 1 or both the index finger with tip Id = 1 and the middle finger with tip Id = 2 are
up, the mouse cursor is made to move around the window of the computer using the AutoPy
package of Python, as shown in Figure 7.

4.5.2. For the Mouse to Perform Left Button Click. If both the index finger with tip Id = 1 and
the thumb finger with tip Id = 0 are up and the distance between the two fingers is lesser than
30px, the computer is made to perform the left mouse button
click using the pyautogui Python package, as shown in Figures 8 and 9.
12

FIGURE 3: Flowchart of the real-time AI virtual mouse system.

13

FIGURE 4: Capturing video using the webcam (computer vision).

FIGURE 5: Rectangular box for the area of the computer screen where we can move the
cursor.

FIGURE 6: Detection of which finger is up.

FIGURE 7: Mouse cursor moving around the computer window.

14

4.5.3. For the Mouse to Perform Right Button Click. If both the index finger with tip Id = 1 and
the middle finger with tip Id = 2 are up and the distance between the two fingers is lesser
than 40 px, the computer is made to perform the right mouse button click using the pyautogui
Python package, as shown in Figure 10.

4.5.4. For the Mouse to Perform Scroll up Function. If both the index finger with tip Id = 1 and
the middle finger with tip

FIGURE 8: Gesture for the computer to perform left button click.

FIGURE 9: Gesture for the computer to perform left button click.

Id = 2 are up and the distance between the two fingers is greater than 40 px and if the two
fingers are moved up the page, the computer is made to perform the scroll up mouse function
using the PyAutoGUI Python package, as shown in Figure 11.

4.5.5. For the Mouse to Perform Scroll down Function. If both the index finger with tip Id = 1
and the middle finger with tip Id = 2 are up and the distance between the two fingers is
greater than 40px and if the two fingers are moved down the page, the computer is made to
perform the scroll down mouse function using the PyAutoGUI Python package, as shown in
Figure 12.

4.5.6. For No Action to be Performed on the Screen. If all the fingers are up with tip Id = 0, 1,
2, 3, and 4, the computer is made to not perform any mouse events in the screen, as shown
in Figure 13.
15

5. Experimental Results and Evaluation

In the proposed AI virtual mouse system, the concept of advancing the human-computer
interaction using computer vision is given.
Cross comparison of the testing of the AI virtual mouse system is di fficult because only
limited numbers of datasets are available. The hand gestures and finger tip detection have
been tested in various illumination conditions and also been tested with di fferent distances
from the webcam for tracking of the hand gesture and hand tip detection. An experimental
test has been conducted to summarize the results shown in Table 1. The test was performed
25 times by 4 persons resulting in 600 gestures with manual labelling, and this test

FIGURE 10: Gesture for the computer to perform right button click.

FIGURE 11: Gesture for the computer to perform scroll up function.

16

FIGURE 12: Gesture for the computer to perform scroll down function.
17

has been made in different light conditions and at di fferent distances from the screen, and
each person tested the AI virtual mouse system 10 times in normal light conditions, 5 times in
faint light conditions, 5 times in close distance from the webcam, and 5 times in long distance
from the webcam, and the experimental results are tabulated in Table 1.

From Table 1, it can be seen that the proposed AI virtual mouse system had achieved an
accuracy of about 99%. From this 99% accuracy of the proposed AI virtual mouse system, we
come to know that the system has performed well. As seen in Table 1, the accuracy is low for
“Right Click” as this is the hardest gesture for the computer to understand. The accuracy for
right click is low because the gesture used for performing the par-ticular mouse function is
harder. Also, the accuracy is very good and high for all the other gestures. Compared to
previous approaches for virtual mouse, our model worked very well with 99% accuracy. The
graph of ac-curacy is shown in Figure 14.

Table 2 shows a comparison between the existing models and the proposed AI virtual
mouse model in terms of accuracy.

From Table 2, it is evident that the proposed AI virtual mouse has performed very well in
terms of accuracy when compared to the other virtual mouse models. The novelty of the
proposed model is that it can perform most of the mouse functions such as left click, right
click, scroll up, scroll down, and mouse cursor movement using finger tip detection, and also,
the model is helpful in controlling the PC like a physical mouse but in the virtual mode. Figure
15 shows a graph of comparison between the models.
18

Accuracy
100
90
80
70
60
50
40
30
20
10
0

Performed
No Action
Le Button

Scroll Up
Movement

Function

Function
Button

Down
Scroll
Right
Click

Click
Mouse

Accuracy

FIGURE 14: Graph of accuracy.

TABLE 2: Comparison with existing models.

19

6. Future Accuracy Scope

Existing models (%)
Virtual mouse system using
The proposed RGB-D AI virtual mouse has some
limitations images and fingertip detection 96.13 such as small decrease in
accuracy of [16] the right click mouse function
Palm and finger recognition
and also the model has some difficulties in
based [17] 78
executing Hand gesture-based virtual clicking and dragging to select
the text. mouse [18] 78 These are some of the
limitations of The proposed AI virtual mouse the proposed AI virtual mouse
system, and system 99 these limitations will be
overcome in our future work.
Furthermore, the proposed method can be developed to handle the keyboard
functionalities along with the mouse functionalities virtually which is another future scope of
Human-Computer Interaction (HCI).

The above-mentioned points are the enhancements that can be done to increase the
applicability and usage of this project. We have left all the options open so that if there is any
other future requirement in the system by the user or students for the enhancement of the
application then it is possible to implement them. In the last, we would like to thank all the
persons involved in the development of the application directly or indirectly. We hope that the
project will serve the purpose for which it is developed thereby underlining the success of the
process.
20

7. Applications

The AI virtual mouse system is useful for many applications; it can be used to reduce the
space for using the physical mouse, and it can be used in situations where we cannot use the
physical mouse. The system eliminates the usage of devices, and it improves the human-
computer interaction.
Major applications:
(i) The proposed model has a greater accuracy of 99% which is far greater than the that
of other proposed.

(ii) Amidst the COVID-19 situation, it is not safe to use the devices by touching them
because it may result in a possible situation of spread of the virus by touching the
devices, so the proposed AI virtual mouse can be used to control the PC mouse
functions without using the physical mouse
21

(iii) The system can be used to control robots and automation systems without the usage
of devices

(iv)2D and 3D images can be drawn using the AI virtual system using the hand gestures.

(v)AI virtual mouse can be used to play virtual reality-and augmented reality-based
games without the wireless or wired mouse devices.

(vi)Persons with problems in their hands can use this system to control the mouse
functions in the computer.

(vii)In the field of robotics, the proposed system like HCI can be used for controlling
robots.

8. Conclusions

The main objective of the AI virtual mouse system is to control the mouse cursor
functions by using the hand ges-tures instead of using a physical mouse. The
proposed system can be achieved by using a webcam or a built-in camera
which detects the hand gestures and hand tip and processes these frames to
perform the particular mouse functions.
From the results of the model, we can come to a con-clusion that the
proposed AI virtual mouse system has performed very well and has a greater
accuracy compared to the existing models and also the model overcomes most
of the limitations of the existing systems. Since the proposed model has greater
accuracy, the AI virtual mouse can be used for real-world applications.
22

REFERENCES

[1]. D. L. Quam, “Gesture recognition with a Data Glove,” IEEE Conference on Aerospace
and Electronics, vol. 2, pp. 755–760, 1990.View at: Publisher Site | Google Scholar

[2]. S. U. Duchene, “Cursor control system using hand gesture recognition,” IJARCCE, vol. 2,
no. 5, 2013.View at: Google Scholar

[3]. L. Thomas, “Virtual mouse using hand gesture,” International Research Journal of
Engineering and Technology (IRJET, vol. 5, no. 4, 2018.View at: Google Scholar

[4]. J. Jaya and K. Thanushkodi, “Implementation of certain system for medical image
diagnosis,” European Journal of Scientific Research, vol. 53, no. 4, pp. 561–567, 2011.View
at: Google Scholar

[5]. P. Nandhini and J. Jaya, “Image segmentation for food quality evaluation using computer
vision system,” International Journal of Engineering Research and Applications, vol. 4, no. 2,
pp. 1–3, 2014.View at: Google Scholar © 2022 JETIR April 2022, Volume 9, Issue 4
www.jetir.org (ISSN-2349-5162) JETIR2204334 Journal of Emerging Technologies and
Innovative Research (JETIR) www.jetir.org d288

[6]. J. Jaya and K. Thanushkodi, “Implementation of classification system for medical

images,” European Journal of Scientific Research, vol. 53, no. 4, pp. 561–569, 2011.View at:
Google Scholar

[7]. Google,MP: https://ai.googleblog.com/2019/08/on -device-real-time-hand-tracking-

with.html.

[8]. https://google.github.io/mediapipe/solutions/hands : Mediapipe

[9]. https://www.autopy.org/documentation/apireference/mouse.html:Autopy