International Journal of Advance Research, Ideas and Innovations in Technology

ISSN: 2454-132X
Impact Factor: 6.078
(Volume 8, Issue 3 - V8I3-1162)
Available online at: https://www.ijariit.com

Cursor Movement on Object Motion

Davuluri Manikanta K. Harith
[email protected] [email protected]
Jain University, Bengaluru, Karnataka Jain University, Bengaluru, Karnataka
S. Sameer Basha M. Sumanth Reddy
[email protected] [email protected]
Jain University, Bengaluru, Karnataka Jain University, Bengaluru, Karnataka

ABSTRACT naturally is becoming even more important in many fields of

Human Computer Interaction.
This paper proposes a new approach for controlling mouse
movement using a real-time camera. Most of the existing The idea which the paper proposes uses gesture to operate the
approaches involve changing mouse parts such as adding more cursor. Gestures are a major form of human communication. The
buttons or changing the position of the tracking ball. Instead, primary goal of gesture recognition is to create a system that can
it proposes to change the hardware design. The method uses a identify specific human gesture and use them to correctly
camera and computer vision technology, such as image identify a corresponding action for controlling the device. A user
segmentation and gesture recognition, to control mouse tasks can control a computer by making a specific gesture in front of a
(left click and right click, selection, scroll and drag). Hand video camera which is linked to the computer.
gestures are acquired using a Web camera based on colour
detection. In this research work, three colour tapes are used on This paper is based on human computer interaction. To make
fingers. The tapes will be used for clicking events of the mouse. fewer human efforts to operate their computer and to use hand
Through a Web camera, the real-time video is captured. Image gestures to communicate with the computer:
processing is performed on each frame of that video to detect • The paper emphasizes to command your computer
the colour and mouse tasks are performed. using natural hands gesture
• No remote controls required
Keywords: Hand Gesture, Colour Detection, Human • No wearable devices required
Computer Interaction (HCI), HSV (Hue, Saturation, Value) • Only one thing required: YOUR HAND
Colour Model, Web Camera, RYB (Red, Yellow, Blue). The aim is to create a virtual mouse system that is a userfriendly
1. INTRODUCTION device and an alternative to a touch screen. By applying
Human Computer Interaction, mainly deals with natural computer vision technology and controlling the mouse by natural
hand gestures, one can reduce the workspace required. This
interfaces. The Graphical User Interface (GUI) on Personal
Computers (PCs) is developed, providing an efficient interface paper proposes a novel approach that uses real-time video
for a user to interact with the computer and access the various capturing to control the mouse movement.
applications effortlessly.
2. LITERATURE SURVEY
Today, the technology used for interaction by mobile phone is Many researchers in the robotics and human computer
mainly touch screen. But this technology is still not that cheap to interaction fields have tried to control mouse movement using
be used in laptops and desktops. The aim is to create a virtual various devices. However, different technologies have different
mouse system having a user-friendly interface and an alternative methods to simulate the mouse movement.
to a touch screen.
In [1], the wearable unit will consist of buttons on the fingertip
Human Computer Interaction is not just limited to keyboard and for a click, an accelerometer inside the glove for sensing the
mouse interaction. The interaction between human beings comes motion and getting the direction of motion. This would be
from different sensory modes like gesture, speech, facial and connected to a microcontroller and a transmitter to communicate
body expressions. Being able to interact with the system wirelessly to the base unit. The accelerometer reads the tilt of

© 2022, www.IJARIIT.com All Rights Reserved Page |178

 International Journal of Advance Research, Ideas and Innovations in Technology
each axis and outputs each as an analog voltage. Push buttons are The aim of human computer interaction is to develop a system
used for enabling and disabling the mouse and for making a left that is easy to interact through natural interfaces. Presently,
click and right click. RF module is used to provide wireless computing environments include wired or wireless devices
communication. connected to your computer. Though many users may find this
interaction limited or inconvenient to always use a device.
In [2], Zhi-hua Chen uses real-time hand gesture recognition
using the finger segmentation method. The hand region is There are different methods to apply other techniques and
extracted from the background with the background subtraction sensors to enrich user experience. For example, a video camera
method Then, the palm and fingers are segmented to detect and and computer vision techniques may be used to capture the shape
recognize the fingers. Finally, a rule classifier is applied to and movements of the hand. The movements of the hand may be
predict the labels of hand gestures. Basically, what the position analyzed to effortlessly communicate with the system. This
of the fingers is based on that it will only show the pattern of approach may lead to a faster, more natural style of interaction
fingers on the screen. to perform certain tasks.

In [3], Chu-Feng Lien used only the fingertips to control the The paper, Mouse Simulation Using Three Colored Tapes is in
mouse cursor and click. His clicking method was based on image ubiquitous computing. Here, three colored tapes will be wrapped
density. It required the user to hold the mouse cursor on the around the fingers. One of the tapes will be used for controlling
desired spot for a short period of time. It starts with retrieving cursor movement while the relative distance between the center
images from camera drivers. Upon a valid image, it first finds of specific two-colored tapes will be used for the click events of
the location of the possible projected screen by Canny Edge the mouse. Thus, the system will provide a new experience for
detection. After finding the target screen, it calculates the users in interacting with the computer.
position mapping between device resolution and camera
resolution. The projected screen detection is executed after every 4.2 Technical Overview:
10 seconds in case the target screen or the camera is moved The platform used for implementation is Anaconda3. Anaconda3
accidentally. After the basic environment setup process, the is an open source distribution of Python language. Anaconda
system reads images in a ring buffer and shows the differences Navigator (virtual environment manager) helps in eliminating
between frames based on MHI. the need to learn to install each library independently also it gives
high-performance computing.
In [4], Kamran Niyazi used a web camera to detect colour tapes
for cursor movement. The clicking actions were performed by The program uses Image Processing, it is a method to perform
calculating the distance between two coloured tapes in the some operations on an image, in order to get an enhanced image
fingers. For left click on the very first step, the system records of the palm or to extract some useful data. It is a type of
the distance (say D) between the yellow and red tapes in the processing in which image or characteristics/features are
index finger and the thumb respectively. Here, the index and extracted associated with that image to perform mouse tasks. The
thumb must be apart as much as possible to get maximum program is implemented in Python. It uses image processing
distance. This distance is regarded as the threshold distance for module OpenCV and implements the mouse actions using
the event. Now, as the thumb moves towards the index finger, python specific library PyAutoGUI.
the distance between the fingertips or in other words, the distance
between yellow and red tapes is decreased. In the second step, Packages installed for successful compilation of the program are
when the thumb is closer to the index finger the system records Python interpreter to read and execute the code, OpenCV an
the reduced distance. When the distance between the tapes is open source computer vision is used for capturing real-time
reduced to D' or less then consider the event as the left click event video and for the analysis of features of the hand. NumPy for
of the mouse cursor. The concept of waiting time is used to numerical operations and PyAutoGUI is a library function that
simulate the right click event of the cursor. If the yellow tape on provides a method for controlling the mouse. Video captured by
the index finger is waiting for 7 seconds(say) in front of the the webcam is processed and only the three colored tips are
camera pointing at the same location, then the event is extracted. The distance between the centers of a particular color
recognized as the right click event of the mouse. Here, the is calculated using the method of moments and depending upon
distance between the red and yellow tapes should be between D their relative positions mouse actions are performed. # finding
and D' respectively. In the same way, the double click event of the centroid using the method of 'moments'
the cursor can also be made, considering the waiting time.

In [5], A. Erdem uses a fingertip tracking to control the motion

of the mouse. A click of the mouse button was implemented by
defining a screen such that a click occurred when a user’s hand
(fingertip) passed over the region.

3. PROBLEM STATEMENT
The hand gesture mouse is based on Human Computer
Interaction (HCI) and Image Processing which performs the
mouse functions such as cursor movement, left click, right click,
scrolling using hand gestures. The hand gesture is captured using
a web camera and image processing is done using OpenCV. The
system identifies the colored region on the fingertips.

4. PROPOSED SYSTEM OVERVIEW

4.1 System Overview Figure 1: Proposed system architecture

© 2022, www.IJARIIT.com All Rights Reserved Page |179

 International Journal of Advance Research, Ideas and Innovations in Technology
• If Red, Green, Blue are detected together, then Drag
action is performed.

4.3 Pseudo Code

Step 1. Conversion to HSV (Hue, Saturation, Value)
The video captured is converted into HSV format. This helps to
detect an object with a certain colour.

4.2.1 Capturing the real-time video

The web camera will be used as a sensor. It will be used to Step 2. Colour Calibration
capture the real-time video and to detect the hand movements. Colour calibration is used to measure and/or adjust the colour
The web camera captures the movements through a fixed frame response of a device. Input for colour range of each colour is in
size. Users can use an inbuilt camera or can externally connect a inRange().
camera.

4.2.2 4Hand Recognition and Colour Tape Detection

The first step is to separate the essential hand pixels from the
non-hand pixels. This can be done by a background subtraction
scheme which segments any potential foreground hand
information from the non-changing background. The
background image is captured and segmented, only the
Each colour range is calibrated by using cv2.inRange() function.
appropriate background is used for further processing. After
background subtraction, the process of skin segmentation is
done. Here, a histogrambased skin classifier assigns each of the
RYB pixels in the training set to either a 3D skin histogram or a
non-skin histogram. The skin segmentation process outputs an Step 3. Noise Removal
image that is ready for the detection of color tapes on the fingers. The noise removal algorithms reduce or remove the visibility of
For this, an algorithm based on HSV color space is used which noise by smoothing the entire image, leaving areas near contrast
is very effective to select a certain color out of an image. The boundaries. It is done in two step morphism i.e. erosion and
idea is to convert the RYB pixels into the HSV color plane so dilation. Dilation adds pixels to the boundaries of objects in an
that it is less affected by variations in shades of similar color. image, while erosion removes pixels on object boundaries.
Then, a tolerance mask is used over the converted image in the
saturation and hue plane. The resulting binary image is then run
through a convolution phase to reduce the noise introduced. Step 4. Centroid Detection
The centers of each colour is detected using the method of
4.2.3 Cursor Position moments.
Defining cursor position with yellow. The user might be left or
right handed, so a video frame of rectangular subportion of
smaller size is considered. Due to the noise captured and
vibrations of the hand, the centers keep dislocating the mean
position. It uses setcursorposition() so that the new center
position is set close to the old center. Now the three centers are
sent for deciding what action to be performed depending upon
their relative position. This is done by the chooseAction() in the Step 5. Cursor Position
code depending upon its output. The performAction() carries out The yellow coloured finger is used for cursor movement. Due to
either of the following pyaotogui library function: click, select, the noise captured while calibrating the colour the centroid
drag, scroll. vibrates, to reduce the vibrations of the cursor it compares the
new position with the previous position of the cursor using the
4.2.4 Controlling mouse def setCursorPos(). The new cursor position is calculated in such
a) Moving mouse cursor a way that the mean deviation from the desired steady state is
For finding the cursor on the screen and for the cursor moment reduced. Step 6. Perform Mouse Movements
any specific color (yellow) on a finger is used to perform the Based on the position of the cursor left click, right click, scroll
movement on the screen. b) Clicking Action up, scroll down, drag/select are performed.
Clicking action is based on the detection of the colors and center
tip of the finger
• If Blue along with Red is detected, then Left Click
action is performed.
• If Red along with Green is detected, then Right Click
action is performed.
• If Red is detected and the movement of the finger is in
the downward direction than Down Scroll action is performed.
• If Red is detected and the movement of the finger is in
the upward direction than Up Scroll action is performed.

© 2022, www.IJARIIT.com All Rights Reserved Page |180

 International Journal of Advance Research, Ideas and Innovations in Technology
5.3.2 Right click
When the Red and Yellow tapes are brought together and cross
a certain threshold distance, right-click action is performed.

5. RESULT AND DISCUSSION

5.1 Calibration
Each colour value is adjusted so that centroid of each colour is
detected.

Figure 5: Hand gesture for Right click action

5.3.3 Drag and select

All three tapes should be together for selection and dragging. The
distance should be minimized between the centers of the colour.
Figure 2: Calibration

5.2 Centroid Detection

Centers of yellow, red, blue are detected.

Figure 6: Hand Gesture for Drag and Select

5.3.4 Scrolling up
Move the blue coloured finger in the downward direction to
scroll up.
Figure 3: Centroid detection

5.3 Mouse Function

5.3.1 Left click
When the Red and Blue tapes are brought together and cross a
certain threshold distance left click action is performed.

Figure 7: Hand gesture for Scroll up operation

5.3.5 Scrolling down

Move the red coloured finger in the downward direction to scroll
down.
Figure 4: Hand gesture for Left click action

© 2022, www.IJARIIT.com All Rights Reserved Page |181

 International Journal of Advance Research, Ideas and Innovations in Technology
windows, closing the window, etc. will be implemented. This
method is created under a standardized operating system.
Various application programs can be developed exclusively from
this technology to create a wide range of applications with the
minimum requirements of resources.

8. REFERENCES
[1] Chetana S. Ingulkar, A.N. Gaikwad; Hand Data Glove: A
wearable real-time device for human computer Interaction;
J. International Journal of View publication stats Science
and Engineering, ISSN-2347-2200, Volume 1, Number 2;
2013.
[2] Zhi-hua Chen, Jung-Tae Kim, Jianning Liang, Jing Zhang,
and Yu-Bo Yuan; Research Article Real-Time Hand
Gesture Recognition Using Finger Segmentation;
Figure 8: Hand gesture for Scroll down operation Department of Computer Science and Engineering, East
6. FUTURE SCOPE China University of Science and Technology, Shanghai
Future work can involve the introduction of an infrared sensor, 200237, China;25 June 2014.
so it can detect hands perfectly when the lighting conditions are [3] Chu-Feng Lien; "Portable Vision-Based HCI – A Real-
not good. In a progressive implementation, this method can be time Hand Mouse System on Handheld Devices"; National
implemented without using coloured tapes. The application can Taiwan University, Computer Science and Information
be further extended to gaming. Gestures for page zoom in and Engineering Department.
zoom out can be implemented. This technology has wide [4] Kamran Niyazi, Vikram Kumar, Swapnil Mahe, and
applications in the fields of augmented reality, biomedical Swapnil Vyawahare; Mouse Simulation Using Two
instrumentation, computer graphics, prosthetics, and computer Coloured Tapes; Department of Computer Engineering,
gaming. AISSMS COE, University of Pune, India International
Journal of Information Sciences and Techniques (IJIST)
7. CONCLUSION Vol.2, No.2; March 2012.
The system implemented will simulate mouse movement using [5] A. Erdem, E. Yardimci, Y. Atalay, V. Cetin; Computer
hand gestures and a real-time camera. All the mouse tasks such vision-based mouse, A. E. Acoustics, Speech, and Signal
as left clicking, right clicking, and scrolling are implemented. In Processing; Proceedings. (ICASS). IEEE International
the future, additional features such as enlarging and shrinking Conference; 2002.

© 2022, www.IJARIIT.com All Rights Reserved Page |182