ISSN (Online) 2581-9429

IJARSCT
International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

Volume 3, Issue 2, March 2023

Impact Factor: 7.301

Computer Vision Based Virtual Mouse and

Keyboard System
Akanksha Pardeshi1, Darshani Kadam2, Nishant Thorat3, Utkarsh Pardeshi4, Prof. Priyanka Khalate5
Students, Department of Computer Engineering1,2,3,4
Assistant Professor, Department of Computer Engineering5
Smt. Kashibai Navale College of Engineering, Vadagaon(BK) Pune, Maharashtra, India
Savitribai Phule Pune University, Pune

Abstract: Gesture recognition is a technology that uses computer algorithms to interpret human gestures
as commands for a computer or other device. A virtual mouse and keyboard project that uses gesture
recognition would involve creating a system that can recognize specific hand or body movements and
translate them into actions on the computer, such as moving the cursor or typing on a virtual keyboard. The
abstraction for this project would involve developing the algorithms and programming necessary to detect
and interpret the gestures, as well as designing the user interface and integrating the system with the
computer's operating system.

Keywords: Convex Hull, Defects, Image Processing, Frame Extraction

I. INTRODUCTION
Gesture recognition technology is becoming an increasingly popular way for people to interact with their devices. In
this project, we propose to develop a virtual mouse and keyboard using gesture recognition technology. The system will
be able to recognize specific hand or eye movements and translate them into actions on the computer, such as moving
the cursor or typing on a virtual keyboard. The goal of this project is to design and implement a gesture recognition
system that is accurate, responsive and easy to use. We will be using visual gesture recognition, where the system uses
cameras or other imaging devices to detect and interpret hand or eye movements. The project will involve researching
and evaluating different image processing and computer vision techniques, as well as developing and testing a user
interface that is intuitive and easy to use. Once the gesture recognition system and user interface have been developed,
the final step will be to integrate the system with the computer's operating system. The end result of this project will be
a virtual mouse and keyboard that allows users to control their computer using natural hand and eye movement .

II. MOTIVATION
Gestures- Common in personal communication. Computer technology has tremendously grown over the past decade
and has become a necessary part of everyday live. The most natural and intuitive technique for HCI, that is a viable
replacement for the computer mouse is with the use of Eye Movements and Keyboard with hand gestures . This project
is therefore aimed at investigating and developing a Computer Control (CC) system using Eye Movements and Hand
Gestures. Gives a wide scope in developing a unique way of Human-Machine Interaction.

III. LITERATURE SURVEY

In general, the Human Computer Interaction progresses toward interfaces that seem to be natural and intuitive to use
rather than the customary usage of keyboard and mouse is Discussed in paper
[1]To improve the recognition of human hand postures in a HCI applications , the reducing the time computing and to
improve the user computing regarding the used human hand postures. In this they have developed the application for
computer mouse control and the application is based on proposed algorithm , hand pad colour and in this way they had
increase comfort in use of system due to proposed hand postures and also system works well having the same behaviour
under very low illuminance level and high illuminance level .

Copyright to IJARSCT DOI: 10.48175/IJARSCT-8905 693

www.ijarsct.co.in
 ISSN (Online) 2581-9429
IJARSCT
International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

Volume 3, Issue 2, March 2023

Impact Factor: 7.301

[2]A recognition system, which can be helpful for a blind person. Hand gesture recognition system and face recognition
system has been implemented in this paper using which various tasks can be performed. Dynamic images are being
taken from a dynamic video and is being processed according to certain algorithms. In the Hand gesture system Skin
colour detection has been done in YCbCr colour space and to discover hand convex defect character point of hand is
used where different features like fingertips, angle between fingers are being extracted. According to gesture
Recognized, various tasks can be performed like turning on the fan or lights. While in face recognition, Haar Cascade
Classifiers and LBPH recognizer are being used for face detection and recognition respectively. With the help of
OpenCV, The research has been implemented. Various hand gestures and human faces have been detected and
identified using this system.
[3]Solutions to the problem of accuracy and it presents a combined hand gesture recognition system that uses a hand
detector to detect hand in the frame and then switches to gesture classifier if a hand was detected. The paper illustrates
the proposed combined algorithm. Descriptions of used hand detector and gesture recognition algorithms also are given.
Equations for the evaluation of potential performance increase and experimental results are presented. The proposed
system is tested on publicly accessible gesture bases and on video sequences prepared by the authors. The experimental
results are consistent with theoretical estimates and demonstrate the benefits of the proposed gesture recognition system
design.
[4] The visualization of the 3D models is a scorching topic in computer vision and human-computer interaction. The
demands for 3D models have been increased due to high involvement in animated characters, virtual reality and
augmented reality. To interact with 3D models with the help of mouse and keyboard is a very hectic, less efficient and
complex process because of multiple types of operations required by the models to view properly in all sides. So it is
essential to improve the user interaction with the 3D system. In this paper, a new method is introduced by using the
Microsoft Kinect v2 to detect the human body and joints. First, we trained the Kinect to understand the specific
gestures, and then recognize to perform the specific task on an object in the proposed environment.

[5]presents the maneuver of mouse pointer and performs various mouse operations such as left click, right click, double
click, drag etc using gestures recognition technique. Recognizing gestures is a complex task which involves many
aspects such as motion modeling, motion analysis, pattern recognition and machine learning. Colour caps have been
used for fingers to distinguish it from the background colour such as skin colour. Thus recognizing the gestures various
mouse events have been performed. The application has been created on MATLAB environment with operating system
as windows.
[6] We can measure the eye movement activity using eye tracking technology. Eye tracking gives us information about
where do we look? What is ignored and how the pupil reacts to different stimuli. The eye tracking concept is basic but
its process and interpretation can be very diverse and complex. ET measures the gaze points generated by our eye
relative to the head. Eye trackers are available in either remote or mobile forms. It tracks and records where do we look
and how we move the gaze. One can analyse, visualize and interpret this information with the help of software. We
have gone through the common use of fingerprint analysis and applications, eye tracking also would be a great
biometric tool for analysis in various applications. In this paper they discuss eye tracking technology and its various
applications. Now days, ET is being employed in almost all field including psychology, human computer interaction,
marketers, designers, academics, medical, research and many more.
[7]Using, system transformation algorithm which converts the coordinates of the fingers from webcam screen to
computer window then the fingers are detected and according to that by recognizing the gestures further action will be
done .
[8] And now finally we are going to do the computer vision based virtual mouse system with eye movements.In this we
are using Linear Regression Algorithm and Haar cascade algorithm and addition we are doing virtual keyboard feature.

IV. PROPOSED SYSTEM

The algorithm for a gesture recognition based virtual mouse and keyboard project would depend on thespecific
techniques and methods used for gesture recognition. However, a general outline of a possible algorithm could include
the following steps:
Copyright to IJARSCT DOI: 10.48175/IJARSCT-8905 694
www.ijarsct.co.in
 ISSN (Online) 2581-9429
IJARSCT
International Journal of Advanced Research in Science,, Communication and Technology (IJARSCT)

Volume 3, Issue 2, March 2023

Impact Factor: 7.301

1. Capture and Pre-processing

processing of Image Data:-The
Data: The system captures image data using a camera or other imaging
device. The image data is then pre-processed
pre processed to remove noise and improve the quality of the image.
2. Feature extraction:- The system extracts features
featu from the pre-processed
processed image data that can be used for
gesture recognition. This may include extracting information about the shape, size, and position of the hand or
body.
3. Classification:- The system classifies the extracted features as a specific gesture.
gesture. This may be done using
machine learning techniques such as neural networks or decision trees.
4. Gesture Interpretation:- The system interprets the classified gesture and maps it to a specific action on the
computer, such as moving the cursor or typing on a virtual keyboard.
5. Feedback:- The system provides feedback to the user, such as highlighting the recognized gesture on the
screen.
6. Continuous Monitoring:-TheThe system continuously monitors for new gestures and repeat the steps above.
It's important to note that the above algorithm is a general outline and the specific steps and methods used will depend
on the chosen technique, the system's design and other factors. Additionally, it could be necessary to include additional
steps to improve the performanceance and robustness of the system, such as handling occlusions, dealing with different
lighting conditions .

Fig .1 Proposed System Architecture

A possible system for a gesture recognition based virtual mouse and keyboard project could include a camera or other
imaging device to capture image data of the user's hand or eye movements. This image data would then be pre pre-
processed and analyzed using image processing and computer vision techniques to extract relevant features for gesture
recognition. A machinee learning module would then be used to classify these features as specific gestures.
The system would also include a gesture recognition module to interpret the classified gestures and map them to
specific actions on the computer such as moving the cursor or typing on a virtual keyboard. A user interface module
would provide feedback to the user, displaying the recognized gestures on the screen. Additionally, an operating system
integration module would allow for seamless interaction between the gesture recognition
recognition system and the computer's

Copyright to IJARSCT DOI: 10.48175/IJARSCT-8905 695

www.ijarsct.co.in
 ISSN (Online) 2581-9429
IJARSCT
International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

Volume 3, Issue 2, March 2023

Impact Factor: 7.301

operating system. The system could also include data storage and analysis module for performance improvement over
time.

V. CONCLUSION
This paper is proposing a system to recognize the hand gesture and replace the mouse and keyboard function. That
includes the movement of the mouse cursor, the drag and click with the keyboard features like printing alphabets and
other keyboard functions . The process of skin segmentation is utilized to separate the colour/image of hand with its
background. In general, the proposed algorithm can detect and recognize hand gesture so that it can operate mouse and
keyboard features and also create a real world user interface. 3d printing, Architectural drawings and even doing
medical operations from anywhere to everywhere.

VI. ACKNOWLEDGMENGT
We would like to express our gratitude to the researchers and publishers for making their resources available. We also
appreciate the guidance and feedback provided by the reviewers and thank the college authorities for providing the
necessary infrastructure and support throughout the research process.

REFERENCES
[1]. Mr.abhinavrajsurendra prasad “cursor movement by hand gesture” international research journal of
modernization in engineering technology and science on may 2022.
[2]. Mr.rachit puri “gesture recognition based mouse events” international journal of computer science and
information technology on 2021.
[3]. Mr. thomasledwon “drivermhg: a multi-modal dataset for dynamic recognition of driver micro hand gestures
and a real-time recognition framework” 2020 15th ieee international conference on automatic face and gesture
recognition (fg 2020) on 2020.
[4]. Mr.muhammadjehanzeb “a new 3d viewer system based on hand gesture recognition for smart interaction”
2020 international conference on computing and information technology, university of tabuk, kingdom of
saudiarabia. volume: 01, issue: iccit- 1441, page no.: 116- 119, on 9* & 10* sep. 2020.
[5]. Mr.dmitryvorotnev “combining hand detection and gesture recognition algorithms for minimizing
computational cost” ieeexplore(national research university of electronic technology moscow, russia) on 2020.
[6]. Mr.Saransh Sharma “A Static hand gesture andface recognition system for blind people”6th international
conference on signal processing and integrated networks(spin) on 2019.
[7]. Mr.Gubbala Prasanth “virtual mouse implementation using opencv”journal of emerging technologies and
innovative research(JETIR)on June 2020.
[8]. Mr.S.Shrirams Rao k “DL based real time AI virtual mouse system using hand gesture”Hindvi journal of
healthcare engineering on 2021.
[9]. Mr.Shrinivas “Hand gesture recognition and appliance control using transfer learning”International journal of
engineering research and applications on 2021.

Copyright to IJARSCT DOI: 10.48175/IJARSCT-8905 696

www.ijarsct.co.in