International Journal of Computer Applications (0975 – 8887)

Volume 72– No.17, June 2013

Implementation of Hand Detection based Techniques for

Human Computer Interaction

Amiraj Dhawan, Vipul Honrao

Dept of Computer Engineering
Fr. C. Rodrigues Institute of Technology, Vashi
Navi Mumbai, India

ABSTRACT interaction with machines explained under Extraction of Input

The computer industry is developing at a fast pace. With this in implementation. These three different types of interaction
development almost all of the fields under computers have are: Finger Counting, Hand Orientation and Finger Tracking.
advanced in the past couple of decades. But the same Further the paper explains a few applications where these
technology is being used for human computer interaction that different interaction mechanisms are used to enhance the
was used in 1970’s. Even today the same type of keyboard interaction between the user and machine.
and mouse is used for interacting with computer systems.
With the recent boom in the mobile segment touchscreens
2. TECHNOLOGIES USED
have become popular for interaction with cell phones. But 2.1 OpenCV
these touchscreens are rarely used on traditional systems. This OpenCV (Open Source Computer Vision Library) is a library
paper tries to introduce methods for human computer mainly aims at real-time computer vision systems, developed
interaction using the user’s hand which can be used both on by Intel. The library is platform independent. It mainly
traditional computer platforms as well as cell phones. The focuses on real time Image Processing and Computer Vision.
methods explain how the user’s detected hand can be used as Computer Vision is a science and technology that is used to
input for applications and also explain applications that can process images to obtain relevant information. OpenCV
take advantage of this type of interaction mechanism [1] [2]. library is written in C language with its primary application
interface available in C. It provides wrapper classes for use in
General Terms C++ language as well. Also there are full interfaces available
Image Processing, Human Computer Interaction for various other languages such as Python, Java, MATLAB
and Octave. It provides basic data structures for Matrix
Keywords operations and Image Processing with efficient optimizations.
OpenCV, Thresholding, Contour, Convex Hull, Convexity It provides fast and efficient data structures and algorithms
Defects, Gesture Controlled Robot, Pick and Place Robot, which can be used for complex real time Image Processing
Finger Tracking, Hand Orientation and Computer Vision applications [3].
1. INTRODUCTION 2.2 C++
In today’s age a lot of research is done for finding effective C++ (pronounced as see plus plus) is a multi-paradigm,
techniques and methods to make existing systems more general-purpose and compiled programming language. It
reliable and efficient. One of the most important parameter to comprises of a combination of high-level language features
make system efficient and reliable is Human Computer like OOPS concepts and low-level language features like
Interaction (HCI). Many systems provide simple techniques memory access using pointers etc. It added object oriented
for HCI, most common techniques for input to the systems concepts like classes to the C language. We use the C++
include use of mouse, keyboard etc. These are physically in interface of OpenCV for the implementation. The C++
contact with the system. Recently new techniques have been interface provides a powerful data structure Mat (short for
developed to make interaction with the system more efficient. Matrix) which is efficient, portable and easy to use. Mat is
This paper presentsthree interactive techniques for Human basically just a multi-channel and multi-dimensional array
Computer Interaction using the hand detection. The paper which can be used to store images, intermediate image
provides efficient and reliable hand based interaction transformations, values just like a normal Array.
techniques, which can be implemented using specific
methodologies according to user’s applicationrequirement. 3. IMPLEMENTATION
The techniques described in the paper requires preprocessing
like transformation of the users hand image from one form to
3.1 Hand Segmentation
Hand segmentation deals with separating the user’s hand from
another, enhancements etc for implementation.These
the background in the image. This can be done using various
preprocessing tasks can be done using various methods
different methods. The most important step for hand
depending on the environment around in which the system is
segmentation is Thresholding which is used in most of the
being used by the specific user. This paper tries to compare
methods described below to separate the hand from the
the different methods for all the preprocessing tasks required
background.
and then create a pipeline of these preprocessing tasks and
then use the detected hand as an interaction device for HCI Thresholding can be used to extract an object from its
applications. Next part contains the three different types of background by assigning intensity values for each pixel such

6
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

even in constant lighting conditions during every system use,

the system might fail depending on the user’s hand color. If
the user’s hand is also darker in color, the system might not be
able to separate the user’s hands and the dark background.
The figures 2 and 3 below show the thresholdedinput image
from figure 1 using a static threshold value of 70 and 20. The
noise introduced in figure 3 clearly shows how using a bad
thresholding value can introduce noise which can reduce the
accuracy of hand detection.

Figure 1: Input Image Frame

that eachpixel is either classified as an object pixel or a

background pixel. Thresholding is done on the input image
according to a threshold value. Any pixel with intensity less
than the threshold value is set to 0 and any pixel with intensity
more than the threshold value is set to 1. Thus the output of
thresholding is a binary image with all pixels 0 belonging to
the background and pixels 1 represent the hand. Therefore the
white blob that is pixels having value 1 is the object area. In
Figure 2:Thresholded Image with threshold value as 70
our case the object is the user’s hand. The most important
component for thresholding is the threshold value. There are
various methods to select the appropriate threshold value for
better result explained in the further points [5].

Hand Segmentation methods are basically divided into 2

broad types, one is with background constraints and the other
one is with relaxed background constraints. These two types
are explained as follows:

3.1.1 With Background Constraint

In this type of segmentation, some constraints are put on the
background to extract hand blob without much noise. In this
type of hand segmentation, intensity of the pixels is used for
segmenting the user’s hand. Typically intensity of hand is
much higher, so by keeping background dark, hand can be
easily segmented. This type includes the following methods
for segmentation:

3.1.1.1 Static Threshold Value

Image frame is taken as input from the webcam in the RGB Figure 3: Thresholded Image with threshold value as 20
format. This image is converted into grayscale. Then either a
static threshold value is used or a threshold value is selected Thresholding can be performed in C++ using OpenCV using
from 0 to 255 according to the user specification which acts as the following function [6]:
the threshold value. This threshold value should be chosen by
the user in such a way that the white blob of the hand is doublethreshold(InputArraysrc,
segmented with minimum noise possible. A Trackbar can be OutputArraydst,
provided to adjust the threshold value for the current usage double thresh,
scenario. doublemaxval,
int type);
For every usage, either the thresholdingvalue is static that is
each time same value is used or the user is required to set the  src - is of data type InputArray and contains the
threshold value to ensure good level of hand segmentation. source image or frame
Thus this method is not used since it puts the systems success
or failure dependant on the user setting a proper threshold  dst - is of data type OutputArray and is the
value or on the quality of the static threshold value.This destination space for the thresholded image
method is useful where the intensity of the hand is almost
similar whenever the system is used. Also the background  thresh - is of type double and it is the threshold
intensity should be similar every time the system is used. But value that is to be selected

7
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

 maxval - is only used in some of the different ways Using OpenCV library this is done by passing a special
of applying the thresholding function. In the current case parameter THRESH_OTSU as the variable type in the
of binary thresholding does not use this variable thresholding function given in 3.1.1.1 Static Thresholding.
The thresholding function ignores thevalue in the variable
 type - defines the type of thresholding applied on the thresh which was used to pass the threshold value in earlier
image. In our case this is THRESH_BINARY cases. But in place of that the function calculates the threshold
value using the Otsu’s Method and threshold the image as per
this threshold value which minimizes the within group
3.1.1.2 Incremental Thresholding Value variance.
In this method, same pre-processing as in the static
thresholding value is done on the image input frame,
converting from RGB to Grayscale. Instead of using a
constant value for every input image frame, the threshold
value is incremented till a condition is not met. For this
method, a minimum threshold value is set and then the input
image frame is thresholded using this value. If the current
thresholding value does not fulfil the condition, then the
thresholding value is incremented and again the same
procedure is followed till the condition is met. The condition
to detect hand is until only one white blob is present in the
thresholded image. The detected white blob can also be some
other object so whether the detected object is a hand or not is
decided by the hand detection part explained further.

This method can automatically select the threshold value. This

method generally gives good results especially in the
environment where intensity values of input image frame
Figure 4: Thresholded Image using Otsu Thresholding
changes continuously. This method ensures that the entire
hand is detected as a whole blob without any internal
The advantage is that this method works well under any
fragmentation. But on the negative side, sometimes the
circumstances until the hand and background pixels create
background pixels near the hand might also get included in
distinct peaks in the histogram of the image. The only
the white blob. Also if the background is not constantly dark,
problem with this method is that if the user’s hand is not in
some areas of the background might also add up in with the
view, the method would give a threshold value which breaks
hand in the white blob at certain threshold values and still
up the background pixels into two separate classes making it
make only one white blob. That is even though it would pass
difficult to understand that the hand is not in view. This
the condition that only white blob is present but the white
problem can be solved again by using the tests explained
blob would consist of the hand and the lighter background
further to make sure the detected white blob is a hand only.
areas that are connected to the hand.
Since the chances of the background getting thresholded in a
way that the white blob passes the hand detection test are
To remove these problems a test has to be conducted to find if
extremely less, the system practically does not give any false
the white blob has a structure similar to hand or not using
positives.
convexity defects explained further.

In this method, the function described in 3.1.1.1 is used i.e. 3.1.1.4 Dynamic Thresholding using Color at
Static Thresholding. But in this case the thresholding value Real Time
keeps on incrementing, that is the variable thresh is
incremented by value ‘1’ in a range from 20 to 160, till we Unlike previous methods of thresholding, in this method color
detect only one contour in the input image. based thresholding is done. This can also be termed as color
level slicing. Initially the user has to give some dummy input
3.1.1.3 Thresholding using Otsu’s Method image frames with the hand to be detected in the central part
Otsu’s Methodis used to automatically select a threshold value of the image. The system would do the analysis on these
based on the shape of the histogram of the image. This dummy input frames and generating dynamic threshold values
algorithm assumes that the image contains two dominant in RGB. In this analysis, a small central circular part, with
peaks of pixel intensities in the histogram that is two classes arbitrary radius, of the dummy input frames is considered
of pixels. The histogram should be bi-model for using this initially. The first two pixels of the central part are set as
method. The two classes are foreground and background. The minimum pixel value and maximum pixel values. Then rest
algorithm tries to find the optimal threshold value using which all pixels in the central part are processed. For every pixel
the two classes are separated in such a way that their intra- value that is scanned, it is compared with the minimum and
class variance or combined spread is minimal. The threshold maximum pixel values. If the scanned pixel value is less than
value given by the Otsu’s Method thus in our case works well the minimum pixels value then the minimum pixel value is
since the images contain two type of pixels, background updated to the scanned pixel value. Similarly if scanned pixel
pixels and hand pixels. Thus the two classes are background value is more than that of maximum pixel value, then the
and hand. So the threshold value tries to separate the peaks in maximum pixel value is updated to the scanned pixel value.
order to give minimal intra-class variance. The example The range defined by the Minimum and Maximum pixel value
output is given in the figure 4. is used to threshold the image, whichever pixel comes
between this ranges is considered as hand pixel.

8
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

This method is very accurate to segment the hand if the gradient is always perpendicular to the contour lines. When
intensity of the hand does not alter much during usage. It can the lines are close together, the magnitude of the gradient is
detect any color of hand thus making it independent of the usually very large. Contours are straight lines or curves
user’s skin color. The dummy input frames should have the describing the intersection of one or more horizontal planes
hand in the central part else the entire system collapses since with a real or hypothetical surface with.
the range decided is not actually for the hand. The background
should not contain pixels with values that fall in between the The contour is drawn around the white blob of the hand that is
decided range as they too would be included as hand pixels. found out by thresholding the input image. There can be
possibilities that more than one blob will be formed in the
3.1.2 Relaxed Background Constraints image due to noise in the background. So the contours are
drawn on such smaller white blobs too. Considering all blobs
3.1.2.1 Color based Thresholding formed due to noise are small, thus the large contour is
In color based thresholding, static values of hand color are considered for further processing specifying it is contour of
considered for thresholding. RGB values of hand are taken hand.
with minimum RGB value and maximum RGB value as a
range. These ranges are selected after analysing the general In this implementation, after preprocessing of the image
range of color of human hands. Then the input image frame frame, white blob is formed. Contour is drawn around this
uses these two minimum and maximum RGB values for
thresholding. Any pixel between this range is considered as
the hand pixel so it is set to 1 and pixels outside this range are
considered as background pixel is set to 0. As there are no
background constraints in this method, it is highly prone to
noise.

This method can also work on general backgrounds with a

slight constraint that the background should not contain pixels
that lie between the ranges specified. Else extra processing
like selecting the largest contour explained further is required
to ensure such white blobs are not detected as hand. The
thresholding values are very tricky to select. For some
user’s,the color of the hand could vary a lot and be outside the
specified range thus making the system unable to detect such
user’s hand.

3.1.2.2 Background Subtraction

This method is useful when the background remains static Figure 5:Detected Contour for the Input Image
throughout usage. There are no special constraints on the
background except there should not be any or much change in whiteblob. Vector contains set of contour points in the
it. An image of the background is provided to the system coordinate form. Figure 5 shows the detected contour for the
without the user’s hand in view. Using this background image input image.
provided, the process of background subtraction is done.
Whenever a new image frame is processed, it is converted For finding contours in OpenCV using C++ as the language
into gray scale. Then the gray level background image is the following function is used [6]:
subtracted from the gray level of the image frame. The
difference between the background image and the current voidfindContours(
image frame is the hand which is in view of the image frame InputOutputArray image,
but not in the background image. Thus the user’s hand is OutputArrayOfArrays contours,
easily segmented in any type of background. int mode,
int method,
This system works very well with any kind of background. Point offset=Point());
The hand is very accurately segmented irrespective of the
color of the hand or the backgroundcolor. The only negative  image– is the source image
with this system is that most of the times it is impossible to
have a completely static background. If the background  contours– is a vector of vector of Point data structure
changes much in the current image frames, the difference defined by OpenCV
between the current background and the initial background
can also become significant enough to cause the system to  mode – is the contour retrieval mode. We use
give false positives. CV_RETR_EXTERNAL which retrieves only the
extreme outer contours. Other modes are also available
3.2 Hand Detection like CV_RETR_TREE, CV_RETR_CCOMP,
CV_RETR_LIST
3.2.1 Contours
A contour is the curve for a two variables function along  method– is to select the contour approximation method.
which the function has a constant value. A contour joins We use CV_CHAIN_APPROX_SIMPLE which is
points above a given level and of equal elevation. A contour capable of compressing vertical, horizontal and diagonal
map illustratesthe contour using contour lines, which shows segments and stores only their end points. This provides
the steepness of slopes and valleys and hills. The function’s sufficient accuracy with lesser storage requirement.

9
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

Other methods are CV_CHAIN_APPROX_NONE,

CV_CHAIN_APPROX_TC89_L1,
CV_CHAIN_APPROX_TC89_KCOS

 offset – this is an optional parameter to add an offset

to each point stored for the contour

To draw the detected contours on a new white colored image,

we use the following function provided in OpenCV for C++:
voiddrawContours(
InputOutputArray image,
InputArrayOfArrays contours,
intcontourIdx,
const Scalar&color,
int thickness=1,
intlineType=8,
InputArray hierarchy=noArray(),
intmaxLevel=INT_MAX, Figure 6: Calculated ConvexHull of the Input Image
Point offset=Point() );
For detecting the convex hull of a contour the following
 image – is the input image on which the contours function is used [6]:
should be drawn. In our case this is a new white image of
void convexHull(
same size as input image
InputArray points,
OutputArray hull,
 contours– is a vector of vector of Point data structure bool clockwise=false,
defined by OpenCV. These are the contours that we want boolreturnPoints=true );
to draw
 points –this is the set of points for which the convex
 contourIdx – index of the contour to be drawn from hull is calculated. In our case we would pass the largest
the contours vector. If this value is negative then all the contour as a set of points to find the convex hull around
contours in the parameter contours are drawn the said contour

 color – is of the data type Scalar and it is a constant  hull – this acts as the storage for the hull output.
value. This is the color to be used to draw the contours Either it is a vector of indices of the points parameter if
all the hull points are present in the points vector or the
 thickness – defines the thickness of the line to be points itself if all the hull points do not exist in the points
used to draw the contours. If this value is negative, then parameter
the contours are filled with the color specified. Default
value is set to 1  clockwise – this is the orientation flag. If it is true
then the output convex hull is oriented in the clockwise
 lineType – is to select how the lines are drawn direction. Else in the anti-clockwise direction. By default
whether they are 8-connected line, 4-connected line and it is set to false
CV_AA for antialiased line
 returnPoints – if the points parameter is a vector
 heirarchy – this is optional information only in place of a matrix which it is in our case, the
required when only some of the contours are required to returnPoints variable is ignored. Else in case of a matrix,
be drawn this flag decided whether the hull output storage stores
the indices of points or the points itself. By default this is
set to true
 offset – optional offset to add to contour points
while drawing The convex hulls are also drawn on an image using the same
3.2.2 Convex Hull function drawContours as explained in section 3.2.1 Contours.
The convex hull of a set of points in the euclidean space is the This is because both, contours and convex hulls are nothing
smallest convex set that contains all the set of given points. but a collection of points which needs to be connected with
For example, when this set of points is a bounded subset of straight lines.
the plane, the convex hull can be visualized as the shape
formed by a rubber band stretched around this set of points. 3.2.3 Convexity Defects
When the convex hull is drawn around the contour of the
Convex hull is drawn around the contour of the hand, such hand, it fits set of contour points of the hand within the hull. It
that all contour points are within the convex hull. This makes uses minimum points to form the hull to include all contour
anenvelope around the hand contour. Figure 6 shows the points inside or on the hull and maintain the property of
convex hull formed around the detected hand. convexity. This causes the formation of defects in the convex
hull with respect to the contour drawn on hand.

10
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

A defect is present wherever the contour of the object is away between two fingers, the system may not detect some fingers
from the convex hull drawn around the same contour. if the user’s hand is far away from the web camera.
Convexity defect gives the set of values for every defect in the
form of vector. This vector contains the start and end point of
the line of defect in the convex hull. These points indicate
3.3 Extraction of Input
After all pre-processing on the input image frame mentioned
indices of the coordinate points of the contour. These points
in hand segmentation and hand detection is done, useful
can be easily retrieved by using start and end indices of the
information is extracted from the user’s hand for input
defect formed from the contour vector. Convexity defect also
purposes. For extracting useful input, three main things that
includes index of the depth point in the contour and its depth
are mentioned in hand detection. Contour and convexity
value from the line. Figure 7 shows an example of convexity
defects in the image frame are mainly used for extraction of
defects calculated in the detected hand using the input image
inputs. For interaction with the computer system, various
of figure 1.
commands can be given

In this paper we propose three techniques for interactions as

follows:

3.3.1 Finger Counting

In this method, the count of fingers from the user’s hand is
extracted. It makes use of convexity defects for detecting the
finger count. As mentioned above, the data structure of
convexity defect gives the depth of the defects. Many such
defects occur in a general hand image due to wrist position
and orientation. But some defects have far greater depth than
others. These are the gaps between the two fingers. As shown
in the figure 8, count given by the user is two. There are many
defects that are formed, but the depth of defect formed due to
the gap between two fingers is much greater and thus can be
separated from other non-important defects. For two adjacent
fingers, there is one such defect.
Figure 7:Major Convexity DefectsCalculated using the
Input Image of figure 1

For finding convexity defects using the contour and its convex
hull we use the following function [6]:

void convexityDefects(
InputArray contour,
InputArrayconvexhull,
OutputArrayconvexityDefects);

 contour – contains the largest interesting contour

points

 convexhull – contains the calculated convex hull of

the contour passed Figure 8: Finger Count of 2

 convexityDefects – acts as the storage for the

output convexity defects. This storage is a vector of
convexity defects. Every convexity defect is represented
by a 4 element integer vector that is Vec4i defined by
OpenCV. These four integer points are start_index,
end_index, farthest_pt_index and fixpt_depth. The
start_index and end_index are the indices of points in the
contour parameter which are the start and end point of a
particular convexity defect. The farthest_pt_index is also
an index of a point in the contour which does not lie on
the convex hull and is the farthest distance from it for a
particular convexity defect. fixpt_depth is the
approximation of the distance between the farthest point
and the hull.

The fixpt_depth is the parameter which we use to remove Figure 9: Finger Count of 3
convexity defects which are not large enough to be considered
as the space between two fingers. Due to this requirement on Similarly in figure 9, count given by the user is three that is
the minimum depth required for consideration as the space having two large defects in the image. So finger count is
easily determined by adding one to the count of large defects

11
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

in the images. This technique fails when there is no large is towards right. And if the difference between midpoint and
defect in the input image frame. This situation occurs when depth point is negative then orientation of hand is towards
there is one finger or no finger in the image. In both the cases left.
there is no large defect, so it cannot be determined whether no
count that is count 0 is given by the user or count one is given
by the user. To overcome this issue we design the commands
from count of 2 to 5. The count of 1 and 0 can also be used as
a command but they both should represent the same
command.

3.3.2 Hand Orientation

In this method of interaction, the orientation of hand is taken
into consideration. Four possible orientations can be
recognized such as up, down, right and left. Same as previous
method the data structure of convexity defects are used to
determine the orientation of hand. In the vector of convexity
defect the defect with the maximum depth is found out. This
gives the index of starting and ending points of line of the
largest defect in the contour. These two point’s coordinate can
be easily extracted from the contour vector using the index Figure 11: Right Orientation
values. Using these two points, the midpoint of the line is
calculated, using 3.3.3 Finger Tracking
In this method the fingertip of the user is used as the input
(𝑠𝑡𝑎𝑟𝑡_𝑝𝑜𝑖𝑛𝑡 + 𝑒𝑛𝑑_𝑝𝑜𝑖𝑛𝑡)
𝑚𝑖𝑑_𝑝𝑜𝑖𝑛𝑡 = pointer. The input pointer is just like a mouse pointer. First
2 the detected hand contour is tested for structural properties
which include the no of convexity defects and the
This midpoint is compared with depth point of the defect, and
approximate depth of these defects in proportion to the hand
orientation of hand is determined.
contour size. Then the top most point of the contour is used as
the mouse pointer. This top most point is the fingertip of the
Calculated midpoint and depth point are compared in two
user. To emulate a click, the user is required to keep the
ways by checking initial conditions. For this the difference
fingertip stable with slight in acceptable error range. The user
between x-coordinate and y-coordinate of midpoint and depth
interface should have large clickable areas so that the number
point is used. If there is small difference in x-coordinate and
of miss-clicks is less. The acceptable error gives a circular
large difference in y-coordinate then the orientation of hand is
area around the detected fingertip in which the fingertip
up or down. For detecting whether it is up or down, difference
should be for a fixed number of frames in order to register it
between y-coordinates is used. As origin of the image is at top
as a click. Sometimes the fingertip is not detected in
left, so if the difference between y-coordinates of the midpoint
consecutive frames due to various reasons so a relaxation is
and depth point is negative then orientation of hand is up. And
provided of 3 frames. So if the fingertip is inside the
if it is positive then orientation is down. As shown in the
acceptable error area for a fixed number of frames with the
figure 10, difference between y-coordinate of the midpoint
relaxation of those 3 consecutive frames may not have the
and depth point is positive, so the orientation of hand is down.
fingertip in the acceptable area.

Figure 12:User’s tracked hand. The green tip acts as the

Figure 10: Down Orientation pointer

Similarly in figure 11, the line of hull with defect having Every time the fingertip goes outside the current acceptable
maximum depth is considered. The difference between x- area, a counter is set to 0. The new location in the current
coordinates of the midpoint and depth point is much larger frame becomes the center of the acceptable area. Now for
than the difference between y-coordinate. So orientation of each frame if the fingertip is detected in the acceptable area
the hand is either right or left. If the difference between then the counter is incremented. If the fingertip is not detected
midpoint and depth point is positive then orientation of hand in the acceptable area then a different variable anti-counter is

12
 International Journal of Computer Applications (0975 – 8887)
Volume 72– No.17, June 2013

incremented. Then if anti-counter reaches the value of 2 then select the object which should be picked up and then also
counter is reset and the current location is made the center of select the destination for this object. This can be done by
the acceptable area. If the fingertip is detected in the showing the fingertip overlay over the images coming from
acceptable area then anti-counter is reset and counter is the robot's camera. When a click is registered inside a
incremented. If the counter reaches a fixed value then the detected object, the object is marked as the target object and
click is registered and the counter reset. Anti-counter variable the arm picks up the object and then again the destination is
is used to implement the relaxation of 2 consecutive frames the point where the next click is registered from the finger
not having fingertip in the acceptable area due to any reason. tracking module. The object is thus placed at the destination.

4. APPLICATIONS 5. FUTURE WORK

The current system does not have a good background
4.1 Gesture Controlled Robot using Image subtraction method which can be used in any situation. Thus
Processing[8] the current system puts lots of constraints on the user for
Service robotics involves directly interacting with people successful working. The future work includes reducing these
through various traditional interaction techniques like Remote constraints so that the system is usable in more scenarios.
Control etc. Now a days there is a need to find a more natural Also the detection can be improved to be more accurate and
and easy to use interaction system through which systems can produce better and precise results.
be made user friendly. The implementation of various Human
Computer Interaction described in this paper can be used for 6. CONCLUSION
controlling robots through gesture for giving commands. Out In the current system we have implemented various
of 3 methods of HCI specified above two of them can be used techniques for efficient human computer interaction. We have
for controlling robots through gesture as follow. also provided different techniques for preprocessing of input
images. These provided techniques can be used accurately and
One method that can be used is Orientation of Hand. As there efficiently applying some constraints.
can be four possible orientations of hand that can be detected
in the provided system, each of these orientations can be used 7. ACKNOWLEDGMENTS
to specify the direction of movement to the robot to move in We would like to thank Mr. H.K. Kaura, HOD of computers
particular direction and navigate in the environment. Another department Fr.C.R.I.T and Prof. Krutika the department
method that can be used is Finger Count. Each Finger Count coordinatorfor the their support and guidance provided during
can be associated with a command for the robot's movement. the work on the implementation of these techniques and while
By changing the finger count, the direction of the robot can be writing this paper.
changed according to it. For all five finger counts commands
like Forward, Backward, Right, Left and Stop can be given to 8. REFERENCES
the robot. [1] Chao Hy Xiang Wang, Mrinal K. Mandal, Max Meng,
and Donglin Li, "Efficient Face and Gesture Recognition
Techniques for Robot Control", CCECE, 1757-1762,
2003.
[2] AsanterabiMalima, ErolOzgur, and Mujdat Cetin, "A
Fast Algorithm for Vision-Based Hand Gesture
Recognition for Robot Control", IEEE International
Conference on Computer Vision, 2006.
[3] O'reilly, Learning OpenCV, Computer Vision in C++
with the OpenCV Library, Adrian Kaebler and Gary
Bradski.
[4] OpenCV usage documentationhttp://docs.opencv.org
[5] Thresholdinghttp://www.cse.unr.edu/~bebis/CS791E/Not
es/Thresholding.pdf
[6] Threshold Function using
OpenCVhttp://docs.opencv.org/modules/imgproc/doc/mi
scellaneous_transformations.html
[7] Contours, Convex Hull and Convexity Defects
Figure 13: Selection of Object using the User’s Hand http://docs.opencv.org/modules/imgproc/doc/structural_a
Pointer for Pick and Place Robot nalysis_and_shape_descriptors.html
[8] Harish Kumar Kaura, VipulHonrao, SayaliPatil,
4.2 Pick and Place Robot [9] PravishShetty, "Gesture Controlled Robot using Image
A pick and place robot is basically a robotic arm that is used Processing", IJARAI 2013, Volume 2 No 5, Page 69-77.
to pick up object and place them at a destination. This
application uses finger tracking for interaction between the [9] AmirajDhawan, AnuradhaBhat, Soumya Sharma, Harish
user and the system. The camera on top of the robotic arm can Kumar Kaura, "Automated Robot with Object
remotely send the work envelope of the robot to the user at a Recognition and Handling Features", IJECSE Volume 2,
remote location. At the remote location the user is supposed to Number 3, 2013, Page 861-873.

13
IJCATM : www.ijcaonline.org