Sign Language Recognition Using Smart Glove

Snehal Savant1 Vijaya Bhusare2 Yogita Lokhane3 Nikita Dhonnar4

1,2,3,4
Student, Department of Computer Engineering, Sandip Foundation’s
Sandip Institute of Technology and Research Centre, Nashik, India

Abstract— Sign Language Recognition by technology


and Object (SVO) [2]. For example, if in normal usage
is an overlooked concept despite there being a large we say “I am not feeling well”, in sign language
social group that could benefit from it. There are 2.78% grammar is neglected and said as “Not feeling well.
of the total population of India who can’t even speak. The advances in the field of Machine Learning. Cloud
Using Gesture-based communication, Individuals Computing and other domain make it possible to
communicate with each other to pass on their thoughts classify easily the hand gesture patterns. Smart glove
to the general population around them. It is a mode of identifies the motions of the hand and fingers on an
correspondence for the rest population who are either individual basis. If outcomes of the system are in
having speaking or hearing incompetency. Ordinary digital signals, are mapped to a particular value for
individuals don't become familiar with gesture-based detecting each character. These characters are then used
communication. It causes a conveyance gap between in the formation of words and phrases. On this glove,
especially impaired people and normal people. The past different sensors are set to identify the positions of the
system of this project used to involve image processing hand. As a result, using this glove speech or hearing
concepts. But these past frameworks were non-portable impaired can convey ideas to others easily. So, the
and excessively costly the main propaganda of this significance of the gloves is to create the life kind of
project is to interpret Indian Sign Language Standards speech or hearing-impaired folks straightforward.
and American Sign Language Standards and Convert
gestures into voice and text, also assist the person to 2. LITERATURE SURVEY
communicate with other people from a remote location.
According to the analysis done that, one out of every
This glove has been actualized with the assistance of
five people is having speech or hearing incompetency
Flex sensors, Cloud, Gyroscope Sensor, ESP32
on this planet is an Indian. In India about more than
Microcontrollers/Micro bit, Accelerometer,25 LED
1.5 million with hearing-impaired utilize Sign
Matrix Actuators/Output &, etc. Language as a method of correspondence. Normal
parents of special incompetency or vice versa use a
Keywords: Sign Language Recognition, Flex sensor,
gesture-based conversation with others. However, due
Indian Sign Language, American Sign Language, to this type of complication an automatic Sign-to-
ESP32 bit Micro-Controller Embedded System, Speech/text language interpretation framework could
Networking, Cloud Computing, and Machine assist to make more details accessible to the hearing
Learning/Artificial Intelligence impaired. Glove Talk II translates hand gestures to
speech, which is based on the gesture format model
1. INTRODUCTION developed by Sidney Fels and Geoffrey Hinton,
Communication is the activity of conveying Department of Computer Science of the University of
information. Over the years, communication played a Toronto. Neural networks were used to implement an
vital role in the exchange of information and feelings. adaptive interface, called Glove Talk II, which
Indian Sign Language is standardized and commonly contains hand gestures to control the parameters of a
used all over the nation. It is communicated using hand parallel formant speech synthesizer to allow a user to
gestures made by Single hands and Double hands [5]. speak with his hands. It is used to implement an
In this, grammar usage is not considered and only main artificial vocal tract. Glove Talk II is a system that
words are taken into consideration. Articles such as ‘a’, translates hand gestures to speech through an adaptive
‘an’, ‘the’ are omitted and also don’t include tense interface. Hand gestures are mapped continuously to
form. The sentence structure of Sign Language is- 10 control parameters of a parallel format speech
Subject, Object, Verb (SOV), which is very much synthesizer. The mapping allows the hand to act as an
different from the sentence structure of English artificial vocal tract that produces speech in real-time.
language whose sentence structure is – Subject, Verb, This gives an unlimited vocabulary, multiple
languages in addition to direct control of fundamental

frequency and volume. Currently, the best version of human-understandable form such as textual messages.
Glove Talk II uses several input devices (including a The hardware section of their proposed design has its
Cyber glove, a contact glove, a Polhemus sensor, and constituent electronics components as a bend sensor,
a foot pedal), a parallel formant speech synthesizer, hall-effect sensor, accelerometer, and machine
and three neural networks. learning algorithms used for gesture recognition. The
bend sensor output is fed to the analog multiplexer
The gesture to speech task is divided into vowel and a (HEF4051 by NXP semiconductors). The output of
consonant neural network. The gating network and the this multiplexer is given to a current to voltage
consonant network are trained with examples from the converter circuit. Since the voltage output of the Hall
user. The vowel network implements a fixed, user- Sensor is low, an amplifier is needed. Sensor outputs
defined relationship between hand position and vowel obtained are given to the inbuilt ADC for sampling
sound and does not require any training examples the values given by sensors, which is also used for
from the user. Volume, fundamental frequency and interfacing the glove with a computer running the
stop consonants are produced with a fixed mapping machine learning algorithms. The data acquisition
from the input devices. Harmeet Kaur, et al. in their process starts with the processor sending control
paper, presented a brief description of the past signals to a multiplexer for receiving values from the
attempts that were made to convert sign language to different sensors sequentially and temporarily storing
understandable form. In their paper, they have them in an array. These stored values are transmitted
thoroughly scrutinized the previous attempts over this to the computer using universal asynchronous
technology and also suggested various possible ways reception and transmission (UART) connection for
to implement the design of a simple smart glove. further processing and decoding of the received
Speak Jet is a sound synthesizer that is used to convert signals. While transmitting the values in UART
text data into voice. It uses Mathematical Sound automatic repeat request scheme has been
Architecture (MSA) technique to control five-channel incorporated for avoiding information loss because of
sound synthesizers to generate speech signals. It has transmission errors.
72 speech elements, 43 sound effects, and 12 (Dual
Tone Multi-Frequency) DTMF touch tones, pitch rate, 3. PROPOSED ARCHITECTURE
bend, and volume parameters which can generate
various sound effects. They tried to develop an
Electronic speaking glove, designed to facilitate easy
communication through synthesized speech for the
benefit of speechless patients. Gestures of fingers of a
user of this glove will be converted to synthesized
speech to convey an audible message to others, for
example in critical communication with doctors.

The glove is internally equipped with multiple flex

sensors that are made up of “bend-sensitive resistance
elements”, For each specific gesture, internal flex Figure 1 System Architecture
sensors produce a proportional change in resistance of
various elements. The processing of this information The work of this project starts from the movement of
sends a unique set of signals to the PIC hand gloves where the flex sensors are attached, and
microcontroller and speak jet IC which is pre- the value of the sensor changes when it experiences
programmed to speak desired sentences. Tushar bending. The changing value of the sensor is
Chouhan et al. implemented a wired interactive glove, depending upon the resistance and applied angle of
interfaced with a computer running MATLAB or the bending when we bend the sensor at some angle.
octave, with a high degree of accuracy for gesture we can see the value of the resistance is increased and
recognition. The glove maps the orientation of the accordingly the output gets reduced. We are also
hand and fingers with the help of bend sensors, Hall using a gyroscope to get the exact position of the hand
Effect sensors, and an accelerometer. The data is then & also implementing a fuzzy logic-based algorithm to
transmitted to a computer using automatic repeat predict the word from sensors value. Arduino Nano is
requests as an error-controlling scheme. The system is employed to assemble signals from the flex sensors
modeled for the differently-abled section of the and accelerometer placed on the glove. Then the
society to help convert sign language to a more processed output is sent over to display the text output
associated conjointly via a Bluetooth link to a
Computer consisting of text to speech package and
speech output is obtained. After looking at the c) Gyroscope: A gyroscope is a device used for
changing value of the output, the value gets recorded measuring or maintaining orientation and
by the ESP32 controller/Micro-bit and Send on AWS angular velocity. It is a spinning wheel or disc
cloud. AWS sends these data to the application in in which the axis of rotation (spin axis) is free
real-time using AWS IoT Core Service with the help to assume any orientation by itself.
of MQTT Protocol.
d) Bluetooth module: The Bluetooth module
The Smart Glove is integrated with the following sends serial data to the target device and the
Components: target device displays that data on the screen.
The data sent to the Bluetooth module is the
a) Flex Sensor: Flex sensor measures the amount alphabets, numerical values, etc.
of bending and flexing done by the user to e) AWS Cloud: The AWS is a comprehensive,
communicate with the use of sign language. evolving cloud computing platform provided
This sensor works on the bending strip by that includes a mixture of infrastructure as a
principle. This flex sensor is connected with a service (IaaS), platform as a service (PaaS),
10k resistor. Here the angular displacement is and packaged software as a service (SaaS)
measured using the voltage from the resistor. offering. It allows users to take benefit from all
Every flex sensor is connected with 10k of these technologies, without the need for
resistors. deep knowledge about or expertise with each
one of them. The aim to cut costs and helps the
users mainly focus on their business instead of
being impeded by IT obstacles.

4. CONCLUSION

“Sign Language Recognition using Smart Glove” is

one of the useful ways to ease the communication
between the hearing incompetent and speech
incompetent people, normal society. The gloves area
b) ESP32 Micro-Controller/Micro-bit: ESP32 unit is moveable with minimum weight and low
is a series of the low-cost, low-power system power. The system converts the hand gestures to the
on a chip microcontroller with integrated Wi-Fi text. With this speech or hearing, incompetent
and dual-mode Bluetooth. The micro bit can be communities will use the gloves to make gestures in
programmed to do several different things, it keeping with signs. Thus, this system plays an
can be a digital watch, fitness tracker, or a important role for especially impaired people. In Fig
game console &, etc. The device features 25 no 1: Block diagram of the Sign Language
LED lights and two programmable buttons, Recognition using Smart Glove the near future this
which can be used. glove is extended by implementing with inbuilt
sensors like temperature sensor is used to measure the
body temperature, the pressure sensor is used to view
the heartbeat of body.

REFERENCES
[1] Cao Dong, Ming C. Leu, Zhao Zheng Yin,
“American sign language Alphabet
Recognition Using Microsoft Kinect”,
Computer Vision and Pattern Recognition
 Workshop, IEEE conference, pp: 2015 JSPM’s Rajashri Shahu College of
http://lemelson.mit.edu/winners/thomaspr Engineering, Pune, India.
yor-and-navid-azodi. [10] “Sign Language to Speech Conversion
http://www.ijste.org/articles/IJSTEV2I908 Using Arduino”, International Journal of
9 Recent Innovation in Engineering and
[2] Nguyen Dang Binh, Toshiaki Ejima “A Research Scientific Journal Impact Factor
new Approach Dedicated to Hand Gesture - 3.605 by Yash Jhunjhunwala1, Pooja
Recognition” (Intelligence Media Shah2, Pradnya Patil3, and Jyoti Waykule
Laboratory, Kyushu Institute of Sou. Sushila Danchand Ghodawat
Technology), Japan 2006IEEE int. Conf. Charitable Trust’sSanjay Ghodawat
on cognitive informatics (icci‟06) Y.Y. Group of Institutions.
Yao, Z.Z. Shi, Y. Wang, and W. kinsner [11] Abhinandan Das, Lavish Yadav,
(Eds.). Mayank Singhal, Raman Sachan, Hemang
[3] “Conversation of Sign Language to Goyal, Keshav Taparia RaghavGulati,
Speech with Human Gestures'', Ankit Singh, Gaurav Trivedi “Smart
ISBCC’2015 by Rajaganapathy. S, Glove for Sign Language
Aravind. B, Keerthana. B, Sivagami.M. Communications” Indian Institute of
[4] Wen Gao and Gaolin Fanga,” A Chinese Technology Guwahati, Assam 781039,
sign language recognition system based India. Published on 978-1-5090-4291-
on SOFM/SRN/HMM. Journal of Pattern 3/16/$31.00 ©2016IEEE.
Recognition”.2389-2402,2004. [12] Paul D. Rosero-Montalvo1,2, Pamela
[5] Nicholas Born., “Senior Project Sign GodoyTrujillo1, Edison Flores-
Language Glove”, electrical engineering Bosmediano1, Jorge Carrascal-Garc´ıa3,
department. California Polytechnic State Santiago Otero-Potosi3, Henry Benitez-
University,1- 49,2010. Pereira3 and Diego H. PeluffoOrd´
[6] Kirsten Ellis and Jan Carlo Barca.” o˜nez4“Sign Language Recognition Based
Exploring Sensor Gloves for Teaching on Intelligent Glove Using Machine
Children Sign Language. Advances in Learning Techniques” published on 978-
Human-Computer Interaction”.1-8.,2012 1-5386- 6659/18/$31.00 ©2018 IEEE.
[7] Solanki Kumar, “Indian Sign Language [13] Professor Alexander Emanuel advisor
using Flex sensor Glove” International “Prototyping a Portable, Affordable Sign
Journal of Engineering Trends and Language Glove
Technology (IJETT)vol.4, n0.6 June 2013. http://www.wpi.edu/acdamics/project.
[8] “Translating Indian Sign Language to text [14] Neha Rajput1, Mrs. Neetu Sikarwar
and voice messages using flex sensors”, “A REVIEW ON IOT BASED SIGN
International Journal of Advanced LANGUAGE CONVERSION” © 2018,
Research in Computer and IRJET ISO 9001:2008Certified Journal.
Communication Engineering Vol. 4, Issue [15] Golda Jeyasheeli P and Annapoorani
5, May 2015 by Sachin Bhat, Amruthesh’s K” IoT Based Sign Language
M, Ashik, Chidanand Das, Sujith. Interpretation System” IOP Publishing
[9] “Hand Gesture Recognition to Speech doi:10.1088/1742- 6596/1362/1/01203
Conversion in Regional Language”,
International Journal of Computer Science
and Network, Volume 4, Issue 1, February
2015 by B.D. Jadhav, 2 Nipun Munot, 3
Madhura Hambarde, 4 Juli Ashtikar