,QWHUQDWLRQDO&RQIHUHQFHRQ,QQRYDWLYH7UHQGVLQ&RPPXQLFDWLRQDQG&RPSXWHU(QJLQHHULQJ ,7&(¶ $VZDQ(J\SW)HE

Smart glove-based gestures recognition system for

Arabic sign language

Neven Saleh*, Mostafa Farghaly, Eslam Elshaaer Amr Mousa

Biomedical and systems engineering department Biomedical engineer
Higher Institute of Engineering in El-Shorouk city Alkan Company
Cairo, Egypt Cairo, Egypt
*Corresponding author e-mail: nevensaleh76@gmail.com

Abstract—Sign language is a communication way that Various works were introduced under the umbrella of
enables deaf and dump people to communicate with others. sign language translation. In the work [3], the authors
Different communication systems are developed to cover presented gestures translator system composed of 4 modules.
various languages including Arabic language. The goal of this The modules were sensing unit, processing unit, voice
study was to design a low cost smart glove system that storage unit, and wireless communication unit. Through a set
recognizes hands gestures in Arabic sign language. Through a of flex sensors, the gestures were processed by PIC16F877A.
set of flex sensors and a tilting sensing module for right and left Based on a vision-based technique, continuous Arabic Sign
hands, gestures were converted into a vocalized speech. Language (ArSL) recognition was introduced by Assaleh et
Further, by developing Android application “Smart Gloves” a
al [1]. The system was developed using spatio-temporal
text speech was appeared. It facilitates user interface with the
feature extraction and Hidden Markov Models (HMM). The
proposed system. Indeed, Bluetooth module was used for
wireless communication between the system and a mobile average recognition rate of words was 94%. Another
phone. The glove system has been designed for words level and approach was adopted in [5] to recognize continuous ArSL
sentences level as well. Results point to 90% of recognition using two DG5-V hand data gloves. A digital camera was
rate. The system has proven its ability to translate hands utilized to synchronize hand movement with words. Further,
gestures into Arabic Sign Language in spoken and written words classification has been carried out by a Modified K-
ways. Nearest Neighbor (MKNN) approach. This approach
achieved sentences recognition rate of 98.9%. In a relevant
Keywords—Arabic sign language; glove; Hands Gestures; study, isolated Arabic words were translated into sign
Communication; Bluetooth. language [7]. The vision-based technique was utilized by two
modes. The first one has been used to translate from deaf
I. INTRODUCTION and dump people to the vocal people and the second one was
vice versa.
One of the most significant challenges that face deaf and
dump people is to communicate with others. Different The aim of this work was to develop a low-cost smart
techniques are developed to facilitate communication with glove system to translate hands gestures into a vocalized
those people using sign language. In general, the utilized speech. Moreover, a smart phone android application was
techniques are classified to be either vision-based or glove- developed to translate gesture as a text speech. It facilitates
based [1, 2]. Through vision-based methodology, gestures user interface with the glove system. Thus, a low cost
are captured by a camera and then images features are solution is presented for deaf and dumb individuals to enable
extracted to interpret gestures meaning [3, 4]. In glove-based them for communication with normal people in a consistent
technique, the user wears a glove and through a set of way.
sensors, gestures are translated into audio or text speech [5,
The article is formed as follows. Materials and methods
6].
are described and discussed in section II. Testing the system
Although image-based techniques are more realistic, they and results are presented in section III. Finally, section IV
require complex processes such as features extraction and concludes this work and the future work is covered.
images processing. Moreover, the background and the
position of the camera are crucial for this technique [2]. II. MATERIALS AND METHODS
Unlikely, glove-based technique is characterized by a less
computational complexity, fast response, and portability [2]. The system is designed to translate hands gestures into
In general, both techniques are utilized to interpret the sign ArSL. A glove-based technique was chosen for this purpose.
language in forms of isolated words or continuous words [1, In implementation, a hardware module and a software
7, 8]. module were integrated to form the proposed system. The
hardware module consists of a pair of tailored gloves,
microcontroller (Arduino MEGA), Bluetooth module, MP3

978-1-7281-4801-4/20/$31.00 ©2020 IEEE.

303
shield, and a loudspeaker. The left hand is used as a slave a) Flex sensors: Flex sensor as shown in Fig. 3 is a
(transmitter), whereas the right hand is used as a master variable resistance that is used for gesture recognition. It
(receiver). On the other hand, the software module was tranforms bending of the finger into analog signal based on
developed to encode and decode the recognized signs. bending degree [4]. It maps the degree of bending of 0ම, 45ම,
and 90ම to variable resistance varies from 10 ȍ to 30 K ȍ
A. Hardware Module [2]. Output voltage is calculted using (1); where Rf is flex
The main part of hardware module is the glove. sensor resistance and Rt is the total resistance. The values of
Literature shows that three types of ready-made gloves are flex sensors outputs are converted into ADC levels. Initial
utilized; cyber glove, DG-5V glove, and 5DT glove [9]. Due values of ADC levels at the three positions are presented in
to high costs of these kinds, we decided to use a hand-made Table I. Therefore, the initial values (thresholds) of ADC
glove to produce a low cost system. An overview of the levels were saved on Arduino board to be comparable with
system is depicted in Fig.1. The proposed system starts with other generated levels in order to recognize the gestures.
the left glove (slave). Once the glove fingers are bending, the
flex sensors give their outputs to the microcontroller ADC
channels. The processed ADC values are then compared with
thresholds values to recognize the gesture. The
microcontroller transmits the signals through UART wireless
module to the right glove (master) in a serial way.
Therefore, it gives its commands to the MP3 shield to come
out the vocalized gestures.

Fig.3. Flex sensors circuit of one glove

Vo = Vi ( ) (1)

Fig.1. Block diagram of the proposed system TABLE I. STANDARD MEASUREMNTS OF FINGERS BENDING AT 3 DIFFERENT
ANGLES

Finger F1 F2 F3 F4 F5
1) The glove : Tailored pair of gloves were designed for 0 degree 295 317 390 360 204
right and left hands using 5 flex sensors and a tilting sensing
45 degree 210 235 275 210 160
module each. They were embedded togther on a nylon
material-based glove as shown in Fig.2. Each sensor was 90 degree 155 165 200 145 110
fixed on thumb, index, middle, ring, and pinky fingers
respectively. In addition a gyroscope was mounted on the b) Gyroscope: Arabic words have their own sign
glove to indicate orientation of the hand palm. language. In fact, some words have similar finger bending
but they are different in hand motion. Thus, these words
require differention in fingers bending orientation and hand
palm motion. Therefore, a trible-axis micro electro
mechanical system (MEMS) module MPU-6050 is used. It
contains a gyroscope to distinguish hand orientation in X ,Y,
and Z directions and accelerometer to detect hand palm
motion. It provides a wide range of features including 16-bit
ADCs.
2) The microcontroller : Arduino Mega 2560 has been
selected to be a microcontroller. It features with a low
power consumption as well as low cost[10]. It has 16
analog input pins, 54 digital input/output pins, 4 UARTs,
and USB connection.
3) HC-05 Bluetooth: A wireless communication was
carried out by using HC-05 Bluetooth module. It allows
Fig.2. Designed glove model
sending data from the microcontroller to a mobile phone or a

304
computer. In our case, the data was sent from Arduino board shown in Fig. 5, while the simulation design of one glove
to a mobile phone asynchronously. (the master) is illustrated in Fig. 6.
4) Arduino MP3 shield: Arduino MP3 shield is an
onboard 3w/channel stereo audio amplifier [11]. It is
employed to derive a loudspeaker to come out with the sound
of translated gestures. In this application, a 3W/4ȍ
loudspeaker was used.
B. Software Module
System development as shown in Fig.4 starts with hands
gestures that are converted by flex sensors to analog signals.
The analog signals are converted to digital signals and
processed by the microcontroller. Each word/sentence has
its gesture; thereby each one has its own serial measurements
[12]. By recording these measurements and comparing them
with coded template of serial values, words are recognized
and vocalized; otherwise the sequence is repeated. The code
is written by Arduino C language as it supported by Arduino. Fig.5. Overall proposed system containing the gloves, the
Real challenge of this study was writing the code to read microcontroller, and the power supply units
Arabic words. In general, Arduino software is written only
for English; therefore a new code was developed to
recognize and vocalize Arabic words.

Fig.6. System design simulation for the master glove

In implementation, the system was tested for words and

expressions. Finger bending and hand palm orientation are
detected by the flex sensors and the gyroscope respectively.
Examples of some words and expressions in terms of
measured ADE levels are illustrated in Fig. 7 and Fig.8
respectively. Fig. 7 displays the word “ήμϣ” or “Egypt”, and
Fig.8 presents the expression “ ϢϜϋΎϤΘγ· ϦδΤϟ ΍ήϜη” or “Thank
you for your attention”. The X-axis presents ten fingers and
the directions in X, Y, and Z for both hands, while the Y-axis
presents the ADC levels. Taking into account "m" stands for
the master glove and "s" stands for the slave one.
The system has been tested using 2 users with different
physical characteristics. A total of 30 words and 15
expressions have been programed to be tested as shown in
Table II. In fact, by making several trials for both users, the
Fig.4. Flowchart of gestures recognition technique
average recognition rate was 90 %. In other words, for every
10 hands gestures recognition trails, one trail has failed. This
is due to adaption of system's user and how the board part is
III. RESULTS well tightened with his/her arm. Therefore, the rate of
accuracy can be improved considering the physical
The proposed system was designed to interpret hands characteristics of the user. One advantage of this system is no
gestures in forms of words and expressions. Arabic language prior training is required for the user.
is as other languages, some expressions are expressed by one
hand, while others by the two hands. In implementation, the
user is asked to wear the two gloves, while the
microcontroller board and the power supply unit are put in a
pod and hanged on the user's arm. The overall system is

305
 TABLE II. SAMPLES OF TRANSLATED GESTURES IN FORMS OF WORDS AND
EXPRESSIONS

No Word No Expression
1 ϲΑ΃ (My father) 1 ϢϜϋΎϤΘγ΍ ϦδΤϟ ΍ήϜη (Thanks for
your attention)
2 ϰϣ΃ (My mother) 2 ϯΩήϔϤΑ ΔγέΪϤϟ΍ ϰϟ· ΐϫΫ΃ (Go to the
school alone)
3 ήμϣ (Egypt) 3 ϰΑ΃ ΐΣ΃ Ύϧ΃ (I love my father)
4 ΍ήϜη ( Thank you) 4 ϰϣ΃ ΐΣ΃ Ύϧ΃ (I love my mother)
5 ΔγέΪϣ (School) 5 ˯ΎϤδϟ΍ ϰϓ ΔόρΎγ βϤθϟ΍ (The sun is
shining in the sky)
6 ϥΎϋϮΟ (Hungry) 6 ϡΎότϟ΍ Ϊϳέ΃ϭ ϥΎϋϮΟ Ύϧ΃ (I am hungry
and I need food)
7 ϥΎθτϋ (Thirsty) 7 ΏΎΘϜϟ΍ ΃ήϗ΃ (I read the book)
8 ϒ΋ΎΧ (Afraid) 8 ϡΪϘϟ΍ Γήϛ ΐόϟ΍ (I play football)
9 ϞϤόϟ΍ (Work) 9 ϰϨϤϴϟ΍ ϯΪϴΑ ΐΘϛ΃ (I write with my
right hand)
10 ΐϫΫ΍ (Go) 10 ΐόΘϟΎΑ ήόη΃ (I am tired)
11 ϢγϹ΍ (Name) 11 ϡϼψϟ΍ Ϧϣ ϒ΋ΎΧ Ύϧ΃ ( I afraid of the
dark)
12 ΐΣ΍ (Love) 12 ϞϤόϟ΍ ϥΎϘΗ· ΐΠϳ (You must master
the work)
Fig.7 A signal of vocalized word “Egypt” in Arabic language 13 ϡΎότϟ΍ (Food) 13 ωέΎθϟ΍ ήΒϋ΍ (Cross the street)
14 ˯ΎϤδϟ΍ (Sky) 14 ϢϠϘϟ΍ Ϊϳέ΃ (I need the pen)
15 βϤθϟ΍ (Sun) 15 Δγ΍έΪϟ΍ ϲϓ ϰϧΪϋΎδϳ ϰΧ΃ (My brother
helps me to study)
16 ωέΎθϟ΍ (Street)
17 ΐΘϛ΃ (Write)
18 ΍ήϗ΃ (Read)
19 Ϊϴϟ΍ (Hand)
20 ϢϠϘϟ΍ (Pen)

Android application was developed to receive the

recognized gestures via Bluetooth. The application is called
“Smart Gloves”. It receives Arabic isolated words separately
as well as the expressions. It stores the words in a matrix and
by a space counter, the expression is recognized. The
recognized gestures are converted into a vocalized speech as
shown in Fig. 9. Moreover, if some words have similar
vocalization it will be corrected automatically by auto-
correction Android option.
It is worthy to mention that the real cost of the glove-
based system is very cheap comparing with commercial
ones. For instance, the price of DG-5V glove is 3000$ [9].
The cost of this smart glove is approximately 120 $. It can be
calculated as follows; one glove is (10$), one flex sensor is
(5$), one MPU 6050 is (5$), Arduino mega 2560 is (25$),
Bluetooth is (11$), and MP3 shield is (4$). Therefore, the
proposed system is characterized by its low cost and ease of
utilization.
Fig.8. A signal of vocalized expression “Thank you for your attention”
in Arabic language

306
 ACKNOWLEDGMENT
Sincere thanks and appreciation go to Eng. Rabie Gouda,
Eng. Abdullah Taha, and Eng. Ahmed Bahaa for their
assistance in writing the code and testing of the system.

REFERENCES
[1] K. Assaleh, T. Shanableh, M. Fanaswala, F. Amin, and H. Bajaj,
“Continuous Arabic sign language recognition in user dependent
mode,” Journal of Intelligent Learning Systems and Applications, vol.
2, pp. 19-27, 2010.
[2] P. Lokhande, R. Prajapati, and S. Pansare, “Data gloves for sign
language recognition system,” in Proc. of National Conference on
Emerging Trends in Advanced Communication Technologies, June
2015, pp. 1-4.
[3] K. Gunasekaran and R. Manikandan, “Sign language to speech
translation system using PIC microcontroller,” International Journal
of Engineering and Technology. vol. 5, no. 4, pp. 1024-1028, 2013.
[4] K.V. Fale, A. Phalke, P. Chaudhari, and P. Jadhav, “Smart glove:
gesture vocalizer for deaf and dump people,” International Journal of
Innovative Research in Computer and Communication Engineering,
vol.4 no.4, pp. 6800-6806, 2016.
[5] N. Tubaiz, T. Shanableh, and K. Assaleh, “Glove-based continuous
Arabic sign language recognition in user dependent mode,” IEEE
Transactions on Human Machine Ssystems, vol. 45, no. 4, pp. 526-
533, 2015.
[6] D. J. Sturman and D. Zeltzer, “A survey of glove-based input,” IEEE
Fig.9. Screenshot of Smart Gloves application
Computer Graphics and Applications, vol. 14, no. 1, pp. 30-39, 1994.
[7] N.B. Ibrahim, M.M. Selim, and H.H. Zayed, “An automatic Arabic
sign language recognition system,” Journal of King Saud Uninversity-
Computer and Information Sciences, vol. 30, no. 4, pp.470-477, 2018.
IV. CONCLUSION [8] Suharjito, R. Anderson, F. Wiryana, M. Chandra, and G. P. Kusuma,
A smart glove-based system is designed to reduce the “Sign language recognition application systems for deaf-mute people:
a review based on input-process-output,” in Proc. of 2nd International
communication gap between the deaf/dump people and the Conference on Computer Science and Computational Intelligence,
normal people. It translates hands gestures into ArSL in the Bali, Indonesia, October 2017, pp. 441-448.
forms of words and expressions as well. The outcome is a [9] M. I. Sadek, M. N. Michael, and H.A. Mansour, “Glove-based
vocalized speech. Moreover, Android application “Smart continuous Arabic sign language recognition in user dependent
Gloves” is developed to translate gestures into a text speech. mode,” in Proc. of 34th National Radio Science Conference, Port Said,
In fact, the system is characterized by ease of interface, Egypt, March 2017, pp. 380-388.
portability, no prior training, and low costs. [10] V. Oza and P. Mehta, “Arduino robotic hand: survey paper,” in Proc.
of International Conference on Smart City and Emerging Technology,
The accuracy of the system is relatively high. It depends Mumbai, India, January 2018, pp. 1-5.
on the physical characteristics of the user. Once, the system [11] A. Nayyar and V. Puri, “A review of Arduino board's, Lilypad's &
is well tightened on the user, it acts in a good way. The Arduino shields,” in Proc. of 3rd International Conference on
Computing for Sustainable Global Development, New Delhi, India,
system can be generalized for other sign languages such as March 2016, pp. 1485-1492.
American Sign Language. In addition, the system can be
[12] A. Sengupta, T. Mallick, and A. Das, “A cost effective designand
programmed to translate more gestures for both words and implementation of Arduino based sign language interpreter ,” in Proc.
expressions levels. One limitation of this system is to adapt of Devcies for Integrated Circuits, Kalyani, India, March 2019, pp.
for more than one user in the same time. It requires 12-15.
modifying size of the board pod. In general, the proposed
system presents an efficient way for communication with
deaf/dump people by interpreting their hands gestures in
spoken and written forms.

307