Automatic Language Translation Software For Aiding Communication Between Indian Sign Language And Spoken

English Using Labview Yellapu Madhuri*, G.Anitha**

* 2nd year M.Tech, ** Assistant Professor

Department of Biomedical Engineering,

SRM University, Kattankulathur-603203, Tamilnadu, India

www.srmuniv.ac.in

Sign Language (SL) is the natural way of communication of speech and/or

hearing-impaired people. A sign is a movement of one or both hands,

accompanied with facial expression, which corresponds to a specific meaning.

This paper presents SIGN LANGUAGE TRANSLATION software for

automatic translation of Indian sign language into spoken English and vice

versa to assist the communication between speech and/or hearing impaired

people and hearing people. It could be used by deaf community as a translator

to people that do not understand sign language, avoiding by this way the

intervention of an intermediate person for interpretation and allow

communication using their natural way of speaking. The proposed software is

standalone executable interactive application program developed using

LABVIEW software that can be implemented in any standard windows

operating laptop, desktop or an IOS mobile phone to operate with the camera,

processor and audio device. For sign to speech translation, the one handed SL

gestures of the user are captured using camera; vision analysis functions are

performed in the operating system and provide corresponding speech output

through audio device. For speech to SL translation the speech input of the user

is acquired by microphone; speech analysis functions are performed and

provide SL gesture picture display of corresponding speech input. The

experienced lag time for translation is little because of parallel processing and

allows for instantaneous translation from finger and hand movements to speech

and speech inputs to SL gestures. This system is trained to translate one handed

SL representations of alphabets (A-Z), numbers (1-9) to speech and 165 word

phrases to SL gestures The training database of inputs can be easily extended to

expand the system applications. The software does not require the user to use

any special hand gloves. The results are found to be highly consistent,

reproducible, with fairly high precision and accuracy.

AIM :

To develop a mobile interactive application program for automatic

translation of Indian sign language into spoken English and vice-versa to assist

the communication between Deaf people and hearing people. The SL translator

should be able to translate one handed Indian Sign language finger spelling input

of alphabets (A-Z) and numbers (1-9) to spoken English audio output and 165

spoken English word input to Indian Sign language picture display output.

OBJECTIVES:

•To acquire one handed SL finger spelling of alphabets (A to Z) and numbers (1

to 9) to produce spoken English audio output.

•To acquire spoken English word input to produce Indian Sign language picture

display output.

•To create an executable file to make the software a standalone application.

•To implement the software and optimize the parameters to improve the accuracy

of translation.

•To minimize hardware requirements and thus expense while achieving high

precision of translation.

There is a need for monitoring cerebral perfusion

MATERIALS

Software Tools used: National Instruments

LabVIEW and toolkits

•LABVIEW 2012 version

•Vision Development Module

•Vision acquisition module

Hardware tools used

•Laptop inbuilt webcamera- Acer Crystal Eye

•Laptop inbuilt speaker-Acer eAudio

METHOD:

The software is a standalone application. To install the file, follow the

instructions that appear in the executable installer file. After installing the

application, a Graphical user interfacing (GUI) window opens, from which the

full application can be used. The GUI has been created to run the entire

application from a single window. It has four pages, each page corresponds to a

specific application.

PAGE 1 gives a detailed demo of the total software usage.

PAGE 2 is for speech to sign language translation.

When the “start” button is pressed, a command is sent to the Windows 7 inbuilt

Speech Recognizer and it opens a mini window at the top. The first time it is

started, a tutorial session begins which gives instructions to setup the microphone

and recognize the user’s voice input. Configure the speech recognition software.

After the initial training, from the next time the program is executed, it starts

speech recognition automatically. To train the system for a different user or

change the microphone settings, right click on the Speech Recognizer window

and select “Start Speech Tutorial”. To stop the speech recognition software say

“Stop listening”. To start speech recognition again say “Start Listening”. When

the user utters any of the words listed in the “Phrases” it is displayed in the

“Command” indicator. A SL gesture picture corresponding to the speech input is

displayed in the “Sign” picture indicator. The score of speech input correlation

with the trained word is displayed in the “Score” numeric indicator. Use the exit

button to exit the application of speech to SL translation.

PAGE 3 is for template preparation for sign to speech translation.

To execute the template preparation module, press the “Start” button.

Choose the camera to acquire images to be used as templates, from the “Camera

Name” list. The acquired image is displayed on “Image” picture indicator. If the

display image is good to be used for preparing a template, press “Snap frame”.

The snapped image is displayed on “Snap Image” picture display. Draw a region

of interest to prepare the template and press “Learn”. The image region in the

selected portion of the snapped frame is saved to the folder specified for

templates. The saved template image is displayed on “Template Image” picture

display. Press “Stop” button to stop execution of template preparation module.

PAGE 4 is for Sign to speech translation.

Press the “Start” button to start the program. Choose the camera to acquire

images to be used for pattern matching, from the “Camera Name” list. The

captured images are displayed on the “Input Image” picture display. Press the

“Match” button to start comparing the acquired input image with the template

images in the data base. In each iteration the input image is checked for pattern

match with one template. When the input image matches with the template image,

the loop halts. The “Match” LED glows and the matched template is displayed on

the “Template Image” indicator. The loop iteration count is used for triggering a

case structure. Depending on the iteration count value a specific case is selected

and gives a string output. Otherwise the loop continues to next iteration where the

input image is checked for pattern match with a new template. The information

in the string output from case structure is displayed on the “Matched Pattern”

alphanumeric indicator. It also initiates the .NET speech synthesizer to give an

audio output through the speaker.

Figure 1.1 Events involved in hearing Figure 1.2 Speech chain

Figure 1.3 Graphical Abstract

[1]. Yellapu Madhuri, G.Anitha (2013) “VISION-BASED SIGN

LANGUAGE TRANSLATION DEVICE” International Conference on

Information Communication & Embedded systems ICICES 2013 in association

with IEEE, S.A engineering College, Chennai. ISBN No. 978-1-4673-5787-6G.

Tracking Id: 13cse213.

[2]. Yellapu Madhuri, G.Anitha (2013) “Automatic Language Translation

Software for Interpreting Sign Language and Speech in English”, has been

awarded Silver medal in paper presentation Research Day 2013 at SRM

University, Chennai.

[3]. Yellapu Madhuri, G.Anitha (2013) submission entitled "SIGN

LANGUAGE TRANSLATOR" has been assigned the following manuscript.

number: IMAVIS-D-13-00011 by Elsevier Editorial Systems- Image and

Vision Computing journal imavis@elsevier.com.

[4]. Yellapu Madhuri, G.Anitha (2013) submission entitled “VISION-BASED

SIGN LANGUAGE TRANSLATOR” is Accepted for publication

in International Journal of Engineering and Science Invention (IJESI)

www.ijesi.org.Review report of manuscript id: A11023.

[5]. Yellapu Madhuri, G.Anitha (2013) submission entitled “SIGN

LANGUAGE TRANSLATION DEVICE” is Accepted for publication

in The International Journal of Engineering and Science (THE IJES)

www.theijes.com. Review report of manuscript id: 13026.

[6]. Yellapu Madhuri, G.Anitha (2013) submission entitled "Automatic

Language Translation Software for Interpreting Sign Language and Speech in

English" has been assigned a tracking number of NCOMMS-13-02048 by

Nature Communications naturecommunications@nature.com.

[1]. Jose L. Hernandez-Rebollar1, Nicholas Kyriakopoulos1, Robert W. Lindeman2 ‘A New Instrumented Approach For Translating American Sign Language Into Sound And Text’, Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition (FGR’04) 0-7695-2122-3/04 $ 20.00 © 2004 IEEE. [2]. K. Abe, H. Saito, S. Ozawa: Virtual 3D Interface System via Hand Motion Recognition From Two Cameras. IEEE Trans. Systems, Man, and Cybernetics, Vol. 32, No. 4, pp. 536–540, July 2002. [3]. Paschaloudi N. Vassilia, Margaritis G. Konstantinos "Listening to deaf': A Greek sign language translator’, 0-7803-9521-2/06/$20.00 §2006IEEE

Name: YELLAPU MADHURI

Reg.No:1651110002

M.Tech (BiomedialEngineering)

Mobile no: 09441571241

E.Mail:ymadhury@rediffmail.com

In this work, a vision based sign language recognition system using LABVIEW for

automatic sign language translation has been presented. This approach uses the

feature vectors which include whole image frames containing all the aspects of the

sign. This project has investigated the different issues of this new approach to SL

recognition to recognize on the hand sign language alphabets and numbers using

appearance based features which are extracted directly from a video stream recorded

with a conventional camera making recognition system more practical. Although

sign language contains many different aspects from manual and non-manual cues,

the position, the orientation and the configuration or shape of the dominant hand of

the signer conveys a large portion of the information of the signs. Therefore, the

geometric features which are extracted from the signers’ dominant hand, improve

the accuracy of the system to a great degree. This project did not focus on facial

expressions although it is well known that facial expressions convey important part

of sign-languages. The facial expressions can e.g. be extracted by tracking the

signers’ face. Then, the most discriminative features can be selected by employing a

dimensionality reduction method and this cue could also be fused into the

recognition system.

The sign language translator is able to translate alphabets (A-Z) and

numbers (1-9). All the signs can be translated real-time. But signs that are

similar in posture and gesture to another sign can be misinterpreted,

resulting in a decrease in accuracy of the system. The current system has

only been trained on a very small database. Since there will always be

variation in either the signers hand posture or motion trajectory, a larger

database accommodating a larger variety of hand posture for each sign is

required. The speech recognition program requires the user to take up a

tutorial of 10 minutes. During the training, the program learns the accent of

the user for speech recognition. It is observed that, the longer the user used

the program , the higher the accuracy of speech recognition.

This paper presents a novel approach for gesture detection. This approach

has two main steps: i) template preparation, and ii) gesture detection. The

template preparation technique presented here has some important features

for gesture recognition including robustness against slight rotation, small

number of required features and device independence. For gesture detection,

a pattern matching technique is used. The gesture recognition technique

presented here can be used with a variety of front-end input systems such as

vision based input , hand and eye tracking, digital tablet, mouse, and digital

glove. Much previous work has focused on isolated sign language

recognition with clear pauses after each sign. These pauses make it a much

easier problem than continuous recognition without pauses between the

individual signs, because explicit segmentation of a continuous input stream

into the individual signs is very difficult. For this reason, and because of co-

articulation effects, work on isolated recognition often does not generalize

easily to continuous recognition. But the proposed software captures the

input images as an AVI sequence of continuous images. This allows for

continuous input image acquisition without pauses. But each image frame is

processed individually and checked for pattern matching. This technique

overcomes the problem of processing continuous images at the same time

having input stream without pauses.

For Speech to SL translation words of similar pronunciation are

sometimes misinterpreted. This problem can be avoided by clearly

pronouncing the words and with extended training and increasing usage. The

speech recognition technique introduced in this article can be used with a

variety of front-end input systems such as computer and video games,

precision surgery, domestic applications and wearable computers.

Figure 1.4 Block diagram of SL Figure 1.5 Block diagram of Speech

to speech translation to sign language translation Figure 1.6 PAGE 3-GUI of template

preparation

Figure 1.8 PAGE 2-GUI of Speech to

SL translation Figure 1.9 GUI of windows

speech recognition tutorial

Figure 1.7 PAGE 4-GUI of SL to

speech translation

Figure 1.10 Database of SL finger spelling Alphabets and Numbers