§ AvaluabledevicewouldbeabletotranslatesignlanguageintospokenwordstoeasethecommunicationbarrierbetweenpeoplewhoonlyknowsignlanguageandpeoplewhoonlyknowEnglish

§ MicrosofthasdemonstratedasignlanguageinterpreterusingitsKinectproduct,thoughthedevicehadtroubleidentifyingallthefingerssoweintendtoimproveupontheirresultsbyusingtheLeapMotiondevice

§ TheLeapisdesirableasitfocusesonhandmovementsandissmallerandmoreportable. Itisalsocheaperandoperateseasilyonthethreemajoroperatingsystems

§ WeusearecurrentneuralnetworktotrainandanalyzesignlanguagedatainputfromtheLeapobserving372featuresfromthehandincludingthethreedimensionalcoordinatesofallthejointsinthehandandthecenterofthepalm

§ Wehaveachieved95%accuracyover26classes,namelythelettersofthealphabet

ABSTRACT

§ AmericanSignLanguage(ASL): Thislanguageisusedbyapproximately500thousandto2millionpeopleacrosstheUnitedStates.Itismostcommonamongthedeafcommunity.

§ Previouswork:Historically,Microsoft’sKinecthasbeenusedinconjunctionwithrandomforest toclassifystationaryASLletterswith92%accuracy.Further,theLeaphasbeenusedwithsupportvectormachineattaining80%accuracy.

§ LeapMotion: ThisisapieceofhardwarethatconnectstoacomputerviaUSB.Weuseittocollectapproximately60framespersecondwhereeachframeholds372featuresonthehands.Thedatatakestheformofagrayscalestereoimageofthenear-infraredlightspectrum,separatedintotheleftandrightcameras.

BACKGROUND

§ Wehaveachieved95%accuracyinidentifyingthe26lettersinthealphabet

§ Confusionmatrix:

RESULTS AND FUTURE WORK

§Dong,C.,Leu,M.C.,&Yin,Z.(2015).AmericanSignLanguageAlphabetRecognitionUsingMicrosoftKinect.2015ComputerVisionandPatternRecognition.IEEE.

§ T.Kim,J.Keane,W.Wang,H.Tang,J.Riggle,G.Shakhnarovich,D.Brentari,andK.Livescu,"Lexicon-FreeFingerspellingRecognitionfromVideo:Data,Models,andSignerAdaptation”arXiv:1609.07876v12016

§H.Sakoe andS.Chiba,“Dynamicprogrammingalgo- rithmoptimizationforspokenwordrecognition,”IEEETrans.Acoust.,Speech,SignalProcess.,vol.26,no.1,pp.43–49,1978.

§ C.H.Chuan,E.ReginaandC.Guardino,"AmericanSignLanguageRecognitionUsingLeapMotionSensor,"MachineLearningandApplications(ICMLA),201413thInternationalConferenceon,Detroit,MI,2014,pp.541-544

REFERENCES

SIGN LANGUAGE LEAPS TO ENGLISHNicholasFrost,WilliamGrant,Kien Nguyen,andParth Parikh

{naf77,wrg34,khn22,prp60}@scarletmail.rutgers.eduAdvisor:ProfessorAnand Sarwate

RutgersUniversity,DepartmentofElectricalandComputerEngineering

METHODOLOGY

Leap:Usehandtosignaletter

Datareaderandrawdatalistener:ProducesaJSONfilewithallthe

informationonthehand

Transformer:Sample,filter,andnormalizeraw

data

RecurrentNeuralNetwork:Trainand

classifyletterviaKeraslibrary

OutputEnglishletterandhistogramwithtopfiveclassificationsviaafriendlyuserinterface

WewouldliketoacknowledgeProfessorHanaGodrich andundergraduatestudentMichaelSoskind fortheirconsistentsupportthroughouttheproject.WefurtherwouldliketothanktheRutgersSignLanguageClubandspecificallyIsabeau Touchard forbeingthesourceofauthenticsignlanguagedata.

ACKNOWLEDGEMENTS

DatacollectionexampleandLeapdiagnosticviewer:

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwjM-K-bjKfTAhXJ7CYKHfr9CgkQjRwIBw&url=https%3A%2F%2Fwww.pubnub.com%2Fblog%2F2015-08-19-motion-controlled-servos-with-leap-motion-raspberry-pi%2F&psig=AFQjCNGjqqDvWcMW45-rLNyC52-9QbXGSQ&ust=1492367466269697

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

§ Keytakeaway:WehavedemonstratedproofofconceptforaLeapdevicetobeusedasasignlanguagetranslator.Thisdirectlyimpactsthedeafpopulationacrosstheworld.

§ Futurework§ Extendthecapabilityofthesoftwaretobeableto

identifymoreclassesincludingwordsandphrases§ Includevideodatatoidentifysignsthatdependon

relativelocationofbody

§ Conventionalclassificationmethodsdonotworkwellwithtimeseriesdata(i.e.letterswithmotion)

§ Wesolvethisbyemployingarecurrentneuralnetwork,specificallylongshorttermmemorytoextracttemporalfeatures