§ AvaluabledevicewouldbeabletotranslatesignlanguageintospokenwordstoeasethecommunicationbarrierbetweenpeoplewhoonlyknowsignlanguageandpeoplewhoonlyknowEnglish
§ MicrosofthasdemonstratedasignlanguageinterpreterusingitsKinectproduct,thoughthedevicehadtroubleidentifyingallthefingerssoweintendtoimproveupontheirresultsbyusingtheLeapMotiondevice
§ TheLeapisdesirableasitfocusesonhandmovementsandissmallerandmoreportable. Itisalsocheaperandoperateseasilyonthethreemajoroperatingsystems
§ WeusearecurrentneuralnetworktotrainandanalyzesignlanguagedatainputfromtheLeapobserving372featuresfromthehandincludingthethreedimensionalcoordinatesofallthejointsinthehandandthecenterofthepalm
§ Wehaveachieved95%accuracyover26classes,namelythelettersofthealphabet
ABSTRACT
§ AmericanSignLanguage(ASL): Thislanguageisusedbyapproximately500thousandto2millionpeopleacrosstheUnitedStates.Itismostcommonamongthedeafcommunity.
§ Previouswork:Historically,Microsoft’sKinecthasbeenusedinconjunctionwithrandomforest toclassifystationaryASLletterswith92%accuracy.Further,theLeaphasbeenusedwithsupportvectormachineattaining80%accuracy.
§ LeapMotion: ThisisapieceofhardwarethatconnectstoacomputerviaUSB.Weuseittocollectapproximately60framespersecondwhereeachframeholds372featuresonthehands.Thedatatakestheformofagrayscalestereoimageofthenear-infraredlightspectrum,separatedintotheleftandrightcameras.
BACKGROUND
§ Wehaveachieved95%accuracyinidentifyingthe26lettersinthealphabet
§ Confusionmatrix:
RESULTS AND FUTURE WORK
§Dong,C.,Leu,M.C.,&Yin,Z.(2015).AmericanSignLanguageAlphabetRecognitionUsingMicrosoftKinect.2015ComputerVisionandPatternRecognition.IEEE.
§ T.Kim,J.Keane,W.Wang,H.Tang,J.Riggle,G.Shakhnarovich,D.Brentari,andK.Livescu,"Lexicon-FreeFingerspellingRecognitionfromVideo:Data,Models,andSignerAdaptation”arXiv:1609.07876v12016
§H.Sakoe andS.Chiba,“Dynamicprogrammingalgo- rithmoptimizationforspokenwordrecognition,”IEEETrans.Acoust.,Speech,SignalProcess.,vol.26,no.1,pp.43–49,1978.
§ C.H.Chuan,E.ReginaandC.Guardino,"AmericanSignLanguageRecognitionUsingLeapMotionSensor,"MachineLearningandApplications(ICMLA),201413thInternationalConferenceon,Detroit,MI,2014,pp.541-544
REFERENCES
SIGN LANGUAGE LEAPS TO ENGLISHNicholasFrost,WilliamGrant,Kien Nguyen,andParth Parikh
{naf77,wrg34,khn22,prp60}@scarletmail.rutgers.eduAdvisor:ProfessorAnand Sarwate
RutgersUniversity,DepartmentofElectricalandComputerEngineering
METHODOLOGY
Leap:Usehandtosignaletter
Datareaderandrawdatalistener:ProducesaJSONfilewithallthe
informationonthehand
Transformer:Sample,filter,andnormalizeraw
data
RecurrentNeuralNetwork:Trainand
classifyletterviaKeraslibrary
OutputEnglishletterandhistogramwithtopfiveclassificationsviaafriendlyuserinterface
WewouldliketoacknowledgeProfessorHanaGodrich andundergraduatestudentMichaelSoskind fortheirconsistentsupportthroughouttheproject.WefurtherwouldliketothanktheRutgersSignLanguageClubandspecificallyIsabeau Touchard forbeingthesourceofauthenticsignlanguagedata.
ACKNOWLEDGEMENTS
DatacollectionexampleandLeapdiagnosticviewer:
https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwjM-K-bjKfTAhXJ7CYKHfr9CgkQjRwIBw&url=https%3A%2F%2Fwww.pubnub.com%2Fblog%2F2015-08-19-motion-controlled-servos-with-leap-motion-raspberry-pi%2F&psig=AFQjCNGjqqDvWcMW45-rLNyC52-9QbXGSQ&ust=1492367466269697
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
§ Keytakeaway:WehavedemonstratedproofofconceptforaLeapdevicetobeusedasasignlanguagetranslator.Thisdirectlyimpactsthedeafpopulationacrosstheworld.
§ Futurework§ Extendthecapabilityofthesoftwaretobeableto
identifymoreclassesincludingwordsandphrases§ Includevideodatatoidentifysignsthatdependon
relativelocationofbody
§ Conventionalclassificationmethodsdonotworkwellwithtimeseriesdata(i.e.letterswithmotion)
§ Wesolvethisbyemployingarecurrentneuralnetwork,specificallylongshorttermmemorytoextracttemporalfeatures
