sign language recognition
DESCRIPTION
Sign Language Recognition. Overview. Average person typing speed Composing: 19 words per minute Transcribing: 33 words per minute Sign speaker Full sign language: 200—220 wpm Spelling out: est. 50-55 wpm Faster by 1.5x to 3x. Purpose and Scope. Native signers can input faster Benefits: - PowerPoint PPT PresentationTRANSCRIPT
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Sign Language Sign Language RecognitionRecognition
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
OverviewOverview Average person typing speed
Composing: 19 words per minute Transcribing: 33 words per minute
Sign speaker Full sign language: 200—220 wpm Spelling out: est. 50-55 wpm Faster by 1.5x to 3x
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Purpose and ScopePurpose and Scope Native signers can input faster
Benefits: Hearing & speaking disabled Sign interpreters
Just letters & numbers for now
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
ResearchResearch Related projects
Using mechanical gloves, colored gloves Tracking body parts Neural network-based application
Still images: 92% accuracy Motion: less than 50% accuracy
Feature vector-based application Also about 90% accuracy on stills No motion tests
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
More ResearchMore Research Image techniques:
Edge detection (Robert’s Cross) Line detection (Hough transform) Line interpretation methods
Chaining groups of lines Macro-scale templates
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Program ArchitectureProgram Architecture Webcam interface
Edge detection
Line detection
Line interpretation
Hand attribute matching
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Testing ModelTesting Model Human interaction necessary General testing model:
bhood@bulusan: ~/syslab-tech $ \> ./main images/hand.pgm in.pgm
[DEBUG] Edge detect time: 486 ms[DEBUG] Wrote image file to `in.pgm’
Errors: 0 Warnings: 0
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Code sampleCode sample Robert’s Cross edge detection: int row,col;// loop through the imagefor(row=0; row<rows; row++) { for(col=0; col<cols; col++) { // avoid the problems from trying to calculate on an edge if(row==0 || row==rows-1 || col==0 || col==cols-1) { image2[row][col]=0; // so set value to 0 } else { int left = image1[row][col-1]; // some variables int right = image1[row][col+1]; // to make the final int top = image1[row-1][col]; // equation seem a int bottom = image1[row+1][col]; // little bit neater image2[row][col]=(int)(sqrt(pow(left – right , 2) + pow(top - bottom, 2))); } }}
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Edge Detection ResultsEdge Detection Results Results:
Shows the important edges but little else
Robert’s Cross method the optimal balance
Original image(800 x 703)
Final image(800 x 703)
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Cropping ResultsCropping Results Remove useless
rows with no features
Very large optimization Area difference Input/output
Original(800 x 703)
Result(633 x 645)
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Line detectionLine detection Mostly complete
Still some bugs to iron out
Shows nonexistent lines occasionally (see image!)
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
Line InterpretationLine Interpretation Chaining groups of lines
Templates Generation Template-based comparison
Hand matching
Byron Hood | version 0.3Computer Systems Lab Project 2007-2008
The Mysterious FutureThe Mysterious Future Finish line interpretation
Building method for interpreting lines Working on template-based recognition Write XML template files
Finish camera-computer interaction Device control must be precise, picky