craig newman curtin university of technology translating australian sign language into speech
TRANSCRIPT
Craig NewmanCurtin University of Technology
Translating Australian Sign Language into Speech
Translating Australian Sign Language into Speech
QuickTime™ and a Video decompressor are needed to see this picture.
• Introduction To Auslan• Elements Of Auslan Signs• Hardware Used• Hardware Control Algorithm• Elements Of Sign Recognition Algorithm• Future Intentions
Presentation OverviewPresentation Overview
• British and Irish origins• More than 4000 commonly used signs • 0.1% of Australians use Auslan
Introduction to AuslanIntroduction to Auslan
• Handshapes• Location• Orientation• Movement• Expression
Elements of Auslan signsElements of Auslan signs
• Large Scale• Straight Line• Series of Straight Lines• Arcs• Circles
• Small Scale• Changes in Orientation• Changes in Fingers New Handshape
MovementMovement
• Known through the P5• Degree of Finger Bend• Position Primary Locations, Movement• Orientation
• Required• Contacting Secondary Locations
What was Needed?What was Needed?
• Collect Data Every 17ms• 60 Hertz• Synchronisation with P5
• Test Pins Not Logged• Write Pin High, Read Remaining Pins• Log Connected Pins• Debounce: Test For Consistency• Encapsulate for Transmission• Transmit When Required
Contact Sensor AlgorithmContact Sensor Algorithm
Working Example 1Working Example 1
QuickTime™ and a Video decompressor are needed to see this picture.
Working Example 2Working Example 2
INSERT CLIP: IM
QuickTime™ and a Video decompressor are needed to see this picture.
• 3 Samples Stored in stop_buffer• Position• Orientation• Finger Bend• Contacts
• Check Last 2 Samples within Range
When was the hand considered Stopped?
When was the hand considered Stopped?
• Wait Until Hands Stop• Take First Sample From stop_buffer• Filter Through if/else Structure to find
Correct Combination• Return Handshape
Identifying HandshapesIdentifying Handshapes
Working Example 1Working Example 1
INSERT CLIP: GUNQuickTime™ and a Video decompressor are needed to see this picture.
Working Example 2Working Example 2
INSERT CLIP: MOTHERQuickTime™ and a DV - PAL decompressor are needed to see this picture.
• Record Position of • Head• Shoulder• Bicep• Forearm
• Extrapolate Primary Locations• Else is Neutral Space
• Problem• Requires Dynamic Position Tracking
Identifying Primary LocationsIdentifying Primary Locations
• Wait Until “not stopped”• While “not stopped” copy stop_buffer
• Allow for Slight Pauses
• Filter Data By Testing For Inherent Geometric Structure
• Output Movement Type
Identifying Movement TypeIdentifying Movement Type
• Identify Initial Handshape, Location, Orientation
• Filter Through if/else Structure
• If Sign Recognised
• Output Sign
• Else
• Identify Movement
• Identify Final Handshape, Location, Orientation
• Filter through if/else Structure
• Output Sign
Putting it all together: Recognizing Signs
Putting it all together: Recognizing Signs
Working ExampleWorking Example
INSERT CLIP: WRAPUP2QuickTime™ and a Video decompressor are needed to see this picture.
• Design Composite Glove• Dynamic Tracking Of Primary Locations• Introduce Learning Algorithms
• Identifying Handshapes• Identifying Movement
• Translate Signing Context Into English Text
• Integrate Text To Voice Synthesizer
Future IntentionsFuture Intentions