Project title : Automated Detection of Sign Language Patterns

Faculty: Sudeep Sarkar, Barbara Loeding, Students: Sunita Nayak, Alan Yang

Department of Computer Science and Engineering, Department of Special Education

Center of Excellence in Pattern Recognition

Goal and Impact Statement

Unsupervised Learning of Sign Models

Representation that does not require tracking

Publications and Acknowledgement

Movement Epenthesis Aware Matching

• R. Yang; S. Sarkar, B. Loeding, Enhanced Level Building Algorithm for the Movement Epenthesis Problem in Sign Language Recognition, to be presented at IEEE Conf. on Computer Vision and Pattern Recognition, 2007.• R. Yang; Sarkar, S., “Gesture Recognition using Hidden Markov Models from Fragmented Observations,”

IEEEConference on Computer Vision and Pattern Recognition pp. 766- 773, 17-22 June 2006.• R. Yang and S. Sarkar, “Detecting Coarticulation in Sign Language using Conditional Random Fields,” International Conference on Pattern Recognition vol.2, pp. 108- 112, 20-24 Aug. 2006• S. Nayak, S. Sarkar, and B. Loeding, “Unsupervised Modeling of Signs Embedded in Continuous Sentences,” IEEE Workshop on Vision for Human-Computer Interaction, vol. 3, pp. 81, June 2005.• R. Yang, S. Sarkar, B. L. Loeding, A. I. Karshmer: Efficient Generation of Large Amounts of Training Data for Sign Language Recognition: A Semi-automatic Tool. ICCHP 2006: 635-642• B. L. Loeding, S. Sarkar, A. Parashar, A. Karshmer: Progress in Automated Computer Recognition of Sign Language. ICCHP 2004: 1079-1087

This work was supported in part by the National Science Foundation under ITR grant IIS 0312993.

Goal: To advance the design of robust computer representations and algorithms for recognizing American Sign Language from video.Broader Impact:

• To facilitate the communication between the Deaf and the hearing population.

• To bridge the gap in access to next generation Human Computer Interfaces.

Intellectual Merit: We are developing representations and approaches that can

• Handle hand and face segmentation (detection) errors, • Learn, without supervision, sign models from

examples, • Recognize in the presence of movement epenthesis,

i.e. hand movements that appear between two signs.

Learn sign model given example sentences with one sign in common. In the following two sentences, the target sign model to be learned is HOUSE (marked in red)

SHE WOMAN HER HOUSE FIRE

fs-JOHN CAN BUY HOUSE FUTURE

S1

O3O2O1

S3S2

},{ 12

11

11 ppg

},{ 13

12

12 ppg ……

},{ 22

21

21 ppg

},{ 23

22

22 ppg ……

},{ 32

31

31 ppg

},{ 33

32

32 ppg ……

...

...

Frag-Hidden Markov Models:• Groups across frames are linked• Best match is a path in this

induced graph over groups• Matching involves optimization

over states AND groups for each frame

Movement epenthesis is the gesture movement that bridges two consecutive signs. This effect can be over a long duration and involve variations in hand shape, position, and movement, making it hard to model explicitly these intervening segments. This has been a problem when trying to match individual signs to full sentences. We have overcome this with a novel matching methodology that do not require modeling of movement epenthesis segments.

The error rates for enhanced Level Building (eLB) (our method), which accounts for movement epenthesis, and classical Level Building (LB) that does not account for movement epenthesis.

Segmentation Aware Matching

• We have proposed a novel representation that captures the Gestalt configuration of edges and points in an image.

• It can work with fragmented noisy low-level outputs such as edges and regions

• It captures the statistics of the relations between the low-level primitives• Distance and orientation between edge primitive.• Vertical and horizontal displacement • Relationships between short motion tracks

• Normalized RD is an estimate of Prob (Any two primitives in the image exhibit a relationship)

• The shape of the RD changes as parts of the objects move.

• Relational distributions over time model high-level motion patterns.