movement tracking in real-time hand gesture recognition

Movement Tracking in

Real-time Hand Gesture Recognition

PresentersPurohit Pankaj (W579074)Salagar Muaaz (W579080)Kulkarni Pranav (2010BCS203)Desale Ritesh (2010BCS211)

Seminar GuideMrs. S. S. Solapure

Agenda IntroductionReference Paper and Its Contents

ApplicationConclusionQuestions

Agenda IntroductionReferenced Paper and Its Contents

ApplicationConclusionQuestions

Introduction

Introduction

Gesture Recognition

Hand Gesture Recognition

Introduction (Contd.)

What is it?

How it works?

Introduction (Contd.)

What is its NEED?

Advantages Natural Interaction Builds a Richer Bridge Remote Interaction Wonderful Gaming Experience

Agenda IntroductionReference Paper and Its Contents

ApplicationConclusionQuestions

Reference Paper

Movement Tracking in Real-time Hand Gesture

RecognitionAuthored by

Hong-Min Zhu & Chi-Man PunDepartment of Computer and Information

ScienceUniversity of Macau, Macau SAR, China

{ma86560, cmpun} [at] umac.mo

9th IEEE/ACIS International Conference on Computer and Information Science

Reference PaperWhat does it say?This paper deals with overcoming of SCHD technique for Hand Gesture Recognition using newly improved Algorithm, IFDHD

Procedures in General Framework of Gesture Recognition

Previous Work DoneTemporal Hand GestureAssumptions

Camera User Synchronization Uniform Lightening Condition Simple Background Features Frame Rate – Gesture Speed

Coordination

S C H D Skin Color based Hand Detection

J. Kovac and P. Peer – Designed Skin Classifier

Rules Pixel is classified as a skin pixel if:

Value of Red > 95, Green > 40 and Blue > 20 & max{R, G, B} - min{R, G, B} > 15 & |R - G| > 15 and R > G and R > B

Proposed Solution Problems with SCHD

Computationally Expensive Skin-like Object Ambiguity Illumination Parameters Skin Color Variation

Solution – Motivated from BSHD

IFDHDInter-Frame Difference based Hand Detection

Proposed Solution

Hand DetectionModule

Motion TrackingModule

Hand Detection Module

Figure 3.1 Zoomed Mode for Hand Detection Module

Algorithm for Hand Detection

Input: Frames Fi = 1..N from video segmentSteps:

1. Convert frame F1 to grayscale2. Repeat (until end of video segment)

1. Convert frame Fi to grayscale2. Intensity difference image D0 = |Fi – F1|3. Binary image I = (D0 > T0)4. Do image opening on I followed by closing5. Splitting of Large regions into max size boundary box

as 60x806. Calculate center co-ordinate

Output: Center coordinate of each region in each frame

Experimental ResultsSCHD based Hand DetectionLightening Condition IFDHD based Hand DetectionSCHD based Movement Tracking IFDHD based Movement TrackingEfficiency Measurement

Fig. 4. Result of SCHD(a) original frame

Fig. 4. Result of SCHD(b) Skin Pixel Classification

Fig. 4. Result of SCHD(c) De-noise & Region

Connection

Fig. 4. Result of SCHD(d) Region Splitting

Fig. 4. Result of SCHD(e) Centers of Each Region

Experimental ResultsSCHD based Hand DetectionLightening Condition IFDHD based Hand DetectionSCHD based Movement Tracking IFDHD based Movement TrackingEfficiency Measurement

Fig. 5. Effect of Lightening Condition

(a) Original Frame

Fig. 5. Effect of Lightening Condition

(b) Skin Pixel Classification

Experimental ResultsSCHD based Hand DetectionLightening Condition IFDHD based Hand DetectionSCHD based Movement Tracking IFDHD based Movement TrackingEfficiency Measurement

Fig. 6. Result of IFDHD

(a) 1st Frame

Fig. 6. Result of IFDHD

(b) 11th Frame

Fig. 6. Result of IFDHD

(c) Subtraction: (b) - (a)

Fig. 6. Result of IFDHD

(d) Threshold of (c)

Fig. 6. Result of IFDHD

(e) De-noise

Fig. 6. Result of IFDHD

(f) Region Splitting

Fig. 6. Result of IFDHD

(g) Centers of Regions

System Domain (Contd.)

Movement Tracking Module

Figure 3.2 Zoomed Mode for Movement Tracking Module

Algorithm for Movement Tracking

Input: Region centers Detected in each frameSteps:

1. Initialize the start of frame2. Repeat (for each frame > 1)

1. Identify tail locations and store2. Calculate matrix of distances between centers and

tail locations3. Repeatedly select – min(Distance ( I ), Distance

( J )) 4. If Distance( I ) < Threshold then append Center to

Gesture and delete Distance( I ) Else initialize new Gesture start location

5. Select Gesture Frame that has the maximal standard deviation

6. Smooth movement track Gesture and interpolate it to Number of Center Coordinate falls coordinates

Output: Encoding of movement track

Experimental ResultsSCHD based Hand DetectionLightening Condition IFDHD based Hand DetectionSCHD based Movement Tracking IFDHD based Movement TrackingEfficiency Measurement

Fig. 7. SCHD Based Movement TrackingFirst Row: Last Frame of Video Segment

Fig. 7. SCHD Based Movement TrackingSecond Row: Detected Digit Track

Fig. 7. SCHD Based Movement TrackingThird Row: Smoothed Track

Experimental ResultsSCHD based Hand DetectionLightening Condition IFDHD based Hand DetectionSCHD based Movement Tracking IFDHD based Movement TrackingEfficiency Measurement

Fig. 8. IFDHD Based Movement TrackingFirst Row: Last Row of Video Segment

Fig. 8. IFDHD Based Movement TrackingSecond Row: Detected Digit Track

Fig. 8. IFDHD Based Movement TrackingThird Row: Smoothed Track

Experimental ResultsSCHD based Hand DetectionLightening Condition IFDHD based Hand DetectionSCHD based Movement Tracking IFDHD based Movement TrackingEfficiency Measurement

Efficiency Measurement

Table 1. Comparing the Efficiencies of SCHD and IFDHD

Outline IntroductionReferenced PaperApplication ScenarioConclusionQuestions

Reference PaperAmerican Sign Language Recognition System for Hearing Impaired People Using Cartesian

Genetic ProgrammingAuthored ByFahad Ullah

Department of Computer Systems Engineering,University Of Engineering & Technology,

Peshawar, Pakistan

Proceeding of 5th International Conference on Automation, Robotics and Applications, New

Zealand

Application Scenario

Why the interfaces are changing ?

How many Apps Out there? Have you tried X-box,PSP-2,Mac-OSX January 9, 2012, 66 million Xbox 360

consoles have been sold worldwide.

New era of Interfaces

Application Scenario What if you can’t speak? ASL CGP (Cartesian Genetic Programming) How it works? Genetic programming an Overview:

Probabilistic search Darwinian principle of natural

selection Naturally occurring genetic operations

such as crossover and mutation.

• Better individuals are preferred• Best is not always picked• Worst is not necessarily excluded• Nothing is guaranteed• Mixture of greedy exploitation and

adventurous exploration• Similarities to simulated annealing

(SA)

Probabilistic Selection Based On Fitness

Workflow

ASL using CGP

26 English language alphabets are trained and Identified

The system uses 26 binary images representing the different alphabets

Mentioned system with a Dictionary correction ability in order to increase the overall accuracy of the system.

Outline IntroductionReferenced PaperApplicationConclusionQuestions

ConclusionProposed IFDHDServing Feature Extraction Stage

Overcoming the pitfalls of SCHD

Outline IntroductionReferenced PaperApplicationConclusionQuestions

Questions, IF ANY?

Q?Thank You