Robust Hand Tracking with Refined CAMShift Based on Combination of Depth and Image FeaturesWenhuan Cui, Wenmin Wang, and Hong Liu

International Conference on Robotics and Biomimetics, IEEE, 2012

2

Outline• Introduction • Related Work• Proposed Method• Experimental Results• Conclusion

3

Introduction

4

Introduction• Hand Tracking:

• Essential for HCI• Most researchers simplify the issue by restrictions:

• On users’ clothing

• On the scene complexity

• On hand motion

Zhou Ren, Junsong Yuan, , Jingjing Meng, M, and Zhengyou Zhang, "Robust Part-Based Hand Gesture Recognition Using Kinect Sensor", IEEE TRANSACTIONS ON MULTIMEDIA, AUGUST 2013

5

Introduction• In this paper:

• Propose a robust hand tracking method

• Focus on reducing restrictions

• Combining:

• Depth cues• Color cues• (Motion cues)

Refined CAMShift tracking

6

Related Work

Related work• Tracking:

7

[a] fingertip

‧Seed Point‧Predicted hand position

[b] hand

Geodesicdistance

GSP points

Neighbordepth

Related work• Difficulties:

8

[c]

[d]

-- (Red) : Side-modeㄧ (Blue) : Frontal-mode

Related work• [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking

in Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012

• [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011

• [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012

• [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013

9

10

ProposedMethod

Flow Chart

11

Hand Detection Hand Tracking

12

Foreground Segmentation• Codebook model

• Codeword:

• Motion detection(Foreground):

K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.

Down-Sampled

13

Foreground Segmentation

14

Histogram-based Segmentation• Stretch ahead

• Depth histogram

• Stretch laterally

• X-projection histogram

Mask

Mask

15

Histogram-based Segmentation• Stretch ahead

• Depth histogram

depth

16

Histogram-based Segmentation• Stretch laterally

• X-projection histogramLower boundary

Upper boundary

j-th bin

x

17

Histogram-based Segmentation• Histogram Analysis

• Depth histogram & X-projection histogram

• Foothill algorithm:

18

Histogram-based Segmentationmax

max01

01

01 10 01 10

01

Depth histogram

X-projection histogram

000000111111011100000

19

Histogram-based Segmentation

Scaled x-mask

X-projection histogram

20

Skin Color Feature

‧

‧

‧

Mask

21

Integration of Features• Hand Detection:

skin

depth(stretch ahead)

X-projection(stretch laterally)

22

• Like mean-shift

• 1. Back projection• Choose an object → probability map → back projection

• 2.Mean-shift (frame-frame)

CAMShift

23

Refined CAMShift Tracking• Probability map:

• Weights:

• s1 : depth mask• s2 : x-mask

blob size < threshold

otherwise

skin depth(stretch ahead)

X-projection(stretch laterally)

24

• Ecliptic shape representation

Refined CAMShift Tracking

Aspect ratio:

Search window for the next frame:

25

• Blob refinement

Refined CAMShift Tracking

1. Choose proper reference line

2.

3. Reduce the l of the ellipse, untill a proper aspect ratio l/w is obtained.

26

• Aspect ratio based blob refinement

Refined CAMShift Tracking

27

• Tracking fast movementDetection +

Tracking

28

• Face & Hand Detection +

Tracking

29

ExperimentalResults

30

Experimental Results• Comparison of overall performance

[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674

‧Training: 4.8s / 10FPS

31

• A) Refined CAMShift with color cue

32

• B) Multi-cue CAMShift without refinement

33

• C) The proposed approach

34

Experimental Results• Video description experimental results

35

Conclusion

36

Conclusion• Focus on reducing restrictions

• Hand Segmentation:• Depth + Skin + (Motion)• Histogram analysis

• Hand tracking• CAMShift• Blob refinement