Robust Hand Tracking with Refined CAMShift Based on Combination of Depth and Image FeaturesWenhuan Cui, Wenmin Wang, and Hong Liu
International Conference on Robotics and Biomimetics, IEEE, 2012
2
Outline• Introduction
• Related Work
• Proposed Method
• Experimental Results
• Conclusion
3
Introduction
4
Introduction• Hand Tracking:
• Essential for HCI
• Most researchers simplify the issue by restrictions:
• On users’ clothing
• On the scene complexity
• On hand motion
Zhou Ren, Junsong Yuan, , Jingjing Meng, M, and Zhengyou Zhang, "Robust Part-Based Hand Gesture Recognition Using Kinect Sensor", IEEE TRANSACTIONS ON MULTIMEDIA, AUGUST 2013
5
Introduction• In this paper:
• Propose a robust hand tracking method
• Focus on reducing restrictions
• Combining:
• Depth cues
• Color cues
• (Motion cues)
Refined CAMShift tracking
6
Related Work
Related work• Tracking:
7
[a] fingertip
‧Seed Point‧Predicted hand position
[b] hand
Geodesicdistance
GSP points
Neighbordepth
Related work• Difficulties:
8
[c]
[d]
-- (Red) : Side-modeㄧ (Blue) : Frontal-mode
Related work• [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking
in Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012
• [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011
• [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012
• [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013
9
10
ProposedMethod
Flow Chart
11
Hand Detection Hand Tracking
12
Foreground Segmentation• Codebook model
• Codeword:
• Motion detection(Foreground):
K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.
Down-Sampled
13
Foreground Segmentation
14
Histogram-based Segmentation• Stretch ahead
• Depth histogram
• Stretch laterally
• X-projection histogram
Mask
Mask
15
Histogram-based Segmentation• Stretch ahead
• Depth histogram
depth
16
Histogram-based Segmentation• Stretch laterally
• X-projection histogramLower boundary
Upper boundary
j-th bin
x
17
Histogram-based Segmentation• Histogram Analysis
• Depth histogram & X-projection histogram
• Foothill algorithm:
18
Histogram-based Segmentationmax
max0
1
0
1
01 10 01 10
01
Depth histogram
X-projection histogram
000000111111011100000
19
Histogram-based Segmentation
Scaled x-mask
X-projection histogram
20
Skin Color Feature
‧
‧
‧
Mask
21
Integration of Features• Hand Detection:
skin
depth(stretch ahead)
X-projection(stretch laterally)
22
• Like mean-shift
• 1. Back projection• Choose an object → probability map → back projection
• 2.Mean-shift (frame-frame)
CAMShift
23
Refined CAMShift Tracking• Probability map:
• Weights:
• s1 : depth mask
• s2 : x-mask
blob size < threshold
otherwise
skin depth(stretch ahead)
X-projection(stretch laterally)
24
• Ecliptic shape representation
Refined CAMShift Tracking
Aspect ratio:
Search window for the next frame:
25
• Blob refinement
Refined CAMShift Tracking
1. Choose proper reference line
2.
3. Reduce the l of the ellipse, untill a proper aspect ratio l/w is obtained.
26
• Aspect ratio based blob refinement
Refined CAMShift Tracking
27
• Tracking fast movementDetection +
Tracking
28
• Face & Hand Detection +
Tracking
29
ExperimentalResults
30
Experimental Results• Comparison of overall performance
[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674
‧Training: 4.8s / 10FPS
31
• A) Refined CAMShift with color cue
32
• B) Multi-cue CAMShift without refinement
33
• C) The proposed approach
34
Experimental Results• Video description experimental results
35
Conclusion
36
Conclusion• Focus on reducing restrictions
• Hand Segmentation:• Depth + Skin + (Motion)• Histogram analysis
• Hand tracking• CAMShift• Blob refinement