a fast and robust fingertips tracking algorithm for vision-based multi-touch interaction
DESCRIPTION
A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction. Qunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu. 2013 10th IEEE International Conference on Control and Automation (ICCA). Outline. Introduction Related Work Proposed Method - PowerPoint PPT PresentationTRANSCRIPT
A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch InteractionQunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu
2013 10th IEEE International Conference on Control and Automation (ICCA)
2
Outline• Introduction
• Related Work
• Proposed Method• Hand localization
• Fingertips tracking
• The Multi-touch system
• Experimental Results
• Conclusion
3
Introduction
4
• Multi-touch technology:
• Sensor Based• Directly receive finger touch as input
• High cost → limits its application to some extent
• Computer Vision Based• Good scalability as well as good performance
Introduction
Image: Oka, K, Sato, Y, Koike, H. "Real-time fingertip tracking and gesture recognition", IEEE Computer Graphics and Applications, 2012
5
Introduction• In this paper:
• Propose a robust fingertip tracking algorithm:
• Real-time• Stereovision-based 3D multi-touch interaction system• Skin / Depth / Geometry structure
Hand Detection
FingertipTracking
Multi-touchSystem
6
Related Work
Related work• Geometry properties:
• Curvature• Edge or shape• Build a model
• Image Analysis• Template matching• Color Segmentation
7
L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, 2007.
D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.
Related work• Palm Center:
• Fingertip Detection
8
𝜴𝜺
[a]
[a]
[b]
[c] [d]
Geodesicdistance
GSP points
Neighbordepth
Related work• [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking in
Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012
• [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011
• [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012
• [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013
9
10
ProposedMethod
11
Hand Segmentation• Skin Color filter
• YCbCr color space
• Gaussian Mixture Model
• Describe the skin-color distribute• Single Gaussian Model:
• Gaussian Mixture Model:
Training data:
Weight of each Gaussian model:
color vector
12
Hand Segmentation• Skin Color filter
• : how skin-like the color is
• Expectation Maximization(EM) algorithm
13
Hand Segmentation• Depth Cue:
• The points with minimum depth are picked as seeds
• Region grow algorithm
skin
depth skin + depth
14
• Divide wrist and hand:• By a boundary curve [18]
• Minimum depth
• Boundary curve
Hand Segmentation
[18] Z. Mo and U. Neumann, “Real-time hand pose recognition using low-resolution depth images,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2.
r : row indexc : column indexz(r,c) : depth value
,
range threshold(related to palm size)
boundary
15
Palm Region Extraction• Observation : Palm is a rectangle-like region
• Method : Project the hand region in all directions
𝟎° 𝟗𝟎° 𝐢𝐧𝐭𝐞𝐫𝐬𝐞𝐜𝐭𝐢𝐨𝐧𝐡𝐚𝐧𝐝
16
Palm Region Extraction
Intersection
17
Palm Center Localization•
𝟎°
>
>
𝟗𝟎°
X
𝟎°
>
𝟗𝟎°
、、
18
Palm Center Localization• Palm Center:
• The point with maximum distance from the closest palm boundary[18].
• The size of palm R:
palm region palm boundary
19
Fingertip Localization• Fingertip : The point with maximum distance to the palm center (on the contour of each finger)
• Candidate set F:P : contour pointC0 : palm centerd2 : distanceR : palm size
1.2
F
20
Fingertip Localization• Assign an index to each point in candidate set:
• Sort candidate set by
: index F : candidate setC0 : palm center the angle of with negative x-axis
21
Fingertip Localization• Distance between successive points :
• If > → Start & End point subset• Fingertips : maximum distance in each subset
22
Multi-touch system• TUIO (Tangible User Interface Object)
[24] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza, “Tuio:A protocol for table-top tangible user interfaces,” in Proc. of the The 6th Intl Workshop on Gesture in Human-Computer Interaction and Simulation, 2005.
23
ExperimentalResults
24
Experimental Results• Xeon 3.07Ghz workstation
• frame rate : 20Hz on average (real-time)
• Modules• Fingertip tracking• TUIO server• TUIO client
[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674
27
Conclusion
28
Conclusion• Fast and robust fingertip tracking• Without pressuring sensing device & extra marks
• Hand Segmentation• Depth / Skin
• Fingertip Detection• Palm region projection• Palm center distance from the boundary• Fingertip : assign index (angle)