gesture recognition?

47
Gesture Recognition B98902037 邱邱邱 [email protected] +886937201724

Upload: dayo-choul

Post on 28-Jan-2015

555 views

Category:

Entertainment & Humor


1 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Gesture Recognition?

Gesture Recognition

B98902037 邱大祐[email protected]

+886937201724

Page 2: Gesture Recognition?

Abstract

• In my presentation, I will introduce a hand gesture recognition system to capture the hand movements and simple gesture.

• The feature of my method: low cost and low complexity.

Page 3: Gesture Recognition?

Motion Capturing Devices

1. Microsoft Kinect + XBOX ($US 330)

2. Leap Motion ($US 69.99)

Page 4: Gesture Recognition?

Using Mono Webcam

• Under $US 15 , it’s cheap and easy to set up.

Page 5: Gesture Recognition?

Flow Diagram

Page 6: Gesture Recognition?

Hand Detection (1)

• 1. Detect Motion Region• 2. Edge Detection• 3. Threshold• 4. Skin Color Estimation• 5. Cross Comparison• 6. Redraw Hand Shape

Page 7: Gesture Recognition?

The Flow Diagram in hand detection

And Operator

Motion Region Image

Skin Color Estimation

Image

Edge Detection

Binary Image

Histogram Equalization

Image

Page 8: Gesture Recognition?

Hand Detection (2)

• Hand moves significantly more than any other area in gesture capturing.

• Di(x, y) = |Fi(x, y) – Fi+1(x, y)|

• Fi(x, y): the i-th frame image.• X, y: the coordinates on the frame. • Di(x, y): the difference between Fi(x, y) and Fi+1(x, y).• Thresholding• Get ROI (Region Of Interest)

Page 9: Gesture Recognition?

Hand Detection (3)

• Di(x, y) = |Fi(x, y) – Fi+1(x, y)|

• Fi(x, y): the i-th frame image. • x, y: the coordinates on the frame. • Di(x, y): the difference between Fi(x, y) and

Fi+1(x, y).• Calculate the absolute value of frame

difference.• Bounding Box

Page 10: Gesture Recognition?

Hand Detection (4)

• Noise Removal in frame difference calculation.• Noise could be slightly moving objects, e.g.

slightly moving edge.• Set a threshold to remove noise (=80)• Di(x, y) = 1, if |Fi(x, y) – Fi+1(x, y)|> Threshold

Di(x, y) = 0, otherwise.• Otsu Method: Not good.

Page 11: Gesture Recognition?

Hand Detection (5)

• Calculate difference to find the motion region and run bounding box algorithm.

Page 12: Gesture Recognition?

Hand Detection (6)

Page 13: Gesture Recognition?

Hand Detection (7)

• Skin Color Estimation• In RGB color space, constrain the skin color, e.g.

R > G > B… I use HSV color space.• (0<=H<=20, 30<=S<=150, 60<=V<=255)• This method might include a wide range of

colors, but that doesn’t matter because we will do a cross comparison later. The noise including wall, poster, … could be removed based on motion region detection.

HSV: Hue, Saturation, ValueRGB: Red, Green, Blue

Page 14: Gesture Recognition?

Hand Detection (8)

• Label the estimated skin color as red.

Page 15: Gesture Recognition?

Hand Detection (9)

• Do morphology closing on skin color estimation image.

• Histogram equalization on source image.• Binarize on source image.• Edge detection (Sobel) on source image.

Page 16: Gesture Recognition?

Hand Detection (10)

• Finally, we can do a cross comparison by using motion region, binary image, skin color estimation image, and edge detection image.

• After that, we’ve done hand detection and calibration. We could use these results to track the basic hand gesture now.

Page 17: Gesture Recognition?

Hand Detection (11)

• After cross comparison, we could get a full hand shape.

Page 18: Gesture Recognition?

Background Subtraction (1)

• Till now, the result is not very stable because webcam has no light.

• The environment luminance sometimes causes variance. e.g. part of face.

• Further, we focus on removing the ambiguous match.

Page 19: Gesture Recognition?

Binary or Skin Image

• In dark environment, we can’t detect skin color in the palm region because it’s full of white color pixels.

• We need to enhance the dark-environment condition.

• Use mathematical morphology (closing).

Page 20: Gesture Recognition?

Ambiguous Match Problem

Arm moves significantly with palm. This image is taken in the dark environment. Such condition causes matching area to be larger than we expected.

Page 21: Gesture Recognition?

Mathematical Morphology

• Dilation and Erosion operation.

Page 22: Gesture Recognition?

Background Subtraction (2)

• Simple estimation 1. F1(x, y): background.

Fi(x, y): background if no ROI is detected.

2. By using height, width, and the centroid of white pixels to estimate the proper hand region.

Page 23: Gesture Recognition?

Background Subtraction (3)

• Object tracking: to be implemented.• Motion Vector: A motion-based recognition.

Page 24: Gesture Recognition?

Search Space (1)

A sketh map of search space and a group of search space and hand region.

Page 25: Gesture Recognition?

Search Space (2)

• Search space is composed of neighborhood relationship.

• Search space can be used to predict next movement of the target object and reduce the tracking time.

• Once when there is any variation in search space, we can simply regard it as direction moves.

Page 26: Gesture Recognition?

Motion Vector

• Use correlation to do image matching.• find to minimize dydx,

Ryx

dyydxximbPIXyximaPIX),(

|),,(),,(|

Page 27: Gesture Recognition?

Feature Detection

• Feature detection can detect the finger.• We can use feature points to build a convex

hull and feedback to hand detection image.• Ideally, we could detect multiple hands and

detect hand regions precisely.• Also, we can use these feature points to

detect hand gesture more precisely.

Page 28: Gesture Recognition?

Feature Detection (1)• cvfindContours: contour detection• SIFT, SURF algorithm: feature detection• 三點法 ( 或 11 點法 )• Assume contour has N points, P1, P2, …, Pn.

• Pi = (x, y).• The approximate curvature:

2/3)(22

ii

iiiii

yx

xyyxC

SIFT: Scale-Invariant Feature TransformSURF:

Page 29: Gesture Recognition?

Feature Detection (2)

2/3)(22

ii

iiiii

yx

xyyxC

iii xxx 1 iiii xxxx 211

11 iii yyyiiii yyyy 211

Page 30: Gesture Recognition?

The Detection Method we chosed

• The Viola–Jones object detection framework: the first object detection framework to provide competitive object detection rates in real-time proposed in 2001 by Paul Viola and Michael Jones.

• It was motivated primarily by the problem of face detection.

• This algorithm is implemented in OpenCV as cvHaarDetectObjects().

Page 31: Gesture Recognition?

Feature Detection (4)

• Feature types used by Viola and Jones.

Page 32: Gesture Recognition?

SIFT and SURF Detection

• Too slow for a real-time system with DEFAULT settings.

• In fact, in gesture recognition, we don’t need too many features.

Page 33: Gesture Recognition?

The feature we use• Haar-like features

• Different colors denote different Haar-like features.

Page 34: Gesture Recognition?

Haar-like features modularization

• We use the spatial relationship to modularize a finger model.

Page 35: Gesture Recognition?

Recognition and Tracking (1)

Page 36: Gesture Recognition?

Recognition and Tracking (2)

Rotate(Vf);

Scale(Vf);

 for( i = 1 to n){ɵ= arcos(Vf˙Vei / |Vf||Vei|);

if (ɵ  0)≒break; /*matched with Vei */

}

Page 37: Gesture Recognition?

Overcome stereo problems

• For a user sitting in front of a laptop or PC, when he gestures, the hand structure must be palm, wrist, and arm in the top-down order in an image.

Page 38: Gesture Recognition?

PART II

How to reduce the CPU loading

Page 39: Gesture Recognition?

Simple Detection Algorithm (1)

• To capture general hand motion, this is module is enough

• Low-Complexity• DO NOT USE significant

feature detection to detectmotion variation in the beginning.

Page 40: Gesture Recognition?

Simple Detection Algorithm (2)

• DO NOT USE significant feature detection to detect motion variation in the beginning.

• Why?• An example by my classmate, they use feature

detection first, to form hand region. This is NOT recommended.

• More complex, less prior.

Page 41: Gesture Recognition?

Simple method, simple processing

Hand Region

Camera View

Page 42: Gesture Recognition?

The processing diagram

Detect hand region size variation Start Tracking

Analyze Path and output the recognition movement

Iteration method

Page 43: Gesture Recognition?

Iteration control (1)

• Initializing: This is to prevent log overflow• Timing: When to start, when to end?

If we enter tracking phase, each frame should be logged if hand region is detected.Our parameter: frame > 20, time > 0.2 sec.

Done yet?No

Page 44: Gesture Recognition?

Iteration control (2)

• When analyzing path, path length is critical parameter.

• If the path we set is too long, that would cause serious latency.

• If the path we set is too short, that is too sensitive, not good for User Experience.

The path looks like this:

Page 45: Gesture Recognition?

Iteration control (3)

• The path segmentation is critical. • Should be tuned again and again in order to

give response at the exactly time user want.

Page 46: Gesture Recognition?

To be continued…

Page 47: Gesture Recognition?

Thank you