Company

LOGO

Mobile Motion Tracking using Onboard Camera

SupervisorProf. LYU, Rung Tsong Michael

Presented byLam Man Kit, Wong Yuk Man

Outline

1. Motivation & Objective1. Motivation & Objective

2. Motion Input 2. Motion Input

6. Experimental Result 6. Experimental Result

5. Applications 5. Applications

3. Previous Work3. Previous Work

4. Improvement & New Components4. Improvement & New Components

7. Demo7. Demo

Motivation

Rapid increase in the use of camera-phone.

Camera-phones’ processing power is increasing

Symbian OS makes programming on mobile phone possible

Objective

Develop motion input method on Symbian phone without requiring additional hardware

This acts as an innovative input method for applications such as: Camera mouse to control the cursor New input method for interactive games Gesture input

What is Motion Input?

Why Motion Input?

Motion input is quick and convenient Common way to play game on mobile

phone is to use joystick or keypad Limited freedom of movement Lack of maneuverability Difficult to press the small-sized keypad

Motion input doesn’t have these problems!

Motion Tracking

Motion tracking is a process to find the motion vector of the current frame from the reference frame

We use block matching algorithm to find the motion vector

Block Matching Algorithm

Comparison of blocks

(dx,dy)

The motion vector which corresponds to

the best match

New Position

Block Matching Algorithm

Evaluate the "goodness" of a match by Sum of Absolute Difference

Select the candidate block with the lowest error

N

i

N

j

jiYjiXyxSAD1 1

|),(),(|),(

Motion Tracking

Block matching algorithm Traditional block matching algorithm is slow Our solution: Hybrid-type fast block matching

algorithm That is “Adaptive Spiral SEA PPNM SSD” algorithm

Adaptive Search Window Spiral Scan Method Successive Elimination Algorithm Progressive Partial Norm Matching Sum of Squared Difference

Previous Work

In the previous semester, Developed a testing platform in both Window and

Symbian OS Developed our own block matching algorithm and

feature selection algorithm Built a real-time translational motion tracking engine Developed some applications using the engine

Improvement on Feature Selection

A good feature block: High variance -> complex block Not in a repeated pattern

If a good feature block is found, the performance of the motion tracking will be improved

We have made improvements to our existing algorithm

squares

Old Feature Selection

Divide the current frame into squares Apply Feature Selection Algorithm to each square However, the best feature block may appear between

two squares

Best feature block

New Feature Selection

More blocks are sampled Sample 26x26 squares with size 15x15 in a

54x54 window Old feature selection only samples 12x12

squares

Search in spiral way and stop searching when selection criteria are matched Prefer to find a feature block in the center Prevent out of bound problem (block appear

out of screen)

New Feature Selection

If a feature block is found on the edge, the following will happen:

The tracking algorithm will fail to find the exact match

Solution: Apply additional

constraint

New Feature selection

In order to find a feature block that is not on the edge, we apply a checking algorithm

Important difference in all directions -> interest point

New Feature Selection

Now our newest feature selection algorithm becomes: Find a block with a large variance which

indicates the complexity of the block Check if the block is on the edge or not by

calculating the SAD between the candidate block and its 4 neighbors

If either one of the SAD is small -> the block is on edge -> reject Else the block is not on edge

Rotation Tracking Engine

Observation and Motivation As we rotate the phone,

the object can still be tracked correctly (center of object always roughly equals center of green box)

First approach Track one block by

translational motion tracking engine, rotate the current block and then find which rotation of the current block best match with the previous block

Fail if tracking object is the same for different angle

One-block approach

Rotation Tracking Engine

Our approach If two blocks are

tracked simultaneously, angle of line connecting the two blocks indicates the rotation angle of the phone

Use only translational motion tracking

Two-block approach

Rotation Tracking Engine

Rotation Tracking Engine

Modification of adaptive search window method to suit rotation tracking

Phone’s motion can be both linear motion and circular motion

Linear Adaptive method used in translational motion tracking is not used in this engine

Why need adaptive search window method? To increase performance based on the fact that

during block matching, reaching the best candidate block earlier increases the elimination effect of SEA, PPNM, PDE

Rotation Tracking Engine

What to predict? Predict the coordinates of the blocks in the next frame Given the coordinates of the tracking blocks in the current frame

and the previous frames When to predict?

After each run of rotation tracking algorithm

θθ

Line L2

Line L1

Line L3

Rotation Tracking Engine

Solution Angle between L2 and L3 = θ

θ = Tan-1(slope(L2)) – Tan-1(slope(L3))

Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix

2

2

1

1

cossin

sincos

L

L

L

L

y

x

y

x

Rotation Tracking Engine

Solution The prediction of the positions of the next tracking blocks

should also take the translational movement into account Horizontal displacement = Tx

Tx = ( xL21 + xL22

– xL31 -

xL32

)/2

Vertical displacement = Ty Ty = ( yL21

+ yL22– yL31

- yL32

)/2

Coordinates of the next tracking block (xL1, yL1) is calculated by

1111

cossin

sincos

12

2

1

1

L

L

L

L

y

x

Ty

Tx

y

x

Rotation Tracking Engine

Reducing error by using level Increase/decrease one level only

when change is large To give less sensitive but more

desirable output for game e.g. skiing game: skier face only

to 7 directions Reduce difficulty of the game,

increase reliability of the engine

40°

80°

120°

- 40° 0°Level 1

Level 2

Level 3

Level 0

Level -1

Rotation Tracking Engine

Some screenshot showing how rotation tracking is carried out and how the result is displayed

Rotation Tracking Engine

A short video demo

Tailor-made background for both engines

Objective Max. performance measurement Increase usability

Condition Feature selection can always find a

good feature point Within certain distance

No repeat pattern Very distinct pattern

Pattern is nearly the same when rotated. E.g. Circle is good for rotation detection

Virtual Mouse

An application that make full use of the translational motion tracking engine

Remote control the mouse of PC by mobile phone using motion as input

Advantages: It allows input in many directions and provides high

levels of control Many buttons left for other purposes

E.g. Joystick can be used as course adjustment of mouse cursor

E.g. Some buttons can be used as shortcuts of applications

Virtual Mouse

Blu

etoo

th C

omm

unic

atio

n C

hann

el

Cable

Virtual Mouse

Server Configure Bluetooth device to provide

RFComm Service and regards the Bluetooth transmission port as Comm. Port

Server receives message from that Comm. Port

Call function in MouseAction.h to make the mouse move and trigger mouse click event

Virtual Mouse

Client Search for Bluetooth device nearby Connect to the selected Bluetooth device Every time motion tracking algorithm

finishes, results are sent to server (14 times/sec)

Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)

Car Racing Game

A “Car Racing Game” is developed using the motion tracking engine.

Game lags. It is because: The engine takes time to find the motion vector CPU speed of the Symbian phones is low

Solutions: Double Buffering Direct Screen Access

Skiing Game

We have developed a skiing game that makes use of the rotation tracking engine Rotate the phone to control the angle of the

skier

After using the rotation tracking engine: The game becomes more interactive The game has more degree of freedom for

rotating the skier

Skiing Game

Skiing Game

The process of using the rotation tracking engine is very simple Create an instance of the tracking engine Call the function of the engine to find the

motion vector Use the motion vector for the game logic

Skiing Game

Experimental Result on Symbian phone

Testing Environment Platform

Symbian Phone Nokia 6600 Algorithm

Our final (hybrid-type) algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)

Algorithm parameter Block size         = 17 x 17 (pixels) Search window size     = 16 x 16 (pixels)

Number of block to track in each run of algorithm = 1 block ONLY

Result

Run time of translational motion tracking algorithms

0

10

20

30

40

50

60

background

Tim

e (

ms)

Full Search (ms)

Partial Sprial (ms)

Adaptive SEA (ms)

Experimental Result on Symbian phone

Testing Result (avg time to run) Final algorithm

7ms Partial Spiral algorithm

22ms Full Exhaustive Search algorithm (the most

simplest one) 55ms

Result

Accuracy of the rotational engine

505560657075808590

bg 1 bg 2 bg 3 bg 4 bg 5 bg 6 bg 7 bg 8

measured angle

Conclusion

We have developed: The real-time motion tracking algorithm and

feature selection method The translational and rotational motion

tracking engine on Symbian A series of applications/games that make full

use of our engines

Demo

Virtual Mouse

Demo

Act as an innovative input device for games

Q & A

Thank You !!

Experimental Result1 - Backgrounds

Experimental Result2 - Backgrounds

Stereo

If we want to track the movement in 3D, we have to: Use two cameras Know the focal length of the cameras Know the distance between two cameras

Depth can be recovered with two images Depth can be recovered with two images and triangulationand triangulation

Stereo

Therefore, stereo has two steps Finding matching points in the images Then using them to compute depth.

OO22

P’P’22

PP

OO11

P’P’11

Stereo

Suppose we have found the matching point P. Now the step to compute the depth is as follow:

f: focal length of the cameras

b: distance between two cameras

Note: y-axis is perpendicular to the slide

Stereo

If b is the distance between the two cameras, and f is the focal length of the cameras, we have

Use similar triangular, we can solve to get

z

bx

f

xl 2/'

z

bx

f

xr 2/'

z

y

z

y

f

y rl ''

''

'' 2/)(

rl

rl

xx

xxbx

''

'' 2/)(

rl

rl

yy

yyby

''

rl xx

fbz

Mono

In our project, only one camera is available in the Symbian phone

If we want to find the z-axis movement, we can find it by comparing the size of the capture image Getting closer -> larger image Getting farther -> smaller image

However, we have tried that using this algorithm is not accurate enough