Mobile Motion Tracking using Onboard Camera

Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit

Wong Yuk Man

Agenda

Motivation & ObjectivePrevious Work Improvement & New components Sample Applications

Conclusion

Q&A

Motivation

Rapid increase in the use of camera-phone

Camera-phone can perform image processing tasks on the device itself

Enhance human-computer interaction by mobile phone

Objective

Implement real-time motion tracking on Symbian phone, without requiring additional hardware

Translational motion trackingRotational motion tracking

To develop a tracking engine for other mobile devices developers to use

Previous Work

A translational motion tracking engineBlocking matching• SEA (Successive Elimination Algorithm)• PPNM• PDE (Partial Distortion Elimination)• Adaptive Window Search• Spiral Scan

Feature selection

A translational motion tracking engine

Block Matching Algorithm

Block Matching Algorithm

Evaluate the "goodness" of a match by Sum of Absolute Difference

Select the candidate block with the lowest error

N

i

N

j

jiYjiXyxSAD1 1

|),(),(|),(

Feature SelectionA good feature block:

High variance -> complex blockNot in a repeated pattern

If a good feature block is found, the performance of the motion tracking is improved

We have made improvements to our existing algorithm

squares

Old Feature Selection

Divide the current frame into squares

Apply Feature Selection Algorithm to each squaresHowever, the best feature block may appear between two squares

New Feature Selection

More blocks are sampledSample 26x26 variances of blocks with size 15x15 in a 54x54 window

Search in spiral way and stop searching when selection criteria are matched

Prefer to find a feature block in the centerPrevent out of bound problem (block appear out of screen)

New Feature SelectionIf a feature block is found on the edge, the following will happen:The tracking algorithm will fail to find the exact matchSolution:

Apply additional constraint

New Feature selectionIn order to find a feature block that is not on the edge, we apply a checking algorithmImportant difference in all directions => interest point

New Feature Selection

Now our newest feature selection algorithm becomes:

Find a block with a large variance -> which indicate the complexity of the blockCheck if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors

• If either one of the SAD is small -> the block is on edge -> reject

• Else the block is not on edge

SSD is used instead of SAD

SSD = Sum of Squared DifferenceLast term, SAD is used as matching criteriaSAD is commonly used because of its lower complexity (faster)But we chose to use SSD finally

SSD is used instead of SAD

Experiment shows that SSD has better performance in accuracy andAlgorithm with SSD run as fast as that with SADReason (of why as fast as SAD):

Elimination effects from SEA, PPNM & PDE become large when SSD is used Compensate the higher complexity of SSD

SSD is used instead of SADSEA and PPNM lower bound is adjusted using the inequality (12) proposed in the following paper

J.J. Francis and G. de Jager. A Sum Square Error based Successive Elimination Algorithm for Block Motion Estimation (2002) 

Experimental Result on Symbian phone

Frame rateMaximum frame rate supported by Nokia 6600 is about 14 frames/secRunning our algorithm (1/0.007 frames/sec) once only for each frame do not slow down the displaying frame rate, in other word, the frame rate can still be 14 frames/sec, our algorithm doesn’t lag the display

Rotation Tracking Engine

As we rotate the phone, the object can still be tracked correctly (center of object roughly equals center of green box)

Observation and Motivation

Rotation Tracking Engine

Our approachIf two blocks are tracked at the same time, angle of line connecting the two blocks reflect the tiling angle of the phone

Two-block approach

Rotation Tracking Engine

Another approachTrack one block by simple motion tracking engine, then find which rotation gives the best matchFail if tracking object is the same for different angle

One-block approach

Rotation Tracking Engine

Modification of motion tracking algorithm to suit rotation tracking

Linear Adaptive method used in translation motion tracking is not used here because phone’s motion can be both linear motion and circular motion

Rotation Tracking EngineWhat to predict?

Given the coordinates of the tracking blocks in the last 2 previous framesPredict the coordinates of the blocks in the next frame

θθ

A simplified mathematic problem (the following assumptions are approximately correct)

All three lines pass through the circle’s center

End points of the three lines lie on the circle

Line L2

Line L1

Line L3

Rotation Tracking Engine

SolutionAngle between L2 and L3 = θ

• θ = Tan-1(slope(L2)) – Tan-1(slope(L3))

Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix

2

2

1

1

cossin

sincos

L

L

L

L

y

x

y

x

Rotation Tracking EngineSolution

The prediction of the position of the next tracking block should also take the translational movement into accountHorizontal displacement = Tx

• Tx = ( xL21 + xL22

– xL31 - xL32

)/2

Vertical displacement = Ty • Ty = ( yL21

+ yL22– yL31

- yL32

)/2

Coordinates of the next tracking block (xL1, yL1) is calculated by

1111

cossin

sincos

12

2

1

1

L

L

L

L

y

x

Ty

Tx

y

x

Rotation Tracking Engine

A simple drawing to show the detected rotation angle of the phone

Rotation Tracking Engine

Reducing error by using levelApply threshold on output, increase/decrease one level only when change is largeTo give less sensitive but more desirable output for game e.g. skiing game: skier face only to 7 directionsReduce difficulty, increase reliabilityA small rotation or a small detection error will not increase/decrease one level

Conditions to be a good background for both engines

ObjectiveMax. performance measurementIncrease usability

Condition:Feature selection can always find a good feature pointWithin certain distance

• No repeat pattern• Very distinct pattern

Pattern is nearly the same when rotated. E.g. Circle which is good for rotation detection

Virtual Mouse

An application that make full use of the translational motion detection engineRemote control the mouse of PC by Symbian phone using motion trackingAdvantages:

It allows input in all directions It provides high levels of control Joystick can still be used as rough moving while camera input can be used as fine translation

Virtual Mouse

ServerIn server side (Windows), the Bluetooth is configured to provide RFComm Service and server regards the Bluetooth transmission port as Comm. Port Server receives message from that Comm. PortCall function in MouseAction.h to make the mouse move and trigger mouse click event

Virtual Mouse

ClientSearch for Bluetooth device nearbyConnect to the selected Bluetooth deviceEvery time motion tracking algorithm finishes, results are sent to server (12 times/sec)Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)

Car Racing GameA “Car Racing Game” is developed using the motion tracking engine.Game lags. It is because:

The engine takes time to find the motion vectorCPU speed of the Symbian phones are lowThe game engine thread uses most of the CPU power while the thread for updating the screen cannot proceed

Solutions:Double BufferingDirect Screen AccessSingle Thread

Double Buffered AreaDouble-buffering – two complete color buffers

One is displayed while the other is calculating next scene to be drawn,When the drawing of the next frame is completed, the content of the back buffer is copied to the front screen

Computing Frame 2

Show Frame 2

Buffer 1 Buffer 2copy

Frame 1

Computing……

Frame 2

Front Screen

Frame 2 ready

Direct Screen AccessIn Symbian, traditional drawing requires the help of Window Server

Overhead in context switchingLower speed

Access the screen directly in 3 ways:Creating and using CfbsScreenDeviceAccessing screen memory directly.Using CdirectScreenAccess.

Using DSA can bypass the Window ServerGet rid of context switchingFaster

Traditional Graphic Programming

Update Display

Device Screen and Keypad

Window Server

RWsSession RWsSessionRWsSession

Application 1 Application 1 Application 1

Key Presses

Direct Screen Access

2. If successful, a region for drawing to is returned to the application.

Window Server

1. Request to Window Server to work in Direct Screen Access mode

Application

Single Thread

Prevent context switchingNo need to do synchronization

Car Racing Game

With the use of efficient graphic programming algorithms, the tracking engine will not affect the performance of the game

It shows that we can use the engine to develop other applications without performance degradation

Skiing Game

Sample program from the book “Developing Series 60 Applications: A Guide for Symbian Os C++ Developers”Based on this game, we plug in our rotation tracking engine into it

The game becomes more interactiveRotate the phone to control the angle of the skier

Skiing Game

Skiing Game

The process of using the engine is very simple

Create an instance of the tracking engineCall the function of the engine to find the motion vectorUse the motion vector for the game logic

Skiing Game

Demo

Experimental Result on Symbian phone

Testing EnvironmentPlatform

• Symbian Phone Nokia 6600 Algorithm

• Our final (hybrid-type) Algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)

Algorithm parameter• Block size         = 17 x 17 (pixels)• Search window size     = 16 x 16 (pixels)

Number of block to track in each run of algorithm = 1 block ONLY

Experimental Result on Symbian phone

Testing Result (avg time to run)Final algorithm • 7ms

Final algorithm without adaptive window method and SEA, PPNM method • 22ms

Full Exhaustive Search algorithm (the most simplest one) • 55ms

Q & A