1

Non

rigid

Surf

ace

Mod

ellin

gan

d Fa

st R

ecov

ery

Zhu

Jian

ke

Sup

ervi

sor:

Pro

f. M

icha

el R

. Lyu

Com

mitt

ee: P

rof.

Leo

J. J

iaan

d P

rof.

K. H

. Won

g

Dep

artm

ent o

f Com

pute

r Sci

ence

and

Eng

inee

ring

The

Chi

nese

Uni

vers

ity o

f Hon

g K

ong

May

11,

200

7

2D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

3D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Out

line

•In

trodu

ctio

n

•Tw

o-st

age

Sch

eme

with

Act

ive

App

eara

nce

Mod

els

for n

onrig

id s

urfa

ce re

cove

ry

•P

rogr

essi

ve F

inite

New

ton

App

roac

h to

non

rigid

su

rface

det

ectio

n

•C

oncl

usio

n

4D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Intr

oduc

tion

•N

onrig

idsu

rface

mod

ellin

gan

d de

tect

ion

are

esse

ntia

lly th

e co

mpu

ter v

isio

n ta

sks

in a

var

iety

of a

pplic

atio

ns:

–im

age

alig

nmen

t–

med

ial i

mag

ing

–au

gmen

ted

real

ity–

hum

an c

ompu

ter i

nter

actio

n –

digi

tal e

nter

tain

men

t

5D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Non

rigid

Surf

ace

Mod

ellin

g

•D

ata

embe

ddin

g: P

rinci

pal C

ompo

nent

Ana

lysi

s–

Act

ive

App

eara

nce

Mod

els

–A

ctiv

e S

hape

Mod

els

–3D

Mor

phab

leM

odel

s•

Phy

sica

l mod

el: F

inite

Ele

men

t Mod

el (F

EM

)–

Act

ive

Con

tour

s (S

nake

s)•

Thin

-Pla

te S

plin

e–

TPS

-RP

M–

Sha

pe C

onte

xt–

Gau

ssia

n M

ixtu

re M

odel

s ba

sed

poin

ts s

et re

gist

ratio

n

6D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Non

rigid

Surf

ace

Rec

over

y

•Ap

pear

ance

-bas

ed M

etho

d–

Als

o kn

own

as d

irect

met

hod

–Lu

cas-

Kan

ade

algo

rithm

bas

ed m

etho

ds [L

K IJ

CA

I’81]

–A

ctiv

e A

ppea

ranc

e M

odel

s [C

oote

sP

AM

I’01]

–3D

Mor

phab

leM

odel

s [V

ette

r Sig

grap

h’99

, PA

MI’0

1]–

2D+3

D A

AM

s[X

iao

CV

PR

’04]

•Fe

atur

e-ba

sed

Met

hod

–S

hape

con

text

[Bel

ongi

eP

AM

I’02]

–TP

S-R

PM

[Hui

CV

IU’0

3]–

Ker

nel c

orre

latio

n [E

CC

V’04

]–

Gau

ssia

n m

ixtu

re m

odel

s [J

ian

ICC

V’0

5]–

Sem

i-im

plic

it op

timiz

atio

n sc

hem

e [P

iletC

VP

R’0

5, IJ

CV

’07]

7D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Par

t I

A T

wo-

stag

e S

chem

e fo

r Non

rigid

Sur

face

R

ecov

ery

with

Act

ive

App

eara

nce

Mod

els

8D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Part

I: S

truc

ture

•M

otiv

atio

n

•Ex

tend

ed A

ctiv

e A

ppea

ranc

e M

odel

s (A

AM

s) fi

tting

•Tw

o st

age

sche

me

for n

onrig

id s

urfa

ce re

cove

ry–

Offl

ine

cons

truct

ion

of 3

D s

hape

mod

el–

Estim

ate

3D p

ose

and

non-

rigid

sha

pe p

aram

eter

s

•Ex

perie

men

tal R

esul

ts

•S

umm

ary

of P

art I

9D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Mot

ivat

ion

•N

onrig

id s

hape

reco

very

for A

ugm

ente

d R

ealit

y–

Rig

id O

bjec

t •

L. V

acch

etti

et a

l. (P

AM

I‘04)

pro

pose

d an

effi

cien

t sol

utio

n fo

r 3D

rig

id o

bjec

t tra

ckin

g•

Two

2D A

AM

s ap

proa

ch fo

r rig

id o

bjec

t pos

e es

timat

ion

–N

on-r

igid

Obj

ect

•V

.Bla

ndz:

3D

Mor

phab

le M

odel

s•

J.A

hlbe

rg: 3

D A

AM

with

gen

eric

Mod

el.

•Ji

ng X

. (C

VP

R‘0

5) 2

D+3

D A

AM

10D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Act

ive

App

eara

nce

Mod

els

•A

AM

sis

def

ined

by

its s

hape

and

text

ure

•Th

e 2D

sha

pe a

nd te

xtur

e ar

e co

ntro

lled

by a

sta

tistic

al m

odel

. The

y ca

n be

re

pres

ente

d as

a b

ase

plus

a li

near

com

bina

tion

of v

aria

tions

:

•D

irect

App

eara

nce

Mod

els

11D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Exte

nded

AA

Ms

Fitti

ng A

lgor

ithm

•O

bjec

t: ||

I –Im

||•

Idea

: the

pro

pose

d ap

proa

ch p

redi

cts

shap

e di

rect

ly fr

om te

xtur

e

whe

re r

= gi

-gm

, is

the

resi

dual

imag

e

12D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Exte

nded

AA

M F

ittin

g

13D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

AA

M F

ittin

g Sa

mpl

e

The

AA

Ms

are

built

up

with

140

stil

l fac

e im

age

belo

ngin

g to

20

indi

vidu

als,

7 im

ages

for e

ach.

The

fitti

ng e

xper

imen

t is

perfo

rmed

on

an

AA

M w

ith 1

4 sh

ape

para

met

ers,

68

text

ure

para

met

ers,

and

363

35 c

olor

pi

xels

.

14D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Two-

stag

e Sc

hem

e N

onrig

idSu

rfac

e R

ecov

ery

•Tr

aini

ng a

nd b

uild

ing

offli

ne b

asis

Acq

uire

the

2D s

hape

of o

bjec

ts u

sing

the

AA

M fi

tting

alg

orith

m,t

hen

cons

truct

the

3D s

hape

bas

is.

•O

nlin

e tra

ckin

g–

Ste

p 1.

2D

AA

M fi

tting

and

trac

king

–S

tep

2. e

stim

ate

the

3D p

ose

and

shap

e pa

ram

eter

s si

mul

tane

ousl

yvi

a lo

cal b

undl

e ad

just

men

t by

build

ing

up th

e po

int c

orre

spon

denc

es

betw

een

2D a

nd 3

D.

15D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Bui

ldin

g O

fflin

e 3D

Mod

el

•3D

sha

pe

•W

eak-

pers

pect

ive

proj

ectio

n

16D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Bui

ldin

g O

fflin

e 3D

Mod

el

•Fa

ctor

izat

ion

met

hod

[Bre

gler

CV

PR

’00,

CV

PR

’01]

17D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Onl

ine

Alg

orith

m

•Th

e op

timiz

atio

n pr

oble

m c

an b

e de

rived

as:

whe

re

deno

tes

the

proj

ectio

n of

3D

sha

pe g

iven

the

para

met

ers

A, R

and

T.

18D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Expe

riem

enta

l Res

ults

I

Pen

tium

III 1

GH

z C

PU

, 20

0ms

per i

mag

e of

siz

e 35

2x28

8. A

AM

fitti

ng ta

kes

40m

s an

d 3D

re

cove

ry s

tep

take

s 74

ms.

The

AA

M w

ith 1

0 sh

ape

para

met

ers,

52

text

ure

para

met

ers.

6

cam

era

para

met

ers

and

6 3D

sha

pe p

aram

eter

s.

19D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Expe

rimen

tal R

esul

ts II

•D

eter

min

e th

e nu

mbe

r of 3

D s

hape

bas

is

20D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Aut

omat

ic F

ace

Mod

ellin

gSc

hem

e

21D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Ren

derin

g in

Diff

eren

t Vie

w

22D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Re-

text

urin

g fa

ce

23D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Dis

cuss

ion

•R

igid

vs.

Non

-Rig

id–

The

prio

r mod

el e

mpl

oyed

by

L. V

acch

etti

et a

l. is

onl

y fo

r rig

id o

bjec

ts o

r def

orm

able

ob

ject

s w

ith s

mal

l var

iatio

ns.

–P

. Mitt

rapi

yanu

mic

et a

l. do

not

take

full

adva

ntag

e of

AA

M’s

defo

rmat

ion

pow

er•

Offl

ine

vs. O

nlin

e–

In c

ontra

st to

the

offli

ne a

lgor

ithm

s, o

ur p

ropo

sed

met

hod

is a

ble

to w

ork

onlin

e by

ex

ploi

ting

the

3D s

hape

mod

els

that

can

be

cons

truct

ed o

fflin

e ef

fect

ivel

y by

AA

M

track

ing

•A

dvan

tage

s fo

r AR

app

licat

ions

–G

ener

ic v

s. p

erso

n sp

ecifi

c: h

andl

e la

rge

text

ure

varia

tions

, fitt

ing

to d

iffer

ent

indi

vidu

als

–C

ombi

ned

2D+3

D A

AM

: wea

k-pe

rspe

ctiv

e m

odel

•D

isad

vant

ages

and

Fut

ure

Wor

k–

does

not

take

full

adva

ntag

e of

3D

info

rmat

ion

for s

peed

ing

up A

AM

con

verg

ence

.–

Larg

e ro

tatio

n m

ay b

e co

mpe

nsat

ed b

y th

e 3D

line

ar m

ode,

ther

efor

e, th

e es

timat

ed

pose

is n

ot s

o ac

cura

te.

–Tr

aini

ng th

e 3D

AA

M w

ith th

e al

igne

d 3D

sha

pes

inst

ead

of 2

D s

hape

s.

24D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Sum

mar

y of

Par

t I

•A

nov

el tw

o-st

age

sche

me

for o

nlin

e no

n-rig

id s

hape

reco

very

tow

ard

Aug

men

ted

Rea

lity

appl

icat

ions

usi

ng A

AM

s.

•O

btai

n un

brok

en p

oint

cor

resp

onde

nces

acr

oss

mul

tiple

fram

es to

co

nstru

ct 3

D s

hape

mod

els

•P

rovi

de 2

D to

3D

ver

tex

corr

espo

nden

ces

in th

e on

line

track

ing.

•A

n ef

ficie

nt a

lgor

ithm

is p

ropo

sed

to e

stim

ate

both

3D

pos

e an

d no

n-rig

id s

hape

par

amet

ers

via

loca

l bun

dle

adju

stm

ent.

25D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Par

t II

Pro

gres

sive

Fin

ite N

ewto

n A

ppro

ach

to

Non

rigid

Sur

face

Det

ectio

n

26D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Part

II: S

truc

ture

•M

otiv

atio

n•

Pro

gres

sive

Fin

ite N

ewto

n A

ppro

ach

to n

onrig

id s

urfa

ce d

etec

tion

–2D

non

rigid

sur

face

mod

el–

Feat

ure-

base

d no

nrig

id s

urfa

ce re

cove

ry–

Fini

te N

ewto

n fo

rmul

atio

n–

Opt

imiz

atio

n•

Expe

riem

enta

l Res

ults

–C

ompu

tatio

nal e

ffici

ency

and

non

rigid

sur

face

det

ectio

n–

Aug

men

ted

real

ity–

Med

ical

imag

e re

gist

ratio

n•

Sum

mar

y of

Par

t II

27D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Mot

ivat

ion

•N

onrig

idsu

rfac

e de

tect

ion:

rec

over

ing

the

expl

icit

surfa

ce w

ith a

few

de

form

atio

n pa

ram

eter

s an

d fin

ding

out

the

corr

ect c

orre

spon

denc

es

from

noi

sy d

ata

sim

ulta

neou

sly.

•U

nlik

e th

e rig

id o

bjec

t, it

is d

iffic

ult t

o di

rect

ly e

mpl

oy a

robu

st e

stim

ator

to

rem

ove

the

spur

ious

mat

ches

for n

onrig

idsu

rface

det

ectio

n.

•Th

e ite

rativ

e m

etho

ds, s

uch

as T

PS

-RP

M [C

VIU

’03]

and

Sha

pe C

onte

x[P

AM

I’02]

, are

eith

er s

ensi

tive

to in

itial

con

ditio

ns a

nd p

aram

eter

cho

ices

, or

invo

lve

too

man

y ite

ratio

ns a

nd a

com

plex

opt

imiz

atio

n pr

oced

ure.

•S

emi-i

mpl

icit

met

hod:

an

auto

mat

ed a

ppro

ach,

and

can

be

appl

ied

for t

he

real

-tim

e A

ugm

ente

d R

ealit

y. [

Pile

tet a

l CV

PR

’05,

IJC

V’0

7]

28D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Rob

ust M

etho

ds

•R

AN

SA

C

•P

RO

SA

C

•M

-est

imat

or

•H

ough

tran

sfor

m

29D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Whe

re

a

re th

e ba

ryce

ntric

coor

dina

tes

for t

he p

oint

m.

2D N

onrig

idSu

rfac

e M

odel

30D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Non

rigid

Surf

ace

Rec

over

y

•E

nerg

y fu

nctio

n

–Ee

(s) i

s th

e su

m o

f the

wei

ghte

d sq

uare

err

or re

sidu

als

for t

he m

atch

ed p

oint

s.

–E

r(s)

is th

e re

gula

rizat

ion

term

that

repr

esen

ts th

e su

rface

def

orm

atio

n en

ergy

–is

a re

gula

rizat

ion

coef

ficie

nt.

31D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Reg

ular

izat

ion

Term

•E

r(s)

, als

o kn

own

as in

tern

al fo

rce

in S

nake

s, is

co

mpo

sed

of th

e su

m o

f the

squ

ared

sec

ond-

orde

r de

rivat

ives

of t

he m

esh

verte

x co

ordi

nate

s.

whe

re K

is a

spa

rse

and

band

ed m

atrix

whi

ch is

de

term

ined

by

the

stru

ctur

e of

the

mes

h m

odel

.

32D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Rob

ust E

stim

ator

•D

efin

e th

e re

sidu

al e

rror

•Th

e ro

bust

est

imat

or

M1

cont

ains

the

inlie

rmat

ches

M2

is th

e se

t of t

he o

utlie

rsTh

e or

der n

det

erm

ines

the

scal

e of

the

resi

dual

.

33D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Fini

te N

ewto

n Fo

rmul

atio

n

•Th

e ro

bust

est

imat

or fu

nctio

n is

not

con

vex

•Th

e as

soci

ated

pen

alty

func

tion

appr

oxim

atio

n pr

oble

m

beco

mes

a h

ard

com

bina

tiona

l opt

imiz

atio

n pr

oble

m

•Ta

ckle

this

pro

blem

und

er th

e fin

ite N

ewto

n op

timiz

atio

n fra

mew

ork

34D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Fini

te N

ewto

n Fo

rmul

atio

n

•An

aug

men

ted

vect

or:

•Th

en,

whe

re (u

, v) a

re th

e co

ordi

nate

s of

m1.

Re-

writ

e th

e er

ror t

erm

as

whe

re q

is th

e nu

mbe

r of o

utlie

rs

35D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Fini

te N

ewto

n Fo

rmul

atio

n

•D

efin

e ve

ctor

b a

s:

•M

atrix

A (N

by

N):

•E

nerg

y fu

nctio

n

36D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Fini

te N

ewto

n Fo

rmul

atio

n

•Th

e fin

ite g

radi

ent o

f the

ene

rgy

func

tion

E w

ith

resp

ect t

o s

•H

essi

an

•R

ewrit

e gr

adie

nt

37D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Fini

te N

ewto

n Fo

rmul

atio

n

•U

pdat

e eq

uatio

n:•

Set

r =

1, a

nd o

btai

n th

e lin

ear e

quat

ion

38D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Opt

imiz

atio

n

•C

oars

e-to

-fine

sch

eme

to d

eal w

ith la

rge

outli

ers

–Su

ppor

t

of ro

bust

est

imat

or is

pro

gres

sive

ly d

ecay

ed

at a

con

stan

t rat

e–

is k

ept c

onst

ant

–R

esul

t is

used

as

the

initi

al s

tate

for n

ext m

inim

izat

ion

–S

top

whe

n

is

clo

se to

the

expe

cted

pre

cisi

on–

Rep

ort a

suc

cess

ful d

etec

tion

whe

n th

e nu

mbe

r of i

nlie

rm

atch

es is

abo

ve a

giv

en th

resh

old.

39D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Opt

imiz

atio

n

•S

tart

from

a s

uffic

ient

ly la

rge

supp

ort

–To

avo

id g

ettin

g st

uck

at lo

cal m

inim

a–

Nee

ds a

few

iter

atio

ns to

com

pens

ate

for t

he e

rror

s du

e to

pos

e va

riatio

ns•

Mod

ified

RA

NS

AC

–C

lose

d-fo

rm s

olut

ion

–D

raw

from

pro

gres

sive

ly la

rger

set

s of

top-

rank

ed

corre

spon

denc

es–

Sam

ple

size

40D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Expe

rimen

tal S

etup

–In

ord

er to

regi

ster

the

mes

h m

odel

con

veni

ently

, a m

odel

im

age

is a

cqui

red

whe

n th

e no

nrig

idsu

rface

con

tain

s no

de

form

atio

n.–

A ra

ndom

-tree

s ba

sed

met

hod

[Lep

etit

PA

MI’0

6] is

use

d to

bu

ild th

e co

rres

pond

ence

s be

twee

n th

e m

odel

imag

e an

d in

put

imag

e.–

A s

et o

f syn

thet

ic d

ata

is u

sed

to s

elec

t the

par

amet

ers

–Th

e be

st re

gula

rizat

ion

coef

ficie

nt is

foun

d by

grid

sea

rchi

ng.

–n

= 4,

dec

ay ra

te is

0.5

–P

entiu

m-4

3.0

GH

z P

C w

ith 1

GB

RA

M–

A D

V c

amer

a si

ze o

f 720

x576

41D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Com

puta

tiona

l Effi

cien

cy

•Th

e co

mpl

exity

of t

he p

ropo

sed

met

hod

is m

ainl

y de

term

ined

by

the

num

ber o

f ver

tices

N•

Ano

ther

fact

or is

the

num

ber o

f inl

ierm

atch

es•

Pro

pose

d m

etho

d: 8

iter

atio

ns, a

roun

d 18

fram

es p

er s

econ

d•

Sem

i-im

plic

it m

etho

d [C

VP

R’0

5, IJ

CV

’07]

: 40

itera

tions

, ab

out 9

fram

es p

er s

econ

d on

Cof

fee

mat

vid

eo

42D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Cof

fee

mat

[Vid

eo d

emo]

43D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Re-

text

urin

g a

T-sh

irt [V

ideo

dem

o]

44D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Pape

r [Vi

deo

dem

o]

45D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Med

ical

imag

e

46D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Sum

mar

y

•A

nov

el p

rogr

essi

ve s

chem

e to

sol

ve th

e no

n-rig

id s

urfa

ce

dete

ctio

n pr

oble

m•

In c

ontra

st to

the

prev

ious

sem

i-exp

licit

met

hod,

we

dire

ctly

sol

ve

the

unco

nstra

ined

qua

drat

ic o

ptim

izat

ion

prob

lem

by

an e

ffici

ent

fact

oriz

atio

n •

Mod

ified

RA

NS

AC

sch

eme

can

hand

le h

igh-

dim

ensi

onal

spa

ces

with

noi

sy d

ata.

•W

e ha

ve c

ondu

cted

ext

ensi

ve e

xper

imen

tal e

valu

atio

ns o

n di

vers

e ob

ject

s w

ith d

iffer

ent m

ater

ials

.•

The

prop

osed

met

hod

is v

ery

fast

and

robu

st, a

nd c

an h

andl

e la

rge

defo

rmat

ions

and

illu

min

atio

n ch

ange

s.

47D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Con

clus

ion

•Tw

o di

ffere

nt a

ppro

ache

s to

non

rigid

surfa

ce re

cove

ry h

ave

been

inve

stig

ated

•A

ppea

ranc

e-ba

sed

met

hod:

–U

sing

mor

e in

form

atio

n, m

ay b

e m

ore

accu

rate

–O

fferin

g im

age

codi

ng c

apab

ility

–Te

nd to

be

com

puta

tiona

l exp

ensi

ve–

Eas

y to

stu

ck a

t the

loca

l opt

ima

•Fe

atur

e-ba

sed

met

hod

–A

utom

atic

sol

utio

n–

Ess

entia

lly fa

st–

Taki

ng a

dvan

tage

of t

he a

dvan

ces

in fe

atur

e m

atch

ing

and

obje

ct

reco

gniti

on–

Err

ors

occu

r in

the

regi

on la

ckin

g te

xtur

e

48D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Futu

re w

ork

•Fu

sion

app

roac

h

•E

xplo

re n

ew re

gula

rizat

ion

met

hod

•Fi

nd fa

st a

nd a

ccur

ate

feat

ure

mat

chin

g al

gorit

hm

•In

clud

ing

mor

e fe

atur

es, s

uch

as e

dges

and

si

lhou

ette

49D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

The

End

Tha

nk y

ou

50D

epar

tmen

t of C

ompu

ter S

cien

ce a

nd E

ngin

eerin

g,

Facu

lty o

f Eng

inee

ring,

Th

e C

hine

se U

nive

rsity

of H

ong

Kong

Ref

eren

ces

•T.

Coo

tes,

G. E

dwar

ds, a

nd C

. Tay

lo. A

ctiv

e ap

pear

ance

mod

els.

IEE

E T

rans

. on

Pat

tern

A

naly

sis

and

Mac

hine

Inte

llige

nce,

23(

6), J

une

2001

.•

J. X

iao,

S. B

aker

, I. M

atth

ews,

and

T. K

anad

e. R

eal-t

ime

com

bine

d 2d

+3d

activ

e ap

pear

ance

mod

els.

In P

roc.

Con

f. C

ompu

ter V

isio

n an

d P

atte

rn R

ecog

nitio

n, v

olum

e 2,

pa

ges

535–

542,

200

4.•

L. V

acch

etti,

V. L

epet

it, a

nd P

. Fua

. Sta

ble

real

-tim

e 3d

trac

king

usi

ng o

nlin

e an

d of

fline

in

form

atio

n. IE

EE

Tra

ns. O

n P

atte

rn A

naly

sis

and

Mac

hine

Inte

llige

nce,

26(

6), O

ct. 2

004.

•J.

Pile

t, V

. Lep

etit,

and

P. F

ua. F

ast n

on-r

igid

sur

face

det

ectio

n, re

gist

ratio

n, a

nd re

alis

tic

augm

enta

tion.

Int’l

J. C

ompu

ter V

isio

n, 2

007.

•J.

Zhu

, S. C

. Hoi

, E. Y

au, a

nd M

. R. L

yu. A

utom

atic

3d

face

mod

elin

g us

ing

2d a

ctiv

e ap

pear

ance

mod

els.

In P

roc.

13th

Pac

ific

Con

f. C

ompu

ter G

raph

ics

and

App

licat

ions

, pag

es

133–

135,

200

5.•

J. Z

hu, S

. C. H

oi, a

nd M

. R. L

yu. R

eal-t

ime

non-

rigid

sha

pe re

cove

ry v

ia a

ctiv

e ap

pear

ance

m

odel

s fo

r aug

men

ted

real

ity. I

n P

roc.

Eur

opea

n C

onf.

Com

pute

r Vis

ion,

pag

es 1

86–1

97,

2006

.•

J. Z

hu a

nd M

. R. L

yu. P

rogr

essi

ve fi

nite

new

ton

appr

oach

to re

al-ti

me

nonr

igid

surfa

ce

dete

ctio

n. In

Pro

c. C

onf.

Com

pute

r Vis

ion

and

Pat

tern

Rec

ogni

tion,

200

7.•

J. Z

hu, S

. C. H

oi, a

nd M

. R. L

yu. A

mul

ti-sc

ale

tikho

nov

regu

lariz

atio

n sc

hem

e fo

r im

plic

it su

rface

mod

ellin

g. In

Pro

c. C

onf.

Com

pute

r Vis

ion

and

Pat

tern

Rec

ogni

tion,

200

7.