Model-Based Stereo with Occlusions

Fabiano Romeiro and Todd Zickler

Introduction

Varying illumination

Varying pose

Occlusions

Varying expressions

Introduction

Eigenfaces [Turk and Pentland, 1991]

[Belhumeur et al, 1997]Fisherfaces

Past Work: Image-based

For example:

3D Morphable Models (3DMMs)

[Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006]

Introduction

2D AAMs

[Cootes et al, 1998; Baker et al, 2004; Mathews et al, 2004; Gross et al, 2006]

2D+3D AAMs

[Xiao et al, 2004]

Past Work: Model-based

Introduction

Pros

- Self-occlusion handled by model itself-- Allows direct modeling of illumination

Cons

- Difficult and expensive fitting process

Past Work: 3DMMs3D Morphable Models (3DMMs)

[Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006]

Introduction

- Texture model not needed

Past Work: Stereo 3DMMs

Our work[Fransens et al, 2005]

- Stereo based cost

d(I 1; I 2)

-Stereo fitting with both shape and texture

→ Improved Accuracy

→ Robust to foreign-body occlusions

Outline

• 3DMM Background• Joint Shape and Texture Stereo Fitting• Handling Occlusions• Conclusions

Background

3DMMs

Vectorization of laser scans:

PCA performed:

F Si = [[X i1Y

i1Z i

1]T :::[X i

N Y iN Z i

N ]T ]T

- Basis for shape fSi gi=1;::;m

F Ti = [[R i1G

i1B

i1]

T :::[R iN Gi

N B iN ]T ]T

f Ti gi=1;::;m- Basis for texture[Blanz and Vetter, 1999]

Representation of face shape and texture:

3DMMs

Background

S = S0 +mX

i=1

®i Si

Prior probabilities on the coefficients:

P (~®) / exp(¡12

mX

i=1

(®i

¾i)2)

P (~̄) / exp(¡12

mX

i=1

(¯ i

°i)2) [Blanz and Vetter, 1999]

T = T0 +mX

i=1

¯ i Ti

Stereo Match

I 1(P1sk) ¼I 2(P2sk)

Shape (®)Pose (R;t)

sk

P1 P2

I 1 I 2

Texture Match

P1 P2

I m(k) ¼ ¹I (sk) = I 1(P1sk )+I 2(P2sk )2

I m(k) Texture (¯)Shape (®)

Lighting (K o®set, K amb, K dir)K =fR,G,Bg

skShape (®)Pose (R;t)

I 1 I 2

Joint Shape and Texture Stereo Fitting of 3DMMs

E =

P (Shape;Texture;P ose;L ightingjI 1; I 2) =

= P (Shape;P osejI 1; I 2)¢

X

kjvk 2V

jjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

| {z }Stereo Match

+mX

i=1

(®i

¾i)2

| {z }Shape Prior

+

+X

kjvk 2V

jjI m(k) ¡ ¹I (sk)jj2

¾2t

| {z }Texture Match

+mX

i=1

(¯ i

°i)2

| {z }TexturePrior

P (Texture;L ightingjI 1; I 2;Shape;P ose)

X

kjvk 2V

ha

µjjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

+jjI m(k) ¡ ¹I (sk)jj2

¾2t

¶X

kjvk 2V

jjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

| {z }Stereo Match

Binary occlusion map O : f1;::;N g ! f0;1gN

Robust Stereo Fitting of 3DMMs

E = +mX

i=1

(®i

¾i)2

| {z }Shape Prior

+

+X

kjvk 2V

jjI m(k) ¡ ¹I (sk)jj2

¾2t

| {z }TextureMatch

+mX

i=1

(¯ i

°i)2

| {z }Texture Prior

+jjI m(k) ¡ ¹I (sk)jj2

¾2t

+

P (Shape;Texture;P ose;L ighting;OjI 1; I 2) =

= P (Shape;P ose;OjI 1; I 2)¢P (Texture;L ightingjI 1; I 2;Shape;P ose;O)

Eo =X

kjvk 2V

ha

µjjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

+jjI m(k) ¡ ¹I (sk)jj2

¾2t

¶

+mX

i=1

(®i

¾i)2 +

mX

i=1

(¯ i

°i)2

Optimization Procedure

Initial fitE f- Fit Shape, Pose to minimize reprojection error

on selected feature points

- Rough initial estimates of Shape and Pose

Optimization procedure

Eo + ¸ ¢E f

Eo

4 experiments

Stereo and texture

Eo Stereo

Eo Robust Stereo and texture

Eo Robust Stereo

Eo =X

kjvk 2V

jjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

| {z }Stereo Match

+mX

i=1

(®i

¾i)2

| {z }ShapePrior

+

X

kjvk 2V

jjI m(k) ¡ ¹I (sk)jj2

¾2t

| {z }TextureMatch

+mX

i=1

(¯ i

°i)2

| {z }TexturePrior

Eo =X

kjvk 2V

ha

µjjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

¶

+mX

i=1

(®i

¾i)2

Eo =X

kjvk 2V

jjI 1(P1sk) ¡ I 2(P2sk)jj2

¾2s

| {z }Stereo Match

+mX

i=1

(®i

¾i)2

| {z }Shape Prior

Results

480 recovered shape models (60 individuals, 8 poses)

K.U. Leuven Stereo face database

First 2 experiments: Stereo and Texture vs. Stereo

[Fransens et al, 2005]

ResultsQualitative Results

Stereo

Stereo and Texture

Stereo Matching CostStereo 280:77Stereo+Texture 340:17

Results

Stereo

Stereo+texture

Qualitative Results

Results

Stereo Stereo+Texturetr(S¡ 1

w Sb) 69:4101 104:0478det(S¡ 1

w Sb) 1:3418e¡ 11 2:9640e¡ 5

Quantitative Results

Results

Occluder classi¯cation Half Full near Full fartextureless (non-skincolor) Stereo Texture Texturetextured Stereo Texture Stereo+Texturetextureless (skincolor) X X X

Half-Occlusion Full Occlusion near Full-Occlusion far

Under Occlusions

Results

Occluder classi¯cation Half Full near Full fartextureless (non-skincolor) Stereo Texture Texturetextured Stereo Texture Stereo+Texturetextureless (skincolor) X X X

Under Occlusions

Input

Shape Estimate

Occlusion Map

Robust Stereo Robust S+T Robust Stereo Robust S+T

Conclusions

Robust stereo fitting of 3DMMs

- Uses both stereo constraint, texture information

- Increased accuracy of fit

- Ability to handle occlusions

Future Work

- More sophisticated stereo matching term

- Different feature spaces

- Break model into segments respecting occlusion boundaries