Algebraic Functions of Views for 3D Object Recognition

CS773C Advanced Machine Intelligence ApplicationsCS773C Advanced Machine Intelligence Applications

Spring 2008: Object RecognitionSpring 2008: Object Recognition

Object Appearance

• The appearance of an object can have a large range of variation due to:– Photometric effects

– Scene clutter

– Changes in shape (e.g., non-rigid objects)

– Viewpoint changes

Algebraic Functions of Views (AFoVs)

• A powerful mathematical foundation for investigating variations in the geometrical appearance of an object due to viewpoint changes.

““the variety of of 2D views depicting the geometrical the variety of of 2D views depicting the geometrical appearance of a 3D object can be expressed as a appearance of a 3D object can be expressed as a combination of a small number of 2D views of the combination of a small number of 2D views of the object”object”

S. Ullman and R. Basri, "Recognition by Linear Combinations of Models", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 10, pp. 992-1006, 1991.

Orthographic Projection

• Case of

3D rigid

transformations

(3 ref. views)

' '' '''1 2 3 4

' '' '''1 2 3 4

x a x a x a x a

y b y b y b y b

Orthographic Projection

• Case of 3D linear transformations

' ' ''1 2 3 4

' ' ''1 2 3 4

x a x a y a x a

y b x b y b x b

(2 ref views)

More Results …

• Perspective projection (2 ref. views, obtained under orthographic projection)

• Objects with smooth surfaces and non-rigid objects– More reference views are required.

'' ' ' '' ' '5 6 7 8 5 6 7 8

'' ' ' '' ' '1 2 3 4 1 2 3 4

a x a x a y a b x b x b y bx y

a x a x a y a b x b x b y b

A. Shashua, “Algebraic functions for recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 8, pp. 779-789, 1995.

A Word of Caution!

• Only common features in the reference views can be predicted in a novel view.

novel view

reference viewreference view

Recognition Framework Using AFoVs

““novel 2D views of a 3D object can be recognized novel 2D views of a 3D object can be recognized by matching them to combinations of a small by matching them to combinations of a small number of known 2D views of the object”number of known 2D views of the object”

Representation and Matching using AFoVs

• Representation– Objects are represented by a small number of views.

– Each view is represented by some geometric features (e.g., points)

• Matching– Predict the geometric appearance of an object in a novel view by

combining a small number of reference views of the object.

Advantages of the Method

• No 3D models or camera calibration are required.

• Only a small number of 2D views are required.

• Novel views can be different from the stored ones.

• Simpler verification scheme.

• More general framework (“family” of methods).

• Evidence that the human visual system works similarly.

Main Challenges

• Which model views to combine to predict a novel view?

• How to establish the correspondences between novel and reference views?

• How to find the coefficients of the combination?.

• How to handle occlusions?

• How to choose the reference views?

Integrate AFoVs with Indexing!

Overview of the Method (cont’d)

Which Model Groups to Choose?

• Cluster geometric features into higher level descriptions.

• Consider properties that are unlikely to occur at random.

• Property used in our work: convexity convexity

Which Model Groups to Choose? (cont’d)

How to generate the appearances of a group?

• Estimate each parameter’s range of values

• Sample the space of parameter values

• Generate a new appearance for each sample of values' ' ''

1 2 3 4

' ' ''1 2 3 4

x a x a y a x a

y b x b y b x b

1 2 3 4( , , , )a a a a 1 2 3 4( , , , )b b b b

min max: [ , ]i i ia a a min max: [ , ]i i ib b b

Estimate the Range of Values of the Parameters

' ' ''1 11 11 1 1

' ' ''2 22 22 2 2

3 3

' ' ''4 4

1

1

... ...... ... ... ...

1 N NN N N

x ya bx y x

x ya bx y x

a b

x ya bx y y

1 xPc p 2 yPc por

1 ( )P

Pi xiP

i ii

u pc v

w

Using SVD:

2 ( )Pi y P

iPi ii

u pc v

wand

and

TP P PP U W V

Estimate the Range of Values of the Parameters (cont’d)

• Assume normalized coordinates:

• Use Interval Arithmetic (Moore, 1966)

(note that the solutions will be identical)

[0,1]xp [0,1]yp

1I I

xPc p 2I I

yPc p

1 2,I Ic c

1 1 2 2

1 1 1 2 2 1 2 2 1 1 1 2 2 1 2 2

[ , ]

* [min( , , , ),max( , , , )]

t r t r t r

t r t r t r t r t r t r t r t r t r

1 2 1 2[ , ] [ , ]t t t r r r

Example

Preconditioning the Reference Views :

• Transform the original views to new views

nPC Psuch that has the best possible condition.nP

effect of the conditionnumber of P on the intervals

Preconditioning the Reference Views (cont’d)

• Choosing:

• This implies:

• Thus:

( )( ) ( )n n nT T T

P P P C C C P P PU W V U W V U W V

C PU V

( ) ( )n n nT T

P P C C P P PU W W V U W V

nP C PW W W

1C PW W

Example(preconditioned views)

Decouple Image Coordinates

• Same transformation generates the x- and y-coordinates:

• Represent only the x-coordinates in the index table.

• For each group, store the following entry:

1 2(mod , , , )el v v group

' ' ''1 2 3 4

' ' ''1 2 3 4

x a x a y a x a

y b x b y b x b

' ' ''1 2 3 4x a x a y a x a

Hypothesis Generation and Verification

1.take intersection of hypotheses

model

2. apply constraintsto reject invalidhypotheses

How to Choose the Scene Groups?

• Using convex groupingconvex grouping to extract salient scene groups.

Implementation Issues

• Space requirements– select salient groups

– reject groups giving rise to bad conditioned matrices

– coarse sampling of parameters

• Index computation and table size

Important Implementation Issues (cont’d)

• Sampling step (i.e., parameters of AFoVs)

• Noise tolerance

' ' ''1 2 3 4i i i ix a x a y a x a

' ' ''1 2 3 4ˆ ˆ ˆ ˆ ˆi i i ix a x a y a x a

actual:

predicted:

* * *| ( ) | | ( ) | | |i j i j i jx e q x q e x q e

make additional entries in a neighborhood around the indexed location

Experiments and Results

model objectsand referenceviews used inour experiments

Experiments and Results (cont’d)

reference views reference views

novel view novel view

Experiments and Results (cont’d)

reference views

novel view novel view

Experiments and Results (cont’d)

reference views reference views

novel viewnovel view

Criticism of the Method

• Relies heavily on feature extraction

• It has high memory requirements.

• The index table might represent unrealistic model appearances.

• Indexing based on hashing is not very efficient.

• No explicit ranking of hypotheses.

Improving AFoVs Recognition Framework

• Reject unrealistic appearances

• Reduce storage requirements and improve speed

• Develop a probabilistic hypothesis generation scheme– Learn shape appearance

– Rank hypotheses

• Represent object appearance more efficiently using improved indexing schemes and probabilistic models.

W. Li, G. Bebis, and N. Bourbakis, "Integrating Algebraic Functions of Views with Indexing and Learning for 3D Object Recognition", IEEE Workshop on Learning in Computer Vision and Patter Recognition (in conjunction with CVPR04), Washington DC, June 28, 2004.

Combine Indexing with Learning

• Sample the space of appearances sparsely and represent the samples in a K-d tree

• Sample the space of views densely and represent the samples using probabilistic models.

• Given a novel view:(1) Use K-d tree to retrieve a small number of candidate models

(2) For each candidate model, compute the probability that it might have produced the novel view

(3) Verify most likely hypotheses first

Combine Indexing with Learning (cont’d)

• The first stage provides hypothetical matches fast.

• The second stage evaluates the feasibility of hypothetical matches fast, without having to apply verification explicitly.

• Only “highly likely” hypotheses are verified explicitly.

Improved Framework

Reference views

Extract model groups

Estimate the range of AFoVs parameters

Sampling AFoVs parameter space

New image

Extract image groups

K-d Tree

Rank hypotheses

Estimate AFoVs parameters

Verify hypotheses

TRAINING PHASE RECOGNITION PHASE

Using SVD & IA

Access

Retrieve

Validate views

Hypothetical matches

Low-dimensional representation

Manifold learning using EM

Recognition results

Random Projection

dense coarsedense

coarse

Eliminate Unrealistic Model Appearances

• Under the assumption of linear transformations, many unrealistic views could be generated.

• Impose rigidity constraints to eliminate them.– Storage requirements can be reduced significantly.

– Recognition becomes faster and more efficient.

0)()( 122332111331332211 rbabarbababababa

0)(2)(2 12323211313123

22

21

23

22

21 raabbraabbbbbaaa

Eliminate Unrealistic Model Appearances

Unrealistic Views (without constraints)

Realistic Views (with constraints)

Indexing Appearances

• Sample the space of views “coarsely” and represent the samples in an index table.

• Hashing might not very well in this case ...

• Need an improved indexing scheme.

Range Search vs Nearest Neighbor Search

Nearest Neighbor Search Range Search

• Range search is not appropriate when storing a sparse number of views.

• K-d trees perform a nearest-neighbor search.

K-d Trees for Indexing

P1

P2

P3

P4

P1 P2 P3 P4

• K-d trees perform a nearest-neighbor search.

Learning Geometric Appearance

• We can pre-compute the views that an object can produce off-line.

• These views form a manifold in lower dimensional space.

• Model object appearance using a pdf.– Sample the space of appearances.– Fit a parametric model (e.g., mixtures of Gaussians using EM).– Use mutual information theory to choose the number of components.

• EM has problems when the dimensionality of the data is high.• Apply “Random Projection” first, then run EM algorithm.

Manifolds of Real Objects: An Example

• Need to store a small number of parameters only for each model

Hypothesis Ranking

)))|((max)|((log(max

))|()|(log()|(

iyiixi

jyjxjo OpOp

OpOpOp

.,,1 Hi where

• Each hypothesis generated by the K-d tree is ranked by computing its probability using mixture models.

• For each test group, we compute two probabilities, one from x coordinates, and the other from y coordinates.

• The overall probability for a particular hypothesis is computed according to the following equation:

Reference Views

1st Reference view

2nd Reference view

Reference Views (cont’d)

1st Reference view

2nd Reference view

Test Views

(a) (b) (c)

(d) (e) (f)

Test Views (cont’d)

Hypothesis rejected Hypothesis rejected

Integrate Geometric Appearance with Intensity Appearance

• Using geometrical information only does not provide enough discrimination for objects having similar “geometric” appearance but probably different “intensity” appearance.

• Integrating geometric and intensity apperance during hypothesis verification to improve discrimination power and robustness.

W. Li, G. Bebis, and N. Bourbakis, "3D Object Recognition Using 2D Views", IEEE Transactions on Image Processing (under revision).

Dense Correspondences

• For each group of corresponding points, apply triangulation recursively to get denser correspondences.

• Divide triangles into four sub-triangles by considering the middle point of each side of each triangle.

Refine AFoVs parameters

(before refinement) (after refinement)

Predict Intensity Appearance - Example

Reference view 1 Reference view 2

Test view Prediction

Predict Intensity Appearance - Example

Reference view 1 Reference view 2

Test view Prediction

Predict Intensity Appearance - Example

(hypothesis rejected)

(hypothesis accepted)