Semi-Local Affine PartsSemi-Local Affine Partsfor Object Recognitionfor Object Recognition

Svetlana Lazebnik, Jean PonceSvetlana Lazebnik, Jean PonceUniversity of Illinois at Urbana-ChampaignUniversity of Illinois at Urbana-Champaign

Cordelia Schmid Cordelia Schmid INRIA RhINRIA Rhôône-Alpesne-Alpes

BMVC 2004BMVC 2004

OverviewOverview• Goal:

– Learning models for recognition of 3D object classes• Challenges:

– Geometric invariance– Robustness to clutter, occlusion– Weakly supervised learning

• Proposed approach: – An object representation using semi-local affine parts

Low-Level Features: Local Affine RegionsLow-Level Features: Local Affine Regions

• This work: Laplacian detector (Gårding & Lindeberg, 1996)• Other detectors: Kadir et al. (2004), Matas et al. (2002),

Mikolajczyk & Schmid (2002), Tuytelaars & Van Gool (2004), etc.

• In practice: two-image matching followed by validation

Learning PartsLearning Parts• Ideal approach: simultaneous correspondence search

across entire training set

validation setinitial pair

candidate part

Two-Image MatchingTwo-Image Matching• Goal: to find collections of local affine regions that can be

mapped onto each other using a single affine transformation

• Implementation: greedy search based on geometric and photometric consistency constraints– Returns multiple correspondence hypotheses

– Automatically determines number of regions in correspondence

– Works on unsegmented, cluttered images (weakly supervised learning)

A

Matching: DetailsMatching: Details• Initialization:

– Identify triples of neighboring regions (i, j, k) in first image– Find all triples (i', j', k') in the second image such that i'

(resp. j', k') is a potential match of i (resp. j, k), and j', k' are neighbors of i'

i

j

k

i'

j'

k'

Matching (cont.)Matching (cont.)• Beginning with each seed triple, iterate:

– Estimate the affine transformation between centers of corresponding regions in current group of matches

A

Matching (cont.)Matching (cont.)• Beginning with each seed triple, iterate:

– Estimate the affine transformation between centers of corresponding regions in current group of matches

– Determine geometric consistency of current group of matches

• Geometric consistency criteria:– Distance between ellipse centers

(residual)– Difference of major and minor axis

lengths– Difference of ellipse orientations

Matching (cont.)Matching (cont.)• Beginning with each seed triple, iterate:

– Estimate the affine transformation between centers of corresponding regions in current group of matches

– Determine geometric consistency of current group of matches

– Search for additional matches in the neighborhood of the current group

Matching: 3D ObjectsMatching: 3D Objects

Matching: 3D ObjectsMatching: 3D Objects

closeup closeup

Matching: FacesMatching: Faces

spurious match ???

Finding Repeated Patterns and Finding Repeated Patterns and SymmetriesSymmetries

Learning Object Models for RecognitionLearning Object Models for Recognition• Match multiple pairs of training images to produce a

set of candidate parts• Use additional validation images to evaluate

repeatability of parts and individual regions • Retain a fixed number of parts having the best

repeatability score

Recognition Experiment: ButterfliesRecognition Experiment: Butterflies

• 26 training images per class– 8 initial pairs– 10 validation images

• 437 test images• 619 images total

Admiral Swallowtail Machaon Monarch 1 Monarch 2 Peacock Zebra

Butterfly PartsButterfly Parts

RecognitionRecognition

• Top 10 parts per class used for recognition• Relative repeatability score:• Classification results:

total number of regions detectedtotal part size

Total part size (smallest/largest)

Classification Rate vs. Classification Rate vs. Number of PartsNumber of Parts

Detection Results (ROC Curves)Detection Results (ROC Curves)

Circles: reference relative repeatability rates. Red square: ROC equal error rate (in parentheses)

Successful Detection ExamplesSuccessful Detection ExamplesTraining images

Test images (blue: occluded regions)

All regions found in the test images

Unsuccessful Detection ExamplesUnsuccessful Detection ExamplesTraining images

Test images (blue: occluded regions)

All regions found in the test images

Future WorkFuture Work• Goal:

– Recognize highly variable, non-rigid object categories

• Proposed approach: – Treat semi-local affine parts as “black boxes”– Model spatial relations between parts– Learn these relations from training data in a weakly

supervised fashion