modeling and recognition of landmark image collections using iconic scene graphs

I. Goal: Design an efficient and scalable system for dataset collection, scene summarization, 3D reconstruction, and recognition for landmark photo collections

II. Modeling by combining 2D appearance and 3D geometry1) Appearance-based clustering: k-means with gist descriptors2) Geometric cluster verification and iconic image selection3) Construction of iconic scene graph (nodes: iconic images;

edges: fundamental matrices or homographies; edge weights: inlier numbers)

4) Tag-based rejection of isolated graph nodes5) Computation of iconic scene graph components (graph cut)

III. Applications1) Structure from motion (reconstruct components separately

and merge)2) Summarization and hierarchical browsing3) Recognition (gist or vocabulary tree followed by geometric

verification)

Notre Dame: 8 models with 580 images registered in total; largest model has 337 views and 30802 points

Modeling and Recognition of Landmark Image Collections Using Iconic Scene GraphsXiaowei Li, Changchang Wu, Christopher Zach, Svetlana Lazebnik, and Jan-Michael Frahm

University of North Carolina, Chapel Hill

Hierarchical Scene Browsing

Stage 1: gist clustering; Stage 2: per-cluster geometric verification; Stage 3: per-image geometric verification; Stage 4: tag filtering

Overview

Modeling and Recognition

Iconic Images and 3D Models

DatasetModeling Recognition

Unlabeled Pos. Neg. Pos. Neg.

Statue of Liberty 42983 1369 932 646 446

San Marco 38332 2094 3131 379 715

Notre Dame 9760 545 535 541 503

San Marco: 4 models with 1213 images registered in total; largest model has 749 views and 39307 points

Iconic imagesInitial dataset

Appearance-based clustering and

geometric verification

Pairwise matching of iconic images

Graph cut

Structure from motion

Level 1: iconic scene graph components Level 2: all iconic images inside a given component Level 3: all images in the gist cluster of a given iconic

Browse online at http://www.cs.unc.edu/iconic-scene-graphs

Components of iconic scene graphIconic scene graph

Reconstructed components

Las Vegas

New York

Tokyo

Statue of Liberty: 196 iconic images

(downloaded from Flickr.com via keyword searches)

Statue of Liberty: 6 models with 1068 images registered in total; largest model has 871 views and 18675 points

Level 1 Level 2

Level 3