3D Shape AttributesDavid F. Fouhey1, Abhinav Gupta1, Andrew Zisserman2

Overview

Sculpture Data

PASCAL VOC Results

Mental Rotation

CMU Robotics Institute1, University of Oxford2

Pla

nar

ity

Most Least

…

…

…

Pt/

Lin

e C

on

t.

…

PlanarSurfaces

Point/Line Contact

ThinStructures

Rough Surfaces

HasHoles

Positives NegativesImage Samples

Base Network

…

Attr.Only

FullModel

or ……

Idea: do two images show the same object but rotated?

Evaluate as classification over ~100M pairs of imagesCosine similarity = classification prediction for featurePositive = same work; Negative = different work

Easy setting: all pairs (ignore same viewpoint cluster)Hard setting: remove “easy” positives via BOW+SIFT

Sculpture Attribute Results

Curvature Contact

Planar !Planar Cylinder Rough Pt/Line Multiple

82.8 77.2 56.9 76.0 74.4 76.4

Occupancy

Empty 2+ Pieces Holes Is Thin Mirr. Sym Cubic Ratio

87.0 60.4 69.3 85.8 60.8 60.3

ZPlanar = YHoles = Y

…2+ Contacts = N

Is Empty = N…

Indirect Baselines

KDES

Eigen ‘14

58.5

HHA

Barron ‘15

59.461.2 62.5 72.3

End-to-end

KDES HHA

…

Thin Structures

Point/Line ContactMost Least

…

Rough Surfaces

…

No Planar Surfaces

…

Qualitative Quantitative (AUROC)

Learning

12D Shape

Attributes

Conv. Layers FC LayersInput

VGG-M

1024D Shape

Embedding

a p

n

Label: {0, 1, invalid}

Prediction [0,1]

Anchor’s feature

Positive’s feature

Negative’s feature

Anchor Positive Negative

Objective: learn features such thattwo views of same work are closer than two views of different works by a margin

Objective: describe 3D shape of generic objects interms of qualitative, higher-order properties

- -

0 +

+ +

0 -

- +

0 0

Historical motivation

Not planarSmooth surface1 point of contactNot point contactHas holeNot thin structures…

Contrast objective with

Category Specific Metric 2.5D Maps (Depth, Normals)

Gathering Data

H. Moore

…

A. Calder

…

Two Forms

Sheep Piece

Locking Piece

5 Swords

Eagle

…

242 Artists 2197 Works

Sculpture list from

…

…

…

143K Images in 9352 Viewpoint Clusters

…

Princeton Columbus Toronto

YorkshireMalagaLondon

Mountains & Clouds

Five Swords

Red Feather

By Alexander Calder

Artist/Shape Correlation

Diverse Materials, Locations

…

Modern Sculpture

Great shape diversity

Categories we can’t describe

Ordinary Objects

Limited shape diversity

Category/shape correlation

Attributes

Curvature Properties

Contact Properties

Volumetric Properties

Cubic Asp. RatioHas Hole Thin Structures Mirror Symm.

Has RoughnessHas Planarity No Planarity Has Cylindrical

Multiple PiecesPoint/line contact Multiple Contacts Mainly Empty

Data source possibilities

We investigate twelve 3D shape attributes inspired by past work in vision

Shape Embedding

Images from

False Positive Rate

Tru

e P

osit

ive

Rat

e

False Positive Rate

Easy Setting (0.9% Positives) Hard Setting (0.3% Positives)

Tru

e P

osit

ive

Rat

e

Evaluation done on 35K images by held-out artists