course: structure-aware shape processing hao (richard) zhang simon fraser university (sfu), canada...
TRANSCRIPT
Course: Structure-Aware Shape Processing
Hao (Richard) Zhang Simon Fraser University (SFU), Canada
Structural Hierarchies
Course: Structure-Aware Shape Processing
Course: Structure-Aware Shape Processing
•Efficiency and flexibility: hierarchy offers *both*
high-level abstraction and granularity, adaptable
as desired
•Natural choice: human perception of structures
*is* hierarchical [Palmer 1977, Hoffman & Richards 1984]
•Hierarchies are the right model to capture
diversities in a set: coarse levels common; fine
details vary
Why hierarchy?
Course: Structure-Aware Shape Processing
•Image pyramids
Familiar examples
•Hierarchical
segmentation
Course: Structure-Aware Shape Processing
•What is the right structural hierarchy?
•Challenges:
•Many possible hierarchies for a given shape: larger search
space than (flat) shape segmentation
•Many possible criteria for “best” structural hierarchy
•Even human observers can disagree on what is ground truth
Key question
…
Course: Structure-Aware Shape Processing
•… not just any structural hierarchy
•… but one that best explains one or more
structures
•Exploit power of symmetry for individual analysis [Martinet 2007, Simari 2006, Wang 2011, Zhang 2013]
•Exploit power of a set for co-hierarchical analysis
[van Kaick 2013]
We are interested in …
Course: Structure-Aware Shape Processing
Two basic solution paradigms:
•Top-down
•Recursively split or divide an input shape
•Bottom-up
•Start with a fine segmentation into primitive parts
•Then recursively group parts, e.g., via graph
contraction
Key ingredient: how to prioritize the operations
Hierarchy construction
Course: Structure-Aware Shape Processing
•Symmetric parts tend to perform the same
function
Why symmetry?
Course: Structure-Aware Shape Processing
•Symmetry helps explain structures
•Symmetry leads to redundancy
•Discovering and removing redundancy leads to
compactness
•A compact representation tends to be the right one
•Occam’s Razor: simplest, i.e., most compact, explanation
tends to be the best explanation
•Works on minimal grammar and minimum description
length
Why symmetry?
Course: Structure-Aware Shape Processing
•Perceptual grouping via Gestalt law of
symmetry
Why symmetry?
Course: Structure-Aware Shape Processing
•Symmetry embeds semantic knowledge:
•Role in functional grouping
•Role in offering simplest (best) explanation
•Role in perceptual grouping
•However, symmetry is
•a purely geometric notion: explicitly detectable w/o
training
•but requires more global analysis
Semantics from geometry analysis
Course: Structure-Aware Shape Processing
•Detect major reflection symmetry
•Divide shape into symmetric part A
(full dinosaur) and the rest B
•Recursively process B and half of A
•Top-down: folding mesh
hierarchy
Folding meshes [Simari et al. 2006]
Key idea: “fold” reflection symmetries for compact shape representation
Course: Structure-Aware Shape Processing
Key idea: symmetry guides grouping and assembly of shape parts for a meaningful hierarchical meaningful part organization.
Symmetry hierarchy [Wang et al. 2011]
Course: Structure-Aware Shape Processing
Pre-segmentation
Hierarchy construction
Symmetry detection
Course: Structure-Aware Shape Processing
Rotational symmetry
Reflection symmetryConnectivity
Initial graph
Course: Structure-Aware Shape Processing
Grouping by symmetry
Assembly by
proximity
Two operations:
Bottom-up recursive graph contraction
Course: Structure-Aware Shape Processing
Two operations:Grouping by symmetry
Assembly by
proximity
Bottom-up recursive graph contraction
Course: Structure-Aware Shape Processing
March 30, 2012, CSC Talk, SFU
17
•Guiding principles
• Perceptual grouping: Gestalt law of symmetry
• Compactness of representation: Occam’s Razor
• Results in a set of “precedence rules”
How to order contractions?
Course: Structure-Aware Shape Processing
•Grouping-assembly mixing rules
• E.g., symmetry grouping takes precedence over
assembly
A1A1 A2A2
BB
Example of precedence rules
Course: Structure-Aware Shape Processing
•Symmetry grouping rules
• E.g., rot-symmetry takes precedence over ref-
symmetry
Example of precedence rules
Course: Structure-Aware Shape Processing
• Assembly rules
• E.g., assemble parts that preserve more symmetries first
A1
A2
B
Example of precedence rules
Course: Structure-Aware Shape Processing
Tree traversal following reversed order of graph
contractions
Application: hierarchical segmentation
Course: Structure-Aware Shape Processing
Application: structural editing
Course: Structure-Aware Shape Processing
•Construction is not the result of an optimization, but based on hand-crafted precedence rules
•Inconsistent hierarchies for objects in the same class
Limitations
Course: Structure-Aware Shape Processing
•To explain irregular 2D facades
•Use hierarchical decomposition as generative model
•Decomposition stops when reaching a regular grid (maximal symmetry) as it requires no explanation
Symmetry maximization [Zhang et al. 2013]
Goal: a generative model that best explains a structure
Course: Structure-Aware Shape Processing
• Hierarchical 2-way decomposition
Structural hierarchy
Course: Structure-Aware Shape Processing
• 2-way splits along two lateral directions
Input
+
Two substructuresSplit
Two operations
Course: Structure-Aware Shape Processing
InputLayering
+
Two substructures (layers)
Structure completion
• Layering: “split” along depth direction
Two operations
Course: Structure-Aware Shape Processing
Input Split only8 ops.
Split + Layering4 ops.
• Layering leads to more compact (simpler) explanation
Layering+split vs. split only
Course: Structure-Aware Shape Processing
Key ideas:
•Best explanation = simplest explanation = fewest decompositions
•Decomposition should then maximize the symmetry (SYMAX) of the substructures obtained
SYMAX is consistent with Gestalt law of symmetry
min. # ops.
Best explanation via SYMAX
Course: Structure-Aware Shape Processing
•Symmetry maximization at each decomposition
How symmetric the two substructures are
How good is the decomposition
SYMAX
Course: Structure-Aware Shape Processing
•Sum of symmetry measures at all internal nodes
Objective function
Course: Structure-Aware Shape Processing
• Difficult combinatorial search: genetic algorithm
Technical details
Course: Structure-Aware Shape Processing
• Difficult combinatorial search: genetic algorithm
• Symmetry measure
• Continuous measure for a discrete 2D pattern
• Behaves well at both ends of the symmetry spectrum
Perfectly symmetric asymmetric
Technical details
Course: Structure-Aware Shape Processing
• Difficult combinatorial search: genetic algorithm
• Symmetry measure
• Continuous measure for a discrete 2D pattern
• Behaves well at both ends of the symmetry spectrum
• Integral symmetry: integrating 1D symmetry profiles
X
Y
Technical details
Course: Structure-Aware Shape Processing
SYMAX
• involves explicit optimization
• symmetry hierarchy with an objective function
Both still confined to analyzing individual models
SYMAX vs. symmetry hierarchy
Course: Structure-Aware Shape Processing
• Extends symmetry hierarchy to co-analysis of a set
• A unified explanation of the shape structures
Co-hierarchical analysis
Goal: obtain a structural hierarchy that best explains a set of objects belonging to the same class
Course: Structure-Aware Shape Processing
Similarity and diversity
• Coarse level: similarity across set
• Finer level: individual shape variations
Course: Structure-Aware Shape Processing
Each object can have many possible hierarchies.
Need to select one hierarchy per shape.
…
Challenges
Course: Structure-Aware Shape Processing
There can be much geometric variability in the set.
Need to bypass geometry and analyze shape structures.
Challenges
Course: Structure-Aware Shape Processing
There can also be much structural variability; a single explanation cannot be sufficient
We account for that by clustering the hierarchies.
Challenges
Course: Structure-Aware Shape Processing
Using a minimal set of four shapes
Algorithm illustrated
Course: Structure-Aware Shape Processing
Compute symmetry hierarchies [Wang 2011] per shape
Per-shape symmetry hierarchies
Course: Structure-Aware Shape Processing
Sample from population of per-shape hierarchies
Sampling
Course: Structure-Aware Shape Processing
Multi-instance clustering: cluster the shapes while each shape has multiple instances (the
samples)
Key problem: cluster …
Course: Structure-Aware Shape Processing
Select one representative hierarchy per shape
… and then select
Course: Structure-Aware Shape Processing
• Traditional clustering: maximize within-cluster similarity and between-cluster dissimilarity
• In our problem: all shapes in the set are related
• So wish to discover maximal similarity among all
• But allow clusters to characterize structural variability
• Trick in representative selection: selections should maximize both within-cluster and between-cluster similarities
• Then resample hierarchies (and then cluster-select) close to the selections so the clusters as a whole also get closer
Objective of cluster-and-select
Course: Structure-Aware Shape Processing
Selections maximize within-cluster similarity only
Representative selection
Course: Structure-Aware Shape Processing
Selections maximize both within- and between-cluster similarity
Representative selection
Course: Structure-Aware Shape Processing
New samples are closer to the
representatives
Resample per-shape hierarchies
Course: Structure-Aware Shape Processing
New cluster-select result. Repeat the process until little movement.
Iterate cluster-select-resample
Course: Structure-Aware Shape Processing
A co-hierarchy implies correspondence between structures ---- functional correspondence.
Structure correspondence
Course: Structure-Aware Shape Processing
Functional correspondence
A co-hierarchy implies correspondence between structures ---- functional correspondence.
Course: Structure-Aware Shape Processing
• Still most works deal with flat structural organizations
• There is much to be done on hierarchical analysis:
• All covered techniques are unsupervised
• Supervised: symmetry measure is hard; ground truths for and comparison between hierarchical models are not easy
• Deep learning?
• Extension from 3D objects to more complex data, e.g., indoor scenes, and mixed object categories
• Toying with functionality analysis, but nothing explicit yet
Short “now and beyond”
Course: Structure-Aware Shape Processing
Go to: http://tinyurl.com/m2uqqfb
We welcome your feedback!