coherent laplacian 3d protrusion segmentation oxford brookes vision group queen mary, university of...
TRANSCRIPT
Coherent Laplacian 3D protrusion segmentation
Oxford Brookes Vision Group
Queen Mary, University of London, 11/12/2009
Fabio Cuzzolin
The problem
to recognize actions we need to extract features
segmenting moving articulated 3D bodies into parts
along sequences, in a consistent way
in an unsupervised fashion
robustly, with respect to changes of the topology of the moving body
as a building block of a wider motion analysis and capture framework
ICCV-HM'07, CVPR'08, to submit to IJCV
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Spectral methods
given a dataset of points {Xi, i=1,..,N}
compute an affinity matrix A(i,j) = d(Xi,Xj)
apply SVD to this affinity matrix
this yields a list of eigenvalues and associated eigenvectors
a number of eigenvectors are selected, and used to build an “embedded cloud” of points {Yi, i=1,..,N}
Laplacian methods
the affinity matrix is the Laplacian operator, or some function of it
Graph Laplacian: operator on functions f defined on sets X of points of the form
L[f]i = jN(i) wij (fi fj)
maps each such function f to another function L[f]
N(i) is the set of neighbors of Xi
fi is the value of the function f on Xi
Laplacian eigenfunctions
Laplacian eigenfunctions/values have nice topological properties
eigenvalues are invariant for volume-preserving transformations
eigenfunctions for a “base” for all functions on X
their zero-level sets are related to protrusions and symmetries of the underlying cloud
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Locally Linear Embedding
• for each data point we compute the weights Wij that best reconstruct Xi from its neighbors:
argminW i |X
i
j W
ij X
j|2
Low-dim embeddings Y_i are obtained by
argminY i |Y
i
j W
ij Y
j|2
i.e., local neighbors are the same, subject to • affinity matrix M = (IW)T(IW)
optimal embedding → bottom d+1 eigevectors (but last one)
LLE algorithm
Protrusion preservation
protrusions are high-curvature surface regions
by definition LLE leaves unchanged the weights of each neighborhood (the affine coordinates of X_i in the base of its neighbors)
weights depend on pairwise distances between points
preserving weights means preserving distances up to a scale
this happens in surface neighborhoods too
if they are all roughly the same size ...
... curvature distribution is preserved: protrusions!
Lower dimensionality
protrusion preservation is an effect of local isometry (the first constraint of LLE)
the covariance constraint has the effect of producing a lower-dimensional embedded cloud
LLE is a constrained minimization problem
in physics constraints G(X)=0 are associated with a force orthogonal to the constraint surface
the covariance constraint is associated with a force that pulls the cloud of points outward, reducing the chain of neighborhoods to a “string”
Clustering in the embedding space
Locally Linear Embedding: preserves the local Locally Linear Embedding: preserves the local structure of the datasetstructure of the dataset
generates a lower-dim embedded cloudpreserves protrusionsless sensitive to topology changes than other methods
40 50 60 70
60
8010
20
30
40
50
60
70
80
3D shape
0.05
0
0.05
0.060.04
0.020
0.02
0.05
0
0.05
0.1
LLE space
50
40
30
20
10
010
20
30
40
40
30
20
10
0
10
20
30
40
50
ISOMAP space
Pose invariance with LLE
generates a lower-dim, widely separated embedded cloudless sensitive to topology changes than other methodsless expensive then ISOMAP (refs. Jenkins, Chellappa)
rigid part
rigid part
moving joint area
unaffected neighborhoods
unaffected neighborhoods
affected neighborhoods
local neighbourhoods stable under articulated motion
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Algorithm
due to their low dimensionality, protrusions are detected in the embedding spacethey can be clustered as sets of collinear points using k-wise clusteringsegmentation is brought back to 3D
K-wise clustering
LLE maps the 3D shape to a lower-dimensional shapeIdea: clustering collinear points together
• K-wise clustering:K-wise clustering:
a hypergraph H is built by measuring the affinity of all triadsa weighted graph G which approximates H is constructed by constrained linear least square optimizationthe approximating graph is partitioned by spectral clustering (n-cut)
Protrusion detection
protrusions can be easily detected after embeddingdue to low dimensionality
Branch termination not detected Branch termination detected
an embedded point is a termination if its projection on the line interpolating its neighborhood is an extremum
Seed propagation along time
To ensure time consistency clusters’ seeds have to be propagated along time
Old positions of clusters in 3D are added to new cloud and embedded
Result: new seeds
Merging/splitting clusters
At each t all branch terminations of Y(t) are detected;
if t=0 they are used as seeds for k-wise clustering;
otherwise (t>0) standard k-means is performed on Y(t) using branch terminations as seeds, yielding a rough partition of the embedded cloud into distinct branches;
propagated seeds in the same partition are merged;
for each partition of Y(t) not containing any old seed a new seed is defined as the related branch termination.
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Results on real sequences 1
for real sequences ground truth is difficult to gather
we can still visually appreciate the quality and consistency of the resulting segmentation
Results on real sequences 2
for real sequences ground truth is difficult to gather
we can still visually appreciate the quality and consistency of the resulting segmentation
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Ground truth for synthetic data
For synthetic sequences of sequences for which pose has been estimated, ground truth can be gathered
performance indicators: compare the obtained segmentation with the three “natural” ones on the right
Scores for synthetic sequences
Scores for synthetic sequences
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Handling topology changes
when topology changes occur cluters merge and/or split to accommodate them
Handling missing data
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Vs EM clustering
EM clustering fits a multi-Gaussian distribution to the data through the EM algorithmnumber of cluster is automatically estimatedLeft: our algo; right: EM clustering
Vs ISOMAP
the same propagation scheme can be applied in the ISOMAP space
• extremely sensitive to topology changes
ISOMAP computes an embedding by applying MDS to the affinity matrix of all pairwise geodesic distances
Performance comparison
segmentation scores for two other real sequencessolid: our method; dashed: EM; dashdot: ISOMAP
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Estimating neighborhood size
the number of neighbors can be estimated from the data sequenceadmissible k: yields neighborhoods which do not span different bodyparts
0 20 40 60 80 1000
1
2
3
4
5
6
7
05
1015
20
05
1015
0
5
10
15
20
25
30
35
40
45
50
0
10
20
0
10
0
5
10
15
20
25
30
35
40
45
50
0 5 10 15 20 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Eigenvector selection
according to the eigenvectors we select after decomposition, we obtain different unsupervised segmentations
Stability with respect to parameter values
• consistency, segmentation and average scores for 2 sequences, as a function of parameter values k and d
Coherent 3D Laplacian protrusion segmentation
Laplacian methods
Locally linear embedding
An unsupervised algorithm
Results on real sequences
Results on synthetic sequences
Topology changes and missing data
Comparisons
Influence of parameters
Applications
Model recovery example
a sequence representing a counting hand is portrait
along time, the algorithm “learns” the object is formed by more and more rigid segments
Laplacian matching of dense meshes or voxelsets
as embeddings are pose-invariant (for articulated bodies)
they can then be used to match dense shapes by simply aligning their images after embedding
ICCV '07 – NTRL, ICCV '07 – 3dRR, CVPR '08, to submit to PAMI
Eigenfunction Histogram assignment
Algorithm:
compute Laplacian embedding of the two shapesfind assignment between eigenfunctions of the two shapesthis selects a section of the embedding spaceembeddings are orthogonally aligned there by EM
Results
Appls: graph matching, protein analysis, motion capture To propagate bodypart segmentation in timeMotion field estimation, action segmentation
Conclusions
Unsupervised bodypart segmentation algorithm which ensure consistency along timeSpectral method: clustering is performed in the embedding space (in particular after LLE) as shape becomes lower-dim and different bodyparts are widely separatedSeeds are propagated along time and merged/splitted according to topology variationsCompares favorably with other techniquesFirst block of motion analysis framework (matching, action recognition, etc.)