structure, complexity and learning - simbad...
TRANSCRIPT
![Page 1: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/1.jpg)
Structure, complexity and
learning
Edwin Hancock
Department of Computer Science
University of York
Supported by a Royal Society
Wolfson Research Merit Award
![Page 2: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/2.jpg)
Aims
• How to probe or characterise structure of similarity data
in the form of graphs using random walks.
• How to characterise complexity of structures using ideas
from information theory (entropy).
• How to use complexity level characterisations to learn
variations in structure.
![Page 3: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/3.jpg)
Outline
• Random walks, graph spectra and
structural characteristics (zeta functions).
• Complexity and graph kernels.
• Learning generative models.
![Page 4: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/4.jpg)
Structural Variations
![Page 5: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/5.jpg)
Learning with graph data
• Problems based on graphs arise in areas such as
language processing, proteomics/chemoinformatics,
data mining, computer vision and complex systems.
• Relatively little methodology available, and vectorial
methods from statistical machine learning not easily
applied since there is no canonical ordering of the nodes
in a graph.
• Can make considerable progress if we develop
permutation invariant characterisations of variations in
graph structure.
![Page 6: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/6.jpg)
Learning with graph data
• Problems based on graphs arise in areas such as
language processing, proteomics/chemoinformatics,
data mining, computer vision and complex systems.
• Relatively little methodology available, and vectorial
methods from statistical machine learning not easily
applied since there is no canonical ordering of the nodes
in a graph.
• Can make considerable progress if we develop
permutation invariant characterisations of variations in
graph structure.
![Page 7: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/7.jpg)
Learning with graph data
• Problems based on graphs arise in areas such as
language processing, proteomics/chemoinformatics,
data mining, computer vision and complex systems.
• Relatively little methodology available, and vectorial
methods from statistical machine learning not easily
applied since there is no canonical ordering of the nodes
in a graph.
• Can make considerable progress if we develop
permutation invariant characterisations of variations in
graph structure.
![Page 8: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/8.jpg)
Protein-Protein Interaction Networks
![Page 9: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/9.jpg)
Characterising graphs
• Topological: e.g. average degree, degree distribution, edge-density, diameter, cycle frequencies etc.
• Spectral or algebraic: use eigenvalues of adjacency matrix or Laplacian, or equivalently the co-efficients of characteristic polynomial.
• Complexity: use information theoretic measures of structure (e.g. Shannon entropy).
![Page 10: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/10.jpg)
Learning is difficult because
• Graphs are not vectors: There is no natural ordering of nodes and edges. Correspondences must be used to establish order.
• Structural variations: Numbers of nodes and edges are not fixed. They can vary due to segmentation error.
• Not easily summarised: Since they do not reside in a vector space, mean and covariance hard to characterise.
![Page 11: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/11.jpg)
Learning with graphs Work with (dis) similarities: Can perform pariwise clustering or embed
sets of graphs in a vector space using multidimensional scaling on
similarities. Non metricity of similarities may pose problems.
Embed individual graphs in a low dimensional space: Characterise
structural variations in terms of statistical variation in a point-pattern.
Learn modes of structural variation: Understand how edge
(connectivity) structure varies for graphs belonging to the same class.
Requires correspondences of raw structure or alignment of an
embedded one. Can also be effected using permutation invariant
characteristics (path length, commute-times, cycle frequencies).
Construct generative model: Borrow ideas from graphical model to
construct model for raw structures or point distribution model to for
embedded graphs.
![Page 12: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/12.jpg)
Methods
• Graph-spectra lead to straightforward methods for
characterising structure and embedding, that do not
require correspondences and are permutation invariant.
• Random walks provide an intuitive way of understanding
spectral methods in terms of distributions of path and
cycle length.
• Complexity characterisations can by used in information
theoretic settings and allow tasks such as kernelisation
and learning to be addressed in a principled way.
![Page 13: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/13.jpg)
Aims
• Show how random walks can be used as
probes of graph structure.
• Explain links with spectral graph theory.
• Show links with complexity
characterisations, and information theory.
![Page 14: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/14.jpg)
Our contributions
• IJCV 2007 (Torsello, Robles-Kelly, Hancock) –shape classes from edit distance using pairwise clustering.
• PAMI 06 and Pattern Recognition 05 (Wilson, Luo and Hancock) – graph clustering using spectral features and polynomials.
• PAMI 07 (Torsello and Hancock) – generative model for variations in tree structure using description length.
• CVIU09 (Xiao, Wilson and Hancock) – generative model from heat-kernel embedding of graphs.
• QIC09 (Emms, Wilson and Hancock) quantum version of commute time.
• PR09a,b,c,(Emms, Wilson and Hancock) graph matching using quantum walks, lifting cospectrality of graphs using quantum walks.
• PR109(Xiao, Wilson and Hancock) graph characteristics from heat kernel trace.
• TNN11 (Ren, Wilson and Hancock) Ihara zeta function as graph characterisation,
![Page 15: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/15.jpg)
Random walks
And links to graph spectra.
![Page 16: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/16.jpg)
Problem studied • How can we find efficient means of characterising
graph structure which does not involve exhaustive
search? Enumerate properties of graph structure without
explicit search, e.g. count cycles, path length frequencies,
etc..
• Can we analyse the structure of sets of graphs
without solving the graph-matching problem? Inexact graph matching is computational bottleneck for
most problems involving graphs.
• Answer: let a random walker do the work.
![Page 17: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/17.jpg)
Graph Spectral Methods Use eigenvalues and eigenvectors of adjacency graph (or
Laplacian matrix) - Biggs, Cvetokovic, Fan Chung
Singular value methods for exact graph-matching and
point-set alignment). (Umeyama, Scott and Longuet-Higgins,
Shapiro and Brady).
Use of eigenvectors for image segmentation (Shi and Malik)
and for perceptual grouping (Freeman and Perona, Sarkar
and Boyer).
Graph-spectral methods for indexing shock-trees (Dickinson
and Shakoufandeh)
![Page 18: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/18.jpg)
Random walks on graphs
• Determined by the Laplacian spectrum
(and in continuous time case by heat-
kernel).
• Can be used to interpret, and analyse,
spectral methods since they can be
understood intuitively as path-based.
![Page 19: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/19.jpg)
Graph spectra and
random walks
Use spectrum of Laplacian matrix
to compute hitting and commute
times for random walk on a graph
![Page 20: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/20.jpg)
Laplacian Matrix
• Weighted adjacency matrix
• Degree matrix
• Laplacian matrix
otherwise
EvuvuwvuW
0
),(),(),(
Vv
vuWuuD ),(),(
WDL
![Page 21: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/21.jpg)
Laplacian spectrum
• Spectral Decomposition of Laplacian
• Element-wise
)()(),( vuvuL kk
k
k
T
k
k
kk
TL
),....,( ||1 Vdiag )|.....|( ||1 V
||21 ....0 V
![Page 22: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/22.jpg)
Properties of the Laplacian
• Eigenvalues are positive and smallest eigenvalue is zero
• Multiplicity of zero eigenvalue is number connected components of graph.
• Zero eigenvalue is associated with all-ones vector.
• Eigenvector associated with the second smallest eigenvalue is Fiedler vector.
||21 .....0 V
![Page 23: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/23.jpg)
Continuous time random walk
![Page 24: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/24.jpg)
Heat Kernels
• Solution of heat equation and measures
information flow across edges of graph
with time:
• Solution found by exponentiating
Laplacian eigensystem
tt Lht
h
TT
kk
k
kt tth ]exp[]exp[
TWDL
![Page 25: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/25.jpg)
Heat kernel and random walk
• State vector of continuous time random
walk satisfies the differential equation
• Solution
tt Lp
t
p
00]exp[ phpLtp tt
![Page 26: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/26.jpg)
Heat kernel and path lengths
• In terms of number of paths of
length k from node u to node v
!),(]exp[),(
|
1 k
tvuPtvuh
k
k
kt
),( vuPk
)()()1(),( vuvuP ii
k
Vi
ik
![Page 27: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/27.jpg)
Example.
Graph shows spanning tree of heat-kernel. Here weights of graph are
elements of heat kernel. As t increases, then spanning tree evolves from
a tree rooted near centre of graph to a string (with ligatures).
Low t behaviour dominated by Laplacian, high t behaviour dominated by
Fiedler-vector.
![Page 28: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/28.jpg)
Moments of the heat-kernel
trace
….can we characterise graph by
the shape of its heat-kernel trace
function?
![Page 29: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/29.jpg)
Heat Kernel Trace
Time (t)->
Trace
]exp[][ thTri
it
Shape of heat-kernel
distinguishes
graphs…can we
characterise its shape
using moments
![Page 30: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/30.jpg)
Rosenberg Zeta function
• Definition of zeta function
s
k
k
s
)()(0
![Page 31: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/31.jpg)
Heat-kernel moments
• Mellin transform
• Trace and number of connected components
• Zeta function
dttts
i
ss
i ]exp[)(
1
0
1
dttts s ]exp[)(0
1
]exp[][0
tChTri
it
dtChTrts
s t
ss
i
i
][)(
1)(
0
1
0
C is multiplicity of zero
eigenvalue or number of
connected components in
graph.
Zeta-function is related
to moments of heat-
kernel trace.
![Page 32: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/32.jpg)
Zeta-function behavior
![Page 33: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/33.jpg)
Objects
72 views of each object taken in 5 degree intervals as camera moves
in circle around object.
Feature points extracted using corner detector.
Construct Voronoi tesselation image plane using corner points as
seeds.
Delaunay graph is region adjacency graph for Voronoi regions.
![Page 34: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/34.jpg)
Heat kernel moments
(zeta(s), s=1,2,3,4)
![Page 35: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/35.jpg)
PCA using zeta(s), s=1,2,3,4)
![Page 36: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/36.jpg)
PCA on Laplace spectrum
![Page 37: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/37.jpg)
![Page 38: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/38.jpg)
![Page 39: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/39.jpg)
Ox-Caltech database
![Page 40: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/40.jpg)
Line-patterns
• Use Huet+Hancock representation (TPAMI-99).
• Extract straight line segments from Canny edge-map.
• Weight computed using continuity and proximity.
• Captures arrangement Gestalts.
![Page 41: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/41.jpg)
![Page 42: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/42.jpg)
Zeta function derivative
• Zeta function in terms of natural exponential
• Derivative
• Derivative at origin
]lnexp[)()(00
kk
k
s
k ss
]lnexp[ln)('0
k
kk ss
0
0
1lnln)0('
k
k k
k
![Page 43: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/43.jpg)
Meaning
• Number of spanning trees in graph
)](exp[)( ' od
d
G
Vu
u
Vu
u
![Page 44: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/44.jpg)
COIL
![Page 45: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/45.jpg)
Ox-Cal
![Page 46: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/46.jpg)
Eigenvalue polynomials (COIL)
Trace
Determinant
![Page 47: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/47.jpg)
Eigenvalue Polynomials (Ox-
Cal)
![Page 48: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/48.jpg)
Spectral polynomials (COIL)
![Page 49: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/49.jpg)
Spectral Polynomials (Ox-Cal)
![Page 50: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/50.jpg)
COIL: node and edge frequency
![Page 51: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/51.jpg)
Ox-Cal: node+edge frequency
![Page 52: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/52.jpg)
Performance
• Rand index=correct/(correct+wrong).
Zeta func. 0.92
Sym. Poly.
(evals)
0,90
Sym. Poly.
(matrix)
0.88
Laplacian
spectrum
0.78
![Page 53: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/53.jpg)
Deeper probes of structure
Ihara zeta function
![Page 54: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/54.jpg)
Zeta functions
• Used in number theory to characterise
distribution of prime numbers.
• Can be extended to graphs by replacing
notion of prime number with that of a
prime cycle.
![Page 55: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/55.jpg)
Ihara Zeta function
• Determined by distribution of prime cycles.
• Transform graph to oriented line graph (OLG) with edges as nodes and edges indicating incidence at a common vertex.
• Zeta function is reciprocal of characteristic polynomial for OLG adjacency matrix.
• Coefficients of polynomial determined by eigenvalues of OLG adjacency matrix.
• Coefficients linked to topological quantity, i.e. prime cycle frequencies.
![Page 56: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/56.jpg)
Oriented Line Graph
- Frobenius operator
![Page 57: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/57.jpg)
Links to walks on graphs
• Transition matrix of OLG determines
discrete time quantum walk on graph (with
Haddamard coin).
• Can be used to define backtrackless
classical walk.
![Page 58: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/58.jpg)
Ihara Zeta Function
• Ihara Zeta Function for a graph G(V,E)
– Defined over prime cycles of graph
– Rational expression in terms of characteristic polynomial of oriented line-graph
A is adjacency matrix of line digraph
Q =D-I (degree matrix minus identity
![Page 59: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/59.jpg)
Characteristic Polynomials from IZF
• Perron-Frobenius operator is the adjacency matrix TH
of the oriented line graph
• Determinant Expression of IZF
– Each coefficient,i.e. Ihara coefficient, can be derived from the
elementary symmetric polynomials of the eigenvalue set
• Pattern Vector in terms of
![Page 60: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/60.jpg)
Analysis of determinant
• From matrix logs
• Tr[T^k] is symmetric polynomial of
eigenvalues of T
]][exp[]det[
1)(
1 k
sTTr
TsIs
k
k
k
N
N
N
N
TTr
TTr
TTr
.............][
.....
...][
........][
21
2
21
2
1
2
1
1
![Page 61: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/61.jpg)
Distribution of prime cycles
• Frequency distribution for cycles of length l
• Cycle frequencies
l
l
lsNsds
ds )(ln
][)(ln)!1(
10
l
sl
l
l TTrsds
d
lN
![Page 62: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/62.jpg)
Experiments: Edge-weighted Graphs
Feature Distance
& Edit Distance
Three Classes of Randomly
Generated Graphs
![Page 63: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/63.jpg)
Experiments: Hypergraphs
![Page 64: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/64.jpg)
Complexity
Information theory, graphs and
kernels.
![Page 65: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/65.jpg)
Protein-Protein Interaction Networks
![Page 66: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/66.jpg)
Characterising graphs
• Topological: e.g. average degree, degree distribution, edge-density, diameter, cycle frequencies etc.
• Spectral or algebraic: use eigenvalues of adjacency matrix or Laplacian, or equivalently the co-efficients of characteristic polynomial.
• Complexity: use information theoretic measures of structure (e.g. Shannon entropy).
![Page 67: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/67.jpg)
Complexity characterisation
• Information theory: entropy measures
• Structural pattern recognition: graph
spectral indices of structure and topology.
• Complex systems: measures of centrality,
separation, searchability.
![Page 68: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/68.jpg)
Information theory
• Entropic measures of complexity:
Shannon , Erdos-Renyi, Von-Neumann.
• Description length: fitting of models to
data, entropy (model cost) tensioned
against log-likelihood (goodness of fit).
• Kernels: Use entropy to computeJensen-
Shannon divergence
![Page 69: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/69.jpg)
Entropy on graphs
• Permutation structure: numbers of
different colourings, projection onto
Birkhoff polytopes.
• Graph spectral: Von Neumann entropy
over Laplacian spectrum.
• Embedding: Entropy associated with
embedding as a point-set.
![Page 70: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/70.jpg)
Von-Neumann Entropy
• Derived from normalised Laplacian
spectrum
• Comes from quantum mechanics and is
entropy associated with density matrix.
2
ˆln
2
ˆ||
1
i
V
i
iVNH
TDADDL ˆˆˆ)(ˆ 2/12/1
![Page 71: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/71.jpg)
Approximation
• Quadratic entropy
• In terms of matrix traces
||
1
2||
1
||
1
ˆ4
1ˆ2
1
2
ˆ1
2
ˆ V
i
i
V
i
ii
V
i
iVNH
]ˆ[4
1]ˆ[
2
1 2LTrLTrHVN
![Page 72: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/72.jpg)
Computing Traces
• Normalised Laplacian
• Normalised Laplacian squared
||]ˆ[ VLTr
Evu vudd
VLTr),(
2
4
1||]ˆ[
![Page 73: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/73.jpg)
Simplified entropy
Evu vu
VNdd
VH),( 4
1||
4
1
Collect terms together, von Neumann
entropy reduces to
![Page 74: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/74.jpg)
Homogeneity index
Evuvuvu
Evu
vu
ddddVVG
ddG
),(
2
),(
2/12/1
211
1||2||
1)(
)()(
Based on degree
statistics
![Page 75: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/75.jpg)
Homogeneity meaning
Evu
vuAvuCTG),(
),(2),(~)(
Limit of large degree
Largest when commute time differs from 2
due to large number of alternative
connecting paths.
![Page 76: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/76.jpg)
Polytopal complexity
• Decompose doubly stochastic kernel matrix into
sum of convex polytopes (permutation matrices)
• Entropy defined over expansion coefficients
PpK
ppS ln
![Page 77: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/77.jpg)
Thermodynamic Depth Complexity
• Simulate heat flow on graph using continuous time
random walk.
• Characterise nodes by their thermodynamic depth (time
walk takes to reach node).
• Measure heat flow dependence at each node with time.
Record maximum.
• Compute homogeneity statistics over thermodynamic
depth.
![Page 78: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/78.jpg)
Phase transition
• As time evolves complexity undergoes
phase transition.
• Corresponds to maximum flow at a node.
• Maximum of entropy.
![Page 79: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/79.jpg)
Uses
• Complexity-based clustering (especially
protein-protein interaction networks).
• Defining information theoretic (Jensen-
Shannon) kernels.
• Controlling complexity of generative
models of graphs.
![Page 80: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/80.jpg)
Protein-Protein Interaction Networks
![Page 81: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/81.jpg)
Experiment
![Page 82: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/82.jpg)
![Page 83: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/83.jpg)
![Page 84: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/84.jpg)
Non-extensive kernel
Based on von Neumann entropy
of a graph
![Page 85: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/85.jpg)
Graph kernels
• Avoid correspondence problem.
• Path length kernel (Gartner+etc) from powers of
adjacency matrix.
• Random walk kernel from product graph
(Borgwardt+Smola etc).
• Cycle length kernel (Horvarth+Gartner), Subtree kernel,
frequent subgraphs etc.
![Page 86: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/86.jpg)
Information theoretic kernels
• Rely on probability distributions, their
entropy and mutual information (Jenssen,
Figueiredo).
• Entropy is complexity level chracterisation
of graph structure.
• Extensive =>additive entropy,
Non-extensive=> non additive entropy
![Page 87: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/87.jpg)
Jensen-Shannon Kernel
• Defined in terms of J-S divergence
• Properties: extensive, positive.
)()()(),(
),(2ln),(
jijiji
jijiJS
GHGHGGHGGJS
GGJSGGK
![Page 88: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/88.jpg)
Computation
• Construct direct product graph for each
graph pair.
• Compute von-Neumann entropy difference
between product graph and two graphs
individually.
• Construct kernel matrix over all pairs.
![Page 89: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/89.jpg)
![Page 90: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/90.jpg)
![Page 91: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/91.jpg)
Generative Models
• Structural domain: define probability distribution over
prototype structure. Prototype together with parameters
of distribution minimise description length (Torsello and
Hancock, PAMI 2007) .
• Spectral domain: embed nodes of graphs into vector-
space using spectral decomposition. Construct point
distribution model over embedded positions of nodes
(Bai, Wilson and Hancock, CVIU 2009).
![Page 92: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/92.jpg)
Deep learning
• Deep belief networks (Hinton 2006, Bengio 2007).
• Compositional networks (Amit+Geman 1999, Fergus
2010).
• Markov models (Leonardis 200
• Stochastic image grammars (Zhu, Mumford, Yuille)
• Taxonomy/category learning (Todorovic+Ahuja, 2006-
2008).
![Page 93: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/93.jpg)
Aim
• Combine spectral and structural methods.
• Use description length criterion.
• Apply to graphs rather than trees.
![Page 94: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/94.jpg)
Prior work
• IJCV 2007 (Torsello, Robles-Kelly, Hancock) –shape classes from edit distance using pairwise clustering.
• PAMI 06 and Pattern Recognition 05 (Wilson, Luo and Hancock) – graph clustering using spectral features and polynomials.
• PAMI 07 (Torsello and Hancock) – generative model for variations in tree structure using description length.
• CVIU09 (Xiao, Wilson and Hancock) – generative model from heat-kernel embedding of graphs.
![Page 95: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/95.jpg)
Structural learning
Using description length
![Page 96: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/96.jpg)
Description length
• Wallace+Freeman: minimum message
length.
• Rissanen: minimum description length.
Use log-posterior probability to locate model that is
optimal with respect to code-length.
![Page 97: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/97.jpg)
Similarities/differences
• MDL: selection of model is aim; model
parameters are simply a means to this
end. Parameters usually maximum
likelihood. Prior on parameters is flat.
• MML: Recovery of model parameters is
central. Parameter prior may be more
complex.
![Page 98: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/98.jpg)
Coding scheme
• Usually assumed to follow an exponential
distribution.
• Alternatives are universal codes and predictive
codes.
• MML has two part codes (model+parameters). In
MDL the codes may be one or two-part.
![Page 99: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/99.jpg)
Method
• Model is supergraph (i.e. Graph prototypes) formed by graph union.
• Sample data observation model: Bernoulli distribution over nodes and edges.
• Mode: complexity: Von-Neumann entropy of supergraphs.
• Fitting criterion:
MDL-like-make ML estimates of the Bernoulli parameters
MML-like: two-part code for data-model fit + supergraph complexity.
![Page 100: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/100.jpg)
Model overview
• Description length criterion
code-length=negative + model code-length
log-likelihood (entropy)
Data-set: set of graphs G
Model: prototype graph+correspondences with it
Updates by expectation maximisation:
Model graph adjacency matrix (M-step)
+ correspondence indicators (E-step).
)()|(),( HGLLGL
![Page 101: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/101.jpg)
Learn supergraph using MDL
• Follow Torsello and Hancock and pose the problem of learning generative model for graphs as that of learning a supergraph representation.
• Required probability distributions is an extension of model developed by Luo and Hancock.
• Use von Neumann entropy to control supergraph’s complexity.
• Develop an EM algorithm in which the node correspondences and the supergraph edge probability matrix are treated as missing data.
![Page 102: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/102.jpg)
Probabilistic Framework
V1
V4
V3V2
V1 V2 V3 V4
0
A=
011
1 0 1
110
0
0
1
001 V1
V2
V3
V4
Here the structure of the sample graphs and the supergraph are
represented by their adjacency matrices
![Page 103: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/103.jpg)
Given a sample graph and a supergraph
along with their assignment matrix,
the a posteriori probabilities of the sample graphs given the
structure of the supergraph and the node correspondences is
defined as
Observation model
![Page 104: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/104.jpg)
Data code-length
• For the sample graph-set and the supergraph ,
the set of assignment is . Under the assumption
that the graphs in are independent samples from the distribution,
the likelihood of the sample graphs can be written
• Code length of observed data
![Page 105: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/105.jpg)
Overall code-length
According to Rissanen and Grunwald’s minimum description length
criterion, we encode and transmit the sample graphs and the
supergraph structure. This leads to a two-part message whose total
length is given
We consider both the node correspondence information between
graphs S and the structure of the supergraph M as missing data and
locate M by minimizing the overall code-length using EM algorithm.
![Page 106: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/106.jpg)
EM – code-length criterion
![Page 107: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/107.jpg)
Expectation + Maximization • M-step :
Recover correspondence matrices: Take partial derivative of the weighted log-likelihood function and soft assign.
Modify supergraph structure :
• E-step: Compute the a posteriori probability of the nodes in the sample graphs being matching to those of the supergraph.
![Page 108: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/108.jpg)
Experiments
Delaunay graphs from images of different objects.
COIL dataset Toys dataset
![Page 109: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/109.jpg)
Experiments---validation COIL dataset: model complexity increase, graph data log-likelihood
increase, overall code length decrease during iterations.
Toys dataset: model complexity decrease, graph data log-likelihood
increase, overall code length decrease during iterations.
![Page 110: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/110.jpg)
Experiments---classification task
We compare the performance of our learned
supergraph on classification task with two alternative
constructions , the median graph and the supergraph
learned without using MDL. The table below shows
the average classification rates from 10-fold cross
validation, which are followed by their standard
errors.
![Page 111: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/111.jpg)
Experiments---graph embedding
Pairwise graph distance based on the
Jensen-Shannon divergence and the von
Neumann entropy of graphs
![Page 112: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/112.jpg)
Experiments---graph embedding
Edit distance JSD distance
![Page 113: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/113.jpg)
Generative model
• Train on graphs with set of predetermined
characteristics.
• Sample using Monte-Carlo.
• Reproduces characteristics of training set,
e.g. Spectral gap, node degree
distribution, etc.
![Page 114: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/114.jpg)
Erdos Renyi
![Page 115: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/115.jpg)
Barabasi Albert (scale free)
![Page 116: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/116.jpg)
Dealunay Graphs
![Page 117: Structure, complexity and learning - Simbad Projectsimbad-fp7.eu/images/tutorial/03-ECCV2012Tutorial.pdf · Structure, complexity and learning ... sets of graphs in a vector space](https://reader031.vdocuments.site/reader031/viewer/2022030506/5ab4fd7b7f8b9a0f058c5675/html5/thumbnails/117.jpg)
Experiments---generate new samples