learning convolutional neural networks for graphs
TRANSCRIPT
![Page 1: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/1.jpg)
Learning Convolutional Neural Networks for GraphsMathias Niepert Mohamed Ahmed Konstantin Kutzkov
NEC Laboratories Europe
GA-653449
![Page 2: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/2.jpg)
2 Learning Convolutional Neural Networks for Graphs
Representation Learning for Graphs
Telecom Safety Transportation Industry Smart cities
(Edge) deployment
Deep Learning System
Intermediate Representation: Graphs
?
![Page 3: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/3.jpg)
3 Learning Convolutional Neural Networks for Graphs
Problem Definition
▌ Input: Finite collection of graphs
● Nodes of any two graphs are not necessarily in correspondence● Nodes and edges may have attributes (discrete and continuous)
▌Problem: Learn a representation for classification/regression
▌ Example: Graph classification problem
…
[ ] = ?class
![Page 4: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/4.jpg)
4 Learning Convolutional Neural Networks for Graphs
State of the Art: Graph Kernels
▌Define kernel based on substructures● Shortest paths● Random walks● Subtrees● ...
▌ Kernel is similarity function on pairs of graphs● Count the number of common substructures
▌Use graph kernels with SVMs
![Page 5: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/5.jpg)
5 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
![Page 6: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/6.jpg)
6 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
![Page 7: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/7.jpg)
7 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
![Page 8: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/8.jpg)
8 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
![Page 9: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/9.jpg)
9 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
![Page 10: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/10.jpg)
10 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
![Page 11: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/11.jpg)
11 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
![Page 12: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/12.jpg)
12 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood assembly (at least k=4 nodes)
![Page 13: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/13.jpg)
13 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood assembly (at least k=4 nodes)
![Page 14: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/14.jpg)
14 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood assembly (at least k=4 nodes)
![Page 15: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/15.jpg)
15 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood assembly (at least k=4 nodes)
![Page 16: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/16.jpg)
16 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood assembly (at least k=4 nodes)
![Page 17: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/17.jpg)
17 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood assembly (at least k=4 nodes)
![Page 18: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/18.jpg)
18 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
node sequence selection (w=6 nodes)
neighborhood normalization (exactly k=4 nodes)
1
2
3
4
1
2
3
41
23
41 2
3
41
2
3
4
neighborhood assembly (at least k=4 nodes)
12 3
4
![Page 19: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/19.jpg)
19 Learning Convolutional Neural Networks for Graphs
Patchy: Learning CNNs for Graphs
neighborhood normalization
● normalized neighborhoods serve as receptive fields● node and edge attributes correspond to channels
Convolutional architecture
1234
1 2 3 4 a 1 … a m
1 2 3 4 a 1 … a n
1234
1 2 3 4 a 1 … a m
1 2 3 4 a 1 … a n
1234
1 2 3 4 a 1 … a m
1 2 3 4 a 1 … a n
…
1
2
3
4
1
2
3
41
23
41 2
3
41
2
3
4
12 3
4
![Page 20: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/20.jpg)
20 Learning Convolutional Neural Networks for Graphs
Node Sequence Selection
▌We use centrality measures to generate the node sequences▌Nodes with similar structural roles are aligned across graphs
node sequence selection
A: Betweenness centrality B: Closeness centrality C: Eigenvector centrality D: Degree centrality
![Page 21: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/21.jpg)
21 Learning Convolutional Neural Networks for Graphs
▌ Simple breadth-first expansion until at least k nodes added, or no additional nodes to add
Neighborhood Assembly
neighborhood assembly
![Page 22: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/22.jpg)
22 Learning Convolutional Neural Networks for Graphs
▌Nodes of any two graphs should have similar position in the adjacency matrices iff their structural roles are similar
▌Result: For several distance measure pairs it is possible to efficiently compare labelingmethods without supervision
▌ Example: ||A – A’||1 and edit distanceon graphs
Graph Normalization Problem
1
2
3
4
normalization
distance measures in matrix and graph space
adjacency matrices under labeling
labeling method(centrality etc.)
![Page 23: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/23.jpg)
23 Learning Convolutional Neural Networks for Graphs
Graph Normalization
Distance to root node
1
2
2
3
3
3
3
![Page 24: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/24.jpg)
24 Learning Convolutional Neural Networks for Graphs
Graph Normalization
Distance to root node
Centrality measures, etc.
1
2
3
5
5
6
4
1
2
2
3
3
3
3
![Page 25: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/25.jpg)
25 Learning Convolutional Neural Networks for Graphs
Graph Normalization
Distance to root node
Centrality measures, etc.
Canonicalization (break ties)
1
2
3
6
5
7
4
1
2
3
5
5
6
4
1
2
2
3
3
3
3
![Page 26: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/26.jpg)
26 Learning Convolutional Neural Networks for Graphs
Computational Complexity
▌ At most linear in number of input graphs▌ At most quadratic in number of nodes for each graph
(depends on maximal node degree and centrality measure)
![Page 27: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/27.jpg)
27 Learning Convolutional Neural Networks for Graphs
Convolutional Architecture
1 2 3 4 a 1 … a n
1 2 3 4 a 1 … a n
1 2 3 4 a 1 … a n
1 2 3 4 a 1 … a n
1 2 3 4 a 1 … a n …
v 1 …
v M
field size: 4, stride: 4, filters: M
![Page 28: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/28.jpg)
28 Learning Convolutional Neural Networks for Graphs
Convolutional Architecture
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n …
v 1 …
v M
field size: 4, stride: 4, filters: M
1 2 3 4 1 2 3 4 1 2 3 41 2 3 4 1 2 3 4
![Page 29: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/29.jpg)
29 Learning Convolutional Neural Networks for Graphs
Convolutional Architecture
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n …
v 1 …
v M
field size: 4, stride: 4, filters: M
v 1 …
v M…
1 2 3 4 1 2 3 4 1 2 3 41 2 3 4 1 2 3 4
![Page 30: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/30.jpg)
30 Learning Convolutional Neural Networks for Graphs
Convolutional Architecture
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n …
field size: 3, stride: 1, N filters
v 1 …
v N
field size: 4, stride: 4, filters: M
v 1 …
v M
v 1 …
v M …
1 2 3 4 1 2 3 4 1 2 3 41 2 3 4 1 2 3 4
![Page 31: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/31.jpg)
31 Learning Convolutional Neural Networks for Graphs
Convolutional Architecture
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n …
field size: 3, stride: 1, N filters
v 1 …
v N
field size: 4, stride: 4, filters: Mv 1
… v M
v 1 …
v M…
1 2 3 4 1 2 3 4 1 2 3 41 2 3 4 1 2 3 4
![Page 32: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/32.jpg)
32 Learning Convolutional Neural Networks for Graphs
Convolutional Architecture
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n
a 1 … a n …
field size: 3, stride: 1, N filters
field size: 4, stride: 4, filters: Mv 1
… v M
v 1 …
v N
v 1 …
v N
v 1 …
v M…
…
1 2 3 4 1 2 3 4 1 2 3 41 2 3 4 1 2 3 4
![Page 33: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/33.jpg)
33 Learning Convolutional Neural Networks for Graphs
Experiments - Graph Classification
▌ Finite collection of graphs and their class labels
● Nodes of any two graphs are not necessarily in correspondence● Nodes and edges may have attributes (discrete and continuous)
▌ Learn a function from graphs to class labels
…
[ ] = ?class
class = 1 class = 0 class = 1
![Page 34: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/34.jpg)
34 Learning Convolutional Neural Networks for Graphs
Experiments - Convolutional Architecture
1 2 … 10 a 1 … a n
1 2 … 10 a 1 … a n
1 2 … 10 a 1 … a n
1 2 … 10 a 1 … a n
1 2 … 10 a 1 … a n …
field size: k, stride: k, filters: 16
field size: 10, stride: 1, filters: 8
flatten, dense 128 units
softmax
1 … 16
1 … 1
6…1
… 8
1 … 8…
![Page 35: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/35.jpg)
35 Learning Convolutional Neural Networks for Graphs
Classification Datasets
▌ MUTAG: Nitro compounds where classes indicate mutagenic effect on a bacterium (Salmonella Typhimurium)
▌ PTC: Chemical compounds where classes indicate carcinogenicity for male and female rats
▌ NCI: Chemical compounds where classes indicate activity against non-small cell lung cancer and ovarian cancer cell lines
▌ D&D: Protein structures where classes indicate whether structure is an enzyme or not
▌ ...
![Page 36: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/36.jpg)
36 Learning Convolutional Neural Networks for Graphs
▌Q: How efficient and effective compared to graph kernels?▌ Apply Patchy to typical graph classification benchmark data
Experiments - Graph Classification
![Page 37: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/37.jpg)
37 Learning Convolutional Neural Networks for Graphs
Experiments - Visualization
▌Q: What do learned edge filters look like?▌ Restricted Boltzmann machine applied to graphs▌ Receptive field size of hidden layer: 9
small instances of graphs weights of hidden nodes
graphs sampled from RBM
![Page 38: Learning Convolutional Neural Networks for Graphs](https://reader033.vdocuments.site/reader033/viewer/2022042722/588916211a28ab4a5c8b6677/html5/thumbnails/38.jpg)
38 Learning Convolutional Neural Networks for Graphs
Discussion
▌ Pros:● Graph kernel design not required● Outperforms graph kernels on several datasets (speed and accuracy)● Incorporates node and edge features (discrete and continuous)● Supports visualizations (graph motifs, etc.)
▌Cons:● Prone to overfitting on smaller data sets (graph kernel benchmarks)● Shift from designing graph kernels to tuning hyperparameters● Graph normalization not part of learning
code to be released: patchy.neclab.eu