tsinghua university - peng cui (cui, peng)'s...
TRANSCRIPT
![Page 1: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/1.jpg)
Peng Cui
Tsinghua University
Perspectives and Outlook on
Network Embedding and GCN
![Page 2: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/2.jpg)
2
The general description of data and their relations.
Network (Graph)
![Page 3: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/3.jpg)
3
Many types of data are networks
Social Networks Biology Networks Finance Networks
Internet of Things Information Networks Logistic Networks
![Page 4: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/4.jpg)
4
Why network is important?
In few cases, you only care about a subject but not its
relations with other subjects.
Reflected by relational subjects Decided by relational subjects
Target
Target
Image Characterization Social Capital
![Page 5: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/5.jpg)
5
G = ( V, E )
Networks are not learning-friendly
Links Topology
Inapplicability of
ML methods
Network
Data
Feature
Extraction
Pattern
Discovery
Network
Applications
Pipeline for network analysis
![Page 6: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/6.jpg)
6
Learning from networks
Network
EmbeddingGCN
![Page 7: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/7.jpg)
7
G = ( V, E ) G = ( V )Vector Space
generate
embed
• Easy to parallel
• Can apply classical ML methods
Network Embedding
![Page 8: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/8.jpg)
8
Goal Support network inference in vector space
Reflect network
structure
Maintain network
properties
B
A C
Transitivity
The goal of network embedding
Transform network nodes into vectors that are fit for
off-the-shelf machine learning models.
![Page 9: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/9.jpg)
9
Graph Neural Networks
Basic idea: recursive definition of states
A simple example: PageRank
F. Scarselli, et al. The graph neural network model. IEEE TNN, 2009.
Design a learning mechanism on graph.
![Page 10: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/10.jpg)
Main idea: pass messages between pairs of nodes & agglomerate
Stacking multiple layers like standard CNNs:
State-of-the-art results on node classification
10
Graph Convolutional Networks (GCN)
T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
![Page 11: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/11.jpg)
11
A brief history of GNNs
![Page 12: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/12.jpg)
Network Embedding and GCN
12
Graph
Feature
Network
Embedding
GCN
Input
Task results
Model Output
Embedding
Task results
Feature
Topology to Vector
Fusion of Topology and Features
Unsupervised v.s. (Semi-)Supervised
![Page 13: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/13.jpg)
Graph convolutional network v.s. Network embedding
13
• In some sense, they are different.
• Graphs exist in mathematics. (Data Structure)
• Mathematical structures used to model pairwise relations between
objects
• Networks exist in the real world. (Data)
• Social networks, logistic networks, biology networks, transaction
networks, etc.
• A network can be represented by a graph.
• A dataset that is not a network can also be represented by a graph.
![Page 14: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/14.jpg)
GCN for Natural Language Processing
• Many papers on BERT + GNN.
• BERT is for retrieval.
• It creates an initial graph of relevant
entities and the initial evidence.
• GNN is for reasoning.
• It collects evidence (i.e., old messages
on the entities) and arrive at new
conclusions (i.e., new messages on
the entities), by passing the messages
around and aggregating them.
14
Cognitive Graph for Multi-Hop Reading Comprehension at Scale. Ding et al., ACL 2019.
Dynamically Fused Graph Network for Multi-hop Reasoning. Xiao et al., ACL 2019.
![Page 15: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/15.jpg)
GCN for Computer Vision
• A popular trend in CV is to construct a graph during the learning process.
• To process multiple objects or parts in a scene, and to infer their relationships.
• Example: Scene graphs.
15
Scene Graph Generation by Iterative Message Passing. Xu et al., CVPR 2017.
Image Generation from Scene Graphs. Johnson et al., CVPR 2018.
![Page 16: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/16.jpg)
GCN for Symbolic Reasoning
16
• We can view the process of symbolic reasoning as a directed acyclic graph.
• Many recent efforts use GNNs to perform symbolic reasoning.
Learning by Abstraction: The Neural State Machine. Hudson & Manning, 2019.
Can Graph Neural Networks Help Logic Reasoning? Zhang et al., 2019.
Symbolic Graph Reasoning Meets Convolutions. Liang et al., NeurIPS 2018.
![Page 17: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/17.jpg)
GCN for Structural Equation Modeling
• Structural equation modeling, a form of causal modeling, tries to describe the
relationships between the variables as a directed acyclic graph (DAG).
• GNN can be used to represent a nonlinear structural equation and help find
the DAG, after treating the adjacency matrix as parameters.
17
DAG-GNN: DAG Structure Learning with Graph Neural Networks. Yu et al., ICML 2019.
![Page 18: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/18.jpg)
Pipeline for (most) GCN works
18
Raw DataGraph
ConstructionGCN End task
![Page 19: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/19.jpg)
• Co-occurrence (neighborhood)
19
Network embedding: topology to vector
![Page 20: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/20.jpg)
• High-order proximities
20
Network embedding: topology to vector
![Page 21: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/21.jpg)
• Communities
21
Network embedding: topology to vector
![Page 22: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/22.jpg)
• Heterogeneous networks
22
Network embedding: topology to vector
![Page 23: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/23.jpg)
Pipeline for (most) Network Embedding works
23
Network
Data
Network
Embedding
Downstream
ModelEnd task
![Page 24: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/24.jpg)
Learning for Networks vs. Learning via Graphs
24
Learning for networks
Learn
ing V
ia G
raphs
Network
Embedding
GCN
![Page 25: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/25.jpg)
The intrinsic problems NE is solving
25
Reducing representation dimensionality while preserving necessary
topological structures and properties.
Nodes & Links
Node Neighborhood
Community
Pair-wise Proximity
Hyper Edges
Global Structure
Non-transitivity
Asymmetric Transitivity
Dynamic
Uncertainty
Heterogeneity
Interpretability
![Page 26: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/26.jpg)
The intrinsic problem GCN is solving
26
Fusing topology and features in the way of smoothing features with
the assistance of topology.
N
N
N
d
N
d
X =
![Page 27: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/27.jpg)
What if the problem is topology-driven?
27
Since GCN is filtering features, it is inevitably feature-driven
Structure only provides auxiliary information (e.g. for filtering/smoothing)
When feature plays the key role, GNN performs good …
How about the contrary?
Synthesis data: stochastic block model + random features
Method Results
Random 10.0
GCN 18.3±1.1
DeepWalk 99.0±0.1
![Page 28: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/28.jpg)
Network Embedding v.s. GCN
28
Node
FeaturesTopology
Network
Embedding
Feature-based
LearningGCN
There is no better one, but there is more proper one.
![Page 29: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/29.jpg)
Rethinking: Is GCN truly a Deep Learning method?
29
Recall GNN formulation:
𝐻 𝑘+1 = 𝜎 𝑆𝐻 𝑘 𝑊 𝑘 , 𝑆 = ෩𝐷−1/2 ሚ𝐴෩𝐷−1/2
How about removing the non-linear component:
𝐻 𝑘+1 = 𝑆𝐻 𝑘 𝑊 𝑘
Stacking multiple layers and add softmax classification:
𝑌 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝐻 𝐾
= 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑆𝑆…𝑆𝐻 0 𝑊 0 𝑊 1 …𝑊 𝐾−1
= 𝑠𝑜𝑓𝑡𝑚𝑎𝑥 𝑆𝐾𝐻 0 𝑊
Wu, Felix, et al. Simplifying graph convolutional networks. ICML, 2019.
High-order proximity
![Page 30: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/30.jpg)
Rethinking: Is GCN truly a Deep Learning method?
30
This simplified GNN (SGC) shows remarkable results:
Node classification Text Classification
Wu, Felix, et al. Simplifying graph convolutional networks. ICML, 2019.
![Page 31: Tsinghua University - Peng Cui (Cui, Peng)'s Homepagepengcui.thumedialab.com/papers/Perspectives on NE and GCN... · 2019. 8. 7. · Tsinghua University Perspectives and Outlook on](https://reader036.vdocuments.site/reader036/viewer/2022081614/5fc532ceab148a1f746f130d/html5/thumbnails/31.jpg)
Summaries and Conclusions
31
Unsupervised v.s. (Semi-)Supervised
Learning for Networks v.s. Learning via Graphs
Topology-driven v.s. Feature-driven
Both GCN and NE need to treat the counterpart as the baselines