a tensor spectral approach to learning mixed membership ... · tensor decomposition: obtain...
TRANSCRIPT
![Page 1: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/1.jpg)
A Tensor Spectral Approach to Learning Mixed
Membership Community Models
Anima Anandkumar
U.C. Irvine
Joint work with Rong Ge, Daniel Hsu and Sham Kakade.
![Page 2: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/2.jpg)
Community Models in Social Networks
Social Network Modeling
Community: group of individuals
Community formation models: howpeople form communities and networks
Community detection: Discoveringhidden communities from observednetwork
![Page 3: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/3.jpg)
Community Models in Social Networks
Social Network Modeling
Community: group of individuals
Community formation models: howpeople form communities and networks
Community detection: Discoveringhidden communities from observednetwork
Modeling Overlapping Communities
People belong to multiple communities
Challenging to model and learn suchoverlapping communities
MIT
Mic
roso
ft
UC Irvine
Cornell
![Page 4: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/4.jpg)
Stochastic Block Model: Classical Approach
Generative Model
k communities and network size n
Each node belongs to one community:πu = ei if node u is in community i.
ei is the basis vector in ith coordinate.
Probability of an edge from u to v is
π>u Pπv .
Notice that π>u Pπv = Pi,j if πu = ei and
πv = ej.
Independent Bernoulli draws for edges.
Pros: Guaranteed algorithms for learning block models, e.g. spectralclustering, d2 distance based thresholding
Cons: Too simplistic. Cannot handle individuals in multiplecommunities
![Page 5: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/5.jpg)
Mixed Membership Block Model
Generative Model
k communities and network size n
Nodes in multiple communities: for node u, πuis community membership vector
Probability of an edge from u to v is π>u Pπv ,
where P is block connectivity matrix
Independent Bernoulli draws for edges
MIT
Mic
roso
ft
UC Irvine
Cornell
Dirichlet Priors
Each πu drawn independently from Dir(α): P[πu] ∝∏k
j=1 πu(j)αj−1
Stochastic block model: special case when αj → 0.
Sparse regime: αj < 1 for j ∈ [k].
Airoldi, Blei, Fienberg, and Xing. Mixed membership stochastic blockmodels. J. of Machine
Learning Research, June 2008.
![Page 6: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/6.jpg)
Learning Mixed Membership Models
Advantages
Mixed membership models incorporate overlapping communities
Stochastic block model is a special case
Model sparse community membership
Challenges in Learning Mixed Membership Models
Not clear if guaranteed learning can be provided.
Potentially large sample and computational complexities
Identifiability: when can parameters be estimated?
![Page 7: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/7.jpg)
Learning Mixed Membership Models
Advantages
Mixed membership models incorporate overlapping communities
Stochastic block model is a special case
Model sparse community membership
Challenges in Learning Mixed Membership Models
Not clear if guaranteed learning can be provided.
Potentially large sample and computational complexities
Identifiability: when can parameters be estimated?
Solution: Method of Moments Approach
![Page 8: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/8.jpg)
Method of Moments
Inverse moment method: solve equations relating parameters toobserved moments
Spectral approach: reduce equation solving to computing the“spectrum” of the observed moments
Non-convex but computationally tractable approaches
![Page 9: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/9.jpg)
Method of Moments
Inverse moment method: solve equations relating parameters toobserved moments
Spectral approach: reduce equation solving to computing the“spectrum” of the observed moments
Non-convex but computationally tractable approaches
Spectral Approach to Learning Mixed Membership Models
Edge and Subgraph Counts: Moments in a network
Tensor Spectral Approach: Low rank tensor form and efficientdecomposition methods
![Page 10: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/10.jpg)
Summary of Results and Technical Approach
Contributions
First guaranteed learning algorithm for overlapping community models
Correctness under exact moments.
Explicit sample complexity bounds.
Results are tight for Stochastic Block Models
![Page 11: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/11.jpg)
Summary of Results and Technical Approach
Contributions
First guaranteed learning algorithm for overlapping community models
Correctness under exact moments.
Explicit sample complexity bounds.
Results are tight for Stochastic Block Models
Approach
Method of moments: edge counts and 3-star count tensors
Tensor decomposition: Obtain spectral decomposition of the tensor
Tensor spectral clustering: Project nodes on the obtained eigenvectorsand cluster.
![Page 12: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/12.jpg)
Related Work
Stochastic Block Models
Classical approach to modeling communities (White et. al ‘76,Fienberg et. al 85)
Spectral clustering algorithm (McSherry ‘01, Dasgupta ‘04)
d2-distance based clustering (Frieze and Kannan ‘98)I weak regularity lemma: any dense convergent graph can be fitted to a
block model
Random graph models based on subgraph counts
Exponential random graph models
NP-hard in general to learn and infer these models
Overlapping community models
Many empirical works but no guaranteed learning
![Page 13: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/13.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 14: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/14.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 15: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/15.jpg)
Connection to LDA Topic Models
Exchangeable Topic Models
l words in a document x1, . . . , xl.
Document: topic mixture (draw of h).
Word xi generated from topic yi.
Exchangeability: x1 ⊥⊥ x2 ⊥⊥ . . . |h
LDA: h ∼ Dir(α) .
Learning from bigrams and trigrams
Words
Topics
Topic
Mixture
x1 x2 x3 x4 x5
y1 y2 y3 y4 y5
h
A. Anandkumar, R. Ge, D. Hsu, S.M. Kakade and M. Telgarsky “Tensor Decompositions for
Learning Latent Variable Models,” Preprint, October 2012.
![Page 16: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/16.jpg)
Viewing Community Models as Topic Models
Analogy for community model: each person can function both as adocument and a word.
Outgoing links from a node u: node u is a document.
Incoming links to a node v: node v is a word.
Node as a document Node as a word
![Page 17: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/17.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 18: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/18.jpg)
Moments for Spectral Method
Subgraph counts as moments of a random graph distribution
Edge Count Matrix
Consider partition X,A,B,C.
Adjacency Submatrices GX,A, GX,B , GX,C
x
uA B C
X
![Page 19: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/19.jpg)
Moments for Spectral Method
Subgraph counts as moments of a random graph distribution
Edge Count Matrix
Consider partition X,A,B,C.
Adjacency Submatrices GX,A, GX,B , GX,C
x
uA B C
X
3-Star Count Tensor
# of 3-star subgraphs from X to A,B,C.
M3(u, v, w) :=1
|X|# of 3-stars with leaves u,v,w
x
u v wA B C
X
Nodes in A,B,C: words and X: documents.
![Page 20: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/20.jpg)
Moments for Spectral Method
Subgraph counts as moments of a random graph distribution
Edge Count Matrix
Consider partition X,A,B,C.
Adjacency Submatrices GX,A, GX,B , GX,C
x
uA B C
X
3-Star Count Tensor
# of 3-star subgraphs from X to A,B,C.
M3(u, v, w) :=1
|X|# of 3-stars with leaves u,v,w
x
u v wA B C
X
Nodes in A,B,C: words and X: documents.
Learning via Edge and 3-Star Counts
![Page 21: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/21.jpg)
Recall Stochastic Block Model..
k communities and network size n
Each node belongs to one community: for node u, πu = ei if u is incommunity i. ei is the basis vector in ith coordinate.
Probability of an edge from u to v is π>u Pπv , where P is block
connectivity matrix
Independent Bernoulli draws for edges
Probability of edges from X to A is Π>XPΠA , where ΠA has πa,
a ∈ A as column vectors.
Denote FA := Π>AP
> and λi = P[π = ei] .
![Page 22: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/22.jpg)
Moments for Stochastic Block Model
Denote FA := Π>AP
> and λi = P[π = ei] .
Edge Count Matrix
Adjacency Submatrices GX,A, GX,B , GX,C
E[G>X,A|ΠA,X ] = Π>
XPΠA = Π>AP
>ΠX = FAΠX
x
uA B C
X
![Page 23: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/23.jpg)
Moments for Stochastic Block Model
Denote FA := Π>AP
> and λi = P[π = ei] .
Edge Count Matrix
Adjacency Submatrices GX,A, GX,B , GX,C
E[G>X,A|ΠA,X ] = Π>
XPΠA = Π>AP
>ΠX = FAΠX
x
uA B C
X
3-Star Count Tensor
# of 3-star subgraphs from X to A,B,C.
M3 :=1
|X|
∑
i∈X
[G>i,A ⊗G>
i,B ⊗G>i,C ]
E[M3|ΠA,B,C ] =∑
i
λi[(FA)i ⊗ (FB)i ⊗ (FC)i]
x
u v wA B C
X
Goal: Recover FA, FB , FC , ~λ
![Page 24: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/24.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 25: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/25.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 26: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/26.jpg)
Tensor Basics: Multilinear Transformations
For a tensor T , define (for matrices Vi of appropriate dimensions)
[T (W1,W2,W3)]i1,i2,i3 :=∑
j1,j2,j3
(T )j1,j2,j3∏
m∈[3]
Wm(jm, im)
For a matrix M , M(W1,W2) := W>1 MW2 .
For a symmetric tensor T of the form
T =k∑
r=1
λrφ⊗3r
T (W,W,W ) =∑
r∈[k]
λr(W>φr)
⊗3
T (I, v, v) =∑
r∈[k]
λr〈v, φr〉2φr.
T (I, I, v) =∑
r∈[k]
λr〈v, φr〉φrφ>r .
![Page 27: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/27.jpg)
Whiten: Convert to Orthogonal Symmetric Tensor
Assume exact moments are known.
E[G>X,A|ΠA,X ] = FAΠX
E[M3|ΠA,B,C ] =∑
i
λi[(FA)i ⊗ (FB)i ⊗ (FC)i]
![Page 28: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/28.jpg)
Whiten: Convert to Orthogonal Symmetric Tensor
Assume exact moments are known.
E[G>X,A|ΠA,X ] = FAΠX
E[M3|ΠA,B,C ] =∑
i
λi[(FA)i ⊗ (FB)i ⊗ (FC)i]
Use SVD of GX,A, GX,B , GX,C to obtain whitening matricesWA,WB ,WC
Apply multi-linear transformation on M3 using WA,WB ,WC .
T := E[M3(WA,WB ,WC)|ΠA,B,C ] =∑
i
wiµ⊗3i
T is symmetric orthogonal tensor: µi are orthonormal.
Spectral Tensor Decomposition of T
![Page 29: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/29.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 30: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/30.jpg)
Orthogonal Tensor Eigen Analysis
Consider orthogonal symmetric tensor T =∑
iwiµ⊗3i
T =k∑
i=1
wiµ⊗3i . T (I, µi, µi) = wiµi
![Page 31: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/31.jpg)
Orthogonal Tensor Eigen Analysis
Consider orthogonal symmetric tensor T =∑
iwiµ⊗3i
T =k∑
i=1
wiµ⊗3i . T (I, µi, µi) = wiµi
Obtaining eigenvectors through power iterations
u 7→T (I, u, u)
‖T (I, u, u)‖
![Page 32: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/32.jpg)
Orthogonal Tensor Eigen Analysis
Consider orthogonal symmetric tensor T =∑
iwiµ⊗3i
T =k∑
i=1
wiµ⊗3i . T (I, µi, µi) = wiµi
Obtaining eigenvectors through power iterations
u 7→T (I, u, u)
‖T (I, u, u)‖
Challenges and Solution
Challenge: Other eigenvectors presentI Solution: Only stable vectors are basis vectors µi
![Page 33: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/33.jpg)
Orthogonal Tensor Eigen Analysis
Consider orthogonal symmetric tensor T =∑
iwiµ⊗3i
T =k∑
i=1
wiµ⊗3i . T (I, µi, µi) = wiµi
Obtaining eigenvectors through power iterations
u 7→T (I, u, u)
‖T (I, u, u)‖
Challenges and Solution
Challenge: Other eigenvectors presentI Solution: Only stable vectors are basis vectors µi
Challenge: empirical momentsI Solution: robust tensor decomposition methods
![Page 34: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/34.jpg)
Optimization Viewpoint for Tensor Eigen Analysis
Consider Norm Optimization Problem for Tensor T
maxu
T (u, u, u) s.t. u>u = I
Constrained stationary fixed points T (I, u, u) = λu and u>u = I.
u is a local isolated maximizer if w>(T (I, I, u) − λI)w < 0 for all wsuch that w>w = I and w is orthogonal to u.
Review for Symmetric Matrices M =∑
i wiµ⊗2i
Constrained stationary points are the eigenvectors
Only top eigenvector is a maximizer and stable under power iterations
Orthogonal Symmetric Tensors T =∑
i wiµ⊗3i
Stationary points are the eigenvectors (up to scaling)
All basis vectors µi are local maximizers and stable under poweriterations
![Page 35: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/35.jpg)
Tensor Decomposition: Perturbation Analysis
Observed tensor T = T + E, where T =∑
i∈k wiµ⊗3i is orthogonal
tensor and perturbation E, and ‖E‖ ≤ ε.
Recall power iterations u 7→T (I, u, u)
‖T (I, u, u)‖
![Page 36: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/36.jpg)
Tensor Decomposition: Perturbation Analysis
Observed tensor T = T + E, where T =∑
i∈k wiµ⊗3i is orthogonal
tensor and perturbation E, and ‖E‖ ≤ ε.
Recall power iterations u 7→T (I, u, u)
‖T (I, u, u)‖
“Good” initialization vector 〈u(0), µi〉2 = Ω
(ε
wmin
)
![Page 37: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/37.jpg)
Tensor Decomposition: Perturbation Analysis
Observed tensor T = T + E, where T =∑
i∈k wiµ⊗3i is orthogonal
tensor and perturbation E, and ‖E‖ ≤ ε.
Recall power iterations u 7→T (I, u, u)
‖T (I, u, u)‖
“Good” initialization vector 〈u(0), µi〉2 = Ω
(ε
wmin
)
Perturbation Analysis
After N iterations, eigen pair (wi, µi) is estimated up to O(ε) error, where
N = O(log k + log log
wmax
ε
).
A. Anandkumar, R. Ge, D. Hsu, S.M. Kakade and M. Telgarsky “Tensor
Decompositions for Learning Latent Variable Models,” Preprint, October 2012.
![Page 38: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/38.jpg)
Robust Tensor Power Method
T =∑
iwiµ⊗3i +E
Basic Algorithm
Pick random initialization vectors
Run power iterations u 7→T (I, u, u)
‖T (I, u, u)‖
Go with the winner, deflate and repeat
![Page 39: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/39.jpg)
Robust Tensor Power Method
T =∑
iwiµ⊗3i +E
Basic Algorithm
Pick random initialization vectors
Run power iterations u 7→T (I, u, u)
‖T (I, u, u)‖
Go with the winner, deflate and repeat
Further Improvements
Initialization: Use neighborhood vectors for initialization
Stabilization: u(t) 7→ αT (I, u(t−1), u(t−1))
‖T (I, u(t−1), u(t−1))‖+ (1− α)u(t−1)
Efficient Learning Through Tensor Power Iterations
![Page 40: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/40.jpg)
Outline
1 Introduction
2 Tensor Form of Subgraph CountsConnection to Topic ModelsTensor Forms for Network Models
3 Tensor Spectral Method for LearningTensor PreliminariesSpectral Decomposition: Tensor Power Method
4 Conclusion
![Page 41: A Tensor Spectral Approach to Learning Mixed Membership ... · Tensor decomposition: Obtain spectral decomposition of the tensor Tensor spectral clustering: Project nodes on the obtained](https://reader034.vdocuments.site/reader034/viewer/2022051511/6017b10729f77d33e06d6940/html5/thumbnails/41.jpg)
Conclusion
Mixed Membership Models
Can model overlapping communities
Efficient to learn from low order moments:edge counts and 3-star counts.
Tensor Spectral Method
Whitened 3-star count tensor is anorthogonal symmetric tensor
Efficient decomposition through powermethod
Perturbation analysis: tight for stochasticblock model
MIT
Mic
roso
ft
UC Irvine
Cornell