diffusion maps - tauamir1/course2012-2013/dm_presentation.pdf · diffusion maps since p is...
TRANSCRIPT
![Page 1: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/1.jpg)
Diffusion Maps
Aviv Rotbart
Tel Aviv University
December 2012
Aviv Rotbart (TAU) Diffusion Maps December 2012 1 / 40
![Page 2: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/2.jpg)
Data Analysis Scope
Data Analysis
Manifold Learning
Kernel Methods
Diffusion Maps
Aviv Rotbart (TAU) Diffusion Maps December 2012 2 / 40
![Page 3: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/3.jpg)
Data Analysis Scope
Data Analysis
Manifold Learning
Kernel Methods
Diffusion Maps
Aviv Rotbart (TAU) Diffusion Maps December 2012 2 / 40
![Page 4: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/4.jpg)
Outline
1 Data Analysis Background
Aviv Rotbart (TAU) Diffusion Maps December 2012 3 / 40
![Page 5: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/5.jpg)
Manifold learningWhy use manifolds?
availability of data⇓
many observable parameters⇓
high-dimensional recorded data
# parameters = O(10), O(100), . . .︷ ︸︸ ︷···
Data contain dependencies & redundanciesObservable space = non-linear mapping of few underlying factorsUnderlying locally low-dimensional in high-dimensional ambientspace
Aviv Rotbart (TAU) Diffusion Maps December 2012 4 / 40
![Page 6: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/6.jpg)
Manifold learningThe goal
Aviv Rotbart (TAU) Diffusion Maps December 2012 5 / 40
![Page 7: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/7.jpg)
Manifold learningThe goal
Aviv Rotbart (TAU) Diffusion Maps December 2012 5 / 40
![Page 8: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/8.jpg)
Manifold learningThe goal
Aviv Rotbart (TAU) Diffusion Maps December 2012 5 / 40
![Page 9: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/9.jpg)
Manifold learningKernel methods
Aviv Rotbart (TAU) Diffusion Maps December 2012 6 / 40
![Page 10: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/10.jpg)
Manifold learningKernel methods
Aviv Rotbart (TAU) Diffusion Maps December 2012 6 / 40
![Page 11: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/11.jpg)
Manifold learningDiffusion distances
Aviv Rotbart (TAU) Diffusion Maps December 2012 7 / 40
![Page 12: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/12.jpg)
Diffusion maps1
Diffusion process & affinities
Gaussian kernel:k(x , y) , e−
‖x−y‖ε
Degrees: q(x) ,∑ k(x , y)
Transition probabilities:
p(x , y) ,k(x , y)
q(x)
Diffusion affinities:
a(x , y) ,k(x , y)√
q(x)√
q(y)
= q1/2(x)p(x , y)q−1/2(y)
1R.R. Coifman and S. Lafon. “Diffusion Maps”. In: Applied andComputational Harmonic Analysis 21.1 (2006), pp. 5–30.
Aviv Rotbart (TAU) Diffusion Maps December 2012 8 / 40
![Page 13: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/13.jpg)
Diffusion mapsSpectral embedding
1 = λ0 ≥ λ1 ≥ λ2 ≥ · · · ≥ λδ > 0
φ0 φ1 φ2 · · · φδn
Spectrum (eigenvalues) of the diffusion affinity A and its powers
x 7→ Φ(x) , [λ0φ0(x) , λ1φ1(x) , λ2φ2(x) , . . . , λδφδ(x)]T
Aviv Rotbart (TAU) Diffusion Maps December 2012 9 / 40
![Page 14: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/14.jpg)
Diffusion mapsSpectral embedding
1 = λ0 ≥ λ1 ≥ λ2 ≥ · · · ≥ λδ > 0
φ0 φ1 φ2 · · · φδn
Spectrum (eigenvalues) of the diffusion affinity A and its powersx 7→ Φ(x) , [λ0φ0(x) , λ1φ1(x) , λ2φ2(x) , . . . , λδφδ(x)]T
Aviv Rotbart (TAU) Diffusion Maps December 2012 9 / 40
![Page 15: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/15.jpg)
Diffusion mapsSpectral embedding
1 = λ0 ≥ λ1 ≥ λ2 ≥ · · · ≥ λδ > 0
φ0 φ1 φ2 · · · φδn
Spectrum (eigenvalues) of the diffusion affinity A and its powers
x 7→ Φ(x) , [λ0φ0(x) , λ1φ1(x) , λ2φ2(x) , . . . , λδφδ(x)]T
Aviv Rotbart (TAU) Diffusion Maps December 2012 9 / 40
![Page 16: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/16.jpg)
Diffusion mapsEmbedded distances
embedded distance︷ ︸︸ ︷‖Φ(x)− Φ(y)‖ =
diffusion distance︷ ︸︸ ︷‖a(x , ·)− a(y , ·)‖
Aviv Rotbart (TAU) Diffusion Maps December 2012 10 / 40
![Page 17: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/17.jpg)
Gaussian-based diffusion mapsGaussian kernel:
Normalization =⇒ diffusion kernelSpectral analysis =⇒ map from M⊆ Rm to Rδ�m
Aviv Rotbart (TAU) Diffusion Maps December 2012 11 / 40
![Page 18: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/18.jpg)
Gaussian-based diffusion mapsGaussian kernel:
Normalization =⇒ diffusion kernelSpectral analysis =⇒ map from M⊆ Rm to Rδ�m
Aviv Rotbart (TAU) Diffusion Maps December 2012 11 / 40
![Page 19: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/19.jpg)
Gaussian-based diffusion mapsGaussian kernel:
Normalization =⇒ diffusion kernelSpectral analysis =⇒ map from M⊆ Rm to Rδ�m
Aviv Rotbart (TAU) Diffusion Maps December 2012 11 / 40
![Page 20: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/20.jpg)
Gaussian-based diffusion mapsGaussian kernel:
Normalization =⇒ diffusion kernelSpectral analysis =⇒ map from M⊆ Rm to Rδ�m
Aviv Rotbart (TAU) Diffusion Maps December 2012 11 / 40
![Page 21: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/21.jpg)
Gaussian-based diffusion mapsGaussian kernel:
Normalization =⇒ diffusion kernelSpectral analysis =⇒ map from M⊆ Rm to Rδ�m
Aviv Rotbart (TAU) Diffusion Maps December 2012 11 / 40
![Page 22: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/22.jpg)
Gaussian-based diffusion mapsGaussian kernel:
Normalization =⇒ diffusion kernelSpectral analysis =⇒ map from M⊆ Rm to Rδ�m
Aviv Rotbart (TAU) Diffusion Maps December 2012 11 / 40
![Page 23: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/23.jpg)
Numerical rank
λ0 · · · λδ︸ ︷︷ ︸numerical rank
Spectrum (eigenvalues) of the diffusion affinity A and its powers
Aviv Rotbart (TAU) Diffusion Maps December 2012 12 / 40
![Page 24: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/24.jpg)
Numerical rank
λ0 · · · λδ︸ ︷︷ ︸numerical rank
Spectrum (eigenvalues) of the diffusion affinity A and its powers
Aviv Rotbart (TAU) Diffusion Maps December 2012 12 / 40
![Page 25: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/25.jpg)
Numerical rank
λ0 · · · λδ︸ ︷︷ ︸numerical rank
Spectrum (eigenvalues) of the diffusion affinity A and its powers
Aviv Rotbart (TAU) Diffusion Maps December 2012 12 / 40
![Page 26: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/26.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training PhaseTesting Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 27: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/27.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training PhaseTesting Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 28: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/28.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 29: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/29.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 30: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/30.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 31: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/31.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100
∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 32: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/32.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reduction
Out-of-sample ext.
⊆ R3
∈ R3
Training Phase
Testing Phase
⊆ R100
∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 33: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/33.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100
∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 34: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/34.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reduction
Out-of-sample ext.
⊆ R3
∈ R3
Training Phase
Testing Phase
⊆ R100
∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 35: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/35.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 36: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/36.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 37: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/37.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 38: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/38.jpg)
Typical Scenario: Anomaly Detection
100 system parameters
every few minutes
Dim. reductionOut-of-sample ext.
⊆ R3∈ R3
Training Phase
Testing Phase
⊆ R100∈ R100
Parameter SignificanceCPU % High
Process # High... ...
Net I/O LowHD I/O Low
Aviv Rotbart (TAU) Diffusion Maps December 2012 13 / 40
![Page 39: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/39.jpg)
Diffusion Maps
The Training stage is based on diffusion maps.Diffusion maps generate efficient representations of complexgeometric structures in lower dimensional spaces.Use the eigenfunctions of Markov matrices.Revels global geometric information from local structures.Nonlinear reduction of dimensionality.
Aviv Rotbart (TAU) Diffusion Maps December 2012 14 / 40
![Page 40: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/40.jpg)
Diffusion Maps - example ILet’s take a look at this dataset and the two pairs of points on it.
Is it right to measure the distance between the points using euclidiandistance?
Aviv Rotbart (TAU) Diffusion Maps December 2012 15 / 40
![Page 41: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/41.jpg)
Diffusion maps and distance
The distance between two points xi and xj can be defined as theprobability to land in xj after N random steps that start at xi .
(a) Euclidean distance (b) Random walk
This distance measures the connectivity between xi and xj in thedata, while taking into account all possible paths between xi and xj .
Aviv Rotbart (TAU) Diffusion Maps December 2012 16 / 40
![Page 42: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/42.jpg)
Diffusion Maps
Let Γ = {x1, ..., xm} be a set of points in Rn.Construct a non-negative symmetric kernel on the data:
wε(x , y) = e−‖x−y‖2
2ε .
(c) Gaussian (d) Gaussian Kernel
Aviv Rotbart (TAU) Diffusion Maps December 2012 17 / 40
![Page 43: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/43.jpg)
Diffusion MapsA general form of a kernel, with a parameter α that controls thenormalization type, given by
w (α)ε (x , y) =
wε (x , y)
qα (x) qα (y), q (x) =
∑y∈Γ
wε (x , y) (1)
If we set α to 0.5, the normalized kernel will look like:
(e) Gaussian Kernelwε (x , y)
(f) w (0.5)ε (x , y)
Aviv Rotbart (TAU) Diffusion Maps December 2012 18 / 40
![Page 44: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/44.jpg)
Diffusion MapsThe transition matrix is defined by
pαε (x , y) =w (α)ε (x , y)
d (α)ε (y)
(2)
whered (α)ε (y) =
∑x∈Γ
w (α)ε (x , y) . (3)
The transition matrix p0.5ε (x , y) look like this:
Aviv Rotbart (TAU) Diffusion Maps December 2012 19 / 40
![Page 45: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/45.jpg)
Diffusion Maps
The asymptotic behavior, ε −→ 0, generates differentinfinitesimal operator for different values of α:
1 α = 0 is the classical normalized graph Laplacian.2 α = 1 approximates the Laplace-Beltrami operator.3 α = 1
2 approximates the diffusion of the Fokker-Planck equation.
Aviv Rotbart (TAU) Diffusion Maps December 2012 20 / 40
![Page 46: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/46.jpg)
Diffusion Maps
The transition matrix P is conjugate to a symmetric matrix A
a(x , y) =√
d(x)p(x , y)1√
d(y). (4)
A , D 12 PD− 1
2 .The symmetric matrix A has the following spectraldecomposition:
a(x , y) =∑k≥0
λkvk(x)vk(y). (5)
Aviv Rotbart (TAU) Diffusion Maps December 2012 21 / 40
![Page 47: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/47.jpg)
Diffusion Maps
Since P is conjugate to A
φk = D 12 vk , ψk = D− 1
2 vk . (6)
where {φk} and {ψk} are the left and right eigenvectors of P.From the orthonormality of {vi} and Eq. 6 it follows that {φk}and {ψk} are biorthonormal which means 〈φl , ψk〉 = δlk .
This leads to the following eigendecomposition:
pt(x , y) =∑k≥0
λtkψk(x)φk(y). (7)
Because of the fast decay of the spectrum, only a few terms arerequired to achieve sufficient accuracy in the sum.
Aviv Rotbart (TAU) Diffusion Maps December 2012 22 / 40
![Page 48: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/48.jpg)
Diffusion Maps
The family of diffusion maps {Ψt}, is defined by
Ψt(x) = (λt1ψ1(x), λt
2ψ2(x), λt3ψ3(x), · · · ) , (8)
Diffusion maps embeds the dataset into a Euclidean space.
Aviv Rotbart (TAU) Diffusion Maps December 2012 23 / 40
![Page 49: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/49.jpg)
Diffusion Maps
Figure: Top Left: The eigenvectors: ψk(x). Top Right: The eigenvaluesλt
k . Bottom: The transition matrix.
Aviv Rotbart (TAU) Diffusion Maps December 2012 24 / 40
![Page 50: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/50.jpg)
Diffusion Maps
The diffusion distance between two data points xi and xj is theweighted L2 distance
D2t (xi , xj) =
∑xl∈Γ
(pt(xi , xl)− pt(xl , xj))2
φ0(z). (9)
The diffusion distance can be expressed by the eigenvalues andeigenvector in the following way:
D2t (xi , xj) =
∑k≥1
λ2tk (ψk(xi)− ψk(xj))2. (10)
Aviv Rotbart (TAU) Diffusion Maps December 2012 25 / 40
![Page 51: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/51.jpg)
Diffusion Maps
(a) Original dataset (b) Embedded dataset
Aviv Rotbart (TAU) Diffusion Maps December 2012 26 / 40
![Page 52: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/52.jpg)
Diffusion Maps: A simple example
These are a few pictures of a vehicle that belong to a little guy Iknow:
Aviv Rotbart (TAU) Diffusion Maps December 2012 27 / 40
![Page 53: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/53.jpg)
Diffusion Maps: A simple example
What is the main difference between the pictures?Will the Diffusion Maps algorithm find this?
Now apply the Diffusion Maps algorithm to the picture set infollowing way:
Reshaped each picture to be a vector in R600∗800∗3.Built a Gaussian kernel on this high dimensional data.Normalized it, and looked at the valued of the first eigenvector.
Aviv Rotbart (TAU) Diffusion Maps December 2012 28 / 40
![Page 54: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/54.jpg)
The truck
These are the values of the first diffusion maps coordinate.
(c) Before sorting (d) After sorting
Aviv Rotbart (TAU) Diffusion Maps December 2012 29 / 40
![Page 55: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/55.jpg)
The truck
Pictures sorted by the value on the first Diffusion map coordinate:
Aviv Rotbart (TAU) Diffusion Maps December 2012 30 / 40
![Page 56: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/56.jpg)
Toy Exampleunordered
Aviv Rotbart (TAU) Diffusion Maps December 2012 31 / 40
![Page 57: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/57.jpg)
Toy ExampleApplying DM
Aviv Rotbart (TAU) Diffusion Maps December 2012 32 / 40
![Page 58: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/58.jpg)
Toy Exampleordered
Aviv Rotbart (TAU) Diffusion Maps December 2012 33 / 40
![Page 59: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/59.jpg)
Embedded Space
Aviv Rotbart (TAU) Diffusion Maps December 2012 34 / 40
![Page 60: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/60.jpg)
Embedded Space
Aviv Rotbart (TAU) Diffusion Maps December 2012 35 / 40
![Page 61: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/61.jpg)
Embedded Space
Aviv Rotbart (TAU) Diffusion Maps December 2012 36 / 40
![Page 62: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/62.jpg)
Embedded Space
Aviv Rotbart (TAU) Diffusion Maps December 2012 37 / 40
![Page 63: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/63.jpg)
Embedded Space
Aviv Rotbart (TAU) Diffusion Maps December 2012 38 / 40
![Page 64: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/64.jpg)
Acknowledgement
Thanks to Guy Wolf and Neta Rabin for the slides
Aviv Rotbart (TAU) Diffusion Maps December 2012 39 / 40
![Page 65: Diffusion Maps - TAUamir1/COURSE2012-2013/DM_Presentation.pdf · Diffusion Maps Since P is conjugate to A φ k = D 1 2 v k, ψ k = D −1 2 v k. (6) where {φk}and {ψ k}are the](https://reader034.vdocuments.site/reader034/viewer/2022052002/6015652075303f2a780252a6/html5/thumbnails/65.jpg)
Questions
Questions?
Aviv Rotbart (TAU) Diffusion Maps December 2012 40 / 40