![Page 1: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/1.jpg)
Generative Restricted Kernel Machines:
towards fully explainable deep learning
Johan Suykens
KU Leuven, ESAT-STADIUSKasteelpark Arenberg 10
B-3001 Leuven (Heverlee), BelgiumEmail: [email protected]://www.esat.kuleuven.be/stadius/
Data Science meetup LLN, Dec 2019
![Page 2: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/2.jpg)
Outline
• Introduction
• Function estimation, model representations, duality
• Neural networks and kernel machines
• Generative models: GAN, RBM, Deep BM
• Restricted kernel machines (RKM), Gen-RKM, and deep learning
• Latent space exploration
• Explainability
• Recent developments
1
![Page 3: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/3.jpg)
Introduction
1
![Page 4: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/4.jpg)
Self-driving cars and neural networks
in the early days of neural networks:
ALVINN (Autonomous Land Vehicle In a Neural Network)
[Pomerleau, Neural Computation 1991]
2
![Page 5: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/5.jpg)
Self-driving cars and deep learning
(27 million connections)
from: [selfdrivingcars.mit.edu (Lex Fridman et al.), 2017]
3
![Page 6: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/6.jpg)
Convolutional neural networks
[LeCun et al., Proc. IEEE 1998]
Further advanced architectures:
Alexnet (2012): 5 convolutional layers, 3 fully connectedVGGnet (2014): 19 layersGoogLeNet (2014): 22 layersResNet (2015): 152 layers
4
![Page 7: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/7.jpg)
Historical context
1942 McCulloch & Pitts: mathematical model for neuron1958 Rosenblatt: perceptron learning1960 Widrow & Hoff: adaline and lms learning rule1969 Minsky & Papert: limitations of perceptron
1986 Rumelhart et al.: error backpropagating neural networks→ booming of neural network universal approximators
1992 Vapnik et al.: support vector machine classifiers→ convex optimization, kernel machines
1998 LeCun et al.: convolutional neural networks2006 Hinton et al.: deep belief networks2010 Bengio et al.: stacked autoencoders
→ booming of deep neural networks
com
putin
g po
wer
4
![Page 8: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/8.jpg)
Different paradigms
Deep
Learning
Neural
Networks
SVM, LS-SVM &
Kernel methods
5
![Page 9: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/9.jpg)
Different paradigms
Deep
Learning
Neural
Networks
SVM, LS-SVM &
Kernel methods
new synergies?
5
![Page 10: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/10.jpg)
Towards a unifying picture
Model
Dual representation
Primal representation
Duality principle
other
Legendre−Fenchel duality
Lagrange duality
Conjugate feature duality
Kernel−based
other
Parametric
linear, polynomial
finite or infinite dictionary
positive definite kernel
tensor kernel
indefinite kernel
symmetric or non−symmetric kernel
(deep) neural network
[Suykens 2017]
6
![Page 11: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/11.jpg)
Function estimation and model representations
6
![Page 12: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/12.jpg)
Linear function estimation (1)
• Given (xi, yi)Ni=1 with xi ∈ R
d, yi ∈ R, consider y = f(x) where f isparametrized as
y = wTx+ b
with y the estimated output of the linear model.
• Consider estimating w, b by
minw,b
1
2wTw + γ
1
2
N∑
i=1
(yi − wTxi − b)2
→ one can directly solve in w, b
7
![Page 13: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/13.jpg)
Linear function estimation (2)
• ... or write as a constrained optimization problem:
minw,b,e
12w
Tw + γ12
∑
i e2i
subject to ei = yi − wTxi − b, i = 1, ..., N
Lagrangian: L(w, b, ei, αi) = 12w
Tw + γ 12
∑
i e2i −
∑
i αi(ei − yi + wTxi + b)
• From optimality conditions:
y =∑
i
αi xTi x+ b
where α, b follows from solving a linear system[
0 1TN1N Ω + I/γ
] [
b
α
]
=
[
0
y
]
with Ωij = xTi xj for i, j = 1, ..., N and y = [y1; ...; yN ].
8
![Page 14: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/14.jpg)
Linear model: solving in primal or dual?
inputs x ∈ Rd, output y ∈ R
training set (xi, yi)Ni=1
(P ) : y = wTx+ b, w ∈ Rd
րModel
ց(D) : y =
∑
i αi xTi x+ b, α ∈ R
N
9
![Page 15: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/15.jpg)
Linear model: solving in primal or dual?
inputs x ∈ Rd, output y ∈ R
training set (xi, yi)Ni=1
(P ) : y = wTx+ b, w ∈ Rd
րModel
ց(D) : y =
∑
i αi xTi x+ b, α ∈ R
N
9
![Page 16: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/16.jpg)
Linear model: solving in primal or dual?
few inputs, many data points: d≪ N
primal : w ∈ Rd
dual: α ∈ RN (large kernel matrix: N ×N)
10
![Page 17: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/17.jpg)
Linear model: solving in primal or dual?
many inputs, few data points: d≫ N
primal: w ∈ Rd
dual : α ∈ RN (small kernel matrix: N ×N)
10
![Page 18: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/18.jpg)
Feature map and kernel
From linear to nonlinear model:
(P ) : y = wTϕ(x) + bր
Modelց
(D) : y =∑
i αiK(xi, x) + b
Mercer theorem:K(x, z) = ϕ(x)Tϕ(z)
Feature map ϕ, Kernel function K(x, z) (e.g. linear, polynomial, RBF, ...)
• SVMs: feature map and positive definite kernel [Cortes & Vapnik, 1995]
• Explicit or implicit choice of the feature map
• Neural networks: consider hidden layer as feature map [Suykens & Vandewalle, 1999]
• Least squares support vector machines [Suykens et al., 2002]: L2 loss and regularization
11
![Page 19: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/19.jpg)
Least Squares Support Vector Machines: “core models”
• Regression
minw,b,e
wTw + γ∑
i
e2i s.t. yi = wTϕ(xi) + b+ ei, ∀i
• Classification
minw,b,e
wTw + γ∑
i
e2i s.t. yi(wTϕ(xi) + b) = 1− ei, ∀i
• Kernel pca (V = I), Kernel spectral clustering (V = D−1)
minw,b,e
−wTw + γ∑
i
vie2i s.t. ei = wTϕ(xi) + b, ∀i
• Kernel canonical correlation analysis/partial least squares
minw,v,b,d,e,r
wTw + vTv + ν∑
i
(ei − ri)2 s.t.
ei = wTϕ(1)(xi) + bri = vTϕ(2)(yi) + d
[Suykens & Vandewalle, 1999; Suykens et al., 2002; Alzate & Suykens, 2010]
12
![Page 20: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/20.jpg)
Sparsity: through regularization or loss function
• through regularization: model y = wTx+ b
min∑
j
|wj|+ γ∑
i
e2i
⇒ sparse w (e.g. Lasso)
• through loss function: model y =∑
i αiK(x, xi) + b
min wTw + γ∑
i
L(ei)
⇒ sparse α (e.g. SVM)
−ε 0 +ε
13
![Page 21: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/21.jpg)
SVMs and neural networks
x
oooo
x
xx
x
x x
x
x
x
oo
oo Input space
Feature space
ϕ(x)
Parametric
Nonparametric
Primal space
Dual space
y = sign[wTϕ(x) + b]
y = sign[∑#sv
i=1 αiyiK(x, xi) + b]
K(xi, xj) = ϕ(xi)Tϕ(xj) (Mercer)
y
y
w1
wnh
α1
α#sv
ϕ1(x)
ϕnh(x)
K(x, x1)
K(x, x#sv)
x
x
14
![Page 22: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/22.jpg)
SVMs and neural networks
x
oooo
x
xx
x
x x
x
x
x
oo
oo Input space
Feature space
ϕ(x)
Parametric
Nonparametric
Primal space
Dual space
y = sign[wTϕ(x) + b]
y = sign[∑#sv
i=1 αiyiK(x, xi) + b]
K(xi, xj) = ϕ(xi)Tϕ(xj) (“Kernel trick”)
y
y
w1
wnh
α1
α#sv
ϕ1(x)
ϕnh(x)
K(x, x1)
K(x, x#sv)
x
x
Par
amet
ricN
on−p
aram
etric
[Suykens et al., 2002]
14
![Page 23: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/23.jpg)
Interpretation of kernel-based models
Decision making: classification problem (e.g. apples versus tomatoes) Inputdata xi ∈ R
d and class labels yi ∈ −1,+1. N training data.
SVM or LS-SVM classifier: given a new x (e.g. ), obtain
y = sign[∑
i
αiyiK(x, xi) + b]
with xi for i = 1, ..., N :
Here K(x, xi) characterizes the similarity between x and xi.The bias term b can be related to prior class probabilities.
15
![Page 24: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/24.jpg)
Kernels
Wide range of positive definite kernel functions possible:
- linear K(x, z) = xTz- polynomial K(x, z) = (η + xTz)d
- radial basis function K(x, z) = exp(−‖x− z‖22/σ2)
- splines- wavelets- string kernel- kernels from graphical models- kernels for dynamical systems- Fisher kernels- graph kernels- data fusion kernels- additive kernels (good for explainability)- other
[Scholkopf & Smola, 2002; Shawe-Taylor & Cristianini, 2004; Jebara et al., 2004; other]
16
![Page 25: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/25.jpg)
Generative models
16
![Page 26: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/26.jpg)
Generative Adversarial Network (GAN)
Generative Adversarial Network (GAN) [Goodfellow et al., 2014]Training of two competing models in a zero-sum game:
(Generator) generate fake output examples from random noise(Discriminator) discriminate between fake examples and real examples.
source: https://deeplearning4j.org/generative-adversarial-network
17
![Page 27: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/27.jpg)
GAN: example on MNIST
MNIST training data:
GAN generated examples:
source: https://www.kdnuggets.com/2016/07/mnist-generative-adversarial-model-keras.html
18
![Page 28: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/28.jpg)
Restricted Boltzmann Machines (RBM)
• Markov random field, bipartite graph, stochastic binary unitsLayer of visible units v and layer of hidden units hNo hidden-to-hidden connections
• Energy:
E(v, h; θ) = −vTWh− cTv − aTh with θ = W, c, a
Joint distribution:
P (v, h; θ) =1
Z(θ)exp(−E(v, h; θ))
with partition function Z(θ) =∑
v
∑
h exp(−E(v, h; θ))
[Hinton, Osindero, Teh, Neural Computation 2006]
19
![Page 29: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/29.jpg)
Restricted Boltzmann Machines (RBM)
• Markov random field, bipartite graph, stochastic binary unitsLayer of visible units v and layer of hidden units hNo hidden-to-hidden connections
• Energy:
E(v, h; θ) = −vTWh− cTv − aTh with θ = W, c, a
Joint distribution:
P (v, h; θ) =1
Z(θ)exp(−E(v, h; θ))
with partition function Z(θ) =∑
v
∑
h exp(−E(v, h; θ))
[Hinton, Osindero, Teh, Neural Computation 2006]
19
![Page 30: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/30.jpg)
RBM and deep learning
RBM
p(v, h) p(v, h1, h2, h3, ...)
[Hinton et al., 2006; Salakhutdinov, 2015]
20
![Page 31: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/31.jpg)
in other words ...
”sandwich”
E = −vTWh
”deep sandwich”
E = −vTW 1h1 − h1TW 2h2 − h2
TW 3h3
21
![Page 32: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/32.jpg)
RBM: example on MNIST
MNIST training data:
Generating new images:
source: https://www.kaggle.com/nicw102168/restricted-boltzmann-machine-rbm-on-mnist
22
![Page 33: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/33.jpg)
Convolutional Deep Belief Networks
Unsupervised Learning of Hierarchical Representations with Convolutional Deep Belief
Networks [Lee et al. 2011]
23
![Page 34: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/34.jpg)
Restricted kernel machines
23
![Page 35: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/35.jpg)
Restricted Kernel Machines (RKM)
• Kernel machine interpretations in terms of visible and hidden units(similar to Restricted Boltzmann Machines (RBM))
• Restricted Kernel Machine (RKM) representations for
– LS-SVM regression/classification– Kernel PCA– Matrix SVD– Parzen-type models– other
• Based on principle of conjugate feature duality (with hidden featurescorresponding to dual variables)
• Deep Restricted Kernel Machines (Deep RKM)
[Suykens, Neural Computation, 2017]
24
![Page 36: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/36.jpg)
Kernel principal component analysis (KPCA)
−1.5 −1 −0.5 0 0.5 1−2.5
−2
−1.5
−1
−0.5
0
0.5
1
1.5
linear PCA
−1.5 −1 −0.5 0 0.5 1−2
−1.5
−1
−0.5
0
0.5
1
1.5
kernel PCA (RBF kernel)
Kernel PCA [Scholkopf et al., 1998]:take eigenvalue decomposition of the kernel matrix
K(x1, x1) ... K(x1, xN)... ...
K(xN , x1) ... K(xN , xN)
(applications in dimensionality reduction and denoising)
25
![Page 37: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/37.jpg)
Kernel PCA: classical LS-SVM approach
• Primal problem: [Suykens et al., 2002]: model-based approach
minw,b,e
1
2wTw −
1
2γ
N∑
i=1
e2i s.t. ei = wTϕ(xi) + b, i = 1, ..., N.
• Dual problem corresponds to kernel PCA
Ω(c)α = λα with λ = 1/γ
with Ω(c)ij = (ϕ(xi)− µϕ)
T (ϕ(xj)− µϕ) the centered kernel matrix
and µϕ = (1/N)∑N
i=1ϕ(xi).
• Interpretation:1. pool of candidate components (objective function equals zero)2. select relevant components
26
![Page 38: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/38.jpg)
From KPCA to RKM representation (1)
Model:
e =W Tϕ(x)objective J= regularization term Tr(W TW )- (1λ) variance term
∑
i eTi ei
↓ use property eTh ≤ 12λe
Te+ λ2h
Th
RKM representation:
e =∑
j hjK(xj, x)
obtain J ≤ J(hi,W )solution from stationary points of J :∂J∂hi
= 0, ∂J∂W = 0
27
![Page 39: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/39.jpg)
From KPCA to RKM representation (2)
• Objective
J =η
2Tr(W TW )−
1
2λ
N∑
i=1
eTi ei s.t. ei =W Tϕ(xi), ∀i
≤ −N∑
i=1
eTi hi +λ
2
N∑
i=1
hTi hi +η
2Tr(W TW ) s.t. ei =W Tϕ(xi), ∀i
= −N∑
i=1
ϕ(xi)TWhi +
λ
2
N∑
i=1
hTi hi +η
2Tr(W TW ) , J
• Stationary points of J(hi,W ):
∂J
∂hi= 0 ⇒ W Tϕ(xi) = λhi, ∀i
∂J
∂W= 0 ⇒ W =
1
η
∑
i
ϕ(xi)hTi
28
![Page 40: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/40.jpg)
From KPCA to RKM representation (3)
• Elimination of W gives the eigenvalue decomposition:
1
ηKHT = HTΛ
where H = [h1...hN ] ∈ Rs×N and Λ = diagλ1, ..., λs with s ≤ N
• Primal and dual model representations
(P )RKM : e =W Tϕ(x)ր
Mց
(D)RKM : e =1
η
∑
j
hjK(xj, x).
29
![Page 41: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/41.jpg)
Deep Restricted Kernel Machines
29
![Page 42: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/42.jpg)
Deep RKM: example
v
h(1)x ϕ1(x)
yy
e(1)ϕ2(h
(1))e(2)h(2)
ϕ3(h(2))
e(3)h(3)
Deep RKM: KPCA + KPCA + LSSVM [Suykens, 2017]
Coupling of RKMs by taking sum of the objectives
Jdeep = J1 + J2 + J3
Multiple levels and multiple layers per level.
30
![Page 43: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/43.jpg)
in more detail ...
v
h(1)x ϕ1(x)
yy
e(1)ϕ2(h
(1))e(2)h(2)
ϕ3(h(2))
e(3)h(3)
Jdeep = −
N∑
i=1
ϕ1(xi)TW1h
(1)i +
λ12
N∑
i=1
h(1)i
Th(1)i +
η12Tr(W T
1 W1)
−
N∑
i=1
ϕ2(h(1)i )TW2h
(2)i +
λ22
N∑
i=1
h(2)i
Th(2)i +
η22Tr(W T
2 W2)
+
N∑
i=1
(yTi − ϕ3(h(2)i )TW3 − bT )h
(3)i −
λ32
N∑
i=1
h(3)i
Th(3)i +
η32Tr(W T
3 W3)
31
![Page 44: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/44.jpg)
Primal and dual model representations
e(1) =W T1 ϕ1(x)
(P )DeepRKM : e(2) =W T2 ϕ2(Λ
−11 e(1))
ր y =W T3 ϕ3(Λ
−12 e(2)) + b
M
ց e(1) = 1η1
∑
j h(1)j K1(xj, x)
(D)DeepRKM : e(2) = 1η2
∑
j h(2)j K2(h
(1)j ,Λ−1
1 e(1))
y = 1η3
∑
j h(3)j K3(h
(2)j ,Λ−1
2 e(2)) + b
The framework can be used for training deep feedforward neural networksand deep kernel machines [Suykens, 2017].
(Other approaches: e.g. kernels for deep learning [Cho & Saul, 2009], mathematics of
the neural response [Smale et al., 2010], deep gaussian processes [Damianou & Lawrence,
2013], convolutional kernel networks [Mairal et al., 2014], multi-layer support vector
machines [Wiering & Schomaker, 2014])
32
![Page 45: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/45.jpg)
Training process
0 100 20010
-1
100
101
102
103
104
105
106
iteration step
Jdeep,P
stab
Objective function (logarithmic scale) during training on the ion data set:
• black color: level 3 objective only
• Jdeep for cstab = 1, 10, 100 (blue, red, magenta color) in stabilization term
33
![Page 46: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/46.jpg)
Generative RKM
33
![Page 47: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/47.jpg)
RKM objective for training and generating
• RBM energy function
E(v, h; θ) = −vTWh− cTv − aTh
with model parameters θ = W, c, a
• RKM ”super-objective” function (for training and for generating)
J(v, h,W ) = −vTWh+ λ2h
Th+ 12v
Tv + η2Tr(W
TW )
Training: clamp v → Jtrain(h,W )Generating: clamp h,W → Jgen(v)
[Schreurs & Suykens, ESANN 2018]
34
![Page 48: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/48.jpg)
Explainable AI: latent space exploration (1)
hidden units: exploring the whole continuum:
-0.15 -0.1 -0.05 0 0.05 0.1
H1
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
H2
0
1
6
h(1,1) = -0.11 h(1,1) = -0.06 h(1,1) = -0.01 h(1,1) = 0.04 h(1,1) = 0.09
h(1,2) = -0.12 h(1,2) = -0.06 h(1,2) = 0 h(1,2) = 0.06 h(1,2) = 0.12
h(1,3) = -0.11 h(1,3) = -0.05 h(1,3) = 0.01 h(1,3) = 0.06 h(1,3) = 0.12
[figures by Joachim Schreurs]
35
![Page 49: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/49.jpg)
Explainable AI: latent space exploration (2)
-0.05 -0.04 -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0.04 0.05
-0.06
-0.04
-0.02
0
0.02
0.04
0.06
C
B
D
A
Yale Face database - generated faces from different regions A,B,C,D
[Winant, Schreurs, Suykens, BNAIC 2019]
36
![Page 50: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/50.jpg)
Tensor-based RKM for Multi-view KPCA
min 〈W,W〉−N∑
i=1
⟨
Φ(i),W⟩
hi+λN∑
i=1
h2i with Φ(i) = ϕ[1](x[1]i )⊗...⊗ϕ[V ](x
[V ]i )
[Houthuys & Suykens, ICANN 2018]
37
![Page 51: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/51.jpg)
Generative RKM (Gen-RKM) (1)
Train:
Generate:
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
38
![Page 52: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/52.jpg)
Analogy with the human brain
eyes earsnose
shared feature representation
multimodal data
39
![Page 53: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/53.jpg)
Gen-RKM (2)
The objective
Jtrain(hi,U ,V ) =∑N
i=1
(
−φ1(xi)TUhi − φ2(yi)
TV hi +λ2h
Ti hi
)
+η12 Tr(U
TU ) + η22 Tr(V
TV )
results for training into the eigenvalue problem
(1
η1K1 +
1
η2K2)H
T = HTΛ
with H = [h1...hN ] and kernel matrices K1,K2 related to φ1, φ2.
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
40
![Page 54: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/54.jpg)
Gen-RKM (3)
Generating data is based on a newly generated h⋆ and the objective
Jgen(φ1(x⋆), ϕ2(y
⋆)) = −φ1(x⋆)TV h⋆−φ2(y
⋆)TUh⋆+1
2φ1(x
⋆)Tφ1(x⋆)+
1
2φ2(y
⋆)Tφ2(y⋆)
giving
φ1(x⋆) =
1
η1
N∑
i=1
φ1(xi)hTi h
⋆, φ2(y⋆) =
1
η2
N∑
i=1
φ2(yi)hTi h
⋆.
For generating x, y one can either work with the kernel smoother or workwith an explicit feature map using a (deep) neural network or CNN.
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
41
![Page 55: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/55.jpg)
Gen-RKM (4)
Fx Fy
H
X Y
U⊤ U V V ⊤
φ1(·) ψ1(·) ψ2(·) φ2(·)
Gen-RKM schematic representation modeling a common subspace H between two data
sources X and Y. The φ1, φ2 are the feature maps (Fx and Fy represent the feature-
spaces) corresponding to the two data sources. While ψ1, ψ2 represent the pre-image
maps. The interconnection matrices U, V model dependencies between latent variables
and the mapped data sources.
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
42
![Page 56: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/56.jpg)
Gen-RKM: implicit feature map
Obtain
kx⋆ =1
η1K1H
⊤h⋆, ky⋆ =1
η2K2H
⊤h⋆,
with kx⋆ = [k(x1,x⋆), . . . , k(xN ,x
⋆)]⊤.
Using the kernel-smoother:
x = ψ1 (φ1(x⋆)) =
∑nrj=1 k1(xj,x
⋆)xj∑nr
j=1 k1(xj,x⋆)
, y = ψ2 (φ2(y⋆)) =
∑nrj=1 k2(yj,y
⋆)yj∑nr
j=1 k2(yj,y⋆)
,
with k1(xi,x⋆) and k2(yi,y
⋆) the scaled similarities between 0 and 1.nr is the number of closest points based on the similarity defined by kernelsk1 and k2.
43
![Page 57: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/57.jpg)
Gen-RKM: explicit feature map
Parametrized feature maps: φθ(·), ψζ(·) (e.g. CNN and transposed CNN).
Overall objective function, using a stabilization mechanism [Suykens, 2017]:
minθ1,θ2,ζ1,ζ2
Jc = Jtrain +cstab2 J 2
train
+cacc2N
(
∑Ni=1
[
L1(x⋆i , ψ1ζ1
(φ1θ1(x⋆i ))) + L2(y
⋆i , ψ2ζ2
(φ2θ2(y⋆i )))
])
with reconstruction errors
L1(x⋆i , ψ1ζ1
(φ1θ1(x⋆i ))) =
1N‖x⋆i − ψ1ζ1
(φ1θ1(x⋆i ))‖
22,
L2(y⋆i , ψ2ζ2
(φ2θ2(y⋆i ))) =
1N‖y⋆i − ψ2ζ2
(φ2θ2(y⋆i ))‖
22
and with Φx = [φ1(x1), . . . , φ1(xN)],Φy = [φ2(y1), . . . , φ2(yN)], U, Vfrom
[
1η1ΦxΦ
⊤x
1η1ΦxΦ
⊤y
1η2ΦyΦ
⊤x
1η2ΦyΦ
⊤y
]
[
UV
]
=
[
UV
]
Λ.
Hence, joint feature learning and subspace learning.
44
![Page 58: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/58.jpg)
Gen-RKM: examples (1)
MNIST Fashion-MNIST
Generated samples from the model using CNN as explicit feature map in the kernel function.
The yellow boxes show training examples and the adjacent boxes show the reconstructed
samples. The other images (columns 3-6) are generated by random sampling from the
fitted distribution over the learned latent variables.
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
45
![Page 59: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/59.jpg)
Gen-RKM: examples (2)
CIFAR-10 CelebA
Generated samples from the model using CNN as explicit feature map in the kernel function.
The yellow boxes show training examples and the adjacent boxes show the reconstructed
samples. The other images (columns 3-6) are generated by random sampling from the
fitted distribution over the learned latent variables.
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
46
![Page 60: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/60.jpg)
Gen-RKM: multi-view generation (1)
CelebA
Multi-view generation on CelebA dataset showing images and attributes
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
47
![Page 61: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/61.jpg)
Gen-RKM: multi-view generation (2)
MNIST: Implicit feature maps with Gaussian kernel + generation by kernel-smoother
MNIST: Explicit feature maps using Convolutional Neural Networks
CIFAR-10: Explicit feature maps using CNNs + Transposed CNNs
Multi-view Generation (images and labels) using implicit and explicit feature maps
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
48
![Page 62: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/62.jpg)
Gen-RKM: latent space exploration (1)
Exploring the learned uncorrelated-features by traversing along the eigenvectors
Explainability: changing one single neuron’s hidden feature changes the hair color while
preserving face structure! [Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
49
![Page 63: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/63.jpg)
Gen-RKM: latent space exploration (2)
MNIST reconstructed images by bilinear-interpolation in latent space
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
50
![Page 64: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/64.jpg)
Gen-RKM: latent space exploration (3)
CelebA reconstructed images by bilinear-interpolation in latent space
[Pandey, Schreurs & Suykens, 2019, arXiv:1906.08144]
51
![Page 65: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/65.jpg)
Future challenges
• efficient algorithms and implementations for large data
• extension to other loss functions and regularization schemes
• multimodal data, tensor models, coupling schemes
• models for deep clustering and semi-supervised learning
• choice kernel functions, invariances and symmetry properties
• deep generative models
• optimal transport
• synergies between neural networks, deep learning and kernel machines
52
![Page 66: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/66.jpg)
Conclusions
• function estimation: parametric versus kernel-based
• primal and dual model representations
• neural network interpretations in primal and dual
• RKM: new connections between RBM, kernel PCA and LS-SVM
• deep kernel machines
• generative models
• explainability: latent space exploration, understanding the role of eachindividual neuron
53
![Page 67: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/67.jpg)
Acknowledgements (1)
• Current and former co-workers at ESAT-STADIUS:
C. Alzate, Y. Chen, J. De Brabanter, K. De Brabanter, B. De Cooman,L. De Lathauwer, H. De Meulemeester, B. De Moor, H. De Plaen, Ph.Dreesen, M. Espinoza, T. Falck, M. Fanuel, Y. Feng, B. Gauthier, X.Huang, L. Houthuys, V. Jumutc, Z. Karevan, R. Langone, F. Liu, R.Mall, S. Mehrkanoon, G. Nisol, M. Orchel, A. Pandey, P. Patrinos, K.Pelckmans, S. RoyChowdhury, S. Salzo, J. Schreurs, M. Signoretto, Q.Tao, F. Tonin, J. Vandewalle, T. Van Gestel, S. Van Huffel, C. Varon,Y. Yang, and others
• Many other people for joint work, discussions, invitations, organizations
• Support from ERC AdG E-DUALITY, ERC AdG A-DATADRIVE-B, KULeuven, OPTEC, IUAP DYSCO, FWO projects, IWT, iMinds, BIL, COST
54
![Page 68: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/68.jpg)
Acknowledgements (2)
55
![Page 69: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/69.jpg)
Acknowledgements (3)
NEW: ERC Advanced Grant E-DUALITYExploring duality for future data-driven modelling
56
![Page 70: Generative Restricted Kernel Machines: towards fully explainable deep learning · 2019. 11. 28. · Data Science meetup LLN, Dec 2019. Outline • Introduction • Function estimation,](https://reader035.vdocuments.site/reader035/viewer/2022071103/5fdd3b4868935b5cd70e1ef7/html5/thumbnails/70.jpg)
Thank you
57