lecture : variational inference - github pages · vi turns inference into optimization. posit a...
TRANSCRIPT
![Page 1: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/1.jpg)
Lecture : Variational Inference
Riashat IslamSlides courtesy of David Blei
Reasoning and Learning LabMcGill University
31st October 2018
![Page 2: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/2.jpg)
Probabilistic Machine Learning
![Page 3: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/3.jpg)
Approximate Inference
![Page 4: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/4.jpg)
Computational Challenge
![Page 5: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/5.jpg)
Probabilistic Model
![Page 6: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/6.jpg)
Variational Inference
![Page 7: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/7.jpg)
Variational Bound
![Page 8: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/8.jpg)
Variational Bound
![Page 9: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/9.jpg)
Motivation : Topic Modelling
![Page 10: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/10.jpg)
Example : Latent Dirichlet Allocation
![Page 11: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/11.jpg)
Example : Latent Dirichlet Allocation
![Page 12: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/12.jpg)
Example : Latent Dirichlet Allocation
![Page 13: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/13.jpg)
LDA as Graphical Model
![Page 14: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/14.jpg)
Posterior Inference in LDA
![Page 15: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/15.jpg)
The Evidence Lower Bound (ELBO)
![Page 16: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/16.jpg)
Stochastic Gradients of ELBO
![Page 17: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/17.jpg)
Variational Inference Recipe
![Page 18: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/18.jpg)
Variational Inference Recipe
![Page 19: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/19.jpg)
Variational Inference Recipe
![Page 20: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/20.jpg)
Variational Inference Recipe
![Page 21: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/21.jpg)
Variational Inference Recipe
![Page 22: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/22.jpg)
Variational Inference Recipe
![Page 23: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/23.jpg)
Variational Inference Recipe
![Page 24: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/24.jpg)
Black Box Variational Inference
![Page 25: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/25.jpg)
Problem of Classical VI
![Page 26: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/26.jpg)
New VI Recipe
![Page 27: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/27.jpg)
Computing Gradients of Expectations
![Page 28: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/28.jpg)
Score Function Gradients of ELBO
![Page 29: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/29.jpg)
Score Function Estimator
![Page 30: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/30.jpg)
Noisy Unbiased Gradients
![Page 31: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/31.jpg)
Black Box Variational Inference
![Page 32: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/32.jpg)
Pathwise Gradients of ELBO
![Page 33: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/33.jpg)
Pathwise Estimator
![Page 34: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/34.jpg)
Pathwise Estimator
![Page 35: Lecture : Variational Inference - GitHub Pages · VI turns inference into optimization. Posit a variational family of distributions over the latent variables, q(z; v) Fit the variational](https://reader036.vdocuments.site/reader036/viewer/2022062306/5ecbbd6355abea0e5d25cce6/html5/thumbnails/35.jpg)
Score Function vs Pathwise Estimator