![Page 1: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/1.jpg)
Learning the structure of Deep sparse Graphical Model
Ryan Prescott Adams Hanna M Wallach
Zoubin Ghahramani
Presented by Zhengming Xing
Some pictures are directly copied from the paper and Hanna Wallach’s slides
![Page 2: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/2.jpg)
outline
• Introduction
• Finite belief network
• Infinite belief network
• Inference
• Experiment
![Page 3: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/3.jpg)
Introduction Main contribution: combine deep belief network and nonparametric bayesian together.
Main idea: use IBP to learn the structure of the network
Structure of the network include:
Depth
Width
Connectivity
![Page 4: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/4.jpg)
Single layer networkUse Binary matrix to represent the network.
Black refer to 1(two unit were connected)
White refer to 0 (two unit were not connected)
IBP can be used as the prior for infinite columns binary matrix
Z
![Page 5: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/5.jpg)
Review IBP
)(poisson
)1/( nnk
1.First customer tries dishes.
2. Nth customer tries
Tasked dishes K with probability
new dishes))1/(( npoisson
1/ nnk
![Page 6: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/6.jpg)
Multi-layer network
![Page 7: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/7.jpg)
Cascading IBP),( Also parameterize by
Each dishes in the restaurant is also a customer in another Indian buffet process
Each matrix is exchangeable both rows and columns
This chain can reach the state with probability one ( number of unit in layer m)
Properties:
For unit in layer m+1
Expected number of parents:
Expected number of children:
0)( mK
K
k kK
1 1/
)(mK
![Page 8: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/8.jpg)
Sample from the CIBP prior
![Page 9: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/9.jpg)
model)()1(11 )( mmmmm ZWy m refer to the layers and increase upto M.
1)1)/(exp(2(.)
),0(~)( )()()()()(
x
Ny mk
mk
mk
mk
mk
weights bias
Place layer wise Gaussian prior on weights and bias, Gamma prior on noise precision
![Page 10: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/10.jpg)
Inference
Weights, bias, noise variance can be sampled with Gibbs sampler.
![Page 11: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/11.jpg)
Inference( sample Z)Two step:
1.
2.
Sample existing dishes
MH-sample
Add a new unit and, and insert connection to this unit with
For a exist unit remove the connection to this unit with
MH ratio
MH ratio
![Page 12: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/12.jpg)
Experiment result
Olivetti faces
Remove bottom halves of the test image.
![Page 13: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/13.jpg)
Experiment result
MNIST Digits
![Page 14: Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are](https://reader030.vdocuments.site/reader030/viewer/2022032605/56649e715503460f94b6f5bf/html5/thumbnails/14.jpg)
Experiment resultFrey Faces