statistical modeling and analysis of neural data (neu...

27
Lecture 23 Thurs 11/19 VAEs II Statistical modeling and analysis of neural data (NEU 560) Jonathan Pillow Fall 2020

Upload: others

Post on 25-Mar-2021

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Lecture 23 Thurs 11/19

VAEs II

Statistical modeling and analysis of neural data (NEU 560)

Jonathan PillowFall 2020

Page 2: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

math warmupConsider the following function:

1. How would you maximize it for ?

2. How would you maximize it for ?

Page 3: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

math warmupConsider the following function:

1. How would you maximize it for ?

Easy: Monte Carlo integration!

(swap differentiation and integration)

sample from q

average over samples

• use these gradients to climb uphill

Page 4: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

math warmupConsider the following function:

Hmmm…. idea below no longer works

2. How would you maximize it for ?

• the dependence on is inside the samples.• no way to take derivatives w.r.t. !

sample from q

average over samples

Page 5: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

math warmupConsider the following function:

Hmmm…. idea below no longer works

2. How would you maximize it for ?

• the dependence on is inside the samples.• no way to take derivatives w.r.t. !

Solution: the resampling trick

works when q is Gaussian:

mean variance

Page 6: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

The resampling trick:• basic idea: sample from a fixed distribution and then transform the

samples to have the desired distribution

Do NOT:

Instead: sample from standard Gaussian

transform these samples to have the desired distribution

Page 7: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

The resampling trick:• basic idea: sample from a fixed distribution and then transform the

samples to have the desired distribution

Do NOT:

Instead: sample from standard Gaussian

transform these samples to have the desired distribution

Applied to our problem of interest:

a sample from q(x | )• use these gradients to climb uphill

Page 8: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

The resampling trick:

SUMMARY

• use Monte Carlo integration to evaluate integral

• sample from fixed distribution

• transform to get samples of the desired distribution, so dependence on is now explicit!

• take derivative w.r.t.

Page 9: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

probabilistic-PCA model

N (0, I)<latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit><latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit><latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit><latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit>

~z<latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit><latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit><latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit><latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit>

~x<latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit>

latent data~z ⇠ N (0, I)

<latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit><latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit><latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit><latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit>

✏ ⇠ N (0,�2I)<latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit><latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit><latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit><latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit>

(low-D) (hi-D)

Page 10: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

VAE generative model

N (0, I)<latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit><latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit><latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit><latexit sha1_base64="esjwAp714KSkhAv2a3yYX5auRdo=">AAACyXicdZHbbhMxEIad5VSWUwuX3FhESEWCaLdCopcV3ICQqiKRtiibVl5ntrHiw9aeDQmWr3gEbuEFeCPeBiddIZLASJZ+zffP2OMpaykcZtmvTnLt+o2bt7Zup3fu3rv/YHvn4bEzjeXQ50Yae1oyB1Jo6KNACae1BaZKCSfl5M2Cn0zBOmH0R5zXMFTsQotKcIYx9ak4tGo3e/7u2fl2N+tly6CbIm9Fl7RxdL7T+VmMDG8UaOSSOTfIsxqHnlkUXEJIi8ZBzfiEXcAgSs0UuKFfvjjQpzEzopWx8Wiky+zfFZ4p5+aqjE7FcOzW2SL5LzZosNofeqHrBkHzq4uqRlI0dDE+HQkLHOU8CsatiG+lfMws4xg/KU0LDZ+5UYrpkS+EngZ/5l/kIawCtHXw/qxAU4d4BUXLtKuNg1Wb0TCWVfBFFfv7PPi9jUZ/LPhfT9xQ5ItJOZP+cB1PZ5FOgdPZOpi0YBLSuN18fZeb4nivl2e9/MPL7sHrds9b5DF5QnZJTl6RA/KWHJE+4USRb+Q7+ZG8Ty6TWfLlypp02ppHZCWSr78BDuzkWw==</latexit>

~z<latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit><latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit><latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit><latexit sha1_base64="F7a4yIfURwl/DWiC8w6hpefRwMg=">AAACxnicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb00lNVJJJWyqaV15ltTPyxsmfTppYlHoErPAJvxNvgTVeIJDCSpb/m95+xx1PWUjjMsl+95M7de/cf7DxMHz1+8vTZ7t7zkTON5TDkRhp7VjIHUmgYokAJZ7UFpkoJp+X8sOWnC7BOGP0ZlzVMFLvUohKcYUyNigVwenOx288G2Srotsg70SddnFzs9X4WU8MbBRq5ZM6N86zGiWcWBZcQ0qJxUDM+Z5cwjlIzBW7iV88N9HXMTGllbDwa6Sr7d4VnyrmlKqNTMZy5TdYm/8XGDVYfJl7oukHQ/PaiqpEUDW1np1NhgaNcRsG4FfGtlM+YZRzjD6VpoeGKG6WYnvpC6EXw5/5tHsI6QFsH788LNHWIV1C0TLvaOFi3GQ0zWQVfVLG/z4Pf32r0x4L/9RxbFXk7KWfSH2/ixXWk7QqvN8G8A/OQxu3mm7vcFqP9QZ4N8k/v+gcfuz3vkJfkFXlDcvKeHJAjckKGhJMv5Bv5Tn4kR4lOmuTq1pr0upoXZC2Sr78B5k3j8g==</latexit>

~x<latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit>

latent

N (f✓(~z),�2I)

<latexit sha1_base64="r17CA4YOkY+34DtRJcgzhP0PJj8=">AAAC5HicdZHLbhMxFIad4VaGS1NYsrEaIaUSRDMREiwr2MCmKhJpK2WSyON4Eiu+jOwzocHyngU7xJYHYMEWXoW3wZOOEEngSJZ+ne/38eXPS8EtJMmvVnTt+o2bt/Zux3fu3ru/3z54cGZ1ZSgbUC20uciJZYIrNgAOgl2UhhGZC3aeL17V/HzJjOVavYNVyUaSzBQvOCUQWpP2YXZiZLeYZDBnQLrZklH84egJziyfSTLu4zdHk3Yn6SXrwrsibUQHNXU6OWh9y6aaVpIpoIJYO0yTEkaOGOBUMB9nlWUloQsyY8MgFZHMjtz6MR4/Dp0pLrQJSwFed//e4Yi0diXz4JQE5nab1c1/sWEFxYuR46qsgCl6dVBRCQwa1z+Dp9wwCmIVBKGGh7tiOieGUAj/F8eZYu+plpKoqcu4Wno3dk9T7zcBmNI7N85Alz4cgcEQZUtt2aZNKzYXhXdZEea71Lv+zqA/FvivJ0QXeP1SSoQ72cbLy0DrQC+3waIBCx+HdNPtLHfFWb+XJr307bPO8csm5z30CB2iLkrRc3SMXqNTNEAUfUTf0Q/0MyqiT9Hn6MuVNWo1ex6ijYq+/gbUu+5P</latexit><latexit sha1_base64="r17CA4YOkY+34DtRJcgzhP0PJj8=">AAAC5HicdZHLbhMxFIad4VaGS1NYsrEaIaUSRDMREiwr2MCmKhJpK2WSyON4Eiu+jOwzocHyngU7xJYHYMEWXoW3wZOOEEngSJZ+ne/38eXPS8EtJMmvVnTt+o2bt/Zux3fu3ru/3z54cGZ1ZSgbUC20uciJZYIrNgAOgl2UhhGZC3aeL17V/HzJjOVavYNVyUaSzBQvOCUQWpP2YXZiZLeYZDBnQLrZklH84egJziyfSTLu4zdHk3Yn6SXrwrsibUQHNXU6OWh9y6aaVpIpoIJYO0yTEkaOGOBUMB9nlWUloQsyY8MgFZHMjtz6MR4/Dp0pLrQJSwFed//e4Yi0diXz4JQE5nab1c1/sWEFxYuR46qsgCl6dVBRCQwa1z+Dp9wwCmIVBKGGh7tiOieGUAj/F8eZYu+plpKoqcu4Wno3dk9T7zcBmNI7N85Alz4cgcEQZUtt2aZNKzYXhXdZEea71Lv+zqA/FvivJ0QXeP1SSoQ72cbLy0DrQC+3waIBCx+HdNPtLHfFWb+XJr307bPO8csm5z30CB2iLkrRc3SMXqNTNEAUfUTf0Q/0MyqiT9Hn6MuVNWo1ex6ijYq+/gbUu+5P</latexit><latexit sha1_base64="r17CA4YOkY+34DtRJcgzhP0PJj8=">AAAC5HicdZHLbhMxFIad4VaGS1NYsrEaIaUSRDMREiwr2MCmKhJpK2WSyON4Eiu+jOwzocHyngU7xJYHYMEWXoW3wZOOEEngSJZ+ne/38eXPS8EtJMmvVnTt+o2bt/Zux3fu3ru/3z54cGZ1ZSgbUC20uciJZYIrNgAOgl2UhhGZC3aeL17V/HzJjOVavYNVyUaSzBQvOCUQWpP2YXZiZLeYZDBnQLrZklH84egJziyfSTLu4zdHk3Yn6SXrwrsibUQHNXU6OWh9y6aaVpIpoIJYO0yTEkaOGOBUMB9nlWUloQsyY8MgFZHMjtz6MR4/Dp0pLrQJSwFed//e4Yi0diXz4JQE5nab1c1/sWEFxYuR46qsgCl6dVBRCQwa1z+Dp9wwCmIVBKGGh7tiOieGUAj/F8eZYu+plpKoqcu4Wno3dk9T7zcBmNI7N85Alz4cgcEQZUtt2aZNKzYXhXdZEea71Lv+zqA/FvivJ0QXeP1SSoQ72cbLy0DrQC+3waIBCx+HdNPtLHfFWb+XJr307bPO8csm5z30CB2iLkrRc3SMXqNTNEAUfUTf0Q/0MyqiT9Hn6MuVNWo1ex6ijYq+/gbUu+5P</latexit><latexit sha1_base64="r17CA4YOkY+34DtRJcgzhP0PJj8=">AAAC5HicdZHLbhMxFIad4VaGS1NYsrEaIaUSRDMREiwr2MCmKhJpK2WSyON4Eiu+jOwzocHyngU7xJYHYMEWXoW3wZOOEEngSJZ+ne/38eXPS8EtJMmvVnTt+o2bt/Zux3fu3ru/3z54cGZ1ZSgbUC20uciJZYIrNgAOgl2UhhGZC3aeL17V/HzJjOVavYNVyUaSzBQvOCUQWpP2YXZiZLeYZDBnQLrZklH84egJziyfSTLu4zdHk3Yn6SXrwrsibUQHNXU6OWh9y6aaVpIpoIJYO0yTEkaOGOBUMB9nlWUloQsyY8MgFZHMjtz6MR4/Dp0pLrQJSwFed//e4Yi0diXz4JQE5nab1c1/sWEFxYuR46qsgCl6dVBRCQwa1z+Dp9wwCmIVBKGGh7tiOieGUAj/F8eZYu+plpKoqcu4Wno3dk9T7zcBmNI7N85Alz4cgcEQZUtt2aZNKzYXhXdZEea71Lv+zqA/FvivJ0QXeP1SSoQ72cbLy0DrQC+3waIBCx+HdNPtLHfFWb+XJr307bPO8csm5z30CB2iLkrRc3SMXqNTNEAUfUTf0Q/0MyqiT9Hn6MuVNWo1ex6ijYq+/gbUu+5P</latexit>

f✓<latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit>

data~z ⇠ N (0, I)

<latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit><latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit><latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit><latexit sha1_base64="aYM7TR2bG/cXp6vZrB5v/5ACUi8=">AAAC2HicdZHLbhMxFIad4VaGS1OQ2LCxiJCKBNFMhQTLCjawqYpE2opMGnkcT2PFN9lnQoOxxA6x5RFYAi/E2+BJR4gkcCRLv873+/jyl0ZwB1n2q5Ncunzl6rWt6+mNm7dub3d37hw5XVvKBlQLbU9K4pjgig2Ag2AnxjIiS8GOy9nLhh/PmXVcq7ewMGwkyZniFacEYmvcvVfMGcUfcOG4xMWBlbvZ49eP8Ljby/rZsvCmyFvRQ20djnc634uJprVkCqggzg3zzMDIEwucChbSonbMEDojZ2wYpSKSuZFfPiDgh7EzwZW2cSnAy+7fOzyRzi1kGZ2SwNSts6b5LzasoXo+8lyZGpiiFwdVtcCgcfMbeMItoyAWURBqebwrplNiCYX4Z2laKPaeaimJmviCq3nwp/5JHsIqAGuC96cFaBPiERgsUc5ox1ZtWrGpqIIvqjjf58HvbQz6Y4H/emJEkTcvpUT4g3U8P4+0ifR8HcxaMAtpTDdfz3JTHO3186yfv3na23/R5ryF7qMHaBfl6BnaR6/QIRogij6ib+gH+pm8Sz4ln5MvF9ak0+65i1Yq+fobzEHpRw==</latexit>

~x = f✓(~z) + ~✏<latexit sha1_base64="S3/+j2DDixCBzXVsiRg0ntCTugw=">AAAC63icdZHLahRBFIZr2ltsb4ku3RQOQoI4dAdBN0LQjasQwUkC05OhuuZ0ppi6NFWnx4xFP4MLd+LWjXu3+hy+jdWdRpwZPVDw839/nbqcvJTCYZL86kVXrl67fmPrZnzr9p2797Z37h87U1kOQ26ksac5cyCFhiEKlHBaWmAql3CSz183/GQB1gmj3+GyhLFi51oUgjMM1mR7L1sApxf0JS0mGc4A2W7rfNijT2irMiidkE22nwyStuimSDvRJ10dTXZ637Kp4ZUCjVwy50ZpUuLYM4uCS6jjrHJQMj5n5zAKUjMFbuzbN9X0cXCmtDA2LI20df/e4ZlybqnykFQMZ26dNea/2KjC4sXYC11WCJpfHlRUkqKhzQfRqbDAUS6DYNyKcFfKZ8wyjuEb4zjT8J4bpZie+kzoRe3P/NO0rlcB2rL2/ixDU9bhCIqWaVcaB6sxo2Emi9pnRejv09rvbzT6E8H/Zg6tCrx5KWfSH67jxUWg7ZTXwbwD8zoO003XZ7kpjvcHaTJI3z7rH7zq5rxFHpJHZJek5Dk5IG/IERkSTj6S7+QH+Rmp6FP0OfpyGY163Z4HZKWir78BWZDxMw==</latexit><latexit sha1_base64="S3/+j2DDixCBzXVsiRg0ntCTugw=">AAAC63icdZHLahRBFIZr2ltsb4ku3RQOQoI4dAdBN0LQjasQwUkC05OhuuZ0ppi6NFWnx4xFP4MLd+LWjXu3+hy+jdWdRpwZPVDw839/nbqcvJTCYZL86kVXrl67fmPrZnzr9p2797Z37h87U1kOQ26ksac5cyCFhiEKlHBaWmAql3CSz183/GQB1gmj3+GyhLFi51oUgjMM1mR7L1sApxf0JS0mGc4A2W7rfNijT2irMiidkE22nwyStuimSDvRJ10dTXZ637Kp4ZUCjVwy50ZpUuLYM4uCS6jjrHJQMj5n5zAKUjMFbuzbN9X0cXCmtDA2LI20df/e4ZlybqnykFQMZ26dNea/2KjC4sXYC11WCJpfHlRUkqKhzQfRqbDAUS6DYNyKcFfKZ8wyjuEb4zjT8J4bpZie+kzoRe3P/NO0rlcB2rL2/ixDU9bhCIqWaVcaB6sxo2Emi9pnRejv09rvbzT6E8H/Zg6tCrx5KWfSH67jxUWg7ZTXwbwD8zoO003XZ7kpjvcHaTJI3z7rH7zq5rxFHpJHZJek5Dk5IG/IERkSTj6S7+QH+Rmp6FP0OfpyGY163Z4HZKWir78BWZDxMw==</latexit><latexit sha1_base64="S3/+j2DDixCBzXVsiRg0ntCTugw=">AAAC63icdZHLahRBFIZr2ltsb4ku3RQOQoI4dAdBN0LQjasQwUkC05OhuuZ0ppi6NFWnx4xFP4MLd+LWjXu3+hy+jdWdRpwZPVDw839/nbqcvJTCYZL86kVXrl67fmPrZnzr9p2797Z37h87U1kOQ26ksac5cyCFhiEKlHBaWmAql3CSz183/GQB1gmj3+GyhLFi51oUgjMM1mR7L1sApxf0JS0mGc4A2W7rfNijT2irMiidkE22nwyStuimSDvRJ10dTXZ637Kp4ZUCjVwy50ZpUuLYM4uCS6jjrHJQMj5n5zAKUjMFbuzbN9X0cXCmtDA2LI20df/e4ZlybqnykFQMZ26dNea/2KjC4sXYC11WCJpfHlRUkqKhzQfRqbDAUS6DYNyKcFfKZ8wyjuEb4zjT8J4bpZie+kzoRe3P/NO0rlcB2rL2/ixDU9bhCIqWaVcaB6sxo2Emi9pnRejv09rvbzT6E8H/Zg6tCrx5KWfSH67jxUWg7ZTXwbwD8zoO003XZ7kpjvcHaTJI3z7rH7zq5rxFHpJHZJek5Dk5IG/IERkSTj6S7+QH+Rmp6FP0OfpyGY163Z4HZKWir78BWZDxMw==</latexit><latexit sha1_base64="S3/+j2DDixCBzXVsiRg0ntCTugw=">AAAC63icdZHLahRBFIZr2ltsb4ku3RQOQoI4dAdBN0LQjasQwUkC05OhuuZ0ppi6NFWnx4xFP4MLd+LWjXu3+hy+jdWdRpwZPVDw839/nbqcvJTCYZL86kVXrl67fmPrZnzr9p2797Z37h87U1kOQ26ksac5cyCFhiEKlHBaWmAql3CSz183/GQB1gmj3+GyhLFi51oUgjMM1mR7L1sApxf0JS0mGc4A2W7rfNijT2irMiidkE22nwyStuimSDvRJ10dTXZ637Kp4ZUCjVwy50ZpUuLYM4uCS6jjrHJQMj5n5zAKUjMFbuzbN9X0cXCmtDA2LI20df/e4ZlybqnykFQMZ26dNea/2KjC4sXYC11WCJpfHlRUkqKhzQfRqbDAUS6DYNyKcFfKZ8wyjuEb4zjT8J4bpZie+kzoRe3P/NO0rlcB2rL2/ixDU9bhCIqWaVcaB6sxo2Emi9pnRejv09rvbzT6E8H/Zg6tCrx5KWfSH67jxUWg7ZTXwbwD8zoO003XZ7kpjvcHaTJI3z7rH7zq5rxFHpJHZJek5Dk5IG/IERkSTj6S7+QH+Rmp6FP0OfpyGY163Z4HZKWir78BWZDxMw==</latexit>

✏ ⇠ N (0,�2I)<latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit><latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit><latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit><latexit sha1_base64="h578i619FhvO+GEqDTk0pXOJDEw=">AAAC43icdZFNi1MxFIbT69d4/ZiOLt1EizCClnuLoMtBN7oZRrAzA71tSdNz29B8keTWKSFrF+7ErX9AcKu/xX9jbucittUDgTfneZOTkzPRnFmXZb9ayZWr167f2LuZ3rp95+5+++DeqVWVodCniitzPiEWOJPQd8xxONcGiJhwOJssXtf8bAnGMiXfu5WGoSAzyUpGiYupcfthAdoyriQuLBO4ODbiMHtab2aCjHr47ZNxu5N1s3XgXZE3ooOaOBkftL4VU0UrAdJRTqwd5Jl2Q0+MY5RDSIvKgiZ0QWYwiFISAXbo170E/DhmprhUJi7p8Dr79wlPhLUrMYlOQdzcbrM6+S82qFz5cuiZ1JUDSS8LlRXHTuH6Y/CUGaCOr6Ig1LD4VkznxBDq4velaSHhA1VCEDn1BZPL4Ef+WR7CJnBGB+9HhVM6xBLYGSKtVhY2bUrCnJfBF2W83+fB93Yu+mNx//XEWUVed0oJ98fbeHkR6RIovtgGiwYsQhqnm2/Pclec9rp51s3fPe8cvWrmvIceoEfoEOXoBTpCb9AJ6iOKPqLv6Af6mUDyKfmcfLm0Jq3mzH20EcnX3+dC7fQ=</latexit>

Recognition q�(~z|~x) ⇡ p(~z|~x, ✓)<latexit sha1_base64="nIeubnmDPYqx73BRsmoXEPSivXY=">AAAC/XicdZFNbxMxEIad5assH03hyMUiQmoliHYrJJC4VHDhVBWJtJWyaeR1vI0Vf2HPhgSz4rdw4IIQV34CVzjyb3C2K0Q2MJKlV/O8M7ZnciO4gyT51YkuXb5y9drW9fjGzVu3t7s7d46dLi1lA6qFtqc5cUxwxQbAQbBTYxmRuWAn+ezFip/MmXVcq9ewNGwkybniBacEQmrcffZmnJkp383mjOJ3+D2uxWIPZ8QYqxfYtNFDnMGUAdnD424v6Sd14E2RNqKHmjga73Q+ZxNNS8kUUEGcG6aJgZEnFjgVrIqz0jFD6Iycs2GQikjmRr7+ZYUfhMwEF9qGowDX2b8rPJHOLWUenJLA1LXZKvkvNiyheDryXJkSmKIXFxWlwKDxamR4wi2jIJZBEGp5eCumU2IJhTDYOM4Ue0u1lERNfMbVvPJn/lFaVesArKm8P8tAmypcgcES5Yx2bN2mFZuKovJZEfr7tPL7G43+WOC/nkMrA1/9lBLhD9t4vgi0XmYbzBowq+Kw3bS9y01xvN9Pk3766nHv4Hmz5y10D91HuyhFT9ABeomO0ABR9BF9Rz/Qz+hD9Cn6En29sEadpuYuWovo229UzPfY</latexit><latexit sha1_base64="nIeubnmDPYqx73BRsmoXEPSivXY=">AAAC/XicdZFNbxMxEIad5assH03hyMUiQmoliHYrJJC4VHDhVBWJtJWyaeR1vI0Vf2HPhgSz4rdw4IIQV34CVzjyb3C2K0Q2MJKlV/O8M7ZnciO4gyT51YkuXb5y9drW9fjGzVu3t7s7d46dLi1lA6qFtqc5cUxwxQbAQbBTYxmRuWAn+ezFip/MmXVcq9ewNGwkybniBacEQmrcffZmnJkp383mjOJ3+D2uxWIPZ8QYqxfYtNFDnMGUAdnD424v6Sd14E2RNqKHmjga73Q+ZxNNS8kUUEGcG6aJgZEnFjgVrIqz0jFD6Iycs2GQikjmRr7+ZYUfhMwEF9qGowDX2b8rPJHOLWUenJLA1LXZKvkvNiyheDryXJkSmKIXFxWlwKDxamR4wi2jIJZBEGp5eCumU2IJhTDYOM4Ue0u1lERNfMbVvPJn/lFaVesArKm8P8tAmypcgcES5Yx2bN2mFZuKovJZEfr7tPL7G43+WOC/nkMrA1/9lBLhD9t4vgi0XmYbzBowq+Kw3bS9y01xvN9Pk3766nHv4Hmz5y10D91HuyhFT9ABeomO0ABR9BF9Rz/Qz+hD9Cn6En29sEadpuYuWovo229UzPfY</latexit><latexit sha1_base64="nIeubnmDPYqx73BRsmoXEPSivXY=">AAAC/XicdZFNbxMxEIad5assH03hyMUiQmoliHYrJJC4VHDhVBWJtJWyaeR1vI0Vf2HPhgSz4rdw4IIQV34CVzjyb3C2K0Q2MJKlV/O8M7ZnciO4gyT51YkuXb5y9drW9fjGzVu3t7s7d46dLi1lA6qFtqc5cUxwxQbAQbBTYxmRuWAn+ezFip/MmXVcq9ewNGwkybniBacEQmrcffZmnJkp383mjOJ3+D2uxWIPZ8QYqxfYtNFDnMGUAdnD424v6Sd14E2RNqKHmjga73Q+ZxNNS8kUUEGcG6aJgZEnFjgVrIqz0jFD6Iycs2GQikjmRr7+ZYUfhMwEF9qGowDX2b8rPJHOLWUenJLA1LXZKvkvNiyheDryXJkSmKIXFxWlwKDxamR4wi2jIJZBEGp5eCumU2IJhTDYOM4Ue0u1lERNfMbVvPJn/lFaVesArKm8P8tAmypcgcES5Yx2bN2mFZuKovJZEfr7tPL7G43+WOC/nkMrA1/9lBLhD9t4vgi0XmYbzBowq+Kw3bS9y01xvN9Pk3766nHv4Hmz5y10D91HuyhFT9ABeomO0ABR9BF9Rz/Qz+hD9Cn6En29sEadpuYuWovo229UzPfY</latexit><latexit sha1_base64="nIeubnmDPYqx73BRsmoXEPSivXY=">AAAC/XicdZFNbxMxEIad5assH03hyMUiQmoliHYrJJC4VHDhVBWJtJWyaeR1vI0Vf2HPhgSz4rdw4IIQV34CVzjyb3C2K0Q2MJKlV/O8M7ZnciO4gyT51YkuXb5y9drW9fjGzVu3t7s7d46dLi1lA6qFtqc5cUxwxQbAQbBTYxmRuWAn+ezFip/MmXVcq9ewNGwkybniBacEQmrcffZmnJkp383mjOJ3+D2uxWIPZ8QYqxfYtNFDnMGUAdnD424v6Sd14E2RNqKHmjga73Q+ZxNNS8kUUEGcG6aJgZEnFjgVrIqz0jFD6Iycs2GQikjmRr7+ZYUfhMwEF9qGowDX2b8rPJHOLWUenJLA1LXZKvkvNiyheDryXJkSmKIXFxWlwKDxamR4wi2jIJZBEGp5eCumU2IJhTDYOM4Ue0u1lERNfMbVvPJn/lFaVesArKm8P8tAmypcgcES5Yx2bN2mFZuKovJZEfr7tPL7G43+WOC/nkMrA1/9lBLhD9t4vgi0XmYbzBowq+Kw3bS9y01xvN9Pk3766nHv4Hmz5y10D91HuyhFT9ABeomO0ABR9BF9Rz/Qz+hD9Cn6En29sEadpuYuWovo229UzPfY</latexit>

~x<latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit><latexit sha1_base64="ufaMB61CbQdk8cpcYUHFVjid7vA=">AAACxnicdZHNahsxEMfl7Ve6/UqaYy+iptBLzW4ItMeQXHIKCdROwOsErTwbq9bHIs06MULQR+g1eYS+Ud+msrOU2m4HBH/m95+RRlPWUjjMsl+d5NHjJ0+fbT1PX7x89frN9s7bgTON5dDnRhp7UTIHUmjoo0AJF7UFpkoJ5+X0aMHPZ2CdMPorzmsYKXatRSU4w5gaFDPg9PZqu5v1smXQTZG3okvaOL3a6fwsxoY3CjRyyZwb5lmNI88sCi4hpEXjoGZ8yq5hGKVmCtzIL58b6IeYGdPK2Hg00mX27wrPlHNzVUanYjhx62yR/BcbNlh9GXmh6wZB84eLqkZSNHQxOx0LCxzlPArGrYhvpXzCLOMYfyhNCw033CjF9NgXQs+Cv/Sf8hBWAdo6eH9ZoKlDvIKiZdrVxsGqzWiYyCr4oor9fR783kajPxb8r+fEqsgXk3Im/ck6nt1GulzhOpi2YBrSuN18fZebYrDXy7NefrbfPThs97xF3pH35CPJyWdyQI7JKekTTr6RH+SO3CfHiU6a5ObBmnTaml2yEsn33+GT4/A=</latexit>

= N (g�1(~x), H�2(~x))<latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit>

g�1<latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit>

H�2<latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit>

mean

cov

q�(~z|~x)<latexit sha1_base64="GPYctA341qFxPkhGbxClFWJiSdc=">AAAC7HicdZLLbhMxFIad4VaGWwJLNhYRUlkQzVRIsKxgw6oqEmkrxWnkcTyNFd+wPdMkxm/BDnVbHoSn4G1w0hEiEziSpV/n+31sn+NCc2Zdlv3qJLdu37l7b+9++uDho8dPur2nJ1ZVhtAhUVyZswJbypmkQ8ccp2faUCwKTk+L+Yc1P62psUzJz26p6VjgC8lKRrCLqUm392WC9Izto3oFv0JUL15Nuv1skG0C7oq8EX3QxPGk1/mJpopUgkpHOLZ2lGfajT02jhFOQ4oqSzUmc3xBR1FKLKgd+83dA3wZM1NYKhOXdHCT/XuHx8LapSiiU2A3s222Tv6LjSpXvht7JnXlqCQ3B5UVh07BdSPglBlKHF9GgYlh8a6QzLDBxMV2pSmS9JIoIbCcesRkHfy5f52HsA2c0cH7c+SUDvEI6AyWVitLt21K0hkvg0dlrO/z4A92Cv2xuP96joyIfP1Sgrk/auN6EWlNCVy0waoBqzaYN2Ae0jj2vD3kXXFyMMizQf7pTf/wffMB9sBz8ALsgxy8BYfgIzgGQ0DAJbgC1+BHIpNvyffk6saadJo9z8BWJNe/ATBt8wI=</latexit><latexit sha1_base64="GPYctA341qFxPkhGbxClFWJiSdc=">AAAC7HicdZLLbhMxFIad4VaGWwJLNhYRUlkQzVRIsKxgw6oqEmkrxWnkcTyNFd+wPdMkxm/BDnVbHoSn4G1w0hEiEziSpV/n+31sn+NCc2Zdlv3qJLdu37l7b+9++uDho8dPur2nJ1ZVhtAhUVyZswJbypmkQ8ccp2faUCwKTk+L+Yc1P62psUzJz26p6VjgC8lKRrCLqUm392WC9Izto3oFv0JUL15Nuv1skG0C7oq8EX3QxPGk1/mJpopUgkpHOLZ2lGfajT02jhFOQ4oqSzUmc3xBR1FKLKgd+83dA3wZM1NYKhOXdHCT/XuHx8LapSiiU2A3s222Tv6LjSpXvht7JnXlqCQ3B5UVh07BdSPglBlKHF9GgYlh8a6QzLDBxMV2pSmS9JIoIbCcesRkHfy5f52HsA2c0cH7c+SUDvEI6AyWVitLt21K0hkvg0dlrO/z4A92Cv2xuP96joyIfP1Sgrk/auN6EWlNCVy0waoBqzaYN2Ae0jj2vD3kXXFyMMizQf7pTf/wffMB9sBz8ALsgxy8BYfgIzgGQ0DAJbgC1+BHIpNvyffk6saadJo9z8BWJNe/ATBt8wI=</latexit><latexit sha1_base64="GPYctA341qFxPkhGbxClFWJiSdc=">AAAC7HicdZLLbhMxFIad4VaGWwJLNhYRUlkQzVRIsKxgw6oqEmkrxWnkcTyNFd+wPdMkxm/BDnVbHoSn4G1w0hEiEziSpV/n+31sn+NCc2Zdlv3qJLdu37l7b+9++uDho8dPur2nJ1ZVhtAhUVyZswJbypmkQ8ccp2faUCwKTk+L+Yc1P62psUzJz26p6VjgC8lKRrCLqUm392WC9Izto3oFv0JUL15Nuv1skG0C7oq8EX3QxPGk1/mJpopUgkpHOLZ2lGfajT02jhFOQ4oqSzUmc3xBR1FKLKgd+83dA3wZM1NYKhOXdHCT/XuHx8LapSiiU2A3s222Tv6LjSpXvht7JnXlqCQ3B5UVh07BdSPglBlKHF9GgYlh8a6QzLDBxMV2pSmS9JIoIbCcesRkHfy5f52HsA2c0cH7c+SUDvEI6AyWVitLt21K0hkvg0dlrO/z4A92Cv2xuP96joyIfP1Sgrk/auN6EWlNCVy0waoBqzaYN2Ae0jj2vD3kXXFyMMizQf7pTf/wffMB9sBz8ALsgxy8BYfgIzgGQ0DAJbgC1+BHIpNvyffk6saadJo9z8BWJNe/ATBt8wI=</latexit><latexit sha1_base64="GPYctA341qFxPkhGbxClFWJiSdc=">AAAC7HicdZLLbhMxFIad4VaGWwJLNhYRUlkQzVRIsKxgw6oqEmkrxWnkcTyNFd+wPdMkxm/BDnVbHoSn4G1w0hEiEziSpV/n+31sn+NCc2Zdlv3qJLdu37l7b+9++uDho8dPur2nJ1ZVhtAhUVyZswJbypmkQ8ccp2faUCwKTk+L+Yc1P62psUzJz26p6VjgC8lKRrCLqUm392WC9Izto3oFv0JUL15Nuv1skG0C7oq8EX3QxPGk1/mJpopUgkpHOLZ2lGfajT02jhFOQ4oqSzUmc3xBR1FKLKgd+83dA3wZM1NYKhOXdHCT/XuHx8LapSiiU2A3s222Tv6LjSpXvht7JnXlqCQ3B5UVh07BdSPglBlKHF9GgYlh8a6QzLDBxMV2pSmS9JIoIbCcesRkHfy5f52HsA2c0cH7c+SUDvEI6AyWVitLt21K0hkvg0dlrO/z4A92Cv2xuP96joyIfP1Sgrk/auN6EWlNCVy0waoBqzaYN2Ae0jj2vD3kXXFyMMizQf7pTf/wffMB9sBz8ALsgxy8BYfgIzgGQ0DAJbgC1+BHIpNvyffk6saadJo9z8BWJNe/ATBt8wI=</latexit>

(low-D) (hi-D)

Page 11: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Variational EM: (in theory)

train recognition networks and

to minimize

H�2<latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit>

E-step: KL

�q�(~z|~x)|p(~z|~x, ✓)

�<latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit><latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit><latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit><latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit>

g�1<latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit>

M-step: train generative network to maximizef✓<latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit>

Zq�(~z|~x) log p(~x|~z)d~z

<latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit><latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit><latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit><latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit>

⇡ 1k

kX

j=1

log p(~x|~zj)<latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit><latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit><latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit><latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit>

~zj ⇠ q�(~z|~x)<latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit>

⇡ 1k

X1

2�2 ||~x� f✓(~zj)||2<latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit><latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit><latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit><latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit>

Page 12: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Variational EM: (in theory)

train recognition networks and

to minimize

H�2<latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit>

E-step: KL

�q�(~z|~x)|p(~z|~x, ✓)

�<latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit><latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit><latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit><latexit sha1_base64="TKRsiIDf3/gCL0KgE9oS5LBLHC0=">AAADDnicdZLLbhMxFIad4VbCpSks2VhESIkE0UxBgmUFGyRQVSTSVorTyON4MlY8trE9QxJ33oEtL8IOdcGmPAJvgycdITKBI1n+fb7/+B4rzowNw1+t4Nr1Gzdv7dxu37l77/5uZ+/BsZG5JnRIJJf6NMaGcibo0DLL6anSFGcxpyfx/E3FTwqqDZPio10qOs7wTLCEEWx9atJ5/u49itkM9j5NkEpZDxUreA5Rsej7TlXDcz94imxKLe7DytufdLrhIFwH3BZRLbqgjqPJXusHmkqSZ1RYwrExoyhUduywtoxwWrZRbqjCZI5ndOSlwBk1Y7c+XQmf+MwUJlL7JixcZ/+ucDgzZpnF3plhm5omq5L/YqPcJq/GjgmVWyrI1UJJzqGVsLoqOGWaEsuXXmCimd8rJCnWmFh/oe02EvQzkVmGxdQhJorSnblnUVluAqtV6dwZslKVfgloNRZGSUM3bVLQlCelQ4mf30Wl29+a6I/F/tdzqDPPq5MSzN1hExcLTwtK4KIJVjVYNcG8BvOy7Z89aj7ytjjeH0ThIPrwonvwuv4AO+AReAx6IAIvwQF4C47AEBDwFVyAS/Az+BJ8C74HF1fWoFXXPAQbEVz+Bu5D/8c=</latexit>

g�1<latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit>

M-step: train generative network to maximizef✓<latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit>

Zq�(~z|~x) log p(~x|~z)d~z

<latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit><latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit><latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit><latexit sha1_base64="9KgQvTZ7Bzvm9xL/mRyoaLZYnAQ=">AAADBnicdZJNb9MwHMbd8DbCWwdHLhYV0iZElUxIcJzgwmkaEt0m1V3lOE5r1W/YTmjr5c6VL8IN7TrEp+Db4HYRoin8pURPnt/jt7+Tac6sS5JfnejGzVu37+zcje/df/DwUXf38YlVpSF0QBRX5izDlnIm6cAxx+mZNhSLjNPTbPZuxU8raixT8qNbaDoSeCJZwQh2wRp3XyAmHfw0RnrK9lC1vEDVfB8iriZQh+85vIDB3Yd5eI+7vaSfrAtui7QRPdDU8Xi38xPlipSCSkc4tnaYJtqNPDaOEU7rGJWWakxmeEKHQUosqB359alq+Dw4OSyUCU/Y49r9e4THwtqFyEJSYDe1bbYy/8WGpSvejDyTunRUkuuFipJDp+CqRTBnhhLHF0FgYljYKyRTbDBxoZFxjCT9TJQQWOY+dK+q/bl/mdb1JnBG196fI6d0HZaAzmBptbJ0M6YknfKi9qgI8/u09gdbE/2JuP9mjowIfHVSgrk/auNqHmhFCZy3wbIByzaYNWBWx+Ha0/Ylb4uTg36a9NMPr3qHb5sfYAc8Bc/AHkjBa3AI3oNjMAAEfAWX4Ar8iL5E36Lv0eV1NOo0Y56AjYqufgP/3/1O</latexit>

⇡ 1k

kX

j=1

log p(~x|~zj)<latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit><latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit><latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit><latexit sha1_base64="8fmBgNxEUW4kMG1rYiyQXUJQqDI=">AAADEXicdZLLbhMxFIad4VbCLYUlG4sIqSyIZqoK2CBVsGFVFYm0leIkchxP4o5vsj1DEuOnYMuLsENlWV6At8FJB0QSOJKlX+f77WOf45HmzLo0/dlIrl2/cfPWzu3mnbv37j9o7T48sao0hHaJ4sqcjbClnEnadcxxeqYNxWLE6emoeLvkpxU1lin5wc017Qs8kSxnBLuYGrZeIKy1UTOIXG4w8VnwRYDIlmLoz19nYVBAxNUE6j1UzeAniKrF8PwZHLbaaSddBdwWWS3aoI7j4W7jOxorUgoqHeHY2l6Watf32DhGOA1NVFqqMSnwhPailFhQ2/erBwb4NGbGMFcmLungKvv3Do+FtXMxik6B3dRusmXyX6xXuvxV3zOpS0cluSqUlxw6BZfdgmNmKHF8HgUmhsW7QjLFsVEu9rTZRJJ+JEoILMceMVkFP/DPsxDWgTM6eD9ATukQS0BnsLRaWbpuU5JOeR48+j2I/a2D/ljcfz1HRkS+fCnB3B9t4moWaUUJnG2CRQ0Wm6CoQRGacezZ5pC3xcl+J0s72fuD9uGb+gPsgMfgCdgDGXgJDsE7cAy6gIAv4AJcgh/J5+Rr8i25uLImjXrPI7AWyeUvMwoCJA==</latexit>

~zj ⇠ q�(~z|~x)<latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit>

⇡ 1k

X1

2�2 ||~x� f✓(~zj)||2<latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit><latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit><latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit><latexit sha1_base64="0VsvqTfsztWheTWPfOB2M5YpQtk=">AAADJHicdZLLbhMxFIad4VbCLYUlG4sIqSwazURIsKxgA5uqSKStFCeR43gSE99ke4Ykjl+HLQ/Blh3qgk15FZx0VDUJHMnSr//7PR6f46HmzLo0/V1Lbt2+c/fe3v36g4ePHj9p7D89taowhHaI4sqcD7GlnEnaccxxeq4NxWLI6dlw+n7Fz0pqLFPys5tr2hN4LFnOCHbRGjQ+Iqy1UTOIXG4w8Vnw0wCRLcQNp40sGwvcb4flEpUzeAjzAXIT6vABKheDL6+Wy3570GimrXRdcFdklWiCqk4G+7UfaKRIIah0hGNru1mqXc9j4xjhNNRRYanGZIrHtBulxILanl/fOcCX0RnBXJm4pINr9+YOj4W1czGMSYHdxG6zlfkv1i1c/rbnmdSFo5JcHZQXHDoFVw2EI2YocXweBSaGxX+FZIJjp1xsc72OJP1KlBBYjjxisgy+7w+zEDaBMzp430dO6RCPgM5gabWydDOmJJ3wPHh0PYmdD11H3H8zx0ZEvropwdwfb+NyFmlJCZxtg0UFFttgWoFpqMexZ9tD3hWn7VaWtrJPr5tH76oHsAeegxfgAGTgDTgCH8AJ6AACvoMLcAn+JN+Sn8mv5OIqmtSqPc/ARiWXfwE68grA</latexit>

OK

Page 13: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

M step is fine, but E-step is not!(Not obvious how to minimize KL term for and )g�1

<latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit><latexit sha1_base64="mOeej9YPmG12HI+l6M/xxSmInl4=">AAAC5HicdZLLahsxFIbl6S2d3pJ22Y2oKXRTMxMC7TKkm65CCnUS8DhGI2tsYd2QzjhxhN4gu5Bt8jB5ir5NZWco9bg9IPg5368j6RyVRnAHWfarkzx6/OTps63n6YuXr16/2d55e+x0bSnrUy20PS2JY4Ir1gcOgp0ay4gsBTspZ9+W/GTOrONa/YSFYUNJJopXnBKIqcFk5Asz5aM84NF2N+tlq8CbIm9EFzVxNNrp3BdjTWvJFFBBnBvkmYGhJxY4FSykRe2YIXRGJmwQpSKSuaFf3TngjzEzxpW2cSnAq+zfOzyRzi1kGZ2SwNS12TL5Lzaoofo69FyZGpiiDwdVtcCg8bIBeMwtoyAWURBqebwrplNiCYXYpjQtFDunWkqixr7gah78mf+ch7AOwJrg/VkB2oR4BAZLlDPasXWbVmwqquCLKtb3efC7G4X+WOC/nkMrI1++lBLhD9t4fhHpnFF80QaXDbhsg1kDZiGNY8/bQ94Ux7u9POvlP/a6+wfNB9hC79EH9Anl6AvaR9/REeojijS6QbfoLqmSq+Q6uXmwJp1mzzu0Fsntb6Iq8JA=</latexit>

H�2<latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit><latexit sha1_base64="v9StdXA+KDraW2aKGgdJIw16qW0=">AAAC43icdZLLbhMxFIadKZcy3NqyZGMRIbEhmokqwbKCTVdVkUhbkUkjj3OmseKbbE9oavkJ2FXdti/DU/A2OOkIkQkcydKv8/0+9vFxqTmzLst+dZKtBw8fPd5+kj599vzFy53dvROrakNhQBVX5qwkFjiTMHDMcTjTBogoOZyWs89LfjoHY5mSX91Cw0iQC8kqRomLqW+HY1/oKRv3w3inm/WyVeBNkTeii5o4Hu92fhYTRWsB0lFOrB3mmXYjT4xjlENIi9qCJnRGLmAYpSQC7Mivrhzw25iZ4EqZuKTDq+zfOzwR1i5EGZ2CuKlts2XyX2xYu+rjyDOpaweS3h9U1Rw7hZf94wkzQB1fREGoYfGumE6JIdTFV0rTQsJ3qoQgcuILJufBn/v3eQjrwBkdvD8vnNIhHoGdIdJqZWHdpiRMeRV8UcX6Pg++v1Hoj8X913NkROTLTinh/qiN55eRzoHiyza4asBVG8waMAtpHHveHvKmOOn38qyXf9nvHnxqPsA2eo3eoHcoRx/QATpEx2iAKJLoBt2iuwSSH8l1cnNvTTrNnldoLZLb396e8Eg=</latexit>

Alternate approach: use “reparametrization trick”

~zj ⇠ q�(~z|~x)<latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit><latexit sha1_base64="eCe2XHFVeaOB+n5RGexjNfFveok=">AAAC9XicdZLLbhMxGIWd4VaGS1PYwcYiQioLopkKCZYVbFhVRSJtpUw68jiexsQ3bM80ibHEi7BD3bbv0KfgbXDSESIT+CVLR+c7/n0tFKPGJsmvTnTr9p2797buxw8ePnq83d15cmRkpTEZYMmkPimQIYwKMrDUMnKiNEG8YOS4mH5Y8uOaaEOl+Gzniow4OhO0pBjZYOXdZ1m9yL/AzFAOv+aZmtDd4HzL6tmrvNtL+smq4KZIG9EDTR3mO53rbCxxxYmwmCFjhmmi7MghbSlmxMdZZYhCeIrOyDBIgTgxI7c6hIcvgzOGpdRhCAtX7t8zHOLGzHkRkhzZiWmzpfkvNqxs+W7kqFCVJQLfLFRWDFoJlzcCx1QTbNk8CIQ1DXuFeII0wjbcWxxngpxjyTkSY5dRUXt36l6n3q8Dq5V37jSzUvmwBLQaCaOkIesxKciEld5lZejvUu/2Nhr9idj/Zg40D3x5UoyYO2jjehZoTTCctcGiAYs2mDZg6uPw7Gn7kTfF0V4/Tfrppze9/ffNB9gCz8ELsAtS8Bbsg4/gEAwABt/BBbgEV9F59CP6GV3cRKNOM+cpWKvo8jf7N/cW</latexit>

Reparametrization trick:

to get:

do: ~✏j ⇠ N (0, I)<latexit sha1_base64="j/QKF+l/F5RhStE2oPhwsDL6Qp8=">AAAC+nicdZLLbhMxFIad4VaGWwoSGzYWEVKRIJqpkGBZwQY2VZFIWylOI49zpnHry8j2DE1dv0x3qCx5DF6At8FJR4gkcCRLv873+xzbx0UluHVZ9quT3Lh56/adjbvpvfsPHj7qbj7et7o2DAZMC20OC2pBcAUDx52Aw8oAlYWAg+L0w5wfNGAs1+qLm1UwkvRY8ZIz6mJq3H1KGqjs+AQTyyUmu0ZuZa8+vRx3e1k/WwReF3kreqiNvfFm5yeZaFZLUI4Jau0wzyo38tQ4zgSElNQWKspO6TEMo1RUgh35xQUCfhEzE1xqE5dyeJH9e4en0tqZLKJTUje1q2ye/Bcb1q58N/JcVbUDxa4blbXATuP5a+AJN8CcmEVBmeHxrJhNqaHMxTdLU6LgK9NSUjXxhKsm+CP/Og9hGThTBe+PiNNViC2wM1TZSltYtmkFU1EGT8pY3+fBb68V+mNx//XECUU+vymjwu+u4uYs0gYYPlsF5y04XwVx/i0iUXKhVUjj/PPVaa+L/e1+nvXzz296O+/bn7CBnqHnaAvl6C3aQR/RHhoghi7QJbpC35OL5DL5llxdW5NOu+cJWorkx28WNPhp</latexit><latexit sha1_base64="j/QKF+l/F5RhStE2oPhwsDL6Qp8=">AAAC+nicdZLLbhMxFIad4VaGWwoSGzYWEVKRIJqpkGBZwQY2VZFIWylOI49zpnHry8j2DE1dv0x3qCx5DF6At8FJR4gkcCRLv873+xzbx0UluHVZ9quT3Lh56/adjbvpvfsPHj7qbj7et7o2DAZMC20OC2pBcAUDx52Aw8oAlYWAg+L0w5wfNGAs1+qLm1UwkvRY8ZIz6mJq3H1KGqjs+AQTyyUmu0ZuZa8+vRx3e1k/WwReF3kreqiNvfFm5yeZaFZLUI4Jau0wzyo38tQ4zgSElNQWKspO6TEMo1RUgh35xQUCfhEzE1xqE5dyeJH9e4en0tqZLKJTUje1q2ye/Bcb1q58N/JcVbUDxa4blbXATuP5a+AJN8CcmEVBmeHxrJhNqaHMxTdLU6LgK9NSUjXxhKsm+CP/Og9hGThTBe+PiNNViC2wM1TZSltYtmkFU1EGT8pY3+fBb68V+mNx//XECUU+vymjwu+u4uYs0gYYPlsF5y04XwVx/i0iUXKhVUjj/PPVaa+L/e1+nvXzz296O+/bn7CBnqHnaAvl6C3aQR/RHhoghi7QJbpC35OL5DL5llxdW5NOu+cJWorkx28WNPhp</latexit><latexit sha1_base64="j/QKF+l/F5RhStE2oPhwsDL6Qp8=">AAAC+nicdZLLbhMxFIad4VaGWwoSGzYWEVKRIJqpkGBZwQY2VZFIWylOI49zpnHry8j2DE1dv0x3qCx5DF6At8FJR4gkcCRLv873+xzbx0UluHVZ9quT3Lh56/adjbvpvfsPHj7qbj7et7o2DAZMC20OC2pBcAUDx52Aw8oAlYWAg+L0w5wfNGAs1+qLm1UwkvRY8ZIz6mJq3H1KGqjs+AQTyyUmu0ZuZa8+vRx3e1k/WwReF3kreqiNvfFm5yeZaFZLUI4Jau0wzyo38tQ4zgSElNQWKspO6TEMo1RUgh35xQUCfhEzE1xqE5dyeJH9e4en0tqZLKJTUje1q2ye/Bcb1q58N/JcVbUDxa4blbXATuP5a+AJN8CcmEVBmeHxrJhNqaHMxTdLU6LgK9NSUjXxhKsm+CP/Og9hGThTBe+PiNNViC2wM1TZSltYtmkFU1EGT8pY3+fBb68V+mNx//XECUU+vymjwu+u4uYs0gYYPlsF5y04XwVx/i0iUXKhVUjj/PPVaa+L/e1+nvXzz296O+/bn7CBnqHnaAvl6C3aQR/RHhoghi7QJbpC35OL5DL5llxdW5NOu+cJWorkx28WNPhp</latexit><latexit sha1_base64="j/QKF+l/F5RhStE2oPhwsDL6Qp8=">AAAC+nicdZLLbhMxFIad4VaGWwoSGzYWEVKRIJqpkGBZwQY2VZFIWylOI49zpnHry8j2DE1dv0x3qCx5DF6At8FJR4gkcCRLv873+xzbx0UluHVZ9quT3Lh56/adjbvpvfsPHj7qbj7et7o2DAZMC20OC2pBcAUDx52Aw8oAlYWAg+L0w5wfNGAs1+qLm1UwkvRY8ZIz6mJq3H1KGqjs+AQTyyUmu0ZuZa8+vRx3e1k/WwReF3kreqiNvfFm5yeZaFZLUI4Jau0wzyo38tQ4zgSElNQWKspO6TEMo1RUgh35xQUCfhEzE1xqE5dyeJH9e4en0tqZLKJTUje1q2ye/Bcb1q58N/JcVbUDxa4blbXATuP5a+AJN8CcmEVBmeHxrJhNqaHMxTdLU6LgK9NSUjXxhKsm+CP/Og9hGThTBe+PiNNViC2wM1TZSltYtmkFU1EGT8pY3+fBb68V+mNx//XECUU+vymjwu+u4uYs0gYYPlsF5y04XwVx/i0iUXKhVUjj/PPVaa+L/e1+nvXzz296O+/bn7CBnqHnaAvl6C3aQR/RHhoghi7QJbpC35OL5DL5llxdW5NOu+cJWorkx28WNPhp</latexit>

~zj ⇠z }| {N (g�1(~x), H�2(~x))

<latexit sha1_base64="tg3nOmBrTLYBQwjah5+nxYhG2qs=">AAADGHicdZJNaxNBGMcn60trfEv16GUwCClo2A2CvQhFLz2VCqYtZNJldjKbjJmXZWZ2TTLMF/HqFxE8SK/15LdxkixiEn1g4c//95+359ms4MzYOP7ViG7dvnN3b/9e8/6Dh48etw6enBtVakL7RHGlLzNsKGeS9i2znF4WmmKRcXqRTd8v+UVFtWFKfrTzgg4FHkuWM4JtsNLWW1Qt0k8QGSYgUiGZaUyoQ6dadMapQ8WEpYnvoGp2+BKe1EZvbRz6tNWOu/Gq4K5IatEGdZ2lB43vaKRIKai0hGNjBklc2KHD2jLCqW+i0tACkyke00GQEgtqhm71Tg9fBGcEc6XDJy1cuX+vcFgYMxdZSApsJ2abLc1/sUFp86OhY7IoLZVkfVBecmgVXDYNjpimxPJ5EJhoFu4KyQSHTtnQ2mYTSfqZKCGwHDnEZOXdlXuVeL8JrC68c1fIqsKHI6DVWJpCGboZU5JOeO4dysP+LvGut7PRn4j9bybML/DlSwnm7nQbV7NAK0rgbBssarDYBtMaTH0zjD3ZHvKuOO91k7ibfHjdPn5X/wD74Bl4DjogAW/AMTgBZ6APCPgKrsEN+Bl9ib5FP6LrdTRq1Guego2Kbn4DuXgE1A==</latexit><latexit sha1_base64="tg3nOmBrTLYBQwjah5+nxYhG2qs=">AAADGHicdZJNaxNBGMcn60trfEv16GUwCClo2A2CvQhFLz2VCqYtZNJldjKbjJmXZWZ2TTLMF/HqFxE8SK/15LdxkixiEn1g4c//95+359ms4MzYOP7ViG7dvnN3b/9e8/6Dh48etw6enBtVakL7RHGlLzNsKGeS9i2znF4WmmKRcXqRTd8v+UVFtWFKfrTzgg4FHkuWM4JtsNLWW1Qt0k8QGSYgUiGZaUyoQ6dadMapQ8WEpYnvoGp2+BKe1EZvbRz6tNWOu/Gq4K5IatEGdZ2lB43vaKRIKai0hGNjBklc2KHD2jLCqW+i0tACkyke00GQEgtqhm71Tg9fBGcEc6XDJy1cuX+vcFgYMxdZSApsJ2abLc1/sUFp86OhY7IoLZVkfVBecmgVXDYNjpimxPJ5EJhoFu4KyQSHTtnQ2mYTSfqZKCGwHDnEZOXdlXuVeL8JrC68c1fIqsKHI6DVWJpCGboZU5JOeO4dysP+LvGut7PRn4j9bybML/DlSwnm7nQbV7NAK0rgbBssarDYBtMaTH0zjD3ZHvKuOO91k7ibfHjdPn5X/wD74Bl4DjogAW/AMTgBZ6APCPgKrsEN+Bl9ib5FP6LrdTRq1Guego2Kbn4DuXgE1A==</latexit><latexit sha1_base64="tg3nOmBrTLYBQwjah5+nxYhG2qs=">AAADGHicdZJNaxNBGMcn60trfEv16GUwCClo2A2CvQhFLz2VCqYtZNJldjKbjJmXZWZ2TTLMF/HqFxE8SK/15LdxkixiEn1g4c//95+359ms4MzYOP7ViG7dvnN3b/9e8/6Dh48etw6enBtVakL7RHGlLzNsKGeS9i2znF4WmmKRcXqRTd8v+UVFtWFKfrTzgg4FHkuWM4JtsNLWW1Qt0k8QGSYgUiGZaUyoQ6dadMapQ8WEpYnvoGp2+BKe1EZvbRz6tNWOu/Gq4K5IatEGdZ2lB43vaKRIKai0hGNjBklc2KHD2jLCqW+i0tACkyke00GQEgtqhm71Tg9fBGcEc6XDJy1cuX+vcFgYMxdZSApsJ2abLc1/sUFp86OhY7IoLZVkfVBecmgVXDYNjpimxPJ5EJhoFu4KyQSHTtnQ2mYTSfqZKCGwHDnEZOXdlXuVeL8JrC68c1fIqsKHI6DVWJpCGboZU5JOeO4dysP+LvGut7PRn4j9bybML/DlSwnm7nQbV7NAK0rgbBssarDYBtMaTH0zjD3ZHvKuOO91k7ibfHjdPn5X/wD74Bl4DjogAW/AMTgBZ6APCPgKrsEN+Bl9ib5FP6LrdTRq1Guego2Kbn4DuXgE1A==</latexit><latexit sha1_base64="tg3nOmBrTLYBQwjah5+nxYhG2qs=">AAADGHicdZJNaxNBGMcn60trfEv16GUwCClo2A2CvQhFLz2VCqYtZNJldjKbjJmXZWZ2TTLMF/HqFxE8SK/15LdxkixiEn1g4c//95+359ms4MzYOP7ViG7dvnN3b/9e8/6Dh48etw6enBtVakL7RHGlLzNsKGeS9i2znF4WmmKRcXqRTd8v+UVFtWFKfrTzgg4FHkuWM4JtsNLWW1Qt0k8QGSYgUiGZaUyoQ6dadMapQ8WEpYnvoGp2+BKe1EZvbRz6tNWOu/Gq4K5IatEGdZ2lB43vaKRIKai0hGNjBklc2KHD2jLCqW+i0tACkyke00GQEgtqhm71Tg9fBGcEc6XDJy1cuX+vcFgYMxdZSApsJ2abLc1/sUFp86OhY7IoLZVkfVBecmgVXDYNjpimxPJ5EJhoFu4KyQSHTtnQ2mYTSfqZKCGwHDnEZOXdlXuVeL8JrC68c1fIqsKHI6DVWJpCGboZU5JOeO4dysP+LvGut7PRn4j9bybML/DlSwnm7nQbV7NAK0rgbBssarDYBtMaTH0zjD3ZHvKuOO91k7ibfHjdPn5X/wD74Bl4DjogAW/AMTgBZ6APCPgKrsEN+Bl9ib5FP6LrdTRq1Guego2Kbn4DuXgE1A==</latexit>

~zj = (H�2(~x))12 ~✏+ g�1(~x)

<latexit sha1_base64="bkD/q8DcfG3NhRG+K+fR2Nt58KI=">AAADLnicdZLLahsxGIXl6SWpe3PaZTeiJuBQamZModkUQrsJXYQU6iRg2YMsa2w1GklImqltoXfKg3Qd6KKkyz5GZXso9aT9QXD4v6PbkcaKM2Pj+HsjunP33v2d3QfNh48eP3na2nt2ZmShCe0TyaW+GGNDORO0b5nl9EJpivMxp+fjyw8rfl5SbZgUn+1C0WGOp4JljGAbWmnrIyqX6Rf4DiJOM9s5Th1SM5b2fAeV8wOk2XRmD0YOZRoTl3jX8x6ikioDX8FpZU425rTVjrvxuuBtkVSiDao6Tfca39BEkiKnwhKOjRkksbJDh7VlhFPfRIWhCpNLPKWDIAXOqRm69aU93A+dCcykDkNYuO7+PcPh3JhFPg7OHNuZqbNV819sUNjscOiYUIWlgmw2ygoOrYSrBOGEaUosXwSBiWbhrJDMcMjHhpybTSToVyLzHIuJQ0yU3o3c68T7bWC18s6NkJUqJLoPrcbCKGnotk0KOuOZ38q/ttAfi/2v50Tnga9uSjB3J3VczgMtKYHzOlhWYFkH4QNUCAXJuBS+Gd4/qb/2bXHW6yZxN/n0pn30vvoJu+AFeAk6IAFvwRE4BqegDwi4AtfgBvyMrqLr6Ed0s7FGjWrOc7BV0a/fkjwN0Q==</latexit><latexit sha1_base64="bkD/q8DcfG3NhRG+K+fR2Nt58KI=">AAADLnicdZLLahsxGIXl6SWpe3PaZTeiJuBQamZModkUQrsJXYQU6iRg2YMsa2w1GklImqltoXfKg3Qd6KKkyz5GZXso9aT9QXD4v6PbkcaKM2Pj+HsjunP33v2d3QfNh48eP3na2nt2ZmShCe0TyaW+GGNDORO0b5nl9EJpivMxp+fjyw8rfl5SbZgUn+1C0WGOp4JljGAbWmnrIyqX6Rf4DiJOM9s5Th1SM5b2fAeV8wOk2XRmD0YOZRoTl3jX8x6ikioDX8FpZU425rTVjrvxuuBtkVSiDao6Tfca39BEkiKnwhKOjRkksbJDh7VlhFPfRIWhCpNLPKWDIAXOqRm69aU93A+dCcykDkNYuO7+PcPh3JhFPg7OHNuZqbNV819sUNjscOiYUIWlgmw2ygoOrYSrBOGEaUosXwSBiWbhrJDMcMjHhpybTSToVyLzHIuJQ0yU3o3c68T7bWC18s6NkJUqJLoPrcbCKGnotk0KOuOZ38q/ttAfi/2v50Tnga9uSjB3J3VczgMtKYHzOlhWYFkH4QNUCAXJuBS+Gd4/qb/2bXHW6yZxN/n0pn30vvoJu+AFeAk6IAFvwRE4BqegDwi4AtfgBvyMrqLr6Ed0s7FGjWrOc7BV0a/fkjwN0Q==</latexit><latexit sha1_base64="bkD/q8DcfG3NhRG+K+fR2Nt58KI=">AAADLnicdZLLahsxGIXl6SWpe3PaZTeiJuBQamZModkUQrsJXYQU6iRg2YMsa2w1GklImqltoXfKg3Qd6KKkyz5GZXso9aT9QXD4v6PbkcaKM2Pj+HsjunP33v2d3QfNh48eP3na2nt2ZmShCe0TyaW+GGNDORO0b5nl9EJpivMxp+fjyw8rfl5SbZgUn+1C0WGOp4JljGAbWmnrIyqX6Rf4DiJOM9s5Th1SM5b2fAeV8wOk2XRmD0YOZRoTl3jX8x6ikioDX8FpZU425rTVjrvxuuBtkVSiDao6Tfca39BEkiKnwhKOjRkksbJDh7VlhFPfRIWhCpNLPKWDIAXOqRm69aU93A+dCcykDkNYuO7+PcPh3JhFPg7OHNuZqbNV819sUNjscOiYUIWlgmw2ygoOrYSrBOGEaUosXwSBiWbhrJDMcMjHhpybTSToVyLzHIuJQ0yU3o3c68T7bWC18s6NkJUqJLoPrcbCKGnotk0KOuOZ38q/ttAfi/2v50Tnga9uSjB3J3VczgMtKYHzOlhWYFkH4QNUCAXJuBS+Gd4/qb/2bXHW6yZxN/n0pn30vvoJu+AFeAk6IAFvwRE4BqegDwi4AtfgBvyMrqLr6Ed0s7FGjWrOc7BV0a/fkjwN0Q==</latexit><latexit sha1_base64="bkD/q8DcfG3NhRG+K+fR2Nt58KI=">AAADLnicdZLLahsxGIXl6SWpe3PaZTeiJuBQamZModkUQrsJXYQU6iRg2YMsa2w1GklImqltoXfKg3Qd6KKkyz5GZXso9aT9QXD4v6PbkcaKM2Pj+HsjunP33v2d3QfNh48eP3na2nt2ZmShCe0TyaW+GGNDORO0b5nl9EJpivMxp+fjyw8rfl5SbZgUn+1C0WGOp4JljGAbWmnrIyqX6Rf4DiJOM9s5Th1SM5b2fAeV8wOk2XRmD0YOZRoTl3jX8x6ikioDX8FpZU425rTVjrvxuuBtkVSiDao6Tfca39BEkiKnwhKOjRkksbJDh7VlhFPfRIWhCpNLPKWDIAXOqRm69aU93A+dCcykDkNYuO7+PcPh3JhFPg7OHNuZqbNV819sUNjscOiYUIWlgmw2ygoOrYSrBOGEaUosXwSBiWbhrJDMcMjHhpybTSToVyLzHIuJQ0yU3o3c68T7bWC18s6NkJUqJLoPrcbCKGnotk0KOuOZ38q/ttAfi/2v50Tnga9uSjB3J3VczgMtKYHzOlhWYFkH4QNUCAXJuBS+Gd4/qb/2bXHW6yZxN/n0pn30vvoJu+AFeAk6IAFvwRE4BqegDwi4AtfgBvyMrqLr6Ed0s7FGjWrOc7BV0a/fkjwN0Q==</latexit>

Page 14: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

ELBO (alternate form used in VAE)

log p(X) =NX

i=1

log p(~xi)<latexit sha1_base64="OB4wpVzyHjwCAzxiVzuMAMj4dIU=">AAADCnicdZLLbhMxFIad4dIyXJrCko1FBGoXRDMVEt1UqmDDKioSaSNlkpHjeBKrvsn2DEktv0GfpjtUlix5Ad4GJx0hMoEjWfp1vt/H9jmeKEaNTZJfreje/QcPd3YfxY+fPH22195/fm5kqTHpY8mkHkyQIYwK0rfUMjJQmiA+YeRicvlxxS8qog2V4otdKjLiaCZoQTGyIZW332RMzqA6GBzCE5iZkueOnqR+3IM1yKpFTg/zdifpJuuA2yKtRQfUcZbvt35mU4lLToTFDBkzTBNlRw5pSzEjPs5KQxTCl2hGhkEKxIkZufWDPHwdMlNYSB2WsHCd/XuHQ9yYJZ8EJ0d2bppslfwXG5a2OB45KlRpicB3BxUlg1bCVXfglGqCLVsGgbCm4a4Qz5FG2IYexnEmyFcsOUdi6jIqKu/G7m3q/SawWnnnxpmVyocjoNVIGCUN2bRJQeas8C4rQn2Xene0VeiPxf7X09M88NVLMWKu18TVItCKYLhogqsaXDUBUaZGWZCUSeHjMP+0Oe1tcX7UTZNu+vld5/RD/RN2wUvwChyAFLwHp+ATOAN9gME1uAG34Ht0Hd1E36LbO2vUqve8ABsR/fgNOrH+Ug==</latexit><latexit sha1_base64="OB4wpVzyHjwCAzxiVzuMAMj4dIU=">AAADCnicdZLLbhMxFIad4dIyXJrCko1FBGoXRDMVEt1UqmDDKioSaSNlkpHjeBKrvsn2DEktv0GfpjtUlix5Ad4GJx0hMoEjWfp1vt/H9jmeKEaNTZJfreje/QcPd3YfxY+fPH22195/fm5kqTHpY8mkHkyQIYwK0rfUMjJQmiA+YeRicvlxxS8qog2V4otdKjLiaCZoQTGyIZW332RMzqA6GBzCE5iZkueOnqR+3IM1yKpFTg/zdifpJuuA2yKtRQfUcZbvt35mU4lLToTFDBkzTBNlRw5pSzEjPs5KQxTCl2hGhkEKxIkZufWDPHwdMlNYSB2WsHCd/XuHQ9yYJZ8EJ0d2bppslfwXG5a2OB45KlRpicB3BxUlg1bCVXfglGqCLVsGgbCm4a4Qz5FG2IYexnEmyFcsOUdi6jIqKu/G7m3q/SawWnnnxpmVyocjoNVIGCUN2bRJQeas8C4rQn2Xene0VeiPxf7X09M88NVLMWKu18TVItCKYLhogqsaXDUBUaZGWZCUSeHjMP+0Oe1tcX7UTZNu+vld5/RD/RN2wUvwChyAFLwHp+ATOAN9gME1uAG34Ht0Hd1E36LbO2vUqve8ABsR/fgNOrH+Ug==</latexit><latexit sha1_base64="OB4wpVzyHjwCAzxiVzuMAMj4dIU=">AAADCnicdZLLbhMxFIad4dIyXJrCko1FBGoXRDMVEt1UqmDDKioSaSNlkpHjeBKrvsn2DEktv0GfpjtUlix5Ad4GJx0hMoEjWfp1vt/H9jmeKEaNTZJfreje/QcPd3YfxY+fPH22195/fm5kqTHpY8mkHkyQIYwK0rfUMjJQmiA+YeRicvlxxS8qog2V4otdKjLiaCZoQTGyIZW332RMzqA6GBzCE5iZkueOnqR+3IM1yKpFTg/zdifpJuuA2yKtRQfUcZbvt35mU4lLToTFDBkzTBNlRw5pSzEjPs5KQxTCl2hGhkEKxIkZufWDPHwdMlNYSB2WsHCd/XuHQ9yYJZ8EJ0d2bppslfwXG5a2OB45KlRpicB3BxUlg1bCVXfglGqCLVsGgbCm4a4Qz5FG2IYexnEmyFcsOUdi6jIqKu/G7m3q/SawWnnnxpmVyocjoNVIGCUN2bRJQeas8C4rQn2Xene0VeiPxf7X09M88NVLMWKu18TVItCKYLhogqsaXDUBUaZGWZCUSeHjMP+0Oe1tcX7UTZNu+vld5/RD/RN2wUvwChyAFLwHp+ATOAN9gME1uAG34Ht0Hd1E36LbO2vUqve8ABsR/fgNOrH+Ug==</latexit><latexit sha1_base64="OB4wpVzyHjwCAzxiVzuMAMj4dIU=">AAADCnicdZLLbhMxFIad4dIyXJrCko1FBGoXRDMVEt1UqmDDKioSaSNlkpHjeBKrvsn2DEktv0GfpjtUlix5Ad4GJx0hMoEjWfp1vt/H9jmeKEaNTZJfreje/QcPd3YfxY+fPH22195/fm5kqTHpY8mkHkyQIYwK0rfUMjJQmiA+YeRicvlxxS8qog2V4otdKjLiaCZoQTGyIZW332RMzqA6GBzCE5iZkueOnqR+3IM1yKpFTg/zdifpJuuA2yKtRQfUcZbvt35mU4lLToTFDBkzTBNlRw5pSzEjPs5KQxTCl2hGhkEKxIkZufWDPHwdMlNYSB2WsHCd/XuHQ9yYJZ8EJ0d2bppslfwXG5a2OB45KlRpicB3BxUlg1bCVXfglGqCLVsGgbCm4a4Qz5FG2IYexnEmyFcsOUdi6jIqKu/G7m3q/SawWnnnxpmVyocjoNVIGCUN2bRJQeas8C4rQn2Xene0VeiPxf7X09M88NVLMWKu18TVItCKYLhogqsaXDUBUaZGWZCUSeHjMP+0Oe1tcX7UTZNu+vld5/RD/RN2wUvwChyAFLwHp+ATOAN9gME1uAG34Ht0Hd1E36LbO2vUqve8ABsR/fgNOrH+Ug==</latexit>

�NX

i=1

Zq�(~zi|~xi) log

p(~xi, ~zi|✓)q�(~zi|~xi)

�d~zi

<latexit sha1_base64="9+Yq9slQ2HSOR1wRtP14trMz9T4=">AAADW3icdZLfbtMwFMadhj8jdNCBuOLGokLqJJiSCQlukCa44aoaEt0m1W3kuk5jzbEz28naGr8hL8DF3gW3jRBtx5Eiffp+34njczIpOdMmjn8HrfDBw0ePD55ET9uHz553jl5caFkpQgdEcqmuJlhTzgQdGGY4vSoVxcWE08vJ9dcVv6yp0kyKH2ZR0lGBZ4JljGDjrbRzi2b0BiJdFallnxM37kPEhIE3KSpz1kP1MmU/UT1P2THicgYRp5kZQpQpTCwse2v0romZnBp87Ow9zQ4pNsvNCE7XLkw73fgkXhfcF0kjuqCp8/Qo+IWmklQFFYZwrPUwiUszslgZRjh1Eao0LTG5xjM69FLgguqRXU/IwbfemcJMKv/4263dfzssLrReFBOfLLDJ9S5bmfexYWWyTyPLRFkZKsjmoKzi0Ei4GjecMkWJ4QsvMFHMfyskOfbDM34pUYQEvSWyKLCYWj/32tmxfZ84tw2MKp21Y2Rk6fwR0CgsdCk13Y5JQXOeObtZTuLs6d6L/kbMfzN9VXi+uinB3PZ3cT33tKYEznfBsgHLXUBL3SDkJeNSuMjvP9nd9r64OD1J4pPk+4fu2ZfmTzgAr8Eb0AMJ+AjOwDdwDgaAgLsgDNrBYesuDMMobG+iraDpeQm2Knz1Bz4IGsU=</latexit><latexit sha1_base64="9+Yq9slQ2HSOR1wRtP14trMz9T4=">AAADW3icdZLfbtMwFMadhj8jdNCBuOLGokLqJJiSCQlukCa44aoaEt0m1W3kuk5jzbEz28naGr8hL8DF3gW3jRBtx5Eiffp+34njczIpOdMmjn8HrfDBw0ePD55ET9uHz553jl5caFkpQgdEcqmuJlhTzgQdGGY4vSoVxcWE08vJ9dcVv6yp0kyKH2ZR0lGBZ4JljGDjrbRzi2b0BiJdFallnxM37kPEhIE3KSpz1kP1MmU/UT1P2THicgYRp5kZQpQpTCwse2v0romZnBp87Ow9zQ4pNsvNCE7XLkw73fgkXhfcF0kjuqCp8/Qo+IWmklQFFYZwrPUwiUszslgZRjh1Eao0LTG5xjM69FLgguqRXU/IwbfemcJMKv/4263dfzssLrReFBOfLLDJ9S5bmfexYWWyTyPLRFkZKsjmoKzi0Ei4GjecMkWJ4QsvMFHMfyskOfbDM34pUYQEvSWyKLCYWj/32tmxfZ84tw2MKp21Y2Rk6fwR0CgsdCk13Y5JQXOeObtZTuLs6d6L/kbMfzN9VXi+uinB3PZ3cT33tKYEznfBsgHLXUBL3SDkJeNSuMjvP9nd9r64OD1J4pPk+4fu2ZfmTzgAr8Eb0AMJ+AjOwDdwDgaAgLsgDNrBYesuDMMobG+iraDpeQm2Knz1Bz4IGsU=</latexit><latexit sha1_base64="9+Yq9slQ2HSOR1wRtP14trMz9T4=">AAADW3icdZLfbtMwFMadhj8jdNCBuOLGokLqJJiSCQlukCa44aoaEt0m1W3kuk5jzbEz28naGr8hL8DF3gW3jRBtx5Eiffp+34njczIpOdMmjn8HrfDBw0ePD55ET9uHz553jl5caFkpQgdEcqmuJlhTzgQdGGY4vSoVxcWE08vJ9dcVv6yp0kyKH2ZR0lGBZ4JljGDjrbRzi2b0BiJdFallnxM37kPEhIE3KSpz1kP1MmU/UT1P2THicgYRp5kZQpQpTCwse2v0romZnBp87Ow9zQ4pNsvNCE7XLkw73fgkXhfcF0kjuqCp8/Qo+IWmklQFFYZwrPUwiUszslgZRjh1Eao0LTG5xjM69FLgguqRXU/IwbfemcJMKv/4263dfzssLrReFBOfLLDJ9S5bmfexYWWyTyPLRFkZKsjmoKzi0Ei4GjecMkWJ4QsvMFHMfyskOfbDM34pUYQEvSWyKLCYWj/32tmxfZ84tw2MKp21Y2Rk6fwR0CgsdCk13Y5JQXOeObtZTuLs6d6L/kbMfzN9VXi+uinB3PZ3cT33tKYEznfBsgHLXUBL3SDkJeNSuMjvP9nd9r64OD1J4pPk+4fu2ZfmTzgAr8Eb0AMJ+AjOwDdwDgaAgLsgDNrBYesuDMMobG+iraDpeQm2Knz1Bz4IGsU=</latexit><latexit sha1_base64="9+Yq9slQ2HSOR1wRtP14trMz9T4=">AAADW3icdZLfbtMwFMadhj8jdNCBuOLGokLqJJiSCQlukCa44aoaEt0m1W3kuk5jzbEz28naGr8hL8DF3gW3jRBtx5Eiffp+34njczIpOdMmjn8HrfDBw0ePD55ET9uHz553jl5caFkpQgdEcqmuJlhTzgQdGGY4vSoVxcWE08vJ9dcVv6yp0kyKH2ZR0lGBZ4JljGDjrbRzi2b0BiJdFallnxM37kPEhIE3KSpz1kP1MmU/UT1P2THicgYRp5kZQpQpTCwse2v0romZnBp87Ow9zQ4pNsvNCE7XLkw73fgkXhfcF0kjuqCp8/Qo+IWmklQFFYZwrPUwiUszslgZRjh1Eao0LTG5xjM69FLgguqRXU/IwbfemcJMKv/4263dfzssLrReFBOfLLDJ9S5bmfexYWWyTyPLRFkZKsjmoKzi0Ei4GjecMkWJ4QsvMFHMfyskOfbDM34pUYQEvSWyKLCYWj/32tmxfZ84tw2MKp21Y2Rk6fwR0CgsdCk13Y5JQXOeObtZTuLs6d6L/kbMfzN9VXi+uinB3PZ3cT33tKYEznfBsgHLXUBL3SDkJeNSuMjvP9nd9r64OD1J4pPk+4fu2ZfmTzgAr8Eb0AMJ+AjOwDdwDgaAgLsgDNrBYesuDMMobG+iraDpeQm2Knz1Bz4IGsU=</latexit>

=NX

i=1

Zq�(~zi|~xi)

✓log p(~xi|~zi, ✓) + log

p(~zi)

q�(~zi|~xi)

�◆d~zi

<latexit sha1_base64="2NJg/MBBRZx4T9Qq9Cx6zKyRv9M=">AAADenicdZLLbhoxGIU90EtKLyHtshsLVAlEGs1EkdpNpKjddIVSqSSRMIyM8YAVj+3aHgo4fo0+VV+g75JFzYCqAukvjXz0f8fXMyPFmbFx/DuqVB89fvL04Fnt+YuXrw7rR6+vjCw0oT0iudQ3I2woZ4L2LLOc3ihNcT7i9Hp0+3nFr2dUGybFN7tQdJDjiWAZI9iGVlr/eQ6RKfLUsfPED7sQMWHh9xSpKWuh2TJld2g2T1kbIk4z2wqDnEDVKpt3peEY2Sm1uA07awjX1j5EmcbEleZga3v3wLIeaTaZ2gFcj204LilM6834JC4L7otkI5pgU5fpUfQLjSUpcios4diYfhIrO3BYW0Y49TVUGKowucUT2g9S4JyagSsf0MN3oTOGmdThC/cvu//OcDg3ZpGPgjPHdmp22ar5EOsXNvs4cEyowlJB1htlBYdWwlUacMw0JZYvgsBEs3BWSKY4vJsNmdVqSNAfROY5FmMXkpl5N3TvE++3gdXKOzdEVioftoBWY2GUNHTbJgWd8sy7dS6Jd6d7C/212P96ujoPfHVTgrnr7uLZPNAZJXC+C5YbsNwFVJkNQkEyLoWvhfyT3bT3xdXpSRKfJF/PmhefNn/CAXgLGqAFEvABXIAv4BL0AAH3USPqRMeV+2qj2q521tZKtJnzBmxV9ewP8uYlDw==</latexit><latexit sha1_base64="2NJg/MBBRZx4T9Qq9Cx6zKyRv9M=">AAADenicdZLLbhoxGIU90EtKLyHtshsLVAlEGs1EkdpNpKjddIVSqSSRMIyM8YAVj+3aHgo4fo0+VV+g75JFzYCqAukvjXz0f8fXMyPFmbFx/DuqVB89fvL04Fnt+YuXrw7rR6+vjCw0oT0iudQ3I2woZ4L2LLOc3ihNcT7i9Hp0+3nFr2dUGybFN7tQdJDjiWAZI9iGVlr/eQ6RKfLUsfPED7sQMWHh9xSpKWuh2TJld2g2T1kbIk4z2wqDnEDVKpt3peEY2Sm1uA07awjX1j5EmcbEleZga3v3wLIeaTaZ2gFcj204LilM6834JC4L7otkI5pgU5fpUfQLjSUpcios4diYfhIrO3BYW0Y49TVUGKowucUT2g9S4JyagSsf0MN3oTOGmdThC/cvu//OcDg3ZpGPgjPHdmp22ar5EOsXNvs4cEyowlJB1htlBYdWwlUacMw0JZYvgsBEs3BWSKY4vJsNmdVqSNAfROY5FmMXkpl5N3TvE++3gdXKOzdEVioftoBWY2GUNHTbJgWd8sy7dS6Jd6d7C/212P96ujoPfHVTgrnr7uLZPNAZJXC+C5YbsNwFVJkNQkEyLoWvhfyT3bT3xdXpSRKfJF/PmhefNn/CAXgLGqAFEvABXIAv4BL0AAH3USPqRMeV+2qj2q521tZKtJnzBmxV9ewP8uYlDw==</latexit><latexit sha1_base64="2NJg/MBBRZx4T9Qq9Cx6zKyRv9M=">AAADenicdZLLbhoxGIU90EtKLyHtshsLVAlEGs1EkdpNpKjddIVSqSSRMIyM8YAVj+3aHgo4fo0+VV+g75JFzYCqAukvjXz0f8fXMyPFmbFx/DuqVB89fvL04Fnt+YuXrw7rR6+vjCw0oT0iudQ3I2woZ4L2LLOc3ihNcT7i9Hp0+3nFr2dUGybFN7tQdJDjiWAZI9iGVlr/eQ6RKfLUsfPED7sQMWHh9xSpKWuh2TJld2g2T1kbIk4z2wqDnEDVKpt3peEY2Sm1uA07awjX1j5EmcbEleZga3v3wLIeaTaZ2gFcj204LilM6834JC4L7otkI5pgU5fpUfQLjSUpcios4diYfhIrO3BYW0Y49TVUGKowucUT2g9S4JyagSsf0MN3oTOGmdThC/cvu//OcDg3ZpGPgjPHdmp22ar5EOsXNvs4cEyowlJB1htlBYdWwlUacMw0JZYvgsBEs3BWSKY4vJsNmdVqSNAfROY5FmMXkpl5N3TvE++3gdXKOzdEVioftoBWY2GUNHTbJgWd8sy7dS6Jd6d7C/212P96ujoPfHVTgrnr7uLZPNAZJXC+C5YbsNwFVJkNQkEyLoWvhfyT3bT3xdXpSRKfJF/PmhefNn/CAXgLGqAFEvABXIAv4BL0AAH3USPqRMeV+2qj2q521tZKtJnzBmxV9ewP8uYlDw==</latexit><latexit sha1_base64="2NJg/MBBRZx4T9Qq9Cx6zKyRv9M=">AAADenicdZLLbhoxGIU90EtKLyHtshsLVAlEGs1EkdpNpKjddIVSqSSRMIyM8YAVj+3aHgo4fo0+VV+g75JFzYCqAukvjXz0f8fXMyPFmbFx/DuqVB89fvL04Fnt+YuXrw7rR6+vjCw0oT0iudQ3I2woZ4L2LLOc3ihNcT7i9Hp0+3nFr2dUGybFN7tQdJDjiWAZI9iGVlr/eQ6RKfLUsfPED7sQMWHh9xSpKWuh2TJld2g2T1kbIk4z2wqDnEDVKpt3peEY2Sm1uA07awjX1j5EmcbEleZga3v3wLIeaTaZ2gFcj204LilM6834JC4L7otkI5pgU5fpUfQLjSUpcios4diYfhIrO3BYW0Y49TVUGKowucUT2g9S4JyagSsf0MN3oTOGmdThC/cvu//OcDg3ZpGPgjPHdmp22ar5EOsXNvs4cEyowlJB1htlBYdWwlUacMw0JZYvgsBEs3BWSKY4vJsNmdVqSNAfROY5FmMXkpl5N3TvE++3gdXKOzdEVioftoBWY2GUNHTbJgWd8sy7dS6Jd6d7C/212P96ujoPfHVTgrnr7uLZPNAZJXC+C5YbsNwFVJkNQkEyLoWvhfyT3bT3xdXpSRKfJF/PmhefNn/CAXgLGqAFEvABXIAv4BL0AAH3USPqRMeV+2qj2q521tZKtJnzBmxV9ewP8uYlDw==</latexit>

= N (g�1(~x), H�2(~x))<latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit>

N (0, I)<latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit>

• can use analytic formula for KL of 2 Gaussians

(log-likelihood)

(Jensen)

Page 15: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

VAE objective

NOTE: this does not mean that VAE is trying to make

q�(~z|~x) = p(~z)<latexit sha1_base64="dvlpku8Ykz0Bbyg0BgniMrkjGq4=">AAAC/HicdZLLbhMxFIad4VaGWwobJDYWEVK7IJqpkGCDVMGGVVUk0laK08hxPI1V37A9QxLjPk13qCx5Cl6At8GTjhCZwJEs/T7fbx8f2xPNmXVZ9quT3Lh56/adrbvpvfsPHj7qbj8+sqo0hA6I4sqcTLClnEk6cMxxeqINxWLC6fHk/H3NjytqLFPyk1toOhL4TLKCEexiatx9+nmM9IztoGr5FVXzXfgW6nqyO+72sn62Crgp8kb0QBOH4+3OTzRVpBRUOsKxtcM8027ksXGMcBpSVFqqMTnHZ3QYpcSC2pFftRDgi5iZwkKZOKSDq+zfKzwW1i7EJDoFdjPbZnXyX2xYuuLNyDOpS0cluS5UlBw6Bev7gFNmKHF8EQUmhsWzQjLDBhMXby1NkaRfiBICy6lHTFbBn/qXeQjrwBkdvD9FTukQS0BnsLRaWbpuU5LOeBE8KuL+Pg9+b2OjPxb3X8+BEZHXnRLM/UEbV/NIK0rgvA2WDVi2AdW2QShKxpUMaXz/vP3am+Jor59n/fzjq97+u+YnbIFn4DnYATl4DfbBB3AIBoCAC3AJrsD35CK5TL4lV9fWpNOseQLWIvnxGyIY+ZA=</latexit><latexit sha1_base64="dvlpku8Ykz0Bbyg0BgniMrkjGq4=">AAAC/HicdZLLbhMxFIad4VaGWwobJDYWEVK7IJqpkGCDVMGGVVUk0laK08hxPI1V37A9QxLjPk13qCx5Cl6At8GTjhCZwJEs/T7fbx8f2xPNmXVZ9quT3Lh56/adrbvpvfsPHj7qbj8+sqo0hA6I4sqcTLClnEk6cMxxeqINxWLC6fHk/H3NjytqLFPyk1toOhL4TLKCEexiatx9+nmM9IztoGr5FVXzXfgW6nqyO+72sn62Crgp8kb0QBOH4+3OTzRVpBRUOsKxtcM8027ksXGMcBpSVFqqMTnHZ3QYpcSC2pFftRDgi5iZwkKZOKSDq+zfKzwW1i7EJDoFdjPbZnXyX2xYuuLNyDOpS0cluS5UlBw6Bev7gFNmKHF8EQUmhsWzQjLDBhMXby1NkaRfiBICy6lHTFbBn/qXeQjrwBkdvD9FTukQS0BnsLRaWbpuU5LOeBE8KuL+Pg9+b2OjPxb3X8+BEZHXnRLM/UEbV/NIK0rgvA2WDVi2AdW2QShKxpUMaXz/vP3am+Jor59n/fzjq97+u+YnbIFn4DnYATl4DfbBB3AIBoCAC3AJrsD35CK5TL4lV9fWpNOseQLWIvnxGyIY+ZA=</latexit><latexit sha1_base64="dvlpku8Ykz0Bbyg0BgniMrkjGq4=">AAAC/HicdZLLbhMxFIad4VaGWwobJDYWEVK7IJqpkGCDVMGGVVUk0laK08hxPI1V37A9QxLjPk13qCx5Cl6At8GTjhCZwJEs/T7fbx8f2xPNmXVZ9quT3Lh56/adrbvpvfsPHj7qbj8+sqo0hA6I4sqcTLClnEk6cMxxeqINxWLC6fHk/H3NjytqLFPyk1toOhL4TLKCEexiatx9+nmM9IztoGr5FVXzXfgW6nqyO+72sn62Crgp8kb0QBOH4+3OTzRVpBRUOsKxtcM8027ksXGMcBpSVFqqMTnHZ3QYpcSC2pFftRDgi5iZwkKZOKSDq+zfKzwW1i7EJDoFdjPbZnXyX2xYuuLNyDOpS0cluS5UlBw6Bev7gFNmKHF8EQUmhsWzQjLDBhMXby1NkaRfiBICy6lHTFbBn/qXeQjrwBkdvD9FTukQS0BnsLRaWbpuU5LOeBE8KuL+Pg9+b2OjPxb3X8+BEZHXnRLM/UEbV/NIK0rgvA2WDVi2AdW2QShKxpUMaXz/vP3am+Jor59n/fzjq97+u+YnbIFn4DnYATl4DfbBB3AIBoCAC3AJrsD35CK5TL4lV9fWpNOseQLWIvnxGyIY+ZA=</latexit><latexit sha1_base64="dvlpku8Ykz0Bbyg0BgniMrkjGq4=">AAAC/HicdZLLbhMxFIad4VaGWwobJDYWEVK7IJqpkGCDVMGGVVUk0laK08hxPI1V37A9QxLjPk13qCx5Cl6At8GTjhCZwJEs/T7fbx8f2xPNmXVZ9quT3Lh56/adrbvpvfsPHj7qbj8+sqo0hA6I4sqcTLClnEk6cMxxeqINxWLC6fHk/H3NjytqLFPyk1toOhL4TLKCEexiatx9+nmM9IztoGr5FVXzXfgW6nqyO+72sn62Crgp8kb0QBOH4+3OTzRVpBRUOsKxtcM8027ksXGMcBpSVFqqMTnHZ3QYpcSC2pFftRDgi5iZwkKZOKSDq+zfKzwW1i7EJDoFdjPbZnXyX2xYuuLNyDOpS0cluS5UlBw6Bev7gFNmKHF8EQUmhsWzQjLDBhMXby1NkaRfiBICy6lHTFbBn/qXeQjrwBkdvD9FTukQS0BnsLRaWbpuU5LOeBE8KuL+Pg9+b2OjPxb3X8+BEZHXnRLM/UEbV/NIK0rgvA2WDVi2AdW2QShKxpUMaXz/vP3am+Jor59n/fzjq97+u+YnbIFn4DnYATl4DfbBB3AIBoCAC3AJrsD35CK5TL4lV9fWpNOseQLWIvnxGyIY+ZA=</latexit>

= N (g�1(~x), H�2(~x))<latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit>

N (0, I)<latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit>

Page 16: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

VAE objective

= N (g�1(~x), H�2(~x))<latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit>

N (0, I)<latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit><latexit sha1_base64="4k+aPXKh/C25siGFRKkDn+hJN4I=">AAAC63icdZLNbhMxEMed5assXy0cuayIkIoE0W6FBMcKLnCpikTaomxaeZ3Zxqq/ZM+GppafojdUjrwNL8Db4KQrRDYwkqW/5ve3PeNxZQR3mOe/esmNm7du39m4m967/+Dho82txwdON5bBkGmh7VFFHQiuYIgcBRwZC1RWAg6rs/cLfjgD67hWn3FuYCzpqeI1ZxRj6ku5Z+V2/vLji5PNfj7Il5Gti6IVfdLG/slW72c50ayRoJAJ6tyoyA2OPbXImYCQlo0DQ9kZPYVRlIpKcGO/rDhkz2NmktXaxqUwW2b/3uGpdG4uq+iUFKeuyxbJf7FRg/XbsefKNAiKXV9UNyJDnS3azybcAkMxj4Iyy2OtGZtSSxnGR0rTUsFXpqWkauJLrmbBH/tXRQirAK0J3h+XqE2IV2RoqXJGO1i1aQVTUQdf1vF8XwS/s3bQHwv+1xMnFPmiU0aF3+vi2XmkM2DZeRdctOCiC8C4FpVRcqFVSOP8i+6018XBzqDIB8Wn1/3dd+1P2CBPyTOyTQryhuySD2SfDAkjklySK/I9kcll8i25urYmvXbPE7ISyY/fV8jy7g==</latexit>

1

k

NX

i=1

kX

j=1

log p(~xi|~zij , ✓)<latexit sha1_base64="2xDtk+tPlUGT9lnCrM9BKLjRnHA=">AAADK3icdZLLbhMxFIad4VbCLYUlG4sIqUgQzVRIdFOpggWsoiKRtlKcRI7jSdz4JtszJDF+pL4Ju65AZcl74KSjiiRwJEu/z/f7do6HmjPr0vRHLbl1+87dezv36w8ePnr8pLH79MSqwhDaIYorczbElnImaccxx+mZNhSLIaenw+mHJT8tqbFMyS9urmlP4LFkOSPYxdSg8RFClBtMfBb8NEBkCzHw7DAL/TasZufL2RQirsZQ76FyNmDfULmItvPwGrkJdfgVHDSaaStdBdwWWSWaoIrjwW7tOxopUggqHeHY2m6Watfz2DhGOA11VFiqMZniMe1GKbGgtudXLw7wZcyMYK5MHNLBVfbvFR4La+diGJ0Cu4ndZMvkv1i3cPlBzzOpC0cluT4oLzh0Ci7LB0fMUOL4PApMDIt3hWSCYwFdLHK9jiT9SpQQWI48YrIMvu/fZCGsA2d08L6PnNIhHgGdwdJqZem6TUk64XnwNw3a39roxuL+62kbEfnypQRz397E5SzSkhI42wSLCiw2AdW2QihKxpUM9dj/bLPb2+Jkv5Wlrezz2+bR++on7IDn4AXYAxl4B47AJ3AMOoCAC3AJrsCv5CK5TH4mV9fWpFateQbWIvn9B6DCDNQ=</latexit><latexit sha1_base64="2xDtk+tPlUGT9lnCrM9BKLjRnHA=">AAADK3icdZLLbhMxFIad4VbCLYUlG4sIqUgQzVRIdFOpggWsoiKRtlKcRI7jSdz4JtszJDF+pL4Ju65AZcl74KSjiiRwJEu/z/f7do6HmjPr0vRHLbl1+87dezv36w8ePnr8pLH79MSqwhDaIYorczbElnImaccxx+mZNhSLIaenw+mHJT8tqbFMyS9urmlP4LFkOSPYxdSg8RFClBtMfBb8NEBkCzHw7DAL/TasZufL2RQirsZQ76FyNmDfULmItvPwGrkJdfgVHDSaaStdBdwWWSWaoIrjwW7tOxopUggqHeHY2m6Watfz2DhGOA11VFiqMZniMe1GKbGgtudXLw7wZcyMYK5MHNLBVfbvFR4La+diGJ0Cu4ndZMvkv1i3cPlBzzOpC0cluT4oLzh0Ci7LB0fMUOL4PApMDIt3hWSCYwFdLHK9jiT9SpQQWI48YrIMvu/fZCGsA2d08L6PnNIhHgGdwdJqZem6TUk64XnwNw3a39roxuL+62kbEfnypQRz397E5SzSkhI42wSLCiw2AdW2QihKxpUM9dj/bLPb2+Jkv5Wlrezz2+bR++on7IDn4AXYAxl4B47AJ3AMOoCAC3AJrsCv5CK5TH4mV9fWpFateQbWIvn9B6DCDNQ=</latexit><latexit sha1_base64="2xDtk+tPlUGT9lnCrM9BKLjRnHA=">AAADK3icdZLLbhMxFIad4VbCLYUlG4sIqUgQzVRIdFOpggWsoiKRtlKcRI7jSdz4JtszJDF+pL4Ju65AZcl74KSjiiRwJEu/z/f7do6HmjPr0vRHLbl1+87dezv36w8ePnr8pLH79MSqwhDaIYorczbElnImaccxx+mZNhSLIaenw+mHJT8tqbFMyS9urmlP4LFkOSPYxdSg8RFClBtMfBb8NEBkCzHw7DAL/TasZufL2RQirsZQ76FyNmDfULmItvPwGrkJdfgVHDSaaStdBdwWWSWaoIrjwW7tOxopUggqHeHY2m6Watfz2DhGOA11VFiqMZniMe1GKbGgtudXLw7wZcyMYK5MHNLBVfbvFR4La+diGJ0Cu4ndZMvkv1i3cPlBzzOpC0cluT4oLzh0Ci7LB0fMUOL4PApMDIt3hWSCYwFdLHK9jiT9SpQQWI48YrIMvu/fZCGsA2d08L6PnNIhHgGdwdJqZem6TUk64XnwNw3a39roxuL+62kbEfnypQRz397E5SzSkhI42wSLCiw2AdW2QihKxpUM9dj/bLPb2+Jkv5Wlrezz2+bR++on7IDn4AXYAxl4B47AJ3AMOoCAC3AJrsCv5CK5TH4mV9fWpFateQbWIvn9B6DCDNQ=</latexit><latexit sha1_base64="2xDtk+tPlUGT9lnCrM9BKLjRnHA=">AAADK3icdZLLbhMxFIad4VbCLYUlG4sIqUgQzVRIdFOpggWsoiKRtlKcRI7jSdz4JtszJDF+pL4Ju65AZcl74KSjiiRwJEu/z/f7do6HmjPr0vRHLbl1+87dezv36w8ePnr8pLH79MSqwhDaIYorczbElnImaccxx+mZNhSLIaenw+mHJT8tqbFMyS9urmlP4LFkOSPYxdSg8RFClBtMfBb8NEBkCzHw7DAL/TasZufL2RQirsZQ76FyNmDfULmItvPwGrkJdfgVHDSaaStdBdwWWSWaoIrjwW7tOxopUggqHeHY2m6Watfz2DhGOA11VFiqMZniMe1GKbGgtudXLw7wZcyMYK5MHNLBVfbvFR4La+diGJ0Cu4ndZMvkv1i3cPlBzzOpC0cluT4oLzh0Ci7LB0fMUOL4PApMDIt3hWSCYwFdLHK9jiT9SpQQWI48YrIMvu/fZCGsA2d08L6PnNIhHgGdwdJqZem6TUk64XnwNw3a39roxuL+62kbEfnypQRz397E5SzSkhI42wSLCiw2AdW2QihKxpUM9dj/bLPb2+Jkv5Wlrezz2+bR++on7IDn4AXYAxl4B47AJ3AMOoCAC3AJrsCv5CK5TH4mV9fWpFateQbWIvn9B6DCDNQ=</latexit>

~zij ⇠ q�(~z|~xi)<latexit sha1_base64="NZ9GhVXGhr8MtN9ZEF1F7Gs2Cj0=">AAAC/HicdZLLbhMxGIWd4dIyXJrCko0hQioLopkKqSwr2LCqikTaSpl05HE8jYlv2J4hiTFbXoQd6rY8AE/B2+CkI0Qm8EuWjs53/PtaKEaNTZJfnejGzVu3t7bvxHfv3X+w0919eGJkpTEZYMmkPiuQIYwKMrDUMnKmNEG8YOS0mL5Z8tOaaEOleG/niow4uhC0pBjZYOXdJ1m9yB394GFmKIfuY56pCd0L7uesnuX0uc+7vaSfrApuirQRPdDUcb7b+ZmNJa44ERYzZMwwTZQdOaQtxYz4OKsMUQhP0QUZBikQJ2bkVmfx8FlwxrCUOgxh4cr9e4ZD3Jg5L0KSIzsxbbY0/8WGlS1fjRwVqrJE4OuFyopBK+HyYuCYaoItmweBsKZhrxBPkEbYhuuL40yQT1hyjsTYZVTU3p27F6n368Bq5Z07z6xUPiwBrUbCKGnIekwKMmGld1kZ+rvUu/2NRn8i9r+ZI80DX54UI+aO2rieBVoTDGdtsGjAog2mDZj6ODx72n7kTXGy30+TfvruZe/wdfMBtsFj8BTsgRQcgEPwFhyDAcDgK7gEV+BH9CX6Fn2PLq+jUaeZ8wisVXT1GxAA+n0=</latexit><latexit sha1_base64="NZ9GhVXGhr8MtN9ZEF1F7Gs2Cj0=">AAAC/HicdZLLbhMxGIWd4dIyXJrCko0hQioLopkKqSwr2LCqikTaSpl05HE8jYlv2J4hiTFbXoQd6rY8AE/B2+CkI0Qm8EuWjs53/PtaKEaNTZJfnejGzVu3t7bvxHfv3X+w0919eGJkpTEZYMmkPiuQIYwKMrDUMnKmNEG8YOS0mL5Z8tOaaEOleG/niow4uhC0pBjZYOXdJ1m9yB394GFmKIfuY56pCd0L7uesnuX0uc+7vaSfrApuirQRPdDUcb7b+ZmNJa44ERYzZMwwTZQdOaQtxYz4OKsMUQhP0QUZBikQJ2bkVmfx8FlwxrCUOgxh4cr9e4ZD3Jg5L0KSIzsxbbY0/8WGlS1fjRwVqrJE4OuFyopBK+HyYuCYaoItmweBsKZhrxBPkEbYhuuL40yQT1hyjsTYZVTU3p27F6n368Bq5Z07z6xUPiwBrUbCKGnIekwKMmGld1kZ+rvUu/2NRn8i9r+ZI80DX54UI+aO2rieBVoTDGdtsGjAog2mDZj6ODx72n7kTXGy30+TfvruZe/wdfMBtsFj8BTsgRQcgEPwFhyDAcDgK7gEV+BH9CX6Fn2PLq+jUaeZ8wisVXT1GxAA+n0=</latexit><latexit sha1_base64="NZ9GhVXGhr8MtN9ZEF1F7Gs2Cj0=">AAAC/HicdZLLbhMxGIWd4dIyXJrCko0hQioLopkKqSwr2LCqikTaSpl05HE8jYlv2J4hiTFbXoQd6rY8AE/B2+CkI0Qm8EuWjs53/PtaKEaNTZJfnejGzVu3t7bvxHfv3X+w0919eGJkpTEZYMmkPiuQIYwKMrDUMnKmNEG8YOS0mL5Z8tOaaEOleG/niow4uhC0pBjZYOXdJ1m9yB394GFmKIfuY56pCd0L7uesnuX0uc+7vaSfrApuirQRPdDUcb7b+ZmNJa44ERYzZMwwTZQdOaQtxYz4OKsMUQhP0QUZBikQJ2bkVmfx8FlwxrCUOgxh4cr9e4ZD3Jg5L0KSIzsxbbY0/8WGlS1fjRwVqrJE4OuFyopBK+HyYuCYaoItmweBsKZhrxBPkEbYhuuL40yQT1hyjsTYZVTU3p27F6n368Bq5Z07z6xUPiwBrUbCKGnIekwKMmGld1kZ+rvUu/2NRn8i9r+ZI80DX54UI+aO2rieBVoTDGdtsGjAog2mDZj6ODx72n7kTXGy30+TfvruZe/wdfMBtsFj8BTsgRQcgEPwFhyDAcDgK7gEV+BH9CX6Fn2PLq+jUaeZ8wisVXT1GxAA+n0=</latexit><latexit sha1_base64="NZ9GhVXGhr8MtN9ZEF1F7Gs2Cj0=">AAAC/HicdZLLbhMxGIWd4dIyXJrCko0hQioLopkKqSwr2LCqikTaSpl05HE8jYlv2J4hiTFbXoQd6rY8AE/B2+CkI0Qm8EuWjs53/PtaKEaNTZJfnejGzVu3t7bvxHfv3X+w0919eGJkpTEZYMmkPiuQIYwKMrDUMnKmNEG8YOS0mL5Z8tOaaEOleG/niow4uhC0pBjZYOXdJ1m9yB394GFmKIfuY56pCd0L7uesnuX0uc+7vaSfrApuirQRPdDUcb7b+ZmNJa44ERYzZMwwTZQdOaQtxYz4OKsMUQhP0QUZBikQJ2bkVmfx8FlwxrCUOgxh4cr9e4ZD3Jg5L0KSIzsxbbY0/8WGlS1fjRwVqrJE4OuFyopBK+HyYuCYaoItmweBsKZhrxBPkEbYhuuL40yQT1hyjsTYZVTU3p27F6n368Bq5Z07z6xUPiwBrUbCKGnIekwKMmGld1kZ+rvUu/2NRn8i9r+ZI80DX54UI+aO2rieBVoTDGdtsGjAog2mDZj6ODx72n7kTXGy30+TfvruZe/wdfMBtsFj8BTsgRQcgEPwFhyDAcDgK7gEV+BH9CX6Fn2PLq+jUaeZ8wisVXT1GxAA+n0=</latexit>

for

MC integration

= 1k

X

ij

12�2 ||~xi � f✓(~zij)||2

<latexit sha1_base64="+Cjv0OxG1vySMLcU4BIHsyHivW8=">AAADKXicdZLLbhMxFIad4VbCLYUlG6sRUlk0momQYINUwYZVVCTSVoqTkeN4EhNfRrZnSOL4idjyBjwBO+i2PEiddFQ1CRzJ0q//+z0en+NhzpmxcfynFt25e+/+g72H9UePnzx91th/fmpUoQntEsWVPh9iQzmTtGuZ5fQ81xSLIadnw+nHFT8rqTZMyS92ntO+wGPJMkawDVba6LyHENlMY+IS76YeIlOI1LGv/pbdRoaNBR60/XKJylnK4BHMUmQn1OJDVC7W+dfL5aCdNppxK14X3BVJJZqgqpN0v/YTjRQpBJWWcGxML4lz23dYW0Y49XVUGJpjMsVj2gtSYkFN360v7uGr4IxgpnRY0sK1e3uHw8KYuRiGpMB2YrbZyvwX6xU2e9d3TOaFpZJcH5QVHFoFV12EI6YpsXweBCaahX+FZIJDu2zodb2OJP1GlBBYjhxisvRu4I4S7zeB1bl3boCsyn04AlqNpcmVoZsxJemEZ96hm3HsfOgmYv+b6WgR+OqmBHPX2cblLNCSEjjbBosKLLbBtAJTXw9jT7aHvCtO260kbiWf3zSPP1QPYA+8BAfgECTgLTgGn8AJ6AICfoALcAn+Rt+jX9Hv6OI6GtWqPS/ARkWXV6gMDKw=</latexit><latexit sha1_base64="+Cjv0OxG1vySMLcU4BIHsyHivW8=">AAADKXicdZLLbhMxFIad4VbCLYUlG6sRUlk0momQYINUwYZVVCTSVoqTkeN4EhNfRrZnSOL4idjyBjwBO+i2PEiddFQ1CRzJ0q//+z0en+NhzpmxcfynFt25e+/+g72H9UePnzx91th/fmpUoQntEsWVPh9iQzmTtGuZ5fQ81xSLIadnw+nHFT8rqTZMyS92ntO+wGPJMkawDVba6LyHENlMY+IS76YeIlOI1LGv/pbdRoaNBR60/XKJylnK4BHMUmQn1OJDVC7W+dfL5aCdNppxK14X3BVJJZqgqpN0v/YTjRQpBJWWcGxML4lz23dYW0Y49XVUGJpjMsVj2gtSYkFN360v7uGr4IxgpnRY0sK1e3uHw8KYuRiGpMB2YrbZyvwX6xU2e9d3TOaFpZJcH5QVHFoFV12EI6YpsXweBCaahX+FZIJDu2zodb2OJP1GlBBYjhxisvRu4I4S7zeB1bl3boCsyn04AlqNpcmVoZsxJemEZ96hm3HsfOgmYv+b6WgR+OqmBHPX2cblLNCSEjjbBosKLLbBtAJTXw9jT7aHvCtO260kbiWf3zSPP1QPYA+8BAfgECTgLTgGn8AJ6AICfoALcAn+Rt+jX9Hv6OI6GtWqPS/ARkWXV6gMDKw=</latexit><latexit sha1_base64="+Cjv0OxG1vySMLcU4BIHsyHivW8=">AAADKXicdZLLbhMxFIad4VbCLYUlG6sRUlk0momQYINUwYZVVCTSVoqTkeN4EhNfRrZnSOL4idjyBjwBO+i2PEiddFQ1CRzJ0q//+z0en+NhzpmxcfynFt25e+/+g72H9UePnzx91th/fmpUoQntEsWVPh9iQzmTtGuZ5fQ81xSLIadnw+nHFT8rqTZMyS92ntO+wGPJMkawDVba6LyHENlMY+IS76YeIlOI1LGv/pbdRoaNBR60/XKJylnK4BHMUmQn1OJDVC7W+dfL5aCdNppxK14X3BVJJZqgqpN0v/YTjRQpBJWWcGxML4lz23dYW0Y49XVUGJpjMsVj2gtSYkFN360v7uGr4IxgpnRY0sK1e3uHw8KYuRiGpMB2YrbZyvwX6xU2e9d3TOaFpZJcH5QVHFoFV12EI6YpsXweBCaahX+FZIJDu2zodb2OJP1GlBBYjhxisvRu4I4S7zeB1bl3boCsyn04AlqNpcmVoZsxJemEZ96hm3HsfOgmYv+b6WgR+OqmBHPX2cblLNCSEjjbBosKLLbBtAJTXw9jT7aHvCtO260kbiWf3zSPP1QPYA+8BAfgECTgLTgGn8AJ6AICfoALcAn+Rt+jX9Hv6OI6GtWqPS/ARkWXV6gMDKw=</latexit><latexit sha1_base64="+Cjv0OxG1vySMLcU4BIHsyHivW8=">AAADKXicdZLLbhMxFIad4VbCLYUlG6sRUlk0momQYINUwYZVVCTSVoqTkeN4EhNfRrZnSOL4idjyBjwBO+i2PEiddFQ1CRzJ0q//+z0en+NhzpmxcfynFt25e+/+g72H9UePnzx91th/fmpUoQntEsWVPh9iQzmTtGuZ5fQ81xSLIadnw+nHFT8rqTZMyS92ntO+wGPJMkawDVba6LyHENlMY+IS76YeIlOI1LGv/pbdRoaNBR60/XKJylnK4BHMUmQn1OJDVC7W+dfL5aCdNppxK14X3BVJJZqgqpN0v/YTjRQpBJWWcGxML4lz23dYW0Y49XVUGJpjMsVj2gtSYkFN360v7uGr4IxgpnRY0sK1e3uHw8KYuRiGpMB2YrbZyvwX6xU2e9d3TOaFpZJcH5QVHFoFV12EI6YpsXweBCaahX+FZIJDu2zodb2OJP1GlBBYjhxisvRu4I4S7zeB1bl3boCsyn04AlqNpcmVoZsxJemEZ96hm3HsfOgmYv+b6WgR+OqmBHPX2cblLNCSEjjbBosKLLbBtAJTXw9jT7aHvCtO260kbiWf3zSPP1QPYA+8BAfgECTgLTgGn8AJ6AICfoALcAn+Rt+jX9Hv6OI6GtWqPS/ARkWXV6gMDKw=</latexit>

= 1k

X

ij

12�2 ||~xi � f✓

�H�2(~xi)

12~✏ij + g�1(~xi)

�||2

<latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit>

N (0, I)<latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit><latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit><latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit><latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit>

1st term:

(reparametrization trick)

Page 17: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

VAE objective

= 1k

X

ij

12�2 ||~xi � f✓

�H�2(~xi)

12~✏ij + g�1(~xi)

�||2

<latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pwSn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit>

N (0, I)<latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit><latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit><latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit><latexit sha1_base64="Hsgx9mc5P4dsV4w8AVWBzlla5Ik=">AAAC7HicdZLNbhMxEMed5assXy0cuVhESEWCaLdCgmMFF7hURSJtRTatvM5sY9VrW/ZsaGr5LXpD5cjT8AK8DU66QmQDI1n6a35/2zMel0YKh1n2q5fcuHnr9p2Nu+m9+w8ePtrcenzgdGM5DLmW2h6VzIEUCoYoUMKRscDqUsJhefZ+wQ9nYJ3Q6jPODYxrdqpEJTjDmPpS7Nl6O3tJP7442exng2wZdF3kreiTNvZPtno/i4nmTQ0KuWTOjfLM4Ngzi4JLCGnRODCMn7FTGEWpWA1u7JclB/o8Zia00jYuhXSZ/XuHZ7Vz87qMzprh1HXZIvkvNmqwejv2QpkGQfHri6pGUtR00T+dCAsc5TwKxq2ItVI+ZZZxjK+UpoWCr1zXNVMTXwg1C/7Yv8pDWAVoTfD+uEBtQryComXKGe1g1aYVTGUVfFHF830e/M7aQX8s+F9PHFHki045k36vi2fnkc6A0/MuuGjBRReAcS0qohRSq5DG+efdaa+Lg51Bng3yT6/7u+/an7BBnpJnZJvk5A3ZJR/IPhkSThS5JFfke6KSy+RbcnVtTXrtnidkJZIfvwHP5PMY</latexit>

This is the key expression!

Learning Algorithm:

1. sample for i = 1:N, j = 1:k2. compute gradient w.r.t.3. take small step in direction of gradient4. repeat to convergence

✓,�1,�2<latexit sha1_base64="aFXXcJJDeqMRSOzTm3PneJPg5Ho=">AAAC+nicdZLNbhMxEMed5asNH01B4sLFIkLiANFuhATHCi6cqiKRtlI2jbzObNeq17bs2dDU3ZfpDZUjj8EL8DZ10hUiGxjJ8l/z+9vjsZ0ZKRzG8e9OdOfuvfsPtra7Dx89frLT23166HRlOYy4ltoeZ8yBFApGKFDCsbHAykzCUXb2acmP5mCd0OorLgxMSnaqRC44w5Ca9p6nWACyNzQ1hZgmzTyc9vrxIF4F3RRJI/qkiYPpbudXOtO8KkEhl8y5cRIbnHhmUXAJdTetHBjGz9gpjINUrAQ38asGavoqZGY01zYMhXSV/XuFZ6VzizILzpJh4dpsmfwXG1eYf5h4oUyFoPhtobySFDVd3gadCQsc5SIIxq0IZ6W8YJZxDHfW7aYKvnFdlkzNfCrUvPYn/m1S1+sAram9P0lRmzqUoGiZckY7WLdpBYXMa5/mYX+f1H64sdEfC/7Xs2/LwJedcib9fhvPzwOdA6fnbXDRgIs2AOMalAYppFZ1N7x/0n7tTXE4HCTxIPnyrr/3sfkJW+QFeUlek4S8J3vkMzkgI8LJJbki1+RHdBldRd+j61tr1GnWPCNrEf28AVyh+Ic=</latexit><latexit sha1_base64="aFXXcJJDeqMRSOzTm3PneJPg5Ho=">AAAC+nicdZLNbhMxEMed5asNH01B4sLFIkLiANFuhATHCi6cqiKRtlI2jbzObNeq17bs2dDU3ZfpDZUjj8EL8DZ10hUiGxjJ8l/z+9vjsZ0ZKRzG8e9OdOfuvfsPtra7Dx89frLT23166HRlOYy4ltoeZ8yBFApGKFDCsbHAykzCUXb2acmP5mCd0OorLgxMSnaqRC44w5Ca9p6nWACyNzQ1hZgmzTyc9vrxIF4F3RRJI/qkiYPpbudXOtO8KkEhl8y5cRIbnHhmUXAJdTetHBjGz9gpjINUrAQ38asGavoqZGY01zYMhXSV/XuFZ6VzizILzpJh4dpsmfwXG1eYf5h4oUyFoPhtobySFDVd3gadCQsc5SIIxq0IZ6W8YJZxDHfW7aYKvnFdlkzNfCrUvPYn/m1S1+sAram9P0lRmzqUoGiZckY7WLdpBYXMa5/mYX+f1H64sdEfC/7Xs2/LwJedcib9fhvPzwOdA6fnbXDRgIs2AOMalAYppFZ1N7x/0n7tTXE4HCTxIPnyrr/3sfkJW+QFeUlek4S8J3vkMzkgI8LJJbki1+RHdBldRd+j61tr1GnWPCNrEf28AVyh+Ic=</latexit><latexit sha1_base64="aFXXcJJDeqMRSOzTm3PneJPg5Ho=">AAAC+nicdZLNbhMxEMed5asNH01B4sLFIkLiANFuhATHCi6cqiKRtlI2jbzObNeq17bs2dDU3ZfpDZUjj8EL8DZ10hUiGxjJ8l/z+9vjsZ0ZKRzG8e9OdOfuvfsPtra7Dx89frLT23166HRlOYy4ltoeZ8yBFApGKFDCsbHAykzCUXb2acmP5mCd0OorLgxMSnaqRC44w5Ca9p6nWACyNzQ1hZgmzTyc9vrxIF4F3RRJI/qkiYPpbudXOtO8KkEhl8y5cRIbnHhmUXAJdTetHBjGz9gpjINUrAQ38asGavoqZGY01zYMhXSV/XuFZ6VzizILzpJh4dpsmfwXG1eYf5h4oUyFoPhtobySFDVd3gadCQsc5SIIxq0IZ6W8YJZxDHfW7aYKvnFdlkzNfCrUvPYn/m1S1+sAram9P0lRmzqUoGiZckY7WLdpBYXMa5/mYX+f1H64sdEfC/7Xs2/LwJedcib9fhvPzwOdA6fnbXDRgIs2AOMalAYppFZ1N7x/0n7tTXE4HCTxIPnyrr/3sfkJW+QFeUlek4S8J3vkMzkgI8LJJbki1+RHdBldRd+j61tr1GnWPCNrEf28AVyh+Ic=</latexit><latexit sha1_base64="aFXXcJJDeqMRSOzTm3PneJPg5Ho=">AAAC+nicdZLNbhMxEMed5asNH01B4sLFIkLiANFuhATHCi6cqiKRtlI2jbzObNeq17bs2dDU3ZfpDZUjj8EL8DZ10hUiGxjJ8l/z+9vjsZ0ZKRzG8e9OdOfuvfsPtra7Dx89frLT23166HRlOYy4ltoeZ8yBFApGKFDCsbHAykzCUXb2acmP5mCd0OorLgxMSnaqRC44w5Ca9p6nWACyNzQ1hZgmzTyc9vrxIF4F3RRJI/qkiYPpbudXOtO8KkEhl8y5cRIbnHhmUXAJdTetHBjGz9gpjINUrAQ38asGavoqZGY01zYMhXSV/XuFZ6VzizILzpJh4dpsmfwXG1eYf5h4oUyFoPhtobySFDVd3gadCQsc5SIIxq0IZ6W8YJZxDHfW7aYKvnFdlkzNfCrUvPYn/m1S1+sAram9P0lRmzqUoGiZckY7WLdpBYXMa5/mYX+f1H64sdEfC/7Xs2/LwJedcib9fhvPzwOdA6fnbXDRgIs2AOMalAYppFZ1N7x/0n7tTXE4HCTxIPnyrr/3sfkJW+QFeUlek4S8J3vkMzkgI8LJJbki1+RHdBldRd+j61tr1GnWPCNrEf28AVyh+Ic=</latexit>

{✏ij}<latexit sha1_base64="18pWSf9DxOEhZbHNnWwVJwO+McI=">AAAC9XicdZLLbhMxFIad4VaGWwLLbiwiJDZEMxUSLCvYsKqKRNpKmTTyOGcaU99ke5Kmlh+lO1SWvAgvwNvUSQdEJnAkS7/P9x/fjkvNmXVZ9quT3Ll77/6DnYfpo8dPnj7r9p4fWVUbCkOquDInJbHAmYShY47DiTZARMnhuDz/uOLHczCWKfnFLTWMBTmTrGKUuJiadHuFL0BbxuPEs6+hCJNuPxtk68DbIm9EHzVxOOl1fhZTRWsB0lFOrB3lmXZjT4xjlENIi9qCJvScnMEoSkkE2LFfnz3gVzEzxZUycUiH19m/KzwR1i5FGZ2CuJlts1XyX2xUu+r92DOpaweS3m5U1Rw7hVcPgafMAHV8GQWhhsWzYjojhlAXnytNCwkLqoQgcuoLJufBn/o3eQibwBkdvD8tnNIhboGdIdJqZWHTpiTMeBV8UcX1fR783tZCfyzuv54DIyJf3ZQS7g/aeH4R6RwovmiDywZctkHsfIN+f4KQxv7n7W5vi6O9QZ4N8s9v+/sfmp+wg3bRS/Qa5egd2kef0CEaIooW6Apdo+/JIrlKviXXt9ak09S8QBuR/LgBAVn4IA==</latexit><latexit sha1_base64="18pWSf9DxOEhZbHNnWwVJwO+McI=">AAAC9XicdZLLbhMxFIad4VaGWwLLbiwiJDZEMxUSLCvYsKqKRNpKmTTyOGcaU99ke5Kmlh+lO1SWvAgvwNvUSQdEJnAkS7/P9x/fjkvNmXVZ9quT3Ll77/6DnYfpo8dPnj7r9p4fWVUbCkOquDInJbHAmYShY47DiTZARMnhuDz/uOLHczCWKfnFLTWMBTmTrGKUuJiadHuFL0BbxuPEs6+hCJNuPxtk68DbIm9EHzVxOOl1fhZTRWsB0lFOrB3lmXZjT4xjlENIi9qCJvScnMEoSkkE2LFfnz3gVzEzxZUycUiH19m/KzwR1i5FGZ2CuJlts1XyX2xUu+r92DOpaweS3m5U1Rw7hVcPgafMAHV8GQWhhsWzYjojhlAXnytNCwkLqoQgcuoLJufBn/o3eQibwBkdvD8tnNIhboGdIdJqZWHTpiTMeBV8UcX1fR783tZCfyzuv54DIyJf3ZQS7g/aeH4R6RwovmiDywZctkHsfIN+f4KQxv7n7W5vi6O9QZ4N8s9v+/sfmp+wg3bRS/Qa5egd2kef0CEaIooW6Apdo+/JIrlKviXXt9ak09S8QBuR/LgBAVn4IA==</latexit><latexit sha1_base64="18pWSf9DxOEhZbHNnWwVJwO+McI=">AAAC9XicdZLLbhMxFIad4VaGWwLLbiwiJDZEMxUSLCvYsKqKRNpKmTTyOGcaU99ke5Kmlh+lO1SWvAgvwNvUSQdEJnAkS7/P9x/fjkvNmXVZ9quT3Ll77/6DnYfpo8dPnj7r9p4fWVUbCkOquDInJbHAmYShY47DiTZARMnhuDz/uOLHczCWKfnFLTWMBTmTrGKUuJiadHuFL0BbxuPEs6+hCJNuPxtk68DbIm9EHzVxOOl1fhZTRWsB0lFOrB3lmXZjT4xjlENIi9qCJvScnMEoSkkE2LFfnz3gVzEzxZUycUiH19m/KzwR1i5FGZ2CuJlts1XyX2xUu+r92DOpaweS3m5U1Rw7hVcPgafMAHV8GQWhhsWzYjojhlAXnytNCwkLqoQgcuoLJufBn/o3eQibwBkdvD8tnNIhboGdIdJqZWHTpiTMeBV8UcX1fR783tZCfyzuv54DIyJf3ZQS7g/aeH4R6RwovmiDywZctkHsfIN+f4KQxv7n7W5vi6O9QZ4N8s9v+/sfmp+wg3bRS/Qa5egd2kef0CEaIooW6Apdo+/JIrlKviXXt9ak09S8QBuR/LgBAVn4IA==</latexit><latexit sha1_base64="18pWSf9DxOEhZbHNnWwVJwO+McI=">AAAC9XicdZLLbhMxFIad4VaGWwLLbiwiJDZEMxUSLCvYsKqKRNpKmTTyOGcaU99ke5Kmlh+lO1SWvAgvwNvUSQdEJnAkS7/P9x/fjkvNmXVZ9quT3Ll77/6DnYfpo8dPnj7r9p4fWVUbCkOquDInJbHAmYShY47DiTZARMnhuDz/uOLHczCWKfnFLTWMBTmTrGKUuJiadHuFL0BbxuPEs6+hCJNuPxtk68DbIm9EHzVxOOl1fhZTRWsB0lFOrB3lmXZjT4xjlENIi9qCJvScnMEoSkkE2LFfnz3gVzEzxZUycUiH19m/KzwR1i5FGZ2CuJlts1XyX2xUu+r92DOpaweS3m5U1Rw7hVcPgafMAHV8GQWhhsWzYjojhlAXnytNCwkLqoQgcuoLJufBn/o3eQibwBkdvD8tnNIhboGdIdJqZWHTpiTMeBV8UcX1fR783tZCfyzuv54DIyJf3ZQS7g/aeH4R6RwovmiDywZctkHsfIN+f4KQxv7n7W5vi6O9QZ4N8s9v+/sfmp+wg3bRS/Qa5egd2kef0CEaIooW6Apdo+/JIrlKviXXt9ak09S8QBuR/LgBAVn4IA==</latexit>

(reparametrization trick)

Page 18: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

putting it all together:

ELBO:

DNN #1 (generator)

DNN #2 (mean)

iid standardnormal noise

DNN #3 (variance)

differentiate w.r.t. and take gradient steps!

datalatents

encoder

Geometricpicture:

= 1k

X

ij

12�2 ||~xi � f✓

�H�2(~xi)

12~✏ij + g�1(~xi)

�||2

<latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit>

= N (g�1(~x), H�2(~x))<latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit>

Page 19: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

putting it all together:

DNN #2 (mean)

iid standardnormal noise

DNN #3 (variance)

differentiate w.r.t. and take gradient steps!

encoder

decoder

• in practice, often use 1 sample (k=1)

compute MSEbetween andGeometric

picture:

datalatents

= N (g�1(~x), H�2(~x))<latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit><latexit sha1_base64="EIA10/QKrFpumu2uiU4ySFhOt2Y=">AAADA3icdZJNixMxHMbT8W0dX7arRy/BInRBy0wR9CIsetnTsoLdXeh0hzTNtKF5I8mM7YYcvfpFvMle14ufwm9j2g5ip/qHwMPze/L6z1gxamyS/GpFt27fuXtv73784OGjx/vtgydnRpYakwGWTOqLMTKEUUEGllpGLpQmiI8ZOR/PP6z4eUW0oVJ8sktFRhxNBS0oRjZYebv7DmYnmnenucvUjOap72bV4vAlPK6N/sY4zNudpJesC+6KtBYdUNdpftD6mU0kLjkRFjNkzDBNlB05pC3FjPg4Kw1RCM/RlAyDFIgTM3LrK3n4IjgTWEgdhrBw7f49wyFuzJKPQ5IjOzNNtjL/xYalLd6OHBWqtETgzUZFyaCVcPU+cEI1wZYtg0BY03BWiGdII2zDK8ZxJshnLDlHYuIyKirvLt2r1PttYLXyzl1mVioftoBWI2GUNGQ7JgWZscK7rAjru9S7/s5CfyL2v5nQvsBXN8WIuZMmrhaBVgTDRRNc1eCqCeY1mPs4tD1tNnlXnPV7adJLP77uHL2vP8AeeAaegy5IwRtwBI7BKRgADL6Ca3ADfkRfom/R9+h6E41a9ZynYKuim9+cvftJ</latexit>

= 1k

X

ij

12�2 ||~xi � f✓

�H�2(~xi)

12~✏ij + g�1(~xi)

�||2

<latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit><latexit sha1_base64="ObuWvJNr4YFjBmwk5u7tLcXh820=">AAADcnicdZLdbtMwFMedlY8Rvjq4GzeGMqkTW5VUSHCDNMHNrqYh0W1S3UaO67SmsR3FTmjn+hV4Ml6A99gD4HRhWjM4UqS/zu9/TpJzTpylTOkg+O1tte7df/Bw+5H/+MnTZ8/bOy/OlCxyQgdEpjK/iLGiKRN0oJlO6UWWU8zjlJ7H8y8VPy9prpgU3/QyoyOOp4IljGDtUlH75ycIkU5yTExozdxCpAoeGfbd3kr3kWJTjsd9u1qhchExeAiTCOkZ1RjFbNo9jgzKZizq2+6a748Nuim2rlVJM3Xd9R2c1ubwr7lqsb9ajfu+H7U7QS9YB7wrwlp0QB2n0Y73C00kKTgVmqRYqWEYZHpkcK4ZSan1UaFohskcT+nQSYE5VSOzHpyFey4zgYnM3SM0XGdvVxjMlVry2Dk51jPVZFXyX2xY6OTjyDCRFZoKcv2ipEihlrDaApywnBKdLp3AJGfuWyGZYTcx7Xbl+0jQH0RyjsXEICZKa8bmMLR2E+g8s8aMkZaZm/Ee1DkWKpOKbtqkoLM0sRsbaTS6sej/ek5y7nj1pw Sn5qSJy4WjJSVw0QSXNbhsAncSNUJOslQKW+0/bG77rjjr98KgF3593zn6XF/CNngF3oAuCMEHcASOwSkYAAKuvF2v473dumrttl636rPZ8uqal2AjWgd/AAWTItk=</latexit>

DNN #1 (generator)

ELBO:

Page 20: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Results [Kingma & Welling 2014]

2D latent z

f✓(z)<latexit sha1_base64="IxSrNj0WA+0HZLX4Ci9qDaljS44=">AAAC7XicdZLNahsxEMfl7Ve6/UraYy9LTSE91OyGQHsM6aWnkEKdBLyOkeXZWERfSLNObKHHyK2kx75MX6BvU9lZSr1uBwR/5vfXSJrR2AjuMM9/dZJ79x88fLT1OH3y9NnzF9s7L0+cri2DPtNC27MxdSC4gj5yFHBmLFA5FnA6vvy05KczsI5r9RXnBoaSXihecUYxpgbVqMQpIN1dvBttd/NevopsUxSN6JImjkc7nZ/lRLNagkImqHODIjc49NQiZwJCWtYODGWX9AIGUSoqwQ396s4hexszk6zSNi6F2Sr79w5PpXNzOY5OSXHq2myZ/Bcb1Fh9HHquTI2g2N1BVS0y1NmyAdmEW2Ao5lFQZnm8a8am1FKGsU1pWiq4YlpKqia+5GoW/Ll/X4SwDtCa4P15idqEeESGlipntIN1m1YwFVXwZRXr+yL4vY1Cfyz4X8+RlZEvX8qo8EdtPLuOdAYsu26DRQMWbQDGNaiMkgutQhrnX7SnvSlO9npF3iu+7HcPDpufsEVekzdklxTkAzkgn8kx6RNGNLkht+R7opOb5Ftye2dNOs2eV2Qtkh+/AXPG9IU=</latexit><latexit sha1_base64="IxSrNj0WA+0HZLX4Ci9qDaljS44=">AAAC7XicdZLNahsxEMfl7Ve6/UraYy9LTSE91OyGQHsM6aWnkEKdBLyOkeXZWERfSLNObKHHyK2kx75MX6BvU9lZSr1uBwR/5vfXSJrR2AjuMM9/dZJ79x88fLT1OH3y9NnzF9s7L0+cri2DPtNC27MxdSC4gj5yFHBmLFA5FnA6vvy05KczsI5r9RXnBoaSXihecUYxpgbVqMQpIN1dvBttd/NevopsUxSN6JImjkc7nZ/lRLNagkImqHODIjc49NQiZwJCWtYODGWX9AIGUSoqwQ396s4hexszk6zSNi6F2Sr79w5PpXNzOY5OSXHq2myZ/Bcb1Fh9HHquTI2g2N1BVS0y1NmyAdmEW2Ao5lFQZnm8a8am1FKGsU1pWiq4YlpKqia+5GoW/Ll/X4SwDtCa4P15idqEeESGlipntIN1m1YwFVXwZRXr+yL4vY1Cfyz4X8+RlZEvX8qo8EdtPLuOdAYsu26DRQMWbQDGNaiMkgutQhrnX7SnvSlO9npF3iu+7HcPDpufsEVekzdklxTkAzkgn8kx6RNGNLkht+R7opOb5Ftye2dNOs2eV2Qtkh+/AXPG9IU=</latexit><latexit sha1_base64="IxSrNj0WA+0HZLX4Ci9qDaljS44=">AAAC7XicdZLNahsxEMfl7Ve6/UraYy9LTSE91OyGQHsM6aWnkEKdBLyOkeXZWERfSLNObKHHyK2kx75MX6BvU9lZSr1uBwR/5vfXSJrR2AjuMM9/dZJ79x88fLT1OH3y9NnzF9s7L0+cri2DPtNC27MxdSC4gj5yFHBmLFA5FnA6vvy05KczsI5r9RXnBoaSXihecUYxpgbVqMQpIN1dvBttd/NevopsUxSN6JImjkc7nZ/lRLNagkImqHODIjc49NQiZwJCWtYODGWX9AIGUSoqwQ396s4hexszk6zSNi6F2Sr79w5PpXNzOY5OSXHq2myZ/Bcb1Fh9HHquTI2g2N1BVS0y1NmyAdmEW2Ao5lFQZnm8a8am1FKGsU1pWiq4YlpKqia+5GoW/Ll/X4SwDtCa4P15idqEeESGlipntIN1m1YwFVXwZRXr+yL4vY1Cfyz4X8+RlZEvX8qo8EdtPLuOdAYsu26DRQMWbQDGNaiMkgutQhrnX7SnvSlO9npF3iu+7HcPDpufsEVekzdklxTkAzkgn8kx6RNGNLkht+R7opOb5Ftye2dNOs2eV2Qtkh+/AXPG9IU=</latexit><latexit sha1_base64="IxSrNj0WA+0HZLX4Ci9qDaljS44=">AAAC7XicdZLNahsxEMfl7Ve6/UraYy9LTSE91OyGQHsM6aWnkEKdBLyOkeXZWERfSLNObKHHyK2kx75MX6BvU9lZSr1uBwR/5vfXSJrR2AjuMM9/dZJ79x88fLT1OH3y9NnzF9s7L0+cri2DPtNC27MxdSC4gj5yFHBmLFA5FnA6vvy05KczsI5r9RXnBoaSXihecUYxpgbVqMQpIN1dvBttd/NevopsUxSN6JImjkc7nZ/lRLNagkImqHODIjc49NQiZwJCWtYODGWX9AIGUSoqwQ396s4hexszk6zSNi6F2Sr79w5PpXNzOY5OSXHq2myZ/Bcb1Fh9HHquTI2g2N1BVS0y1NmyAdmEW2Ao5lFQZnm8a8am1FKGsU1pWiq4YlpKqia+5GoW/Ll/X4SwDtCa4P15idqEeESGlipntIN1m1YwFVXwZRXr+yL4vY1Cfyz4X8+RlZEvX8qo8EdtPLuOdAYsu26DRQMWbQDGNaiMkgutQhrnX7SnvSlO9npF3iu+7HcPDpufsEVekzdklxTkAzkgn8kx6RNGNLkht+R7opOb5Ftye2dNOs2eV2Qtkh+/AXPG9IU=</latexit>

Page 21: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Results [Kingma & Welling 2014]

Page 22: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Results

[Rezende, Mohamed, & Wierstra 2014]

g�1(~x)<latexit sha1_base64="UdQcUVOkQf5Ylbd81gwGlCBF5B8=">AAAC83icdZLLbhMxFIad4VaGS1NYshkRIZUF0UyFBMuqbFhVRSJtpUw68jhnEqu+yfZMk1p+ku5QWfImvABvg5OOEJnAkSz9Ot/vY/scl4pRY9P0Vy+6d//Bw0c7j+MnT5893+3vvTg1stYERkQyqc9LbIBRASNLLYNzpQHzksFZeflpxc8a0IZK8dUuFUw4nglaUYJtSBX93VnhcjWnReb382bxtugP0mG6jmRbZK0YoDZOir3ez3wqSc1BWMKwMeMsVXbisLaUMPBxXhtQmFziGYyDFJiDmbj1zX3yJmSmSSV1WMIm6+zfOxzmxix5GZwc27npslXyX2xc2+rjxFGhaguC3B1U1SyxMlm1IZlSDcSyZRCYaBrumpA51pjY0Kw4zgVcEck5FlOXU9F4d+HeZd5vAquVd+4it1L5cERiNRZGSQObNilgzirv8irUd5l3B1uF/ljsfz3Hmge+einBzB13cbMItAGSLLrgugXXXQDKtCgPkjIpfBzmn3WnvS1OD4ZZOsy+vB8cHrU/YQe9Qq/RPsrQB3SIPqMTNEIE1egG3aLvUR3dRN+i2ztr1Gv3vEQbEf34DX2/9mI=</latexit><latexit sha1_base64="UdQcUVOkQf5Ylbd81gwGlCBF5B8=">AAAC83icdZLLbhMxFIad4VaGS1NYshkRIZUF0UyFBMuqbFhVRSJtpUw68jhnEqu+yfZMk1p+ku5QWfImvABvg5OOEJnAkSz9Ot/vY/scl4pRY9P0Vy+6d//Bw0c7j+MnT5893+3vvTg1stYERkQyqc9LbIBRASNLLYNzpQHzksFZeflpxc8a0IZK8dUuFUw4nglaUYJtSBX93VnhcjWnReb382bxtugP0mG6jmRbZK0YoDZOir3ez3wqSc1BWMKwMeMsVXbisLaUMPBxXhtQmFziGYyDFJiDmbj1zX3yJmSmSSV1WMIm6+zfOxzmxix5GZwc27npslXyX2xc2+rjxFGhaguC3B1U1SyxMlm1IZlSDcSyZRCYaBrumpA51pjY0Kw4zgVcEck5FlOXU9F4d+HeZd5vAquVd+4it1L5cERiNRZGSQObNilgzirv8irUd5l3B1uF/ljsfz3Hmge+einBzB13cbMItAGSLLrgugXXXQDKtCgPkjIpfBzmn3WnvS1OD4ZZOsy+vB8cHrU/YQe9Qq/RPsrQB3SIPqMTNEIE1egG3aLvUR3dRN+i2ztr1Gv3vEQbEf34DX2/9mI=</latexit><latexit sha1_base64="UdQcUVOkQf5Ylbd81gwGlCBF5B8=">AAAC83icdZLLbhMxFIad4VaGS1NYshkRIZUF0UyFBMuqbFhVRSJtpUw68jhnEqu+yfZMk1p+ku5QWfImvABvg5OOEJnAkSz9Ot/vY/scl4pRY9P0Vy+6d//Bw0c7j+MnT5893+3vvTg1stYERkQyqc9LbIBRASNLLYNzpQHzksFZeflpxc8a0IZK8dUuFUw4nglaUYJtSBX93VnhcjWnReb382bxtugP0mG6jmRbZK0YoDZOir3ez3wqSc1BWMKwMeMsVXbisLaUMPBxXhtQmFziGYyDFJiDmbj1zX3yJmSmSSV1WMIm6+zfOxzmxix5GZwc27npslXyX2xc2+rjxFGhaguC3B1U1SyxMlm1IZlSDcSyZRCYaBrumpA51pjY0Kw4zgVcEck5FlOXU9F4d+HeZd5vAquVd+4it1L5cERiNRZGSQObNilgzirv8irUd5l3B1uF/ljsfz3Hmge+einBzB13cbMItAGSLLrgugXXXQDKtCgPkjIpfBzmn3WnvS1OD4ZZOsy+vB8cHrU/YQe9Qq/RPsrQB3SIPqMTNEIE1egG3aLvUR3dRN+i2ztr1Gv3vEQbEf34DX2/9mI=</latexit><latexit sha1_base64="UdQcUVOkQf5Ylbd81gwGlCBF5B8=">AAAC83icdZLLbhMxFIad4VaGS1NYshkRIZUF0UyFBMuqbFhVRSJtpUw68jhnEqu+yfZMk1p+ku5QWfImvABvg5OOEJnAkSz9Ot/vY/scl4pRY9P0Vy+6d//Bw0c7j+MnT5893+3vvTg1stYERkQyqc9LbIBRASNLLYNzpQHzksFZeflpxc8a0IZK8dUuFUw4nglaUYJtSBX93VnhcjWnReb382bxtugP0mG6jmRbZK0YoDZOir3ez3wqSc1BWMKwMeMsVXbisLaUMPBxXhtQmFziGYyDFJiDmbj1zX3yJmSmSSV1WMIm6+zfOxzmxix5GZwc27npslXyX2xc2+rjxFGhaguC3B1U1SyxMlm1IZlSDcSyZRCYaBrumpA51pjY0Kw4zgVcEck5FlOXU9F4d+HeZd5vAquVd+4it1L5cERiNRZGSQObNilgzirv8irUd5l3B1uF/ljsfz3Hmge+einBzB13cbMItAGSLLrgugXXXQDKtCgPkjIpfBzmn3WnvS1OD4ZZOsy+vB8cHrU/YQe9Qq/RPsrQB3SIPqMTNEIE1egG3aLvUR3dRN+i2ztr1Gv3vEQbEf34DX2/9mI=</latexit>

z spacex space

Page 23: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

Neuroscience Application[Gao, Archer, Paninski & Cunningham 2016]

linear dynamical system (LDS)

p(~z)<latexit sha1_base64="MZOuI/NENoDOZeMpnWVDWer1nac=">AAAC63icdZLLbhMxFIad4VaGW1uWbCwipLIgmqmQYFnBpquqSKQtyqSVxznTWPVNtic0sfwU3aGy7Nv0BXgbPOkIkQkcydKv8/22z/FxqTmzLst+9ZJ79x88fLTxOH3y9NnzF5tb20dW1YbCkCquzElJLHAmYeiY43CiDRBRcjguLz43/HgGxjIlv7q5hrEg55JVjBIXU9/0TjEDihdvzzb72SBbBl4XeSv6qI3Ds63ebTFRtBYgHeXE2lGeaTf2xDhGOYS0qC1oQi/IOYyilESAHftlxQG/iZkJrpSJSzq8zP69wxNh7VyU0SmIm9oua5L/YqPaVR/HnkldO5D07qKq5tgp3LSPJ8wAdXweBaGGxVoxnRJDqIuPlKaFhO9UCUHkxBdMzoI/9e/yEFaBMzp4f1o4pUO8AjtDpNXKwqpNSZjyKviiiuf7PPjdtYP+WNx/PQdGRN50Sgn3B108u4y0meFlFyxasOgC0LZFRZSMKxnSOP+8O+11cbQ7yLNB/uV9f+9T+xM20Cv0Gu2gHH1Ae2gfHaIhokigK3SNfiYiuUp+JNd31qTX7nmJViK5+Q1/RvNk</latexit><latexit sha1_base64="MZOuI/NENoDOZeMpnWVDWer1nac=">AAAC63icdZLLbhMxFIad4VaGW1uWbCwipLIgmqmQYFnBpquqSKQtyqSVxznTWPVNtic0sfwU3aGy7Nv0BXgbPOkIkQkcydKv8/22z/FxqTmzLst+9ZJ79x88fLTxOH3y9NnzF5tb20dW1YbCkCquzElJLHAmYeiY43CiDRBRcjguLz43/HgGxjIlv7q5hrEg55JVjBIXU9/0TjEDihdvzzb72SBbBl4XeSv6qI3Ds63ebTFRtBYgHeXE2lGeaTf2xDhGOYS0qC1oQi/IOYyilESAHftlxQG/iZkJrpSJSzq8zP69wxNh7VyU0SmIm9oua5L/YqPaVR/HnkldO5D07qKq5tgp3LSPJ8wAdXweBaGGxVoxnRJDqIuPlKaFhO9UCUHkxBdMzoI/9e/yEFaBMzp4f1o4pUO8AjtDpNXKwqpNSZjyKviiiuf7PPjdtYP+WNx/PQdGRN50Sgn3B108u4y0meFlFyxasOgC0LZFRZSMKxnSOP+8O+11cbQ7yLNB/uV9f+9T+xM20Cv0Gu2gHH1Ae2gfHaIhokigK3SNfiYiuUp+JNd31qTX7nmJViK5+Q1/RvNk</latexit><latexit sha1_base64="MZOuI/NENoDOZeMpnWVDWer1nac=">AAAC63icdZLLbhMxFIad4VaGW1uWbCwipLIgmqmQYFnBpquqSKQtyqSVxznTWPVNtic0sfwU3aGy7Nv0BXgbPOkIkQkcydKv8/22z/FxqTmzLst+9ZJ79x88fLTxOH3y9NnzF5tb20dW1YbCkCquzElJLHAmYeiY43CiDRBRcjguLz43/HgGxjIlv7q5hrEg55JVjBIXU9/0TjEDihdvzzb72SBbBl4XeSv6qI3Ds63ebTFRtBYgHeXE2lGeaTf2xDhGOYS0qC1oQi/IOYyilESAHftlxQG/iZkJrpSJSzq8zP69wxNh7VyU0SmIm9oua5L/YqPaVR/HnkldO5D07qKq5tgp3LSPJ8wAdXweBaGGxVoxnRJDqIuPlKaFhO9UCUHkxBdMzoI/9e/yEFaBMzp4f1o4pUO8AjtDpNXKwqpNSZjyKviiiuf7PPjdtYP+WNx/PQdGRN50Sgn3B108u4y0meFlFyxasOgC0LZFRZSMKxnSOP+8O+11cbQ7yLNB/uV9f+9T+xM20Cv0Gu2gHH1Ae2gfHaIhokigK3SNfiYiuUp+JNd31qTX7nmJViK5+Q1/RvNk</latexit><latexit sha1_base64="MZOuI/NENoDOZeMpnWVDWer1nac=">AAAC63icdZLLbhMxFIad4VaGW1uWbCwipLIgmqmQYFnBpquqSKQtyqSVxznTWPVNtic0sfwU3aGy7Nv0BXgbPOkIkQkcydKv8/22z/FxqTmzLst+9ZJ79x88fLTxOH3y9NnzF5tb20dW1YbCkCquzElJLHAmYeiY43CiDRBRcjguLz43/HgGxjIlv7q5hrEg55JVjBIXU9/0TjEDihdvzzb72SBbBl4XeSv6qI3Ds63ebTFRtBYgHeXE2lGeaTf2xDhGOYS0qC1oQi/IOYyilESAHftlxQG/iZkJrpSJSzq8zP69wxNh7VyU0SmIm9oua5L/YqPaVR/HnkldO5D07qKq5tgp3LSPJ8wAdXweBaGGxVoxnRJDqIuPlKaFhO9UCUHkxBdMzoI/9e/yEFaBMzp4f1o4pUO8AjtDpNXKwqpNSZjyKviiiuf7PPjdtYP+WNx/PQdGRN50Sgn3B108u4y0meFlFyxasOgC0LZFRZSMKxnSOP+8O+11cbQ7yLNB/uV9f+9T+xM20Cv0Gu2gHH1Ae2gfHaIhokigK3SNfiYiuUp+JNd31qTX7nmJViK5+Q1/RvNk</latexit>

f✓<latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit><latexit sha1_base64="Z+vg35/w+JUTRAbd7H0eJs+88mA=">AAACyHicdZHNbhMxEMed5assXy0cuVhESFyIdiskOFb0UnGoikTaSNk0cpzZxoq/ZM+mjSxfeASu8AS8EW+Dk64QSWAkS3/N7z9jj2dipfBYFL862Z279+4/2HuYP3r85Omz/YPn5940jkOfG2ncYMI8SKGhjwIlDKwDpiYSLibz4xW/WIDzwugvuLQwUuxKi1pwhik1qMcVzgDZeL9b9Ip10F1RtqJL2jgbH3R+VlPDGwUauWTeD8vC4igwh4JLiHnVeLCMz9kVDJPUTIEfhfWDI32dMlNaG5eORrrO/l0RmPJ+qSbJqRjO/DZbJf/Fhg3WH0ZBaNsgaH57Ud1IioaupqdT4YCjXCbBuBPprZTPmGMc0x/leaXhmhulmJ6GSuhFDJfhbRnjJkBnYwiXFRob0xUUHdPeGg+bNqNhJusYqjr1D2UMhzuN/ljwv55TpxJfTcqZDKfbeHGT6AI4vdkG8xbMY562W27vclecH/bKold+ftc9+tjueY+8JK/IG1KS9+SInJAz0iecSPKNfCc/sk+Zza6z5a0167Q1L8hGZF9/A5t+5Qk=</latexit>

spike trains

~x|~z ⇠ Poiss(f✓(~z))<latexit sha1_base64="4i1nrjq/7y1byM3ieY4EA3zaejY=">AAADEHicdZLLbhMxFIad4VbCLYUlG4sIKZUgmqmQ6LKCDasqSKStFKeR43gaq76M7DPTJMYv0afpDpUl4gV4G5x0hEgCR7L063y/j32OPS6kcJCmvxrJnbv37j/Yedh89PjJ02et3efHzpSW8T4z0tjTMXVcCs37IEDy08JyqsaSn4wvPi75ScWtE0Z/gXnBh4qea5ELRiGmRq03pJrhr5hUC0ycUJgoClOrfM8I50InHxGYcqCdaNjbG7XaaTddBd4WWS3aqI7eaLfxk0wMKxXXwCR1bpClBQw9tSCY5KFJSscLyi7oOR9EqanibuhXbQX8OmYmODc2Lg14lf17h6fKubkaR+fy1m6TLZP/YoMS8oOhF7oogWt2e1BeSgwGL2eEJ8JyBnIeBWVWxLtiNqWWMoiTbDaJ5pfMKEX1xBOhq+DP/NsshHUAtgjenxEwRYhHYLBUu8I4vm4zmk9lHjzJY32fBb+/VeiPBf7rObIq8mWnjEp/tImrWaQVZ3i2CRY1WGwCXrgakSiFNDo04/tnm6+9LY73u1nazT6/ax9+qH/CDnqJXqEOytB7dIg+oR7qI4au0DW6Qd+Tq+Q6+Zbc3FqTRr3nBVqL5Mdv4woCVA==</latexit><latexit sha1_base64="4i1nrjq/7y1byM3ieY4EA3zaejY=">AAADEHicdZLLbhMxFIad4VbCLYUlG4sIKZUgmqmQ6LKCDasqSKStFKeR43gaq76M7DPTJMYv0afpDpUl4gV4G5x0hEgCR7L063y/j32OPS6kcJCmvxrJnbv37j/Yedh89PjJ02et3efHzpSW8T4z0tjTMXVcCs37IEDy08JyqsaSn4wvPi75ScWtE0Z/gXnBh4qea5ELRiGmRq03pJrhr5hUC0ycUJgoClOrfM8I50InHxGYcqCdaNjbG7XaaTddBd4WWS3aqI7eaLfxk0wMKxXXwCR1bpClBQw9tSCY5KFJSscLyi7oOR9EqanibuhXbQX8OmYmODc2Lg14lf17h6fKubkaR+fy1m6TLZP/YoMS8oOhF7oogWt2e1BeSgwGL2eEJ8JyBnIeBWVWxLtiNqWWMoiTbDaJ5pfMKEX1xBOhq+DP/NsshHUAtgjenxEwRYhHYLBUu8I4vm4zmk9lHjzJY32fBb+/VeiPBf7rObIq8mWnjEp/tImrWaQVZ3i2CRY1WGwCXrgakSiFNDo04/tnm6+9LY73u1nazT6/ax9+qH/CDnqJXqEOytB7dIg+oR7qI4au0DW6Qd+Tq+Q6+Zbc3FqTRr3nBVqL5Mdv4woCVA==</latexit><latexit sha1_base64="4i1nrjq/7y1byM3ieY4EA3zaejY=">AAADEHicdZLLbhMxFIad4VbCLYUlG4sIKZUgmqmQ6LKCDasqSKStFKeR43gaq76M7DPTJMYv0afpDpUl4gV4G5x0hEgCR7L063y/j32OPS6kcJCmvxrJnbv37j/Yedh89PjJ02et3efHzpSW8T4z0tjTMXVcCs37IEDy08JyqsaSn4wvPi75ScWtE0Z/gXnBh4qea5ELRiGmRq03pJrhr5hUC0ycUJgoClOrfM8I50InHxGYcqCdaNjbG7XaaTddBd4WWS3aqI7eaLfxk0wMKxXXwCR1bpClBQw9tSCY5KFJSscLyi7oOR9EqanibuhXbQX8OmYmODc2Lg14lf17h6fKubkaR+fy1m6TLZP/YoMS8oOhF7oogWt2e1BeSgwGL2eEJ8JyBnIeBWVWxLtiNqWWMoiTbDaJ5pfMKEX1xBOhq+DP/NsshHUAtgjenxEwRYhHYLBUu8I4vm4zmk9lHjzJY32fBb+/VeiPBf7rObIq8mWnjEp/tImrWaQVZ3i2CRY1WGwCXrgakSiFNDo04/tnm6+9LY73u1nazT6/ax9+qH/CDnqJXqEOytB7dIg+oR7qI4au0DW6Qd+Tq+Q6+Zbc3FqTRr3nBVqL5Mdv4woCVA==</latexit><latexit sha1_base64="4i1nrjq/7y1byM3ieY4EA3zaejY=">AAADEHicdZLLbhMxFIad4VbCLYUlG4sIKZUgmqmQ6LKCDasqSKStFKeR43gaq76M7DPTJMYv0afpDpUl4gV4G5x0hEgCR7L063y/j32OPS6kcJCmvxrJnbv37j/Yedh89PjJ02et3efHzpSW8T4z0tjTMXVcCs37IEDy08JyqsaSn4wvPi75ScWtE0Z/gXnBh4qea5ELRiGmRq03pJrhr5hUC0ycUJgoClOrfM8I50InHxGYcqCdaNjbG7XaaTddBd4WWS3aqI7eaLfxk0wMKxXXwCR1bpClBQw9tSCY5KFJSscLyi7oOR9EqanibuhXbQX8OmYmODc2Lg14lf17h6fKubkaR+fy1m6TLZP/YoMS8oOhF7oogWt2e1BeSgwGL2eEJ8JyBnIeBWVWxLtiNqWWMoiTbDaJ5pfMKEX1xBOhq+DP/NsshHUAtgjenxEwRYhHYLBUu8I4vm4zmk9lHjzJY32fBb+/VeiPBf7rObIq8mWnjEp/tImrWaQVZ3i2CRY1WGwCXrgakSiFNDo04/tnm6+9LY73u1nazT6/ax9+qH/CDnqJXqEOytB7dIg+oR7qI4au0DW6Qd+Tq+Q6+Zbc3FqTRr3nBVqL5Mdv4woCVA==</latexit>

Page 24: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

https://www.dropbox.com/s/cluev4fzfsob4q9/video_fLDS.mp4?dl=0 video:

Page 25: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

black-box variationalinference (BBVI)

• variational inference / maximizing the ELBO: approximate ML fitting method

• Monte Carlo integration

Summary

Page 26: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

VAE

black-box variationalinference (BBVI)

Summary

• variational inference / maximizing the ELBO: approximate ML fitting method

• Monte Carlo integration

• reparametrization trick

• Use DNNs to parametrize generative model p(x|z) and recognition distribution q(z|x)

Page 27: Statistical modeling and analysis of neural data (NEU 560)pillowlab.princeton.edu/teaching/statneuro2020/slides/...math warmup Consider the following function: Hmmm…. idea below

VAE

black-box variationalinference (BBVI)

Summary

• variational inference / maximizing the ELBO: approximate ML fitting method

• Monte Carlo integration

• reparametrization trick

• Use DNNs to parametrize generative model p(x|z) and recognition distribution q(z|x)

• many extensions: - structured priors for p(x)- alternative paramemetrizations of q(z|x)- non-Gaussian outputs- sequential models for time series (eg LFADS)- control variates, alternate objectives, improved training methods