machine learning 2 - northeastern universitypioter piotr substitution (i for e) insertion (o)...
TRANSCRIPT
Machine Learning 2DS 4420 - Spring 2018
From clustering to EMByron C. Wallace
Clustering
Four Types of Clustering1. Centroid-based (K-means, K-medoids)
Notion of Clusters: Voronoi tesselation
Four Types of Clustering2. Density-based (DBSCAN, OPTICS)
Notion of Clusters: Connected regions of high density
Four Types of Clustering3. Connectivity-based (Hierarchical)
Notion of Clusters: Cut off dendrogram at some depth
Four Types of Clustering4. Distribution-based (Mixture Models)
Notion of Clusters: Distributions on features
Hierarchical Clustering
Dendrogram
Root
Internal Branch
Terminal Branch
LeafInternal Node
Root
Internal Branch
Terminal Branch
LeafInternal Node
Similarity of A and B is represented as height of lowest shared internal node
(a.k.a. a similarity tree)
Dendrogram
Similarity of A and B is represented as height of lowest shared internal node
(a.k.a. a similarity tree)
(Bovine: 0.69395, (Spider Monkey: 0.390, (Gibbon:0.36079,(Orang: 0.33636, (Gorilla: 0.17147, (Chimp: 0.19268, Human: 0.11927): 0.08386): 0.06124): 0.15057): 0.54939);
D(A,B)
Dendrogram
Natural when measuring genetic similarity, distance to common ancestor
(a.k.a. a similarity tree)
(Bovine: 0.69395, (Spider Monkey: 0.390, (Gibbon:0.36079,(Orang: 0.33636, (Gorilla: 0.17147, (Chimp: 0.19268, Human: 0.11927): 0.08386): 0.06124): 0.15057): 0.54939);
D(A,B)
Example: Iris data
https://en.wikipedia.org/wiki/Iris_flower_data_set
Iris Setosa
Iris versicolor
Iris virginica
Hierarchical Clustering
https://en.wikipedia.org/wiki/Iris_flower_data_set
(Euclidian Distance)
Edit Distance Change dress color, 1 point Change earring shape, 1 point Change hair part, 1 point
D(Patty, Selma) = 3
Change dress color, 1 point Add earrings, 1 point Decrease height, 1 point Take up smoking, 1 point Lose weight, 1 point
D(Marge,Selma) = 5
Distance Patty and Selma
Distance Marge and Selma
Can be defined for any set of discrete features
Edit Distance for Strings
Peter
Piter
Pioter
Piotr
Substitution (i for e)
Insertion (o)
Deletion (e)
• Transform string Q into string C, using only Substitution, Insertion and Deletion.
• Assume that each of these operators has a cost associated with it.
• The similarity between two strings can be defined as the cost of the cheapest transformation from Q to C.
Similarity “Peter” and “Piotr”?
Substitution 1 UnitInsertion 1 UnitDeletion 1 Unit
D(Peter,Piotr) is 3
Piot
r
Pyot
r
Petro
s
Piet
ro
Pedr
o Pi
erre
Pier
o
Pete
r
Hierarchical Clustering(Edit Distance)
Pio
tr P
yotr
Pet
ros
Pie
troPedro
Pie
rre P
iero
Pet
erPe
der
Pek
a P
eada
rM
ichal
isM
ichae
lMiguel
Mick
Cristovao
Chris
toph
erCh
risto
phe
Chris
toph
Crisd
ean
Crist
obal
Crist
ofor
oKr
istof
fer
Krys
tof
Pedro (Portuguese)Petros (Greek), Peter (English), Piotr (Polish), Peadar (Irish), Pierre (French), Peder (Danish), Peka (Hawaiian), Pietro (Italian), Piero (Italian Alternative), Petr (Czech), Pyotr (Russian)
Cristovao (Portuguese)Christoph (German), Christophe (French), Cristobal (Spanish), Cristoforo (Italian), Kristoffer(Scandinavian), Krystof (Czech), Christopher (English)
Miguel (Portuguese)Michalis (Greek), Michael (English), Mick (Irish)
A Demonstration of Hierarchical Clustering using String Edit Distance Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Meaningful Patterns
Pedro(Portuguese/Spanish)Petros (Greek), Peter (English), Piotr(Polish), Peadar (Irish), Pierre (French), Peder (Danish), Peka (Hawaiian), Pietro(Italian), Piero (Italian Alternative), Petr(Czech), Pyotr (Russian)
Slide from Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Edit distance yields clustering according to geography
Spurious Patterns
18
ANGUILLAAUSTRALIA St. Helena &Dependencies
South Georgia &South Sandwich Islands
U.K.Serbia & Montenegro(Yugoslavia)
FRANCE NIGER INDIA IRELAND BRAZIL
Hierarchal clustering can sometimes show patterns that are meaningless or spurious
The tight grouping of Australia, Anguilla, St. Helena etc is meaningful; all these countries are former UK colonies
However the tight grouping of Niger and India is completely spurious; there is no connection between the two
Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
In general clusterings will only be as meaningful as your distance metric
Spurious Patterns
18
ANGUILLAAUSTRALIA St. Helena &Dependencies
South Georgia &South Sandwich Islands
U.K.Serbia & Montenegro(Yugoslavia)
FRANCE NIGER INDIA IRELAND BRAZIL
Hierarchal clustering can sometimes show patterns that are meaningless or spurious
The tight grouping of Australia, Anguilla, St. Helena etc is meaningful; all these countries are former UK colonies
However the tight grouping of Niger and India is completely spurious; there is no connection between the two
Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
In general clusterings will only be as meaningful as your distance metric
Former UK colonies No relation
“Correct” Number of Clusters
19
We can look at the dendrogram to determine the “correct” number of clusters.
Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
“Correct” Number of Clusters
19
We can look at the dendrogram to determine the “correct” number of clusters.
Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering AlgorithmsDetermine number of clusters by looking at distance
Detecting Outliers
20
Outlier
One potential use of a dendrogram: detecting outliersThe single isolated branch is suggestive of a data point that is very different to all others
Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Bottom up vs. Top downBottom-up (agglomerative): Each item starts as its own cluster; greedily merge
Bottom up vs. Top downBottom-up (agglomerative): Each item starts as its own cluster; greedily merge
Top-down (divisive): Start with one big cluster (all data); recursively split
Distance Matrix
22
0 8 8 7 7
0 2 4 4
0 3 3
0 1
0
D( , ) = 8D( , ) = 1
We begin with a distance matrix which contains the distances between every pair of objects in our database.
Slide based on one by Eamonn Keogh
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms 26
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms 26
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Can you now implement this?
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms 26
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
Distances between examples (can calculate using metric)
Bottom-up (Agglomerative Clustering)
25
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms 26
…Consider all possible merges…
Choose the best
Consider all possible merges… …
Choose the best
Consider all possible merges…
Choose the best…
Bottom-Up (agglomerative):Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together.
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
How do we calculate the distance to a cluster?
Clustering CriteriaSingle link:
(Closest point)d(A, B) = min
a2A,b2Bd(a, b)
Complete link: (Furthest point)
d(A, B) = maxa2A,b2B
d(a, b)
Group average: (Average distance)
d(A, B) =1|A||B|X
a2A,b2B
d(a, b)
Centroid: (Distance of average)
d(A, B) = d(µA,µB) µX =1|X |X
x2X
x
Hierarchical Clustering Summary
+ No need to specify number of clusters+ Hierarchical structure maps nicely onto
human intuition in some domains - Scaling: Time complexity at least O(n2)
in number of examples - Heuristic search method:
Local optima are a problem - Interpretation of results is (very) subjective
Hierarchical Clustering Summary
+ No need to specify number of clusters+ Hierarchical structure maps nicely onto
human intuition in some domains - Scaling: Time complexity at least O(n2)
in number of examples - Heuristic search method:
Local optima are a problem - Interpretation of results is (very) subjective
Hierarchical Clustering Summary
+ No need to specify number of clusters+ Hierarchical structure maps nicely onto
human intuition in some domains - Scaling: Time complexity at least O(n2)
in number of examples - Heuristic search method:
Local optima are a problem - Interpretation of results is (very) subjective
Hierarchical Clustering Summary
+ No need to specify number of clusters+ Hierarchical structure maps nicely onto
human intuition in some domains - Scaling: Time complexity at least O(n2)
in number of examples - Heuristic search method:
Local optima are a problem - Interpretation of results is (very) subjective
Hierarchical Clustering Summary
+ No need to specify number of clusters+ Hierarchical structure maps nicely onto
human intuition in some domains - Scaling: Time complexity at least O(n2)
in number of examples - Heuristic search method:
Local optima are a problem - Interpretation of results is (very) subjective
Evaluation?
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 83
Cluster Validity
O For supervised classification we have a variety of measures to evaluate how good our model is
– Accuracy, precision, recall
O For cluster analysis, the analogous question is how to evaluate the “goodness” of the resulting clusters?
O But “clusters are in the eye of the beholder”!
O Then why do we want to evaluate them?– To avoid finding patterns in noise– To compare clustering algorithms– To compare two sets of clusters– To compare two clusters
© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 84
Clusters found in Random Data
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
y
Random Points
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
y
K-means
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
y
DBSCAN
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x
yComplete Link
Clustering CriteriaInternal Quality Criteria Measure compactness of clusters • Sum of Squared Error (SSE) • Scatter Criteria
External Quality Criteria• Precision-Recall Measure • Mutual Information
Clustering CriteriaInternal Quality Criteria Measure compactness of clusters • Sum of Squared Error (SSE) • Scatter Criteria
External Quality Criteria• Precision-Recall Measure • Mutual Information
From K-means to Mixture Models
From K-means to Mixture ModelsLet’s come back to K-means for a momentK-means Algorithm
Input: X = {x1, x2, . . . , xN}Number of clusters K
Initialize: K random centroids µ1, µ2, . . . , µK
Repeat Until Convergence
1 For i = 1, . . . ,K do
Ci = {x 2 X |i = arg min1jK
k x� µj k2}2 For i = 1, . . . ,K do
µi = argminz
Px2Ci
k z� x k2}
Output: C1,C2, . . . ,CK
Yijun Zhao DATA MINING TECHNIQUES Clustering Algorithms
A probabilistic view• K-means feels a bit heuristic
• What if we instead took a probabilistic view of clustering?
• Mixture models define a “generative story” for the data observed
Some slides derived from Matt Gormley and Eric Xing (CMU)
K-Means vs Gaussian Mixture Models
μ1
μ2
μ3
Idea: Learn both means μk and covariances Σk
μ3 Σ3 μ2 Σ2
μ1 Σ1
Don’t just learn where the center of the cluster is, but also how big it is, and what shape it has.
μ1
μ2
μ3
�nk = I[zn = k]<latexit sha1_base64="93Lo9JLGfOSZ3crmU0IM1QPo76Q=">AAAF7HicfZRPb9MwFMC9scIo/za4IHGp6IVDNSVj2nZBqoYm4Dam/ZOaqnKc19Sq4wTb2dZa5lNwQ9wQN/gEfBG+DU4biSaOcKTkye/3/vm9OMwYlcrz/qyt39lo3b23eb/94OGjx0+2tp9eyDQXBM5JylJxFWIJjHI4V1QxuMoE4CRkcBlO3xb6y2sQkqb8TM0yGCY45nRMCVZ2a7T1PIhxkuCR5lPTedP5MJiPuP1Oh6OtrrfjLVbHFfxS6PbRcp2Mtjd+B1FK8gS4IgxLOfC9TA01FooSBqYd5BIyTKY4hoEVOU5ADvWiBNOpaM/8oR6nXAEnFTONE5lgNXE2C1hWd8nEBgZRDVtuDnXhJQJJY161ChPTbgcRjO1xLjLTUchyMPr03ZHRXm//dc/fPTA1REBUEv6h17NPHYgFAC+Rw72ev3/oMlkuMgb/IK/AimwEcLghqe0Sj3RwDcQM7PkEwGUuoChEB2Giu74xxoGXqLVZ6NvBqvLWaB1UEgyu9a2pY7MVrCjUQjMHmjf5mjvYpyYsUBNQuCF71UzjvIHNHTZ3IeFAop4hNMaETFKWcqee8Qq9mJOxG5StMGWPC5fM/qMRdjxmk2Y8m1CHPa115tQU47JKYBEn2PY5SDMQWKWi+OluqJowmlAldak3rhXl/7ey+nqwY1MdyuIdhvrYOCQJ2WIwq2fnTigRUZUrqmzAYlHFlo1rALMaWB5wQdr7zq/fbq5wsbvjW/njXrd/VN58m+gFeoleIR8doD56j07QOSLoM/qOfqJfLd760vra+rZE19dKm2eoslo//gJWbira</latexit><latexit sha1_base64="93Lo9JLGfOSZ3crmU0IM1QPo76Q=">AAAF7HicfZRPb9MwFMC9scIo/za4IHGp6IVDNSVj2nZBqoYm4Dam/ZOaqnKc19Sq4wTb2dZa5lNwQ9wQN/gEfBG+DU4biSaOcKTkye/3/vm9OMwYlcrz/qyt39lo3b23eb/94OGjx0+2tp9eyDQXBM5JylJxFWIJjHI4V1QxuMoE4CRkcBlO3xb6y2sQkqb8TM0yGCY45nRMCVZ2a7T1PIhxkuCR5lPTedP5MJiPuP1Oh6OtrrfjLVbHFfxS6PbRcp2Mtjd+B1FK8gS4IgxLOfC9TA01FooSBqYd5BIyTKY4hoEVOU5ADvWiBNOpaM/8oR6nXAEnFTONE5lgNXE2C1hWd8nEBgZRDVtuDnXhJQJJY161ChPTbgcRjO1xLjLTUchyMPr03ZHRXm//dc/fPTA1REBUEv6h17NPHYgFAC+Rw72ev3/oMlkuMgb/IK/AimwEcLghqe0Sj3RwDcQM7PkEwGUuoChEB2Giu74xxoGXqLVZ6NvBqvLWaB1UEgyu9a2pY7MVrCjUQjMHmjf5mjvYpyYsUBNQuCF71UzjvIHNHTZ3IeFAop4hNMaETFKWcqee8Qq9mJOxG5StMGWPC5fM/qMRdjxmk2Y8m1CHPa115tQU47JKYBEn2PY5SDMQWKWi+OluqJowmlAldak3rhXl/7ey+nqwY1MdyuIdhvrYOCQJ2WIwq2fnTigRUZUrqmzAYlHFlo1rALMaWB5wQdr7zq/fbq5wsbvjW/njXrd/VN58m+gFeoleIR8doD56j07QOSLoM/qOfqJfLd760vra+rZE19dKm2eoslo//gJWbira</latexit><latexit sha1_base64="93Lo9JLGfOSZ3crmU0IM1QPo76Q=">AAAF7HicfZRPb9MwFMC9scIo/za4IHGp6IVDNSVj2nZBqoYm4Dam/ZOaqnKc19Sq4wTb2dZa5lNwQ9wQN/gEfBG+DU4biSaOcKTkye/3/vm9OMwYlcrz/qyt39lo3b23eb/94OGjx0+2tp9eyDQXBM5JylJxFWIJjHI4V1QxuMoE4CRkcBlO3xb6y2sQkqb8TM0yGCY45nRMCVZ2a7T1PIhxkuCR5lPTedP5MJiPuP1Oh6OtrrfjLVbHFfxS6PbRcp2Mtjd+B1FK8gS4IgxLOfC9TA01FooSBqYd5BIyTKY4hoEVOU5ADvWiBNOpaM/8oR6nXAEnFTONE5lgNXE2C1hWd8nEBgZRDVtuDnXhJQJJY161ChPTbgcRjO1xLjLTUchyMPr03ZHRXm//dc/fPTA1REBUEv6h17NPHYgFAC+Rw72ev3/oMlkuMgb/IK/AimwEcLghqe0Sj3RwDcQM7PkEwGUuoChEB2Giu74xxoGXqLVZ6NvBqvLWaB1UEgyu9a2pY7MVrCjUQjMHmjf5mjvYpyYsUBNQuCF71UzjvIHNHTZ3IeFAop4hNMaETFKWcqee8Qq9mJOxG5StMGWPC5fM/qMRdjxmk2Y8m1CHPa115tQU47JKYBEn2PY5SDMQWKWi+OluqJowmlAldak3rhXl/7ey+nqwY1MdyuIdhvrYOCQJ2WIwq2fnTigRUZUrqmzAYlHFlo1rALMaWB5wQdr7zq/fbq5wsbvjW/njXrd/VN58m+gFeoleIR8doD56j07QOSLoM/qOfqJfLd760vra+rZE19dKm2eoslo//gJWbira</latexit><latexit sha1_base64="NcJZo1QTOBeujZmLoiTOXO9RCk0=">AAAF7HicfZRLb9QwEIDd0oWyvFq4IHFZsRcOqyopVdsLUlVUAbdS9SVtVivHmc1a6zjBdvqyzK/ghrghbvAL+CP8G+xtJDZxhCMlI883L8/EccGoVEHwZ2n5zkrn7r3V+90HDx89frK2/vRU5qUgcEJylovzGEtglMOJoorBeSEAZzGDs3j21unPLkBImvNjdV3AKMMppxNKsLJb47XnUYqzDI81n5nem96H4c2Y2+9sNF7rBxvBfPV8IayEPqrW4Xh95XeU5KTMgCvCsJTDMCjUSGOhKGFgulEpocBkhlMYWpHjDORIz0swvZr2OBzpSc4VcFIz0ziTGVZTb9PBsr5LpjYwiHrYanOknZcEJE153SrOTLcbJTCxxznPTCcxK8Hoo3f7RgeD7deDcHPHNBABSUWEu8HAPk0gFQC8Qna3BuH2rs8UpSgY/IMCh7lsBHC4JLntEk90dAHEDO35RMBlKcAVoqM40/3QGOPBt6i1meu70aLyymgd1RKMLvSVaWLXC5gr1ELXHnTT5uvGwz61YZGagsIt2at2GpctbOmxpQ8JDxLNDKE1JhSSspx79UwW6PmcTPygbIGpeuxcMvuPJtjzWEzb8WJKPfao0Zkj48ZlkcAizbDtc5QXILDKhfvpLqmaMppRJXWlN74V5f+3svpmsANTH0r3jmN9YDySxGw+mPWz8yeUiKTOuSpbsFTUsdvGtYBFA6wO2JH2vgubt5svnG5uhFb+uNXf269uvlX0Ar1Er1CIdtAeeo8O0Qki6DP6jn6iXx3e+dL52vl2iy4vVTbPUG11fvwFBJ8qmg==</latexit>
Hard assignments to clusters
(one-hot vector)
Idea: Replace hard assignments with soft assignments
Soft assignments to clusters
(posterior probability)�nk = p(zn=k | xn)
<latexit sha1_base64="BJ/xva1AWL+z+wmBUMTzPebmzMI=">AAAF/XicfZRLb9QwEIBd6EJZXi0cuaTspZVWVVKqtpdKVVEFx1L1JdWrlePM7lrrOMFx+rIsfgK/ghvihrgg+B38G5xsJDZxhCNFI883L8/YYcpZpnz/z8K9+4udBw+XHnUfP3n67PnyyouzLMklhVOa8ERehCQDzgScKqY4XKQSSBxyOA+nbwv9+RXIjCXiRN2mMIjJWLARo0TZreHyOh6TOCZDLabG2/PStbuh8PDqHl71ph6OWeThK31jhmJ9uNzzN/xyea4QVEJvH83W0XBl8SeOEprHIBTlJMsuAz9VA02kYpSD6eI8g5TQKRnDpRUFiSEb6LIm49W0J8FAjxKhQNCamSZxFhM1cTYLOKvv0okNDLIettoc6MJLBBkbi7pVGJtuF0cwsudbZqajkOdg9PG7A6P9/vabfrC5YxqIhKgigl2/b78mMJYAokJ2t/rB9q7LpLlMOfyD/AIrspEg4Jomtm0i0vgKqLm054NBZLmEohCNw1j3AmOMA89Qa1Pqu3heeWO0xrUEy943sds5rCjUQrcOdNfm687BPrZhWE1AkZbsVTtN8hY2d9jchaQDyWaG0BoT0ozxRDj1jObock5GblA+x1Q9Llxye2kj4nhMJ+14OmEOe9zozLEpxmWeIHIcE9tnnKQgiUpkcemumZpwFjOV6UpvXCsm/m9l9c1gh6Y+lMU/DPWhcUga8nIw62fnTiiVUZ0rqmzBxrKOzRrXAqYNsDrggrTvXdB83VzhbHMjsPKHrd7+QfXyLaFX6DVaQwHaQfvoPTpCp4iiz+g7+oV+dz51vnS+dr7N0HsLlc1LVFudH38BUqsxIw==</latexit><latexit sha1_base64="BJ/xva1AWL+z+wmBUMTzPebmzMI=">AAAF/XicfZRLb9QwEIBd6EJZXi0cuaTspZVWVVKqtpdKVVEFx1L1JdWrlePM7lrrOMFx+rIsfgK/ghvihrgg+B38G5xsJDZxhCNFI883L8/YYcpZpnz/z8K9+4udBw+XHnUfP3n67PnyyouzLMklhVOa8ERehCQDzgScKqY4XKQSSBxyOA+nbwv9+RXIjCXiRN2mMIjJWLARo0TZreHyOh6TOCZDLabG2/PStbuh8PDqHl71ph6OWeThK31jhmJ9uNzzN/xyea4QVEJvH83W0XBl8SeOEprHIBTlJMsuAz9VA02kYpSD6eI8g5TQKRnDpRUFiSEb6LIm49W0J8FAjxKhQNCamSZxFhM1cTYLOKvv0okNDLIettoc6MJLBBkbi7pVGJtuF0cwsudbZqajkOdg9PG7A6P9/vabfrC5YxqIhKgigl2/b78mMJYAokJ2t/rB9q7LpLlMOfyD/AIrspEg4Jomtm0i0vgKqLm054NBZLmEohCNw1j3AmOMA89Qa1Pqu3heeWO0xrUEy943sds5rCjUQrcOdNfm687BPrZhWE1AkZbsVTtN8hY2d9jchaQDyWaG0BoT0ozxRDj1jObock5GblA+x1Q9Llxye2kj4nhMJ+14OmEOe9zozLEpxmWeIHIcE9tnnKQgiUpkcemumZpwFjOV6UpvXCsm/m9l9c1gh6Y+lMU/DPWhcUga8nIw62fnTiiVUZ0rqmzBxrKOzRrXAqYNsDrggrTvXdB83VzhbHMjsPKHrd7+QfXyLaFX6DVaQwHaQfvoPTpCp4iiz+g7+oV+dz51vnS+dr7N0HsLlc1LVFudH38BUqsxIw==</latexit><latexit sha1_base64="BJ/xva1AWL+z+wmBUMTzPebmzMI=">AAAF/XicfZRLb9QwEIBd6EJZXi0cuaTspZVWVVKqtpdKVVEFx1L1JdWrlePM7lrrOMFx+rIsfgK/ghvihrgg+B38G5xsJDZxhCNFI883L8/YYcpZpnz/z8K9+4udBw+XHnUfP3n67PnyyouzLMklhVOa8ERehCQDzgScKqY4XKQSSBxyOA+nbwv9+RXIjCXiRN2mMIjJWLARo0TZreHyOh6TOCZDLabG2/PStbuh8PDqHl71ph6OWeThK31jhmJ9uNzzN/xyea4QVEJvH83W0XBl8SeOEprHIBTlJMsuAz9VA02kYpSD6eI8g5TQKRnDpRUFiSEb6LIm49W0J8FAjxKhQNCamSZxFhM1cTYLOKvv0okNDLIettoc6MJLBBkbi7pVGJtuF0cwsudbZqajkOdg9PG7A6P9/vabfrC5YxqIhKgigl2/b78mMJYAokJ2t/rB9q7LpLlMOfyD/AIrspEg4Jomtm0i0vgKqLm054NBZLmEohCNw1j3AmOMA89Qa1Pqu3heeWO0xrUEy943sds5rCjUQrcOdNfm687BPrZhWE1AkZbsVTtN8hY2d9jchaQDyWaG0BoT0ozxRDj1jObock5GblA+x1Q9Llxye2kj4nhMJ+14OmEOe9zozLEpxmWeIHIcE9tnnKQgiUpkcemumZpwFjOV6UpvXCsm/m9l9c1gh6Y+lMU/DPWhcUga8nIw62fnTiiVUZ0rqmzBxrKOzRrXAqYNsDrggrTvXdB83VzhbHMjsPKHrd7+QfXyLaFX6DVaQwHaQfvoPTpCp4iiz+g7+oV+dz51vnS+dr7N0HsLlc1LVFudH38BUqsxIw==</latexit><latexit sha1_base64="3RCsbNdr5mqlxZKhKDsXVBb7jcw=">AAAF/XicfZRLa9wwEICVNNum21fSHntxupcElmAnIcklEFJCe0xDXhAtiyzP7oqVZVeS8xKiP6G/orfSW+mltL+j/6a2Y+jaMpXBDJpvXpqRwpQzpX3/z9z8g4XOw0eLj7tPnj57/mJp+eWZSjJJ4ZQmPJEXIVHAmYBTzTSHi1QCiUMO5+H0baE/vwKpWCJO9G0Kg5iMBRsxSnS+NVxaw2MSx2RoxNR6e166ejcUHl7Zwyve1MMxizx8ZW7sUKwNl3r+ul8uzxWCSuihah0Nlxd+4iihWQxCU06Uugz8VA8MkZpRDraLMwUpoVMyhstcFCQGNTBlTdaraU+CgRklQoOgNTNDYhUTPXE2C1jVd+kkDwyyHrbaHJjCSwSKjUXdKoxtt4sjGOXnW2ZmopBnYM3xuwNr/P72Zj/Y2LENREJUEcGu38+/JjCWAKJCdrf6wfauy6SZTDn8g/wCK7KRIOCaJnnbRGTwFVB7mZ8PBqEyCUUhBoex6QXWWge+R3ObUt/Fs8obawyuJVj2vondzmBFoTl060B3bb7uHOxjG4b1BDRpyV630yRrYTOHzVxIOpBsZgitMSFVjCfCqWc0Q5dzMnKD8hmm6nHhkueXNiKOx3TSjqcT5rDHjc4c22JcZgkixzHJ+4yTFCTRiSwu3TXTE85ippWp9Na1YuL/Vrm+GezQ1oey+IehObQOSUNeDmb97NwJpTKqc0WVLdhY1rH7xrWAaQOsDrgg8/cuaL5urnC2sR7k8oet3v5B9fItotfoDVpFAdpB++g9OkKniKLP6Dv6hX53PnW+dL52vt2j83OVzStUW50ffwEA3DDj</latexit>
μ3 Σ3 μ2 Σ2
μ1 Σ1
K-Means vs Gaussian Mixture Models
Mixture models
Gaussian Mixture Models
μ3 Σ3 μ2 Σ2
μ1 Σ1
Idea 1: Points in each cluster are sampled for a Gaussian
Idea 2: Compute probability that point belongs to each cluster
zn ⇠ Discrete(⇡1, . . . ,⇡K)
xn | zn=k ⇠ Norm(µk,⌃k)<latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="3cbWSxdmV0r9eqXo0fV9T0T9Hos=">AAAFxHicfZRPb9MwFMC9scIIA7Yzl4leOFRTMtDWIwJNcBxl3SY11eQ4r601xwm2062zLCGOXPk6fBG+DXYaiSaOcKToye/3/P74+SUFo1KF4Z+t7Uc7vcdPdp8Gz/aC5y9e7u9dyrwUBMYkZ7m4TrAERjmMFVUMrgsBOEsYXCW3H53+aglC0pxfqFUB0wzPOZ1RgpXdOr/Z74dHYbUOfSGqhT6q183Bzu84zUmZAVeEYSknUVioqcZCUcLABHEpocDkFs9hYkWOM5BTXcVpDhvai2iqZzlXwEnDTONMZlgtvE0Hy+YuWVjHIJpu682pdqekIOmcN62SzARBnMLM1qyKTKcJK8Ho0acPRoeDk7eD6PjUtBABaU1Ew3BgvzYwFwC8RobvBtHJ0GeKUhQM/kGhw1w0AjjckTzLME91vARiJrY+MXBZCnCJ6DjJdD8yxnjwGrU2lT6IN5X3Ruu4EWC81Pemja02MJeohVYe9NB11oOHfevCYrUAhTuiV900LjvY0mNLHxIeJNoRQqdPKCRlOffymW3QVZ/MfKdsg6nv2B3J7ENMsXdisejGiwX12FHrZkbGtcsmgcU8w/ae47wAgVUu3KO7o2rBaEaV1LXe+FaU/9/K6tvOzkyzKd0/SfSZ8UiSsKoxm7XzO5SItMm5LDuwuWhi64vrAIsWWBfYkXbcRe3h5guXx0eRlb+EaBe9Qq/RGxShU/QefUbnaIwIStFP9Kv3tbfqfV+Pxe2tej4eoMbq/fgLSA0fDA==</latexit><latexit sha1_base64="AoyELedfKaAu50cE2ilfZ8Dwl30=">AAAGN3icfZRPb9MwFMDTbYURBmxcuXRMQptUTclAWy9ICJhAQkJj7J9UV5XjvLZWYyfYztbN8jfjyEfgwlfghrhxwuki0cQRjpS8+P3eH/s9O8oSKlUQ/GgtLa+079xdveffX3vw8NH6xtqZTHNB4JSkSSouIiwhoRxOFVUJXGQCMIsSOI+mbwr9+SUISVN+oq4zGDA85nRECVZ2arjOb4a88wxJyjpIwUzpt1QSAQrMNsroMOx2UBKnStqv/f2w4yPU72Vq4M+sHWI07lgHaPMl2pxW3HxMBbMuWD6cWtPPdMzwcLrjD9e3gt1gPjquEJbClleOo+HGylcUpyRnwBVJsJT9MLDhNRaKkgSMj3IJGSZTPIa+FTlmIAd6vjGmU9GehAM9SrkCTipmGjPJsJo4kwUsq7NkYgODqIYtJwe68BKDpGNetYqY8X0Uw8gWaZ6ZjqMkB6OP3702OujuP++GewemhgiISyLsBV371IGxAOAl0nvRDfd7LpPlIkvgHxQUWJGNAA5XJGUM81ijSyCmb/cHAZe5gGIhGkVMb4XGGAe+Ra3NXO+jReXMaI0qCaJLPTN17HoBKxZqoWsHumnydeNgX5owpCagcEP2qpnGeQObO2zuQsKBRD1DaIwJmaRJyp31jBboeZ+M3KDJAlPWuHCZ2JMfY8djNmnGswl12ONaZY5N0S6LBBb2NNs6ozQDgVUqikN3RdUkoYwqqUu9ca0o/7+V1deDHZpqUxbvKNKHxiFJlMwbs7p3bocSEVe5YpUN2FhUsdvCNYBZDSw3uCDtfRfWbzdXONvbDa38KfBWvSfeU2/bC70D75X33jvyTj3ifff+tJZbK+1v7Z/tX7c341KrvCIfe5XR/v0XBSNH4g==</latexit><latexit sha1_base64="AoyELedfKaAu50cE2ilfZ8Dwl30=">AAAGN3icfZRPb9MwFMDTbYURBmxcuXRMQptUTclAWy9ICJhAQkJj7J9UV5XjvLZWYyfYztbN8jfjyEfgwlfghrhxwuki0cQRjpS8+P3eH/s9O8oSKlUQ/GgtLa+079xdveffX3vw8NH6xtqZTHNB4JSkSSouIiwhoRxOFVUJXGQCMIsSOI+mbwr9+SUISVN+oq4zGDA85nRECVZ2arjOb4a88wxJyjpIwUzpt1QSAQrMNsroMOx2UBKnStqv/f2w4yPU72Vq4M+sHWI07lgHaPMl2pxW3HxMBbMuWD6cWtPPdMzwcLrjD9e3gt1gPjquEJbClleOo+HGylcUpyRnwBVJsJT9MLDhNRaKkgSMj3IJGSZTPIa+FTlmIAd6vjGmU9GehAM9SrkCTipmGjPJsJo4kwUsq7NkYgODqIYtJwe68BKDpGNetYqY8X0Uw8gWaZ6ZjqMkB6OP3702OujuP++GewemhgiISyLsBV371IGxAOAl0nvRDfd7LpPlIkvgHxQUWJGNAA5XJGUM81ijSyCmb/cHAZe5gGIhGkVMb4XGGAe+Ra3NXO+jReXMaI0qCaJLPTN17HoBKxZqoWsHumnydeNgX5owpCagcEP2qpnGeQObO2zuQsKBRD1DaIwJmaRJyp31jBboeZ+M3KDJAlPWuHCZ2JMfY8djNmnGswl12ONaZY5N0S6LBBb2NNs6ozQDgVUqikN3RdUkoYwqqUu9ca0o/7+V1deDHZpqUxbvKNKHxiFJlMwbs7p3bocSEVe5YpUN2FhUsdvCNYBZDSw3uCDtfRfWbzdXONvbDa38KfBWvSfeU2/bC70D75X33jvyTj3ifff+tJZbK+1v7Z/tX7c341KrvCIfe5XR/v0XBSNH4g==</latexit><latexit sha1_base64="gmSp9yGGdLqLCSOOcgzNoK+siuI=">AAAGQnicfZTNb9MwFMBTtsIIXxscuXRUQptUTclAWy9I02ACCQmNsS+priIneW2txk6wna2b5f+MI38CF/4Fbogb4oDTRaKJIxwpefH7vQ/7PTvMEiKk531v3Vpabt++s3LXvXf/wcNHq2uPT0Wa8whOojRJ+XmIBSSEwYkkMoHzjAOmYQJn4fR1oT+7AC5Iyo7lVQZDiseMjEiEpZkKVtl1wDrPkSC0gyTMpHpDRMRBgt5AGQn8XgclcSqF+Zrf95suQoN+JofuzNghSuKOcYDWX6H1acXNh5RT44LmwdSYfiJjioPpphusdr0tbz46tuCXQtcpx2GwtvwFxWmUU2AySrAQA98z4RXmkkQJaBflAjIcTfEYBkZkmIIYqvnG6E5Fe+wP1ShlElhUMVOYCorlxJosYFGdjSYmMPBq2HJyqAovMQgyZlWrkGrXRTGMTJHmmak4THLQ6ujtvlZeb+dFz9/e1TWEQ1wSft/rmacOjDkAK5H+y56/07eZLOdZAv8gr8CKbDgwuIxSSjGLFbqASA/M/iBgIudQLEShkKqur7W24BvU2Mz1LlpUzrRSqJIgulAzXceuFrBioQa6sqDrJl/XFva5CUNyAhI3ZC+baZw3sLnF5jbELYjXM4TGmJAJkqTMWs9ogZ73ycgOmiwwZY0Ll4k5+TG2PGaTZjybEIs9qlXmSBftskhgbk6zqTNKM+BYprw4dJdEThJCiRSq1GvbirD/Wxl9PdiBrjZl8Q5DdaAtMgqTeWNW987u0IjHVa5YZQM25lXspnANYFYDyw0uSHPf+fXbzRZOt7d8I3/0unv75c234jx1njkbju/sOnvOO+fQOXEi55vzu7XUWm5/bf9o/2z/ukFvtUqbJ05ltP/8BUlrSY8=</latexit><latexit sha1_base64="iq7hPoss7PQskHtd23AE9/yVMFM=">AAAGQnicfZTNb9MwFMDTbYURvjo4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOF4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0ePW2tPzkSS8RBOwyRO+EWABcSEwakkMoaLlAOmQQznwfR1rj+/BC5Iwk7kdQoDiseMjEiIpZkattjNkLVfIEFoG0mYSfWGiJCDBL2BUjL0u20UR4kU5mt+32+6CPV7qRy4M2OHKInaxgFaf4XWpyU3HxJOjQuaDafG9BMZUzycbrrDVsfb8uajbQt+IXScYhwN11a+oCgJMwpMhjEWou97JrzCXJIwBu2iTECKwykeQ9+IDFMQAzXfGN0uaU/8gRolTAILS2YKU0GxnFiTOSzKs+HEBAZeDltMDlTuJQJBxqxsFVDtuiiCkSnSPDMVBXEGWh2/PdDK6+6+7Prbe7qCcIgKwu95XfNUgTEHYAXS2+n6uz2bSTOexvAP8nIsz4YDg6swoRSzSKFLCHXf7A8CJjIO+UIUCqjq+FprC75Fjc1c76JF5UwrhUoJoks101XsegHLF2qgawu6qfN1Y2Gf6zAkJyBxTfaynsZZDZtZbGZD3IJ4NUOojQmpIHHCrPWMFuh5n4zsoPECU9Q4dxmbkx9hy2M6qcfTCbHY40pljnXeLosE5uY0mzqjJAWOZcLzQ3dF5CQmlEihCr22rQj7v5XRV4Md6nJT5u8gUIfaIsMgnjdmee/sDg15VObyVdZgY17GbgtXA6YVsNjgnDT3nV+93WzhbHvLN/LHnc7+QXHzrTrPnOfOhuM7e86+8845ck6d0Pnm/G4sN1aaX5s/mj+bv27RpUZh89Qpjeafv0qrSZM=</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="+E5CbyieIrGPo7yn0AlANbzw84Y=">AAAGQnicfZTNb9MwFMDTbYURvjY4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOG4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0eP19afnIsk4yGchUmc8MsAC4gJgzNJZAyXKQdMgxgugsnrXH9xBVyQhJ3KmxT6FI8YGZIQSzM1WGO3A9Z6gQShLSRhKtUbIkIOEvQmSsnA77RQHCVSmK/5fb/lItTrprLvTo0doiRqGQdo4xXamJTcfEg4NS5oNpgY009kRPFgsuUO1tretjcbLVvwC6F94MzH8WB95QuKkjCjwGQYYyF6vmfCK8wlCWPQLsoEpDic4BH0jMgwBdFXs43RrZL21O+rYcIksLBkpjAVFMuxNZnDojwbjk1g4OWwxWRf5V4iEGTEylYB1a6LIhiaIs0yU1EQZ6DVydtDrbzO3suOv7OvKwiHqCD8rtcxTxUYcQBWIN3djr/XtZk042kM/yAvx/JsODC4DhNKMYsUuoJQ98z+IGAi45AvRKGAqravtbbgOWpsZnoXLSqnWilUShBdqamuYjcLWL5QA91Y0G2dr1sL+1yHITkGiWuyl/U0zmrYzGIzG+IWxKsZQm1MSAWJE2atZ7hAz/pkaAeNF5iixrnL2Jz8CFse03E9no6JxZ5UKnOi83ZZJDA3p9nUGSUpcCwTnh+6ayLHMaFEClXotW1F2P+tjL4a7EiXmzJ/B4E60hYZBvGsMct7Z3doyKMyl6+yBhvxMjYvXA2YVsBig3PS3Hd+9XazhfOdbd/IH3fbB4fFzbfqPHOeO5uO7+w7B84759g5c0Lnm/O7sdxYaX5t/mj+bP6ao0uNwuapUxrNP38BnHpJ0w==</latexit><latexit sha1_base64="iq7hPoss7PQskHtd23AE9/yVMFM=">AAAGQnicfZTNb9MwFMDTbYURvjo4cumohDapmpIxbb0gTYMJJCQ0xr6kuqqc5LW1GjvBdrZulv8zjvwJXPgXuCFuiANOF4kmjnCk5MXv9z7s9+wgjYmQnve9sbS80rxzd/Wee//Bw0ePW2tPzkSS8RBOwyRO+EWABcSEwakkMoaLlAOmQQznwfR1rj+/BC5Iwk7kdQoDiseMjEiIpZkattjNkLVfIEFoG0mYSfWGiJCDBL2BUjL0u20UR4kU5mt+32+6CPV7qRy4M2OHKInaxgFaf4XWpyU3HxJOjQuaDafG9BMZUzycbrrDVsfb8uajbQt+IXScYhwN11a+oCgJMwpMhjEWou97JrzCXJIwBu2iTECKwykeQ9+IDFMQAzXfGN0uaU/8gRolTAILS2YKU0GxnFiTOSzKs+HEBAZeDltMDlTuJQJBxqxsFVDtuiiCkSnSPDMVBXEGWh2/PdDK6+6+7Prbe7qCcIgKwu95XfNUgTEHYAXS2+n6uz2bSTOexvAP8nIsz4YDg6swoRSzSKFLCHXf7A8CJjIO+UIUCqjq+FprC75Fjc1c76JF5UwrhUoJoks101XsegHLF2qgawu6qfN1Y2Gf6zAkJyBxTfaynsZZDZtZbGZD3IJ4NUOojQmpIHHCrPWMFuh5n4zsoPECU9Q4dxmbkx9hy2M6qcfTCbHY40pljnXeLosE5uY0mzqjJAWOZcLzQ3dF5CQmlEihCr22rQj7v5XRV4Md6nJT5u8gUIfaIsMgnjdmee/sDg15VObyVdZgY17GbgtXA6YVsNjgnDT3nV+93WzhbHvLN/LHnc7+QXHzrTrPnOfOhuM7e86+8845ck6d0Pnm/G4sN1aaX5s/mj+bv27RpUZh89Qpjeafv0qrSZM=</latexit>
�nk = p(zn=k | xn)<latexit sha1_base64="s6BIwT4drCR6QCX+SZn7AuZMIgA=">AAAF/HicfZRLb9QwEIDd0oWyvFo4ctmylyKtqqRUbS+VqqIKjqXqS6pXK8eZzVrrOMF2+rLMT+BXcEPcECfgf/BvSLKR2MQRjhSNPN+8PGMHKWdKe96fhcV7S537D5Yfdh89fvL02crq8zOVZJLCKU14Ii8CooAzAaeaaQ4XqQQSBxzOg+nbQn9+BVKxRJzo2xSGMYkEGzNKdL41WlnHEYljMjJiant7vXT9biR6eG0Pr017OGZhD1+ZGzsSr0crfW/DK1fPFfxK6O+j2ToarS79xGFCsxiEppwodel7qR4aIjWjHGwXZwpSQqckgstcFCQGNTRlSbZX0574QzNOhAZBa2aGxComeuJsFrCq79JJHhhkPWy1OTSFlxAUi0TdKohtt4tDGOfHW2ZmwoBnYM3xuwNrvMH2m4G/uWMbiISwIvxdb5B/TSCSAKJCdrcG/vauy6SZTDn8g7wCK7KRIOCaJnnXRGjwFVB7mZ8PBqEyCUUhBgex6fvWWgeeoblNqe/ieeWNNQbXEix738Ru57Ci0By6daC7Nl93DvaxDcN6Apq0ZK/baZK1sJnDZi4kHUg2M4TWmJAqxhPh1DOeo8s5GbtB+RxT9bhwyfM7GxLHYzppx9MJc9jjRmeObTEu8wSRUUzyPuMkBUl0IotLd830hLOYaWUqvXWtmPi/Va5vBju09aEs/kFgDq1D0oCXg1k/O3dCqQzrXFFlCxbJOjZrXAuYNsDqgAsyf+/85uvmCmebG34uf9jq7x9UL98yeoleoXXkox20j96jI3SKKPqMvqNf6HfnU+dL52vn2wxdXKhsXqDa6vz4C3FeMPk=</latexit><latexit sha1_base64="s6BIwT4drCR6QCX+SZn7AuZMIgA=">AAAF/HicfZRLb9QwEIDd0oWyvFo4ctmylyKtqqRUbS+VqqIKjqXqS6pXK8eZzVrrOMF2+rLMT+BXcEPcECfgf/BvSLKR2MQRjhSNPN+8PGMHKWdKe96fhcV7S537D5Yfdh89fvL02crq8zOVZJLCKU14Ii8CooAzAaeaaQ4XqQQSBxzOg+nbQn9+BVKxRJzo2xSGMYkEGzNKdL41WlnHEYljMjJiant7vXT9biR6eG0Pr017OGZhD1+ZGzsSr0crfW/DK1fPFfxK6O+j2ToarS79xGFCsxiEppwodel7qR4aIjWjHGwXZwpSQqckgstcFCQGNTRlSbZX0574QzNOhAZBa2aGxComeuJsFrCq79JJHhhkPWy1OTSFlxAUi0TdKohtt4tDGOfHW2ZmwoBnYM3xuwNrvMH2m4G/uWMbiISwIvxdb5B/TSCSAKJCdrcG/vauy6SZTDn8g7wCK7KRIOCaJnnXRGjwFVB7mZ8PBqEyCUUhBgex6fvWWgeeoblNqe/ieeWNNQbXEix738Ru57Ci0By6daC7Nl93DvaxDcN6Apq0ZK/baZK1sJnDZi4kHUg2M4TWmJAqxhPh1DOeo8s5GbtB+RxT9bhwyfM7GxLHYzppx9MJc9jjRmeObTEu8wSRUUzyPuMkBUl0IotLd830hLOYaWUqvXWtmPi/Va5vBju09aEs/kFgDq1D0oCXg1k/O3dCqQzrXFFlCxbJOjZrXAuYNsDqgAsyf+/85uvmCmebG34uf9jq7x9UL98yeoleoXXkox20j96jI3SKKPqMvqNf6HfnU+dL52vn2wxdXKhsXqDa6vz4C3FeMPk=</latexit><latexit sha1_base64="s6BIwT4drCR6QCX+SZn7AuZMIgA=">AAAF/HicfZRLb9QwEIDd0oWyvFo4ctmylyKtqqRUbS+VqqIKjqXqS6pXK8eZzVrrOMF2+rLMT+BXcEPcECfgf/BvSLKR2MQRjhSNPN+8PGMHKWdKe96fhcV7S537D5Yfdh89fvL02crq8zOVZJLCKU14Ii8CooAzAaeaaQ4XqQQSBxzOg+nbQn9+BVKxRJzo2xSGMYkEGzNKdL41WlnHEYljMjJiant7vXT9biR6eG0Pr017OGZhD1+ZGzsSr0crfW/DK1fPFfxK6O+j2ToarS79xGFCsxiEppwodel7qR4aIjWjHGwXZwpSQqckgstcFCQGNTRlSbZX0574QzNOhAZBa2aGxComeuJsFrCq79JJHhhkPWy1OTSFlxAUi0TdKohtt4tDGOfHW2ZmwoBnYM3xuwNrvMH2m4G/uWMbiISwIvxdb5B/TSCSAKJCdrcG/vauy6SZTDn8g7wCK7KRIOCaJnnXRGjwFVB7mZ8PBqEyCUUhBgex6fvWWgeeoblNqe/ieeWNNQbXEix738Ru57Ci0By6daC7Nl93DvaxDcN6Apq0ZK/baZK1sJnDZi4kHUg2M4TWmJAqxhPh1DOeo8s5GbtB+RxT9bhwyfM7GxLHYzppx9MJc9jjRmeObTEu8wSRUUzyPuMkBUl0IotLd830hLOYaWUqvXWtmPi/Va5vBju09aEs/kFgDq1D0oCXg1k/O3dCqQzrXFFlCxbJOjZrXAuYNsDqgAsyf+/85uvmCmebG34uf9jq7x9UL98yeoleoXXkox20j96jI3SKKPqMvqNf6HfnU+dL52vn2wxdXKhsXqDa6vz4C3FeMPk=</latexit><latexit sha1_base64="TPTl4FzVvO4VXZrW2Hc+rC2QUOU=">AAAF/HicfZRLb9QwEIDd0oWyvLZw5JKylyKtqqRUbS+VqqIKjqXqS6pXKyeZ3bXWcYLt9GWZn8Cv4Ia4IU7A/+Df4KSR2MQRjhSNPN+8PGOHGaNS+f6fhcV7S537D5Yfdh89fvL0WW/l+alMcxHBSZSyVJyHRAKjHE4UVQzOMwEkCRmchbO3hf7sEoSkKT9WNxkMEzLhdEwjouzWqLeGJyRJyEjzmfF2vWztdsQ9vLqLV2ceTmjs4Ut9bUb89ajX99f9cnmuEFRCH1XrcLSy9BPHaZQnwFXEiJQXgZ+poSZC0YiB6eJcQkaiGZnAhRU5SUAOdVmS8Wra42CoxylXwKOamSaJTIiaOpsFLOu70dQGBlEPW20OdeElBkknvG4VJqbbxTGM7fGWmek4ZDkYffRu32h/sPVmEGxsmwYiIK6IYMcf2K8JTAQAr5CdzUGwteMyWS4yBv8gv8CKbARwuIpS2zUea3wJkbmw54OBy1xAUYjGYaL7gTHGge9Qa1Pqu3heeW20xrUEy943sZs5rCjUQjcOdNvm69bBPrZhWE1BkZbsVTtN8hY2d9jchYQDiWaG0BoTMklZyp16xnN0OSdjNyibY6oeFy6ZvbMxcTxm03Y8m1KHPWp05sgU4zJPEDFJiO0zTjMQRKWiuHRXVE0ZTaiSutIb14ry/1tZfTPYgakPZfEPQ31gHDIKWTmY9bNzJzQScZ0rqmzBJqKO3TWuBcwaYHXABWnfu6D5urnC6cZ6YOUPm/29/erlW0Yv0Su0hgK0jfbQe3SITlCEPqPv6Bf63fnU+dL52vl2hy4uVDYvUG11fvwFH48wuQ==</latexit>
KX
k=1
�nk = 1<latexit sha1_base64="kB+keM/zdRKnmv9weAjlrJQEy8Q=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt4u3XCoVBVVIHEpVV/SZlk5yWw22sQJttOX5V/CT+AvcOGGuCFOwJ/ByQaxiSMcKRrNfPOwZ2w3jUIuLOvnyuqttdbtO+t32/fuP3j4aGPz8SlPMubBiZdECTt3CYcopHAiQhHBecqAxG4EZ+78VW4/uwDGw4Qei+sUxjEJaDgNPSK0arJxIJ0iyIgF7lhaWzvDF9i2e9ZW38Y2HmjBermNh33l8CyeyPkuVu/fdpyAxDGZSDpXnd0OVpONrgaL1TEFXArdPbRYh5PNtc+On3hZDFR4EeF8hK1UjCVhIvQiUG0n45ASb04CGGmRkhj4WBalqk7FeozHcppQAdSruEkS85iImaHMYV7VejOdGFg1bakcyzyKDzwMaNXLjVW77fgw1WdfVCZ9N8pAyaPX+0pavZ3tHu4PVQ1h4JcEtq2e/upAwABoidiDHt6xTSbNWBrBP8jKsbwaBhQuvUR3h/rSuQBPjfT5OEB5xiDfiHTcWHaxUsqAF6j2KextZ9l4pWQ5KX8LdC7klapj10tYvlENXRvQTVOsGwP70IQ5YgaCNFQvmmmSNbCZwWYmxAyI1SuExpyQ8jBKqLGf6RJdzMnUTBotMWWP85CRvtA+MSKms2Y8nYUGe1TrzJHKx2WZICyIie6zk6TAiEhYfukuQzGLwjgUXJZ2ZXqF9P9e2l5PdqCqQ5n/XVceKIP03KgYzOrZmRPqMb/K5btswAJWxRaNawDTGlgecE7q9w7XXzdTOO1vYS2/G3T39suXbx09Rc/Qc4TREO2hN+gQnSAPfULf0S/0u/Wx9aX1tfVtga6ulD5PUGW1fvwBmCw5yA==</latexit><latexit sha1_base64="kB+keM/zdRKnmv9weAjlrJQEy8Q=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt4u3XCoVBVVIHEpVV/SZlk5yWw22sQJttOX5V/CT+AvcOGGuCFOwJ/ByQaxiSMcKRrNfPOwZ2w3jUIuLOvnyuqttdbtO+t32/fuP3j4aGPz8SlPMubBiZdECTt3CYcopHAiQhHBecqAxG4EZ+78VW4/uwDGw4Qei+sUxjEJaDgNPSK0arJxIJ0iyIgF7lhaWzvDF9i2e9ZW38Y2HmjBermNh33l8CyeyPkuVu/fdpyAxDGZSDpXnd0OVpONrgaL1TEFXArdPbRYh5PNtc+On3hZDFR4EeF8hK1UjCVhIvQiUG0n45ASb04CGGmRkhj4WBalqk7FeozHcppQAdSruEkS85iImaHMYV7VejOdGFg1bakcyzyKDzwMaNXLjVW77fgw1WdfVCZ9N8pAyaPX+0pavZ3tHu4PVQ1h4JcEtq2e/upAwABoidiDHt6xTSbNWBrBP8jKsbwaBhQuvUR3h/rSuQBPjfT5OEB5xiDfiHTcWHaxUsqAF6j2KextZ9l4pWQ5KX8LdC7klapj10tYvlENXRvQTVOsGwP70IQ5YgaCNFQvmmmSNbCZwWYmxAyI1SuExpyQ8jBKqLGf6RJdzMnUTBotMWWP85CRvtA+MSKms2Y8nYUGe1TrzJHKx2WZICyIie6zk6TAiEhYfukuQzGLwjgUXJZ2ZXqF9P9e2l5PdqCqQ5n/XVceKIP03KgYzOrZmRPqMb/K5btswAJWxRaNawDTGlgecE7q9w7XXzdTOO1vYS2/G3T39suXbx09Rc/Qc4TREO2hN+gQnSAPfULf0S/0u/Wx9aX1tfVtga6ulD5PUGW1fvwBmCw5yA==</latexit><latexit sha1_base64="kB+keM/zdRKnmv9weAjlrJQEy8Q=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt4u3XCoVBVVIHEpVV/SZlk5yWw22sQJttOX5V/CT+AvcOGGuCFOwJ/ByQaxiSMcKRrNfPOwZ2w3jUIuLOvnyuqttdbtO+t32/fuP3j4aGPz8SlPMubBiZdECTt3CYcopHAiQhHBecqAxG4EZ+78VW4/uwDGw4Qei+sUxjEJaDgNPSK0arJxIJ0iyIgF7lhaWzvDF9i2e9ZW38Y2HmjBermNh33l8CyeyPkuVu/fdpyAxDGZSDpXnd0OVpONrgaL1TEFXArdPbRYh5PNtc+On3hZDFR4EeF8hK1UjCVhIvQiUG0n45ASb04CGGmRkhj4WBalqk7FeozHcppQAdSruEkS85iImaHMYV7VejOdGFg1bakcyzyKDzwMaNXLjVW77fgw1WdfVCZ9N8pAyaPX+0pavZ3tHu4PVQ1h4JcEtq2e/upAwABoidiDHt6xTSbNWBrBP8jKsbwaBhQuvUR3h/rSuQBPjfT5OEB5xiDfiHTcWHaxUsqAF6j2KextZ9l4pWQ5KX8LdC7klapj10tYvlENXRvQTVOsGwP70IQ5YgaCNFQvmmmSNbCZwWYmxAyI1SuExpyQ8jBKqLGf6RJdzMnUTBotMWWP85CRvtA+MSKms2Y8nYUGe1TrzJHKx2WZICyIie6zk6TAiEhYfukuQzGLwjgUXJZ2ZXqF9P9e2l5PdqCqQ5n/XVceKIP03KgYzOrZmRPqMb/K5btswAJWxRaNawDTGlgecE7q9w7XXzdTOO1vYS2/G3T39suXbx09Rc/Qc4TREO2hN+gQnSAPfULf0S/0u/Wx9aX1tfVtga6ulD5PUGW1fvwBmCw5yA==</latexit><latexit sha1_base64="1JyZBQ5Z7F2Urhr97RI+dmF3euU=">AAAGGXicfZRLb9QwEIDd0oWyvFo4clmxFw6rKt6WbjhUqooqkLiUqi9ps6ycZDYbreME2+nL8i/hJ/AXuHBD3BAn4M/gbIPYxBGOFI1mvnnYM7af0VhIx/m5tHxrpXX7zurd9r37Dx4+Wlt/fCLSnAdwHKQ05Wc+EUBjBscylhTOMg4k8Smc+rNXhf30HLiIU3YkrzIYJSRi8SQOiDSq8dq+8uZBhjzyR8rZ2B68wK7bczb6LnbxlhGcl5t40NeeyJOxmu1g/f5tx4tIkpCxYjPd2elgPV7rGnC+OraAS6GLynUwXl/57IVpkCfAZECJEEPsZHKkCJdxQEG3vVxARoIZiWBoREYSECM1L1V3KtYjPFKTlElgQcVNkUQkRE4tZQGLqjaYmsTAq2lL5UgVUUIQccSqXn6i220vhIk5+3llKvRpDlodvt7Tyultb/Zwf6BrCIewJLDr9MxXByIOwErE3erhbddmspxnFP5BToEV1XBgcBGkpjssVN45BHpozscDJnIOxUaU5yeqi7XWFnyDGp+5ve0tGi+1Kiflb4HeubrUdexqASs2aqArC7puinVtYR+aME9OQZKG6mUzTfIGNrfY3Ia4BfF6hdCYEzIR05RZ+5ks0PM5mdhJ6QJT9rgISc2FDokVMZs249k0ttjDWmcOdTEuiwThUUJMn700A05kyotLdxHLKY2TWApV2rXtFbP/exl7Pdm+rg5l8fd9ta8tMvDpfDCrZ2dPaMDDKlfssgGLeBW7aVwDmNXA8oAL0rx3uP662cJJfwMb+d1Wd3evfPlW0VP0DD1HGA3QLnqDDtAxCtAn9B39Qr9bH1tfWl9b327Q5aXS5wmqrNaPP0ZdOYg=</latexit>
Weights sum to 1:
Algorithm
Initialize parameters to
Repeat until convergence
1. Update cluster assignments
2. Update parameters
✓ := {µ1:K ,⌃1:K ,⇡}
“Hard EM” with Gaussians
Parameter Updates
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 znk xn
⌃k =1Nk
PNn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 znk xn
⌃k =1Nk
PNn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
Assignment Update
“Hard EM” with Gaussians
Initialize parameters randomlywhile not converged
1. E-Step:Set the latent variables to the the values that maximizes likelihood, treating parameters as observed
2. M-Step:Set the parameters to the values that maximizes likelihood, treating latent variables as observed
Slide credit: Matt Gormley and Eric Xing (CMU)
“Hard” EM: General
Hallucinate labels
Initialize parameters randomlywhile not converged
1. E-Step:Set the latent variables to the the values that maximizes likelihood, treating parameters as observed
2. M-Step:Set the parameters to the values that maximizes likelihood, treating latent variables as observed
Slide credit: Matt Gormley and Eric Xing (CMU)
“Hard” EM: General
Hallucinate labels
Train (as if supervised)
Initialize parameters randomlywhile not converged
1. E-Step:Set the latent variables to the the values that maximizes likelihood, treating parameters as observed
2. M-Step:Set the parameters to the values that maximizes likelihood, treating latent variables as observed
Slide credit: Matt Gormley and Eric Xing (CMU)
“Hard” EM: General
Algorithm 1 Hard EM for MMs
1: procedure H EM(D = {t(i)}Ni=1)
2: Randomly initialize parameters, �, �3: while not converged do4: E-Step:
z(i) � �`;K�tz
HQ; p(t(i)|z; �) + HQ; p(z; �)
5: M-Step:
� � �`;K�t�
N�
i=1
HQ; p(z(i); �)
� � �`;K�t�
N�
i=1
HQ; p(t(i)|z; �)
6: return (�, �)
Slide credit: Matt Gormley and Eric Xing (CMU)
Algorithm 1 Hard EM for MMs
1: procedure H EM(D = {t(i)}Ni=1)
2: Randomly initialize parameters, �, �3: while not converged do4: E-Step:
z(i) � �`;K�tz
HQ; p(t(i)|z; �) + HQ; p(z; �)
5: M-Step:
� � �`;K�t�
N�
i=1
HQ; p(z(i); �)
� � �`;K�t�
N�
i=1
HQ; p(t(i)|z; �)
6: return (�, �)
Just loop over potential assignments
Slide credit: Matt Gormley and Eric Xing (CMU)
Algorithm 1 Hard EM for MMs
1: procedure H EM(D = {t(i)}Ni=1)
2: Randomly initialize parameters, �, �3: while not converged do4: E-Step:
z(i) � �`;K�tz
HQ; p(t(i)|z; �) + HQ; p(z; �)
5: M-Step:
� � �`;K�t�
N�
i=1
HQ; p(z(i); �)
� � �`;K�t�
N�
i=1
HQ; p(t(i)|z; �)
6: return (�, �)
Supervised learning
Just loop over potential assignments
Slide credit: Matt Gormley and Eric Xing (CMU)
Algorithm 1 Hard EM for GMMs
1: procedure H EM(D = {t(i)}Ni=1)
2: Randomly initialize parameters, �, µ,�3: while not converged do4: E-Step:
z(i) � �`;K�tz
HQ; p(t(i)|z; µ,�) + HQ; p(z; �)
5: M-Step:
�k � 1
N
N�
i=1
I(z(i) = k), �k
µk ��N
i=1 I(z(i) = k)t(i)
�Ni=1 I(z(i) = k)
, �k
�k ��N
i=1 I(z(i) = k)(t(i) � µk)(t(i) � µk)T
�Ni=1 I(z(i) = k)
, �k
6: return (�, µ,�)
Slide credit: Matt Gormley and Eric Xing (CMU)
Algorithm 1 Hard EM for GMMs
1: procedure H EM(D = {t(i)}Ni=1)
2: Randomly initialize parameters, �, µ,�3: while not converged do4: E-Step:
z(i) � �`;K�tz
HQ; p(t(i)|z; µ,�) + HQ; p(z; �)
5: M-Step:
�k � 1
N
N�
i=1
I(z(i) = k), �k
µk ��N
i=1 I(z(i) = k)t(i)
�Ni=1 I(z(i) = k)
, �k
�k ��N
i=1 I(z(i) = k)(t(i) � µk)(t(i) � µk)T
�Ni=1 I(z(i) = k)
, �k
6: return (�, µ,�)
Slide credit: Matt Gormley and Eric Xing (CMU)
Algorithm 1 Hard EM for GMMs
1: procedure H EM(D = {t(i)}Ni=1)
2: Randomly initialize parameters, �, µ,�3: while not converged do4: E-Step:
z(i) � �`;K�tz
HQ; p(t(i)|z; µ,�) + HQ; p(z; �)
5: M-Step:
�k � 1
N
N�
i=1
I(z(i) = k), �k
µk ��N
i=1 I(z(i) = k)t(i)
�Ni=1 I(z(i) = k)
, �k
�k ��N
i=1 I(z(i) = k)(t(i) � µk)(t(i) � µk)T
�Ni=1 I(z(i) = k)
, �k
6: return (�, µ,�)
Slide credit: Matt Gormley and Eric Xing (CMU)
Parameter Updates
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 znk xn
⌃k =1Nk
PNn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 znk xn
⌃k =1Nk
PNn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
Assignment Update
How can we deal with overlapping clusters in a better way?
“Hard EM” with Gaussians
Learn Soft Assignments to ClustersPosterior on Cluster Assignments (from Bayes’ Rule)
�nk = p(zn=k | xn) =p(xn | zn=k)p(zn=k)
p(xn)<latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="01dvNLe0oQZXqM25sO+shPAxN0c=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0EPVOhqtLn3GQULzGLikEcmyS89N5VARIRmNQHdxnkFK6JSEcGlETmLIhqo8E+3UrCfeUI0TLoHTmpsicRYTObGUBZzVtXRiEoOop62UQ1VECSBjIa97+bHudnEAY9OfsjIV+FEOWh2/P9DKHWy/HXibO7qBCAgqwtt1B+ZrAqEA4BWyuzXwtndtJs1FGsE95BZYUY0ADtc0MW3ngcJXQPWlOR8MPMsFFBtR2I9Vz9NaW/AMNT6lvYvnjTdaKVwrsGx+E7udw4qNGujWgu7aYt1Z2Kc2DMsJSNJSvWynSd7C5hab25CwINGsEFpzQpqxKOHWfsZzdDknYztpNMdUPS5CRubSB8SKmE7a8XTCLPa40ZljXYzLPEFEGBPTZ5ykIIhMRHHprpmcRCxmMlOVXdtejP/fy9ibyQ51fSiLv++rQ22R1I/KwayfnT2hVAR1rthlCxaKOjZrXAuYNsDqgAvSvHde83WzhbPNDc/IH7d6+wfVy7eMXqM3aB15aAftow/oCJ0iir6iX+gP+tv50vne+dH5OUMXFyqfV6i2Or//AdjXSHE=</latexit>
Likelihood Prior
Marginal LikelihoodPosterior
Learn Soft Assignments to ClustersPosterior on Cluster Assignments (from Bayes’ Rule)
�nk = p(zn=k | xn) =p(xn | zn=k)p(zn=k)
p(xn)<latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="adDTJFR8pS/pk6Yf7RSUEzzGvIU=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0NtHs3U0Wl36jIOE5jFwSSOSZZeem8qhIkIyGoHu4jyDlNApCeHSiJzEkA1VeSbaqVlPvKEaJ1wCpzU3ReIsJnJiKQs4q2vpxCQGUU9bKYeqiBJAxkJe9/Jj3e3iAMamP2VlKvCjHLQ6fn+glTvYfjvwNnd0AxEQVIS36w7M1wRCAcArZHdr4G3v2kyaizSCe8gtsKIaARyuaWLazgOFr4DqS3M+GHiWCyg2orAfq56ntbbgGWp8SnsXzxtvtFK4VmDZ/CZ2O4cVGzXQrQXdtcW6s7BPbRiWE5CkpXrZTpO8hc0tNrchYUGiWSG05oQ0Y1HCrf2M5+hyTsZ20miOqXpchIzMpQ+IFTGdtOPphFnscaMzx7oYl3mCiDAmps84SUEQmYji0l0zOYlYzGSmKru2vRj/v5exN5Md6vpQFn/fV4faIqkflYNZPzt7QqkI6lyxyxYsFHVs1rgWMG2A1QEXpHnvvObrZgtnmxuekT9u9fYPqpdvGb1Gb9A68tAO2kcf0BE6RRR9Rb/QH/S386XzvfOj83OGLi5UPq9QbXV+/wMqtUix</latexit><latexit sha1_base64="01dvNLe0oQZXqM25sO+shPAxN0c=">AAAGPXicfZRLa9wwEICVNNum21fSHntxupcsLMFOQ5JLIKSE9piGvCBaFlme9Yq1ZVeW8xL6Z4X+hJ77A3orvZXe2speQ9aWqQxmmPnmIc1IfhqxTLrut4XFB0udh4+WH3efPH32/MXK6suzLMkFhVOaRIm48EkGEeNwKpmM4CIVQGI/gnN/+q6wn1+ByFjCT+RtCsOYhJyNGSXSqEYrDIckjslI8al29px0/W7EHby2h9ecqYNjFjj4St3oEe8bKx4LQlW6Xqlm9jmHft2/r+/Zvh6t9NwNt1yOLXiV0EPVOhqtLn3GQULzGLikEcmyS89N5VARIRmNQHdxnkFK6JSEcGlETmLIhqo8E+3UrCfeUI0TLoHTmpsicRYTObGUBZzVtXRiEoOop62UQ1VECSBjIa97+bHudnEAY9OfsjIV+FEOWh2/P9DKHWy/HXibO7qBCAgqwtt1B+ZrAqEA4BWyuzXwtndtJs1FGsE95BZYUY0ADtc0MW3ngcJXQPWlOR8MPMsFFBtR2I9Vz9NaW/AMNT6lvYvnjTdaKVwrsGx+E7udw4qNGujWgu7aYt1Z2Kc2DMsJSNJSvWynSd7C5hab25CwINGsEFpzQpqxKOHWfsZzdDknYztpNMdUPS5CRubSB8SKmE7a8XTCLPa40ZljXYzLPEFEGBPTZ5ykIIhMRHHprpmcRCxmMlOVXdtejP/fy9ibyQ51fSiLv++rQ22R1I/KwayfnT2hVAR1rthlCxaKOjZrXAuYNsDqgAvSvHde83WzhbPNDc/IH7d6+wfVy7eMXqM3aB15aAftow/oCJ0iir6iX+gP+tv50vne+dH5OUMXFyqfV6i2Or//AdjXSHE=</latexit>
Likelihood Prior
Marginal LikelihoodPosterior
p(zn=k) = ⇡k
p(xn | zn=k) =1p
2⇡|⌃|e�
12 (xn�µk)>⌃�1(xn�µk)
p(4vxn) =KX
k=1
p(xn | zn=k)p(zn=k)<latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="QaDSdZsdnnw46aiXjE1zxHFSMx8=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMiykoj4Vllum6qCsbext+3v7I/sxwwmOaaNLW8KhIPOd246x8eNfZIw2/5dW3iw+LD+aOmx+eTps+cvlldeniRRShE+RpEf0TMXJtgnIT5mhPn4LKYYBq6PT93Je6U/vcQ0IVF4xKYx7gdwFJIhQZDJq8Hyn3j9ZhBaYLULVq1J0zLXuhaIyWBiAtDrxKxvxuvgkl8LBQXEsySdwZOmuWZ2TTCkEHFHcJBcUMbbyti6lRbgkIwCKG6FMPE537jj2sK6c7hhKTBIxWDSPAcsiq17Q2nj/AMVMjeZ1uZM1bRUykkaDPik64jzT5ae8X19xXIHyw37rZ0dSxecXGgY+dkfrCz+Al6E0gCHDPkwSXqOLR+JQ8oI8rHMLE1wDNEEjnBPiiEMcNLnWZ+EVdAeOX0+jEKGQ1Qw4zBIAsjG2qWCk+ItGsvAmBbD5pd9rrx4OCGjsGjlBsI0gYeHcmayzLjn+ikW/ODDruB2a+tdy2lvixJCsZcTTsduyV8ZGFGMwxzpbLacrY7OxCmNfXwP2QpT2VAc4isUBQEMPQ4uMRI9+T4Ah0lKsSqEAzfgDUfIeSrDM1TaZHoTzCuvBeegkGA2G2VsOoepQiU01aCbKl83GnZRhQE2xgxWZM+qaZhWsKnGpjpENYiWM8SVMXGcED8KtXqGc3Q2J0M9qD/H5D1WLn25iDyoeYzH1Xg8Jhp7UOrMgVDjMk9AKveF7DOIYkwhi6j66K4IG/skICzhuV7oViT8v5XUl4PtieJQqn/X5XtCI5HrZ4NZfDt9QhH1ipyqsgIb0SI2a1wFGJfA/IEVKfedU95uunDSfutI+fNmY2c333xLxmvjjbFuOMa2sWN8NPaNYwPVUO1b7UftZ/2wPq1/qX+doQu13OaVUTj1738B/zV22Q==</latexit>
p(zn=k) = ⇡k
p(xn | zn=k) =1p
2⇡|⌃|e�
12 (xn�µk)>⌃�1(xn�µk)
p(4vxn) =KX
k=1
p(xn | zn=k)p(zn=k)<latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="QaDSdZsdnnw46aiXjE1zxHFSMx8=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMiykoj4Vllum6qCsbext+3v7I/sxwwmOaaNLW8KhIPOd246x8eNfZIw2/5dW3iw+LD+aOmx+eTps+cvlldeniRRShE+RpEf0TMXJtgnIT5mhPn4LKYYBq6PT93Je6U/vcQ0IVF4xKYx7gdwFJIhQZDJq8Hyn3j9ZhBaYLULVq1J0zLXuhaIyWBiAtDrxKxvxuvgkl8LBQXEsySdwZOmuWZ2TTCkEHFHcJBcUMbbyti6lRbgkIwCKG6FMPE537jj2sK6c7hhKTBIxWDSPAcsiq17Q2nj/AMVMjeZ1uZM1bRUykkaDPik64jzT5ae8X19xXIHyw37rZ0dSxecXGgY+dkfrCz+Al6E0gCHDPkwSXqOLR+JQ8oI8rHMLE1wDNEEjnBPiiEMcNLnWZ+EVdAeOX0+jEKGQ1Qw4zBIAsjG2qWCk+ItGsvAmBbD5pd9rrx4OCGjsGjlBsI0gYeHcmayzLjn+ikW/ODDruB2a+tdy2lvixJCsZcTTsduyV8ZGFGMwxzpbLacrY7OxCmNfXwP2QpT2VAc4isUBQEMPQ4uMRI9+T4Ah0lKsSqEAzfgDUfIeSrDM1TaZHoTzCuvBeegkGA2G2VsOoepQiU01aCbKl83GnZRhQE2xgxWZM+qaZhWsKnGpjpENYiWM8SVMXGcED8KtXqGc3Q2J0M9qD/H5D1WLn25iDyoeYzH1Xg8Jhp7UOrMgVDjMk9AKveF7DOIYkwhi6j66K4IG/skICzhuV7oViT8v5XUl4PtieJQqn/X5XtCI5HrZ4NZfDt9QhH1ipyqsgIb0SI2a1wFGJfA/IEVKfedU95uunDSfutI+fNmY2c333xLxmvjjbFuOMa2sWN8NPaNYwPVUO1b7UftZ/2wPq1/qX+doQu13OaVUTj1738B/zV22Q==</latexit>
p(zn=k) = ⇡k
p(xn | zn=k) =1p
2⇡|⌃|e�
12 (xn�µk)>⌃�1(xn�µk)
p(4vxn) =KX
k=1
p(xn | zn=k)p(zn=k)<latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="QaDSdZsdnnw46aiXjE1zxHFSMx8=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMiykoj4Vllum6qCsbext+3v7I/sxwwmOaaNLW8KhIPOd246x8eNfZIw2/5dW3iw+LD+aOmx+eTps+cvlldeniRRShE+RpEf0TMXJtgnIT5mhPn4LKYYBq6PT93Je6U/vcQ0IVF4xKYx7gdwFJIhQZDJq8Hyn3j9ZhBaYLULVq1J0zLXuhaIyWBiAtDrxKxvxuvgkl8LBQXEsySdwZOmuWZ2TTCkEHFHcJBcUMbbyti6lRbgkIwCKG6FMPE537jj2sK6c7hhKTBIxWDSPAcsiq17Q2nj/AMVMjeZ1uZM1bRUykkaDPik64jzT5ae8X19xXIHyw37rZ0dSxecXGgY+dkfrCz+Al6E0gCHDPkwSXqOLR+JQ8oI8rHMLE1wDNEEjnBPiiEMcNLnWZ+EVdAeOX0+jEKGQ1Qw4zBIAsjG2qWCk+ItGsvAmBbD5pd9rrx4OCGjsGjlBsI0gYeHcmayzLjn+ikW/ODDruB2a+tdy2lvixJCsZcTTsduyV8ZGFGMwxzpbLacrY7OxCmNfXwP2QpT2VAc4isUBQEMPQ4uMRI9+T4Ah0lKsSqEAzfgDUfIeSrDM1TaZHoTzCuvBeegkGA2G2VsOoepQiU01aCbKl83GnZRhQE2xgxWZM+qaZhWsKnGpjpENYiWM8SVMXGcED8KtXqGc3Q2J0M9qD/H5D1WLn25iDyoeYzH1Xg8Jhp7UOrMgVDjMk9AKveF7DOIYkwhi6j66K4IG/skICzhuV7oViT8v5XUl4PtieJQqn/X5XtCI5HrZ4NZfDt9QhH1ipyqsgIb0SI2a1wFGJfA/IEVKfedU95uunDSfutI+fNmY2c333xLxmvjjbFuOMa2sWN8NPaNYwPVUO1b7UftZ/2wPq1/qX+doQu13OaVUTj1738B/zV22Q==</latexit>
p(zn=k) = ⇡k
p(xn | zn=k) =1p
2⇡|⌃|e�
12 (xn�µk)>⌃�1(xn�µk)
p(4vxn) =KX
k=1
p(xn | zn=k)p(zn=k)<latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="oGWb7C43Dd5WdkHiuM9XKD/emTg=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMi2koj4ospy21QVjL2NvW1/Z39kP2YwyTFtbHlTIBx0vnPTOT4uCXDCbPt3beHB4sP6o6XH5pOnz56/WF55eZLEKfXQsRcHMT1zYYICHKFjhlmAzghFMHQDdOpO3iv96SWiCY6jIzYlqB/CUYSH2INMXg2W/5D1m0FkgdUuWLUmTctc61qA4MHEBKDXIaxvknVwya+FgkLsW5LO4EnTXDO7JhhS6HFHcJBcUMbbyti6lRbgEI9CKG6FMNE537jj2sK6c7hhKTBMxWDSPAcsJta9obRx/oEKmZtMa3Omaloq5SQNB3zSdcT5J0vP+L6+YrmD5Yb91s6OpQtOLjR2jNnZH6ws/gJ+7KUhipgXwCTpObZ8JA4pw16AZGZpggj0JnCEelKMYIiSPs/6JKyC9sjp82EcMRR5BTMOwySEbKxdKjgp3npjGRjRYtj8ss+VFx8leBQVrdxQmCbw0VDOTJYZ990gRYIffNgV3G5tvWs57W1RQijyc8Lp2C35KwMjilCUI53NlrPV0RmSUhKge8hWmMqGoghdeXEYwsjn4BJ5oiffB6AoSSlShXDghrzhCDlPZXiGSptMb4J55bXgHBQSzGajjE3nMFWohKYadFPl60bDLqowwMaIwYrsWTUN0wo21dhUh6gG0XKGqDImIgkO4kirZzhHZ3My1IMGc0zeY+UykIvIh5pHMq7GyRhr7EGpMwdCjcs8AancF7LPICaIQhZT9dFdYTYOcIhZwnO90K1w9H8rqS8H2xPFoVT/rsv3hEZ6bpANZvHt9An1qF/kVJUV2IgWsVnjKkBSAvMHVqTcd055u+nCSfutI+XPm42d3XzzLRmvjTfGuuEY28aO8dHYN44Nr+bVvtV+1H7WD+vT+pf61xm6UMttXhmFU//+F1ETdxk=</latexit><latexit sha1_base64="QaDSdZsdnnw46aiXjE1zxHFSMx8=">AAAG0XicfZRba9swFICdrss679Zuj3txFygNpMPOSpuXQOkoG+yl6x2iNMiykoj4Vllum6qCsbext+3v7I/sxwwmOaaNLW8KhIPOd246x8eNfZIw2/5dW3iw+LD+aOmx+eTps+cvlldeniRRShE+RpEf0TMXJtgnIT5mhPn4LKYYBq6PT93Je6U/vcQ0IVF4xKYx7gdwFJIhQZDJq8Hyn3j9ZhBaYLULVq1J0zLXuhaIyWBiAtDrxKxvxuvgkl8LBQXEsySdwZOmuWZ2TTCkEHFHcJBcUMbbyti6lRbgkIwCKG6FMPE537jj2sK6c7hhKTBIxWDSPAcsiq17Q2nj/AMVMjeZ1uZM1bRUykkaDPik64jzT5ae8X19xXIHyw37rZ0dSxecXGgY+dkfrCz+Al6E0gCHDPkwSXqOLR+JQ8oI8rHMLE1wDNEEjnBPiiEMcNLnWZ+EVdAeOX0+jEKGQ1Qw4zBIAsjG2qWCk+ItGsvAmBbD5pd9rrx4OCGjsGjlBsI0gYeHcmayzLjn+ikW/ODDruB2a+tdy2lvixJCsZcTTsduyV8ZGFGMwxzpbLacrY7OxCmNfXwP2QpT2VAc4isUBQEMPQ4uMRI9+T4Ah0lKsSqEAzfgDUfIeSrDM1TaZHoTzCuvBeegkGA2G2VsOoepQiU01aCbKl83GnZRhQE2xgxWZM+qaZhWsKnGpjpENYiWM8SVMXGcED8KtXqGc3Q2J0M9qD/H5D1WLn25iDyoeYzH1Xg8Jhp7UOrMgVDjMk9AKveF7DOIYkwhi6j66K4IG/skICzhuV7oViT8v5XUl4PtieJQqn/X5XtCI5HrZ4NZfDt9QhH1ipyqsgIb0SI2a1wFGJfA/IEVKfedU95uunDSfutI+fNmY2c333xLxmvjjbFuOMa2sWN8NPaNYwPVUO1b7UftZ/2wPq1/qX+doQu13OaVUTj1738B/zV22Q==</latexit>
p(zn=k) = ⇡k
p(xn | zn=k) =1p
2⇡|⌃|e�
12 (xn�µk)>⌃�1(xn�µk)
p(xn) =KX
k=1
p(xn | zn=k)p(zn=k)<latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="RgVg34VZMZaqts3thXMJPnaoqpI=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FhpWf3eHS/C8QJH4aoZj7IWSs77bVEwlIOfZDpPJKGSLQn8Ix6isxhhFiA5F1SToF7YE7EKMk5ij2C2YCRiyCfGJcapgVb/2JCoxoMWx+ORDaS4AYHsdFKy+Stg0CNFITk2UmAi9MkRT7H7alaLc23rXczqYsIRQFOeF22y31KwNjilCcI931lrvRNRmSUhKie6itMZ0NRTG68JMognEgwDnyZV+9D0AxSynShQjgRaLhSjVNZfgWVTaZ3gazykspBCgkmM1GGbuawXShCroyoOsqX9cGdlaFAT5BHFZkz6tpmFawqcGmJkQNiJYzRJUxEWE4TGKjntEMnc3JyAwazjB5j7XLUK2hABoeyaQaJxNssPulzuxLPS6zBKRqW6g+g4QgCnlC9Ud3gfkkxBHmTOR6aVrh+P9WSl8OtiOLQ6n/PU/sSIP0vTAbzOLbmRPq06DI6SorsDEtYreNqwBJCcwfWJNq37nl7WYKR523rpL31htb2/nmW7BeW2+sVcu1Nq0t66O1ax1afg3WvtV+1H7W9+oX9S/1r7foXC23eWUVTv37X2OLdhs=</latexit>
p(zn=k) = ⇡k
p(xn | zn=k) =1p
2⇡|⌃|e�
12 (xn�µk)>⌃�1(xn�µk)
p(xn) =KX
k=1
p(xn | zn=k)p(zn=k)<latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="zKK2cQwncjaaX5ViqjhoiyDPtGo=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FxpZ1e3aHS/O/QJD4aYRi7oeQsb7bVk8kIOXYD5HKK2WIQH8Kx6ivxBhGiA1E1iXpFLQH7kCMkpij2C+YCRixCPKJcalhVrz1JyowosWw+eVAaC8BYngcF628SNo2CNBITUyWmQi8MEVS7H/YlqLd2njXcjubsoRQFOSE22231K8MjClCcY5011vuRtdkSEpJiO6htsZ0NhTF6MJPogjGgQDnyJd99T4AxSylSBcigBeJhivVNJXhW1TZZHobzCovpRCgkGA2G2XsagbThSroyoCuq3xdG9hZFQb4BHFYkT2vpmFawaYGm5oQNSBazhBVxkSE4TCJjXpGM3Q2JyMzaDjD5D3WLkO1hgJoeCSTapxMsMHulzqzL/W4zBKQqm2h+gwSgijkCdUf3QXmkxBHmDOR66VpheP/Wyl9OdiOLA6l/vc8sSMN0vfCbDCLb2dOqE+DIqerrMDGtIjdNq4CJCUwf2BNqn3nlrebKRx13rpK3ltvbG3nm2/Bem29sVYt19q0tqyP1q51aPk1WPtW+1H7Wd+rX9S/1L/eonO13OaVVTj1738BtVp2Ww==</latexit><latexit sha1_base64="RgVg34VZMZaqts3thXMJPnaoqpI=">AAAGz3icfZRbb9MwFIDTMcoItw0eecmoNK1Sh5oybX2pNA1NIPGyjd2kuqucxG2t5uLZzm6eEW+IN/g//BF+Cy/YWbQ1ccCVqiOf79x8To5HQsx4u/27Nvdg/mH90cJj+8nTZ89fLC69PGJJSn106CdhQk88yFCIY3TIMQ/RCaEIRl6Ijr3pe60/PkeU4SQ+4FcEDSI4jvEI+5Crq+HiH7J6PYwdsNwDy8606dgrPQcQPJzaAPS7hA9ssgrOxaXUUIQDR9EZPG3aK3bPBiMKfeFKAdgZ5aKjjZ0bZQE+43EE5Y2UNjoVa3dcRzp3DtccDUapHE6bp4AnxLk3VDbuP1CpclNpZYqmoxNmaTQU054rTz85Zr731RWLHS422m/b2XFMwc2FhpWf3eHS/C8QJH4aoZj7IWSs77bVEwlIOfZDpPJKGSLQn8Ix6isxhhFiA5F1SToF7YE7EKMk5ij2C2YCRiyCfGJcapgVb/2JCoxoMWx+ORDaS4AYHsdFKy+Stg0CNFITk2UmAi9MkRT7H7alaLc23rXczqYsIRQFOeF22y31KwNjilCcI931lrvRNRmSUhKie6itMZ0NRTG68JMognEgwDnyZV+9D0AxSynShQjgRaLhSjVNZfgWVTaZ3gazykspBCgkmM1GGbuawXShCroyoOsqX9cGdlaFAT5BHFZkz6tpmFawqcGmJkQNiJYzRJUxEWE4TGKjntEMnc3JyAwazjB5j7XLUK2hABoeyaQaJxNssPulzuxLPS6zBKRqW6g+g4QgCnlC9Ud3gfkkxBHmTOR6aVrh+P9WSl8OtiOLQ6n/PU/sSIP0vTAbzOLbmRPq06DI6SorsDEtYreNqwBJCcwfWJNq37nl7WYKR523rpL31htb2/nmW7BeW2+sVcu1Nq0t66O1ax1afg3WvtV+1H7W9+oX9S/1r7foXC23eWUVTv37X2OLdhs=</latexit>
Prior
Likelihood
MarginalLikelihood
Learn Gaussian for Each Cluster Maximum Likelihood Estimation
µ⇤,⌃⇤,⇡⇤ = argmaxµ,⌃,⇡
log p(x1, . . . , xN | µ,⌃,⇡)<latexit sha1_base64="VGcx1UxYqdEnlbi/OsG2rnPTTNM=">AAAGZHicfZTrbtMwFIDTsZVRBmxM/AKhiAo0UDUlY9r2B2kamuAXGmM3aSmV45y21pwLtrOb5T0DD8SL8AI8Ar+x02g0cTRXqk7P+c7FPqcnzCjhwvN+t2buzc61788/6DxcePT4yeLS0yOe5gzDIU5pyk5CxIGSBA4FERROMgYoDikch2cfjf34HBgnaXIgrjLox2iUkCHBSGjVYPFncC6DOFff3/VcI34joxjd/sqIFt0PboCYVl8OZElPs7ekcm9uApqO3GxFay7VwNcmGqWCF4hWfHGDmETuXUHeDha73qpXHNcW/FLobjuTszdYmv0VRCnOY0gEpojzU9/LRF8iJgimoDpBziFD+AyN4FSLCYqB92XxdMqtWA/8vhymiYAEV9wkinmMxNhSGphXtXisEwOrpi2VfWmiRMDJKKl6hbHqdIIIhrqNRWUyCmkOSu5/2lHS62287/lrm6qGMIhKwt/yevpTB0YMICmRrfWev7FlM1nOMgr/Ic9gphoGCVzgNI5REunGA1an+n0CSHjOwFxEBmEsu75SyoInqPYp7J1g2nippAwqBRbTUceupjBzUQ1dWdB1U6xrC/vRhAViDAI1VC+aaZQ3sLnF5jbELIjVK4TGnJBxQtPEus9wii7mZGgnpVNM2WMTkurdECErYjZuxrMxsdj9Wmf2lRmXaWKyLpQM0gwYEikzf7oLIsaUxERwWdqV7UWSu720vZ5sV1WH0nyHodxVFolDWgxm9e3sCcUsqnLmlg3YiFWxSeMawKwGlg9sSL3v/Pp2s4WjtVVfy1/Xu9s75eabd547r5wVx3c2nW3ns7PnHDrY+dt60XrdejP3p73QXm4/m6AzrdJn2amc9st/8VFVCw==</latexit><latexit sha1_base64="VGcx1UxYqdEnlbi/OsG2rnPTTNM=">AAAGZHicfZTrbtMwFIDTsZVRBmxM/AKhiAo0UDUlY9r2B2kamuAXGmM3aSmV45y21pwLtrOb5T0DD8SL8AI8Ar+x02g0cTRXqk7P+c7FPqcnzCjhwvN+t2buzc61788/6DxcePT4yeLS0yOe5gzDIU5pyk5CxIGSBA4FERROMgYoDikch2cfjf34HBgnaXIgrjLox2iUkCHBSGjVYPFncC6DOFff3/VcI34joxjd/sqIFt0PboCYVl8OZElPs7ekcm9uApqO3GxFay7VwNcmGqWCF4hWfHGDmETuXUHeDha73qpXHNcW/FLobjuTszdYmv0VRCnOY0gEpojzU9/LRF8iJgimoDpBziFD+AyN4FSLCYqB92XxdMqtWA/8vhymiYAEV9wkinmMxNhSGphXtXisEwOrpi2VfWmiRMDJKKl6hbHqdIIIhrqNRWUyCmkOSu5/2lHS62287/lrm6qGMIhKwt/yevpTB0YMICmRrfWev7FlM1nOMgr/Ic9gphoGCVzgNI5REunGA1an+n0CSHjOwFxEBmEsu75SyoInqPYp7J1g2nippAwqBRbTUceupjBzUQ1dWdB1U6xrC/vRhAViDAI1VC+aaZQ3sLnF5jbELIjVK4TGnJBxQtPEus9wii7mZGgnpVNM2WMTkurdECErYjZuxrMxsdj9Wmf2lRmXaWKyLpQM0gwYEikzf7oLIsaUxERwWdqV7UWSu720vZ5sV1WH0nyHodxVFolDWgxm9e3sCcUsqnLmlg3YiFWxSeMawKwGlg9sSL3v/Pp2s4WjtVVfy1/Xu9s75eabd547r5wVx3c2nW3ns7PnHDrY+dt60XrdejP3p73QXm4/m6AzrdJn2amc9st/8VFVCw==</latexit><latexit sha1_base64="VGcx1UxYqdEnlbi/OsG2rnPTTNM=">AAAGZHicfZTrbtMwFIDTsZVRBmxM/AKhiAo0UDUlY9r2B2kamuAXGmM3aSmV45y21pwLtrOb5T0DD8SL8AI8Ar+x02g0cTRXqk7P+c7FPqcnzCjhwvN+t2buzc61788/6DxcePT4yeLS0yOe5gzDIU5pyk5CxIGSBA4FERROMgYoDikch2cfjf34HBgnaXIgrjLox2iUkCHBSGjVYPFncC6DOFff3/VcI34joxjd/sqIFt0PboCYVl8OZElPs7ekcm9uApqO3GxFay7VwNcmGqWCF4hWfHGDmETuXUHeDha73qpXHNcW/FLobjuTszdYmv0VRCnOY0gEpojzU9/LRF8iJgimoDpBziFD+AyN4FSLCYqB92XxdMqtWA/8vhymiYAEV9wkinmMxNhSGphXtXisEwOrpi2VfWmiRMDJKKl6hbHqdIIIhrqNRWUyCmkOSu5/2lHS62287/lrm6qGMIhKwt/yevpTB0YMICmRrfWev7FlM1nOMgr/Ic9gphoGCVzgNI5REunGA1an+n0CSHjOwFxEBmEsu75SyoInqPYp7J1g2nippAwqBRbTUceupjBzUQ1dWdB1U6xrC/vRhAViDAI1VC+aaZQ3sLnF5jbELIjVK4TGnJBxQtPEus9wii7mZGgnpVNM2WMTkurdECErYjZuxrMxsdj9Wmf2lRmXaWKyLpQM0gwYEikzf7oLIsaUxERwWdqV7UWSu720vZ5sV1WH0nyHodxVFolDWgxm9e3sCcUsqnLmlg3YiFWxSeMawKwGlg9sSL3v/Pp2s4WjtVVfy1/Xu9s75eabd547r5wVx3c2nW3ns7PnHDrY+dt60XrdejP3p73QXm4/m6AzrdJn2amc9st/8VFVCw==</latexit><latexit sha1_base64="Nnf+J7NPD+a2xcEGiekcvI6Ye+A=">AAAGZHicfZTdbtMwFIDTsZVRBmxMXIFQRAUaqJqSMW27QZqGJrhCY+xPmkvlJKetNScxtrM/y3sGHogX4QV4BK5x0mg0cTRXqk7P+c6PfU5PwCgR0vN+t2buzc61788/6DxcePT4yeLS0yORZjyEwzClKT8JsABKEjiURFI4YRxwHFA4Ds4+5vbjc+CCpMmBvGLQj/EoIUMSYmlUg8Wf6FyhONPf3/XcXPxGRjG+/cWIEd0PLsLcqC8HqqSn2VtSuzc3iKYjl60YzaUe+MZEo1SKAjGKLy6KSeTeFeTtYLHrrXrFcW3BL4WuU569wdLsLxSlYRZDIkOKhTj1PSb7CnNJQgq6gzIBDIdneASnRkxwDKKviqfTbsV64PfVME0kJGHFTeFYxFiOLWUOi6o2HJvEwKtpS2Vf5VEiEGSUVL2CWHc6KIKhaWNRmYoCmoFW+592tPJ6G+97/tqmriEcopLwt7ye+dSBEQdISmRrvedvbNkMyzij8B/yciyvhkMCF2EaxziJTOMh1KfmfRAkIuOQX0ShIFZdX2ttwRPU+BT2Dpo2XmqlUKXAYjrq2NUUll/UQFcWdN0U69rCfjRhSI5B4obqZTONswY2s9jMhrgF8XqF0JgTmCA0Taz7DKfoYk6GdlI6xZQ9zkNSsxsibEVk42acjYnF7tc6s6/zcZkmJutCK5Qy4FimPP/TXRA5piQmUqjSrm0vktztZez1ZLu6OpT5dxCoXW2RYUCLway+nT2hIY+qXH7LBmzEq9ikcQ0gq4HlA+ek2Xd+fbvZwtHaqm/kr+vd7Z1y8807z51XzorjO5vOtvPZ2XMOndD523rRet16M/envdBebj+boDOt0mfZqZz2y3+fglTL</latexit>
Nk =X
n
�nk<latexit sha1_base64="SDjl3WfkWXVthwbBKbP9SkNtmcg=">AAAGF3icfZRLb9NAEMe3pYESXi0cuVjkwiGK7PQRc0CqWlVwQqXqS4qjaG1PHCv22uyu+1rtB+Ej8B2QuCFuiBv9NqwdR8Rew0aKRju//8zsznjdNAoZN827ldV7a637D9Yfth89fvL02cbm8zOWZNSDUy+JEnrhYgZRSOCUhzyCi5QCjt0Izt3ZQe4/vwTKwoSc8JsURjEOSDgJPczV1njjQDhFkCEN3JEwe7uDHcu2u2avb1u2ta0M882WNejLD+OZ8dZwWBaPBZGGE+A4xsqcSTne6CyExkJoLISG1TOL1dlD83U03lz76viJl8VAuBdhxoaWmfKRwJSHXgSy7WQMUuzNcABDZRIcAxuJolBpVLwn1khMEsKBeBWZwDGLMZ9qmznMqrveVCUGWk1bbo5EHsUHFgakqnJj2W47PkzUzReVCd+NMpDi+N2+FGZ3d6tr9QeyhlDwS8Kyza761YGAApASsbe71q6tM2lG0wj+QmaO5dVQIHDlJaozxBfOJXhyqO7HAcIyCvlBhOPGomNJKTV4jipN4W87y85rKco5WRToXIprWcdulrD8oAq60aDbpli3GvapCXP4FDhuqJ430zhrYDONzXSIahCtVwiNOSFlYZQQ7TyTJbqYk4meNFpiyh7nISP1OftYi5hOm/F0Gmrsca0zxzIfl2UC0yDGqs9OkgLFPKH5R3cV8mkUxiFnovRLXRWS/6uUv57sUFaHMv93XXEoNdJzo2Iwq3enT6hH/SqXn7IBC2gVmzeuAUxrYHnBOaneu8WjZvzbOOv3LGV/3O7s7Zcv3zp6iV6h18hCA7SH3qMjdIo89AX9RL/RXetz61vre+vHHF1dKTUvUGW1fv0B0Hc52A==</latexit><latexit sha1_base64="SDjl3WfkWXVthwbBKbP9SkNtmcg=">AAAGF3icfZRLb9NAEMe3pYESXi0cuVjkwiGK7PQRc0CqWlVwQqXqS4qjaG1PHCv22uyu+1rtB+Ej8B2QuCFuiBv9NqwdR8Rew0aKRju//8zsznjdNAoZN827ldV7a637D9Yfth89fvL02cbm8zOWZNSDUy+JEnrhYgZRSOCUhzyCi5QCjt0Izt3ZQe4/vwTKwoSc8JsURjEOSDgJPczV1njjQDhFkCEN3JEwe7uDHcu2u2avb1u2ta0M882WNejLD+OZ8dZwWBaPBZGGE+A4xsqcSTne6CyExkJoLISG1TOL1dlD83U03lz76viJl8VAuBdhxoaWmfKRwJSHXgSy7WQMUuzNcABDZRIcAxuJolBpVLwn1khMEsKBeBWZwDGLMZ9qmznMqrveVCUGWk1bbo5EHsUHFgakqnJj2W47PkzUzReVCd+NMpDi+N2+FGZ3d6tr9QeyhlDwS8Kyza761YGAApASsbe71q6tM2lG0wj+QmaO5dVQIHDlJaozxBfOJXhyqO7HAcIyCvlBhOPGomNJKTV4jipN4W87y85rKco5WRToXIprWcdulrD8oAq60aDbpli3GvapCXP4FDhuqJ430zhrYDONzXSIahCtVwiNOSFlYZQQ7TyTJbqYk4meNFpiyh7nISP1OftYi5hOm/F0Gmrsca0zxzIfl2UC0yDGqs9OkgLFPKH5R3cV8mkUxiFnovRLXRWS/6uUv57sUFaHMv93XXEoNdJzo2Iwq3enT6hH/SqXn7IBC2gVmzeuAUxrYHnBOaneu8WjZvzbOOv3LGV/3O7s7Zcv3zp6iV6h18hCA7SH3qMjdIo89AX9RL/RXetz61vre+vHHF1dKTUvUGW1fv0B0Hc52A==</latexit><latexit sha1_base64="SDjl3WfkWXVthwbBKbP9SkNtmcg=">AAAGF3icfZRLb9NAEMe3pYESXi0cuVjkwiGK7PQRc0CqWlVwQqXqS4qjaG1PHCv22uyu+1rtB+Ej8B2QuCFuiBv9NqwdR8Rew0aKRju//8zsznjdNAoZN827ldV7a637D9Yfth89fvL02cbm8zOWZNSDUy+JEnrhYgZRSOCUhzyCi5QCjt0Izt3ZQe4/vwTKwoSc8JsURjEOSDgJPczV1njjQDhFkCEN3JEwe7uDHcu2u2avb1u2ta0M882WNejLD+OZ8dZwWBaPBZGGE+A4xsqcSTne6CyExkJoLISG1TOL1dlD83U03lz76viJl8VAuBdhxoaWmfKRwJSHXgSy7WQMUuzNcABDZRIcAxuJolBpVLwn1khMEsKBeBWZwDGLMZ9qmznMqrveVCUGWk1bbo5EHsUHFgakqnJj2W47PkzUzReVCd+NMpDi+N2+FGZ3d6tr9QeyhlDwS8Kyza761YGAApASsbe71q6tM2lG0wj+QmaO5dVQIHDlJaozxBfOJXhyqO7HAcIyCvlBhOPGomNJKTV4jipN4W87y85rKco5WRToXIprWcdulrD8oAq60aDbpli3GvapCXP4FDhuqJ430zhrYDONzXSIahCtVwiNOSFlYZQQ7TyTJbqYk4meNFpiyh7nISP1OftYi5hOm/F0Gmrsca0zxzIfl2UC0yDGqs9OkgLFPKH5R3cV8mkUxiFnovRLXRWS/6uUv57sUFaHMv93XXEoNdJzo2Iwq3enT6hH/SqXn7IBC2gVmzeuAUxrYHnBOaneu8WjZvzbOOv3LGV/3O7s7Zcv3zp6iV6h18hCA7SH3qMjdIo89AX9RL/RXetz61vre+vHHF1dKTUvUGW1fv0B0Hc52A==</latexit><latexit sha1_base64="uhIbF8M0t4WkHuwHZFnMrV1hqBU=">AAAGF3icfZTNbtNAEMe30EAJXy0cuVjkwiGK7LQ05oBUtarghErVLymOorU9cays12Z33a/VPgiPwDsgcUPcEDf6NqxdR8RZw0aKRju//8zsznj9jMRc2PbNyp27q61799cetB8+evzk6frGsxOe5iyA4yAlKTvzMQcSUzgWsSBwljHAiU/g1J/tFf7Tc2A8TumRuMpglOCIxpM4wEJvjdf3pFcGGbLIH0m7tz147bhu1+71Xcd1trRhv9l0Bn31YTyz3loez5OxpMryIpwkWJszpcbrnbnQmgutudByena5OqhaB+ON1a9emAZ5AlQEBHM+dOxMjCRmIg4IqLaXc8hwMMMRDLVJcQJ8JMtClVXzHjkjOUmpABrUZBInPMFiamwWMK/vBlOdGFg9bbU5kkWUEHgc0brKT1S77YUw0TdfViZDn+Sg5OG7XSXt7vZm1+kP1BLCIKwIx7W7+rcMRAyAVoi71XW2XZPJcpYR+AvZBVZUw4DCRZDqztBQeucQqKG+Hw8ozxkUB5Gen8iOo5Qy4FtUa0p/21t0XipZzcm8QO9cXqpl7GoBKw6qoSsDum6KdW1gn5owT0xB4IbqRTON8wY2N9jchJgBseUKoTEnZDwmKTXOM1mgyzmZmEnJAlP1uAhJ9OccYiNiNm3Gs2lssIdLnTlUxbgsEphFCdZ99tIMGBYpKz66i1hMSZzEgsvKr0xVTP+v0v7lZPuqPpTFv+/LfWWQgU/KwazfnTmhAQvrXHHKBixidey2cQ1gtgRWF1yQ+r2bP2rWv42Tfs/R9setzs5u9fKtoRfoJXqFHDRAO+g9OkDHKEBf0E/0G920Pre+tb63ftyid1YqzXNUW61ffwB+qDmY</latexit>
µk =1Nk
X
n
�nkxn<latexit sha1_base64="wLZYBBbHUes2ZdhmkUiMsX4ibyA=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqbS9IVVEFJ1Sq/knNauU4s1lrEyfYzvbH8hvwCDwFNwQnxA1egLfBzkZiE0c4UjSa+cYz4xk7zBMqpOf9WVm9s9a5e2/9fvfBw0ePn2xsPj0TWcEJnJIsyfhFiAUklMGppDKBi5wDTsMEzsPZG2s/nwMXNGMn8iaHUYpjRieUYGlU441hMFdBWujxrPe6F0w4JsrX6v14pnuBKNKxYkaIcZpiI1rlXF3rsXHse1teuXqu4FdCfx8t1tF4c+17EGWkSIFJkmAhLn0vlyOFuaQkAd0NCgE5JjMcw6URGU5BjFRZoO7VrCf+SE0yJoGRmpvCqUixnDpKC4u6lkxNYOD1sJVypOwuEQgas7pXmOpuN4hgYg67zExFYVKAVsdvD7TyBsNXA397VzcQDlFF+HvewHxNIOYArEL2dgb+cM9l8oLnCfyDPIvZbDgwuCKZaRCLVDAHoi/N+QTARMHBFqKCMFV9X2vtwAvU+JT2brBsvNZKBbUEy9Y3sZslzBZqoBsHum3b69bBPrZhgZyCxC3Zy3YaFy1s4bCFC3EH4s0MoTUm5IImGXPqmSzR5ZxM3KDJElP12G6ZmBscYWfHfNqO51PqsMeNzhxrOy7LBOZxik2fgywHjmXG7aW7onKa0JRKoSq7dr0o+7+XsTeDHer6UNp/GKpD7ZAkTMrBrJ+dO6GER3XOVtmCxbyOLRrXAuYNsDpgS5r3zm++bq5wtr3lG/nDTn//oHr51tFz9AK9RD7aRfvoHTpCp4igz+gH+oV+dz51vnS+dr4t0NWVyucZqq3Oz78GQDi+</latexit><latexit sha1_base64="wLZYBBbHUes2ZdhmkUiMsX4ibyA=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqbS9IVVEFJ1Sq/knNauU4s1lrEyfYzvbH8hvwCDwFNwQnxA1egLfBzkZiE0c4UjSa+cYz4xk7zBMqpOf9WVm9s9a5e2/9fvfBw0ePn2xsPj0TWcEJnJIsyfhFiAUklMGppDKBi5wDTsMEzsPZG2s/nwMXNGMn8iaHUYpjRieUYGlU441hMFdBWujxrPe6F0w4JsrX6v14pnuBKNKxYkaIcZpiI1rlXF3rsXHse1teuXqu4FdCfx8t1tF4c+17EGWkSIFJkmAhLn0vlyOFuaQkAd0NCgE5JjMcw6URGU5BjFRZoO7VrCf+SE0yJoGRmpvCqUixnDpKC4u6lkxNYOD1sJVypOwuEQgas7pXmOpuN4hgYg67zExFYVKAVsdvD7TyBsNXA397VzcQDlFF+HvewHxNIOYArEL2dgb+cM9l8oLnCfyDPIvZbDgwuCKZaRCLVDAHoi/N+QTARMHBFqKCMFV9X2vtwAvU+JT2brBsvNZKBbUEy9Y3sZslzBZqoBsHum3b69bBPrZhgZyCxC3Zy3YaFy1s4bCFC3EH4s0MoTUm5IImGXPqmSzR5ZxM3KDJElP12G6ZmBscYWfHfNqO51PqsMeNzhxrOy7LBOZxik2fgywHjmXG7aW7onKa0JRKoSq7dr0o+7+XsTeDHer6UNp/GKpD7ZAkTMrBrJ+dO6GER3XOVtmCxbyOLRrXAuYNsDpgS5r3zm++bq5wtr3lG/nDTn//oHr51tFz9AK9RD7aRfvoHTpCp4igz+gH+oV+dz51vnS+dr4t0NWVyucZqq3Oz78GQDi+</latexit><latexit sha1_base64="wLZYBBbHUes2ZdhmkUiMsX4ibyA=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqbS9IVVEFJ1Sq/knNauU4s1lrEyfYzvbH8hvwCDwFNwQnxA1egLfBzkZiE0c4UjSa+cYz4xk7zBMqpOf9WVm9s9a5e2/9fvfBw0ePn2xsPj0TWcEJnJIsyfhFiAUklMGppDKBi5wDTsMEzsPZG2s/nwMXNGMn8iaHUYpjRieUYGlU441hMFdBWujxrPe6F0w4JsrX6v14pnuBKNKxYkaIcZpiI1rlXF3rsXHse1teuXqu4FdCfx8t1tF4c+17EGWkSIFJkmAhLn0vlyOFuaQkAd0NCgE5JjMcw6URGU5BjFRZoO7VrCf+SE0yJoGRmpvCqUixnDpKC4u6lkxNYOD1sJVypOwuEQgas7pXmOpuN4hgYg67zExFYVKAVsdvD7TyBsNXA397VzcQDlFF+HvewHxNIOYArEL2dgb+cM9l8oLnCfyDPIvZbDgwuCKZaRCLVDAHoi/N+QTARMHBFqKCMFV9X2vtwAvU+JT2brBsvNZKBbUEy9Y3sZslzBZqoBsHum3b69bBPrZhgZyCxC3Zy3YaFy1s4bCFC3EH4s0MoTUm5IImGXPqmSzR5ZxM3KDJElP12G6ZmBscYWfHfNqO51PqsMeNzhxrOy7LBOZxik2fgywHjmXG7aW7onKa0JRKoSq7dr0o+7+XsTeDHer6UNp/GKpD7ZAkTMrBrJ+dO6GER3XOVtmCxbyOLRrXAuYNsDpgS5r3zm++bq5wtr3lG/nDTn//oHr51tFz9AK9RD7aRfvoHTpCp4igz+gH+oV+dz51vnS+dr4t0NWVyucZqq3Oz78GQDi+</latexit><latexit sha1_base64="iUdAi1B2oPRFFR3Sl7J0S6e2ggw=">AAAGCnicfZTNbtQwEIDd0oWy/LVw5LJiLxxWVVKqthekqqiCEypV/6R6tXKc2V1rEyfYzvbH8hvwCDwFNwQnxA1egLfBTiOxiSMcKRrNfOOZ8Ywd5QmTKgj+LC3fWencvbd6v/vg4aPHT9bWn57KrBAUTmiWZOI8IhISxuFEMZXAeS6ApFECZ9HsjbOfzUFIlvFjdZ3DMCUTzsaMEmVVo7VtPNc4Lcxo1nvdw2NBqA6Nfj+amR6WRTrS3AoTkqbEik4511dmZB37wUZQrp4vhJXQR9U6HK2vfMdxRosUuKIJkfIiDHI11EQoRhMwXVxIyAmdkQlcWJGTFORQlwWaXs16HA71OOMKOK25aZLKlKipp3SwrGvp1AYGUQ9bKYfa7RKDZBNe94pS0+3iGMb2sMvMdBwlBRh99Hbf6GCw/WoQbu6YBiIgrohwNxjYrwlMBACvkN2tQbi96zN5IfIE/kGBw1w2Ajhc0sw2iMcaz4GaC3s+GLgsBLhCNI5S3Q+NMR58i1qf0t7Fi8YrozWuJVi2voldL2CuUAtde9BN2143HvaxDcNqCoq0ZK/aaVK0sIXHFj4kPEg0M4TWmJBLlmTcq2e8QJdzMvaDJgtM1WO3ZWJvcEy8HfNpO55PmcceNTpzZNy4LBJETFJi+4yzHARRmXCX7pKpacJSpqSu7Mb3Yvz/XtbeDHZg6kPp/lGkD4xH0igpB7N+dv6EUhHXOVdlCzYRdey2cS1g3gCrA3akfe/C5uvmC6ebG6GVP2z19/arl28VPUcv0EsUoh20h96hQ3SCKPqMfqBf6HfnU+dL52vn2y26vFT5PEO11fn5F7RiOH4=</latexit>
⇡k =Nk
N<latexit sha1_base64="4s/F4xsMBYlMi+mf8agPCZwO+oo=">AAAF6XicfZRLbxMxEIDd0kAJrxTEiUtELhyiardUbS5IVVEFp6pUfUl1FHm9k8SK12tsb1+WfwQ3xA1xg7/AH+HfsJusRHa9wiutRjPfvOyxI8mZNkHwZ2X13lrr/oP1h+1Hj588fdbZeH6m00xROKUpT9VFRDRwJuDUMMPhQiogScThPJq9L+znV6A0S8WJuZUwTMhEsDGjxOSqUecllmw0677r4rEi1B6OZs4eulGnF2wG89X1hbAUentosY5GG2u/cZzSLAFhKCdaX4aBNENLlGGUg2vjTIMkdEYmcJmLgiSgh3Zev+tWrCfh0I5TYUDQipsliU6ImXrKAtZVLZ3miUFV05bKoS2ixKDZRFS9osS12ziGcb6X88psHPEMnD3+sO9s0N952w+3dl0NURCXRDgI+vlXByYKQJTIYLsf7gx8RmZKcvgHBQVWVKNAwDVNk4SI2OIroO4y3x8MQmcKikYsjhLbC51zHrxAc5+5vY2XjTfOWlwpEF/ZG1fHbpewotEcuvWgu6ZYdx72uQnDZgqGNFRvmmmSNbCZx2Y+pDxI1SuExpwgNeOp8PoZL9HzORn7SfkSU55xEZLnFzQmXkQ5bcbllHnsce1kjl0xLssEUZOE5OeMUwmKmFQVl+6amSlnCTPalnbnezHxf6/cXk924KpDWfyjyB44j6QRnw9mde/8CaUqrnJFlw3YRFWxxcE1gLIGlhtckPl7F9ZfN18429oMc/nTdm9vv3z51tEr9Bq9QSHaRXvoIzpCp4gii76jn+hXa9b60vra+rZAV1dKnxeoslo//gJT2CpK</latexit><latexit sha1_base64="4s/F4xsMBYlMi+mf8agPCZwO+oo=">AAAF6XicfZRLbxMxEIDd0kAJrxTEiUtELhyiardUbS5IVVEFp6pUfUl1FHm9k8SK12tsb1+WfwQ3xA1xg7/AH+HfsJusRHa9wiutRjPfvOyxI8mZNkHwZ2X13lrr/oP1h+1Hj588fdbZeH6m00xROKUpT9VFRDRwJuDUMMPhQiogScThPJq9L+znV6A0S8WJuZUwTMhEsDGjxOSqUecllmw0677r4rEi1B6OZs4eulGnF2wG89X1hbAUentosY5GG2u/cZzSLAFhKCdaX4aBNENLlGGUg2vjTIMkdEYmcJmLgiSgh3Zev+tWrCfh0I5TYUDQipsliU6ImXrKAtZVLZ3miUFV05bKoS2ixKDZRFS9osS12ziGcb6X88psHPEMnD3+sO9s0N952w+3dl0NURCXRDgI+vlXByYKQJTIYLsf7gx8RmZKcvgHBQVWVKNAwDVNk4SI2OIroO4y3x8MQmcKikYsjhLbC51zHrxAc5+5vY2XjTfOWlwpEF/ZG1fHbpewotEcuvWgu6ZYdx72uQnDZgqGNFRvmmmSNbCZx2Y+pDxI1SuExpwgNeOp8PoZL9HzORn7SfkSU55xEZLnFzQmXkQ5bcbllHnsce1kjl0xLssEUZOE5OeMUwmKmFQVl+6amSlnCTPalnbnezHxf6/cXk924KpDWfyjyB44j6QRnw9mde/8CaUqrnJFlw3YRFWxxcE1gLIGlhtckPl7F9ZfN18429oMc/nTdm9vv3z51tEr9Bq9QSHaRXvoIzpCp4gii76jn+hXa9b60vra+rZAV1dKnxeoslo//gJT2CpK</latexit><latexit sha1_base64="4s/F4xsMBYlMi+mf8agPCZwO+oo=">AAAF6XicfZRLbxMxEIDd0kAJrxTEiUtELhyiardUbS5IVVEFp6pUfUl1FHm9k8SK12tsb1+WfwQ3xA1xg7/AH+HfsJusRHa9wiutRjPfvOyxI8mZNkHwZ2X13lrr/oP1h+1Hj588fdbZeH6m00xROKUpT9VFRDRwJuDUMMPhQiogScThPJq9L+znV6A0S8WJuZUwTMhEsDGjxOSqUecllmw0677r4rEi1B6OZs4eulGnF2wG89X1hbAUentosY5GG2u/cZzSLAFhKCdaX4aBNENLlGGUg2vjTIMkdEYmcJmLgiSgh3Zev+tWrCfh0I5TYUDQipsliU6ImXrKAtZVLZ3miUFV05bKoS2ixKDZRFS9osS12ziGcb6X88psHPEMnD3+sO9s0N952w+3dl0NURCXRDgI+vlXByYKQJTIYLsf7gx8RmZKcvgHBQVWVKNAwDVNk4SI2OIroO4y3x8MQmcKikYsjhLbC51zHrxAc5+5vY2XjTfOWlwpEF/ZG1fHbpewotEcuvWgu6ZYdx72uQnDZgqGNFRvmmmSNbCZx2Y+pDxI1SuExpwgNeOp8PoZL9HzORn7SfkSU55xEZLnFzQmXkQ5bcbllHnsce1kjl0xLssEUZOE5OeMUwmKmFQVl+6amSlnCTPalnbnezHxf6/cXk924KpDWfyjyB44j6QRnw9mde/8CaUqrnJFlw3YRFWxxcE1gLIGlhtckPl7F9ZfN18429oMc/nTdm9vv3z51tEr9Bq9QSHaRXvoIzpCp4gii76jn+hXa9b60vra+rZAV1dKnxeoslo//gJT2CpK</latexit><latexit sha1_base64="/a7eQ6x7KnFFyD0d9IcX9lnNdw0=">AAAF6XicfZRPb9MwFMC9scIo/zYQJy4VvXCopmRM2y5I09AEp2lU6zZprirHeW2tOk6wnf2z/CG4IW6IG3wFvgjfhriNRBNHOFL09N7v/bOfHWWcKR0Ef1ZW76217j9Yf9h+9PjJ02cbm8/PVJpLCgOa8lReREQBZwIGmmkOF5kEkkQczqPZe2c/vwKpWCpO9W0Gw4RMBBszSnShGm28xBkbzTrvOngsCTXHo5k1x3a00Q22gvnq+EJYCl1UrpPR5tpvHKc0T0BoyolSl2GQ6aEhUjPKwbZxriAjdEYmcFmIgiSghmZev+1UrKfh0IxToUHQipshiUqInnpKB6uqlk6LxCCraUvl0LgoMSg2EVWvKLHtNo5hXOzlvDITRzwHa/ofDq0Jertve+H2nq0hEuKSCPeDXvHVgYkEECWyv9MLd/d9JstlxuEfFDjMVSNBwDVNk4SI2OAroPay2B8MQuUSXCMGR4nphtZaD16ghc/c3sbLxhtrDK4UiK/Mja1jt0uYa7SAbj3orinWnYd9bsKwnoImDdXrZprkDWzusbkPSQ+S9QqhMSdkivFUeP2Ml+j5nIz9pHyJKc/YheTFBY2JFzGbNuPZlHlsv3YyfevGZZkgcpKQ4pxxmoEkOpXu0l0zPeUsYVqZ0m59Lyb+71XY68mObHUo3T+KzJH1SBrx+WBW986fUCrjKue6bMAmsootDq4BzGpgucGOLN67sP66+cLZ9lZYyJ92ugeH5cu3jl6h1+gNCtEeOkAf0QkaIIoM+o5+ol+tWetL62vr2wJdXSl9XqDKav34CwIJKgo=</latexit>
Cluster Mean
Cluster Covariance
Fraction of pointsin each cluster
Idea: Use weights γnk = p(zn=k | xn) to compute estimates
⌃k =1Nk
X
n
�nk(xn �µk)(xn �µk)><latexit sha1_base64="m+VqJWgg9/bBz5V+ool4bizAanE=">AAAGWHicfZRLb9QwEICzpd22y2tbTsAlYi9FWlBSqrYXpKqoghMqpS9pvUSOM5u1NnGC4/Rl+cbP4o/AT+BX4GRTsYkjfIhGM9887JmMn0Y0E47zq7P0YHmlu7q23nv46PGTp/2NzfMsyTmBM5JECb/0cQYRZXAmqIjgMuWAYz+CC3/2obBfXAHPaMJOxW0K4xiHjE4owUKrvP4PicogIx76Y+m8dcozNASFriT6SsMYK29mv7fRhGMiXSU/ezNloyyPPcm0EOI4xlrUyi3tcqM8Zr+xUZx7s9em5hsSSWorrz+4z2ObglsJgwNrfo69jeWfKEhIHgMTJMJZNnKdVIwl5oKSCFQP5RmkmMxwCCMtMhxDNpblTZVds566YzlJmABGam4Sx1mMxdRQFnBW15KpTgy8nrZSjmURJYCMhqzu5ceq10MBTHTryspk4Ec5KHny8VBJZ7j7buhu76kGwiGoCHdft2foNIGQA7AK2d8Zurv7JpPmPI3gH+QUWFENBwbXJNE9ZIFEV0DUSL8PApblHIqLSOTHcuAqpQx4jmqf0t5Di8YbJatBuy+wHIUmdruAFRfV0K0B3bXFujOw720YElMQuKV60U7jvIXNDTY3IW5AvFkhtOaENKNRwoz7TBbock4mZtJogal6XISM9D4IsBExnbbj6ZQa7EmjMyeqGJdFAnO9HHSfUZICxyLhxU93TcU0ojEVmazsyvSi7P9e2t5MdqTqQ1l8fV8eKYMkflQOZv3tzAklPKhzxS1bsJDXsXnjWsC0AVYPXJB637nN7WYK59tvXS1/2RkcHFabb816ab2ytizX2rMOrE/WsXVmEetPp9953nmx8rtrdVe763N0qVP5PLNqp7v5F1y6S/g=</latexit><latexit sha1_base64="m+VqJWgg9/bBz5V+ool4bizAanE=">AAAGWHicfZRLb9QwEICzpd22y2tbTsAlYi9FWlBSqrYXpKqoghMqpS9pvUSOM5u1NnGC4/Rl+cbP4o/AT+BX4GRTsYkjfIhGM9887JmMn0Y0E47zq7P0YHmlu7q23nv46PGTp/2NzfMsyTmBM5JECb/0cQYRZXAmqIjgMuWAYz+CC3/2obBfXAHPaMJOxW0K4xiHjE4owUKrvP4PicogIx76Y+m8dcozNASFriT6SsMYK29mv7fRhGMiXSU/ezNloyyPPcm0EOI4xlrUyi3tcqM8Zr+xUZx7s9em5hsSSWorrz+4z2ObglsJgwNrfo69jeWfKEhIHgMTJMJZNnKdVIwl5oKSCFQP5RmkmMxwCCMtMhxDNpblTZVds566YzlJmABGam4Sx1mMxdRQFnBW15KpTgy8nrZSjmURJYCMhqzu5ceq10MBTHTryspk4Ec5KHny8VBJZ7j7buhu76kGwiGoCHdft2foNIGQA7AK2d8Zurv7JpPmPI3gH+QUWFENBwbXJNE9ZIFEV0DUSL8PApblHIqLSOTHcuAqpQx4jmqf0t5Di8YbJatBuy+wHIUmdruAFRfV0K0B3bXFujOw720YElMQuKV60U7jvIXNDTY3IW5AvFkhtOaENKNRwoz7TBbock4mZtJogal6XISM9D4IsBExnbbj6ZQa7EmjMyeqGJdFAnO9HHSfUZICxyLhxU93TcU0ojEVmazsyvSi7P9e2t5MdqTqQ1l8fV8eKYMkflQOZv3tzAklPKhzxS1bsJDXsXnjWsC0AVYPXJB637nN7WYK59tvXS1/2RkcHFabb816ab2ytizX2rMOrE/WsXVmEetPp9953nmx8rtrdVe763N0qVP5PLNqp7v5F1y6S/g=</latexit><latexit sha1_base64="m+VqJWgg9/bBz5V+ool4bizAanE=">AAAGWHicfZRLb9QwEICzpd22y2tbTsAlYi9FWlBSqrYXpKqoghMqpS9pvUSOM5u1NnGC4/Rl+cbP4o/AT+BX4GRTsYkjfIhGM9887JmMn0Y0E47zq7P0YHmlu7q23nv46PGTp/2NzfMsyTmBM5JECb/0cQYRZXAmqIjgMuWAYz+CC3/2obBfXAHPaMJOxW0K4xiHjE4owUKrvP4PicogIx76Y+m8dcozNASFriT6SsMYK29mv7fRhGMiXSU/ezNloyyPPcm0EOI4xlrUyi3tcqM8Zr+xUZx7s9em5hsSSWorrz+4z2ObglsJgwNrfo69jeWfKEhIHgMTJMJZNnKdVIwl5oKSCFQP5RmkmMxwCCMtMhxDNpblTZVds566YzlJmABGam4Sx1mMxdRQFnBW15KpTgy8nrZSjmURJYCMhqzu5ceq10MBTHTryspk4Ec5KHny8VBJZ7j7buhu76kGwiGoCHdft2foNIGQA7AK2d8Zurv7JpPmPI3gH+QUWFENBwbXJNE9ZIFEV0DUSL8PApblHIqLSOTHcuAqpQx4jmqf0t5Di8YbJatBuy+wHIUmdruAFRfV0K0B3bXFujOw720YElMQuKV60U7jvIXNDTY3IW5AvFkhtOaENKNRwoz7TBbock4mZtJogal6XISM9D4IsBExnbbj6ZQa7EmjMyeqGJdFAnO9HHSfUZICxyLhxU93TcU0ojEVmazsyvSi7P9e2t5MdqTqQ1l8fV8eKYMkflQOZv3tzAklPKhzxS1bsJDXsXnjWsC0AVYPXJB637nN7WYK59tvXS1/2RkcHFabb816ab2ytizX2rMOrE/WsXVmEetPp9953nmx8rtrdVe763N0qVP5PLNqp7v5F1y6S/g=</latexit><latexit sha1_base64="Rmhbw9pD1UBLs2lCJVC602qBVzs=">AAAGWHicfZRLb9QwEICzpd22y6PbcgIuEXsp0lIlpWp7QaqKKjihUvqS1kvkOLPZaBMn2E5flm/8LP4I/AR+BU6aik0c4UM0mvnmYc9k/CyOuHCcX52FR4tL3eWV1d7jJ0+frfXXN855mjMCZySNU3bpYw5xROFMRCKGy4wBTvwYLvzZh8J+cQWMRyk9FbcZjBMc0mgSESy0yuv/kKgMMmKhP5bOllOeoSEodCXR1yhMsPJm9nsbTRgm0lXyszdTNuJ54kmqhRAnCdaiVm5qlxvlUfutjZLcm70xNd+QSDNbef3BQx7bFNxKGFjVOfbWF3+iICV5AlSQGHM+cp1MjCVmIiIxqB7KOWSYzHAIIy1SnAAfy/Kmyq5ZT92xnKRUACU1N4kTnmAxNZQFzOtaMtWJgdXTVsqxLKIEwKOQ1r38RPV6KICJbl1ZmQz8OAclTz4eKukMd98N3e091UAYBBXh7uv2DJ0mEDIAWiH7O0N3d99kspxlMfyDnAIrqmFA4Zqkuoc0kOgKiBrp90FAec6guIhEfiIHrlLKgO9R7VPae2jeeKNkNWgPBZaj0MRu57Diohq6NaC7tlh3Bva9DUNiCgK3VC/aaZy3sLnB5ibEDIg1K4TWnJDxKE6pcZ/JHF3OycRMGs8xVY+LkLHeBwE2ImbTdjybRgZ70ujMiSrGZZ7ATC8H3WeUZsCwSFnx011HYhpHSSS4rOzK9Iro/720vZnsSNWHsvj6vjxSBkn8uBzM+tuZE0pYUOeKW7ZgIatj941rAbMGWD1wQep95za3mymcb2+5Wv6yMzg4rDbfivXKem1tWq61Zx1Yn6xj68wi1p9Ov/Oi83Lpd9fqLndX79GFTuXz3Kqd7sZfCutLuA==</latexit>
Parameter Updates
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 znk xn
⌃k =1Nk
PNn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 znk xn
⌃k =1Nk
PNn=1 znk (xn �µk)(xn �µk)>
Assignment Update
Idea: Replace hard assignments with soft assignments
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
“Hard EM” with Gaussians
Nk :=PN
n=1 �nk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 �nk xn
⌃k =1Nk
PNn=1 �nk (xn �µk)(xn �µk)>
Nk :=PN
n=1 �nk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 �nk xn
⌃k =1Nk
PNn=1 �nk (xn �µk)(xn �µk)>
Nk :=PN
n=1 �nk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
PNn=1 �nk xn
⌃k =1Nk
PNn=1 �nk (xn �µk)(xn �µk)>
Parameter Updates
Gaussian Mixture Models
znk := I[zn = k] Nk :=PN
n=1 znk
⇡ = (N1/N , . . . , NK/N)
µk =1Nk
Pn=1 znk xn
⌃k =1Nk
Pn=1 znk (xn �µk)(xn �µk)>
Soft Assignment Update
Idea: Replace hard assignments with soft assignments
EM for Gaussian Mixtures
Credit: Andrew Moore
EM for Gaussian Mixtures
Credit: Andrew Moore
EM for Gaussian Mixtures
Credit: Andrew Moore
EM for Gaussian Mixtures
Credit: Andrew Moore
EM for Gaussian Mixtures
Credit: Andrew Moore
EM for Gaussian Mixtures
Credit: Andrew Moore
EM for Gaussian Mixtures
Credit: Andrew Moore
Consider Naive Bayes
evaluate the classifier. One good evaluation metric is accuracy, which is defined as
# correctly predicted labels in the test set# of instances in the test set
(6)
2.2 Naive Bayes model
The naive Bayes model defines a joint distribution over words and classes–that is, the prob-ability that a sequence of words belongs to some class c.
p(c, w1:N |⇡, ✓) = p(c|⇡)NY
n=1
p(wn|✓c) (7)
⇡ is the probability distribution over the classes (in this case, the probability of seeing spame-mails and of seeing ham e-mails). It’s basically a discrete probability distribution overthe classes that sums to 1. ✓c is the class conditional probability distribution–that is, givena certain class, the probability distribution of the words in your vocabulary. Each class hasa probability for each word in the vocabulary (in this case, there is a set of probabilities forthe spam class and one for the ham class).Given a new test example, we can classify it using the model by calculating the conditionalprobability of the classes given the words in the e-mail p(c | wn) and see which class ismost likely. There is an implicit independence assumption behind the model. Since we’remultiplying the p(wn | ✓c) terms together, it is assumed that the words in a document areconditionally independent given the class.
2.3 Class prediction with naive Bayes
We classify using the posterior distribution of the classes given the words, which is propor-tional to the joint distribution since P (X|Y ) = P(X, Y)/P(Y) / P(X, Y). The posteriordistribution is
p(c|w1:N , ⇡, ✓) / p(c|⇡)NY
n=1
p(wn|✓c) (8)
Note that we don’t need a normalizing constant because we only care about which proba-bility is bigger. The classifier assigns the e-mail to the class which has highest probability.
2.4 Fitting a naive Bayes model with maximum likelihood
We find the parameters in the posterior distribution used in class prediction by learningon the labeled training data (in this case, the ham and spam e-mails). We use the maxi-mum likelihood method method in finding parameters that maximize the likelihood of theobserved data set. Given data {wd,1:N , cd}D
d=1, the likelihood under a certain model withparameters (✓1:C , ⇡) is
p(D|✓1:C , ⇡) =DY
d=1
p(cd|⇡)NY
n=1
p(wn|✓cd)!
3
evaluate the classifier. One good evaluation metric is accuracy, which is defined as
# correctly predicted labels in the test set# of instances in the test set
(6)
2.2 Naive Bayes model
The naive Bayes model defines a joint distribution over words and classes–that is, the prob-ability that a sequence of words belongs to some class c.
p(c, w1:N |⇡, ✓) = p(c|⇡)NY
n=1
p(wn|✓c) (7)
⇡ is the probability distribution over the classes (in this case, the probability of seeing spame-mails and of seeing ham e-mails). It’s basically a discrete probability distribution overthe classes that sums to 1. ✓c is the class conditional probability distribution–that is, givena certain class, the probability distribution of the words in your vocabulary. Each class hasa probability for each word in the vocabulary (in this case, there is a set of probabilities forthe spam class and one for the ham class).Given a new test example, we can classify it using the model by calculating the conditionalprobability of the classes given the words in the e-mail p(c | wn) and see which class ismost likely. There is an implicit independence assumption behind the model. Since we’remultiplying the p(wn | ✓c) terms together, it is assumed that the words in a document areconditionally independent given the class.
2.3 Class prediction with naive Bayes
We classify using the posterior distribution of the classes given the words, which is propor-tional to the joint distribution since P (X|Y ) = P(X, Y)/P(Y) / P(X, Y). The posteriordistribution is
p(c|w1:N , ⇡, ✓) / p(c|⇡)NY
n=1
p(wn|✓c) (8)
Note that we don’t need a normalizing constant because we only care about which proba-bility is bigger. The classifier assigns the e-mail to the class which has highest probability.
2.4 Fitting a naive Bayes model with maximum likelihood
We find the parameters in the posterior distribution used in class prediction by learningon the labeled training data (in this case, the ham and spam e-mails). We use the maxi-mum likelihood method method in finding parameters that maximize the likelihood of theobserved data set. Given data {wd,1:N , cd}D
d=1, the likelihood under a certain model withparameters (✓1:C , ⇡) is
p(D|✓1:C , ⇡) =DY
d=1
p(cd|⇡)NY
n=1
p(wn|✓cd)!
3
The model
In-class exercise: How would we use EM here?
Initialize parameters randomlywhile not converged
1. E-Step:Set the latent variables to the the values that maximizes likelihood, treating parameters as observed
2. M-Step:Set the parameters to the values that maximizes likelihood, treating latent variables as observed
In-class exercise
Slide credit: Matt Gormley and Eric Xing (CMU)
Let’s review (on board)
Summing up• Mixture models can be used to perform probabilistic clustering
• General idea: Assume instances are generated from distinct components. Each component has its own model parameters.
• Fitting: More difficult here than in standard supervised learning because we do not observe z. (One) Solution: Expectation-Maximization.
Summing up• Mixture models can be used to perform probabilistic clustering
• General idea: Assume instances are generated from distinct components. Each component has its own model parameters.
• Fitting: More difficult here than in standard supervised learning because we do not observe z. (One) Solution: Expectation-Maximization.
Summing up• Mixture models can be used to perform probabilistic clustering
• General idea: Assume instances are generated from distinct components. Each component has its own model parameters.
• Fitting: More difficult here than in standard supervised learning because we do not observe z. (One) Solution: Expectation-Maximization.