SDP-MARCH-Talk

K-means and Gaussian Mixture Model

王养浩2013年 11月 20日

Outline

• K-means• Gaussian Mixture Model• Expectation Maximum

K-means

• Gather data points to a few cohesive ‘Clusters’

• Unsupervised Learning

K-means

K-means

K-means

• Easy• Fast

• Euclidean distance?• K needs input？• Convergence？

Determination of K

• Rule of Thumb：• Elbow Method• Cross Validation

2/nk

K-means Convergence

• x(i) data points • μc(i) cluster centroids

• Coordinate descent

Coordinate Descent

K-means Convergence

• Local minimum– The optimization object is non-convex

Gaussian Mixture Model

• Mixture of Gaussian distribution

Gaussian Mixture Model

• Log likelihood

• Maximum likelihood – Expectation Maximum

Expectation Maximum

Expectation Maximum

• Jenson inquality

Expectation Maximum

• Training set {}• Hidden variables {}• Parameter θ

Expectation Maximum

• Construct lower bound

Expectation Maximum

• Maximum lower bound– Coordinate Ascent on J

• Repeat until convergence –Maximum given fixed θ–Maximum θ given fixed

Expectation Maximum

• Repeat until convergence

Generalized Expectation Maximum

• Difficulty in M-step

Summary

• K-means– Coordinate descent

• Gaussian Mixture Model– Expectation Maximum

• Expectation Maximum–MLE for models with latent variables– Generalized EM

• Thanks!