ece 5984: introduction to machine learnings15ece5984/slides/l8_regression.pptx.pdf · ece 5984:...
TRANSCRIPT
ECE 5984: Introduction to Machine Learning
Dhruv Batra Virginia Tech
Topics: – Regression
Readings: Barber 17.1, 17.2
Administrativia • HW1
– Due on Sun 02/15, 11:55pm – http://inclass.kaggle.com/c/VT-ECE-Machine-Learning-HW1
• Project Proposal – Due: Tue 02/24, 11:55 pm – <=2pages, NIPS format
• HW2 – Out today – Due on Friday 03/06, 11:55pm – Please please please please please start early – Implement linear regression, Naïve Bayes, Logistic
Regression (C) Dhruv Batra 2
Recap of last time
(C) Dhruv Batra 3
Learning a Gaussian • Collect a bunch of data
– Hopefully, i.i.d. samples – e.g., exam scores
• Learn parameters – Mean – Variance
(C) Dhruv Batra 4
MLE for Gaussian • Prob. of i.i.d. samples D={x1,…,xN}:
• Log-likelihood of data:
Slide Credit: Carlos Guestrin (C) Dhruv Batra 5
Your second learning algorithm: MLE for mean of a Gaussian
• What’s MLE for mean?
Slide Credit: Carlos Guestrin (C) Dhruv Batra 6
Learning Gaussian parameters
• MLE:
(C) Dhruv Batra 7
• Conjugate priors – Mean: Gaussian prior – Variance: Inverse Gamma or Wishart Distribution
• Prior for mean:
Slide Credit: Carlos Guestrin (C) Dhruv Batra 8
Bayesian learning of Gaussian parameters
MAP for mean of Gaussian
Slide Credit: Carlos Guestrin (C) Dhruv Batra 9
Plan for Today • Regression
– Linear Regression – Connections with Gaussians
(C) Dhruv Batra 10
New Topic: Regression
(C) Dhruv Batra 11
1-NN for Regression • Often bumpy (overfits)
(C) Dhruv Batra 12 Figure Credit: Andrew Moore
(C) Dhruv Batra 13 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 14 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 15 Slide Credit: Greg Shakhnarovich
Linear Regression • Demo
– http://hspm.sph.sc.edu/courses/J716/demos/LeastSquares/LeastSquaresDemo.html
(C) Dhruv Batra 16
(C) Dhruv Batra 17 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 18 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 19 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 20 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 21 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 22 Slide Credit: Greg Shakhnarovich
(C) Dhruv Batra 23 Slide Credit: Greg Shakhnarovich
• Why sum squared error??? • Gaussians, Watson, Gaussians…
But, why?
24 (C) Dhruv Batra
(C) Dhruv Batra 25 Slide Credit: Greg Shakhnarovich
MLE Under Gaussian Model • On board
(C) Dhruv Batra 26
Is OLS Robust? • Demo
– http://www.calpoly.edu/~srein/StatDemo/All.html
• Bad things happen when the data does not come from your model!
• How do we fix this?
(C) Dhruv Batra 27
Robust Linear Regression • y ~ Lap(w’x, b)
• On board
(C) Dhruv Batra 28
0 0.2 0.4 0.6 0.8 1−6
−5
−4
−3
−2
−1
0
1
2
3
4Linear data with noise and outliers
least squareslaplace
−3 −2 −1 0 1 2 3−0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
L2
L1
huber