regularization for - machine learningmachinelearning.math.rs/jovanovic-multitask.pdf ·...
TRANSCRIPT
![Page 1: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/1.jpg)
Regularization for
Multi-task learning
Miloš Jovanović
some slides taken from:
Zhou,Chen,Ye (2012) Multi-Task Learning:
Theory, Algorithms, and Applications
12th SIAM SDM, 2012
![Page 2: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/2.jpg)
Learning objective
• Minimizing loss function:
– squared error:
– logistic loss:
• Useful properties:
– convexity
– differentiability
– smoothness
![Page 3: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/3.jpg)
Regularization
• Fighting ill-posed problems:
– non-unique solutions
– non-smoothness
• “Penalty”, Lagrangian dual
• In learning:
– Fighting sample variance / overfitting
=> limiting capacity of the model
![Page 4: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/4.jpg)
Null Regularization
• Modified objective:
• “Resistance” for parameters to take large values (Shrinking) – Linear regression
– Logistic regression
• Prior towards the null hypothesis: “no link between input and output” => statistical (scientific) caution (unbiased)
![Page 5: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/5.jpg)
Prior Regularization
• Prior for parameter values:
• Prior belief:
– previous regression parameters!
– prior assumptions
• Penalty for breaking our prior (Bayesian,
Scientific)
– Data vs Knowledge
![Page 6: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/6.jpg)
Multi-task problem
• Global model
• Local model
• Local model + null-regularization
• Best regularization? – more data!
• Regularization for sharing data – penalty for being
different
![Page 7: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/7.jpg)
Zhou,Chen,Ye (2012)
![Page 8: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/8.jpg)
MULTI-TASK MODELS
![Page 9: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/9.jpg)
Zhou,Chen,Ye (2012)
![Page 10: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/10.jpg)
Zhou,Chen,Ye (2012)
![Page 11: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/11.jpg)
Zhou,Chen,Ye (2012)
![Page 12: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/12.jpg)
Zhou,Chen,Ye (2012)
![Page 13: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/13.jpg)
Zhou,Chen,Ye (2012)
![Page 14: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/14.jpg)
Zhou,Chen,Ye (2012)
![Page 15: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/15.jpg)
Zhou,Chen,Ye (2012)
![Page 16: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/16.jpg)
Zhou,Chen,Ye (2012)
![Page 17: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/17.jpg)
Zhou,Chen,Ye (2012)
![Page 18: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/18.jpg)
Zhou,Chen,Ye (2012)
![Page 19: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/19.jpg)
Zhou,Chen,Ye (2012)
![Page 20: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/20.jpg)
Zhou,Chen,Ye (2012)
![Page 21: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/21.jpg)
Zhou,Chen,Ye (2012)
![Page 22: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/22.jpg)
Zhou,Chen,Ye (2012)
![Page 23: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/23.jpg)
Zhou,Chen,Ye (2012)
![Page 24: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/24.jpg)
Zhou,Chen,Ye (2012)
![Page 25: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/25.jpg)
Zhou,Chen,Ye (2012)
![Page 26: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/26.jpg)
Zhou,Chen,Ye (2012)
![Page 27: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/27.jpg)
Zhou,Chen,Ye (2012)
![Page 28: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/28.jpg)
Other Multi-task
• NN hidden layers
• Gaussian processes shared kernel
parameters
![Page 29: Regularization for - Machine Learningmachinelearning.math.rs/Jovanovic-MultiTask.pdf · Regularization for ... Zhou,Chen,Ye (2012) Multi-Task Learning: Theory, Algorithms, and Applications](https://reader034.vdocuments.site/reader034/viewer/2022050313/5f75cb58bfbc4b63340bc63c/html5/thumbnails/29.jpg)
Q&A
Thank you!