a unifying framework for multi-task learning
TRANSCRIPT
![Page 1: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/1.jpg)
A Unifying Framework for
Multi-task LearningCarlo Ciliberto
![Page 2: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/2.jpg)
Sharing Information
![Page 3: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/3.jpg)
Sharing Information
![Page 4: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/4.jpg)
Sharing Information
![Page 5: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/5.jpg)
Without Sharing Information
![Page 6: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/6.jpg)
Sharing Information
![Page 7: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/7.jpg)
Multi-task Learning: AssumptionLeveraging on the tasks relations/structure
reduces the complexity of the problem
![Page 8: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/8.jpg)
![Page 9: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/9.jpg)
![Page 10: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/10.jpg)
Impose known structures[Evgeniou et al. 2005, Fergus et al. 2010, Kadri et al. 2010, Minh et al 2013, Jayaraman et
al., 2014 and many others]
Parametrize and Learn
the structure [Argyriou et al. 2008, Jacob et al. 2009, Zhang et al, 2010 Dinuzzo et al. 2011, Zhong
2012, and many other]
![Page 11: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/11.jpg)
Learning To Learn
Output Representation Learning
![Page 12: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/12.jpg)
To Abstract,
Understand & Organize
![Page 13: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/13.jpg)
Can we design a unifying
(convex) framework for
learning Multiple Tasks and
their structure?
![Page 14: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/14.jpg)
Can we design a unifying
(convex) framework for
learning Multiple Tasks and
their structure?
Yes!
![Page 15: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/15.jpg)
Can we provide a general
meta-strategy for
optimization…
…with convergence
guarantees?
![Page 16: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/16.jpg)
Can we provide a general
meta-strategy for
optimization…
…with convergence
guarantees?
Yes!
![Page 17: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/17.jpg)
Can we derive new models
of tasks structures from
such a framework?
![Page 18: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/18.jpg)
Can we derive new models
of tasks structures from
such a framework?
Yes!
![Page 19: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/19.jpg)
Can we derive new models of tasks
structures from such a framework?
Can we provide a general meta-strategy for
optimization, with convergence guarantees?
Can we design a unifying (convex)
framework for learning multiple-tasks and
their structure?
[Ciliberto et al. - ICML 2015]
[Ciliberto et al. - CVPR 2015]
![Page 20: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/20.jpg)
RKHSfor Vector-Valued functions
![Page 21: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/21.jpg)
Examples
~ Graph Laplacian[Evgeniou et al. 2005, Argyriou et al. 2013]
Low dimensional subspace sharing[Argyriou et al. 2008, Zhang et al. 2010]
Cluster Multi-task learning[Jacob et al. 2009, kwok et al. 2012]
Sparse Kernel Multi-task Learning[Ciliberto et al. 2015]
![Page 22: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/22.jpg)
Can we derive new models of tasks
structures from such a framework?
Can we provide a general meta-strategy for
optimization, with convergence guarantees?
Can we design a unifying (convex)
framework for learning multiple-tasks and
their structure?
[Ciliberto et al. - ICML 2015]
[Ciliberto et al. - CVPR 2015]
![Page 23: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/23.jpg)
Are we done?
![Page 24: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/24.jpg)
![Page 25: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/25.jpg)
![Page 26: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/26.jpg)
![Page 27: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/27.jpg)
Can we find a parametrization
for all Operator-valued
Kernels?
Can we still learn them?
![Page 28: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/28.jpg)
Can we find a parametrization
for all Operator-valued
Kernels?
Can we still learn them?
Spoiler alert: Yes![Ciliberto et al. - In Preparation]
![Page 29: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/29.jpg)
Take home messages
![Page 30: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/30.jpg)
Take home messages
Multi Task LearningIf tasks are related, solving them jointly can be much more favorable!
MTL
![Page 31: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/31.jpg)
Take home messages
Multi Task LearningIf tasks are related, solving them jointly can be much more favorable!
RKHS for vector-valued functionsAre the way to go! you can:
MTL
![Page 32: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/32.jpg)
Take home messages
Multi Task LearningIf tasks are related, solving them jointly can be much more favorable!
RKHS for vector-valued functionsAre the way to go! you can:
Impose prior knowledge on the structureBy designing a suitable structure matrix A
Learn the relations!Imposing a structure penalty F(A) on the problem
MTL
![Page 33: A Unifying Framework for Multi-task Learning](https://reader031.vdocuments.site/reader031/viewer/2022012508/6184c1eb4c6ab7121b5847df/html5/thumbnails/33.jpg)
Take home messages
Multi Task LearningIf tasks are related, solving them jointly can be much more favorable!
RKHS for vector-valued functionsAre the way to go! you can:
Impose prior knowledge on the structureBy designing a suitable structure matrix A
Learn the relations!Imposing a structure penalty F(A) on the problem
Future Work
More complex intra-task relationsImpose or learn more complex input-output relations
MTL