![Page 1: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/1.jpg)
Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang
MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
IEEE International Conference on Multimedia and Expo6-10 July 2020 || London, United Kingdom || Virtual
Beyond without Forgetting: Multi-Task Learning for Classification with Disjoint Datasets
![Page 2: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/2.jpg)
Catalog
1 Multi-task Learning from Disjoint Datasets
2 Learning from Pseudo Labels
3 Selective Augmentation Method
4 Experiments
![Page 3: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/3.jpg)
Catalog
1
2
3
4
Multi-task Learning from Disjoint Datasets
Learning from Pseudo Labels
Selective Augmentation Method
Experiments
![Page 4: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/4.jpg)
Multi-task Learning
![Page 5: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/5.jpg)
Forgetting Effects
Supervised
Unsupervised
![Page 6: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/6.jpg)
Multi-task Learning without ForgettingSoft label vector
Preserving information
![Page 7: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/7.jpg)
Catalog
2
1
3
4
Multi-task Learning from Disjoint Datasets
Learning from Pseudo Labels
Selective Augmentation Method
Experiments
![Page 8: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/8.jpg)
Multi-task Learning with Augmentation DataOne-hot vector
Augmentation information
![Page 9: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/9.jpg)
Noisy Pseudo Labels
Images from AFLW with the pseudo emotion label “angry”
![Page 10: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/10.jpg)
Distribution Mismatch
Images from AFLW with pose label, captured in good light condition
Images from SFEW with emotion label, captured in poor light condition
![Page 11: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/11.jpg)
Catalog
1
4
3
2
Multi-task Learning from Disjoint Datasets
Learning from Pseudo Labels
Selective Augmentation Method
Experiments
![Page 12: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/12.jpg)
Selecting Data with Confident Pseudo Labels
0.09 0.01 0.03 0.04 0.02 0.81
0 0 0 0 0 1
Soft label vector
Pseudo label vector
Max indexConfidence weight 𝑤𝑤𝑖𝑖𝑐𝑐 = 0.81
![Page 13: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/13.jpg)
Selecting Data with Confident Pseudo Labels
Low local density
High local density
= �1,0, otherwise
![Page 14: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/14.jpg)
Selecting Data with Closer Data Distribution
![Page 15: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/15.jpg)
Label Vector Interpolation
selecting data with confident pseudo labels
selecting data with closer data distribution
interpolation
![Page 16: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/16.jpg)
Catalog
1
2
4
3
Multi-task Learning from Disjoint Datasets
Learning from Pseudo Labels
Selective Augmentation Method
Experiments
![Page 17: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/17.jpg)
Comparison with baselines
![Page 18: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/18.jpg)
Ablation Study
![Page 19: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/19.jpg)
Visual Results
Hight weighs,correct thepseudo facialexpressionlabels
Low weighs,incorrect thepseudo facialexpressionlabels
![Page 20: IEEE International Conference on Multimedia and Expobcmi.sjtu.edu.cn/home/niuli/paper/SAMTL_slides.pdf · Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang. MoE Key Lab of Artificial Intelligence,](https://reader033.vdocuments.site/reader033/viewer/2022052008/601d510de8f59d4cca6d5bae/html5/thumbnails/20.jpg)
IEEE International Conference on Multimedia and Expo6-10 July 2020 || London, United Kingdom || Virtual
Thanks for watching!
Yan Hong, Li Niu, Jianfu Zhang, Liqing Zhang
MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
Beyond without Forgetting: Multi-Task Learning for Classification with Disjoint Datasets