large-scale long-tailed recognition in an open worldstellayu/publication/doc/2019... ·...
TRANSCRIPT
![Page 1: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/1.jpg)
Large-Scale Long-Tailed Recognition in an Open World
Ziwei Liu* Zhongqi Miao* Xiaohang Zhan Jiayun Wang Boqing Gong Stella X. Yu
The Chinese University of Hong Kong UC Berkeley / ICSI
![Page 2: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/2.jpg)
Train
Test
Cat
Fox
Panda
CatFoxPanda
CatFoxPanda
![Page 3: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/3.jpg)
Train
Test
Cat
Fox
Panda
CatFoxPanda
CatFoxPanda
?(open class)
(many-shotclass)
(medium-shotclass)
(few-shotclass)
![Page 4: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/4.jpg)
Faces [Zhang et al. 2017] Places [Wang et al. 2017]
Actions [Zhang et al. 2019]Species [Van Horn et al. 2019]
![Page 5: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/5.jpg)
?Open Long-Tailed Recognition
Open World
Head Classes Tail Classes Open Classes
![Page 6: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/6.jpg)
?Open Long-Tailed Recognition
Open World
Head Classes Tail Classes Open Classes
Knowledge Transfer
![Page 7: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/7.jpg)
?Open Long-Tailed Recognition
Open World
Head Classes Tail Classes Open Classes
Sensitivity to NoveltyKnowledge Transfer
![Page 8: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/8.jpg)
?Open Long-Tailed Recognition
Open World
Head Classes Tail Classes Open Classes
Avoid Forgetting
Sensitivity to NoveltyKnowledge Transfer
![Page 9: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/9.jpg)
?Open Long-Tailed Recognition
Open World
Imbalanced Classification
Few-shot Learning
Head Classes Tail Classes Open Classes
Open Set Recognition
![Page 10: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/10.jpg)
Imbalanced Classification(metric learning, re-sampling, re-weighting)
test train testtrain
Few-Shot Learning(meta learning, classifier dynamics)
Open Set Recognition(distribution rectification, out-of-distribution detection)
Open Long-Tailed Recognition(dynamic meta-embedding)
testtrain train test
Sensitivity to Novelty Avoid Forgetting
Knowledge Transfer Knowledge Transfer Sensitivity to Novelty
Avoid Forgetting
![Page 11: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/11.jpg)
train test
Open Long-Tailed Recognition(dynamic meta-embedding)
Knowledge Transfer Sensitivity to Novelty
Avoid Forgetting
![Page 12: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/12.jpg)
FLY
visual memory
top-down attention
enhanced embedding
FLY
familiarity
direct embedding
bottom-up attention
![Page 13: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/13.jpg)
Head Classes Tail Classes
bottom-up attention
top-down attention
familiarity
visual memory
Open Classes
![Page 14: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/14.jpg)
Head Classes Tail Classes
bottom-up attention
top-down attention
familiarity
visual memory
Open Classes
Avoid Forgetting Knowledge Transfer
Sensitivity to Novelty
![Page 15: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/15.jpg)
Head Classes Tail Classes
bottom-up attention
Open Classes
top-down attention
familiarity
visual memory
originalfeature map
attentivefeature map
TenchHand
Fish
![Page 16: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/16.jpg)
Head Classes Tail Classes
bottom-up attention
top-down attention
familiarity
visual memory
Open Classes
direct embedding
enhanced embedding
associative memory
feature selection
![Page 17: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/17.jpg)
Head Classes Tail Classes
familiarity
visual memory
Open Classes
bottom-up attention
embedding
rescaled embedding
Tail Class ‘African Grey’
Tail Class ‘Buckeye’
top-down attention
Open Sample
![Page 18: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/18.jpg)
ImageNet-LT Benchmark
Absolute Performance Gain: ~20%
Places-LT Benchmark
MS1M-LT Benchmark
Absolute Performance Gain: ~10%
Absolute Performance Gain: ~2%
![Page 19: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/19.jpg)
Methods ImageNet-LT Places-LT MS1M-LTPlain Model 0.295 0.366 0.738Sample Re-weighting (Focal Loss) 0.371 0.453 -Metric Learning (Range Loss) 0.373 0.457 0.722Open Set Recognition (OpenMax) 0.368 0.458 -Few-shot Learning (FSLwF) 0.347 0.375 -
Dynamic Meta-Embedding 0.474 0.464 0.745
Overall F1 Score on ImageNet-LT, Places-LT and MS1M-LT Benchmarks
![Page 20: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/20.jpg)
Methods ImageNet-LT Places-LT MS1M-LTPlain Model 0.295 0.366 0.738Sample Re-weighting (Focal Loss) 0.371 0.453 -Metric Learning (Range Loss) 0.373 0.457 0.722Open Set Recognition (OpenMax) 0.368 0.458 -Few-shot Learning (FSLwF) 0.347 0.375 -
Dynamic Meta-Embedding 0.474 0.464 0.745
Overall F1 Score on ImageNet-LT, Places-LT and MS1M-LT Benchmarks
![Page 21: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/21.jpg)
Methods ImageNet-LT Places-LT MS1M-LTPlain Model 0.295 0.366 0.738Sample Re-weighting (Focal Loss) 0.371 0.453 -Metric Learning (Range Loss) 0.373 0.457 0.722Open Set Recognition (OpenMax) 0.368 0.458 -Few-shot Learning (FSLwF) 0.347 0.375 -
Dynamic Meta-Embedding 0.474 0.464 0.745
Overall F1 Score on ImageNet-LT, Places-LT and MS1M-LT Benchmarks
![Page 22: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/22.jpg)
Methods ImageNet-LT Places-LT MS1M-LTPlain Model 0.295 0.366 0.738Sample Re-weighting (Focal Loss) 0.371 0.453 -Metric Learning (Range Loss) 0.373 0.457 0.722Open Set Recognition (OpenMax) 0.368 0.458 -Few-shot Learning (FSLwF) 0.347 0.375 -
Dynamic Meta-Embedding 0.474 0.464 0.745
Overall F1 Score on ImageNet-LT, Places-LT and MS1M-LT Benchmarks
![Page 23: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/23.jpg)
Methods ImageNet-LT Places-LT MS1M-LTPlain Model 0.295 0.366 0.738Sample Re-weighting (Focal Loss) 0.371 0.453 -Metric Learning (Range Loss) 0.373 0.457 0.722Open Set Recognition (OpenMax) 0.368 0.458 -Few-shot Learning (FSLwF) 0.347 0.375 -
Dynamic Meta-Embedding 0.474 0.464 0.745
Overall F1 Score on ImageNet-LT, Places-LT and MS1M-LT Benchmarks
![Page 24: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/24.jpg)
Few shot
![Page 25: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/25.jpg)
New Task Open Long-Tailed Recognition (OLTR)
Train Test
New Approach Dynamic Meta-Embedding
New Benchmarks ImageNet-LT Places-LT MS1M-LT
![Page 26: Large-Scale Long-Tailed Recognition in an Open Worldstellayu/publication/doc/2019... · 2019-06-15 · Cat Fox Panda Cat Fox Panda Cat Fox Panda. Train Test Cat Fox Panda Cat Fox](https://reader034.vdocuments.site/reader034/viewer/2022050303/5f6c63d9aac48e460908877b/html5/thumbnails/26.jpg)
Thanks! Code, models and benchmarks are available at
Project Page: https://liuziwei7.github.io/projects/LongTail.html
Poster #170