mmfruit - openimage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...adj-soft...
TRANSCRIPT
![Page 1: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/1.jpg)
MMfruit - OpenImage 2019 1st solutionYu Liu, Guanglu Song, Yuhang Zang, Yan Gao, Junjie Yan, Chen Change Loy, Xiaogang Wang
Multimedia Laboratory, The Chinese University of Hong KongMultimedia Laboratory, Nanyang Technological UniversitySensetime X-Lab
![Page 2: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/2.jpg)
Team members
MMfruit Team
Yu Liu Guanglu Song Yuhang Zang Yan Gao
Chen Change LoyJunjie Yan Xiaogang Wang
![Page 3: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/3.jpg)
Results Breakdown
● 39 models in total (including exps):● Data distribution:
○ 26 trained by all classes○ 3 expert models (low AP)○ 2 models from COCO○ 8 models from O365
● Framework:○ 27 from pytorch○ 10 from tensorflow
● 32~512 accelerators for each model● 2 images on each accelerator
![Page 4: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/4.jpg)
Solutions
● Multiple Large Backbones● Gradient Decoupling● Class Sampling & Full Batch● Augmentation (Segmentation Label)● Truncated Loss● Multiscale Testing● Adj-soft NMS● Expert Model● Weakly & Fully Supervised Pipeline● Auto Ensemble
![Page 5: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/5.jpg)
Multiple Large Backbones
Model Zoo
ResNet family
ResNet50 ResNet152
ResNext family
ResNext101 ResNext152
DCN-ResNext101 DCN-ResNext152SEResNet family
SEResNext101
DCN-SEResNet154
DCN-SEResNext101
NAS family
NASNet NAS-FPN
EfficientNet family
EfficientNetB7 WideEfficientNetB7
Expert Model family
SEResNet154
![Page 6: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/6.jpg)
Gradient Decoupling
Decoupling Backbone (Naive implementation)
Decoupling Head
Backbone(to stride 8)
Branch A
Branch B
classificationregression
classificationregression
Lower loss weight
Backbone
RPN
ROIPooling classificationregression
ROIPooling
ROIPooling
regression
classification
CML
CML
![Page 7: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/7.jpg)
Gradient DecouplingLearn the offset for classification and regression separately.
CML for classification: for regression:
Exp on OpenImage2019Model DCP Val Public LB
ResNet50 64.64 49.79
ResNet50 √ 68.18 52.55
DCN-ResNext101 68.70 55.05
DCN-ResNext101 √ 71.71 58.59
DCN-SENet154 71.13 57.77
DCN-SENet154 √ 72.19 60.5
Exp on COCO 2017 (FPN)
Model A1 A2 A3 CML DCP mAP(IOU=0.50:0.95)
ResNet50 36.1
ResNet50 √ 37.3
ResNet50 √ 38.0
ResNet50 √ 38.5
ResNet50 √ 39.7
ResNet50 √ √ 40.8
A1: separate classification and regressionA2: Deformable ROIPooling for classification and ROIAlignPooling for regression.A3: Deformable ROIPooling for classification and Deformable ROIPooling for regression.
![Page 8: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/8.jpg)
Class Sampling & Full Batch
Class-aware sampling & full batch:
500 classes
Class 1
Class 2
Class 3
Class 4
……
……
Class 499
Class 500
Sample image for each class
Training model
Batch
![Page 9: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/9.jpg)
Augmentation (Segmentation Label)
Elaborate Augmentation
Select an image and a bounding box
Random
rotation
Sample a specific
scale
Crop
Copy-and-Paste Augmentation
Augmentation
![Page 10: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/10.jpg)
Truncated Loss
Missing (red dashed box):human eye, ear, nose, mouth, ...
Missing (red dashed box):wheel, tire, tree, light, ...
![Page 11: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/11.jpg)
Multiscale Testing
Image Pyramid
Model
[600, 800, 1000, 1333, 1666, 2000]
![Page 12: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/12.jpg)
Adj-soft NMS
Model
Dog: 0.99Dog: 0.9
Dog: 0.5
Dog: 0.99Dog: 0.04Dog: 0.99Dog: 0.5
Step1 Step2
NMS
Soft-NMS
Model Adj-soft NMS Public LB
{4models} 59.40
{4models} √ 60.35
![Page 13: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/13.jpg)
Expert Model
Class centroid matrix
(1)Build Similarity Matrix
Select Neg class(2)Generate Expert Class Group
Initial expert class group
Goldfish
Pig……
final expert class group
Goldfish
Pig
……Neg
Class
Train expert model
![Page 14: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/14.jpg)
Weakly & Fully Supervised Pipeline
ROI-Features
regression
classification
reg-loss
cls-loss
Bounding-box level
annotations
Image classificationannotations
max-MIL
Weakly supervised pipeline
Fully supervised pipeline
![Page 15: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/15.jpg)
Auto Ensemble
Detector
Detectors ZooBuilding model tree
DQN DQN DQN
: Functions set, such as NMS, Adj-soft NMS, and so on
Model inference from leaf to root
Selecting the operation in each via greedy algorithm
![Page 16: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/16.jpg)
Misc.
● Confusing definitions:
● For more please read our solutions
![Page 17: MMfruit - OpenImage 2019 1st solutionstorage.googleapis.com/openimages/challenge_2019/...Adj-soft NMS Expert Model Weakly & Fully Supervised Pipeline Auto Ensemble. Multiple Large](https://reader036.vdocuments.site/reader036/viewer/2022081411/60aa602cbd90de35d97fc16a/html5/thumbnails/17.jpg)
Thanks, Q & A