Milan Šulc Fine-grained Recognition of Plant Species from Images

Fine-grained Recognitionof Plant Species from Images

Milan Šulc

PhD candidate Advisor Jiřiacute Matas

Visual Recognition GroupCenter for Machine PerceptionCzech Technical University in Prague

Milan Šulc Fine-grained Recognition of Plant Species from Images

Many formulations exist depending on

bull Type of observationbull from an image (or images) of a known plant organbull from images of multiple organsbull from images of a larger part of the plant

bull Acquisition conditions bull controlled background viewpoint occlusion vs unconstrained

bull Granularity of the decision Typically fine-grained classification required the classes have

bull Small inter-class differencesbull High intra-class variability

Plant Species Recognition

236

Milan Šulc Fine-grained Recognition of Plant Species from Images

The observation depends on many factors

bull Genotypebull Agebull Seasonbull Local environment

Climate Altitude Illumination

bull Clutter (other plants in the foreground or background)bull Acquisition conditions

bull Device

Plant Species Recognition

336

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Constrained recognition of plants from photos of tree bark photos or scans of leaves

ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo

bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo

bull Re-weighting predictions for different training- and test-time class prior probabilities

bull Test-time class prior estimation

bull Future work Knowledge distillation from Ensembles

Presentation Outline

436

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Bark recognition [1] using texture descriptors based on Local Binary Patterns

bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]

[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011

Our Early Work on Plant Recognition

536

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Fast description Histograms of Completed LBP (Local Binary Patterns)

2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)

3) Improved scale space for multi-scale description and scale invariance

4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]

Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)

Fast Features Invariant to Rotationand Scale of Texture

636

184 129 140

159 156 150

168 80 130

0

rarr00011100

0

0

0

0

1

1

1

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Many formulations exist depending on

bull Type of observationbull from an image (or images) of a known plant organbull from images of multiple organsbull from images of a larger part of the plant

bull Acquisition conditions bull controlled background viewpoint occlusion vs unconstrained

bull Granularity of the decision Typically fine-grained classification required the classes have

bull Small inter-class differencesbull High intra-class variability

Plant Species Recognition

236

Milan Šulc Fine-grained Recognition of Plant Species from Images

The observation depends on many factors

bull Genotypebull Agebull Seasonbull Local environment

Climate Altitude Illumination

bull Clutter (other plants in the foreground or background)bull Acquisition conditions

bull Device

Plant Species Recognition

336

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Constrained recognition of plants from photos of tree bark photos or scans of leaves

ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo

bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo

bull Re-weighting predictions for different training- and test-time class prior probabilities

bull Test-time class prior estimation

bull Future work Knowledge distillation from Ensembles

Presentation Outline

436

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Bark recognition [1] using texture descriptors based on Local Binary Patterns

bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]

[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011

Our Early Work on Plant Recognition

536

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Fast description Histograms of Completed LBP (Local Binary Patterns)

2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)

3) Improved scale space for multi-scale description and scale invariance

4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]

Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)

Fast Features Invariant to Rotationand Scale of Texture

636

184 129 140

159 156 150

168 80 130

0

rarr00011100

0

0

0

0

1

1

1

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

The observation depends on many factors

bull Genotypebull Agebull Seasonbull Local environment

Climate Altitude Illumination

bull Clutter (other plants in the foreground or background)bull Acquisition conditions

bull Device

Plant Species Recognition

336

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Constrained recognition of plants from photos of tree bark photos or scans of leaves

ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo

bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo

bull Re-weighting predictions for different training- and test-time class prior probabilities

bull Test-time class prior estimation

bull Future work Knowledge distillation from Ensembles

Presentation Outline

436

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Bark recognition [1] using texture descriptors based on Local Binary Patterns

bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]

[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011

Our Early Work on Plant Recognition

536

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Fast description Histograms of Completed LBP (Local Binary Patterns)

2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)

3) Improved scale space for multi-scale description and scale invariance

4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]

Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)

Fast Features Invariant to Rotationand Scale of Texture

636

184 129 140

159 156 150

168 80 130

0

rarr00011100

0

0

0

0

1

1

1

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Constrained recognition of plants from photos of tree bark photos or scans of leaves

ldquoOld-school handcrafted features achieve excellent recognition ratesrdquo

bull Plant recognition ldquoin the wildrdquo for up to 10K plant species ldquoAlmost unconstrained scenario Outperforms 59 human experts best results in the LifeCLEF 2018 plant identification challengerdquo

bull Re-weighting predictions for different training- and test-time class prior probabilities

bull Test-time class prior estimation

bull Future work Knowledge distillation from Ensembles

Presentation Outline

436

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Bark recognition [1] using texture descriptors based on Local Binary Patterns

bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]

[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011

Our Early Work on Plant Recognition

536

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Fast description Histograms of Completed LBP (Local Binary Patterns)

2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)

3) Improved scale space for multi-scale description and scale invariance

4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]

Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)

Fast Features Invariant to Rotationand Scale of Texture

636

184 129 140

159 156 150

168 80 130

0

rarr00011100

0

0

0

0

1

1

1

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Bark recognition [1] using texture descriptors based on Local Binary Patterns

bull Fast Features Invariant to Rotation and Scale of Texture (Ffirst) [2]

[1] Kernel-mapped histograms of multi-scale LBPs for tree bark recognition Milan Šulc and Jiřiacute Matas IVCNZ 2013[2] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[3] Automated identification of tree species from images of the bark leaves and needles Stefan Fiel and Robert Sablatnig Master thesis 2010 CVWW 2011

Our Early Work on Plant Recognition

536

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Fast description Histograms of Completed LBP (Local Binary Patterns)

2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)

3) Improved scale space for multi-scale description and scale invariance

4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]

Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)

Fast Features Invariant to Rotationand Scale of Texture

636

184 129 140

159 156 150

168 80 130

0

rarr00011100

0

0

0

0

1

1

1

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Fast description Histograms of Completed LBP (Local Binary Patterns)

2) Rotation-invariant representation ldquoHistogram Fourier featuresrdquo (extended)

3) Improved scale space for multi-scale description and scale invariance

4) Linear SVM classifiers approximating the intersection kernel via explicit feature maps [2]

Very fast descriptor a 200x200px image takes lt 5ms on a laptop CPU[1] Fast Features Invariant to Rotation and Scale of Texture Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Efficient additive kernels via explicit feature maps Andrea Vedaldi and Andrew Zisserman IEEE TPAMI 343 (2012)

Fast Features Invariant to Rotationand Scale of Texture

636

184 129 140

159 156 150

168 80 130

0

rarr00011100

0

0

0

0

1

1

1

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Leaf recognition texture of the leaf interior and border [1][2]

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Our Early Work on Plant Recognition

736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our Early Work on Plant Recognition

836

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Leaf Recognition in the Era of CNNs

936

[1] Texture-Based Leaf Identification Milan Šulc and Jiřiacute Matas ECCV 2014 CVPPP workshop[2] Fine-grained Recognition of Plants from Images Milan Šulc and Jiřiacute Matas Plant Methods 13 (1) 115 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 1000 plant species (herbs trees ferns) no fungibull 113205 images in training set different ldquotypes of viewrdquo

Leaf Leaf Scan Flower Fruit Stem Branch Entirebull Meta-data available

bull Organ known surprisingly organ-specific classifiers performed worsebull ldquoObservation IDrdquo available both in test and trainingbull Author ID sometimes GPS coordinates

bull Contained ldquodistractorsrdquo in the test set

[1] LifeCLEF 2016 multimedia life species identification challenges Alexis Joly et al CLEF 2016[2] Plant identification in an open-world (LifeCLEF 2016) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2016

LifeCLEF 2016 Plant Identification Task

1036

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Resnet-152 [2]~ 113 GFLOPs while 8x deeper than VGG-19 (193 GFLOPs)

bull Maxout [3]

bull 4 pieces of FC layers followed with maxoutbull Dropout is performed on the inputs of the FC layersbull Another FC layer is added on the top for classification

VGG-19 196 billion FLOPsResNet-152 113 billion FLOPs

[1] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016[2] Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun CVPR 2016[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327

LifeCLEF 2016 Our Approach [1]

1136

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonbull Combining test images with the same observation ID was allowed

in the competition and had a significant effect on the final scores

[1] Bluefield (KDE TUT) at LifeCLEF 2016 Plant Identification Task Siang Thye Hang Atsushi Tatsuma Masaki Aono CLEF 2016[2] Open-set Plant Identification Using an Ensemble of Deep Convolutional Neural Networks Mostafa Mehdipour Ghazi Berrin Yanikoglu Erchan Aptoula CLEF 2016[3] Very Deep Residual Networks with MaxOut for Plant Identification in the Wild Milan Šulc Jiřiacute Matas Dmytro Mishkin CLEF 2016

LifeCLEF 2016 Plant Identification Task

Team Single-image recognition [ mAP]

Sum per observation[ mAP]

Bluefield [1] 611 742

SabanciUGebzeTU [2] 738 793

CMP (ours) 710 788

1236

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull 10000 plant species (herbs trees ferns)bull ldquotrustedrdquo training set of ~ 256 000 images

obtained from Encyclopedia Of Life (EoL)

bull ldquonoisyrdquo training set of ~1 400 000 imagesfrom Bing and Google image search

bull test set of 25K images no distractors

bull 80 participants registered only 8 participants submitted results

LifeCLEF 2017 Plant Identification Task

1336

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Random examples from the ldquotrustedrdquo training set

bull Random examples from the ldquonoisyrdquo training set

LifeCLEF 2017 Plant Identification Task

1436

[1] Plant identification based on noisy web data the amazing performance of deep learning (LifeCLEF 2017) Herveacute Goeumlau Pierre Bonnet and Alexis Joly CLEF 2017

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Our approach [1]

bull Inception-ResNet-v2 [2]

bull Maxout [3]

bull Bootstrapping [4] consistency objectives for training on noisy labels

[1] Learning with Noisy and Trusted Labels for Fine-Grained Plant Recognition Milan Šulc and Jiřiacute Matas CLEF 2017[2] Inception-v4 Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke and Alexander A Alemi[3] Maxout Networks Ian J Goodfellow David Warde-Farley Mehdi Mirza Aaron C Courville and Yoshua Bengio ICML (3) 28 (2013) 1319-1327[4] Training deep neural networks on noisy labels with bootstrapping Scott Reed Honglak Lee Dragomir Anguelov Christian Szegedy Dumitru Erhan and Andrew Rabinovich

LifeCLEF 2017 Plant Identification Task

1536

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Results and lessonsbull Increasing size and influence of CNN ensemblesbull Best performing methods used both trusted and noisy data

additional train data (PlantCLEF 2016) helped challenge winner [1]bull No participants made use of GPS coordinates

[1] Image-based plant species identification with deep convolutional neural networks Mario Lasseck CLEF 2017

LifeCLEF 2017 Plant Identification Task

1636

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2014 test set external resources (field guides books) not allowed for experts

1736

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experts vs Machines (after LifeCLEFrsquo17)

Evaluation on part of the PlantCLEF 2017 test set external resources (field guides books) allowed for experts

1836

[1] Plant Identification Experts vs Machines in the Era of Deep Learning Pierre Bonnet Herveacute Goeumlau Siang Thye Hang Mario Lasseck Milan Šulc Alexis Joly Submitted to Multimedia Tools and Applications

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull Scenario similar to the 2017 challenge

bull 10000 plant species (herbs trees ferns)bull Datasets from precious years available for training

2017 ldquotrustedrdquo training set of ~ 256K images (EoL) 2017 ldquonoisyrdquo training set of ~14M images (web) 2017 test set of ~ 25K images with GT labels 2016 training- and test- data ~ 64K images covering a small

subset of the 10k species

bull Test set of 6892 images

bull Organized with the intention to compare deep learning methods against human experts in plant sciences(On a subset of the test data)

LifeCLEF 2018 Plant Identification Task

1936

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Preliminary experiments (using the 2017 test set for validation)

1 Running averages (exponential decay) of trainable variables

684 rarr 802 accuracy (Inception-ResNet-v2 840k iterations with batch size 32)

Interpretation Noisy samples (from web majority of data) in mini-batches

may cause ldquomisleadingrdquo gradients running averages are more robust to such ldquoerrorsrdquo

Learning rate was probably too high

2 Assuming changes in class prior distribution is very important

LifeCLEF 2018 Plant Identification Task

2036

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

0 10 20Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Label Distributions

2136

0 2000 4000 6000 8000 10000Class (sorted by Ntrain)

0

500

1000

1500

2000

2500

im

ag

es

Test set

Training set

Large datasets downloaded from the web typically follow a ldquolong-tail distributionrdquo of classes which may not correspond with the real test-time distributionCan we compensate for this imbalance

Figure PlantCLEF 2017 label distribution in the ldquotrustedrdquo training set

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Training neural networks (f with parameters ) by cross-entropy loss minimization means training it to estimate the posterior probabilities

where

Then

CNN Outputs as Posterior Estimates

2236

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Experiment on selected subsets of CIFAR-100 with different class priorsHow well do the posterior estimates marginalize over dataset samples

CNN Outputs as Posterior Estimates

2336

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Assuming that the probability density function remains unchanged

The mutual relation of the posteriors is

Adjusting Estimates to New Priors

2436

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

When Test Set Priors Are Unknown

2536

How to estimate the test-set priors

Saerens et al proposed a simple EM procedure to maximize the likelihood L(x

0x1x2)

This procedure is equivalent [2] to fixed-point-iteration minimization of

the KL divergence between and

[1] Adjusting the outputs of a classifier to new a priori probabilities a simple procedure Marco Saerens Patrice Latinne and Christine Decaestecker Neural computation 141 (2002) 21-41[2] Semi-supervised learning of class balance under class-prior change by distribution matching Marthinus Christoffel Du Plessis and Masashi Sugiyama Neural Networks 50110ndash119 2014

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Test Set Prior Estimation in LifeCLEF

2636

Preliminary experiments (using the 2017 test set for validation)

When the whole test set is availableInception-ResNet-v2 829 rarr 858Inception-v4 828 rarr 863

On-line [1] after each new test image

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Plant Ident Task Results

2736

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

LifeCLEF 2018 Experts vs Machines

2836

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

2936

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3036

0 200 400 600 800 1000 1200Class (sorted)

00

01

02

03

04

05

s

am

ple

s

Training set

Validation set

0 1000 2000 3000 4000 5000 6000 7000 8000Class (sorted)

000

005

010

015

020

s

am

ple

s

Training set

Validation set

iNaturalist 2018 FGVCx Fungi 2018

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

35

40

45

50

55

60

65

70

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

0 50000 100000 150000 200000 250000 300000 350000 400000Training steps

350

375

400

425

450

475

500

525

Acc

ura

cy [

]

CNN output accuracy

Known (flat) test distr

[1] Improving CNN classifiers by estimating test-time priors Milan Šulc and Jiřiacute Matas arXiv180508235 [csCV] 2018

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

When New Priors Are Known

3136

Note in the iNaturalist 2017 challenge the winning GMV submission [1] approached the change in priors as follows

ldquoTo compensate for the imbalanced training data the models were further fine-tuned on the 90 subset of the validation data that has a more balanced distributionrdquo

[1] The iNaturalist Species Classification and Detection Dataset-Supplementary Material Grant Van Horn Oisin Mac Aodha Yang Song Yin Cui Chen Sun Alex Shepard Hartwig Adam Pietro Perona and Serge Belongie Reptilia 32 no 400 5426

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3236

So far we approached the estimation of new class priorsby Maximum Likelihood (ML)

It is often a good idea to use Maximum a Posteriori (MAP) framework and include additional prior information or assumptions Wersquoll use a Dirichlet distribution as a hyper-prior on the class priors P

k

α ge 1 - favours dense distributions rarr avoids Pkasymp0

- makes Dirichlet log-concave rarr suitable for SGD

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3336

CIFAR-100 experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

Hyper-prior on Class Priors

3436

Fine-grained experiments

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

1) Knowledge distillation from Ensembles

Ensembles of Convolutional Neural Networks outperform individual models That holds even when all models in the ensemble

have the same architecture are trained on the same training set

How to distill knowledge from ensembles to improve single-model accuracy

2) Test set prior estimation

In many cases we cannot assume to remain unchangedEg when training from noisy data or when part of the training images comes from ldquoin vitrordquo sources (encyclopedia on-line stores)

3) Learning embeddings Metric learning

As samples and classes grow focusing on the embedding space(eg training with triplet or contrastive loss) may be practical

3536

Future Work

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion

Milan Šulc Fine-grained Recognition of Plant Species from Images

bull State-of-the-art CNN classifiers (and their ensembles) achievebest results in plant recognition

bull Constrained tasks can be solved by faster features with 99+ accuracy

bull Properly learning from large ldquonoisyrdquo data is an open machine learning problem

bull It is important to take into account change in class prior distribution New priors can be estimated on-line as new test-samples appear

bull Human expert performance reached

bull Q amp A sulcmilacmpfelkcvutcz

3636

Discussion