ACTIVE LEARNING USING CONFORMAL PREDICTORS: APPLICATION TO IMAGE

CLASSIFICATION

Hyp Introduction Hyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

L. Makili1, J. Vega2, S. Dormido-Canto3

1Universidade Katyavala Bwila. Benguela (Angola)2Asociación EURATOM/CIEMAT para Fusión. Madrid (Spain)

3Universidad Nacional de Educación a Distancia (UNED). Madrid (Spain)

7th Workshop on Fusion Data Processing Validation and Analysis

Outline

• Introduction• Concepts overview• Experimental results• Conclusions

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Motivation • 5 – class classification problem

– Classification of TJ – II Thomson Scattering images

• Classifier based on conformal predictors, using SVM as the underlying algorithm– Computationally intensive task

Patterns of TSD images: (a) BKGND, (b) COFF, (c) ECRH, (d) NBI and (e) STRAY

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Goal

• To find out a minimal and good enough training dataset for classification purposes

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Active learning• The learning algorithm must have some

control over the data from which it learns• It must be able to query an oracle, requesting

for labels of data samples that seem to be most informative for the learning process

• Proper selection of samples implies better performances with fewer data

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Settles, B. “Active Learning Literature Survey. Computer Sciences Technical Report 1648”, University of Wisconsin – Madison, 2009. Available at http://research.cs.wisc.edu/tech reports/2009/TR1648.pdf

Uncertainty sampling

• The learning algorithm selects new examples when their class membership is unclear

• Suitable for classifiers that besides making classification decisions, estimates certainty of these decisions

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Lewis, D. and Gale, W., “A Sequential Algorithm for Training Text Classifiers”. In Proceedings of the ACM – SIGIR Conference on Research and Development in Information Retrieval, Croft, W. B. and van Rijbergen, C. J. (eds). New York: Springer – Verlag, 1994, pp. 3 – 12

Conformal prediction• Permits complementation of predictions made

by machine learning algorithms with some measures of reliability

• Besides the label predicted for a new object, it outputs two additional values– Confidence– Credibility

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Vovk, V., Gammerman, A. and Shafer, G., Algorithmic Learning in a Random World, New York: Springer Science + Business Media, Inc., 2005

Conformal prediction

• Used as nonconformity scores the Lagrange multipliers computed during SVM training

• Extended to a multiclass framework in a one-vs-rest approach

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Active learning algorithm• Inputs

– Initial training set T, calibration set C, pool of candidate samples U

– Selection treshold τ, batch size β• Train an initial classifier on T• While a stopping-criterion is not reached

– Apply the current classifier to the pool of samples– Rank the samples in the pool using the uncertainty criterion– Select the top β examples whose certainty level fall under the

selection threshold τ– Ask teacher to label the selected samples and add them to the

training set– Train a new classifier on the expanded training set

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

(Un)Certainty criteria• Credibility:

• Confidence:

• Query-by-transduction:

• Multicriteria:

– Combination 1:

– Combination 2:

credIcr )(x

confIcf )(x

)1()( confcredIqbt x

)()1()()( 21 xxx III

78.0);()1()()( xxx cfqbt III

5.0);()1()()( xxx cfqbt III

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Ho, S. and Wechsler, H., “Query by Transduction”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30 (9), 2008, pp. 1557 – 1571

Setup • 1149 samples divided into

– Initial training: 5 samples– Pool: 794 samples– Calibration set: 150 samples– Test set: 200 samples

• Batch size: 25 • Selection treshold : 0.4

• Used with Qbt, Comb1 and Comb2

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Setup • For Each criterion in (Qbt, Comb1, Comb2)

– For experiment = 1 : 10• Select test set• Run active learning algorithm selecting 700 samples hybridly• For NumTrn = 50 : 50 : 700

– Train CP classifier on the first NumTrn samples – Aplly classifier to the test set

• End for NumTrn

– End For experiment• End For Each

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Results

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Results

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

Conclusions• Active learning was applied to the selection of a

minimal and good enough training dataset for classification purposes

• It allows reaching higher success rates and confidence in predictions with fewer data points, compared to the random selection of the training set

• Combining multiple criteria we can balance the trade-off between success rate and confidence of prediction improvement

Hyp IntroductionHyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions

ACTIVE LEARNING USING CONFORMAL PREDICTORS: APPLICATION TO IMAGE

CLASSIFICATION

THANK YOU

7th Workshop on Fusion Data Processing Validation and Analysis

Hyp Introduction Hyp Conceptual overview

Hyp Experiments and resultsHyp Conclusions