multiple classifier systems
TRANSCRIPT
![Page 1: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/1.jpg)
Multiple Classifier SystemFarzad Vasheghani Farahani – Machine Learning
![Page 2: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/2.jpg)
Outline
Introduction
Decision Making
General Idea
Brief History
Reasons & Rationale
Statistical
Large volumes of data
Too little data
Divide and Conquer
Data Fusion
Multiple Classifier system
Designing
Diversity
Create an Ensemble
Combining Classifiers
Example
Conclusions
References
![Page 3: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/3.jpg)
Ensemble-based Systems in Decision
Making
For many tasks, we often seek second opinion before making a decision,
sometimes many more
Consulting different doctors before a major surgery
Reading reviews before buying a product
Requesting references before hiring someone
We consider decisions of multiple experts in our daily lives
Why not follow the same strategy in automated decision making?
Multiple classifier systems, committee of classifiers, mixture of experts,
ensemble based systems
![Page 4: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/4.jpg)
Ensemble-based Classifiers
How to (i) generate individual components of the ensemble systems
(base classifiers), and (ii) how to combine the outputs of individual
classifiers?
![Page 5: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/5.jpg)
Brief History of Ensemble Systems
Dasarathy and Sheela (1979) partitioned the feature space using two
or more classifiers
Schapire (1990) proved that a strong classifier can be generated by
combining weak classifiers through boosting; predecessor of AdaBoost
algorithm
Two types of combination:
classifier selection
classifier fusion
![Page 6: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/6.jpg)
Why Ensemble Based Systems?
![Page 7: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/7.jpg)
Why Ensemble Based Systems?
1. Statistical reasons
A set of classifiers with similar training performances may have different generalization performances
Combining outputs of several classifiers reduces the risk of selecting a poorly performing classifier
Example:
Suppose there are 25 base classifiers
Each classifier has error rate, = 0.35
Probability that the ensemble classifier makes a wrong prediction:
2525
1
25(1 ) 0.06i i
i i
![Page 8: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/8.jpg)
Why Ensemble Based Systems?
2. Large volumes of data
If the amount of data to be analyzed is too large, a single classifier
may not be able to handle it; train different classifiers on
different partitions of data
![Page 9: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/9.jpg)
Why Ensemble Based Systems?
3. Too little data
Ensemble systems can also be used when there is too little data;
resampling techniques
![Page 10: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/10.jpg)
Why Ensemble Based Systems?
4. Divide and Conquer
Divide data space into smaller & easier-to-learn partitions; each classifier learns only one of the simpler partitions
![Page 11: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/11.jpg)
Why Ensemble Based Systems?
5. Data Fusion
Given several sets of data from various sources, where the nature
of features is different (heterogeneous features), training a single
classifier may not be appropriate (e.g., MRI data, EEG recording,
blood test,..)
![Page 12: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/12.jpg)
Multiple Classifier system Designing
![Page 13: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/13.jpg)
Major Steps
All ensemble systems must have two key components:
Generate component classifiers of the ensemble
Method for combining the classifier outputs
![Page 14: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/14.jpg)
“Diversity” of Ensemble
Objective: create many classifiers, and combine their outputs
to improve the performance of a single classifier
Intuition: if each classifier makes different errors, then their
strategic combination can reduce the total error!
Need base classifiers whose decision boundaries are adequately
different from those of others
Such a set of classifiers is said to be “diverse”
![Page 15: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/15.jpg)
How to achieve classifier diversity?
A. Use different training sets to train individual classifiers
B. Use different training parameters for a classifier
C. Different types of classifiers (MLPs, decision trees, NN
classifiers, SVM) can be combined for added diversity
D. Using random feature subsets, called random subspace
method
![Page 16: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/16.jpg)
Create an Ensemble(Coverage Optimization)
![Page 17: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/17.jpg)
Creating An Ensemble
Two questions:
1. How will the individual classifiers be generated?
2. How will they differ from each other?
![Page 18: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/18.jpg)
Create Ensembles Methods
1. Subsample Approach (Data sample)
Bagging
Random forest
Boosting
Adaboost
Wagging
Rotation forest
RotBoost
Mixture of Expert
2. Subspace Approach (Feature Level)
Random based
Feature reduction
Performance based
3. Classifier Level Approach
![Page 19: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/19.jpg)
Bagging
![Page 20: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/20.jpg)
Boosting
![Page 21: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/21.jpg)
Combining Classifiers(Decision Optimization)
![Page 22: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/22.jpg)
Two Important Concept (i)
(i) trainable vs. non-trainable
Trainable rules: parameters of the combiner, called “weights” determined through a separate training algorithm
Non-trainable rules: combination parameters are available as classifiers are generated; Weighted majority voting is an example
![Page 23: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/23.jpg)
Two Important Concept (ii)
(ii) Type of the output of classifiers
Combine Classifier
Absolute output
Majority Voting
Naïve Bayes
Behavior Knowledge Space
Ranked output
Borda Counting
Maximum Ranking
Continuous output
Algebraic Metohd
Fuzzy Integral
Decesion Template
![Page 24: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/24.jpg)
Example (“Zoo” UCI Data Set)
1. animal name: Unique for each instance2. hair: Boolean3. feathers: Boolean4. eggs: Boolean5. milk: Boolean6. airborne: Boolean7. aquatic: Boolean8. predator: Boolean9. toothed: Boolean10. backbone: Boolean11. breathes: Boolean12. venomous: Boolean13. fins: Boolean14. legs: Numeric (set of values: {0,2,4,5,6,8})15. tail: Boolean16. domestic: Boolean17. catsize: Boolean18. type: Numeric (integer values in range [1,7])
![Page 25: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/25.jpg)
Conclusions
Ensemble systems are useful in practice
Diversity of the base classifiers is important
Ensemble generation techniques: bagging, AdaBoost, mixture of
experts
Classifier combination strategies: algebraic combiners, voting
methods, and decision templates.
No single ensemble generation algorithm or combination rule is
universally better than others
Effectiveness on real world data depends on the classifier diversity
and characteristics of the data
![Page 26: Multiple Classifier Systems](https://reader033.vdocuments.site/reader033/viewer/2022042518/55a6cef51a28abe07a8b477c/html5/thumbnails/26.jpg)
References
[1] Polikar R., “Ensemble Based Systems in Decision Making,” IEEE
Circuits and Systems Magazine, vol.6, no. 3, pp. 21-45, 2006
[2] Polikar R., “Bootstrap Inspired Techniques in Computational Intelligence,” IEEE Signal Processing Magazine, vol.24, no. 4, pp. 56-72, 2007
[3] Polikar R., “Ensemble Learning,” Scholarpedia, 2008.
[4] Kuncheva, L. I. , Combining Pattern Classifiers: Methods and
Algorithms. New York, NY: Wiley, 2004.