learning classifier systems for class imbalance problems
DESCRIPTION
Ester Bernadó-Mansilla analyzes the behavior of LCS on extreme class imbalance problemsTRANSCRIPT
![Page 1: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/1.jpg)
Learning Classifier Systemsfor Class Imbalance
Problems
Ester Bernadó-Mansilla
Research Group in Intelligent SystemsEnginyeria i Arquitectura La Salle
Universitat Ramon LlullBarcelona, Spain
![Page 2: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/2.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Aim
Enhance the applicability of LCSsto knowledge discovery from datasets
Classification problemsReal-world domains
![Page 3: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/3.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Framework
LCSDataset model
+estimated
performance• Representativity of the targetconcept
• Geometrical complexity
• Class imbalance
• Noise
• Evolutionary pressures
• Interpretability
• Domain of applicability
![Page 4: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/4.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Class Imbalance
When one class is represented by a small number ofexamples, compared to other class/es.
Usually the class of that describes the circumscribedconcept (positive class) is the minority class
Where? Rare medical diagnoses Fraud detection Oil spills in satellite images
![Page 5: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/5.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Class Imbalance and Classifiers
Is there a bias towards the majority class?
Probably, because… Most classifier schemes are trained to minimize the global error
As a result They classify accurately the examples from the majority class They tend to misclassify the examples of the minority class,
which are often those representing the target concept.
![Page 6: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/6.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Measures of Performance
Confusion matrix
true negative (TN)false positive (FP)Bfalse negative (FN)true positive (TP)A
BAActual
Prediction
Accuracy = (TP+TN)/(TP+FN+FP+TN)
TN rate = TN / (TN + FP)
TP rate = TP / (FN + TP)
ROC curves
![Page 7: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/7.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
concept: 15 counterpart: 150ratio: 10:1
Dataset 1
concept: 15 counterpart: 45ratio: 3:1
Dataset 2
The Higher Class Imbalance: theHigher Bias?
![Page 8: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/8.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
XCS
Environment
Set of Rules
input class
ReinforcementLearning
GeneticAlgorithms
reward
updatesearch
XCS
Dataset
![Page 9: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/9.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Our Approach with XCS
Bounding XCS’s parameters for unbalanced datasets
Online identification of small disjuncts
Adaptation of parameters for the discovery of smalldisjuncts
![Page 10: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/10.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Unbalanced 11-multiplexer problem
ir=64:1ir=32:1ir=16:1
XCS’s Behavior in UnbalancedDatasets
![Page 11: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/11.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
XCS’s Population
Classifier P Error F Num
###########:0 1000 0.12 0.98 385
###########:1 1.2 10-4
0.074 0.98 366
Most numerous rules, ir=128:1
overgeneral classifiers
estimated error:15.38
highfitness
too highnumerosity
Test examples are classified as belonging to the majority class
estimated prediction:
992.247.75
![Page 12: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/12.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
How Imbalance Affects XCS
Classifier’s error
Stability of prediction and error estimates
Occurrence-based reproduction
![Page 13: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/13.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Classifier’s Error in UnbalancedDatasets
Will an overgeneral classifier be detected as inaccurate if theimbalance ratio is high?
Bound for inaccurate classifier:
Given the estimated prediction and error:
We derive:
where
For
we get maximum imbalance ratio:
0!"!
))(1(||)(||
))(1()(
minmax
minmax
clPRPclPRP
RclPRclPP
cc
cc
!!+!"=
!+=
0)(2 00max
2!"#"#+"# Rppo
0!"!
CCp /!=
1998max
=ir
110000max=!=R
![Page 14: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/14.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Prediction and Error Estimates andLearning Rate
ir=128:1, ###########:0Prediction Error
β=0.
2β=
0.00
2
![Page 15: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/15.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Occurrence-based Reproduction
Probability of occurrence (pocc)
Given ir=maj/min:
Classifier poccB poccI
########### :0 1/2 1/2
########### :1 1/2 1/2
0000#######:0 1/32
occBpir
ir
1
2
+
0001#######:1 1/32
occBpir 1
2
+
0
0,1
0,2
0,3
0,4
0,5
0,6
1 2 4 8 16 32 64 128 256
imbalance ratio
pro
bab
ilit
y o
f o
ccu
rren
ce
00001######:1 00000######:0
###########:0 ###########:1
![Page 16: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/16.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Occurrence-based Reproduction
Probability of reproduction (pGA)
With θGA=20:
!"# <
$
=
otherwiseT
TifTwhere
Tp
occ
GAoccGA
GA
GA
GA
%%
1
1)0:#######0000(
)0:##########(#
occT
GAT
GAGAT
!
!"
1 Assuming non-overlapping
Tocc
θGA
…
Tocc
θGA
…
GA
GA
![Page 17: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/17.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Guidelines for Parameter Tuning
Rmax and є0 determine the threshold between negligible noise andimbalance ratio
β determines the size of the moving window. The window should behigh enough to allow computing examples from both classes:
θGA can counterbalance the reproduction opportunities of most frequent(majority) and least frequent niches (minority):
majf
fk min=!
min
1'f
kGA=!
![Page 18: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/18.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
XCS with Parameters Tuning
ir=64:1ir=32:1ir=16:1 ir=64:1 ir=256:1
XCS with parameter tuningXCS with standard settings
![Page 19: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/19.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
XCS Tuning for Real-world Datasets
How we can estimate the niche frequency?
Estimate from the ratio of majority class instances and minorityclass instances
Problem:
• This may not be related to the distribution of niches in the featurespace
Take the approach to the small disjuncts problem
![Page 20: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/20.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Online Identification of Small Disjuncts
We search for regions that promoteovergeneral classifiers
Estimate ircl based on the classifier’sexperience on each class:
Adapt β and θGA according to ircl
4/20=clir
min
max
exp
exp=
clir
![Page 21: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/21.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Online Parameter Adaptation
ir=256:1
![Page 22: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/22.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
What about UCS?
Supervised XCS: Needs less exploration
Avoids XCS’s fitness dilemma
More robust to parameter settings
Overgeneral classifiers also tend to overcome thepopulation Their probability of occurrence depends on the imbalance ratio Partially minimized with fitness sharing
![Page 23: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/23.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
ir=256:1ir=512:1
What about UCS?
![Page 24: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/24.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Are LCSs more error-prone to class imbalancethan other classifier schemes?
24,99%±30,50%17,07%±9,50%12,87%±41,00%wpbc
4,52%±98,57%6,02%±97,14%11,70%±90,18%wine2c3
5,27%±98,33%5,27%±98,33%8,05%±95,00%wine2c2
0,00%±100,00%0,00%±100,00%16,63%±89,00%wine2c1
6,37%±93,83%2,71%±94,81%5,09%±92,47%wdbc
5,89%±95,83%5,36%±95,42%3,42%±92,95%wbdc
3,65%±87,86%3,48%±89,97%2,38%±77,64%wab2c3
2,57%±78,72%4,05%±84,57%3,89%±72,34%wav2c2
3,43%±87,24%3,20%±88,51%4,06%±75,74%wav2c1
8,05%±90,71%21,35%±33,81%10,34%±90,95%thy2c3
14,93%±90,83%24,92%±54,17%12,45%±94,17%thy2c2
16,10%±90,00%22,50%±76,67%16,10%±90,00%thy2c1
5,72%±92,58%6,17%±84,11%2,14%±95,23%tao
9,75%±55,93%6,42%±53,38%13,27%±55,37%pim
9,78%±80,00%7,03%±80,00%13,29%±75,83%h-s
13,95%±81,79%0,00%±0,00%15,13%±59,82%gls2c6
14,21%±84,29%9,64%±10,00%16,77%±77,14%gls2c5
25,40%±81,67%25,40%±81,67%32,63%±75,00%gls2c4
15,81%±5,00%0,00%±0,00%42,16%±30,00%gls2c3
49,72%±55,00%33,75%±15,00%47,43%±35,00%gls2c2
52,70%±50,00%0,00%±0,00%42,16%±80,00%gls2c1
9,10%±61,38%0,00%±0,00%14,09%±42,95%bpa
6,88%±83,99%5,59%±93,77%6,04%±81,90%Bal2c3
6,00%±81,96%4,64%±93,72%6,83%±81,65%Bal2c2
0,00%±0,00%0,00%±0,00%0,00%±0,00%Bal2c1
XCSSMOC4.5
TP rate
![Page 25: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/25.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
How can we Minimize the Effects ofSmall Disjuncts?
Resampling the dataset: Classical methods:
• Random oversampling• Random undersampling
Heuristic methods:• Tomek links• CNN• One-sided selection• Smote
Cluster-based oversampling
Cost-sensitive classifiers
Addresses smalldisjuncts
Assumes thatclusterization willfind smalldisjuncts andmatch classifier’sapproximation
Could XCSbenefit from theonlineidentification ofsmall disjuncts?
![Page 26: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/26.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Domains of Applicability
Should we use some counterbalancing scheme?
Which learning scheme should we use?
Is there a combination of counterbalancingscheme+learner that beats all others?
How can we know the presence of smalldisjuncts?
Are there other complexity factors mixed up withthe small disjuncts problem?
![Page 27: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/27.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Domains of Applicability
Learn it!
PredictionDataset
characterization
Type of dataset:Geometrical distribution of classes
Possible presence of small disjunctsOther complexity factors
Resampling/Classifier/Resampling+classifier
Suggestedapproach
Where areLCSs
placed?
Dataset
![Page 28: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/28.jpg)
Learning Classifier Systems for Class Imbalance Problems Ester Bernadó-Mansilla
Future Directions
Potential benefit of XCS to discover small disjuncts…and learn from it online
Further analyze UCS
How do LCSs perform w.r.t. other classifiers for unbalanceddatasets?
Measures for small disjuncts identification… and other possible complexity factors
What is noise and what is a small disjunct?
In which cases a LCS is applicable?
![Page 29: Learning Classifier Systems for Class Imbalance Problems](https://reader034.vdocuments.site/reader034/viewer/2022051514/549fde10ac795971328b464e/html5/thumbnails/29.jpg)
Learning Classifier Systemsfor Class Imbalance
Problems
Ester Bernadó-Mansilla
Research Group in Intelligent SystemsEnginyeria i Arquitectura La Salle
Universitat Ramon LlullBarcelona, Spain