pitfalls in benchmarking data stream classification and how to avoid them
DESCRIPTION
TRANSCRIPT
Pitfalls in Benchmarking Data StreamClassification and How to Avoid Them
Albert Bifet1, Jesse Read2, Indre Zliobaite3
Bernhard Pfahringer4, Geoff Holmes4
1Yahoo! Research Barcelona2Universidad Carlos III, Madrid, Spain
3Aalto University and Helsinki Institute for Information Technology (HIIT), Finland4University of Waikato, Hamilton, New Zealand
ECML-PKDD 2013, 25 September 2013
Data Streams
Data StreamsI Sequence is potentially infiniteI High amount of data: sublinear spaceI High speed of arrival: sublinear time per exampleI Once an element from a data stream has been processed
it is discarded or archived
Big Data & Real Time
1. Motivation
Electricity Dataset
I Popular benchmark for testing adaptive classifiersI Collected from the Australian New South Wales Electricity
Market.I Contains 45,312 instances which record electricity prices
at 30 minute intervals.I The class label identifies the change of the price (UP or
DOWN) related to a moving average of the last 24 hours.
Electricity Dataset, Accuracy
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Acc
urac
y,%
VFDT Majority ClassNaive Bayes
Electricity Dataset, Accuracy
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Acc
urac
y,%
Magic Classifier VFDTMajority Class Naive Bayes
Electricity Dataset, Kappa Statistic
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Kap
paS
tatis
tic,%
VFDT Naive Bayes
Electricity Dataset, Kappa Statistic
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Kap
paS
tatis
tic,%
Magic Classifier VFDTNaive Bayes
Electricity Dataset, Accuracy
Algorithm name Acc. (%) Algorithm name Acc. (%)DDM 89.6* Local detection 80.4Learn++.CDS 88.5 Perceptron 79.1KNN-SPRT 88.0 AUE2 77.3GRI 88.0 ADWIN 76.6FISH3 86.2 EAE 76.6EDDM-IB1 85.7 Prop. method 76.1Magic classifier 85.3 Cont. λ-perc. 74.1ASHT 84.8 CALDS 72.5bagADWIN 82.8 TA-SVM 68.9DWM-NB 80.8* tested on a subset
2. Problem
No-Change classifier: Weather classifier
Prediction for tomorrow: the same astoday
Electricity Dataset, Accuracy
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Acc
urac
y,%
No-Change VFDTMajority Class Naive Bayes
Electricity Dataset, Kappa Statistic
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Kap
paS
tatis
tic,%
No-Change VFDTNaive Bayes
Characteristics of the Electricity Dataset
0.5 1 1.5 2 2.5 3 3.5 4 4.5
·104
20
30
40
50
60
Time, instances
Cla
sspr
ior,
%
Characteristics of the Electricity Dataset
20 40 60 80 100 120 140 160 180 200
0
0.5
1
Lag, instances
Aut
ocor
rela
tion
3. Proposal
New Evaluation for Stream Classifiers
Kappa Statistic
I p0: classifier’s prequential accuracyI pc : probability that a chance classifier makes a correct
prediction.I κ statistic
κ =p0 − pc
1 − pc
I κ = 1 if the classifier is always correctI κ = 0 if the predictions coincide with the correct ones as
often as those of the chance classifier
New Evaluation for Stream Classifiers
Kappa Plus Statistic
I p0: classifier’s prequential accuracyI pe: no-change classifier’s prequential accuracy
I κ+ statisticκ+ =
p0 − pe
1 − pe
I κ+ = 1 if the classifier is always correctI κ+ = 0 if the predictions coincide with the correct ones as
often as those of the no-change classifier
Electricity Market Dataset Accuracy
0 1 2 3 4
·104
60
80
100
Time, instances
Acc
urac
y,%
No-Change HATLev. Bagging
Electricity Market Dataset κ
0 1 2 3 4
·104
0
20
40
60
80
100
Time, instances
Kap
paS
tatis
tic,%
No-Change HATLev. Bagging
Electricity Market Dataset κ+
0 1 2 3 4
·104
−300
−200
−100
0
100
Time, instances
Kap
paP
lus
Sta
tistic
,%
No-Change HATLev. Bagging
SWT: Temporally Augmented Classifier
SWT: meta strategy that builds meta instances by augmentingthe original input attributes with the values of recent classlabels from the past
Pr [class is c] ≡ h(x t , ct−`, . . . , ct−1)
for the t-th test instance, where ` is the size of the slidingwindow over the most recent true labels.
Electricity Market Dataset κ+
0 1 2 3 4
·104
−300
−200
−100
0
100
Time, instances
Kap
paP
lus
Sta
tistic
,%
No-Change SWT HATSWT Lev. Bagging
Electricity Market Dataset κ+
0 1 2 3 4
·104
−300
−200
−100
0
100
Time, instances
Kap
paP
lus
Sta
tistic
,%
No-Change HATLev. Bagging
Electricity Market Dataset κ+
0 1 2 3 4
·104
−300
−200
−100
0
100
Time, instances
Kap
paP
lus
Sta
tistic
,%
No-Change SWT HATSWT Lev. Bagging
Forest Cover Type Dataset
0 2 4
·105
60
80
100
Time, instances
Acc
urac
y,%
No-Change HATLev. Bagging
0 2 4
·105
0
20
40
60
80
100
Time, instances
Kap
paS
tatis
tic,%
No-Change HATLev. Bagging
0 2 4
·105
−300
−200
−100
0
100
Time, instances
Kap
paP
lus
Sta
tistic
,%
No-Change HATLev. Bagging
0 2 4
·105
0
20
40
60
80
100
Time, instances
Acc
urac
y,%
No-Change SWT HATSWT Lev. Bagging
0 2 4
·105
0
20
40
60
80
100
Time, instances
Kap
paS
tatis
tic,%
No-Change SWT HATSWT Lev. Bagging
0 2 4
·105
−300
−200
−100
0
100
Time, instances
Kap
paP
lus
Sta
tistic
,%No-Change SWT HATSWT Lev. Bagging
Conclusions
Temporal dependence in data stream mining
I new κ+ measureI a wrapper classifier SWT
Pitfalls in Benchmarking Data StreamClassification and How to Avoid Them
Thanks!
Pitfalls in Benchmarking Data StreamClassification and How to Avoid Them