Statistical Comparison of Two Learning Algorithms

Presented by:

Payam Refaeilzadeh

Overview

How can we tell if one algorithm can learn better than another?– Design an experiment to measure the accuracy of

the two algorithms.– Run multiple trials.– Compare the samples not just their means. Do a

statistically sound test of the two samples.– Is any observed difference significant? Is it due to

true difference between algorithms or natural variation in the measurements?

Statistical Hypothesis Testing

Statistical Hypothesis: A statement about the parameters of one or more populations

Hypothesis Testing: A procedure for deciding to accept or reject the hypothesis– Identify the parameter of interest– State a null hypothesis, H0

– Specify an alternate hypothesis, H1

– Choose a significance level α– State an appropriate test statistic

Statistical Hypothesis Testing Cont

Null Hypothesis (H0): A statement presumed to be true until statistical evidence shows otherwise

Usually specifies an exact value for a parameter Example H0: µ = 30 Kg

Alternate Hypothesis (H1): Accepted if the null hypothesis is rejected

Test Statistic: Particular statistic calculated from measurements of a random sample / experiment

– A test statistic is assumed to follow a particular distribution (normal, t, chi-square, etc)

– That particular distribution can be used to test for the significance of the calculated test statistic.

Error in Hypothesis Testing

Type I error occurs when H0 is rejected but it is in fact true

– P(Type I error)=α or significance level

Type II error occurs when we fail to reject H0 but it is in fact false

– P(Type II error)=β– power = 1-β = Probability of

correctly rejecting H0

– power = ability to distinguish between the two populations

Paired t-Test

Collect data in pairs:– Example: Given a training set DTrain and a test set DTest, train

both learning algorithms on DTrain and then test their accuracies on DTest.

Suppose n paired measurements have been made Assume

– The measurements are independent– The measurements for each algorithm follow a normal

distribution The test statistic T0 will follow a t-distribution with n-1

degrees of freedom

Paired t-Test cont

Trial #

Algorithm 1 Accuracy

X1

Algorithm 2 Accuracy

X2

1 X11 X21

2 X12 X22

… .. …

n X1N X2N

Null Hypothesis:H0: µD = Δ0

Test Statistic:

Assume: X1 follows N(µ1,σ1) X2 follows N(µ2,σ2)Let: µD = µ1 - µ2

Di = X1i - X2i i=1,2,...,n

i

ii XXn

D 21

1

)( 21 iiD XXSTDEVS

DS

nDT 0

0

Rejection Criteria:

H1: µD ≠ Δ0 |t0| > tα/2,n-1

H1: µD > Δ0 t0 > tα,n-1

H1: µD < Δ0 t0 < -tα,n-1

Cross Validated t-test

Paired t-Test on the 10 paired accuracies obtained from 10-fold cross validation

Advantages– Large train set size– Most powerful (Diettrich, 98)

Disadvantages– Accuracy results are not independent

(overlap)– Somewhat elevated probability of

type-1 error (Diettrich, 98)

   

…

 

     

     

     

     

     

     

     

     

     

5x2 Cross Validated t-test

Run 2-fold cross validation 5 times Use results from the first of five replications to

estimate mean difference Use results for all folds to estimate the variance Advantage:

– Lowest Type-1 error (Diettrich, 98)

Disadvantage– Not as powerful as 10 fold cross validated t-test (Diettrich,

98)

Re-sampled t-test

Randomly divide data into train / test sets (usually 2/3 – 1/3)

Run multiple trials (usually 30) Perform a paired t-test between the trial

accuracies This test has very high probability of type-1

error and should never be used.

Calibrated Tests

Bouckaert – ICML 2003:– It is very difficult to estimate the true degrees of

freedom because independence assumptions are being violated

– Instead of correcting for the mean-difference, calibrate on the degrees of freedom

– Recommendation: use 10 times repeated 10-fold cross validation with 10 degrees of freedom

References

R. R. Bouckaert. Choosing between two learning algorithms based on calibrated tests. ICML’03: PP 51-58.

T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10:1895–1924, 1998.

D. C. Montgomery et al. Engineering Statistics. 2nd Edition. Wiley Press. 2001