analysing quantitative data

Web Directions User Experience ‘08

Analysing quantitative data

with Steve BatyUX Strategist

Web Directions User Experience ’08 - Analysing Quantitative Data

Data is important


We expend a lot of effort to gather it


We don’t always use it well


We’ll be looking at:


We’ll be looking at:* time-to-completion



* task completion rates* time-to-completion



* a/b testing* task completion rates* time-to-completion



* page-view data* a/b testing

* task completion rates* time-to-completion


time-to-completion


1 min 24 secs


1 min 23.8 secs


1 min 23.77 secs


1 min 23.768 secs


83.768 secs


User 1 User 2 User...Task 1Task 2Task 3Task 4Task 5Task 6Task 7

83.5 97.3131.1 165.554.5 45.597.8 88.2

118.0 143.3243.9 309.022.9 23.9

Our data might

look like this...


We can calculate...


We can calculate...mean - AVERAGE()variance - VAR()standard dev’n - STDEV()


Low-variability

Medium-variability

High-variability

90.43 s


Compare 2 sets of data- between iterations- between audience segments


Low sample sizes restrict options


non-parametric version == no assumed dist’n


Rank-sum test


Time for a practical demonstration


1 3 3 3 5.5 5.5 7.5 7.5 9 10


1 3 3 3 5.5 5.5 7.5 7.5 9 10

2+3+4=9/3

}5+6

=11/2

}7+8

=15/2

}


1 3

3

3

5.5

5.5

7.5

7.5

10

9

m = 5S1 = 28

S0 = 27n = 5


U0 = nm +n n +1( )2

⎡⎣⎢

⎤⎦⎥− S0

= 5x5 +5 5 +1( )2

⎡⎣⎢

⎤⎦⎥− 27

= 13


U0 = nm +n n +1( )2

⎡⎣⎢

⎤⎦⎥− S0

= 5x5 +5 5 +1( )2

⎡⎣⎢

⎤⎦⎥− 27

= 13

90% --> 3895% --> 4199% --> 45


task completion rates


Only 2 possible values: success or fail


Small samples lead to very broad estimates


4/6 successes = 66.67%

21% - 99.3% with 62.5% most likely


With 30 users47.7% - 81.9% with 64.8% most likely


Most likely =

Range = p ± zp 1− p( )

n

p =s +1n + 2


p ± zp 1− p( )

n


p ± zp 1− p( )

nmost likely


p ± zp 1− p( )

n

confidencelevel


p ± zp 1− p( )

n

variability


A/B Testing

Photo courtesy of www.dorothyphoto.com

http://www.dorothyphoto.com

http://www.dorothyphoto.com


Compare two different approaches to the

same problem


Run both simultaneously;

randomly divert users to option B


Compare using a Chi-squared test


Example: clicks on an ad banner

Ignore Click Total

A

B

10,119 275 10,394

962 38 1,000

Total 11,081 313 11,394


χ 2 =eij − oij( )2eij

∑The test statistic is a measure of distance

between what we expect to see (e), and what we actually observed (o). For each cell, subtract what we expect from what we saw, square it to remove any negative values, and divide it by

the expected value. Add it all together...


Calculated expected valuesFor each cell:

row total x column total/grand total


Ignore Click Total

A

B

10,108 = 10,394x(11,081/11,394)

286 = 10,394x(313/11,394) 10,394

973 = 1,000x(11,081/11,394)

27 = 1,000x(313/11,394) 1,000

Total 11,081 313 11,394


Ignore Click Total

A

B

10,108 - 10,119 = -11 286 - 275 = 11 10,394

973 - 962 = 11 27 - 38 = -11 1,000

Total 11,081 313 11,394


χ 2 =eij − oij( )2eij

∑

=112

10,108+112

286+112

973+112

27= 0.012 + 0.423+ 0.124 + 4.48= 5.04


χα =0.0252 = 5.02 < χ 2

χα =0.012 = 6.63 > χ 2


page viewspre- & post comparison


Can be cyclical


Can be trending


Typically compare the average


But ignores fluctuation


But ignores fluctuation

?


z =x1 − x2( )s12

n1+ s2

2

n2Test 1 : x1, s1

2 ,n1Test 2 : x2 , s2

2 ,n2


z =x1 − x2( )s12

n1+ s2

2

n2Test 1 : x1, s1

2 ,n1Test 2 : x2 , s2

2 ,n2

In order: mean, variance &

number of data points in each

test.


z =x1 − x2( )s12

n1+ s2

2

n2Test 1 : x1, s1

2 ,n1Test 2 : x2 , s2

2 ,n2



test.

Mean difference


z =x1 − x2( )s12

n1+ s2

2

n2Test 1 : x1, s1

2 ,n1Test 2 : x2 , s2

2 ,n2



test.

Mean difference

Combined standard error


z =x1 − x2( )s12

n1+ s2

2

n2Test 1 : x1, s1

2 ,n1Test 2 : x2 , s2

2 ,n2If z < -1.96 or > 1.96 a significance difference exists



test.

Mean difference

Combined standard error


xs2ni

Pre Post

1,288 1,331

1,369 756.25

60 30


z =x1 − x2( )s12

n1+ s2

2

n2

=1288 −1331( )136960

+ 756.2530

=436.93

= 6.205


1,288 1,331


Read more...Statistics without tears by Derek Rowntree

Flaws & Fallacies in statistical thinking by Stephen K Campbell

http://uxstats.blogspot.com


Thank you

analysing quantitative data

Technology

data task

sets of data

p range

ab testing photo courtesy

low sample

variability highvariability90

ijeijthe test statistic

small samples