statistics for programmers

8/19/2019 Statistics for Programmers

1/129

Jake VanderPlas

Jake VanderPlas @jakevdp

Sept 17, 2015


2/129

Jake VanderPlas

About Me Jake VanderPlas (Twitter/Github: jakevdp)

- Astronomer by training

- Data Scientist at UW eScience Institute

- Active in Python science & open source- Blog at Pythonic Perambulations

- Author of two books:


3/129


4/129

Jake VanderPlas

Statistics is Hard.

Using programming skills,it can be easy.


5/129


6/129

Jake VanderPlas

My thesis today:

If you can write a for-loop, you can do statistics


7/129Jake VanderPlas

You toss a coin 30

times and see 22 heads. Is it a fair coin?

Warm-up:

Coin Toss



A fair coin shouldshow 15 heads in 30

tosses. This coin isbiased.

Even a fair coincould show 22 heads

in 30 tosses. It mightbe just chance.



Classic Method:

Assume the Skeptic is correct:test the Null Hypothesis.

Assuming a fair coin, compute

probability of seeing 22 headssimply by chance.



Classic Method:

Start computing probabilities . . .


11/129



Classic Method:

Number of

arrangements(binomialcoefficient) Probability of

N H heads

Probability of N

T tails



Classic Method:



Classic Method:

0.8 %



Classic Method:

0.8 %

Probability of 0.8% (i.e. p = 0.008) of

observations given a fair coin. → reject fair coin hypothesis at p < 0.05



Could there be

an easier way?



Easier Method:

Just simulate it!

M = 0

for i in range(10000):

trials = randint(2, size=30)

if (trials.sum() >= 22):

M += 1

p = M / 10000 # 0.008149

→ reject fair coin at p = 0.008


19/129

Jake VanderPlas

In general . . .

Computing the SamplingDistribution is Hard.


20/129

Jake VanderPlas

In general . . .

Computing the SamplingDistribution is Hard.

Simulating the Sampling

Distribution is Easy.


21/129

Jake VanderPlas

Four Recipes for

Hacking Statistics:1. Direct Simulation

2. Shuffling3. Bootstrapping

4. Cross Validation


22/129

Jake VanderPlas

Now, the Star-Belly Sneetches

had bellies with stars.The Plain-Belly Sneetches had none upon thars . . .

Sneeches:Stars and

Intelligence

*adapted from John Rauser’sStatistics Without All The Agonizing Pain


23/129

Jake VanderPlas

★ ❌

84 72 81 69

57 46 74 61

63 76 56 87

99 91 69 65

66 44

62 69

★ mean: 73.5❌ mean: 66.9

difference: 6.6

Sneeches:Stars and

Intelligence

Test Scores


24/129

Jake VanderPlas

★ mean: 73.5❌ mean: 66.9 difference: 6.6

Is this difference of 6.6statistically significant?


25/129


26/129


27/129

Jake VanderPlas

Classic

Method

(Student’s t distribution)


28/129

Jake VanderPlas

Classic

Method


Degree of Freedom: “The number of independentways by which a dynamic system can move,without violating any constraint imposed on it.”

-Wikipedia


29/129

Jake VanderPlas

Degree of Freedom: “The number of independentways by which a dynamic system can move,without violating any constraint imposed on it.”

-Wikipedia

Classic

Method


https://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equation


30/129

Jake VanderPlas

Classic

Method

( Welch–Satterthwaiteequation)

https://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equationhttps://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equationhttps://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equation


31/129

Jake VanderPlas

Classic

Method

( Welch–Satterthwaiteequation)

https://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equationhttps://en.wikipedia.org/wiki/Welch%E2%80%93Satterthwaite_equation


32/129


33/129

Jake VanderPlas

Classic

Method


34/129

Jake VanderPlas

Classic

Method

1.7959


35/129


36/129


37/129

Jake VanderPlas

Classic

Method


38/129

Jake VanderPlas

“The difference of 6.6 is not

significant at the p=0.05 level”


39/129

Jake VanderPlas


40/129


41/129

Jake VanderPlas

Let’s use a sampling

method instead


42/129

Jake VanderPlas

Problem:

Unlike coin flipping, we don’t have a probabilistic model . . .


43/129

Jake VanderPlas

Problem:

Unlike coin flipping, we don’t have a probabilistic model . . .

Solution:

Shuffling

★ d


44/129

Jake VanderPlas

★ ❌

84 72 81 69

57 46 74 61

63 76 56 87

99 91 69 6566 44

62 69

Idea:Simulate the distributionby shuffling the labelsrepeatedly and computingthe desired statistic.

Motivation:if the labels really don’tmatter, then switchingthem shouldn’t changethe result!

★ Sh ffl L b l


45/129

Jake VanderPlas

★ ❌

84 72 81 69

57 46 74 61

63 76 56 87

99 91 69 6566 44

62 69

1. Shuffle Labels2. Rearrange3. Compute means

★ Sh ffl L b l


46/129

Jake VanderPlas

★ ❌

84 72 81 69

57 46 74 61

63 76 56 87

99 91 69 6566 44

62 69


★ Sh ffl L b l


47/129

Jake VanderPlas

★ ❌

84 81 72 69

61 69 74 57

65 76 56 87

99 44 46 6366 91

62 69


★ Sh ffl L b l


48/129

Jake VanderPlas

★ ❌

84 81 72 69

61 69 74 57

65 76 56 87

99 44 46 6366 91

62 69

★ mean: 72.4❌ mean: 67.6difference: 4.8


★ Sh ffl L b l


49/129

Jake VanderPlas

★ ❌

84 81 72 69

61 69 74 57

65 76 56 87

99 44 46 6366 91

62 69



★ ❌ Sh ffl L b l


50/129

Jake VanderPlas

★ ❌

84 81 72 69

61 69 74 57

65 76 56 87

99 44 46 6366 91

62 69


★ ❌ 1 Sh ffl L b l


51/129

Jake VanderPlas

★ ❌

84 56 72 69

61 63 74 57

65 66 81 87

62 44 46 6976 91

99 69

★ mean: 62.6❌ mean: 74.1difference: -11.6


★ ❌ 1 Sh ffl L b l


52/129

Jake VanderPlas

★ ❌

84 56 72 69

61 63 74 57

65 66 81 87

62 44 46 6976 91

99 69


★ ❌ 1 Shuffle Labels


53/129

Jake VanderPlas

★ ❌

74 56 72 69

61 63 84 57

87 76 81 65

91 99 46 6966 62

44 69





54/129

Jake VanderPlas

★ ❌

84 56 72 69

61 63 74 57

65 66 81 87

62 44 46 6976 91

99 69




55/129

Jake VanderPlas

★ ❌

84 81 69 69

61 69 87 74

65 76 56 57

99 44 46 6366 91

62 72



56/129

1 Shuffle Labels★ ❌


57/129

Jake VanderPlas


★ ❌

84 81 72 69

61 69 74 57

65 76 56 87

99 44 46 6366 91

62 69


58/129

Jake VanderPlas

score difference

n u m b

e r


59/129

Jake VanderPlas

score difference

n u m b

e r


60/129

Jake VanderPlas

16 %

score difference

n u m b

e r


61/129

Jake VanderPlas

“A difference of 6.6 is notsignificant at p = 0.05.”

That day, all the Sneetches

forgot about stars

And whether they had one,

or not, upon thars.

N t Sh ffli


62/129

Jake VanderPlas

Notes on Shuffling:

- Works when the Null Hypothesis assumes two groups are equivalent

- Like all methods, it will only work if your

samples are representative – always becareful about selection biases!

- For more discussion & references, seeStatistics is Easy by Shasha & Wilson


63/129

Jake VanderPlas

Four Recipes for



4. Cross Validation


64/129

How High can Yertle


65/129

Jake VanderPlas

How High can Yertlestack his turtles?

- What is the mean of the number of

turtles in Yertle’s stack?- What is the uncertainty on this

estimate?

48 24 32 61 51 12 32 18 19 24

21 41 29 21 25 23 42 18 23 13

Observe 20 of Yertle’s turtle towers . . .

# o f t u r t l e s

Classic Method:


66/129

Jake VanderPlas

Classic Method:

Sample Mean:

Standard Error of the Mean:


67/129

Jake VanderPlas

What assumptions go intothese formulae?

Can we usesampling instead?


68/129

Jake VanderPlas

Problem:

We need a way to simulatesamples, but we don’t have a

generating model . . .


69/129

Jake VanderPlas

Problem:

We need a way to simulatesamples, but we don’t have a

generating model . . .

Solution:Bootstrap Resampling

Bootstrap Resampling:


70/129

Jake VanderPlas

p p g

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18

Idea:Simulate the distributionby drawing samples withreplacment.

Motivation:The data estimates itsown distribution – wedraw random samplesfrom this distribution.



71/129

Jake VanderPlas

p p g

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18





72/129

Jake VanderPlas

p p g

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21



73/129

Jake VanderPlas

p p g

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19



74/129

Jake VanderPlas

p p g

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25



75/129

Jake VanderPlas

g

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24



76/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23



77/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19



78/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41



79/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41 23


80/129



81/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41 23 41 18


82/129



83/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12



84/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42



85/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42



86/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 13

32 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42 42



87/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 1332 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42 42 19


88/129



89/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 1332 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42 42 19 18 61



90/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 1332 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42 42 19 18 61 29



91/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 1332 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42 42 19 18 61 29 41



92/129

Jake VanderPlas

48 24 51 12

21 41 25 23

32 61 19 24

29 21 23 1332 18 42 18



21 19 25 24 23 19 41 23 41 18

61 12 42 42 42 19 18 61 29 41→ 31.05


93/129

Jake VanderPlas

Repeat this

several thousand times . . .

Recovers The Analytic Estimate!


94/129

Jake VanderPlas


sample = N[randint(20, size=20)]

xbar[i] = mean(sample)mean(xbar), std(xbar)

# (28.9, 2.9)

Height = 29 ± 3 turtles


95/129

Jake VanderPlas

Bootstrap samplingcan be applied even to

more involved statistics


96/129

Bootstrap on LinearR i


97/129

Jake VanderPlas

Regression:


i = randint(20, size=20)

slope, intercept = fit(x[i], y[i])

results[i] = (slope, intercept)

Notes on Bootstrapping:


98/129

Jake VanderPlas

pp g

- Bootstrap resampling rests on sound

theoretical grounds.

- Bootstrapping doesn’t work well for rank-based statistics (e.g. maximum value)

- Works poorly with very few samples(N > 20 is a good rule of thumb)

- Always be careful about selection biases!

Four Recipes for


99/129

Jake VanderPlas

Four Recipes for



4. Cross Validation

Onceler Industries:


100/129

Jake VanderPlas

Sales of Thneeds

I'm being quite useful! This thing is a Thneed.

A Thneed's a Fine-Something-

That-All-People-Need!


101/129

Jake VanderPlas

Thneed sales seem to show atrend with temperature . . .


102/129

Jake VanderPlas

y = a + b x

y = a + b x + c x2

But which model is a better fit?


103/129

Jake VanderPlas

y = a + b x

y = a + b x + c x2

Can we judge by root-mean-square error?

RMS error = 63.0

RMS error = 51.5

In general, more flexible models will


104/129

Jake VanderPlas

always have a lower RMS error.

y = a + b x

y = a + b x + c x2

y = a + b x + c x2 + d x3

y = a + b x + c x2 + d x3 + e x4

y = a + ⋯

RMS error does not


105/129

Jake VanderPlas

y = a + b x + c x2 + d x3 + e x4 + f x5 + ⋯ + n x14

tell the whole story.


106/129

Jake VanderPlas

Not to worry:

Statistics has figured this out.

Classic Method


107/129

Jake VanderPlas

Difference in Mean

Squared Error follows

chi-square distribution:

Classic Method


108/129

Jake VanderPlas

Difference in Mean



Can estimate degrees of

freedom easily because the

models are nested . . .

Classic Method


109/129

Jake VanderPlas

Difference in Mean-

Squared-Error follows





Now plug in all our numbers and . . .

Classic Method


110/129

Jake VanderPlas

Difference in Mean






Now plug in all our numbers and . . .


111/129

Jake VanderPlas

Easier Way:

Cross Validation

Cross-Validation


112/129

Jake VanderPlas

Cross-Validation


113/129

Jake VanderPlas

1. Randomly Split data

Cross-Validation


114/129

Jake VanderPlas

1. Randomly Split data

Cross-Validation


115/129

Jake VanderPlas

2. Find the best model for each subset

Cross-Validation


116/129

Jake VanderPlas

3. Compare models across subsets


117/129

Cross-Validation


118/129

Jake VanderPlas


Cross-Validation


119/129

Jake VanderPlas


Cross-Validation


120/129

Jake VanderPlas

4. Compute RMS error for each

RMS = 48.9RMS = 55.1

RMS estimate = 52.1

Cross-Validation


121/129

Jake VanderPlas

5. Compare cross-validated RMS for models:

Cross-Validation


122/129

Jake VanderPlas

Best model minimizes thecross-validated error.

5. Compare cross-validated RMS for models:


123/129

Jake VanderPlas

. . . I biggered the loads of the thneeds I shipped out! I was shipping them forth,to the South, to the East to the West, to the North!

Notes on Cross-Validation:


124/129

Jake VanderPlas

- This was “2-fold” cross-validation; other

CV schemes exist & may perform betterfor your data (see e.g. scikit-learn docs)

- Cross-validation is the go-to method formodel evaluation in machine learning,as statistics of the models are often notknown in the classical sense.

Four Recipes for


125/129

Jake VanderPlas

p



4. Cross Validation

Sampling Methods


126/129

Jake VanderPlas

Sampling Methods

allow you to replacenon-intuitive statistical rules withintuitive computational approaches!

If you can write a for-loop

you can do statistical analysis.

Things I didn’t have time for:


127/129

Jake VanderPlas

- Bayesian Methods: very intuitive & powerful

approaches to more sophisticated modeling.(see Bayesian Methods for Hackers by Cam Davidson-Pillon)

- Selection Bias: if you get data selection

wrong, you’ll have a bad time.(See Chris Fonnesbeck’s Scipy 2015 talk, Statistical Thinking for Data Science )

- Detailed considerations on use of sampling,shuffling, and bootstrapping.(I recommend Statistics Is Easy , a book by Shasha & Wilson)


128/129

~ Thank You! ~


129/129

Email: [email protected]

Twitter: @jakevdp

Github: jakevdp

Web: http://vanderplas.com/

Blog: http://jakevdp.github.io/

Slides available ath k d k j k d i i f h k
https://speakerdeck.com/jakevdp/statistics-for-hackers/https://speakerdeck.com/jakevdp/statistics-for-hackers/

statistics for programmers

Documents