1 nonparametric methods iii henry horng-shing lu institute of statistics national chiao tung...
TRANSCRIPT
1
Nonparametric Methods III
Henry Horng-Shing LuInstitute of Statistics
National Chiao Tung [email protected]
http://tigpbp.iis.sinica.edu.tw/courses.htm
2
PART 4: Bootstrap and Permutation Tests Introduction References Bootstrap Tests Permutation Tests Cross-validation Bootstrap Regression ANOVA
3
References Efron, B.; Tibshirani, R. (1993). An Introduction
to the Bootstrap. Chapman & Hall/CRC. http://cran.r-project.org/doc/contrib/Fox-Co
mpanion/appendix-bootstrapping.pdf http://cran.r-project.org/bin/macosx/2.1/chec
k/bootstrap-check.ex http://bcs.whfreeman.com/ips5e/content/cat
_080/pdf/moore14.pdf
4
Hypothesis Testing (1) A statistical hypothesis test is a method of m
aking statistical decisions from and about experimental data.
Null-hypothesis testing just answers the question of “how well the findings fit the possibility that chance factors alone might be responsible.”
This is done by asking and answering a hypothetical question.
http://en.wikipedia.org/wiki/Statistical_hypothesis_testing
5
Hypothesis Testing (2) Hypothesis testing is largely the product of Ronald Fisher,
Jerzy Neyman, Karl Pearson and (son) Egon Pearson. Fisher was an agricultural statistician who emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an (extended) hybrid of the Fisher vs. Neyman/Pearson formulation, methods and terminology developed in the early 20th century.
6
Hypothesis Testing (3)
7
Hypothesis Testing (4)
8
Hypothesis Testing (5)
9
Hypothesis Testing (7) Parametric Tests:
Nonparametric Tests: Bootstrap Tests Permutation Tests
10
Confidence Intervals vs. Hypothesis Testing (1)
Interval estimation ("Confidence Intervals") and point estimation ("Hypothesis Testing") are two different ways of expressing the same information.
http://www.une.edu.au/WebStat/unit_materials/c5_inferential_statistics/confidence_interv_hypo.html
11
Confidence Intervals vs. Hypothesis Testing (2)
If the exact p-value is reported, then the relationship between confidence intervals and hypothesis testing is very close. However, the objective of the two methods is different: Hypothesis testing relates to a single
conclusion of statistical significance vs. no statistical significance.
Confidence intervals provide a range of plausible values for your population.
http://www.nedarc.org/nedarc/analyzingData/advancedStatistics/convidenceVsHypothesis.html
12
Confidence Intervals vs. Hypothesis Testing (3)
Which one? Use hypothesis testing when you want to do a
strict comparison with a pre-specified hypothesis and significance level.
Use confidence intervals to describe the magnitude of an effect (e.g., mean difference, odds ratio, etc.) or when you want to describe a single sample.
http://www.nedarc.org/nedarc/analyzingData/advancedStatistics/convidenceVsHypothesis.html
13
P-value
http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf
14
Achieved Significance Level (ASL)
*0 0
0
Definition:
The (ASL) is defined as:
ˆ ˆASL = ( | H ).
The smaller ASL, the stronger is the evidence of false.
Th
achieved significance
e ASL is an estimate o
l
f
e
the p-value by
vel o
perm
f the test
P
H
uation and bootstrap methods.
0
Definition
A is a way of deciding whether or not the data decisively
reject the h
hypoth
ypothe
esis
sis
test
.
H
https://www.cs.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf
15
Bootstrap Tests Methodology Flowchart R code
16
Bootstrap Tests Beran (1988) showed that bootstrap inference
is refined when the quantity bootstrapped is asymptotically pivotal.
It is often used as a robust alternative to inference based on parametric assumptions.
http://socserv.mcmaster.ca/jfox/Books/Companion/appendix-bootstrapping.pdf
17
Hypothesis Testing by a Pivot
2 1
2
1
:
- 1. : (0, 1), ( , ) , , .
-2. : = (0, 1) , ( , ),
( ,
n
ii
i
Di
n
ii
Examples
XX
A pivot Z N when X iid N and X is knownn
n
XAn asymptotic pivot T N as n when X iid N
Sn
Xwhere X is unknown and S
n
2
1
).
1
n
ii
X X
n
http://en.wikipedia.org/wiki/Pivotal_quantity
Pivot or pivotal quantity: a function of observations whose distribution does
not depend on unknown parameters.
18
T statistics can be regarded as a pivot or an asymptotic pivotal when the data are normally distributed.
Bootstrap T tests can be applied when the data are not normally distributed.
One Sample Bootstrap Tests
19
Bootstrap T tests Flowchart R code
20
·0
1 2 0
ˆˆ ( , , ..., ) ( ),
ˆ( )ndata x x x x x s x and t
Bootstrap B times*Bx*2x*1x
*Bt
*2t
*1t
· *0#{ }/Boot bASL t t B
Flowchart of Bootstrap T Tests
** 0
*
ˆ
ˆ( )
bb
b
t
21
Bootstrap T Tests by R
22
An Example of Bootstrap T Tests by R
23
Bootstrap Tests by The “BCa” The BCa percentile method is an efficient met
hod to generate bootstrap confidence intervals.
There is a correspondence between confidence intervals and hypothesis testing.
So, we can use the BCa percentile method to test whether H0 is true.
Example: use BCa to calculate p-value
24
Use R package “boot.ci(boot)” Use R package “bcanon(bootstrap)” http://qualopt.eivd.ch/stats/?page=bootstrap http://www.stata.com/capabilities/boot.html
BCa Confidence Intervals:
25
http://finzi.psych.upenn.edu/R/library/boot/DESCRIPTION
26
An Example of “boot.ci(boot)” in R
27
http://finzi.psych.upenn.edu/R/library/bootstrap/DESCRIPTION
28
An example of “bcanon(bootstrap)” in R
29
BCa by http://qualopt.eivd.ch/stats/?page=bootstrap
30
Use BCa to calculate p-value by R
31
Two Sample Bootstrap Tests Flowchart R code
32
Bootstrap B times* * *1 1 1( , )d y x
1 2 1: ( , y , ..., y )nSample yy
* * *1 1 1ˆ ( ) ( )s s y x
· *ˆ ˆ (#( )) /Boot bASL B
Flowchart of Two-Sample Bootstrap Tests
1 2 2: ( , x , ..., x )mSample xx
1 2 1ˆ : ( , , ..., , , ..., ) ( , ) ( ) ( )n n n mcombined data d d d d d s s d y x y x
* * *2 2 2( , )d y x
* * *2 2 2
ˆ ( ) ( )s s y x * * *ˆ ( ) ( )B B Bs s y x
m+n=Ncombine
* * *( , )B B Bd y x
33
Two-Sample Bootstrap Tests by R
34
Output (1)
35
Output (2)
36
Permutation Tests Methodology Flowchart R code
37
Permutation In several fields of mathematics, the term
permutation is used with different but closely related meanings. They all relate to the notion of (re-)arranging elements from a given finite set into a sequence.
http://en.wikipedia.org/wiki/Permutation
38
Permutation Tests Permutation test is also called a
randomization test, re-randomization test, or an exact test.
If the labels are exchangeable under the null hypothesis, then the resulting tests yield exact significance levels.
Confidence intervals can then be derived from the tests.
The theory has evolved from the works of R.A. Fisher and E.J.G. Pitman in the 1930s.
http://en.wikipedia.org/wiki/Pitman_permutation_test
39
Applications of Permutation Tests (1)
We can use a permutation test only when we can see how to resample in a way that is consistent with the study design and with the null hypothesis.
http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf
40
Two-sample problems when the null hypothesis says that the two populations are identical. We may wish to compare population means, proportions, standard deviations, or other statistics.
Matched pairs designs when the null hypothesis says that there are only random differences within pairs. A variety of comparisons is again possible.
Relationships between two quantitative variables when the null hypothesis says that the variables are not related. The correlation is the most common measure of association, but not the only one.
http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf
Applications of Permutation Tests (2)
41
2
2a b
0
A tradionnal way is to consider some hypotheses: ~ ( , )
and ~ ( , ), and the null hypothesis becomes = .
ˆUnder , the statistic . = - can be modelled as a normal
distribution with mean
a
b
a b
F N
F N
H X X
* 2
2
2 2ˆ
ˆ ˆ( )
2*
ˆˆ
2 2
2 1 1
1 1 0 and variance ( ).
The ASL is then computed by
ˆ ASL=2
when is unknown and has to be estimated from the data by
( ) ( ).
2We w
n m
ai a bi bi i
m n
ed
X X X X
m n
0ill reject if ASL > .H
Inference by Permutation Tests
https://www.cs.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf
42
Flowchart of The Permutation Test for Mean Shift in One Sample
1 2 1 2 , , ..., , , , ..., n n n n mSample x x x x x x
*11x
*21x
Partition 2 subset B times
(treatment group)
(control group) (treatment group)
(control group)
* * *1 2
ˆ ( ) ( )b b bs s x x
11G 12G
1x 2x11O 12O
1 2ˆ ( ) ( )s s x x
· *ˆ ˆ (#( )) / , and NPerm b nASL B B C
*12x
*22x
21G 22G
*1Bx
*2Bx
1BG 2BG
n m N
43
An Example for One Sample Permutation Test by R
http://mason.gmu.edu/~csutton/EandTCh15a.txt
44
45
An Example of Output Results
46
1 2 2: ( , x , ..., x )mSample xx
*1x
*1y
Partition subset B times
treatment
subgroup
control
subgroup
11G 12G
1 2 1: ( , y , ..., y )nSample yy
m+n=N
1 2 1ˆ : ( , , ..., , , ..., ) ( , ) ( ) ( )n n n mcombined data d d d d d s s d y x y x
combine
* * *ˆ ( ) ( )b b bs s x y
· *ˆ ˆ (#( )) / , and NPerm b nASL B B C
Flowchart of The Permutation Test for Mean Shift in Two Samples
*2x
*2y
21G 22G
*Bx
*By
1BG 2BG
treatment
subgroup
control
subgroup
47
Bootstrap Tests vs. Permutation Tests Very similar results between the
permutation test and the bootstrap test. is the exact probability when . is not an exact probability but is
guaranteed to be accurate as an estimate of the ASL, as the sample size B goes to infinity.
PermASL
BootASL
https://www.cs.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf
NnB C
48
Cross-validation Methodology R code
49
Cross-validation Cross-validation, sometimes called rotation es
timation, is the statistical practice of partitioning a sample of data into subsets such that the analysis is initially performed on a single subset, while the other subset(s) are retained for subsequent use in confirming and validating the initial analysis. The initial subset of data is called the training set. the other subset(s) are called validation or testing s
ets.
http://en.wikipedia.org/wiki/Cross-validation
50
Overfitting Problems In statistics, overfitting is fitting a statistical model that has too
many parameters. When the degrees of freedom in parameter selection exceed the i
nformation content of the data, this leads to arbitrariness in the final (fitted) model parameters which reduces or destroys the ability of the model to generalize beyond the fitting data.
The concept of overfitting is important also in machine learning. In both statistics and machine learning, in order to avoid overfitti
ng, it is necessary to use additional techniques (e.g. cross-validation, early stopping, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization.
http://en.wikipedia.org/wiki/Overfitting
51
library(bootstrap)
?crossval
52
An Example of Cross-validation by R
53
output
54
Bootstrap Regression Bootstrapping pairs:
Resample from the sample pairs { }. Bootstrapping residuals:
1. Fit by the original sample and obtain the residuals.2. Resample from residuals.
, i ix y
ˆi iy x
55
Bootstrapping Pairs by R
http://www.stat.uiuc.edu/~babailey/stat328/lab7.html
56
Output
57
Bootstrapping Residuals by R
http://www.stat.uiuc.edu/~babailey/stat328/lab7.html
58
Bootstrapping residuals
59
ANOVA When random errors follow a normal
distribution: When random errors do not follow a
Normal distribution: Bootstrap tests:Permutation tests:
60
An Example of ANOVA by R (1) Example
Twenty lambs are randomly assigned to three different diets. The weight gain (in two weeks) is recorded. Is there a difference among the diets?
Reference http://mcs.une.edu.au/~stat261/Bootstrap/
bootstrap.R
61
An Example of ANOVA by R (1)
62
An Example of ANOVA by R (2)
63
An Example of ANOVA by R (3)
64
Output (1)
65
Output (2)
66
Output (3)
67
Output (4)
68
Output (5)
69
Output (6)
70
Output (7)
71
The Second Example of ANOVA by R (1)
Data source http://finzi.psych.upenn.edu/R/library/rpart/html/kyp
hosis.html Reference
http://www.stat.umn.edu/geyer/5601/examp/parm.html Kyphosis is a misalignment of the spine. The data are on 8
3 laminectomy (a surgical procedure involving the spine) patients. The predictor variables are age and age^2 (that is, a quadratic function of age), number of vertebrae involved in the surgery and start the vertebra number of the first vertebra involved. The response is presence or absence of kyphosis after the surgery (and perhaps caused by it).
72
The Second Example of ANOVA by R (2)
73
The Second Example of ANOVA by R (3)
74
The Second Example of ANOVA by R (4)
75
Output (1)
Data = kyphosis
76
Output (2)
77
Output (3)
78
Output (4)
79
Output (5)
#deviance
#p-value
80
Output (6)
81
Exercises: Write your own programs similar to those
examples presented in this talk.
Write programs for those examples mentioned at the reference web pages.
Write programs for the other examples that you know.
Practice Makes Perfect!81