2008 chingchun 1 bootstrap chingchun huang ( 黃敬群 ) vision lab, nctu
Post on 21-Dec-2015
227 views
TRANSCRIPT
12008 Chingchun
Bootstrap
Chingchun Huang (黃敬群 )
Vision Lab, NCTU
22008 Chingchun
Introduction
A data-based simulation method For statistical inference
– finding estimators of the parameter in interest – Confidence of the parameter in interest
32008 Chingchun
An example• Two statistics definition for a random variable
• Average: sample mean • Standard error: The standard deviation of the sample means
• Calculation of two statistics • Carry out measurement “many” times
• Observations from these two statistics • Standard error decreases as N increases• Sample mean becomes more reliable as N increases
42008 Chingchun
Central limit theoremCentral limit theorem
• Averages taken from Averages taken from any any distribution distribution (your experimental data) will have a normal (your experimental data) will have a normal distributiondistribution• The error for such an statistic will The error for such an statistic will decrease slowly as the number of decrease slowly as the number of observations increaseobservations increase
52008 Chingchun
Normal distributionNormal distribution Averages of N.D.Averages of N.D.
distributiondistribution Averages of Averages of distribution distribution
62008 Chingchun
Uniform distributionUniform distribution Averages of U.D.Averages of U.D.
72008 Chingchun
Consequences of central limit theoremConsequences of central limit theorem
Bootstrap --- the technique to the rescueBootstrap --- the technique to the rescue
• But nobody tells you how big the sample has to But nobody tells you how big the sample has to be..be..• Should we believe a measurement of Should we believe a measurement of “Average”?“Average”? •How about other objects rather than “Average”How about other objects rather than “Average”
82008 Chingchun
Basic idea of bootstrap• Originally, from some list of data, one
computes an object (e.g.: statistic).• Create an artificial list by randomly drawing
elements from that list. Some elements will be picked more than once. • Nonparametric mode (later)• Parametric mode (later)
• Compute a new object.• Repeat 100-1000 times and look at the
distribution of these objects.
92008 Chingchun
A simple example• Data available comparing grades before and after
leaving graduate school
• Some linear correlation between grades =0.776
• But how reliable is this result (=0.776)?
102008 Chingchun
A simple example
112008 Chingchun
122008 Chingchun
132008 Chingchun
A simple example
142008 Chingchun
Confidence intervalsConsider the similar situation as before The parameter of interest is (e.g. Mean) is an estimator of based on the sample . We are interested in finding the confidence
interval for the parameter.
x
152008 Chingchun
The percentile algorithmInput the level=2 for the confidence interval.Generate B number of bootstrap samples.Compute for b = 1,…, BArrange the new data set with ‘s in order.Compute and percentile for the
new data.C.I. is given by ( th , th )Percentile 5% 10% 16% 50% 84% 90% 95%
Percentile 49.7 56.4 62.7 86.9 112.3 118.7 126.7
of
*b
*b
(1 )
(1 )
*b
*b
*b
162008 Chingchun
How many bootstraps ?• No clear answer to this.
• Rule of thumb : try it 100 times, then 1000 times, and see if your answers have changed by much.
172008 Chingchun
How many bootstraps ?
B 50 100 200 500 1000 2000 3000
Std.
Error
0.0869 0.0804 0.0790 0.0745 0.0759 0.0756 0.0755
182008 Chingchun
Convergence
This histogram is showing the distribution of the correlation coefficient for the bootstrap sample . Here B=200, B=500
0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.750
5
10
15
20
25
30
35
40
45
0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.750
20
40
60
80
100
120
192008 Chingchun
Contd…
B=1000, B=2000
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80
50
100
150
200
250
300
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
100
200
300
400
500
600
202008 Chingchun
Contd…..
B=3000 B=4000
Now it can be seen the sampling distributions of correlation coefficient are more or less identical.
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
100
200
300
400
500
600
700
800
900
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
200
400
600
800
1000
1200
212008 Chingchun
Contd…..
The above graph is showing the similarity in the distribution of the bootstrap distribution and the direct enumeration from random samples from the empirical distribution
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
200
400
600
800
1000
1200
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
200
400
600
800
1000
1200
222008 Chingchun
Is it reliable ?• Observations
• Good agreement for Normal (Gaussian) distributions• Skewed distributions tend to more problematic,
particularly for the tails• A tip: For now nobody is going to shoot you down
for using it.
232008 Chingchun
Schematic representation of bootstrap procedure
1( , , )nx x x
*1x*2x *Bx
*1s*2s *Bs
*s
242008 Chingchun
Bootstrap
The bootstrap can be used either non-parametrically or parametrically
In nonparametric mode, it avoids restrictive and sometimes dangerous parametric assumptions about the form of the underlying population .
In parametric mode it can provide more accurate estimates of errors than traditional methods.
252008 Chingchun
Parametric Bootstrap
Real World
Statistic of interest
Bootstrap World
Estimated Bootstrapprobability sample model
Bootstrap Replication
unknown
probability
Model
1 2( , ,... )nP x x x x
Observed data
ˆ ( )s x
* * * *1 2
ˆ ( , ,... )nP x x x x
* *ˆ ( )s x
(distribution) P x (samples)
262008 Chingchun
Bootstrap
The technique was extended, modified and
refined to handle a wide variety of problems
including:– (1) confidence intervals and hypothesis tests, – (2) linear and nonlinear regression, – (3) time series analysis and other problems
2626
272008 Chingchun
• Fit a cubic spline: (N=50 training data)
7
1
T
j jj
μ x E Y X x β h x h x β
Example: one-dimensional smoothing
TY h x β n 2~ (0, )n N
282008 Chingchun
• Least squares
yHHH TT 1ˆ βxhxμ T ˆˆ where
21ˆ)ˆr(av
HH T
ijij xhH
where
N
i
iN
xy
1
22 ˆˆ
1
1 2ˆˆ ˆ ˆ( )T T Tse x std h x β h x H H h x
xesx ˆˆ96.1ˆ
The bootstrap and maximum likelihood method
292008 Chingchun
• Nonparametric bootstrap
Repeat B=200 times:
- draw a dataset of N=50 with replacement from the training data zi=(xi,yi)
- fit a cubic spline Construct a 95% pointwise confidence interval:
At each xi compute the mean and find the 2,5% and 97,5% percentiles
The bootstrap and maximum likelihood method
302008 Chingchun
• Parametric bootstrap
We assume that the model errors are Gaussian:
Repeat B=200 times:
- draw a dataset of N=50 with replacement from the training data zi=(xi,yi)
- fit a cubic spline on zi : and estimate
- simulate new responses : zi*=(xi,yi
*)
- fit a cubic spline on zi*:
Construct a 95 pointwise confidence interval:
At each xi compute the mean and find the 2,5% and 97,5% percentiles
20,σε~N;εXμY
2ˆ0ˆ σ,~Nε;εxμy *i
*ii
*i
ixμ
*1*ˆ iTTT
i yHHHxhx
21ˆˆˆ σxhHHx,hxμ~Nxμ TT*
2σ
The bootstrap and maximum likelihood method
312008 Chingchun
• Parametric bootstrap
Conclusion: least squares = parametric bootstrap as B
(only because of Gaussian errors)
The bootstrap and maximum likelihood method
322008 Chingchun
Some notations
The Bootstrap is – A computer-based method for assigning measures
of accuracy to statistical estimates. – The basic idea behind bootstrap is very simple, and
goes back at least two centuries.– The bootstrap method is not a way of reducing the
error ! It only tries to estimate it.– Bootstrap methods depend only on the Bootstrap
samples. It does not depend on the underlying distribution.
332008 Chingchun
A general data set-upWe have dealt with
– The standard error – The confidence interval – With the assumption that distribution is either
unknown or very complicated.
The situation can be more general – Like regression , – Sometimes using maximum likelihood estimation.
342008 Chingchun
ConclusionThe bootstrap allow the data analyst to
– Asses the statistical accuracy of complicated procedures, by exploiting the power of the computer.
The use of the bootstrap either– Relief the analyst from having to do complex
mathematical derivation or– Provide an answer where no analytical answer can be
obtained.
352008 Chingchun
Addendum : The Jack-knife• Jack-knife is a special kind of bootstrap.
• Each bootstrap subsample has all but one of the original elements of the list.
• For example, if original list has 10 elements, then there are 10 jack-knife subsamples.
362008 Chingchun
Introduction (continued) Definition of Efron’s nonparametric bootstrap. Given a sample of n independent identically
distributed (i.i.d.) observations X1, X2, …, Xn from a distribution F and a parameter of the distribution F with a real valued estimator
(X1, X2, …, Xn ), the bootstrap estimates the
accuracy of the estimator by replacing F with Fn,
the empirical distribution, where Fn places
probability mass 1/n at each observation Xi.
3636
372008 Chingchun
Introduction (continued)Let X1
*, X2*, …, Xn
* be a bootstrap sample, that is a sample of size n taken with replacement from Fn .
The bootstrap, estimates the variance of
(X1, X2, …, Xn ) by computing or approximating the variance of
* = (X1*, X2
*, …, Xn* ).
3737
382008 Chingchun
Introduction (continued)The bootstrap is similar to earlier techniques
which are also called resampling methods:– (1) jackknife, – (2) cross-validation, – (3) delta method, – (4) permutation methods, and – (5) subsampling..
3838
392008 Chingchun
Bootstrap RemediesIn the past decade many of the problems where
the bootstrap is inconsistent remedies have been found by researchers to give good modified bootstrap solutions that are consistent.
For both problems describe thus far a simple procedure called the m-out-n bootstrap has been shown to lead to consistent estimates .
3939
402008 Chingchun
The m-out-of-n Bootstrap This idea was proposed by Bickel and Ren (1996) for handling
doubly censored data. Instead of sampling n times with replacement from a sample of
size n they suggest to do it only m times where m is much less than n.
To get the consistency results both m and n need to get large but at different rates. We need m=o(n). That is m/n→0 as m and n both → ∞.
This method leads to consistent bootstrap estimates in many cases where the ordinary bootstrap has problems, particularly (1) mean with infinite variance and (2) extreme value distributions.
4040
Don’t know why.
412008 Chingchun
Examples where the bootstrap fails Athreya (1987) shows that the bootstrap
estimate of the sample mean is inconsistent when the population distribution has an infinite variance.
Angus (1993) provides similar inconsistency results for the maximum and minimum of a sequence of independent identically distributed observations.
4141