lecture 16 nonlinear problems: simulated annealing and bootstrap confidence intervals
DESCRIPTION
Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals . Syllabus. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/1.jpg)
Lecture 16
Nonlinear Problems:
Simulated Annealing and Bootstrap Confidence Intervals
![Page 2: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/2.jpg)
SyllabusLecture 01 Describing Inverse ProblemsLecture 02 Probability and Measurement Error, Part 1Lecture 03 Probability and Measurement Error, Part 2 Lecture 04 The L2 Norm and Simple Least SquaresLecture 05 A Priori Information and Weighted Least SquaredLecture 06 Resolution and Generalized InversesLecture 07 Backus-Gilbert Inverse and the Trade Off of Resolution and VarianceLecture 08 The Principle of Maximum LikelihoodLecture 09 Inexact TheoriesLecture 10 Nonuniqueness and Localized AveragesLecture 11 Vector Spaces and Singular Value DecompositionLecture 12 Equality and Inequality ConstraintsLecture 13 L1 , L∞ Norm Problems and Linear ProgrammingLecture 14 Nonlinear Problems: Grid and Monte Carlo Searches Lecture 15 Nonlinear Problems: Newton’s Method Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals Lecture 17 Factor AnalysisLecture 18 Varimax Factors, Empircal Orthogonal FunctionsLecture 19 Backus-Gilbert Theory for Continuous Problems; Radon’s ProblemLecture 20 Linear Operators and Their AdjointsLecture 21 Fréchet DerivativesLecture 22 Exemplary Inverse Problems, incl. Filter DesignLecture 23 Exemplary Inverse Problems, incl. Earthquake LocationLecture 24 Exemplary Inverse Problems, incl. Vibrational Problems
![Page 3: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/3.jpg)
Purpose of the Lecture
Introduce Simulated Annealing
Introduce the Bootstrap Methodfor computing Confidence Intervals
![Page 4: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/4.jpg)
Part 1
Simulated Annealing
![Page 5: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/5.jpg)
Monte Carlo Methodcompletely undirected
Newton’s Methodcompletely directed
![Page 6: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/6.jpg)
Monte Carlo Methodcompletely undirected
Newton’s Methodcompletely directed
slow, but foolproof
fast, but can fall into local minimum
![Page 7: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/7.jpg)
compromise
partially-directed random walk
![Page 8: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/8.jpg)
m(p)
Ehigh EmediumElow
![Page 9: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/9.jpg)
m(p)
Ehigh EmediumElowp(m*|m(p))
![Page 10: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/10.jpg)
m(p)
Ehigh EmediumElow
m*
![Page 11: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/11.jpg)
acceptance of m* as m(p+1) always accept in error is smaller
accept with probability
where T is a parameterif error is bigger
![Page 12: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/12.jpg)
large T 1always accept m*
(undirected random walk)ignores the error completely
![Page 13: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/13.jpg)
small T 0accept m* only when error is smaller
(directed random walk)strictly decreases the error
![Page 14: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/14.jpg)
intermediate Tmost iterations decrease the error
but occasionally allow an m*that increases it
![Page 15: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/15.jpg)
m(p)
Ehigh EmediumElow
large Tundirected random
walk
![Page 16: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/16.jpg)
m(p)
Ehigh EmediumElow
small Tdirected random
walk
![Page 17: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/17.jpg)
strategy
start off with large T
slowly decrease T during iterations
undirectedsimilar to Monte Carlo method
(except more “local”)
directedsimilar to Newton’s method
(except precise gradient direction not used)
![Page 18: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/18.jpg)
strategy
start off with large T
slowly decrease T during iterations
claim is that this strategy helps achieve the global minimum
more random
more directed
![Page 19: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/19.jpg)
analogous to annealing of metals
high temperaturesatoms randomly moving
about due to thermal motions
as temperature decreasesatoms slowly find themselves in a
minimum energy configurationorderly arrangement of a “crystal”
www.sti-laser.com/technology/heat_treatments.html
![Page 20: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/20.jpg)
analogous to annealing of metals
high temperaturesatoms randomly moving
about due to thermal motions
as temperature decreasesatoms slowly find themselves in a
minimum energy configurationorderly arrangement of a “crystal”
hence “simulated annealing”and T called “temperature”
![Page 21: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/21.jpg)
this is just Metroplois-Hastings
(way of producing realizations of a random variable)
applied to the p.d.f.
![Page 22: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/22.jpg)
this is just Metroplois-Hastings
(way of producing realizations of a random variable)
applied to the p.d.f.
sampling a distribution that starts out wide and blurrybut sharpens up as T is decreases
![Page 23: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/23.jpg)
0 100 200 300 4000
100
200
iteration
E
0 100 200 300 4001
2
3
iteration
m1
0 100 200 300 4000
1
2
iteration
m2
0 100 200 300 4000
10
20
iteration
T
0 100 200 300 4000
0.5
1
iteration
p 1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
2
4
x
d
0 0.5 1 1.5 2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
m2
m1
20
40
60
80
100
120
140
160
180
200
220
(A)
(B)
(C)
![Page 24: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/24.jpg)
for k = [1:Niter] T = 0.1 * Eg0 * ((Niter-k+1)/Niter)^2; ma(1) = random('Normal',mg(1),Dm); ma(2) = random('Normal',mg(2),Dm); da = sin(w0*ma(1)*x) + ma(1)*ma(2); Ea = (dobs-da)'*(dobs-da); if( Ea < Eg ) mg=ma; Eg=Ea; p1his(k+1)=1; else p1 = exp( -(Ea-Eg)/T ); p2 = random('unif',0,1); if( p1 > p2 ) mg=ma; Eg=Ea; end end end
![Page 25: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/25.jpg)
Part 2
Bootstrap Method
![Page 26: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/26.jpg)
theory of confidence intervalserror is the data
result inerrors in the estimated model parameters
p(d) ddobsp(m) mmest
m(d)
![Page 27: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/27.jpg)
theory of confidence intervalserror is the data
result inerrors in the estimated model parameters
p(d) d p(m) m95% confidence
m(d)2½% 2½% 2½% 2½%95% confidence
![Page 28: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/28.jpg)
Gaussian linear theoryd = Gmm = G-gdstandard error propagation[cov m]=G-g [cov d] G-gT
univariate Gaussian distribution has95% of error within two σ of its mean
![Page 29: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/29.jpg)
What to do withGaussian nonlinear theory?
One possibilitylinearize theory and use standard error
propagation
d = g(m) m-m(p) ≈ G(p) –g [d- g(m(p)) ][cov m]≈G(p)-g [cov d] G(p)-g
![Page 30: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/30.jpg)
disadvantagesunknown accuracy
andneed to compute gradient of theory G(p)
G(p) not computed when using some solution methods
![Page 31: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/31.jpg)
alternativeconfidence intervals with
repeat datasets
do the whole experiment many times
use results of each experiment to make compute mest
create histograms from many mest’s
derive empirical 95% confidence intervalsfrom histograms
![Page 32: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/32.jpg)
Bootstrap Method
create approximate repeat datasetsby randomly resampling (with duplications)
the one existing data set
![Page 33: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/33.jpg)
example of resampling
1.42.13.83.11.51.7
123456
313251
3.81.43.82.11.51.4
123456
original data set
random integers in range 1-6
resampled data set
![Page 34: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/34.jpg)
example of resampling
1.42.13.83.11.51.7
123456
313251
3.81.43.82.11.51.4
123456
original data set
random integers in range 1-6
new data set
![Page 35: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/35.jpg)
example of resampling
1.42.13.83.11.51.7
123456
313251
3.81.43.82.11.51.4
123456
original data set
random integers in range 1-6
resampled data set
note repeats
![Page 36: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/36.jpg)
rowindex = unidrnd(N,N,1);xresampled = x( rowindex );dresampled = dobs( rowindex );
![Page 37: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/37.jpg)
p(d) p’(d)
sampling
duplication
mixing
interpretation of resampling
![Page 38: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/38.jpg)
0 0.5 1 1.5 2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
m2
m1
50
100
150
200
1.19 1.195 1.2 1.205 1.21 1.2150
20
40
60
80
1.35 1.4 1.45 1.5 1.55 1.60
2
4
6
8
(A) (B)
m1
m2
m1
m2p(m 2
)p(m 1
)
(C)
![Page 39: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/39.jpg)
Nbins=50;m1hmin=min(m1save);m1hmax=max(m1save);Dm1bins = (m1hmax-m1hmin)/(Nbins-1);m1bins=m1hmin+Dm1bins*[0:Nbins-1]';m1hist = hist(m1save,m1bins);pm1 = m1hist/(Dm1bins*sum(m1hist));Pm1 = Dm1bins*cumsum(pm1);m1low=m1bins(find(Pm1>0.025,1));m1high=m1bins(find(Pm1>0.975,1));
![Page 40: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/40.jpg)
Nbins=50;m1hmin=min(m1save);m1hmax=max(m1save);Dm1bins = (m1hmax-m1hmin)/(Nbins-1);m1bins=m1hmin+Dm1bins*[0:Nbins-1]';m1hist = hist(m1save,m1bins);pm1 = m1hist/(Dm1bins*sum(m1hist));Pm1 = Dm1bins*cumsum(pm1);m1low=m1bins(find(Pm1>0.025,1));m1high=m1bins(find(Pm1>0.975,1));
histogram
![Page 41: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/41.jpg)
Nbins=50;m1hmin=min(m1save);m1hmax=max(m1save);Dm1bins = (m1hmax-m1hmin)/(Nbins-1);m1bins=m1hmin+Dm1bins*[0:Nbins-1]';m1hist = hist(m1save,m1bins);pm1 = m1hist/(Dm1bins*sum(m1hist));Pm1 = Dm1bins*cumsum(pm1);m1low=m1bins(find(Pm1>0.025,1));m1high=m1bins(find(Pm1>0.975,1));
empirical p.d.f.
![Page 42: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/42.jpg)
Nbins=50;m1hmin=min(m1save);m1hmax=max(m1save);Dm1bins = (m1hmax-m1hmin)/(Nbins-1);m1bins=m1hmin+Dm1bins*[0:Nbins-1]';m1hist = hist(m1save,m1bins);pm1 = m1hist/(Dm1bins*sum(m1hist));Pm1 = Dm1bins*cumsum(pm1);m1low=m1bins(find(Pm1>0.025,1));m1high=m1bins(find(Pm1>0.975,1));
empirical c.d.f.
![Page 43: Lecture 16 Nonlinear Problems: Simulated Annealing and Bootstrap Confidence Intervals](https://reader035.vdocuments.site/reader035/viewer/2022062218/56816735550346895ddbe211/html5/thumbnails/43.jpg)
Nbins=50;m1hmin=min(m1save);m1hmax=max(m1save);Dm1bins = (m1hmax-m1hmin)/(Nbins-1);m1bins=m1hmin+Dm1bins*[0:Nbins-1]';m1hist = hist(m1save,m1bins);pm1 = m1hist/(Dm1bins*sum(m1hist));Pm1 = Dm1bins*cumsum(pm1);m1low=m1bins(find(Pm1>0.025,1));m1high=m1bins(find(Pm1>0.975,1)); 95%
confidence bounds