sequential experimental designs for sensitivity experiments nist glm conference april 18-20, 2002...
TRANSCRIPT
![Page 1: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/1.jpg)
Sequential Experimental Designs For Sensitivity Experiments
NIST GLM ConferenceApril 18-20, 2002
Joseph G. VoelkelCenter for Quality and Applied StatisticsCollege of EngineeringRochester Institute of Technology
![Page 2: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/2.jpg)
2
CQAS
Sensitivity Experiments ASTM method D 1709–91 Impact resistance of plastic film by free-falling
dart method
Film
D art Pass
Fail
![Page 3: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/3.jpg)
3
CQAS
Objectives
Engineer Specify a probability of
failure – 0.50, 0.10, … Find dart weight x= such
that Prob(F; )=
Statistician Find a strategy for
selecting weights {xi} so that is estimated as precisely as possible
0.1
X
p
Darts are dropped one at a time. Weight of ith dart may depend on results obtained up to date
![Page 4: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/4.jpg)
4
CQAS
Data Collection PossibilitiesNon-sequential Specify n and all the {xi} before any Pass-Fail data {Yi} are
obtained. Find dose of drug at which 5% of mice develop tumors
Group-sequential Example: two-stage. Specify n1 and the {x1i}. Obtain data {Y1i} Use this info to specify n2 and the {x2i}. Obtain data {Y2i}
Same mice example, but with more time.
(Fully) Sequential Use all prior knowledge: x1Y1 x2Y2 x3Y3 x4Y4
Dart-weight example. One machine, one run at a time.
![Page 5: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/5.jpg)
5
CQAS
Model and Objectives
Objective: Example Estimate weight at
which 10% of the samples fail
0.1
X
p
1 ( )
1 (1 exp( / ) )
x xP Y p F x
x
x
ln( /(1 ))
logit(p )x xx p p
logit( )
ln(0.10 / 0.90)
logit(0.10)
= 2.2
ˆvar( ) So, try to set the {xi} to
minimize
![Page 6: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/6.jpg)
6
CQAS
Our Interest
(Fully) Sequential experiments Estimating a corresponding to a given , e.g. 0.10. The real problem. = 0.50? = 0.001?
![Page 7: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/7.jpg)
7
CQAS
A Quick Tour of Some Past Work
Up-Down Method. Dixon and Mood (1948)
Only for =0.50 Robbins-Munro (1951)
wanted {xi} to converge to .
Like Up-Down, but with decreasing increments
far from 0.50 convergence is too slow
151050
7
6
5
Run
Set
ting
151050
7
6
5
Run
Set
ting
Pass
Fail
![Page 8: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/8.jpg)
8
CQAS
A Quick Tour of Some Past Work
Wu’s (1985) Sequential-Solving Method Similar in spirit to the R-M procedure Collect some initial data to get estimates of and Choose the next setting, xn1 , to solve
( )F xn n 1 . So xn n 1
Better than R-M, much better than Up-and-Down Performance depends somewhat heavily on initial runs Asymptotically optimal, in a certain sense
![Page 9: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/9.jpg)
9
CQAS
Some Non-Sequential Bayesian Results
Tsutakawa (1980) How to create design for estimation of for a given . Certain priors on and Some approximations Assumed constant number of runs made at equally
spaced settings. Chaloner and Larntz (1989)
Includes how to create design for estimation of for a given
Some reasonable approximations used Not restricted to constant number of runs or equally
spaced settings.
![Page 10: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/10.jpg)
10
CQAS
Examples of Optimal Designs for estimating when =0.50.
(Based on Chaloner and Larntz)
~ , , .Unif 1 1 14a f ~ , , .Unif 10 10 14a f
100-10x
-10 0 10x
![Page 11: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/11.jpg)
11
CQAS
Examples of Optimal Designs for estimating when =0.05.
(Based on Chaloner and Larntz)
~ , , .Unif 1 1 14a f ~ , , .Unif 10 10 14a f
100-10x
100-10x
![Page 12: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/12.jpg)
12
CQAS
This Talk. Bayesian Sequential Design
A way to specify priors Measures of what we are learning about , ,
and —AII and Information Specifying the next setting, with some insights Some examples and comparisons Rethinking the priors
![Page 13: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/13.jpg)
13
CQAS
Specifying Priors Consider the related tolerance-distribution problem The r.v. Xi represents the (unobservable) speed at which
the ith sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)
1
1 (1 exp( / ) )
xP Y P X x
x
![Page 14: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/14.jpg)
14
CQAS
Specifying Priors
Two-parameter distribution Could specify priors on
(, ) (, ) (, )
For simplicity, want to assume independence so only need to specify marginals of each parameter
, , not ,s X s X z s
0.50
(, )
![Page 15: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/15.jpg)
15
CQAS
Specifying Priors
Instead of (,) … Consider =0.10
example Consider
distance
from to = 2.2
1050
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
x
c5/.7
logit(0.10)= 2.2
Easier for engineer to understand
(, )
![Page 16: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/16.jpg)
16
CQAS
Specifying Priors
Ask engineer for Best guess and 95% range for
5.0 ± 3.0 Best guess and 95% range for – distance
6.6 / 2.0 Translate –=2.2 into terms: 3.0 / 2.0 Translate into normal, independent, priors on
and ln() We used a discrete set of 1515=225 values as
prior distribution of (,) (, )
![Page 17: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/17.jpg)
17
CQAS
Specifying Priors
More natural for engineer to think about priors on and . We let engineer do this as follows.
We created 27 combinations of prior distributions: best guess—10 uncertainty (95% limits)— ± 2, 4, 6. best guess— 1, 3, 5 uncertainty (95% limits)— / 2, 4, 6.
We graphed these in terms of (,)
(, )
![Page 18: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/18.jpg)
18
CQAS
Example of Prior Distributions of =10± 4 /
10 4
= 1 = 3 = 5 2
15105 100-10 100-10 100-10 4
15105 100-10 100-10 100-10 6
15105 100-10 100-10 100-10
![Page 19: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/19.jpg)
19
CQAS
Finding the next setting xn+1 to runAssume we have collected data 1,..., nY Y based on
settings 1,..., nx x (This includes the case of not having any data, for which n=0.)
B a s e d o n t h i s , u p d a t e t h e o r i g i n a l p r i o r s o n a n d ( a n d h e n c e ) t o f o r m 1D i s t , | , . . . , D i s t ,n nY Y ,
o u r c u r r e n t p r i o r o n a n d . We would like to select a new setting 1nx so that if we run
there and get the result xY, we want Varn xY , the (new)
posterior variance of , to be as small as possible. However, we don’t know xY, so can’t find Varn xY . So find 1nx to minimize result on average, or EVarnnxY, where expectation is wrt xY. The optimal setting.
![Page 20: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/20.jpg)
20
CQAS
AII Measure
Before we make the ( )n1th run, current prior variance of
, denoted by 1Var | ,..., =Varn nY Y , is determined. 1Varn (total) information about after n runs. E q u iva le n t w a y to fin d o p tim a l x is th e x th a t m a x im ize s
1 1
A IIV arE (V ar ) nn n x
xY
AII = anticipated increase in information. Plotting AIIxaf versus x gives a good indication for how sensitive our results are to the choice of the next x
![Page 21: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/21.jpg)
21
CQAS
Simple Example
Objective: find the corresponding to =0.10
Prob
8 0.25
9 0.50
10 0.25
Prob
1 0.25
2 0.50
3 0.25
Pr8 1 5.8 .0625
8 2 3.6 .1250
8 3 1.4 .0625
9 1 6.8 .1250
9 2 4.6 .2500
9 3 2.4 .1250
10 1 7.8 .0625
10 2 5.6 .1250
10 3 3.4 .0625
0E 5.8 0.0625 ... 3.4 0.0625 4.6
![Page 22: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/22.jpg)
22
CQAS
Simple Example
![Page 23: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/23.jpg)
23
CQAS
Simple Example
Finding the AII for various x settings
20100
0.02
0.01
0.00
x
AII
![Page 24: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/24.jpg)
24
CQAS
How AII “Thinks”
20100
0.02
0.01
0.00
x
AII
![Page 25: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/25.jpg)
25
CQAS
First Simulation (,)=(8,1.82). Makes =4.0
403020100
6
5
4
3
2
Run
Set
ting,
E(
P assFailSettingX
Setting increment = 1
![Page 26: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/26.jpg)
26
CQAS
Example with a More Diffuse Prior
=10 ± 4, =5 / 6
Simulation againdone with =8,=4
100-10
![Page 27: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/27.jpg)
27
CQAS
Example with a More Diffuse Prior
6050403020100
7
6
5
4
3
2
1
0
Run
Set
ting,
E(
P assFailSettingX
403020100
6
5
4
3
2
Run
Set
ting,
E(
![Page 28: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/28.jpg)
28
CQAS
Behavior of AII after 0, 2, 10, 20, 60 runs
20100
0.010
0.005
0.000
Setting
AII
0
2
10
20
60
![Page 29: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/29.jpg)
29
CQAS
Information on , , = 2.2
6050403020100
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Run
Info
rmat
ion
on
A serious problem—all the information on was obtained through The simulation trusted the relative tight prior on … Another problem: more
objective methods of estimation, such as MLE, will likely not work wellAre there other ways to specify priors that might be better? Two methods…
![Page 30: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/30.jpg)
30
CQAS
Equal-Contribution Priors For =2.2, restrict original prior so that
Var0()=Varo(2.2) Results of another simulation
0 10 20 30 40 50 60
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Run
Info
rmat
ion
on
6050403020100
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
Run
Info
rmat
ion
on
Problem: fails for case =: Var0()=Varo(0)?
![Page 31: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/31.jpg)
31
CQAS
Relative Priors Consider the tolerance-distribution problem The r.v. Xi represents the (unobservable) speed at which
the ith sample of film would have failed. Say from a location-scale family (e.g., logistic, normal, …)
![Page 32: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/32.jpg)
32
CQAS
Relative Priors
We observe only the ( x, Yx ) ’s If we could observe the X ’s, the problem would
be a simple one-sample problem of finding the 100 percentile of a distribution.
Assume the distribution of the X ’s has a finite fourth moment.
![Page 33: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/33.jpg)
33
CQAS
Relative Priors
Using delta-method to find Var(s) and m-1m
22sd sd
4X m s
m
So, to a good approximation
1sd sds X k
2 2 4 22Var Var
1X m s
m m
After m runs, observing X1, X2, …, Xm, we have
![Page 34: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/34.jpg)
34
CQAS
Relative Priors
So, with k1 and k2 known,
1 2ˆ ˆsd = sdk k
So, in this sense it is defensible to specify only the prior precision with which is know, and base the prior precision of upon it.
2ˆ ˆˆ ˆ= , , corr , =0 X k s
Now assume tolerance distribution is symmetric and its shape is know, e.g. logistic. Then
![Page 35: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/35.jpg)
35
CQAS
Logistic Example
6050403020100
1.0
0.5
0.0
Run
Info
rmat
ion
on
![Page 36: Sequential Experimental Designs For Sensitivity Experiments NIST GLM Conference April 18-20, 2002 Joseph G. Voelkel Center for Quality and Applied Statistics](https://reader030.vdocuments.site/reader030/viewer/2022032606/56649eab5503460f94bb14ad/html5/thumbnails/36.jpg)
36
CQAS
Summary
AII as a useful measure of value of making next run at x. Combination of shift in posterior mean & probability that a failure will occur at x
Informal comparison to non-Bayesian methodsBayesian x-strategy is more subtle
Danger of simply using any prior, and recommended way to set priors