a brief introduction to doe

7/30/2019 A Brief Introduction to DoE

1/9

J. K. TELFORD

JOhns hOpKins ApL TEchnicAL DigEsT, VOLumE 27, numbER 3 (2007)224

DA Bief Idui Deig f Expeime

Jacqueline K. Telford

esign o experiments is a series o tests in which purposeul

changes are made to the input variables o a system or pro-

cess and the eects on response variables are measured. Design o

experiments is applicable to both physical processes and computer simulation models.

Experimental design is an eective tool or maximizing the amount o inormation

gained rom a study while minimizing the amount o data to be collected. Factorial

experimental designs investigate the eects o many dierent actors by varying them

simultaneously instead o changing only one actor at a time. Factorial designs allowestimation o the sensitivity to each actor and also to the combined eect o two or

more actors. Experimental design methods have been successully applied to several

Ballistic Missile Deense sensitivity studies to maximize the amount o inormation

with a minimum number o computer simulation runs. In a highly competitive world

o testing and evaluation, an efcient method or testing many actors is needed.

BACKGROUND

Would you like to be sure that you will be able todraw valid and denitive conclusions from your datawith the minimum use of resources? If so, you shouldbe using design of experiments. Design of experiments,also called experimental design, is a structured and orga-nized way of conducting and analyzing controlled teststo evaluate the factors that are affecting a response vari-able. The design of experiments species the particularsetting levels of the combinations of factors at which

the individual runs in the experiment are to be con-ducted. This multivariable testing method varies thefactors simultaneously. Because the factors are variedindependently of each other, a causal predictive modelcan be determined. Data obtained from observationalstudies or other data not collected in accordance with adesign of experiments approach can only establish cor-relation, not causality. There are also problems with thetraditional experimental method of changing one factor


2/9

A bRiEF inTRODucTiOn TO DEsign OF EXpERimEnTs

JOhns hOpKins ApL TEchnicAL DigEsT, VOLumE 27, numbER 3 (2007) 225

at a time, i.e., its inefciency and its inability to deter-mine effects that are caused by several factors acting incombination.

BRIEF HISTORY

Design of experiments was invented by Ronald A.Fisher in the 1920s and 1930s at Rothamsted Experi-mental Station, an agricultural research station 25

miles north of London. In Fishers rst book on designof experiments1 he showed how valid conclusions couldbe drawn efciently from experiments with natural uc-tuations such as temperature, soil conditions, and rainfall, that is, in the presence of nuisance variables. Theknown nuisance variables usually cause systematic biasesin groups of results (e.g., batch-to-batch variation). Theunknown nuisance variables usually cause random vari-ability in the results and are called inherent variabilityor noise. Although the experimental design methodwas rst used in an agricultural context, the methodhas been applied successfully in the military and inindustry since the 1940s. Besse Day, working at the U.S.Naval Experimentation Laboratory, used experimentaldesign to solve problems such as nding the cause of badwelds at a naval shipyard during World War II. GeorgeBox, employed by Imperial Chemical Industries beforecoming to the United States, is a leading developer ofexperimental design procedures for optimizing chemicalprocesses. W. Edwards Deming taught statistical meth-ods, including experimental design, to Japanese scien-tists and engineers in the early 1950s2 at a time whenMade in Japan meant poor quality. Genichi Taguchi,the most well known of this group of Japanese scien-tists, is famous for his quality improvement methods.

One of the companies where Taguchi rst applied hismethods was Toyota. Since the late 1970s, U.S. indus-try has become interested again in quality improvementinitiatives, now known as Total Quality and SixSigma programs. Design of experiments is consideredan advanced method in the Six Sigma programs, whichwere pioneered at Motorola and GE.

FUNDAMENTAL PRINCIPLES

The fundamental principles in design of experi-ments are solutions to the problems in experimentationposed by the two types of nuisance factors and serveto improve the efciency of experiments. Those funda-mental principles are

Randomization Replication Blocking Orthogonality Factorial experimentation

Randomization is a method that protects against anunknown bias distorting the results of the experiment.

An example of a bias is instrument drift in an experimentcomparing a baseline procedure to a new procedure. Ifall the tests using the baseline procedure are conductedrst and then all the tests using the new procedure areconducted, the observed difference between the proce-dures might be entirely due to instrument drift. To guardagainst erroneous conclusions, the testing sequence ofthe baseline and new procedures should be in random

order such asB

,N

,N

,B

,N

,B

,and so on. The instru-ment drift or any unknown bias should average out.Replication increases the sample size and is a method

for increasing the precision of the experiment. Replica-tion increases the signal-to-noise ratio when the noiseoriginates from uncontrollable nuisance variables. Areplicate is a complete repetition of the same experi-mental conditions, beginning with the initial setup. Aspecial design called a Split Plot can be used if some ofthe factors are hard to vary.

Blocking is a method for increasing precision byremoving the effect of known nuisance factors. Anexample of a known nuisance factor is batch-to-batch

variability. In a blocked design, both the baseline andnew procedures are applied to samples of material fromone batch, then to samples from another batch, andso on. The difference between the new and baselineprocedures is not inuenced by the batch-to-batchdifferences. Blocking is a restriction of complete ran-domization, since both procedures are always appliedto each batch. Blocking increases precision since thebatch-to-batch variability is removed from the experi-mental error.

Orthogonality in an experiment results in the factoreffects being uncorrelated and therefore more easilyinterpreted. The factors in an orthogonal experiment

design are varied independently of each other. Themain results of data collected using this design can oftenbe summarized by taking differences of averages and canbe shown graphically by using simple plots of suitablychosen sets of averages. In these days of powerful com-puters and software, orthogonality is no longer a neces-sity, but it is still a desirable property because of the easeof explaining results.

Factorial experimentation is a method in which theeffects due to each factor and to combinations of fac-tors are estimated. Factorial designs are geometricallyconstructed and vary all the factors simultaneously andorthogonally. Factorial designs collect data at the ver-tices of a cube inp-dimensions (p is the number of fac-tors being studied). If data are collected from all of thevertices, the design is a full factorial, requiring 2p runs.Since the total number of combinations increases expo-nentially with the number of factors studied, fractionsof the full factorial design can be constructed. As thenumber of factors increases, the fractions become smallerand smaller (1/2,

1/4,1/8,

1/16, ). Fractional factorialdesigns collect data from a specic subset of all possible


3/9

J. K. TELFORD


vertices and require 2p2qruns, with 22q being the frac-tional size of the design. If there are only three factors inthe experiment, the geometry of the experimental designfor a full factorial experiment requires eight runs, and aone-half fractional factorial experiment (an inscribedtetrahedron) requires four runs (Fig. 1).

Factorial designs, including fractional factorials, haveincreased precision over other types of designs because

they have built-in internal replication. Factor effects areessentially the difference between the average of all runsat the two levels for a factor, such as high and low.Replicates of the same points are not needed in a facto-rial design, which seems like a violation of the replicationprinciple in design of experiments. However, half of allthe data points are taken at the high level and the otherhalf are taken at the low level of each factor, resultingin a very large number of replicates. Replication is alsoprovided by the factors included in the design that turnout to have nonsignicant effects. Because each factoris varied with respect to all of the factors, informationon all factors is collected by each run. In fact, every data

point is used in the analysis many times as well as in theestimation of every effect and interaction. Additionalefciency of the two-level factorial design comes fromthe fact that it spans the factor space, that is, puts half ofthe design points at each end of the range, which is themost powerful way of determining whether a factor hasa signicant effect.

USES

The main uses of design of experiments are

Discovering interactions among factors

Screening many factors Establishing and maintaining quality control Optimizing a process, including evolutionary opera-

tions (EVOP) Designing robust products

Interaction occurs when the effect on the responseof a change in the level of one factor from low to highdepends on the level of another factor. In other words,when an interaction is present between two factors, thecombined effect of those two factors on the responsevariable cannot be predicted from the separate effects.The effect of two factors acting in combination caneither be greater (synergy) or less (interference) thanwould be expected from each factor separately.

Frequently there is a need to evaluate a processwith many input variables and with measured outputvariables. This process could be a complex computersimulation model or a manufacturing process with rawmaterials, temperature, and pressure as the inputs. Ascreening experiment tells us which input variables (fac-tors) are causing the majority of the variability in theoutput (responses), i.e., which factors are the drivers. A

screening experiment usually involves only two levels ofeach factor and can also be called characterization test-ing or sensitivity analysis.

A process is out of statistical control when eitherthe mean or the variability is outside its specications.

When this happens, the cause must be found and cor-rected. The cause is found efciently using an experi-mental design similar to the screening design, exceptthat the number of levels for the factors need not be twofor all the factors.

Optimizing a process involves determining the shapeof the response variable. Usually a screening design isperformed rst to nd the relatively few important fac-tors. A response surface design has several (usually threeor four) levels on each of the factors. This produces amore detailed picture of the surface, especially provid-ing information on which factors have curvature and onareas in the response where peaks and plateaus occur.

The EVOP method is an optimization procedure usedwhen only small changes in the factors can be toleratedin order for normal operations to continue. Examples ofEVOP are optimizing the cracking process on crude oilwhile still running the oil renery or tuning the weldingpower of a welding robot in a car manufacturing assem-bly line.

Product robustness, pioneered by Taguchi, usesexperimental design to study the response surfaces asso-ciated with both the product means and variances tochoose appropriate factor settings so that variance andbias are both small simultaneously. Designing a robustproduct means learning how to make the response vari-able insensitive to uncontrollable manufacturing processvariability or to the use conditions of the product by thecustomer.

MATHEMATICAL FORMULATION AND TERMINOLOGY

The input variables on the experiment are calledfactors. The performance measures resulting from theexperiment are called responses. Polynomial equations

Figue 1. Full actorial and one-hal actorial in three

dimensions.


4/9



are Taylor series approximations to the unknown true functional form of theresponse variable. An often quoted insight of George Box is, All modelsare wrong. Some are useful.3 The trick is to have the simplest model thatcaptures the main features of the data or process. The polynomial equation,shown to the third order in Eq. 1, used to model the response variable Yas afunction of the input factors Xs is

Y= 0+

iX

i+

ijj = 1

p

Xi Xji = 1

i j

p

+ ijkk = 1

p

j = 1i j k

p

i = 1

p

Xi Xj Xk +L ,i = 1

p

(1)

where

0 = the overall mean response,

i = the main effect for factor (i = 1, 2, ... ,p),

ij = the two-way interaction between the ith andjth factors, and

ijk = the three-way interaction between the ith,jth, and kth factors.

Usually, two values (called levels) of the Xs are used in the experiment foreach factor, denoted by high and low and coded as 11 and 21, respectively.A general recommendation for setting the factor ranges is to set the levelsfar enough apart so that one would expect to see a difference in the responsebut not so far apart as to be out of the likely operating range. The use of onlytwo levels seems to imply that the effects must be linear, but the assumptionof monotonicity (or nearly so) on the response variable is sufcient. At leastthree levels of the factors would be required to detect curvature.

Interaction is present when the effect of a factor on the response variabledepends on the setting level of another factor. Graphically, this can be seenas two nonparallel lines when plotting the averages from the four combina-tions of high and low levels of the two factors. The ij terms in Eq. 1 accountfor the two-way interactions. Two-way interactions can be thought of as thecorrections to a model of simple additivity of the factor effects, the modelwith only the iterms in Eq. 1. The use of the simple additive model assumesthat the factors act separately and independently on the response variable,

which is not a very reasonable assumption.Experimental designs can be categorized by their resolution level. Adesign with a higher resolution level can t higher-order terms in Eq. 1 thana design with a lower resolution level. If a high enough resolution level designis not used, only the linear combination of several terms can be estimated,not the terms separately. The word resolution was borrowed from the termused in optics. Resolution levels are usually denoted by Roman numerals,with III, IV, and V being the most commonly used. To resolve all of thetwo-way interactions, the resolution level must be at least V. Four resolutionlevels and their meanings are given in Table 1.

IMPLEMENTATION

The main steps to implementan experimental design are as fol-lows. Note that the subject matterexperts are the main contributors tothe most important steps, i.e., 14,10, and 12.

1. State the objective of thestudyand the hypotheses to betested.

2. Determine the response vari-able(s) of interest that can bemeasured.

3. Determine the controllable fac-tors of interest that might affectthe response variables and thelevels of each factor to be usedin the experiment. It is betterto include more factors in thedesign than to exclude factors,

that is, prejudging them to benonsignicant.

4. Determine the uncontrollablevariables that might affect theresponse variables, blockingthe known nuisance variablesand randomizing the runs toprotect against unknown nui-sance variables.

5. Determine the total numberof runs in the experiment, ide-ally using estimates of variabil-ity, precision required, size of

effects expected, etc., but morelikely based on available timeand resources. Reserve someresources for unforeseen con-tingencies and follow-up runs.Some practitioners recommendusing only 25% of the resourcesin the rst experiment.

6. Design the experiment, remem-bering to randomize the runs.

7. Perform a pro forma analy-sis with response variables asrandom variables to checkfor estimability of the factoreffects and precision of theexperiment.

8. Perform the experiment strictlyaccording to the experimen-tal design, including the initialsetup for each run in a physicalexperiment. Do not swap the runorder to make the job easier.

Table 1. Resolution levels and their meanings.

Resolution level MeaningII Main effects are linearly combined with each other (bi1bj).

III Main effects are linearly combined with two-way interactions

(bi1bjk).

IV Main effects are linearly combined with three-way interac-

tions (bi1bjkl) and two-way interactions with each other

(bij1bkl).

V Main effects and two-way interactions are not linearly com-

bined except with higher-order interactions (bi1bjklm andbij1bklm).


5/9

J. K. TELFORD


9. Analyze the data from the experiment using theanalysis of variance method developed by Fisher.

10. Interpret the results and state the conclusions interms of the subject matter.

11. Consider performing a second, conrmatory experi-ment if the conclusions are very important or arelikely to be controversial.

12. Document and summarize the results and conclu-

sions, in tabular and graphical form, for the reportor presentation on the study.

NUMBER OF RUNS NEEDED FOR FACTORIAL

EXPERIMENTAL DESIGNS

Many factors can be used in a screening experimentfor a sensitivity analysis to determine which factors arethe main drivers of the response variable. However, asnoted earlier, as the number of factors increases, thetotal number of combinations increases exponentially.Thus, screening studies often use a fractional factorial

design, which produces high condence in the sensitiv-ity results using a feasible number of runs.Fractional factorial designs yield polynomial equa-

tions approximating the true response function, withbetter approximations from higher resolution leveldesigns. The minimum number of runs needed for Reso-lution IV and V designs is shown in Table 2 as a functionof the number of factors in the experiment.

There is a simple relationship for the minimumnumber of runs needed for a Resolution IV design:round up the number of factors to a power of two andthen multiply by two. The usefulness of Table 2 is toshow that often there is no penalty for including more

factors in the experiment. For example, if 33 factorsare going to be studied already, then up to 64 factorscan be studied for the same number of runs, namely,128. It is more desirable to conduct a Resolution Vexperiment to be able to estimate separately all thetwo-way interactions. However, for a large number offactors, it may not be feasible to perform the Resolu-tion V design. Because the signicant two-way inter-actions are most likely to be combinations of the sig-nicant main effects, a Resolution IV design can beused rst, especially if it is known that the factors havemonotonic effects on the response variable. Then afollow-up Resolution V design can be performed to

determine if there are any signicant two-way interac-tions using only the factors found to have signicanteffects from the Resolution IV experiment. If a fac-torial design is used as the screening experiment onmany factors, the same combinations of factors neednot be replicated, even if the simulation is stochastic.Different design points are preferable to replicatingthe same points since more effects can be estimated,possibly up to the next higher resolution level.

APPLICATION TO A SIMULATION MODEL

Screening Design

Design of experiments was used as the method foridentifying Ballistic Missile Defense (BMD) system-of-systems needs using the Extended Air Defense Simu-lation (EADSIM) model. The sensitivity analysis pro-ceeded in two steps:

1. A screening experiment to determine the maindrivers

2. A response surface experiment to determine the

shape of the effects (linear or curved)

The primary response variable for the study was pro-tection effectiveness, i.e., the number of threats negateddivided by the total number of incoming threats overthe course of a scenario, and the secondary responsevariables were inventory use for each of the defensiveweapon systems.

The boxed insert shows the 47 factors screened in thestudy. These factors were selected by doing a functional

Table 2. Two-level designs: minimum number of runs as a function of

number of factors.

Factors Runs

Resolution IV

1 2

2 4 = 22

34 8 = 23

58 16 = 24

916 32 = 25

1732 64 = 26

3364 128 = 27

65128 256 = 28

129256 512 = 29

Resolution V

1 2

2 4 = 22

3 8 = 23

45 16 = 24

6 32 = 25

78 64 = 26

911 128 = 27

1217 256 = 28

1822 512 = 29

2331 1,024 = 210

3240 2,048 = 211

4154 4,096 = 212

5570 8,192 = 213

7193 16,394 = 214

94119 32,768 = 215


6/9



Forty-sEvEn FActors to BE scrEEnED to IDEntIFy BMD systEM-oF-systEMs nEEDs

Threat radar cross section GB lower tier 2 reaction time

Satellite cueing system probability of detection GB lower tier 2 Pk

Satellite cueing system network delay GB lower tier 2 Vbo

Satellite cueing system accuracy SB lower tier time to acquire track

Satellite cueing system time to form track SB lower tier time to discriminate

GB upper tier time to acquire track SB lower tier time to commit

GB upper tier time to discriminate SB lower tier time to kill assessment

GB upper tier time to commit SB lower tier probability of correct discrimination

GB upper tier time to kill assessment SB lower tier Pk assessment

GB upper tier probability of correct discrimination SB lower tier launch reliability

GB upper tier probability of kill (Pk) assessment SB lower tier reaction time

GB upper tier launch reliability SB lower tier Pk

GB upper tier reaction time SB lower tier Vbo

GB upper tier Pk Network delay

GB upper tier burnout velocity (Vbo) Lower tier minimum intercept altitude

GB lower tier time to acquire track Upper tier minimum intercept altitude

GB lower tier time to discriminate ABL reaction time

GB lower tier time to commit ABL beam spread

GB lower tier probability of correct discrimination ABL atmospheric attenuation

GB lower tier 1 launch reliability ABL downtime

GB lower tier 1 reaction time GB upper tier downtime

GB lower tier 1 Pk GB lower tier downtime

GB lower tier 1 Vbo SB lower tier downtime

GB lower tier 2 launch reliability

decomposition of the engagement process for each defen-

sive weapon system, that is, a radar must detect, track,discriminate, and assess the success of intercept attemptsand the accuracy, reliability, and timeline factors associ-ated with each of those functions.

A fractional factorial experimental design andEADSIM were used to screen the 47 factors above for theirrelative importance in far-term Northeast Asia (NEA)and Southwest Asia (SWA) scenarios over the rst 10days of a war. A three-tiered defense system was employedfor both scenarios, including an airborne laser (ABL), aground-based (GB) upper tier, and a lower tier comprisingboth ground-based and sea-based (SB) systems.

We initially conducted 512 EADSIM runs to screenthe sensitivities of the 47 factors in the NEA scenario.This is a Resolution IV design and resolves all of the 47main factors but cannot identify which of the 1081 pos-sible two-way interactions are signicant.

After analyzing results from the initial 512 runs, 17additional, separate experimental designs were needed(for a total of 352 additional EADSIM runs) to identifythe signicant two-way interactions for protection effec-tiveness. We learned from the NEA screening study that

more runs were warranted in the initial experiment to

eliminate the number of additional experiments neededto disentangle all the two-way interactions. For the SWAscreening study, we conducted 4096 EADSIM runs tond the 47 main factors and all 1081 two-way interac-tions for the 47 factors. This was a Resolution V design.An added benet of conducting more experiments isthat SWA error estimates are approximately one-thirdthe size of NEA error estimates, i.e., the relative impor-tance of the performance drivers can be identied withhigher certainty in SWA compared to NEA, which canbe seen in Fig. 2. Note that only a very small fraction ofthe total number of possible combinations was run, 1 in275 billion since it is a 247238 fractional factorial, evenfor the Resolution V design.

Figure 2 illustrates the main factor sensitivities tothe 47 factors for both the NEA and SWA scenarios,labeled F1 to F47. The colored dots represent the changeof protection effectiveness to each factor, and the errorbars are 95% condence bounds. The y-axis is the differ-ence in the average protection effectiveness for a factorbetween the good and bad values. Factors are deter-mined to be performance drivers if the 95% condence


7/9

J. K. TELFORD


Figue 2. Change in protection eectiveness: 47 main eects and 95% confdence limits.

bounds do not include zero as a probable result. Factors

shown in red in Fig. 2 were found to be performancedrivers in both scenarios. Factors in blue were found tobe drivers in NEA only, and factors in green were foundto be drivers in SWA only. Factors that were not foundto be drivers in either scenario are shown in gray. (Thefactors in Fig. 2 are not listed in the same order as theyappear in the boxed insert.)

The factors in Fig. 2 are sorted in numerical order oftheir effects in the NEA scenario. The red factors allappear in the left quarter of the SWA graph, indicat-ing that many of the same factors that are most impor-tant in the NEA scenario are also the most importantin the SWA scenario. The important factors that differbetween the two scenarios, coded in blue and green,result from the geographic (geometric, laydown, andterrain) differences in those two theaters.

The two-way interactions (1081) are too numerous toshow in a gure similar to the one for the main effects.However, the vast majority of the two-way interactionsare quite small. An example of a signicant interactioneffect can be seen in Fig. 3, shown graphically by thetwo lines not being parallel. The increase in protection

effectiveness from improvingFactor 6 is large if Factor 9 is atthe low level, but essentially zeroif Factor 9 is at its high level. (Fac-tors 6 and 9 are not the sixth andninth values listed in the boxedinsert.) Data would not havebeen collected at the 11 level

for Factors 6 and 9 in the tradi-tional change-one-factor-at-timeexperiment, starting at the 21level for both factors. The pro-tection effectiveness value at 11for both factors would probablybe overestimated from a change-one-factor-at-time experiment.Only by varying both factors atthe same time (the Factorial prin-ciple) can the actual effect of twofactors acting together be known.

Response Surface DesignOnce a screening experiment

has been performed and theimportant factors determined,the next step is often to performa response surface experimentto produce a prediction modelto determine curvature, detectinteractions among the factors,and optimize the process. Themodel that is frequently used toestimate the response surface is

the quadratic model in Eq. 2:

Y= 0 + i Xi + ijj = 1

p

Xi Xji = 1

i j

p

+ ii Xi2

i = 1

p

i = 1

p

, (2)

where

0 = the overall mean response,i = the main effect for each factor (i = 1, 2, ... ,p),ij = the two-way interaction between the ith andjth

factors, andii = the quadratic effect for the ith factor.

To t the quadratic terms in Eq. 2, at least threelevels for the input X variables are needed, that is, high,medium, and low levels, usually coded as 11, 0, and 21.A total of 3p computer simulations are needed to takeobservations at all the possible combinations of thethree levels of the p factors. If 2p computer simulationsrepresent a large number, then 3p computer simulationsrepresent a huge number. The value of conducting theinitial screening study is to reducep to a smaller number,say k. Even so, 3k computer simulations may still beprohibitively large.


8/9



The minimum number of runs needed for a three-level Resolution V design as a function of the number offactors is shown in Table 3. From the two-level screen-ing designs, 11 main effects were statistically signicantand have at least a 1% effect on protection effectiveness.

Table 3 shows that for 11 factors, a minimum number of243 runs are needed. Notice that 36 factors out of theoriginal 47 have been deemed nonsignicant and will bedropped from further experimentation.

An example of a signicant quadratic main effect(Factor 9) and a signicant two-way interaction betweenFactors 6 and 9 for the three-level fractional factorialresponse surface experiment is shown in Fig. 4. There aredifferent values in protection effectiveness when Factor9 is at the low level (21), depending on whether the levelof Factor 6 is a the low, medium, or high level, but verylittle difference if Factor 9 is at the high level (11). Theshape of the lines in Fig. 4 is curved, indicating that a

quadratic term is needed for Factor 9 in the polynomialequation. (Factors 6 and 9 are not the sixth and ninthfactors listed as in the boxed insert.)

The polynomial equation for protection effectivenesswith quadratic and cross-product terms resulting fromthe 31126 fractional factorial response surface experimentis shown in Eq. 3. The size of a factor effect on protectioneffectiveness is actually twice as large as the coefcientson the X terms since the coefcients are actually slopesand X has a range of 2 (from 21 to11).

Figue 3. Protection eectiveness: two-way interaction between

Factors 6 and 9 rom the screening experiment.

The full study comprised not only an examinationof two theaters (NEA and SWA) but also four forcelevels in each theater. All of the analyses shown previ-ously were conducted at Force Level 4, which is com-parable to a Desert Stormlevel of logistics support

before the operation. Force Level 1 is a rapid responsewith no prior warning and limited weapons available.Force Levels 2 and 3 are intermediate between Levels1 and 4. The response surfaces for the four force levelsin the NEA scenario are shown in Fig. 5. The indi-vidual graphs are the response surfaces for Factors 9and 11, the two largest main effects for Force Level4. There is a very noticeable curvature for Factor 9,especially at Force Levels 1 and 2. As the force levelincreases, protection effectiveness increases. The dif-ferent color bands are 5% increments in protectioneffectiveness: red is between 65% and 70% and orangeis between 90% and 95%. The response surfaces at-

ten out and rise up as the force level increases, that is,protection effectiveness improves and is less sensitiveto changes in the factors. As the force level increases,there are more assets available, so the reliance on theperformance of any individual asset diminishes. (Fac-tors 9 and 11 are not the ninth and eleventh valueslisted in the boxed insert.)

Table 3. Three-level Resolution V designs: minimum number of runs

as a function of number of factors.

Factors Runs

1 3

2 9 = 32

3 27 = 33

45 81 = 34

611 243 = 35

1214 729 = 36

1521 2187 = 37

2232 6561 = 38

Figue 4. Protection eectiveness: quadratic main eect and

two-way interaction between Factors 6 and 9 rom the Response

Surace Experiment.(3)

PE = 0.938 + 0.035X9+ 0.026X

11+ 0.017X

5

+ 0.016X2+ 0.015X

6+ 0.014X

1+ 0.012X

7

+ 0.011X4+ 0.007X

3+ 0.006X

8 0.011X

6X9

0.007X8X9 0.007X

2X5 0.006X

5X7

0.005X3X9 0.005X

5X6 0.005X

1X5

0.019X9

2 0.011X

5

2 0.009X

11

2 0.008X

4

2

0.006X3

2 0.006X

2

2.


9/9

J. K. TELFORD


CONCLUSION

Design of experiments has been applied success-fully in diverse elds such as agriculture (improvedcrop yields have created grain surpluses), the petro-chemical industry (for highly efcient oil reneries),and Japanese automobile manufacturing (giving thema large market share for their vehicles). These develop-ments are due in part to the successful implementationof design of experiments. The reason to use design of

experiments is to implement valid and efcient exper-iments that will produce quantitative results and sup-port sound decision making.

ACKNOWLEDGMENTS. Te ator wold lke to tak RoerL. Wet, barry L. mtell, ad Toa p. sreterafor ter artato te tdy tat wa ed a teexale for t artle.

REFERENCES

1Fisher, R. A., The Design of Experiments, Oliver & Boyd, Edinburgh,Scotland (1935).

2Deming, W. E., Out of the Crisis, MIT Center for Advanced Engineer-ing Study, Cambridge, MA (1982).

3Box, G. E. P., and Draper, N. R., Empirical Model Building and ResponseSurfaces, Wiley, Hoboken, NJ (1987).

Figue 5. Protection eectiveness response suraces or Factors 9 and 11 at our

orce levels in the NEA theater.

Jacqueline K. Telford

The AuthorJacqueline K. Telford is a statistician and a Principal Professional Staff member at APL. She obtained a B.S. in mathematics from

Miami University and masters and Ph.D. degrees in statistics from North Carolina State University. Dr. Telford was employed

at the U.S. Nuclear Regulatory Commission in Bethesda, Maryland, from 1979 to 1983. Since joining

APL in 1983, she has been in the Operational Systems Analysis Group of the Global Engagement

Department working primarily on reliability analysis and testing, test sizing, and planning for Trident

programs, and more recently for the Missile Defense Agency. She has successfully applied statistical

analysis methods to numerous other projects, including evaluation of sensors, models of the hydro-

dynamics of radioisotopes, and lethality effects. She has taught statistics courses in the JHU gradu-

ate engineering program, mostly recently on the topic of design of experiments. Her e-mail address is

[email protected].

a brief introduction to doe

Documents