gore lecture slides-2011

7/28/2019 Gore Lecture Slides-2011

1/49

Gore Lecture - UD 16 March 2011

Design of Experiments: New Methods

and How to Use Them in Design,

Development and Decision-making

Douglas C. Montgomery

Regents Professor of Industrial Engineering & Statistics

ASU Foundation Professor of Engineering

Arizona State University

[email protected]


2/49


Four Eras in the History of DOX

The agricultural origins, 1918 1940s

R. A. Fisher & his co-workers

Profound impact on agricultural science

Factorial designs, ANOVA The first industrial era, 1951 late 1970s

Box & Wilson, response surfaces

Applications in the chemical & process industries

The second industrial era, late 1970s 1990s Quality improvement initiatives in many companies

Taguchi and robust parameter design, process robustness

The modern era, beginning early 1990s


3/49


George E. P. BoxR. A. Fisher (1890 1962)


4/49


Design Optimality

One of the truly great paradigm shifts in DOX

Can create custom designs for almost anysituation

Modern software makes this easy for at leastsome optimality criteria

What optimality criteria should we use?


5/49


Regression and Design (ANOVA)

Models are the Same Thing


6/49


Design Optimality Criteria:

The D-criteria: M =N

XX'

Maximize the determinant ofM

p

effMax

D

/1

|][|

||

=

XX

XX

The G-criteria: Minimize the maximumSPV over R

( ) ( )mmNyNVar

SPV xXX'xx 1)(

2'

)]([ =

= )(max SPV

pG

Rx

eff

=

The I-criteria: Minimize the average prediction variance over R

2

1 [ ( )]

R

NVar yI d

A =

xx

)(

))((

DAPV

DAPVMinIeff =


7/49


Example E-Commerce

Web page design is important

Companies experiment with the designregularly

This can be an ideal application of optimaldesigns

Many factors Factors often have different number of levels

Often lots of categorical factors


8/49


Example E-Commerce

k = 5 categorical factors

One 5-level factor, one 4-level factor, one 3-level factor, and two 2-level factors

Consider a full factorial design:

All possible combinations of factor levels

240 runs Thats a lot of web pages!

What about a fractional factorial?


9/49


This is a D-optimal fractionalfactorial design constructed from

J MP with 57 runs.

It is nearly orthogonal

No main effects are aliased with anytwo-factor interactions

This is still a large experiment

Lets consider a smaller design


10/49


30-run D-optimal fractional factorialdesign constructed from J MP

It is nearly orthogonal


11/49


The 15-run design is nearly orthogonal


12/49


A Standard Factor Screening Problem -

Resolution IV Screening Designs in 16 Runs

These are designs for 6, 7, and 8 factors

Very widely used

The generators for the standard designs are:

For six factors, E = ABC and F = BCD;

for seven factors, E = ABC, F = BCD, and G = ACD;

for eight factors, E = BCD, F =ACD, G = ABC, and H= ABD.


13/49


Alias Relationships: Six Factors


14/49


Alias Relationships: Seven and Eight Factors


15/49


Alias Relationships

In each design, there are seven alias chainsinvolving only twofactor interactions

These are regular fractions

These are the minimum aberration fractions

Because two-factor interactions arecompletely confounded experimenters oftenexperience ambiguity in interpreting results

Resolve with process knowledge Additional experimentation


16/49


17/49


Interpretation:

A, B, C, and E are

probably real effects

AB and CE are aliases

Either some process

knowledge or additional

experimentation isrequired to complete

the interpretation


18/49


Additional Experimentation: Fold-Over

Reverse the signs in column A, add

another 16 runs to the original design

This single-factor fold over allows all

the two-factor interactions involving

the factor whose signs are switched

to be separated:


19/49


Results:


20/49


Other Design Augmentation Tricks

Partial fold-over, requiring only eightadditional runs

Optimal augmentation (using the D-optimalitycriterion)

But what if additional experimentation isnt anoption?


21/49


An Alternative to Additional Experimentation

While strong two-factor interactions may be less likely than strong

main effects, there are many more interactions than main effects inscreening situations.

It is not unusual to find that between 15 and 30% of the effects in ascreening design are active

As a result, the likelihood of at least one significant interactioneffect is high.

There is often substantial institutional reluctance to commitadditional time and material (and sometimes its impossible).Consequently, experimenters would like to avoid the need for afollow-up study.

We show can lower the risk of analytical ambiguity by using a

specific orthogonal but non-regular fractional factorial design. The proposed designs for six, seven, and eight factor studies in 16

runs have no complete confounding of pairs of two-factorinteractions.


22/49


Background

Plackett and Burman (1946) introduced non-regular orthogonal designs

for sample sizes that are a multiple of four but not powers of two. Hall (1961) identified five non-isomorphic orthogonal designs for 15

factors in 16 runs. Our proposed six through eight factor designs areprojections of the Hall designs created by selecting specific sets ofcolumns.

Box and Hunter (1961) introduced the regular fractional factorial designs

that became the standard tools for factor screening. Sun, Li and Ye (2002) catalogued all the non-isomorphic projections of the

Hall designs.

Li, Lin and Ye (2003) used this catalog to identify the best designs to use incase there is a need for a fold-over. For each of these designs they providethe columns to use for folding and the resulting resolution of the

combined design. Loeppky, Sitter and Tang (2007) also used this catalog to identify the best

designs to use assuming that a small number of factors are active and theexperimenter wished to fit a model including the active main effects andall two-factor interactions involving factors having active main effects.


23/49


Metrics for Comparing Screening DesignsConsider the model

y = X + ewhere X contains columns for the intercept, main effects and all two-factor interactions,

is the vector of model parameters, and is the usual error vector.

For six through eight factors in 16 runs, the matrix, X, has more columns than rows. Thus,

it is not of full rank and the usual least squares estimate for does not exist because the

matrix XX is singular. With respect to this model, every 16 run design is supersaturated.

Booth and Cox (1962) introduced the E(s2) criterion as a diagnostic measure for comparing

supersaturated designs:

2

2

( )

( ) , number of columns in( 1)

i j

i j

E s kk k

A A


29/49


Correlation Matrix (a) Regular 26-2 Fractional Factorial (b) the

No-Confounding Design


30/49


Recommended 16-Run Seven Factor


Run A B C D E F G

1 1 1 1 1 1 1 1

2 1 1 1 -1 -1 -1 -1

3 1 1 -1 1 1 -1 -1

4 1 1 -1 -1 -1 1 1

5 1 -1 1 1 -1 1 -1

6 1 -1 1 -1 1 -1 1

7 1 -1 -1 1 -1 -1 1

8 1 -1 -1 -1 1 1 -1

9 -1 1 1 1 1 1 -1

10 -1 1 1 -1 -1 -1 1

11 -1 1 -1 1 -1 1 1

12 -1 1 -1 -1 1 -1 -1

13 -1 -1 1 1 -1 -1 -1

14 -1 -1 1 -1 1 1 1

15 -1 -1 -1 1 1 -1 1

16 -1 -1 -1 -1 -1 1 -1


31/49





32/49


Recommended 16-Run Eight Factor


Run A B C D E F G H

1 1 1 1 1 1 1 1 1

2 1 1 1 1 -1 -1 -1 -1

3 1 1 -1 -1 1 1 -1 -1

4 1 1 -1 -1 -1 -1 1 1

5 1 -1 1 -1 1 -1 1 -1

6 1 -1 1 -1 -1 1 -1 1

7 1 -1 -1 1 1 -1 -1 1

8 1 -1 -1 1 -1 1 1 -1

9 -1 1 1 1 1 1 1 1

10 -1 1 1 -1 1 -1 -1 -1

11 -1 1 -1 1 -1 -1 1 -1

12 -1 1 -1 -1 -1 1 -1 1

13 -1 -1 1 1 -1 -1 -1 1

14 -1 -1 1 -1 -1 1 1 -1

15 -1 -1 -1 1 1 1 -1 -1

16 -1 -1 -1 -1 1 -1 1 1


33/49





34/49


Design Comparison on Metrics

N FactorsDesign

Confounded Effect PairsE(s2) Trace(AA)

6 Recommended 0 7.31 6

Resolution IV 9 10.97 0

7 Recommended 0 10.16 6


8 Recommended 0 12.80 10.5



35/49


Example Revisited

The No-Confounding Design for the Photoresist Experiment

Run A B C D E FThickness

1 1 1 1 1 1 1 4494

2 1 1 -1 -1 -1 -1 4592

3 -1 -1 1 1 -1 -1 4357

4 -1 -1 -1 -1 1 1 4489

5 1 1 1 -1 1 -1 4513

6 1 1 -1 1 -1 1 4483

7 -1 -1 1 -1 -1 1 4288

8 -1 -1 -1 1 1 -1 4448

9 1 -1 1 1 1 -1 4691

10 1 -1 -1 -1 -1 1 4671

11 -1 1 1 1 -1 1 4219

12 -1 1 -1 -1 1 -1 4271

13 1 -1 1 -1 -1 -1 4530

14 1 -1 -1 1 1 1 4632

15 -1 1 1 -1 1 1 4337

16 -1 1 -1 1 -1 -1 4391


36/49


Lock Entered Parameter Estimate nDF SS "F Ratio" "Prob>F"X X Intercept 4462.812

51 0 0.000 1

X A 85.3125 1 77634.37 53.976 2.46e-5

X B -77.6825 1 64368.76 44.753 5.43e-5

X C -34.1875 2 42735.84 14.856 0.00101

D 0 1 31.19857 0.020 0.89184

X E 21.5625 2 31474.34 10.941 0.00304

F 0 1 2024.045 1.474 0.25562

A*B 0 1 395.8518 0.255 0.6259

A*C 0 1 476.1781 0.308 0.59234

A*D 0 2 3601.749 1.336 0.31571

A*E 0 1 119.4661 0.075 0.78986

A*F 0 2 4961.283 2.106 0.18413

B*C 0 1 60.91511 0.038 0.84923

B*D 0 2 938.8809 0.279 0.76337

B*E 0 1 3677.931 3.092 0.11254

B*F 0 2 2044.119 0.663 0.54164

C*D 0 2 1655.264 0.520 0.61321

X C*E 54.8125 1 24035.28 16.711 0.00219

C*F 0 2 2072.497 0.673 0.53667

D*E 0 2 79.65054 0.022 0.97803

D*F 0 0 0 . .

E*F 0 2 5511.275 2.485 0.14476


37/49


If I only knew thenwhat I know now


38/49


A personal experience with DOX: Wine-making.

Original vineyard property purchased in 1983. Objectives

were to begin as a grape supplier to other winemakers,

then develop a winemaking process.

Big problem: none of the partners were winemakers (how

do you make a small fortune in wine).

However, some partners knew the power of designed

experiments.

Wine-making is a chemical processI didnt know how to

make polymer, either, when I took my first job as achemical engineer.


39/49


40/49


The experimental design for 1985 is a fractional factorial:


41/49


Results for 1985:

A: PN Clone

B: Oak TypeC: Barrel AgeD: YeastE: StemsF: Barrel ToastG: Whole ClusterH: Temp

Normal%

probability

E ffec t

-3.98 -2.04 -0.10 1.84 3.77

1

5

10

20

30

50

70

80

90

95

99

A

D

F

G

A D

[AD] = AD + CF + BH + EG

A: PN CloneB: Oak TypeC: Barrel AgeD: YeastE: StemsF: Barrel Toast

G: Whole ClusterH: Temp


42/49


Some of the results are interesting and useful, such as

1. toasting the barrel a little more seems like a good idea, and

2. it doesnt seem to matter much where the oak comes from.

Some results are surprising such as no temperature effect!

There is an interaction: [AD] = AD + CF + BH + EGWhich effect(s) are real? CF and EG are more intuitive than AD, but we

really dont have any process knowledge

How would we normally resolve this?

1. Fold-over (is this even possible)?

2. Partial fold-over?

3. Would the recommended non-regular designs have been better?


43/49


How did we do?

First commercial release, the1990 Pinot Noir, won a goldmedal at the American Wine Competition

1991 release won a silver medal

1992 release won gold a medal, sixth best wine overall (of2000 entries), best Oregon Pinot Noir

Consistently ranked by The Wine Spectatoras among thebest Pinot Noir available, 90+ ratings:Wine Spectator, Dec. 201093 - Corral Creek vineyards, 93 - Stoller Vineyards, 92 - Wind Ridge Vineyards,

91 - 3 Vineyard

Consistently highly rated in the International Pinot NoirCompetition


44/49



45/49


Chehalemwines.com


46/49



47/49


Other Work

Appropriate analysis methods (other thanstepwise regression)?

Whats the power of these designs?

How many two-factor interactions can wedetect?

What about the resolution III case (9-15factors in 16 runs)?


48/49


References Booth, K.H.V., Cox, D.R., (1962). Some systematic supersaturated designs. Technometrics 4,

489495.

Box, G. E. P. and Hunter, J. S. (1961). The 2k-p Fractional Factorial Designs. Technometrics 3,

pp.449458. Bursztyn, D. and Steinberg, D. (2006). Comparison of designs for computer experiments

Journal of Statistical Planning and Inference 136, pp. 1103-1119.

Hall, M. Jr. (1961). Hadamard matrix of order 16.Jet Propulsion Laboratory ResearchSummary, 1, pp. 2126.

Jones, B. and Montgomery, D.C. (2010), Alternatives to Resolution IV Screening Designs in 16Runs, International Journal of Experimental Design and Process Optimisation, Vol. 1, No. 4,

pp. 285-295. Li, W., Lin, D.K.J., Ye, K. (2003) Optimal Foldover Plans for Two-Level Non-regular Orthogonal Designs Technometrics 45, pp.347351. Plackett, R. L. and Burman, J. P. (1946). The Design of Optimum Multifactor Experiments.

Biometrika 33, pp. 305325.

Loeppky, J. L., Sitter, R. R., and Tang, B.(2007) Nonregular Designs With Desirable ProjectionProperties. Technometrics 49, pp.454466.

Montgomery, D.C. (2009). Design and Analysis of Experiments 7th Edition. Wiley Hoboken,New Jersey. Sun, D. X., Li, W., and Ye, K. Q. (2002). An Algorithm for Sequentially Constructing Non-

Isomorphic Orthogonal Designs and Its Applications. Technical Report SUNYSB-AMS-02-13,State University of New York at Stony Brook, Dept. of Applied Mathematics and Statistics.


49/49

Thank You!!

Questions?

gore lecture slides-2011

Documents