strictly positive estimates of variance components for

Strictly Positive Estimates of Variance Componentsfor Measurement Systems Analysis Models

Laura Lancaster and Chris Gotwalt

JMP Research & DevelopmentSAS Institute

Discovery Summit 2015

Laura Lancaster (SAS Institute) Discovery Summit 2015 1 / 39

Outline

1 Measurement Systems Analysis

2 Bayesian Estimation Method

3 Simulation Study

4 Two Factors Crossed Study

5 Two Factors Nested Study

6 Three Factors Staggered Nested Study

7 Conclusions


Measurement Systems Analysis (MSA)

Designed experiments that help determine how muchmeasurement variation is contributing to the overall processvariation.

Model for an observed measurement

X = P + EX = P + EX = P + E

I XXX = Observed Product MeasurementI PPP = Product ValueI EEE = Measurement ErrorI PPP and E are assumed independent with P ∼ N(µp, σ

2p)P ∼ N(µp, σ2p)P ∼ N(µp, σ2p) and

E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)

I We will assume that µe = 0µe = 0µe = 0.

Observed Measurement Variance

σ2x = σ2

p + σ2eσ2

x = σ2p + σ2

eσ2x = σ2

p + σ2e


Measurement Systems Analysis (MSA)

Designed experiments that help determine how muchmeasurement variation is contributing to the overall processvariation.Model for an observed measurement

X = P + EX = P + EX = P + E

I XXX = Observed Product MeasurementI PPP = Product ValueI EEE = Measurement ErrorI PPP and E are assumed independent with P ∼ N(µp, σ

2p)P ∼ N(µp, σ2p)P ∼ N(µp, σ2p) and

E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)E ∼ N(µe, σ2e)

I We will assume that µe = 0µe = 0µe = 0.

Observed Measurement Variance

σ2x = σ2

p + σ2eσ2

x = σ2p + σ2

eσ2x = σ2

p + σ2e


MSA Models

Use random effects models to estimate the sources ofvariation in the measurement process.

Variance components are typically estimated using one of threemethods:

I Average and Range MethodI Expected Means Squares (EMS)I Restricted Maximum Likelihood (REML)

These methods can produce zeroed variance components!


MSA Models

Use random effects models to estimate the sources ofvariation in the measurement process.Variance components are typically estimated using one of threemethods:

I Average and Range MethodI Expected Means Squares (EMS)I Restricted Maximum Likelihood (REML)

These methods can produce zeroed variance components!


Motivation

Wanted to find an estimation method that produces strictlypositive variance components and has good properties for MSAmodels.

There are several non-informative prior methods that producestrictly positive variance components for random effects models.

I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had

posterior means with good properties!

We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.


Motivation



I A flat prior on (0,∞)

- Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had




Motivation



I A flat prior on (0,∞) - Not good

I Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had




Motivation



I A flat prior on (0,∞) - Not goodI Jeffrey’s prior

- Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior - This prior had




Motivation



I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not good

I Portnoy and Sahai’s modified Jeffrey’s Prior - This prior hadposterior means with good properties!



Motivation



I A flat prior on (0,∞) - Not goodI Jeffrey’s prior - Not goodI Portnoy and Sahai’s modified Jeffrey’s Prior

- This prior hadposterior means with good properties!



Motivation







Motivation





We generalized Portnoy and Sahai’s approach and implemented itin JMP’s Variability platform.

Default JMP behavior is to use REML estimates if no variancecomponents have been zeroed and use the Bayesian estimatesotherwise. We will refer to this as a Hybrid method.


Motivation







Research Questions

1 How do the Bayesian estimates compare to the REMLestimates?

I Bias?I Variability?I Classification?

• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method

2 How does our Hybrid method perform?


Research Questions


I Bias?

I Variability?I Classification?




Research Questions


I Bias?I Variability?

I Classification?• Don Wheeler’s Evaluating the Measurement Process (EMP) method• Automotive Industry Action Group’s (AIAG) Gauge R&R method



Research Questions


I Bias?I Variability?I Classification?




Wheeler’s EMP Classification System

Intraclass Correlation Coefficient (ICC) - ρρρ - ratio of productvariance to total variance

ρ =σ2

p

σ2p + σ2

e=σ2

p

σ2x

ρ =σ2

p

σ2p + σ2

e=σ2

p

σ2x

ρ =σ2

p

σ2p + σ2

e=σ2

p

σ2x

Wheeler’s EMP Classifications:

Classification ρ̂̂ρ̂ρ Prob 3σp3σp3σp Shift - Warning Rule 1First Class 0.80− 1.00 0.99− 1.00Second Class 0.50− 0.80 0.88− 0.99Third Class 0.20− 0.50 0.40− 0.88Fourth Class 0.00− 0.20 0.03− 0.40


AIAG’s Classification System

AIAG uses Percent Gauge R&R to classify the health of ameasurement system.

%GRR = 100

√σ̂2

e

σ̂2p + σ̂2

e= 100

σ̂e

σ̂x= 100

√1− ρ̂%GRR = 100

√σ̂2

e

σ̂2p + σ̂2

e= 100

σ̂e

σ̂x= 100

√1− ρ̂%GRR = 100

√σ̂2

e

σ̂2p + σ̂2

e= 100

σ̂e

σ̂x= 100

√1− ρ̂

AIAG Classifications:

Classification %GRR%GRR%GRR ρ̂̂ρ̂ρ

Acceptable 0%− 10% 0.99− 1.00Marginal 10%− 30% 0.91− 0.99Unacceptable 30%− 100% 0.00− 0.91


Portnoy-Sahai Estimator

Modified Jeffrey’s Prior:

πPS(σ2i ) = |F(σ2)|

12 /τ4

i

I σ2i - the i th variance component.

I τ2i - expected mean square associated with the σ2

i .I F(σ2) - Fisher information matrix.

The Portnoy-Sahai estimator, σ̂2i , is the posterior mean of σ2

i usingthe modified Jeffrey’s prior.We use Gauss-Laguerre quadrature to perform the numericalintegration for these estimators.




πPS(σ2i ) = |F(σ2)|

12 /τ4

i





i usingthe modified Jeffrey’s prior.

We use Gauss-Laguerre quadrature to perform the numericalintegration for these estimators.




πPS(σ2i ) = |F(σ2)|

12 /τ4

i





i usingthe modified Jeffrey’s prior.We use Gauss-Laguerre quadrature to perform the numericalintegration for these estimators.


Simulation Study Design

Three Typical MSA ModelsI Two Factors Crossed (balanced)I Two Factors Nested (balanced)I Three Factors Staggered Nested Design (unbalanced)

Range of bad to good measurement systems (using ICC as themetric)

I ICC values in middle of Wheeler’s EMP classifications:0.1, 0.35, 0.65, 0.9

I ICC values in middle of AIAG’s top 2 classifications:0.96 and 0.995

Part variance values:1, 5, and 25



A given part variance, σ2p, and ICC, ρ, determine the measurement

error variance, σ2e:

σ2e =

σ2p(1− ρ)ρ

Each MSA model uses a weighting schema to set the variancecomponents that contribute to the measurement error.

I Select λ1, λ2, . . . , λm ≥ 0 such that∑m

i=1 λi = 1.I Set σ2

i = λiσ2e for i = 1,2, . . . ,m so that

m∑i=1

σ2i = σ2

e



Scripts written in JSL in JMP Pro 13.

Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:

I REMLI Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):

If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.



Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.

Called the Variability platform with following estimation methods:I REMLI Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):




Scripts written in JSL in JMP Pro 13.Used the new JMP Pro 13 Simulate function that makes estimatesimulation very easy.Called the Variability platform with following estimation methods:






I REML

I Bayesian (Portnoy-Sahai)I Hybrid (JMP default setting):





I REMLI Bayesian (Portnoy-Sahai)

I Hybrid (JMP default setting):If zeroed variance components⇒ Bayesian estimates.Otherwise⇒ REML estimates.


Two Factors Crossed Design

Typical MSA design - example from AIAG MSA book3 Operators, 10 Parts, 3 ReplicationsTotal of 90 measurements

500 Simulations


Two Factors Crossed - Mean Bias


Two Factors Crossed - RMSE


Two Factors Crossed - Standard Deviation


Two Factors Crossed - REML Zeroed


Two Factors Crossed - Wheeler’s EMP Classifications


Two Factors Crossed - AIAG Classifications


Two Factors Crossed - Summary

REML and Hybrid estimates are less biased than Bayesianestimates.

Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.



REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.

REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.



REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.

Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.



REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.

AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.



REML and Hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.REML zeroing stays about the same for all ICC (good and badsystems) except for part variance.Wheeler’s EMP Classifications - The Bayesian method does aslightly better job. Classifications are best for good systems.AIAG Classifications - The Bayesian method does a slightly betterjob for marginal and acceptable systems.


Two Factors Nested Design

Typical MSA design - example from Montgomery and Runger(1993)3 Operators, 20 Parts, 2 ReplicationsTotal of 120 measurements

500 Simulations


Two Factors Nested - Mean Bias


Two Factors Nested - RMSE


Two Factors Nested - Standard Deviation


Two Factors Nested - REML Zeroed


Two Factors Nested - Wheeler’s EMP Classifications


Two Factors Nested - AIAG Classifications


Two Factors Nested - Summary

REML and hybrid estimates are less biased than Bayesianestimates.

Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -

I REML and Hybrid methods do a slightly better job classifying badMSA systems, but the Bayesian method does a better job for bettersystems.

I Classifications are more accurate for good systems in general.AIAG Classifications -

I The Bayesian method classifies better for marginal systems butclassifies horribly for acceptable systems (100% wrong).

I REML classifies slightly worse for marginal systems but classifiesmuch better than Bayesian and Hybrid for acceptable systems.

I Overall, the accuracy of the classifications for acceptable systemsis not good.



REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.

Wheeler’s EMP Classifications -I REML and Hybrid methods do a slightly better job classifying bad

MSA systems, but the Bayesian method does a better job for bettersystems.







REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE and standard deviationsthan REML and hybrid estimates.Wheeler’s EMP Classifications -










I Classifications are more accurate for good systems in general.

AIAG Classifications -I The Bayesian method classifies better for marginal systems but

classifies horribly for acceptable systems (100% wrong).I REML classifies slightly worse for marginal systems but classifies

much better than Bayesian and Hybrid for acceptable systems.I Overall, the accuracy of the classifications for acceptable systems

is not good.


Three Factors Staggered Nested Design

Polyethylene pellets MSA design example from Lawson (2008)I 30 lotsI 2 boxes from each lotI 2 preparations within box 1, 1 preparation within box 2I 2 measurements from preparation 1 within box 1, 1 measurement

from preparation 2 within box 1, 1 measurement from preparation 3within box 2.

I Total of 120 measurements of tensile strengthI Highly unbalanced


Three Factors Staggered Nested - Mean Bias


Three Factors Staggered Nested - RMSE


Three Factors Staggered Nested - Standard Deviation


Three Factors Staggered Nested - REML Zeroed


Three Factors Staggered Nested - Wheeler’s EMPClassifications


Three Factors Staggered Nested - AIAGClassifications


Three Factors Staggered Nested - Summary

REML and hybrid estimates are less biased than Bayesianestimates.

Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -

I Bayesian method does a slightly better job classifying bad MSAsystems.

I REML does a better job classifying good systems where Bayesianand Hybrid methods classify quite poorly.

AIAG Classifications -I Bayesian and Hybrid methods do a poor job classifying marginal

and acceptable systems (almost always incorrect).I REML does a better job but is still wrong more than 60% of time!



REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.

Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -







REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.

REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -







REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.

Wheeler’s EMP Classifications -I Bayesian method does a slightly better job classifying bad MSA

systems.I REML does a better job classifying good systems where Bayesian

and Hybrid methods classify quite poorly.AIAG Classifications -

I Bayesian and Hybrid methods do a poor job classifying marginaland acceptable systems (almost always incorrect).

I REML does a better job but is still wrong more than 60% of time!



REML and hybrid estimates are less biased than Bayesianestimates.Bayesian estimates have smaller RMSE than REML and hybridestimates for worse systems, but they are larger for bettersystems.Bayesian estimates have smaller standard deviations than REMLand hybrid estimates.REML zeroing increases as ICC increases.Wheeler’s EMP Classifications -











and acceptable systems (almost always incorrect).

I REML does a better job but is still wrong more than 60% of time!


Conclusions

REML and Hybrid estimates have less mean bias than Bayesianestimates.

Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.

I It will be very difficult to classify your system as marginal oracceptable.

I If this design must be used, we recommend using REML estimates.Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.

I Its performance is a usually a good compromise.I Exception: Two Factors Nested designs when trying to prove a

system is acceptable by AIAG standards.


Conclusions

REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.

Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.






Conclusions

REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.

We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.






Conclusions

REML and Hybrid estimates have less mean bias than Bayesianestimates.Bayesian estimates usually have smaller RMSE than REML andHybrid estimates.Bayesian estimates have smaller standard deviations than REMLand Hybrid estimates.We recommend not using staggered nested designs when at allpossible, especially if AIAG standards are required.






Conclusions



I If this design must be used, we recommend using REML estimates.

Otherwise, we think the hybrid method is a good strategy for TwoFactors Crossed and Two Factors Nested designs.




Conclusions







Conclusions




I Its performance is a usually a good compromise.

I Exception: Two Factors Nested designs when trying to prove asystem is acceptable by AIAG standards.


Conclusions







References

Automotive Industry Action Group (2002), Measurement SystemsAnalysis Reference Manual, 3rd Edition.Lawson (2008), “Bayesian Interval Estimates of VarianceComponents Used in Quality Improvement Studies, ” QualityEngineering, 20, 334-345.Montgomery and Runger (1993), “Gauge Capability Analysis andDesigned Experiments. Part II: Experimental Design Models andVariance Component Estimation,” Quality Engineering, 6(2),289-305.Portnoy (1971), “Formal Bayes Estimation With Application To aRandom Effect Model,” Annals Of Mathematical Statistics, 42,1379-1402.Sahai (1974), “Some Formal Bayes Estimators of VarianceComponents in Balanced Three-Stage Nested Random EffectsModel,” Communications in Statistics, 3, 233-242.


[email protected]@jmp.com


strictly positive estimates of variance components for

Documents