useful tips for presenting data and measurement uncertainty analysis

University of FloridaMechanical and Aerospace Engineering

1

Useful Tips for Presenting Data and Measurement Uncertainty

AnalysisBen Smarslok


2

Outline• Why is presenting data properly important?

• Explain important terminology and definitions– NIST vs. ISO vs. ASME/ASTM– Oberkampf definitions of model uncertainty (not included)

• Experimental scenarios and corresponding methods– Uncertainty propagation– Crossed vs. nested factors (ANOVA vs. VCA)– p-values– Interlaboratory Studies (not included)


3

• Best guess by experimenter• Half the smallest division of measurement• Standard deviation: • Standard error: m = /n• Expanded uncertainty of ± 2 or ± 3 (95%

or 99% confidence interval)• Standard uncertainty: u• Combined standard uncertainty: uc

*(Courtesy of Duane Deardorff presentation from UNC)

m = 75 ± 5 g What is the meaning of ± 5 ?


4

What does x ± u mean?• Engineers think in terms of ±2 (95%) • Physicists generally report ±1 (68% CI)• Chemists report ±2 or ±3 (95% or 99% CI)• Survey/poll margin of error is 95% CI• Accuracy tolerances are often 95% or 99%• NIST Calibration certificate is usually 99%

• Conclusion: The interpretation of ± u is not consistent within a field, let alone between fields– It is very important to explain the statistical relevance

of the uncertainty bounds!!!


5

Presenting Uncertainty Precisely• Choose a standard for presenting uncertainty (I

prefer NIST), and reference the standard• Explain the source of the uncertainty

– Type A – calculated by statistical methods (it is useful to explain the design of experiments and the number of samples involved)

– Type B – determined by other means, such as estimate from experience or manufacturers specifications

• Use terms carefully! – Error vs. Uncertainty: Error is the deviation from the true value

and measured value (never known), which is estimated as uncertainty

– Bias vs. variability (will explain later)• Avoid use of ambiguous ± notation without

explanation• Pet peeve:

– COV = covariancexsCV

x coefficient of variation


6

• Uncertainty classification:– Random uncertainty /

variability – scatter in the measurements (v)

– Systematic uncertainty / bias – systematic departure from the true value (b)

NIST Classification of Measurement Uncertainties

• Type of evaluation:– Type A – calculated by

statistical methods– Type B – determined by

other means, such as estimate from experience

xt = true value of specimen = experimental population

averagex = experimental sample

averagevx = random error of samplex = systematic error of

sample

2Bb Range is at 95% (2)

level of a normal distribution

xx x

ssN

v


7

Uncertainty Analysis Example • Consider our transverse modulus work (E2)

• Hooke’s Law:

• We will work through this problem backwards

P = Load

A = Area

= transverse strain2

2

PEA

1

2

A w t

TotalUncertainty

Bias &Variability Components Contributors


8

Level 1: Total Uncertainty• In general,

– where, vX and bX were propagated from component uncertainties

= Student’s t distribution at 95% confidence level (depends on # of DOF)

• Total uncertainty of E2 at 1 (68%) confidence for comparison to experimental results

• Or, at the commonly accepted 95% level

2 268 2 2 2( ) ( )E E EU b v

2,95% 9.01 0.12E GPa

2 295 95 ( ) ( )X X XU t b v

95t


9

Level 2: Uncertainty Propagation• Law of Propagation of Uncertainties (LPU):

– where, p are the inputs (components) and q is the output• E2 Example:

– Uncertainty contributors were analyzed for each of the components of E2

– Random and systematic effects propagated separately– Only systematic uncertainties can have correlated

effects• Thickness and width are correlated22 2 2

2 2 2 2 22 2 2 22 2

2( ) ( ) ( ) ( ) ( )T T T T

E E E EE P t wP t w

v v v v v

22 2 22 2 2 2 22 2 2 2 2 2

2 22

( ) ( ) ( ) ( ) ( ) 2 ( ) ( )T T T T T TE E E E E Eb E b P b b t b w b t b wP t w t w

2

2 2

1 1 1

( ) ( ) ( , )n n n

i i ji i ji i j

q q qu q u p u p pp p p


10

Level 3: NIST Component Measurement Uncertainty Table


11

Level 4: Contributors of Component Uncertainty

(Further Analysis)• Numerous different methods to analysis the

significance of uncertainty contributors• It is important to use the appropriate

analysis method depending on the design of experiments (DOE)– Either design the experiments properly or match the

corresponding method to the data you already have• Most DOEs fall into one of these two

categories:Crossed

Same patients in each hospital. Patients unique to each hospital.

Nested


12

Crossed Design: ANOVA• Crossed (or factorial) DOEs correspond to

analysis of variance (ANOVA)• Consider thickness in the E2 example

– Since the SAME specimens were measured in the SAME positions with the SAME users, then the factors were crossed

– 3-way ANOVA with crossed, random variables was conducted

Nominal: 0.09 x 1 in.Uncertainty contributors:Specimen – variability from specimen to specimenPosition – variation across

measurement surface User – error from user techniqueMeasurement repeatability


13

Thickness ANOVA• 3-way ANOVA of crossed, random variables

– Statistical software available for ease of use: Excel for 2 factors or SAS for 3 or more

• Factors:– A = specimen a = 4– B = position b = 3– C = user c = 4– Repetitions: n = 3

• ANOVA model:

• ANOVA results were not directly used in uncertainty analysis, but were used to identify significant contributors and validate uncertainty estimates

0:

0:2

20

aH

HHypothesis Test for A:

ijklijkjkikijkjiijkly )()()()(


14

Results: Thickness ANOVA

• Use ANOVA to deterimine the significance of the contributors of uncertainty in thickness

• Position is most significant factor with p-value = 0.006

• Not as interested in interactions in this study

• Used to validate estimated range of uncertainties of thickness and width

~~ ~


15

Nested Design: VCA• Nested DOEs correspond to variance component analysis (VCA)• Consider a two-stage nested design of one specimen for

thickness– Relevant if positions and users were unique each time– Specimens considered individually since the thickness does not

have to be the same from one specimen to the next– Data was organized according to position– y1, y2, and y3 refer to the repeated basic measurements


16

Variance Component Analysis of Thickness

• Goal: Develop a nested design to determine the contribution of each factor in the overall variance

• Variance of the measurement process for one specimen

– Position – the three locations on the specimen where the thickness was measured (unique to each specimen)

– User – four different users per position performed the measurements

– Basic Measurement – three repeated measurements by each user at each position

• Compare the weight of each contributor to determine significance

i

iprocess22 where, i is a component in the process

2222pubmmp


17

Concluding Remarks• Using proper statistical terminology and

representation is necessary to have meaningful results

• You can say your results are “pretty good”, but give what your definition of “pretty good” is!

• Depending on the project, more or less uncertainty analysis may be required

• It is important to design your experiments with the statistical analysis in mind

• Age-old question: How many measurements do I need?– Obviously depends on the circumstances, so there is no straight

forward answer– Best recommendation: Feel comfortable enough with your results

that you can predict the next measurement within a desired range

useful tips for presenting data and measurement uncertainty analysis

Documents