DQOs and the Development DQOs and the Development of MQOsof MQOs

Carl V. GogolakCarl V. Gogolak

USDOEUSDOE

Environmental Measurements Lab Environmental Measurements Lab

22

Data Quality Objectives

DQOs define the performance criteria that limit the probabilities of making decision errors by: considering the purpose of collecting the data; defining the appropriate type of data needed; and specifying tolerable probabilities of making decision errors.

33

Measurement Quality Objectives

MQOs are statements of performance objectives or requirements for a particular analytical method performance characteristic.

MQOs are acceptance criteria for quality attributes, usually measured by project DQIs.

One of the most important of these is the analytical measurement uncertainty (accuracy).

44

UncertaintyUncertainty

UncertaintyUncertainty is defined in the ISO Guide to is defined in the ISO Guide to the Expression of Uncertainty in the Expression of Uncertainty in Measurement (GUM) as:Measurement (GUM) as:

““a parameter associated with the result of a parameter associated with the result of a measurement that characterizes the a measurement that characterizes the dispersion of values that could reasonably dispersion of values that could reasonably be attributed to the measurand.”be attributed to the measurand.”

55

UncertaintyUncertainty

The uncertainty of a measured value is typically The uncertainty of a measured value is typically expressed as an estimated standard deviation, expressed as an estimated standard deviation, called a called a standard uncertainty.standard uncertainty.

The standard uncertainty of a calculated result is The standard uncertainty of a calculated result is usually obtained by propagating the standard usually obtained by propagating the standard uncertainties of a number of other measured uncertainties of a number of other measured values. In this case it is called a values. In this case it is called a combined combined standard uncertainty.standard uncertainty.

66

DQOs and Uncertainty DQOs and Uncertainty

No measurement program or sampling plan No measurement program or sampling plan can be adequately designed without some can be adequately designed without some estimate of the uncertainty in the data estimate of the uncertainty in the data relative to the action level.relative to the action level.

77

Uncertainty and the Action LevelUncertainty and the Action LevelRelatively large uncertainty can be tolerated:

Either more accuracy or more samples are needed:

88

Connecting the MQOs to the DQOsConnecting the MQOs to the DQOs

Decision errors are made because there is Decision errors are made because there is uncertainty in the datauncertainty in the data

One component of the uncertainty is One component of the uncertainty is analytical measurement uncertaintyanalytical measurement uncertainty

To limit decision errors, the analytical To limit decision errors, the analytical measurement uncertainty should be measurement uncertainty should be limited to a level appropriate to the DQOslimited to a level appropriate to the DQOs

99

MARLAP MARLAP emphasizesemphasizes Method UncertaintyMethod Uncertainty as an MQO. as an MQO.

Method UncertaintyMethod Uncertainty refers to the predicted refers to the predicted uncertainty of a measured value that would be uncertainty of a measured value that would be calculated if the method were applied to a calculated if the method were applied to a hypothetical laboratory sample with a specified hypothetical laboratory sample with a specified analyte concentration.analyte concentration.

The Method Uncertainty is a characteristic of the The Method Uncertainty is a characteristic of the analytical method and the measurement analytical method and the measurement process.process.

1010

Developing MQOs for Method Developing MQOs for Method UncertaintyUncertainty

Data are collected so that decisions can be Data are collected so that decisions can be made...made...

Decisions can be made about individual samplesDecisions can be made about individual samples……as for bioassaysas for bioassays

Decisions can be made about the mean of a Decisions can be made about the mean of a sampled populationsampled population……as for MARSSIM final status surveysas for MARSSIM final status surveys

1111

Decisions made about individual Decisions made about individual samplessamples

HH00: Sample contains no radioactivity: Sample contains no radioactivity

HHaa: Sample contains radioactivity: Sample contains radioactivity

Type I error: Decide there is radioactivity when there isn’tType I error: Decide there is radioactivity when there isn’t

Type II error: Decide there is no radioactivity when there isType II error: Decide there is no radioactivity when there is

This is the familiar framework for LLD calculations.This is the familiar framework for LLD calculations.

1212

Desired Limits on Decision ErrorsDesired Limits on Decision Errors

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5

Concentration

Pro

babi

lity

of D

etec

tion

AL 0

Gray Region:The probability of detectiondecreases with decreasingconcentration

Limit the probability of missing a sampleabove the action level

Limit the probability of flagging a blank sample

1313

Lower Limit of DetectionLower Limit of Detection

Distributionof net countsunder nullhypothesis

Distribution of net counts under alternativehypothesis

LC0 LD

1414

-3 -2 -1 0 1 2 3 4 5 6

Concentration (in units of measurement standard deviation)

0.0

0.1

0.2

0.3

0.4

3

1.645

1

The probability of detection increases with increasing concentration

1515

Power CurvePower Curve

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5

Concentration

Pro

babi

lity

of D

etec

tion

LDLC0

Gray Region

1616

The MQO for Method UncertaintyThe MQO for Method UncertaintyThe width of the gray region is = AL – 0 = AL.

To limit the probability of decision errors, and , to 0.05, we have to limit the standard deviation of the analytical method, M

So that / M > 1.645 + 1.645 = 3.29.

The performance requirement is that the upper bound for the measurement standard deviation is MR = / 3.29 = AL / 3.29.

This is essentially the same as requiring that the minimum detectable concentration (MDC) not exceed the action level.

1717

Required Method UncertaintyRequired Method Uncertainty The true standard deviation of the measurement The true standard deviation of the measurement

process, process, M , is a theoretical quantity and is never , is a theoretical quantity and is never

known exactly.known exactly.

The laboratories estimate of The laboratories estimate of M is denoted uM, the method uncertainty.

The value MR is the required upper bound for the

unknown M. Therefore in practice MR is an upper bound on uM,

the method uncertainty.

Used this way, MR is called the required method uncertainty and is denoted by uMR.

1818

Decisions made about the mean of a Decisions made about the mean of a sampled populationsampled population

HH00: The mean exceeds the action level.: The mean exceeds the action level.

HHaa: The mean is below the action level.: The mean is below the action level.

Type I error: Decide mean exceeds AL when it doesn’t.Type I error: Decide mean exceeds AL when it doesn’t.

Type II error: Decide mean does not exceed AL when it does.Type II error: Decide mean does not exceed AL when it does.

1919

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5

Desired Limits on Decision ErrorsDesired Limits on Decision Errors

Mean Concentration

Pro

babi

lity

of D

ecid

ing

Mea

n <

AL

AL DL0

Limit the probability of missing a meanabove the action level

Limit the probability of missing a meanbelow the DL

Gray Region

The widthof the grayregion, ,represents the smallest concentration difference that it is important to detect.

2020

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5

Power CurvePower Curve

Mean Concentration

Pro

babi

lity

of D

ecid

ing

Mea

n <

AL

AL DL0

Gray Region

2121

The MQO for Method UncertaintyThe MQO for Method UncertaintyThe width of the gray region is = AL – DL.The total variance of the data is

222SM

The sampling standard deviation, S , depends on the variability in the spatial distribution of the analyte concentrations and other factors having to do with how the sampling is performed.

The analytical standard deviation, M , is affected by laboratory sample preparation, subsampling and analysis procedures.

2222

The MQO for Method UncertaintyThe MQO for Method Uncertainty

The sample size needed to conduct the hypothesis test with specified limits on and depends on the relative shift, / .

To keep sample sizes reasonable, should be such that 1< / < 3. Ideally, / 3.

The cost in samples rises rapidly when / < 1, but there is little benefit from increasing / above 3.

2323

The MQO for Method UncertaintyThe MQO for Method UncertaintyGenerally it is easier to control M than S.

If S is large, then the best one can do is make M small relative to S.

If M ~ S/3, then the analytical method variance is contributing less than 10% to the total variance 2. Reducing it further will not reduce very much.

This implies that the upper bound for M should be

MRMR u 10/)103/(10/

2424

Required Method UncertaintyRequired Method UncertaintyIf the lower bound of the gray region is zero, then

the required method uncertainty is

10/ALu MRMR This is essentially the same as requiring that the

relative standard deviation of the measurements near the action level be 10%.

In other words, the minimum quantifiable concentration MQC should be no larger than the action level.

2525

Required Method UncertaintyRequired Method Uncertainty The required method uncertainty is The required method uncertainty is

specified at the Action Levelspecified at the Action Level At concentrations below the action level, At concentrations below the action level,

the bound on the standard deviation the bound on the standard deviation ReqReq = = MRMR is constant. is constant.

At concentrations above the action level, At concentrations above the action level, the bound on the the bound on the relativerelative standard standard deviation deviation ReqReq = = MRMR = = MRMR/AL is constant./AL is constant.

The combined standard uncertainty The combined standard uncertainty uucc of of each result may be compared to these each result may be compared to these bounds.bounds.

2626

2727

Required Method UncertaintyRequired Method Uncertainty

The required method uncertainty, The required method uncertainty, uuMRMR , and , and

the required method relative uncertainty, the required method relative uncertainty, MR MR , can be used for both method , can be used for both method

selection and to develop acceptance selection and to develop acceptance criteria for QC sample results.criteria for QC sample results.