distinguishing between analytical precision and assessment … · 2017. 2. 1. · accuracy and...

Distinguishing between analytical

precision and assessment accuracy

in relation to materials

characterisation

Steven Pearce| Principal environmental scientist Perth

Presentation Title

Presentation overview

• Heterogeneity, glossing over the elephant in the data

room

• Paradigm of the lab as the provider of accurate data

• Statistics and data smoothing (professional lies)

• Materials characterisation, adopting a rational approach

Materials characterisation

Presentation Title


• Geochemical and geophysical properties

• Defining specific characteristics that are taken to be

representative of the bulk properties of the material

• Classifying materials into groupings based on

characterisation process

• Examples:

• Landfill waste classification

• Contaminated sites assessment

• Mine waste classification


Presentation Title

General problems of carrying out

material characterisation studies

• Remote location of sites

• Time pressure (everyone has a schedule), and ultimately

time is money

• Need an answer quickly, not recommendations for more

testing

• Size of site, volume of material to sample

• Heterogeneity of materials that require sampling

End result: Defining bulk properties using “snapshots”


Presentation Title

Bulk properties

Particle size distribution

Point source effects

Geological controls

Weathering

Geochemical controls

Fractionation

Mine Closure Materials characterisation

Presentation Title

Characterisation

Concentration

Distribution

Volume

Mine Closure Materials characterisation

Presentation Title

Heterogeneity is the measure of the degree of

compositional variability of a material. This can be divided

into

• inter sample (macro scale): Concentration may occur

within a particular material (for example in a vein of

primary mineralisation), at a particular location, or at a

particular depth (point source contamination)

• intra sample (micro scale): e.g. various mineral phases

may be present and unequally distributed (for example

isolated macro pyrite crystals)


Defining bulk properties, a scalar problem

Presentation Title

Intra sample variability shown from

multiple XRF results from a single sample

(approx 50g dry weight)

*50g sample split into 3 parts: bulk, 2mm (coarse)

Composite

lab result

within 10%

of mean of

XRF

results

Presentation Title

• Typically, for analysis the lab will extract a small 1-10g

sub sample of the 1000g parent sample on which to

complete analysis

• selection of the sub sample may take place after sieving

or crushing of the parent sample.

• Therefore, if intra sample heterogeneity is significant

then clearly it will be unlikely that a single 1-10g sub

sample will be representative (chemically or

mineralogicaly) of the sample as a whole.

• Bias introduced at early stage

Mine Closure

Intra sample variability (analysis bias)


Presentation Title

• Contamination may be concentrated within a particular

material, at a particular location, or at a particular depth

• If random sampling is being employed then it is clear that

inter sample heterogeneity will have a potentially

significant impact on the ability of the sampling

programme to characterise the distribution of

contamination on site

Mine Closure

Inter sample variability


Presentation Title

Inter sample variability (90 sample data

base)

Copper results:

Note wide inter

sample variability

probably not

captured the full

sample population

distribution range

Very large

distribution

“tail”


Presentation Title

Very different terms although commonly mixed up.

• Precision is the repeatability of a testing method

• Accuracy is a reflection of how well the testing characterises

the sample composition

Testing a 10g sub sample of material in a laboratory may

therefore yield high precision result if a duplicate test is

completed on the same 10g sub sample

however if a separate 10g sample of the material is tested

very likely that intra sample variability will be introduced

that will result in increased error (i.e. reduce accuracy).

Mine Closure

Accuracy and precision


Presentation Title

• No analytical technique is 100% precise and so random and systematic

errors will affect the final result

• Generally, the error introduced by modern analytical instrumentation as

analytical bias is likely to be relatively low.

• A study carried out in Europe (CLAIRE technical bulletin 7) indicates that for

a particular case study sampling was by far the greatest cause of uncertainty

rather than analysis.

• Precision was estimated at 83% of the concentration value for the

sampling method, but was much lower at 7.5% for analytical method.

• The overall random component of uncertainty was estimated as being

83.6%, that is to say, the value of any concentration for an individual

location was reproduced to within ± 83.6% of the quoted value (at 95%

confidence).

• Given that analytical precision was only 7.5%, then clearly the majority of

the overall variability was related to sampling rather than analytical

factors.

Mine Closure

Accuracy and precision


Presentation Title Mine Closure

Lithological characterisation, vertical

sampling intervals

-20

0

20

40

60

80

100

0 25 50 75 100 125 150 175 200 225 250

Depth (m)

Est NAPP

NA

PP

(k

g H

2S

O4/t

on

ne



Elevated heavy metals not related to

sulfur…

-20

0

20

40

60

80

100

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

0 25 50 75 100 125 150 175 200 225 250

NA

PP

(kg

H2SO

4/t

on

ne

)

Depth (m)

Mercury

Est NAPP

Metal x



Lithological characterisation, vertical

sampling intervals

-50

0

50

100

150

200

250

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

0 25 50 75 100 125 150 175 200 225 250

Depth (m)

Mercury

NAG pH 7

Est NAPP

Metal x


Presentation Title

Traditional “collect and analyse”

Environmental Sampling: Aiming for high

precision on limited sample numbers

1000 m3 500g 5g

Typical volume of

material represented by

one site sample

Typical weight of

sample collected

Typical weight of

laboratory sub-

sample that is

analysed

Intra/inter

sample

variability

Intra

sample

variability


Presentation Title

Heterogeneity testing (used in mining industry

for grade control)

• Coarse fragment ores (stockpile, mill feed etc)

• 50-100 individual fragments picked one by one

• Each assayed to extinction

• Consecutive results graphed

• Often can see that removal of top few results will drop

mean grade by orders of magnitude

• Caused by heterogenity, small pockets of high grade

material in general low grade background

• Applies to contaminated sites, similar distributions


Presentation Title

Heterogeneity testing profile

0

50

100

150

200

250

0 10 20 30 40 50 60 70 80

mg/

kg

HT group

Inclusion of top 5

results considerable

increases mean grade


Presentation Title

Worldwide sampling and testing standards (e.g. ASTM, USEPA,

UK EA, DEC Australia) based on the premise that collecting

limited samples from the field and using laboratory to analyse to

high precision provides the most “accurate” data

Based on assumption that

• Only laboratory derived data is acceptable

• That analytical precision is the cause of most sampling error

• That statistics can “fill” the data gaps left by low sampling density.

This assumption is flawed however as multiple studies have

shown that sample variability (heterogeneity) has the greatest

impact on accuracy, and that statistics do a poor job of

interpolation (i.e. Data gap filling).

The solution is increase to sampling density

Mine Closure

Current paradigm “The lab as the provider of

accurate information”


Presentation Title

Paradigm reflected in guidance docs

NEPC 1999: 4.7 FIELD TESTING

• “A variety of field testing devices may be used as a

limited contribution to the screening of samples on

contaminated sites”.

• “The role in providing real-time data needs to be

augmented by chemical analysis of soil. Their use as the

sole source of analytical data in the assessment of

potentially contaminated sites is inappropriate as they

may give falsely high or low results”.


Presentation Title

Mine Closure

Embedded assumptions

Precision - measures the reproducibility of measurements under a given set of conditions. The

precision of the data is assessed by calculating the Relative Per cent Difference (RPD) between

duplicate sample pairs.

200(%)

do

do

CC

CCRPD

Where Co = Analyte concentration of the original sample

Cd = Analyte concentration of the duplicate sample

The Environmental Consultant will adopt nominal acceptance criteria of 30% RPD for field duplicates

and splits for inorganics, and nominal acceptance criteria of 50% RPD for field duplicates and splits

for organics, however it is noted that this will not always be achieved, particularly in heterogeneous

soil or fill materials, or at low analyte concentrations.

Question: If analytical techniques are precise why such

high acceptable RPDs? And why therefore is field

testing “unacceptable”?

There is inconsistent logic in this approach which is

embedded in the industry


Presentation Title

• A given site (or location) will have a given sample

population distribution that cant be known (without

testing every gram of material)

• As we can never know the ‘true’ sample population,

taking a few soil samples from the site is therefore is

akin to ‘random’ sampling as nothing prior is known

about the population distribution

• Generally the more samples that are analysed from a

given site (or location) the greater the confidence in the

overall assessment. The direct relationship between

increasing levels of confidence with sample numbers

comes down to simple statistics.

Mine Closure

Statistical considerations


Presentation Title

• Guidance on the number of samples to be taken on a site for the

purposes of contaminated sites assessment based on assumptions

that do not generally apply to most sites.

• The key assumptions include that the occurrence of contamination is

described by normal distribution, and that hotspots present are of

uniform size, shape and vertical profile.

• Common for the 95th percentile value to be quoted by assessors

and requested by regulators as a representative concentration for a

contaminant upon which to base decisions (to portray an illusion of

statistical certainty).

• However, in reality the probability of being able to define anything

close to the true 95th percentile representative concentration for a

site is very unlikely when sampling at densities similar to that

recommended by published guidance

Mine Closure

Statistical considerations


Presentation Title

• Statistics commonly cited as a method to account for variability and

to allow for interpolation to fill data gaps (e.g. US95, outlier tests, non

parametric analysis etc)

• However data created in this way is prone to large error and can

introduce bias into interpretation of data sets.

• Many broad assumptions are made, most commonly analysis

techniques assume a normal distributed data set. Problem is most

data sets are not normally distributed, and in the majority of

instances the data set is to small to define the true mean, median,

and minimum/maximum values.

• Increasing sampling frequency is the only way to accurately fill “data

gaps” and therefore to reduce the error in calculation of descriptive

statistics (mean, minimum etc)

Mine Closure

Statistics (the magic data gap filler)


Presentation Title

Generated concentration profile, however

apparent inter sample variability is in fact likely

to be intra sample heterogeneity


Presentation Title

Increased sampling density

using on site screening

Intra sample

variability assessment

completed on sub

samples (as little as

1g material required)

10 or more

samples

analysed 1000m3


Presentation Title

2D contour plots produced from XRF

data to show distribution of metals

Presentation Title

Inter sample variability (90

sample data base)

Copper results:

Note wide inter

sample variability

probably not

captured the full

sample population

distribution range

Very large

distribution

“tail”

Presentation Title

Intra sample variability shown from XRF

results from a single sample location

Intra sample variability

0

200

400

600

800

1000

1200

1400

Co

pp

er

[mg

/kg

]

Bulk fraction

Fine fraction

Coarse fraction

Lab

Note: Up to

800 mg/kg

variance

*50g sample split into 3 parts: bulk, 2mm (coarse)

Composite

lab result

within 10%

of mean of

XRF

results

Presentation Title

Characterising the distribution of

contamination, the case for more samples

rather than higher precision

Sample population range: Likely to be larger than defined

Intra sample variability >10% of inter sample variability,

difficult to differentiate between the two in some samples

and therefore determine what is the cause of spatial

variation in concentration

Laboratory results rely on compositing, unclear at what

scale is this acceptable given the level of intra sample

variability (>100%)

If we had relied on limited lab samples alone the

distribution would be even more poorly characterised as a

result of smaller data set, and compositing, preventing

understanding of intra sample variability Materials characterisation

Presentation Title

Conclusions

Limitations of standard methodology of taking less

samples but aiming for higher precision (lab ICP)

• Logistical and cost implications of taking physical samples

and sending to laboratory

• Poor understanding of intra sample variability

• Very easy to assume variability between samples is a

function of lateral or vertical distribution, could be simply be

a function of intra sample variability

• Low probability of defining the sample population range and

true mean, but a high chance of thinking you have

• Therefore: Not an ideal method for defining areas or

volumes of material as contaminated


Presentation Title

[email protected]

www.ghd.com

distinguishing between analytical precision and assessment … · 2017. 2. 1. · accuracy and...

Documents