ISO/IEC 17025:2017Statement or Certificate of

Conformance&

Decision rulesBy

John Wilson

Accreditation and Metrology Services (Pty) LTD

John Wilson 2019 1

Introduction• This presentation is presented in conjunction

with the presentation by Steve Sidney who

has covered the new aspects in the 2017

version of ISO/IEC 17025 related to RISK.

• It incorporates some of the material covered

in a previous NLA Test & Measurement

conference but expands on the Decision Rule

and Statements of Conformity now

incorporated in the ISO/IEC 17025:2017 as

well as a summary of what is in the New ILAC

G8:2019 guidance document.

John Wilson 2019 2

What is a decision rule• Let’s start with what is a decision?

John Wilson 2019 3

What is a decision rule• Let’s start with what is a decision?

John Wilson 2019 4

What is a decision rule• Let’s start with what is a decision?

John Wilson 2019 5

What is a decision rule• Let’s start with what is a decision?

John Wilson 2019 6

What is a decision rule• Let’s look at some metrology (measurements)

Years

Where will the next measurement probably be?

John Wilson 2019 7

What is a decision rule• Let’s look at some metrology

Years

Yes Likely

John Wilson 2019 8

What is a decision rule• Let’s look at some metrology

Years

Not likely but possible

John Wilson 2019 9

What is a decision rule• Let’s look at some metrology

Years

Are these measurements “in Specification” ?

John Wilson 2019 10

What is a decision rule• Let’s look at some metrology

Nominal

Are these measurements“in Specification” ?

Before we decide, we need to know the limits (specification)

John Wilson 2019 11

What is a decision rule• Let’s look at some metrology

Nominal

So are the measurements “in Specification” ?

Upper Limit

Lower Limit

John Wilson 2019 12

What is a decision rule• In the 2005 version of ISO/IEC 17025 it states

in clause 5.10.4.2 “When statements of conformance are made, the uncertainty of measurement shall be taken into account”

• A statement of conformance is something

like “The calibration of this instrument shows

that all the readings are within the

manufacturer’s specifications” or “This

instrument complies with the requirements of

the XYZ standard”.

John Wilson 2019 13

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

Lower Limit

On this graph, the uncertainty of measurement has not been taken into account.

John Wilson 2019 14

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

Lower Limit

The Uncertainty bars are now included

John Wilson 2019 15

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

Lower Limit

So are these measurements “in Specification”?

John Wilson 2019 16

What is a decision rule• Let’s look at some metrology

Nominal

Are these measurements “in Specification” ?

Upper Limit

Lower Limit

John Wilson 2019 17

What is a decision rule• Let’s look at some metrology

Nominal

Three different groups of measurements.

Upper Limit

Lower Limit

John Wilson 2019 18

What is a decision rule• Let’s look at some metrology

Nominal

So are the measurements“in Specification” ?

Upper Limit

Lower LimitPass

Fail

Undecided

John Wilson 2019 19

What is a decision rule• Let’s look at some metrology

Nominal

So does this meet the requirements of ISO/IEC 17025?

Upper Limit

Lower LimitPass

Fail

Undecided

John Wilson 2019 20

What is a decision rule• Let’s look at some metrology

Nominal

The undecided could however lead to lots of arguments.

Upper Limit

Lower LimitPass

Fail

Undecided

John Wilson 2019 21

What is a decision rule• Let’s look at some metrology

Nominal

The undecided could however lead to lots of arguments.

Upper Limit

Lower LimitPass

Fail

Undecided

Readings 1 to 4 are clearly “pass” and within Specification.

Reading 7 is clearly “fail” and outside of Specification.

It is readings 5 and 6 that cause the arguments.

John Wilson 2019 22

What is a decision rule• Let’s look at some metrology

Nominal

The undecided could however lead to lots of arguments.

Upper Limit

Lower LimitPass

Fail

Undecided

If you are the user, then the undecided are not acceptable

but if you are the supplier you could argue that that they

are acceptable because of the uncertainty associated with

the readings.

John Wilson 2019 23

What is a decision rule• Let’s look at some metrology

Nominal

The undecided could however lead to lots of arguments.

Upper Limit

Lower LimitPass

Fail

Undecided

In this case we have a series of measurements over a period

of time so we have an understanding of how the artefact

behaves and can reasonably predict what will happen in the

future. This reduces the risk involved in a decision.

John Wilson 2019 24

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

Lower LimitSimilarly – if we had several of the same instruments, such

as a production run, and get the above results, we know a

lot more about the behaviour of the products

John Wilson 2019 25

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

If we had only one (or one set) of measurements,

would we view it any differently?

Is this a pass or fail?

There may be a lot of money or risk resting on the decision.

John Wilson 2019 26

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

In the past many cal labs would not make a decision based

on this information but would rather pass the information to

the customer to decide for themselves.

So this would not be a statement of conformance but merely

a calibration report.

John Wilson 2019 27

What is a decision rule• Let’s look at some metrology

Nominal

Upper Limit

If however the lab was tasked to give a statement

of compliance, what would the decision be?

Clearly it depends who’s side you are on.

Different labs could give different decisions.

So the customer needs to tell you or be told what

“decision rule” you applied to the statement of

compliance.

John Wilson 2019 28

What is a decision rule• The wording of ISO/IEC 17025:2005 leaves it

very vague and open to a wide range of

interpretation.

• The old ILAC-G8:2009 tried to provide some

clarity by writing the guideline on how to look

at pass/fail conformity assessment.

• The JCGM 106:2012 “Evaluation of

Measurement Data- the Role of measurement

uncertainty in conformance assessment is a

further guideline on how to deal with the

problem.

John Wilson 2019 29

What is a decision rule• In the 2005 version of ISO/IEC 17025 it states

in clause 5.10.4.2 “When statements of

conformance are made, the uncertainty of

measurement shall be taken into account”

John Wilson 2019 30

What is a decision rule• The ILAC G8:2009 tried to provide some

clarity by writing this guideline on how to look at pass/fail conformity assessment.

• Compliance: If the specification limit is not breached by the measurement result plus the expanded uncertainty with a 95% coverage probability, then compliance with the specification can be stated (See Case 1 of Fig.1). This can be reported as “Compliance” or “Compliance – The measurement result is within (or below) the specification limit when the measurement uncertainty is taken into account”. In calibration this is often reported as “Pass”;

John Wilson 2019 31

What is a decision rule• The ILAC G8:2009 tried to provide some

clarity by writing this guideline on how to look at pass/fail conformity assessment.

• Non-compliance: If the specification limit is exceeded by the measurement result minus the expanded uncertainty with a 95% coverage probability, then noncompliance with the specification can be stated. (See Case 4 of Fig.1) This can be reported as “Non-compliance” or “Non-compliance – The measurement result is outside (or above) the specification limit when the measurement uncertainty is taken into account”. In calibration this is often reported as “Fail”;

John Wilson 2019 32

What is a decision rule• The ILAC G8:2009 tried to provide some clarity by

writing this guideline on how to look at pass/fail conformity assessment.

• If the measurement result plus/minus the expanded uncertainty with a 95 % coverage probability overlaps the limit, it is not possible to state compliance or non-compliance. The measurement result and the expanded uncertainty with a 95 % coverage probability should then be reported together with a statement indicating that neither compliance nor non-compliance was demonstrated.

• This is what most SANAS labs used in the past if called on for a certificate of conformance or a statement of compliance.

John Wilson 2019 33

What is a decision rule• The ILAC G8:2009 approach can thus still be

used by the laboratories as a guide to their default decision rule.

• Compliance: If the specification limit is not breached by the measurement result plus the expanded uncertainty with a 95% coverage probability, then compliance with the specification can be stated. This can be reported as “Compliance” or “Compliance – The measurement result is within (or below) the specification limit when the measurement uncertainty is taken into account”.

• In calibration this is often just reported as “Pass”;

John Wilson 2019 34

What is a decision rule• The JCGM 106:2012 “Evaluation of

Measurement Data - the Role of measurement

uncertainty in conformance assessment” is a

very comprehensive document.

• The document address a lot more detail and

gives different ways of interpreting results

• As this is a fairly complicated document ILAC

has revised the ILAC-G8:09/2019 document to

assist laboratories (and assessors) on

understanding the requirements of ISO/IEC

17025:2017

John Wilson 2019 35

What is a decision rule• So the bottom line is that much more

knowledge on the subject is required to enter

into an educated discussion with the

customer about the probability of “False

accepts” when a product should fail and

“False rejects” when a product should pass.

• It is suggested that a good study of JCGM

106:2012 is made.

• It is available free off the BIPM web site

• There is also a lot more information available

on the Web related to this subject.

John Wilson 2019 36

What is a decision rule• So why all the fuss? Why did ISO/IEC

17025:2017 address this matter differently?

• It became clear that there were significant

RISK issues at stake and more clarity was

required to protect both the Laboratory User

and the Laboratory itself.

• Again, it depends if you are the equipment

user or the supplier and it boils down to the

issue of the Risk related to the probability of

“False Accept” or “False Reject”. What is the

Risk to all parties?

John Wilson 2019 37

What is a decision rule

• So what is in ISO/IEC 17025:2017 ?…. Definitions …

• Clause 3.7: a decision rule is defined as “a rule that

describes how measurement uncertainty will be

accounted for when stating conformity with a specified

requirement”.

…. Resource requirements – Personnel …

• Clause 6.2.6 requires that the laboratory shall authorize

personnel to perform “analysis of results, including

statements of conformity or opinions and interpretations”.

John Wilson 2019 38

What is a decision rule

• So what is in ISO/IEC 17025:2017 ?…. Review of requests, tenders and contracts …

• Clause 7.1.3 requires that “When the customer requests

a statement of conformity to a specification or standard

for the test or calibration (e.g. pass/fail, in-tolerance/out-

of-tolerance), the specification or standard and the

decision rule shall be clearly defined. Unless inherent in

the requested specification or standard, the decision rule

selected shall be communicated to, and agreed with, the

customer.”

John Wilson 2019 39

What is a decision rule• So what is in ISO/IEC 17025:2017 ?….Specific requirements for test reports ….

….test reports shall, …. for the interpretation of test results, include ..

• Clause 7.8.3.1b) states “where relevant, a statement of conformity with requirements or specifications” and

• Clause 7.8.3.1c) states “where applicable, the measurement uncertainty presented in the same unit as that of the measurand or in a term relative to the measurand (e.g. percent), when it is relevant to the validity or application of the test results, when a customer's instruction so requires, or when the measurement uncertainty affects conformity to a specification limit”.

John Wilson 2019 40

What is a decision rule

• So what is in ISO/IEC 17025:2017 ?….Specific requirements for calibration certificates ….

…..calibration certificates shall include …..

• Clause 7.8.4.1a) states “the measurement uncertainty of

the measurement result presented in the same unit as

that of the measurand or in a term relative to the

measurand (e.g. percent).”

• Clause 7.8.4.1e) also states “where relevant, a

statement of conformity with requirements or

specifications”.

John Wilson 2019 41

What is a decision rule

• So what is in ISO/IEC 17025:2017 ?……Reporting statements of conformity …

• Clause 7.8.6.1 states “When a statement of conformity

to a specification or standard for test or calibration is

provided, the laboratory shall document the decision rule

employed, taking into account the level of risk (such as

false accept and false reject and statistical assumptions)

associated with the decision rule employed and apply

the decision rule.”

John Wilson 2019 42

What is a decision rule

• So what is in ISO/IEC 17025:2017 ?……Reporting statements of conformity …

• Clause 7.8.6.2 requires that “the laboratory shall report

on the statement of conformity, such that the statement

clearly identifies:

• a) to which results the statement of conformity applies;

• b) which specifications, standard or parts thereof are met

or not met;

• c) the decision rule applied (unless it is inherent in the

requested specification or standard).”

John Wilson 2019 43

What is a decision rule• So breaking this down further

When statement of conformity to a specification or

standard for test or calibration is requested, the

laboratory shall:

• The key here is “when it is requested” which

implies it is requested by the customer.

• This means that “contract review” must take

place and a clear definition agreed on

BEFORE the job is started.

John Wilson 2019 44

What is a decision rule• During Contract review

The laboratory needs to be prepared to discuss

what the decision rule options are regarding the

compliance statement.

They also need to understand what the customer

may require and where the risk of “false accept” or

“false reject” lies so that they do not get drawn into

legal battles or compensation claims.

As with all statistical analysis, it is open to different

interpretations and care must be taken to ensure a

correct agreement between parties.

John Wilson 2019 45

What is a decision rule• During Contract review

There are many models which may be applied

and they will vary greatly depending on the

application.

You may be testing a new prototype where no

previous or group history is available or you

may be checking a widely manufactured and

accepted item against a manufacturer’s

specification. The approach will be very

different.

John Wilson 2019 46

What is a decision rule• Different approaches to each condition

John Wilson 2019 47

One instrument

over a long time One batch of the same model instrument

One reading (or set of readings on a prototype

What is a decision rule• Different approaches to each condition

• A knowledge of “Guard Bands” and the way they are

implemented is advisable.

John Wilson 2019 48

Nominal

Upper Limit

Lower Limit

New Limit

New Limit

Guard band

Guard band

What is a decision rule• Different approaches to each condition

• Guard bands and Uncertainties

John Wilson 2019 49

Nominal

Upper Limit

Lower Limit

Guard band Limit - upper

Guard band Limit - lower

What is a decision rule• ILAC G-8:09/2019 covers decision rules and

statements of conformity and tries to capture and explain in one document the new requirements of ISO/IEC 17025:2017.

• Topics covered in ILAC-G8 include:-o ISO/IEC 17025: 2017 references to decision rules and statements

of conformity

o Measurement Uncertainty and Decision Rule overview

o Guard Bands and Decision Rules

o Taking measurement Uncertainty into account

o Decision Rule Selection Flow Chart

o Decision Rule Documentation and Application

o Simple Checklist (for Labs and Assessors)

o Examples of Decision Rules

John Wilson 2019 50

What is a decision rule• Decision Rule Flow Chart (from ILAC-G8)

John Wilson 2019 51

John Wilson 2019 52

What is a decision rule• During Contract review

Once agreed and documented with the customer –

the work can continue.

John Wilson 2019 53

What is a decision rule• In the Lab… the laboratory shall:

a) document the decision rules employed taking into account the level of risk

associated with the decision rule employed (false accept and false reject

and statistical assumptions associated with the decision rule employed);

• Take the time to clearly document what has

been agreed and ensure that lab procedures

allow for the correct treatment of the

readings and uncertainties.

John Wilson 2019 54

What is a decision rule• In the Lab

… the laboratory shall:

a) document the decision rules employed taking into account the level

of risk associated with the decision rule employed (false accept and

false reject and statistical assumptions associated with the decision

rule employed);

• Might need to change test procedures to

cater for the decision rule that will be

applied. It has to be such that anybody in the

lab will know what is required and what the

limits are.

John Wilson 2019 55

What is a decision rule• In the report

… the laboratory shall:

a) document the decision rules employed taking into account the level

of risk associated with the decision rule employed (false accept and

false reject and statistical assumptions associated with the decision

rule employed);

b) apply the decision rule .

• The report must contain either the decision

rule used or a reference to a standard or

agreed decision rule

John Wilson 2019 56

What is a decision rule• In the report

… the laboratory shall:

a) document the decision rules employed taking into account the level

of risk associated with the decision rule employed (false accept and

false reject and statistical assumptions associated with the decision

rule employed);

b) apply the decision rule .

• The appropriate statement of conformity

which has been requested or required by a

standard.

John Wilson 2019 57

What is a decision rule• In the report

• There must be enough detail so that if

another lab is tasked with taking the same

measurements and using the same decision

rule, that they will come to the same

conclusion about a “pass” or “fail”

John Wilson 2019 58

What is a decision rule• The measurement decision RISK

… the laboratory shall:

a) document the decision rules employed taking into account the level

of risk associated with the decision rule employed (false accept and

false reject and statistical assumptions associated with the decision

rule employed);

• There are two classifications of

measurement decision risk.

• They are Specific and Global risk.

• These are different to the Consumer and

Producer risk already discussed.

John Wilson 2019 59

What is a decision rule• The measurement decision RISK

• If a laboratory only measures a single instrument and has no history of calibration for that serial number or has no information on the behaviour of that model number as a population, then it can be considered as having “meagre prior information” it is classified as Specific Risk.

• When calibration records are actively managed by the customer or laboratory for an instrument or population, they can be directed as Global Risk.

John Wilson 2019 60

What is a decision rule• The measurement decision RISK

• Similarly, as discussed earlier, there is the

consumer risk approach and the producer

risk.

• It follows that when using binary decision

rules to reduce consumers risk, it will always

increase the producers risk.

• JCGM 106 has a lot of information on these

various risk based approaches and shows

how to calculate the appropriate risk involved

with a specific decision rule.

John Wilson 2019 61

What is a decision rule• In Summary – it is the balance of risk

John Wilson 2019 62

What is a decision rule• The measurement decision RISK• Taken from ILAC-G8 (in this table U = U95)

John Wilson 2019 63

Appendix B of ILAC-G8 09/2019 give more examples.

What is a decision rule• In general

• This means that if a certificate or statement of conformance is required, a lot more detail is required to be addressed before the measurements are taken and then the Lab will need to be vey clear about the criteria used and this must be defined on the test report, certificate of conformance or statement of compliance.

• The following Checklist from ILAC-G8 09/2019 (Appendix A) gives a simple guideline to follow to ensure that all the key issues are covered.

John Wilson 2019 64

What is a decision ruleThe checklist from ILAC-G8 09/2019

APPENDIX A - Sample Checklist for meeting the requirements of ISO/IEC

17025:2017

a)Documentation and records reflecting customer

agreement requesting statement of conformity to a

specification or standard. (clause 7.1.3 [1])

b)Records of the selection of test limits and associated

tolerances and compatibility with customer

requirements (clause 7.1.3 [1])

John Wilson 2019 65

What is a decision ruleThe checklist from ILAC-G8 09/2019

APPENDIX A - Sample Checklist for meeting the requirements of ISO/IEC

17025:2017

c)A documented decision rule for calculating,

controlling, and reporting levels of risk associated

with statement of conformity. (clause 7.1.3 [1])

d)Documentation of laboratory personnel that include

the knowledge, skill, and authorization to apply the

decision rule and make statements of conformity.

(clause 6.2.6 c [1])

John Wilson 2019 66

What is a decision ruleThe checklist from ILAC-G8 09/2019

e)Documentation of the calculation or estimation of

level of risk and measurement uncertainty. (clause

7.8.6.1 [1])

f) Documentation of other supporting factors such as

statistical assumptions including type of risk (i.e.

specific or global) and measurement uncertainty.

(clause 7.8.6.1 [1]) NOTE: for more information on

specific and global risk see section 5.3 of this

document.

John Wilson 2019 67

What is a decision ruleThe checklist from ILAC-G8 09/2019

g)Documentation of the conformity assessment type

and statements of conformance. (clause 7.8.6.2 [1])

NOTE: for more info see section 4.2 of this document

h) Decision rule documentation to be included with

test and calibration records. (clause 7.8.6.2 [1])

John Wilson 2019 68

What is a decision rule• In general

• This is not so difficult if it is broken down in

simple logical steps.

• However it will require laboratories to

seriously consider how they will approach the

whole job when they are requested to supply

a certificate or statement of conformance

• Suggestion:- The laboratory documents a few

decision rules that they are able to offer their

customers.

John Wilson 2019 69

Practical ExampleAssume a 3½ digit DMM has a specification of

±(0.1% of Reading + 2 Digits)

on the 6 Vdc Range.

The tolerance limits for a nominal of 5.900 Vdc

would be:-

Lower Limit 5.892 Vdc & Upper Limit 5.908 Vdc

If zero guard band was applied, the risk of PFA

is < 50%

If a guard band of U95 is applied the risk of PFA

drops to < 2.5%

John Wilson 2019 70

Practical ExampleMass Verification

OIML R76-1:2006 (NAWIs) cl. 3.7.1

where it is required that

“…the standard masses used for the type

examination or verification of an instrument ….

shall not have an error greater than 1/3 of the

MPE. If they belong to class E2 or better, their

uncertainty is allowed to be not greater than

1/3 of the MPE of the instrument (the

tolerance)”

John Wilson 2019 71

Practical ExampleLiquids (other than water)

OIML R117-1:2007 Dynamic measuring systems

for liquids other than water Part 1: Metrological

and technical requirements A.2

Uncertainties of measurement:

When a test is conducted, the expanded

uncertainty of the determination of errors on

indications of volume or mass shall be less than

one-fifth of the maximum permissible error

(MPE) (the tolerance)

John Wilson 2019 72

Practical ExampleCivils

John Wilson 2019 73

• As we have a few Material Testing labs in

the audience lets consider a Mat Civils

problem. (forgive my lack of knowledge wrt Civils tests)

• What is involved with Specific Density

Tests ? Mass measurements

• Similarly with Marshall Control Tests.

• If no uncertainty is applied to any one

measurement, your risk of False Accept

approaches 50% as you get closer to the

tolerance limit.

• So when you then include multiple

measurements, the situation gets worse.

Practical Example• It should be clear from the previous examples

that in some cases there are clear

instructions on what to do.

• In other cases, the laboratory and the

customer will need to decided what decision

rule must be applied.

• In all cases, this relates to the Risk that

needs to be evaluated in an educated

discussion with the customer about the

probability of “False accepts” when a product

should fail and “False rejects” when a

product should pass.

John Wilson 2019 74

What is a decision rule

John Wilson 2019 75

It is all about communication!

ISO/IEC 17025:2017Statement or Certificate of Conformance

&Decision rules

By

John Wilson

Accreditation and Metrology Services (Pty) LTD

John Wilson 2019 76