method comparison chemometrics department of chemical pathology, university of pretoria, dr r...

Method comparisonMethod comparison

ChemometricsChemometrics

Department of Chemical Pathology, University of Pretoria,

Dr R Delport 2003

Performance characteristics that are taken into account:

Precision, Accuracy, Interference, Working Range, andDetection Limit.

http://www.westgard.com/lesson21.htm


To carry out a good method validation study, you need to do the following:

• Define a quality requirement for the test in the form of the amount of error that is allowable, preferably an allowable total error,

• Select appropriate experiments to reveal the expected types of analytical errors,

• Collect the necessary experimental data,

• Perform statistical calculations on the data to estimate the size of analytical errors,

• Compare the observed errors with the defined allowable error, and

• Judge the acceptability of the observed method performance.



An experimental plan can be formulated by:

• Recognizing the types of errors that need to be assessed for this test and method,

• Identifying the appropriate experiments and the amount of data needed to estimate those types of errors, then

• Organizing these experiments to perform the quick and easy ones first and the ones taking more time and effort last.



Types of errors

• Imprecision or random errors,

• Inaccuracy, bias, or systematic errors, which can be of two types

•Constant systematic error or

•Proportional systematic error.



The dashed line in the middle of the figure represents ideal method performance where the test method and the comparative method give exactly the same results.

Note that these results will also be subject to the random error of the method, therefore the actual data points would scatter about the line as illustrated in the figure. The range of this scatter above and below the line provides some idea of the amount of random error that is present.

The bottom line in the figure shows the effect of a proportional systematic error, where the magnitude of the error increases as the test result gets higher.

The top line shows the effect of a constant systematic error, where the whole line is shifted up and all results are high by the same amount.



Experiments for estimating analytical errors



Description of different experiments.

Replication experiment

• Provides information about random error

• Is performed by making measurements on a series of aliquots of the same test samples within a specified period of time, usually within an analytical run, within a day, or over a period of a month.

• Preliminary experiment involves determining within-run imprecision.

• Final experiment requires at least 20 working days to provide estimate of the total imprecision, which includes within and between run components.




Interference experiment

• Provides information about the constant systematic error caused by the lack of specificity of the method.

• One test sample is prepared by adding the suspected material to a sample containing the analyte.

• A second aliquot of the original sample is diluted by the same amount with solvent, then both samples are analyzed by the test method and the difference determined.





Recovery experiment

• Provides information about the proportional systematic error caused by a competitive reaction.

• Test sample is prepared by adding a standard solution of the analyte being tested to an aliquot of a patient specimen.

• A baseline sample is prepared by adding an equal amount of the solvent used for the standard solution to a second aliquot of the same patient specimen.

• The two samples are then analyzed by the test method and the amount recovered is compared to the amount added. http://www.westgard.com/lesson21.htm




Comparison of methods experiment

• Is primarily used to estimate the average systematic error observed with real patient samples.

• Can also reveal the constant or proportional nature of that error.

• A series of patient specimens are collected and analyzed by both the test method and a comparative analytical method.

• The results are compared to determine the differences between the methods, which are the analytical errors between the methods.





Get the method working and establish an operating protocol.

• Set up the instrument, prepare the reagents, calibrate the methods, and obtain results from test samples.

• Check the standards and be sure the method is properly calibrated, otherwise calibration errors will show up throughout the experimental studies.



Walking tour of the plan.


Determine the working range.

• The working range will vary from test to test and must be defined as part of the specifications for the method.

• Check by analyzing a series of solutions, in duplicate or triplicate, covering the concentrations range of interest.

• If detection limit is a critical characteristic, it may be assessed at this time or in the next phase of preliminary experiments.





Determine within-run imprecision, recovery, and interference.

• The replication experiment might include 20 samples of two or three materials whose concentrations closely match the medical decision levels of interest for the tests.

• Interference experiments should test common problems such as hemolysis, lipemia, and high bilirubin.

• Recovery experiments assess whether there are any competitive reactions due to the matrix or other materials in the native specimens.





Comparison of methods experiment.

• Fresh patient specimens, and stored specimens.

• Minimum of 40 well-chosen patient samples should be tested over a minimum of 5 working days.

• Distributed one-third in the low to low-normal range, one-third in the normal range, and one-third in the high abnormal range.

• Method acceptability should be judged on the basis of the sizes of the random, systematic, and total analytical errors.






• Used as initial graph if the two methods are expected to show one-to-one agreement,

• Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure.





Difference plot


• Used as initial graph if the two methods are expected to show one-to-one agreement,

• Displays the difference between the test minus comparative results on the y-axis versus the comparative result on the x-axis, such as shown in the accompanying figure.

• Differences should scatter around the line of zero differences, half being above and half being below.

• Repeat measurements if indicated.





Difference plot


• The correlation coefficient is a statistic that is almost always calculated and reported to describe the results from a comparison of methods study.

• The new or "test" method values are plotted on y-axis and comparisonvalues on x-axis.





Correlation coefficient



• A value of 1.000 indicates perfect correlation between the results of two methods.

• Other statistics (such as slope, intercept, and standard deviation of the residuals) can also be calculated from the same data to estimate the size of errors occurring between the methods.




Random error, RE, or imprecision

• Can be either positive or negative.

• Direction and exact magnitude cannot be predicted.

• Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements.



Analytical errors

Random error

Random error, RE, or imprecision

• Can be either positive or negative.

• Direction and exact magnitude cannot be predicted.

• Imprecision is quantitated by calculating the standard deviation (SD) from the results of a set of replicate measurements.

• As the SD often increases as the concentration increases, the coefficient of variation (CV) is calculated to express the SD as a percentage of the mean concentration from the replication study.

• Maximum size of a random error is commonly expressed as a 2 SD or 3 SD estimate to help understand the potential size of the error that might occur.



Analytical errors

Random error

Systematic error, SE, or inaccuracy

• Is always in one direction.

• A systematic shift displaces the mean of the distribution from its original value.

• In contrast to random errors that may be either negative or positive and whose direction can not be predicted, systematic errors are in one direction and cause all the test results to be either high or low.



Analytical errors

Systematic error


• How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment.

• Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis.



Analytical errors

Systematic error


• How high or how low can be described by the bias, which is calculated as the average difference, or the difference between averages, between the value by the "test" method and a "comparative" method in a comparison of methods experiment.

• Alternatively, the expected systematic difference may be predicted from the equation of the line that best fits the graphical display of test method values on the y-axis vs comparative method values on the x-axis.

• SE may stay the same over a range of concentrations, in which case it can also be called constant error, or it may change as concentration changes, in which case it can be called proportional error.



Analytical errors

Systematic error

Total Error, TE,

• It is the net or combined effect of random and systematic errors, as shown in the accompanying figure.



Analytical errors

Total error

Total Error, TE,

• It is the net or combined effect of random and systematic errors, as shown in the accompanying figure.

• It represents a "worst-case" situation, or just how far wrong a test result might be due to both random and systematic errors.

• Because laboratories typically only make a single measurement for each test, that measurement can be in error by the expected SE, or bias, plus 2 or 3 SD, depending on how you quantitate the effect of RE.



Analytical errors

Total error

ChemometricsChemometricsanalysed with: Analyse-It + Clinical Laboratory v1.40

Test Bias plots NCCLS EP9-A example dataset Method: Comparative v Test

Performed by Analyse-it Software, Ltd. Date 1 February 1999

n 40

Bias -0.1

95% CI -2.3 to 2.2

95% limits of agreement 95% CI

Lower -13.7 -17.4 to -10.0

Upper 13.6 9.9 to 17.3

Identity line A=B

0

50

100

150

200

250

300

0 100 200 300

Mean of Method - Com parative

Mea

n o

f M

eth

od

- T

est


Zero bias

-15

-10

-5

0

5

10

15

0 100 200 300

Mean of Method - Comparative

Dif

fere

nce

bet

wee

n m

eth

od

s

Identity line A=B

0

50

100

150

200

250

300

0 100 200 300

Mean of Method - Comparative

Mea

n o

f M

eth

od

- T

est

0 5 10 15


Zero bias

-15

-10

-5

0

5

10

15

0 100 200 300

Mean of Method - Com parative

Dif

fere

nce

bet

wee

n m

eth

od

s (%

)

method comparison chemometrics department of chemical pathology, university of pretoria, dr r...

Documents

chemometrics slide

systematic errors

test method

analytical errors http

random errors

chemometrics experiments

proportional systematic

error increases