Investigating Positively Biased Test Results In An ISO 17025 Accredited Chemistry Water Testing Laboratory Using Trend Analysis And Internal

Quality Control Procedures

Myburgh, A.

East Rand Water Care Company (ERWAT) P. O. Box 13106, Norkem Park, Kempton Park, 1631, South Africa

E-mail: annetjiem@erwat.co.za Tel: +27 11 929 7014 Fax: +27 11 929 7065

Abstract:

Having protocols in place to monitor trends in quality control and analytical test results is a requirement from ISO/IEC 17025. ERWAT Laboratory has various procedures and protocols in place to ensure valid test results and identify any trends. Bias can be defined as the deviation from the true test value. When evaluated, two test methods employed by ERWAT Laboratory – i.e. for analysing zinc (Zn) and chloride (Cl) in water and wastewater samples – showed positively biased results. The use of Certified Reference Materials (CRMs), in-house quality controls and participation in Proficiency Testing Schemes (PTS) all proved highly valuable in evaluating trends and identifying biased results. In this case study, daily QC testing and results from participation in PT schemes showed positively biased test results for both the Zn and Cl methods employed by ERWAT Laboratory. Finding the cause of the biased results is challenging and investigations are still underway. Ishikawa diagrams were used to identify possible causes, but none of the identified causes could be attributed to the biased results. However, because ERWAT Laboratory often test samples for compliance to national laws and municipal by-laws, biased results are considered significant and therefor both methods have temporarily been suspended. This paper describes how trend analyses of PTS and quality control data showed positively biased test results and discuss possible action plans needed to rectify the biased results. 1. Introduction: ERWAT Laboratory Services offers a wide range of chemical and microbiological analyses and provides key support to the operation of ERWAT Operations and ERWAT Commercial Business departments, the Ekurhuleni Metropolitan Council and a variety of private customers. The laboratory currently offers clients 34 chemical, 5 microbiological and 3 molecular biology ISO/IEC17025 accredited methods. On average, the laboratory performs 625 000 analyses per year making this a high through-put laboratory. For testing laboratories, assuring the quality of test results is very important and ISO/IEC 17025 requires accredited laboratories to: “…have quality control procedures for monitoring the validity of tests and calibrations undertaken. The resulting data shall be recorded in such a way that trends are detectable and, where practicable, statistical techniques shall be applied to the reviewing of results.”

The ERWAT Laboratory Quality Management System adheres to this requirement and several operating procedures and protocols are in place to monitor test and QC results to identify trends and ensure validity. In analytical testing, bias can generally be defined as the deviation, positively or negatively, of the average value from a series of measurements from the “true” value of the analyte under test [4]. In other words, bias is the numerically expressed degree of “trueness”. It is important to realize that bias is relative and the magnitude thereof is dependent on the method and material used to represent the true value of the analyte [3]. Biased test results can be problematic. ERWAT Laboratory often provide testing services to clients who use these results to assess compliance with municipal by-laws and national standards (e.g. SANS 241). Reporting biased results can inadvertently lead to non-compliances to these standards and laws while the actual (true) value of the analyte tested are within or exceeding stated limits. This paper describes how trend analyses of proficiency testing scheme (PTS) and quality control samples showed positively biased test results for two determinands – i.e. zinc (Zn) and chloride (Cl). It further aims to discuss how the cause of the biased results was investigated and possible action plans needed to rectify biased results. 2. Materials and Methods: 2.1 Analytical procedure for zinc (Zn) analysis Detection and quantification of Zn in water and wastewater samples were performed by inductively coupled plasma optical emission spectrometry (ICP-OES) using an Optima 5300 DV instrument (Perkin Elmer, US). In principle, water samples were dissociated into small droplets using a nebulizer. Droplets entering the plasma stage of the instrument were excited and upon exiting the plasma relaxed again to emit light. The wavelength (in nm) of the emitted light was measured. Wavelength of the emitted light is element species specific and was used to identify the metal(s) present in the water sample. 2.2 Analytical procedure for chloride (Cl) analysis The presence and quantity of Cl in water and wastewater samples were determined colorimetrically using an automated Aquakem 500 Photometric Analyzer (Thermo Scientific, US). Basically, chloride present reacts with mercuric thiocyanate to form a mercuric chloride complex. Any released thiocyanate reacts with iron (Fe) to form red ferric thiocyanate, and when measured (480nm) is proportional to the concentration of chloride present in the sample. 2.3 Quality control (QC) procedures employed by ERWAT Laboratory ISO/IEC 17025 requires accredited laboratories to constantly verify and monitor the quality and validity of test results. The ERWAT Laboratory Quality Management System adheres to this requirement and several operating procedures and protocols are in place to monitor the quality of test results. 2.3.1 Daily QC samples As part of ERWAT’s Quality Management System, standardized QC samples are analysed in combination with test samples. These QC samples are analysed at the start-up of

instrumentation and intermittently between batches of samples. Results from QC samples are plotted on a true value X-chart for routine control and comparison purposes. 2.3.2 Proficiency Testing Scheme (PTS) samples To independently verify test results, ERWAT Laboratory participate in proficiency testing schemes (PTS) provided by the South African Bureau of Standards (SABS). Participation is quarterly and to evaluate test results, Z-scores were calculated as follow:

� − ����� =Laboratorytestresult − Median(truevalue)

RobustStandardDeviation 2.3.3 Unknown samples In addition, “unknown” samples (i.e. analyte concentration is unknown to the analyst) were prepared on a three-monthly basis. The percentage (%) deviation from the known concentration (true value) was calculated as follows:

%��������� =Truevalueofanalyte − Testresultobtained

Truevalueofanalyte X100

2.3.4 Certified Reference Materials (CRMs) Lastly, Certified Reference Material (CRM) samples are purchased and analysed by ERWAT Laboratory on a 3-yearly basis. The specific concentration ranges of the analyte under test are specified by the manufacturer (certificate of analysis) and is traceable to international standards. 3. Results and Discussion: 3.1 Evaluation of QC data and recognition of biased test results Results generated from quality control strategies used by ERWAT Laboratory to monitor and verify analytical test results were evaluated to identify any possible trends as required by ISO/IEC 17025. Quality control procedures included the use of unknown samples (prepared in-house), the use of CRMs and participation in PT schemes. 3.1.1 Results from daily QC samples Daily QC samples are tested continuously while performing the Zn and Cl methods. In the figures below the results for daily QC samples are shown for Zn and Cl analyses respectively for the period January to December 2015. Figure 1 shows the results for daily QC samples used when performing Zn testing at a true value of 1.0mg/L Zn.

Figure 1: Daily QC results for Zn analysis at a true value level of 1.0mg/L (UWL – upper warning limit; UCL – upper control limit; LWL – lower warning limit; LCL – lower control

limit) For the daily QC testing (true Zn value at 1.0mg/L), results appeared to be positively biased with test results constantly higher than the expected 1.0mg/L. The average test result for the period was 1.02mg/L with some test values reaching the upper warning limit of 1.08mg/L. Figures 2 and 3 show the results obtained for daily QC samples with true values of 0.2mg/L and 8.0mg/L Zn.

Figure 2: Daily QC results for Zn analysis at a true value level of 0.2mg/L (UWL – upper warning limit; UCL – upper control limit; LWL – lower warning limit; LCL – lower control

limit)

0.9

0.95

1

1.05

1.1m

g/L

Date: Jan - Dec 2015

Zinc (Zn): Daily QC analysis - true value 1.0mg/L

Result

True valueUCL

UWL

LWL

LCL

0.14

0.19

0.24

mg/

L

Date: Jan - Dec 2015

Zinc (Zn): Daily QC analysis - true value 0.2mg/L

Result

True value

UCL

UWL

LWL

LCL

Figure 3: Daily QC results for Zn analysis at a true value level of 8.0mg/L (UWL – upper warning limit; UCL – upper control limit; LWL – lower warning limit; LCL – lower control

limit) Interestingly, results for daily QC testing using true Zn values of 0.2mg/L and 8.0mg/L appeared to be much less biased as shown in Figures 2 and 3. This suggests that the biased results at a true Zn value of 1.0mg/L might be a result of inadequate instrument calibration at that specific detection level. The results for daily QC samples used in Cl testing (at a true level of 10mg/L) is shown in Figure 4.

Figure 4: Daily QC results for Cl analysis (UWL – upper warning limit; UCL – upper control limit; LWL – lower warning limit; LCL – lower control limit)

7.7

7.75

7.8

7.85

7.9

7.95

8

8.05

8.1

8.15

8.2

8.25

8.3m

g/L

Date: Jan - Dec 2015

Zinc (Zn): Daily QC analysis - true value 8.0mg/L

Result

True value

UCL

UWL

LWL

LCL

7.5

8.5

9.5

10.5

11.5

12.5

mg/

L

Date: Jan - Dec 2015

True value

UCL

UWL

LWL

LCL

Result

Chloride (Cl): Daily QC analysis - true value 10mg/L

Results obtained when performing the daily QC testing for the Cl method were constantly higher than the expected true Cl value of 10mg/L. The same positively biased test results were obtained at a true Cl value of 20mg/L (data not shown). 3.1.2 Results from “unknown” samples The % deviation of results from the true value for the Cl and Zn analytical methods obtained when testing in-house prepared “unknown” samples are shown in Figures 5 and 6 respectively.

Figure 5: Deviation of test results from true Cl concentration in “unknown” samples

Figure 6: Deviation of test results from true Zn concentration in “unknown” samples The acceptable deviation from the true value, according to the ERWAT Quality Management System, was ≤10% (at 1mg/L Cl) and ≤30% (at 30mg/L Cl). Generally, when testing the in-house prepared “unknown” samples using the Zn and Cl methods, results did not deviate excessively from the true concentration. Test results only deviated outside the determined

-30.0

-20.0

-10.0

0.0

10.0

20.0

30.0

40.0

12 O

ct'1

2

7Feb

'13

April

'13

July

'13

Oct

'13

Apr'1

4

July

'14

Oct

'14

Jan'

15

Apr'1

5

July

'15

Oct

'15

Jan'

16

April

'16

% D

evia

tion

Date

Chloride (Cl): Unknown samples

Test range: ≥1.00mg/L

Test range: 0.1-0.9 mg/L

Acceptable limit (≥1mg/L)

Acceptable limit (0.1-0.9mg/L)

-40.0

-20.0

0.0

20.0

40.0

12 O

ct'1

2

16 Ja

n'13

April

'13

April

'13

Augu

st'1

3

Oct

'13

Oct

'13

Jan'

14

Apr'1

4

July

'14

Oct

'14

Jan'

15

Apr'1

5

July

'15

Oct

'15

Oct

'15

Jan'

16

Jan'

16

April

'16

April

'16

% D

evia

tion

Date

Zinc (Zn): Unknown samples

Test range: ≥1.00mg/L

Test range: 0.1-0.9 mg/L

Test range: 0.01-0.09mg/L

Acceptable limit (≥1mg/L)

Acceptable limit (0.1-0.9mg/L)

Acceptable limit (0.01-0.09mg/L)

acceptable limits when testing at low concentration ranges (i.e. 0.1-0.9mg/L for Cl and 0.01-0.09mg/L for Zn). No bias (positively or negatively) could be observed for the Cl or Zn methods. 3.1.3 Results from PTS samples Figures 7 and 8 shows the calculated Z-scores for Zn and Cl testing when participating in proficiency testing schemes.

Figure 7: Z-score results for Zn testing (ICP-OES) using PTS samples

Figure 8: Z-score results for Cl testing (Aquakem) using PTS samples According to ERWAT’s Quality Management System, a Z-score of ≤ 3 ≥ -3 is considered acceptable. From the above figures it’s clear that the Z-scores obtained for both Zn and Cl testing showed a positive bias reaching towards the upper acceptable limit. For Zn testing, positively biased Z-scores seemed to have occurred since participation in the PTS of October 2012, eventually exceeding the upper acceptable limit of 3 during participation in the PTS of

-4.00

-3.00

-2.00

-1.00

0.00

1.00

2.00

3.00

4.00

Jan'

12

Apr'1

2

Jul'1

2

Oct

'12

Jan'

13

Apr'1

3

Jul'1

3

Oct

'13

Jan'

14

Oct

'13

Jan'

14

Apr'1

4

Jul'1

4

Oct

'14

Jan'

15

Jul'1

5

Oct

'15

Jan'

16

Apr'1

6

Z-sc

ore

Date

Zinc (Zn): PTS samples

Sample 1

Sample 2

Sample 3

Acceptable upper limit

Acceptable lower limit

Mean

-4.00

-3.00

-2.00

-1.00

0.00

1.00

2.00

3.00

4.00

Mar

'12

Jun'

12

Sep'

12

Dec'

12

Mar

'13

Jun'

13

Sep'

13

Dec'

13

Mar

'14

Jun'

14

Sep'

14

Dec'

14

Mar

'15

Jun'

15

Sep'

15

Dec'

15

Mar

'16

Z-sc

ore

Date

Chloride (Cl): PTS samples

Sample1

Sample 2

Sample 3

Acceptable upper limit

Acceptable lower limit

True value

October 2013. While Z-scores for Cl testing showed a more gradual increase since participation in the PTS of September 2013, dropping back to a Z-score of 0 during the participation in the PTS of December 2014 and March 2015. 3.1.4 Results for CRMs Zinc and chloride CRMs were purchased and tested by ERWAT Laboratory during 2013. The certified value of the analyte under test are specified when purchasing the CRM. Results are shown in Table 1.

Table 1: Results obtained for CRM testing of Zn and Cl (2013)

Analyte: Certified value: (mg/L) Results Analyst 1: Deviation: Analyst 2: Deviation:

Zn 1060 1070 +0.9% 1060 0.0%

Cl 85.5 80.0 -6.5% 90.0 +5.2%

Cl 59.9 60.0 +0.1% 60.0 +0.1%

Cl 38.2 38.0 -0.5% 38.0 -0.5%

According to the ERWAT Quality Management System a deviation within 10% of the true value is considered acceptable. When testing the Zn and Cl CRMs, results never exceeded a 10% deviation from the true value. 3.2 Evaluation of observed trends and investigating the cause of biased results Graphical representation of quality control data assisted in identifying trends and subsequently the positively biased test results observed for the Zn and Cl methods. To further investigate the biased results, Ishikawa (fishbone) diagrams were prepared (Figure 9) showing possible causes and their effects on the methods.

Figure 9: Ishikawa (fishbone) diagram used to investigate biased Zn and Cl test results

mg/L

Purity of standards Glassware

Sample repeatability Regression

Accuracy of CRMs

Pipettes

Flasks

The accuracy of CRMs were verified and these CRMs were purchased from reputable suppliers. The quality of CRMs used could be confirmed as stated on the certificate of analyses provided. Both the ICP-OES and Aquakem instruments are routinely calibrated and serviced by reputable service providers. From calibration data, the regression of instrument calibration was evaluated. Both the calibration of the ICP-OES and Aquakem instruments showed a high correlation coefficient (r2 ≥ 0.995) indicating a linear relationship between variables. All glassware used by ERWAT Laboratory, and for the execution of the Zn and Cl methods, were validated. High quality glassware were used and the likelihood of glassware as the cause of the biased test results were very low. The quality and purity of the Zn standard solution could be verified. Reputable suppliers were used for the purchase of the Zn standard and the quality of the standard could be traced back to international standards. Also, the quality of reagents used to prepare the Cl standard could be verified. Analytical grade (AR) reagents were purchased from recognized suppliers and the certificate of analyses confirmed purity and quality. Furthermore, all standards and quality control solutions were prepared according to recognized methods (Standard Methods, 2011) using calibrated instruments and glassware. The likelihood of the standards used as a cause for the biased test results are this very low. 3.3 Significance of biased test results and possible future actions Quality control monitoring for both the Zn and Cl methods as employed by ERWAT Laboratory showed positively biased results. In some cases, biased results may be considered insignificant by the testing laboratory depending on the degree of deviation and the type of testing services provided. ERWAT Laboratory provide testing services to a variety of clients within the water and wastewater sectors of South Africa. Often, samples from clients are tested to determine compliance to national laws and municipal by-laws. For example, potable drinking water has to comply with the chemical and microbiological limits as set in the South African National Standard for Drinking Water (SANS241), while industrial effluent has to comply with the limits set by municipal by-laws. Regarding potential impacts on the aquatic environment, target ranges for various chemical parameters are defined by the South African Water Quality Guidelines [2]. The limits regarding Zn and Cl are shown in Table 2.

Table 2: Limits with regards to Zn and Cl according to SANS241 and SAWQG

Determinand: SANS241:2011 SAWQG for aquatic ecosystems (Vol. 7, 1996)

Zn ≤5mg/L ≤2mg/L

Cl ≤300mg/L —

Biased test results could result in the under or over estimation and reporting of the true value of the analyte under test in the particular sample. Considering the limits set by laws and by-laws, under or over reporting of test results could mean the difference between compliance and non-compliance with these limits and ultimately have significant consequences to clients served by the laboratory. As an ISO/IEC 17025 accredited laboratory, ERWAT thus recognizes the importance of accurately test results and considers any biases observed as highly significant. As a result, internal non-compliances were issued by the ERWAT Quality Department regarding the Zn and Cl testing methods and to date, these two methods have been suspended until the cause of the biased results can be determined and rectified. 4. Concluding Remarks: Generally, bias can be defined as the deviation (over time) of analytical test results from the true value of the analyte under test. ERWAT Laboratory recognized positively biased test results for Zn and Cl testing when evaluating quality control data for these two methods – in particular daily QC and PTS samples. The cause of the biased results are still not fully determined and needs further investigation. Because samples received by the laboratory are often tested for compliance to national laws and municipal by-laws, the impact of biased test results can have significant consequences to clients served by the laboratory. ERWAT Laboratory thus suspended the Zn and Cl testing using these particular instruments until the issue of biased results can be resolved. 5. References:

1. American Public Health Association (APHA), American Water Works Association (AWWA) & Water Environment Federation (WEF), Standard Methods for the Examination of Water and Wastewater, 21st Edition, 2005, Washington D.C.

2. Department of Water Affairs and Forestry (DWAF), South African Water Quality

Guidelines, Volume 7 (2nd Edition): Aquatic Ecosystems, 1996, Pretoria.

3. B. Flatlands, K. R. Friedrichs and S. Klenner “Differentiating between analytical and diagnostic performance evaluation with a focus on the method comparison study and identification of bias.” Veterinary Clinical Pathology 43(4), 2014, pp. 475 to 486

4. J. Tate and M. Panteghini “Standardisation – The theory and the practice.” Clinical

Biochemistry Reviews 28, 2007, pp. 127 to 130