PRACTITIONER’S REPORT

Evaluation of proficiency tests in microbiological analysis:enumeration of aerobic microorganisms

In Sun Hwang • So Ra Lee • Sooyeul Cho • Kab Ryong Chae •

Hyoung Jun Park • Ji Hyun Lee • Ok Soon Heo • Seung Yi Hong •

Woo-Seong Kim

Received: 12 July 2013 / Accepted: 19 December 2013 / Published online: 11 January 2014

� Springer-Verlag Berlin Heidelberg 2014

Abstract South Korea’s Ministry of Food and Drug

Safety (MFDS) has been developing programs for the

inspection and accreditation of food sanitation inspection

institutions. Food sanitation inspection institutions such as

MFDS regional offices, the Research Institute of Public

Health and Environment and authorized private service

providers in South Korea must participate in proficiency

testing (PT) programs to comply with the Food Sanitation

Act and MFDS Notification No. 2012-112. As the PT pro-

vider, the MFDS annually plans various microbiological

and chemical PT programs for foods, cosmetics, and phar-

maceutical products in accordance with ISO/IEC 17043.

The aim of this project was to evaluate the performance of

microbiological PT programs to ensure the quality of their

routine test results. The test materials used were freeze-

dried BioBalls from BTF Pty Ltd. Homogeneity and sta-

bility were investigated to assess the adequacy of the

selected test materials. This project also contains data from

inter-laboratory comparisons organized by MFDS in 2011

and 2012. More than 50 laboratories attended the PT pro-

gram and submitted their results. Laboratory results were

rated with z-scores according to the international standard

ISO 13528. The results from 2011 and 2012 revealed that all

participating laboratories had similar levels of proficiency.

Most of the participants received a rating of ‘‘Satisfactory.’’

Moreover, the percentage of participants who received a

rating of ‘‘Unsatisfactory’’ decreased from 3.5 % in 2011 to

2.0 % in 2012.

Keywords Proficiency testing � Microbiological PT �z-score � Inter-laboratory comparisons

Introduction

South Korea’s Ministry of Food and Drug Safety (MFDS)

launched official proficiency testing (PT) programs in the

early 1990s. The PT schemes had initially been developed

to comply with the requirements for food registration stated

in MFDS Notification No. 2012-112, and then gradually

expanded [1]. At first, MFDS only provided a PT program

for the detection of pathogens in food matrices, but in

2010, it started conducting inter-laboratory comparisons

for microbiological enumeration tests.

The aim of the inter-laboratory comparisons is to pro-

vide each laboratory the opportunity to prove and assess

the reliability of their analytical results. These PT programs

were designed to comply with the requirements of ISO/IEC

17043 (general requirements for the competence of pro-

viders of PT schemes) [2] and ISO 13528 (Statistical

methods for use in PT by inter-laboratory comparisons) [3].

The test materials used were BioBalls, a microbiological

reference material for cytometry and freeze-drying produced by

BTF Pty Ltd (Sydney, Australia). Homogeneity tests were per-

formed to ensure consistent numbers of microorganisms. Tests

were also performed to confirm the stability of the reference

material throughout the analysis, including during transport.

Participating public laboratories and authorized private

service providers conducted the enumeration of aerobic

microorganisms in accordance with the Korea Food Code

[4] and were asked to submit their results within 4 weeks.

I. S. Hwang � S. R. Lee � S. Cho � K. R. Chae �H. J. Park � J. H. Lee � O. S. Heo � S. Y. Hong � W.-S. Kim (&)

Advanced Analysis Team, Toxicological Evaluation and

Research Department, National Institute of Food and

Drug Safety Evaluation, Ministry of Food and Drug Safety,

Osong Health Technology Administration Complex,

187 Osongsaengmyeong2-ro, Yeonje-ri, Osong-eup,

Cheongwon-gun, Chungcheongbuk-do 363-700, Korea

e-mail: in-sun-h@hanmail.net

123

Accred Qual Assur (2014) 19:41–46

DOI 10.1007/s00769-013-1030-7

In this project, an evaluation of the test materials toge-

ther with the results of inter-laboratory comparisons of

microorganism enumerations is presented.

Materials and methods

Test materials

The BioBall, a small water-soluble ball that contains a

precise number of viable bacteria, is used for growth pro-

motion testing with nutrient agar. It is produced by BTF

Pty Ltd. (Sydney, Australia), a reference materials pro-

ducer accredited by the National Association of Testing

Authorities, Australia (NATA). For this study, the BioBalls

contained Escherichia coli American Type Culture Col-

lection (ATCC) 11775. BioBalls contained a precise

number of microorganisms. For each lot, a certificate of

analysis is provided containing the reference material data,

the culture information and the strain confirmation. Bio-

Balls from lot number B1746 were found to contain 17 384

colony forming units (CFUs) per BioBall with standard

deviation (SD) of 963.7 CFUs and those were manufac-

tured in 2011. BioBalls from lot number B1980 were found

to contain 8 068 CFUs (SD: 483.6 CFUs) per BioBall, and

those were manufactured in 2012. For all living bacteria

enumeration, nutrient agar (NA) provided by the supplier

was used.

Enumeration of aerobic bacteria

To determine the total number of aerobic bacteria in each

BioBall, a 1.0 g test solution was prepared by rehydrating a

BioBall in 1 ml of distilled water. This solution was then

serially diluted with distilled water and plated onto plate

count agar (PCA, Oxoid, Basingstoke, England) or dry

rehydratable film (3 M Petrifilm Aerobic Count Plates;

3 M Microbiology, St. Paul, MN, USA), which was then

incubated for 48 h at 37 �C in accordance with the Korea

Food Code [4]. All colonies were counted, and they all

were in countable range (15–150 colonies). Both BioBall

reference materials were investigated to obtain quantifica-

tion data for the use of PCA instead of NA.

Homogeneity and stability tests

Between-bottles homogeneity study was checked by enu-

merating aerobic microorganisms from ten samples using

the pour plate method and PCA; the analysis was done in

triplicate (n = 1093). The mean of measurements was

taken as reference value for the PT.

We performed short-term (48 h) stability tests with the

B1980 BioBalls and long-term (12 month) stability tests

with the B1746 BioBalls. The stability tests were per-

formed by enumerating aerobic microorganisms in 3

bottles per time-period in triplicate (n = 3 9 3). For the

short-term stability test, the bottles were placed inside an

ice box packaged with a temperature sensor, 0.6 kg dry ice,

and a 0.7 kg ice pack and the box was monitored for inner

box temperature changes for 48 h. Three bottles of Bio-

Balls were analyzed after 12 h, 24 h and 48 h storage in ice

boxes containing dry ice and ice packs. The long-term

stability test was evaluated after storage of the bottles at

-20 �C for 12 months. The data obtained were compared

to initial data which were obtained for the homogeneity

study of the reference materials.

PT program

Each participant received two bottles of test materials.

They were asked to submit a result of enumeration of

aerobic microorganisms within 4 weeks after the sample

was distributed. Sample bottles were shipped with dry ice

and ice packs to the participants. The laboratories were

allowed to use either PCA or Petrifilm for enumeration

presented in the Korea Food Code. From the submitted

results, there was no significant difference in the number of

colonies obtained using PCA and that obtained using Pet-

rifilm. In total, 57 laboratories participated in the 2011 PT

program, which used materials from lot number B1746 and

started on 6 July 2011. In the 2012 PT program, test

materials from lot number B1980 were distributed on 3

July 2012, and a total of 51 laboratories participated.

Statistical assessment of PT results

The results submitted by participants were statistically

analyzed as per ISO 13528 [3, 5, 6]. According to ISO

13528, the assigned value, xa, may be determined by one of

the following methods:

• formulation (i.e., value assignment on the basis of

proportions used in a solution or other mixture of

ingredients with known analyte content);

• certified value(s) for the certified reference material

(CRM) used as the test material;

• direct comparison of the PT test material with CRMs;

• consensus value from expert laboratories; or

• consensus value from participants.

For this study, consensus values from participants were

used to determine the assigned values, as it is difficult to

obtain certified values for quantitative tests of microor-

ganisms, and enumeration is not a higher-order or primary

method.

The assigned values were then used in combination with

the SD for proficiency, rp, to calculate a z-score for each

42 Accred Qual Assur (2014) 19:41–46

123

result. In normal circumstances, about 95 % of z-scores

will lie in the range of |z| B 2, which indicates a rating of

‘‘Satisfactory.’’ Scores in the range of 2 \ |z| \ 3, which

indicates a rating of ‘‘Questionable,’’ are expected to occur

at a rate of 1/20. Scores in the range of |z| C 3, which

indicate a rating of ‘‘Unsatisfactory,’’ are expected to occur

at a rate of about 1/300 [7].

Participants’ z-scores were calculated as: z = (log10

x - log10 xa)/log10 rp, where

• x denotes the participant’s reported result,

• xa denotes the consensus value (average value from

participants)

• and rp denotes the SD for the proficiency test.

Results and discussion

Homogeneity and stability of test materials

The test materials were assessed for homogeneity and

stability before being distributed to the participants [2, 3,

5]. Homogeneity was checked by using ten samples. Bio-

Balls from lot B1746 had been assigned a mean value of 17

384 CFUs per BioBall by the supplier; in contrast, our

homogeneity tests show a mean value of 16 770 CFUs per

BioBall on PCA (SD: 961.3 CFUs; Table 1). BioBalls from

lot B1980 were assessed by the supplier to have a mean

value of 8 068 CFUs per BioBall; in contrast, our homo-

geneity tests show a mean value of 7 890 CFUs per BioBall

(SD: 529.6 CFUs) on PCA (Table 2). The difference

between the supplier’s mean values and ours may be

because of the use of different media. However, the dif-

ferences in relative standard deviation (RSD) between the

two media are not significant. The RSD on PCA among

bottles of BioBalls from lot B1746 is 5.7 % and among

bottles of BioBalls from lot B1980 is 6.7 %. The RSD

results show that the test materials are adequately homo-

geneous and suitable for use in PT (Tables 1, 2).

The stability of the BioBalls was checked in accor-

dance with ISO Guide 35 to determine whether the PT

samples would be stable during the analytical process,

including during transport [7, 8]. Short-term stability tests

were performed with the BioBalls from lot B1980. No

decrease in mean number of CFUs is seen for each time-

period tested (Fig. 1). We also performed long-term sta-

bility tests with BioBalls from lot B1746. No significant

difference in mean number of CFUs is seen after the

12-month storage period (Fig. 2). SDs from the stability

tests were not included in the uncertainty factors because

both the RSD from the long-term stability test of BioBalls

from lot number B1746 (4.4 %) and that from the short-

Table 1 Results for aerobic colonies count on plate count agar from

selected bottles of BioBalls from lot number B1746

No. Replicate Mean

1 2 3 x log10 x

1 17 200 16 000 16 200 16 467 4.22

2 15 800 15 900 14 200 15 300 4.18

3 15 700 16 400 16 100 16 067 4.21

4 17 900 16 900 15 100 16 633 4.22

5 14 200 15 700 18 100 16 000 4.20

6 18 200 17 300 16 000 17 167 4.23

7 17 900 18 600 18 000 18 167 4.26

8 16 400 17 100 17 300 16 933 4.23

9 18 600 18 900 17 800 18 433 4.27

10 15 200 16 200 18 200 16 533 4.22

The mean of the means 16 770 4.22

The standard deviation of the means 961.3

The relative standard deviation (%) 5.7

To assess the homogeneity of BioBalls from lot number B1746

(manufactured in 2011), enumeration on plate count agar for ten

samples was performed in triplicate. The mean number of CFUs per

BioBall is represented by x

Table 2 Results for aerobic colonies count on plate count agar from

selected bottles of BioBalls from lot number B1980

No. Replicate Mean

1 2 3 x log10 x

1 8 500 9 100 7 900 8 500 3.93

2 8 400 8 700 6 700 7 933 3.90

3 7 000 6 900 7 700 7 200 3.86

4 6 900 8 100 7 100 7 367 3.87

5 6 900 7 300 8 100 7 433 3.87

6 7 600 7 800 7 400 7 600 3.88

7 6 400 8 800 8 900 8 033 3.90

8 8 400 8 500 7 500 8 133 3.91

9 9 100 8 700 8 900 8 900 3.95

10 7 900 7 800 7 700 7 800 3.89

The mean of the means 7 890 3.90

The standard deviation of the means 529.6

The relative standard deviation (%) 6.7

To assess the homogeneity of BioBalls from lot number B1980

(manufactured in 2012), enumeration on plate count agar for ten

samples was performed in triplicate. The mean number of CFUs per

BioBall is represented by x

Accred Qual Assur (2014) 19:41–46 43

123

term stability test of BioBalls from lot number B1980

(1.7 %) were much smaller than the SD obtained from a

between-bottle homogeneity study (5.7 % and 6.7 %,

respectively).

According to the stability tests, the aerobic microor-

ganisms in the BioBalls were stable for up to 48 h in an ice

box with dry ice and ice packs, and for up to 12 months

when stored at -20 �C. Therefore, we designed the ship-

ping conditions an ice box with 0.6 kg dry ice and a 0.7 kg

ice pack (which is the same as the conditions for the short-

term stability test) and storage instructions for the partici-

pants based on the results of the short- and long-term

stability tests.

Statistical assessment of PT results

Results submitted by participants were statistically ana-

lyzed as per ISO 13528 [3, 5, 6]. Whereas the logarithm of

the consensus value from the participants’ 2011 PT results

is log10 xa = 4.189, the reference value xh from the

homogeneity test for BioBalls from lot number B1746

leads to log10 xh = 4.225 (Table 3). The difference

between these two values is due to a misunderstanding on

the part of the participant laboratories of the instructions

for the assay. Participants in the 2011 PT program should

have used all the solution after rehydrating with 1.1 ml of

distilled water; however, they actually used only 1.0 ml of

the fluids after rehydration. To eliminate the confusion, in

the 2012 PT program, we instructed participants to use all

of the rehydrated solution after rehydrating with 1.0 ml of

distilled water. As a result, there were no significant dif-

ferences between the reference value and the consensus

value for the materials from lot number B1930 (Table 3).

Two different approaches to z-score calculation were

compared: one using the reference values and the other

using the consensus value. As the reference values did not

assigned with higher-order methods or primary methods

and the consensus mean values derived from the PT

participants were sufficiently reliable as there were more

than 50 participants, we used the participants’ consensus

values (averages), which are commonly used by many

other PT providers [5]. The SD used in the PT assessment

was derived from the participants’ results in the same

round. Figures 3 and 4 graphically present participants’ z-

scores.

From the 2011 PT program, 94.7 % of results are rated

as ‘‘Satisfactory,’’ 1.8 % of results are rated as ‘‘Ques-

tionable,’’ and 3.5 % of results are rated as

‘‘Unsatisfactory’’ (Table 4). From the 2012 PT program,

94.1 % of results are rated as ‘‘Satisfactory,’’ 3.9 % of

results are rated as ‘‘Questionable,’’ and 2.0 % of results

are rated as ‘‘Unsatisfactory’’ (Table 4).

Fig. 1 Results of the 48-h stability test on BioBalls from lot number

B1980 using PCA (n = 3 9 3). Mean values of log10 x are shown.

The crossed-circles indicate the average value of three BioBalls. Each

dot indicates mean of triplicate measurements per individual BioBall.

The vertical bars indicate standard error of the mean

Fig. 2 Results of the 12-month stability test on BioBalls from lot

number B1746 using PCA (n = 3 9 3). The long-term stability was

tested after 12 months. Mean values of log10 x are shown. The

crossed-circles indicate the average value for three BioBalls. Each

dot indicates the mean of triplicate measurements per individual

BioBall. The vertical bars indicate standard error of the mean

Table 3 Summary statistics for the calculation of z-scores

Lot number B1746 in

2011

Lot number B1930 in

2012

Reference

value

Consensus

value

Reference

value

Consensus

value

Assigned value

(log10 x)

4.225 4.189 3.897 3.902

Standard deviation

(log10 rp)

0.033 0.029

Relative standard

deviation (%)

0.775 0.782 0.740 0.739

44 Accred Qual Assur (2014) 19:41–46

123

As the inspection and accreditation agency for food

sanitation inspection institutions, the MFDS required root

cause analysis for each inspection institution with a result

rated as ‘‘Questionable’’ or ‘‘Unsatisfactory.’’ Laboratories

that had a z-score greater than two could do their own root

cause analyses. From these analyses, it seems the causes

could be attributed to quantitative errors due to poor

operating skills when performing the procedure and errors

due to poor calibration of some of the experimental

equipment such as pipettes, balances, clean bench, and

incubator. Each laboratory was required to take corrective

action based on the root cause analyses so as not to

encounter the same problems again. Moreover, the MFDS

gave the laboratories experimental advice and also per-

formed overall field-checks of their laboratory systems.

Conclusions

The purpose of the project was to assess the reliability of

each laboratory’s analysis result via an inter-laboratory

comparison and to give feedback for better performance.

All the evaluations and statistical assessments observed

international standards. The results from both the 2011 and

2012 PT programs show that most of the laboratories that

participated are skilled in the enumeration of aerobic

microorganisms. Each laboratory whose result was rated

‘‘Questionable’’ or ‘‘Unsatisfactory’’ performed self-cor-

rection based on their root cause analyses so as not to

encounter the same problems. The MFDS acted as a

facilitator and gave these laboratories field-check services

and further education.

Acknowledgments This Project was supported by the Korea Food

and Drug Administration in 2011 and 2012. The authors wish to thank

Euijin Hwang, Ph.D. in the Korea Research Institute of Standards and

Science for his advice concerning assessment of PT results.

References

1. MFDS notification No. 2012-112. Accessed 16 Nov 2012

2. ISO/IEC 17043 (2010) Conformity assessment—general require-

ments for proficiency testing

3. ISO 13528 (2005) Statistical methods for use in proficiency testing

by inter-laboratory comparisons

4. Korea Food Code. Article 10. Analytical methods. Accessed 1 Feb

2012

5. Thompson M, Ellison SLR, Wood R (2006) The international

harmonized protocol for the proficiency testing of analytical

chemistry laboratories. Pure Appl Chem 78:145–196

-4

-3

-2

-1

0

1

2

3

4

z-sc

ore

Participants

4.25

4.19

4.12

Fig. 3 Histogram of z-scores

for the 2011 PT scheme. The z-

scores for the 2011 PT were

calculated by using the means

and standard deviations from

data from 57 participants. One

participant result was rated as

‘‘Questionable’’ and 2 other

participant results as

‘‘Unsatisfactory’’

-4

-3

-2

-1

0

1

2

3

4

z-sc

ore

Participants

3.96

3.84

3.90

Fig. 4 Histogram of z-scores

for the 2012 PT scheme. The z-

scores for the 2012 PT were

calculated by using the means

and standard deviations from

data from 51 participants. Two

participant results were rated as

‘‘Questionable’’ and another

participant result as

‘‘Unsatisfactory’’

Table 4 Percentages of participants’ z-scores

|z| B 2 2 \ |z| \ 3 |z| C 3 Total number of participants

2011 94.7 1.8 3.5 57

2012 94.1 3.9 2.0 51

Accred Qual Assur (2014) 19:41–46 45

123

6. ISO 5725-5 Accuracy (trueness and precision) of measurement

methods and results—part 5: alternative methods for the

determination of the precision of a standard measurement

method

7. EURACHEM/CITAG guide, selection, use and interpretation of

proficiency testing (PT) schemes, (2011), 2nd English edition

8. ISO Guide 35 (2006) Reference materials—general and statistical

principles for certification

46 Accred Qual Assur (2014) 19:41–46

123