external quality assessment of laboratories performing

of 50

External Quality Assessment of laboratories Performing SARS-CoV-2 Diagnostics for the Dutch Population, February 2021

of 50

Colophon © RIVM 2021 Parts of this publication may be reproduced, providedacknowledgement is given to the: National Institute for Public Health and the Environment, and the authors, title and year of publication are cited. John Sluimer1,* Gabriel Goderski1,* Sharon van den Brink1 Lisa Wijsman1 Chantal Reusken1 Marion Koopmans2 Richard Molenkamp2 Adam Meijer1 * Equal contribution 1. National Institute For Public Health and The Environment(RIVM), Centre for Infectious Diseases Research, Diagnostics and laboratory Surveillance, Antonie van Leeuwenhoeklaan 9, 3721 MA Bilthoven, The Netherlands. 2. Erasmus Medical Centre Erasmus MC), Department Viroscience, Dr. Molewaterplein 40, 3015 GD Rotterdam AM coordinated the project. AM, GG and JS conceptualized thestudy, developed and produced the panels and reporting system. SvdB and LW performed reference PCR analysis. JS performed the analysis of the reported data and wrote the initial version of the report. AM critically reviewed the data analysis and report and JS together with AM wrote the final version. Corresponding author Adam Meijer [email protected] This investigation was performed by order, and for the account, of the Ministry of Health, Welfare and Sport, project number V/190002/01/PR.

This is a publication of the: National Institute for Public Health and the Environment P.O. Box 1 | 3720 BA Bilthoven The Netherlands www.rivm.nl/en

of 50

Contents Colophon ................................................................................................................................................. 2

1. Introduction ........................................................................................................................................ 6

2 Materials and methods ........................................................................................................................ 6

2.1 Approach ....................................................................................................................................... 6

2.2 Contents of LEQA2 panel .............................................................................................................. 6

2.3 Scoring the workflows ................................................................................................................... 9

3. Results ............................................................................................................................................... 10

3.1 Aggregated overview .................................................................................................................. 10

3.1 Used volumes, equipment, kits and reagents ............................................................................. 11

3.2 Target genes used for RT‐PCR or other NAAT ............................................................................. 17

3.3 Performance of the workflows ................................................................................................... 19

3.4 Comparing results LEQA2 to LEQA1 ............................................................................................ 31

4. Discussion and conclusion ................................................................................................................ 36

5. References ........................................................................................................................................ 38

6. Supplemental material ...................................................................................................................... 39

6.1 Results obtained per target gene per sample ............................................................................. 39

6.2 Participating laboratories ............................................................................................................ 49

of 50

Summary

Background

Since January 2020 many workflows for molecular diagnostics of SARS‐CoV‐2 were implemented and

checked for performance using specificity and sensitivity panels distributed by the National Institute

for Public Health and the Environment (RIVM). Although panels have been largely similar in load

components for checking SARS‐CoV‐2 Nucleic Acid Amplification Tests (NAAT) performance, they were

not exactly similar in constituents. Because of this heterogeneity in the past, the fact that patchy

quality checks were implemented only when workflows change or laboratories were added to the

network, and because it is considered important by the COVID‐19 WHO reference laboratories at RIVM

and Erasmus Medical Centre (Erasmus MC) and the Dutch Ministry of Health, Welfare and Sport

(‘Dienst Testen’) that the performance of the network of COVID‐19 molecular diagnostic labs is

checked as a whole regularly, a National External Quality Assessment (EQA) (Landelijk EQA; LEQA)

program has been developed. This program consists of three rounds of LEQA. This report includes the

second round of the LEQA program.

Objective

The goal of this LEQA round is to inventorize the quality of the Dutch SARS‐CoV‐2 diagnostics field,

using a panel that consists of 10 simulated clinical specimens, containing heat inactivated SARS‐CoV‐

2, including one variant of concern (VOC) B.1.1.7; 20B/501Y.V1, or other respiratory viruses or genetic

material. Each of the laboratories was asked to conduct molecular detection of SARS‐CoV‐2 according

to their workflows normally used for SARS‐CoV‐2 diagnostics.

Materials and Methods

In January 2021 the LEQA panel was produced and pre‐tested at the RIVM. After obtaining the correct

results per sample, all laboratories performing SARS‐CoV‐2 diagnostics in the Dutch network were

contacted and notified of the distribution of the panel in the first week of February 2021. A number

of workflows, especially the molecular point of care (mPOCT) ones, use expensive cartridges or

pouches of which laboratories only receive a limited number every week. Therefore laboratories with

limited resources that wanted to test their workflows were asked to indicate so. Then they were sent

an email to limit testing for these workflows to samples 3, 5, 8 and 9. Laboratories were asked to

report their results via an online form. Workflows were given a score of 8 for 100% correct results for

the 8 core samples and reduced by 1 point per sample for an incorrect result and 0.5 points for a result

“Indeterminate”, “Equivocal” or “Inconclusive” for a core SARS‐CoV‐2 positive sample. When a

workflow tested the reduced panel (containing samples 3, 5, 8 and 9), 2 points per sample for an

incorrect result and 1 point for a result “Indeterminate”, “Equivocal” or “Inconclusive” for a sample

were subtracted from the maximum score.

Results

Out of 180 workflows reported, 148 scored a 100% correct score for all 8 core specimens (8 points)

and thus met all criteria set for reliable SARS‐CoV‐2 diagnostics, 18 scored between 7‐7.5, making it

likely that only minor adjustments need to be made to meet all criteria and 16 workflows scored a 6.5

or lower, indicating that a lot of improvements still need to be made for these workflows to be reliable

for SARS‐CoV‐2 diagnostics in clinical diagnostic settings and surveillance. For the SARS‐CoV‐2 negative

core samples one false positive result was reported and two workflows gave indeterminate, equivocal

or inconclusive results for non‐SARS‐CoV‐2 containing samples. For the SARS‐CoV‐2‐containing core

samples, false negative results (n=23) and false indeterminate, equivocal or inconclusive results (n=12)

were reported. Some workflows reported a negative result for SARS‐CoV‐2 presence in SARS‐CoV‐2‐

of 50

containing samples due to cut‐off values used in the assay; up to 1.5% of each SARS‐CoV‐2 positive

sample (range: 0% – 1.5%) was reported as SARS‐CoV‐2 negative despite one (or more) of the target

genes against SARS‐CoV‐2 in the assay giving a Ct value. The VOC was detected by 176/177 (99.4%)

workflows. Despite the wide variety of kits, equipment and enzymes that are used in the implemented

workflows, the influence on the quality of molecular diagnostics for SARS‐CoV‐2 was limited.

Compared to the results obtained during the first round of quality control in this program (LEQA1), no

difference in overall quality of the SARS‐CoV‐2 diagnostic laboratory network was detected.

Conclusions

Overall the workflows used for SARS‐CoV‐2 diagnostics perform very well and laboratories using them

provide a reliable network. A small number of workflows should be further optimized to achieve full

potential. The Dutch SARS‐CoV‐2 diagnostics laboratory network performs on a very high level with

the vast majority of workflows detecting the core SARS‐CoV‐2 containing specimens correctly,

including VOC B.1.1.7; 20B/501Y.V1 that has become the major strain (up to 82%) in circulation by

week 9/2021 in The Netherlands. The wide variety of kits, equipment and enzymes used in the Dutch

SARS‐CoV‐2 diagnostic field do not affect adversely the quality of diagnostics. Instead, it allows for

great flexibility during times of shortages in supplies and likely improves the capacity to detect possible

future variants of SARS‐CoV‐2.

of 50

1. Introduction Since January 2020 many workflows for molecular diagnostics of COVID‐19 were implemented and

checked for performance using specificity and sensitivity panels distributed by the National Institute

for Public Health and the Environment (RIVM). Although panels have been largely similar in load

components for checking SARS‐CoV‐2 Nucleic Acid Amplification Tests (NAAT) performance, they

rapidly varied from SARS‐CoV‐1 RNA initially, to SARS‐CoV‐2 RNA, followed by SARS‐CoV‐2 whole heat

inactivated virus particles, depending on when materials became available. Because of this

heterogeneity in the past, the fact that patchy quality checks were implemented only when workflows

changed or laboratories were added to the network, and because it is considered important by the

COVID‐19 WHO reference laboratories at RIVM and Erasmus Medical Centre (Erasmus MC) and the

Dutch Ministry of Health, Welfare and Sport (‘Dienst Testen’) that we check the performance of the

network of COVID‐19 molecular diagnostic labs as a whole regularly, a National External Quality

Assessment (EQA) (Landelijk EQA; LEQA) program has been developed. This program consists of three

rounds of EQA. In the first week of February 2021 the second round of EQA panels was distributed to

all laboratories performing SARS‐CoV‐2 diagnostics on clinical samples derived from Dutch patients.

This panel consisted of 10 simulated clinical specimens that contained either heat inactivated SARS‐

CoV‐2, including one variant of concern (VOC) B.1.1.7; 20B/501Y.V1 that has become the major strain

(up to 82%) in circulation by week 9/2021 in The Netherlands [1], or other respiratory viruses or

genetic material. Each of the laboratories was asked to conduct molecular detection of SARS‐CoV‐2

on this panel according to their workflows normally used for SARS‐CoV‐2 diagnostics. All data had to

be reported back to the RIVM using an online reporting form.

2 Materials and methods 2.1 Approach Except for the VOC containing sample the LEQA panel was produced at the RIVM in October 2020 and

pretested at the RIVM and Erasmus MC. Both centers obtained similar results. In January 2021

aliquoted samples for LEQA2 were thawed and analyzed again at RIVM. This confirmed that the

samples were of unchanged quality. The sample containing the VOC was prepared and pretested at

RIVM in January 2021. All 85 laboratories excluding the RIVM performing SARS‐CoV‐2 diagnostics in

the Dutch network by January 2021 were contacted and notified of the distribution of the panel in the

first week of February. All laboratories were asked to report their findings using an online form using

Formdesk software (Wassenaar, The Netherlands) to allow for a more streamlined method of data

collection. Laboratories had until the 21st of February to report their obtained results. After the 21st of

February laboratories that had not reported their results yet were given one week grace time to report

their results, after which the submission was closed on the 28th of February. A number of workflows,

especially the molecular point of care (mPOCT) ones, use expensive cartridges or pouches of which

laboratories only receive a limited number every week. Therefore laboratories with limited resources

that wanted to test their workflows were asked to indicate so. Then they were sent an email to limit

testing for these workflows to samples 3, 5, 8 and 9 (Table 1), covering three different concentrations

of SARS‐CoV‐2 of which one was the VOC and a hCoV‐NL63 containing sample.

2.2 Contents of LEQA2 panel The LEQA2 panel consisted of 10 simulated clinical specimens (1ml) containing either whole infectious

human respiratory seasonal viruses, genetic material of relevant viruses or heat‐inactivated SARS‐CoV‐

2 viruses. SARS‐CoV‐2 was isolated from clinical specimens on VERO E6 cells and heat‐inactivated by

heat treatment at 60 °C for two hours. The number of detectable copies of SARS‐CoV‐2 positive strand

RNA in the stocks of SARS‐CoV‐2 was back‐calculated from determination of the copy number after

of 50

extraction of RNA by digital SARS‐CoV‐2 E‐gene and RdRP‐gene PCR. For hCoV‐

19/Netherlands/NoordBrabant_10003/2020 at 1.28*10^8 and 1.73*10^8 copies of E‐gene and RdRP‐

gene positive strand RNA/ml, respectively. And for hCoV‐19/Netherlands/NH‐RIVM‐20432/2020 at

3.39*10^9 and 2.32*10^9 copies of E‐gene and RdRP‐gene positive strand RNA/ml, respectively

Because the viruses were not purified from the supernatant, the whole virus preparation contains in

addition to genomic RNA, intermediate replication negative strand genomic RNA and subgenomic E‐

gene RNA that contribute to detection in routine one‐step RT‐qPCR for SARS‐CoV‐2 RNA. Virus

dilutions were made in MEM with Hanks’ salts. HEp2 cells were added to the dilution at a

concentration of 10.000 cells per ml panel sample to simulate a clinical sample. The 10 samples

included in the panel contained the following viruses: SARS‐CoV‐2 (hCoV‐

19/Netherlands/NoordBrabant_10003/2020) in various concentrations, SARS‐CoV‐2

B.1.1.7/20B/501Y.V1 VOC‐202012/01 (UK) variant (hCoV‐19/Netherlands/NH‐RIVM‐20432/2020),

hCoV‐NL63 (kindly provided by Lia van der Hoek, Amsterdam University Medical Hospital), hCoV‐229E

(ATCC), hCoV‐OC43 (ATCC), SARS‐CoV‐1 (RNA) (kindly provided by Bart Haagmans, Erasmus MC) and

a sample without any virus. Except for the A(H3N2) containing sample in LEQA1 [2] that was replaced

by the VOC SARS‐CoV‐2 in LEQA2, all other samples in LEQA1 and 2 had the same content, but in

different order. In Table 1 all samples are listed together with the expected target specific Ct values

obtained at RIVM with routinely used diagnostic RT‐qPCRs for the respective pathogens and the

expected conclusion for SARS‐CoV‐2 detection in the samples. The digital copies of RdRP‐gene and E‐

gene are also listed in Table 1 for the SARS‐CoV‐2 containing samples.

of 50

Table 1: Composition of LEQA2 together with the target specific expected Ct values1 based on the in‐house assay(s) of the RIVM.

1 The expected Ct values shown in this table are based on RT‐qPCR tests performed on the panel samples using the routinely used RIVM in‐house assays. The in‐house real‐

time RT‐qPCRs have been performed using the following reagents and volumes: ThermoFisher TaqMan® Fast Virus 1‐Step Master Mix after extraction of 200 μl sample on

Roche MagNA Pure 96 instrument with Roche MagNA Pure 96 DNA and Viral NA Small Volume Kit, elution in 50 μl and 5 μl extract per RT‐qPCR reaction on Roche LightCycler

480 mark I or II. Extractions and subsequent RT‐qPCRs were performed in 4‐fold; after the average Ct value between brackets () the number of times found positive is shown.

SARS‐CoV‐2 E‐gene Sarbeco specific primers and probes are those published by Corman et al.; the RdRP primers and probes are modified from those published by Corman

et al. to become SARS‐CoV‐2 specific and similar in LOD95 compared to the E‐gene Rt‐qPCR. 2 d1, d3 and d4 indicate that d3 is a 1:100 dilution of d1 and d4 is a 1:10 dilution of d3; SARS‐CoV‐2 is heat inactivated. SARS‐CoV‐1 is RNA stabilized with yeast tRNA. 3 dPCR has been performed on + strand genomic RNA for RdRP‐gene and E‐gene; for E‐gene, subgenomic messengers present are also detected. The one‐step E‐gene and

RdRP‐gene diagnostic RT‐qPCR also detects ‐ strand replicative form genomic RNA and the one‐step E gene RT‐qPCR in addition also detects subgenomic messengers, which

probably increases the actual number of target templates for the diagnostic RT‐qPCR in the sample after extraction. 4 For hCoV‐NL63 and hCoV‐229E N‐gene and hCoV‐OC43 M‐gene; n/a = not applicable. 5 Educational specimen. Laboratories using only the Corman E‐gene Sarbeco specific RT‐PCR will epidemiologically rightly label this specimen as SARS‐CoV‐2 positive. The

combination of low Ct with Sarbeco specific PCR and absence of positive signal with another SARS‐CoV‐2 target would prompt further research. One of the two targets

positive with SARS‐CoV‐2 usually only occurs with very low viral load. 6 Educational specimen: repeats of this specimen may have the E‐gene and/or RdRP‐gene negative; only 54.7% of reported workflows having tested this specimen reported

this specimen positive for SARS‐CoV‐2 7 SARS‐CoV‐2 variant B.1.1.7; 20B/501Y.V1 VOC‐202012/01 (UK); TaqPath assay: S‐dropout; TibMolbiol or other N501Y assay: 501Y variant positive. This is a core specimen.

Panel coding Virus2 Number of copies E gene target/ml specimen, determined with dPCR3

Number of copies RdRP gene target/ml specimen, determined with dPCR3

Target specific Ct4

E‐gene (Sarbeco) Ct

RdRP‐gene (SARS‐CoV‐2) Ct

Conclusion SARS‐CoV‐2

LEQA2_CoV20‐15 SARS‐CoV‐1 ‐ ‐ NA 28.57 (4/4) Neg Negative5

LEQA2_CoV20‐2 SARS‐CoV‐2 (d1) 1.28*10^4 1.73*10^4 NA 28.52 (4/4) 28.37 (4/4) POSITIVE

ELQA2_CoV20‐3 hCoV‐NL63 ‐ ‐ 28.10 (4/4) Neg Neg Negative

LEQA2_CoV20‐4 hCoV‐229E ‐ ‐ 17.22 (4/4) Neg Neg Negative

LEQA2_CoV20‐56 SARS‐CoV‐2 (d4) 1.28*10^1 1.73*10^1 NA 36.95 (2/4) 35.59 (2/4) Weakly POSITIVE


LEQA2_CoV20‐7 hCoV‐OC43 ‐ ‐ 27.77 (4/4) Neg Neg Negative


LEQA2_CoV20‐97 SARS‐CoV‐2 VOC 3.39*10^5 2.32*10^5 NA 26.86 (4/4) 27.33 (4/4) POSITIVE

LEQA2_CoV20‐10 No virus ‐ ‐ Neg Neg Neg Negative

of 50

2.3 Scoring the workflows The performance of each reported workflow was evaluated after which they were scored on a scale

from 0 to 8, with 8 being the best grade. This scoring system was implemented based on the detection

of the core samples present in the panel. All samples except LEQA2_CoV20‐1 and LEQA2_CoV20‐5

(containing SARS‐CoV‐1 RNA and an educational load of SARS‐CoV‐2, respectively) were deemed core

samples (samples with clinically relevant amounts of virus or no virus). The laboratories were given

the option to evaluate samples with the following scores: Positive, Negative, Indeterminate, Equivocal,

Inconclusive or Not tested. Each workflow started with 8 points for all correct results for each of the

8 core samples. For each wrongly determined core sample (being positive for a sample containing no

SARS‐CoV‐2 or vice versa) 1 point was deducted (out of 8). When a core sample was scored with an

“Indeterminate”, “Equivocal” or “Inconclusive” result, 0.5 point was deducted from the final mark of

the workflow.

For some workflows (e.g. molecular point‐of‐care test (mPOCT) workflows) an option was given to test

only a smaller subset of samples specimens in order to be able to make a limited statement about the

sensitivity of detection of SARS‐CoV‐2 with the mPOCT used. These workflows only had to test

LEQA2_CoV20‐3, LEQA2_CoV20‐5, LEQA2_CoV20‐8 and LEQA2_CoV20‐9. When notifying the Dutch

SARS‐CoV‐2 diagnostics network, the laboratories were informed about the option to apply for a

reduced (mPOCT) LEQA panel. Especially for these workflows the score “Not tested” was added as an

option. The workflows testing the reduced panel were also graded according to a scale from 0 – 8

points. For each wrongly determined core sample (being positive for a sample containing no SARS‐

CoV‐2 or vice versa) 2 points were deducted (out of 8). When a sample was scored with an

“Indeterminate”, “Equivocal” or “Inconclusive”, 1 point was deducted from the final mark of the

workflow.

When the entire panel was supposed to be tested using a workflow and a sample was given a score of

“Not tested”, 1 point was deducted from the final score for that workflow. This might have occurred

when a laboratory used a second or more workflows for confirmation of some of the results in the

first workflow used.

A workflow scoring 8 out of 8 passed all criteria set for SARS‐CoV‐2 diagnostics in terms of sensitivity

and specificity deemed necessary for SARS‐CoV‐2 diagnostics in accordance with the set requirements

for new workflows and laboratories. [3] Workflows scoring 7.5 or 7 out of 8 might still be valuable for

SARS‐CoV‐2 diagnostics, but need adjustments in order to perform as desired. Adjustments depend

on the type of result, e.g. an “Indeterminate”, “Equivocal” or “Inconclusive” result for low viral load

LEQA2_CoV20‐6 or LEQA2_CoV20‐8 samples is less severe than detection of SARS‐CoV‐2 targets in

specimens which were SARS‐CoV‐2 negative (false positive). Any workflows scoring below 7 out of 8

points needs serious adjustments in order to be fit for SARS‐CoV‐2 diagnostics.

of 50

3. Results 3.1 Aggregated overview

Eighty‐five laboratories were contacted with the announcement of panel distribution for this second

EQA round. Seventy‐nine (92.9%) of these laboratories reported their findings for 180 workflows. The

workflow conclusions reported for each panel sample are summarized in Table 2 (‘Not tested’ result

excluded). The panel scores obtained per laboratory and by number of workflows used are

summarized in Table 3.

Table 2. Aggregated overview of workflow conclusions by LEQA panel sample.

Panel sample Content № of workflows with test result reported (n=180)1

SARS‐CoV‐2 detection workflow conclusion

№ Positive № Indeterminate, Equivocal or Inconclusive

№ Negative Errors

LEQA2_CoV20‐1 SARS‐CoV‐1 161 70 (43.5%) 11 (6.8%) 80 (49.7%) Not applicable, educational sample

LEQA2_CoV20‐2 SARS‐CoV‐2 (d1) 160 160 (100%) 0 0 None

ELQA2_CoV20‐3 hCoV‐NL63 176 0 1 (0.6%) 175 (99.4%) False indeterminate, equivocal or inconclusive (n=1)

LEQA2_CoV20‐4 hCoV‐229E 160 0 0 160 (100%) None

LEQA2_CoV20‐5 SARS‐CoV‐2 (d4) 179 98 (54.7%) 12 (6.7%) 69 (38.5%) Not applicable, educational sample

LEQA2_CoV20‐6 SARS‐CoV‐2 (d3) 162 147 (90.7%) 5 (3.1%) 10 (6.2%) False indeterminate, equivocal or inconclusive (n=5), false negative (n=10)

LEQA2_CoV20‐7 hCoV‐OC43 160 0 0 160 (100%) None

LEQA2_CoV20‐8 SARS‐CoV‐2 (d3) 176 157 (89.2%) 7 (4.0%) 12 (6.8%) False indeterminate, equivocal or inconclusive (n=7), false negative (n=12)

LEQA2_CoV20‐9 SARS‐CoV‐2 VOC 177 176 (99.4%) 0 1 (0.6%) False negative (n=1)

LEQA2_CoV20‐10 No virus 160 1 (0.6%) 1 (0.6%) 158 (98.8%) False positive (n=1), false indeterminate, equivocal or inconclusive (n=1)

1 15 of the 180 workflows tested were mPOCT testing samples 3, 5, 8 and 9 only. For 5 of 180 workflows laboratories made

another limited selection of panel samples to test. Therefore the number of workflows with test result per sample is less

than 180.

Table 3. Aggregated overview of scores for core specimens obtained by laboratories using various numbers of workflows.

№ of workflows per lab

№ of labs № of workflows per lab with indicated score (No of labs)

Score 8 Score 7 or 7.5 Score < 7

9 1 6 (n=1) 0 3 (n=1)

7 2 5‐6 (n=2) 1 (n=2) 1 (n=1)

6 1 5 (n=1) 1 (n=1) 0

5 4 1‐5 (n=4) 1‐4 (n=2) 1 (n=1)

4 8 3‐4 (n=8) 1 (n=2) 1 (n=1)

3 10 1‐3 (n=10) 1 (n=3) 1 (n=2)

2 16 2 (n=10) 0 0

1 37 1 (n=26) 1 (n=5) 1 (n=6)

of 50

Despite not all workflows obtained fully correct results with the core specimens (Table 2), nearly all

laboratories (73/79; 92.4%) used at least one workflow with which a score 7‐8 was obtained (Table 3).

All laboratories using two or more workflows had at least one workflow with which a score of 8 for

fully correct results was obtained. Of the laboratories that used one workflow only 26/37 (70.3%) used

a workflow with which a score of 8 was obtained. In the subsequent chapters a more detailed insight

in the background of the results and the results themselves are presented.

3.1 Used volumes, equipment, kits and reagents Because the sensitivity of a workflow is partly defined by the sample equivalent input volume in the

RT‐qPCR/other NAAT, a subset of questions revolved around the volumes used for testing of clinical

samples for each specific workflow: volume specimen in nucleic acid extraction; elution volume;

volume RNA/total NA in RT‐qPCR reaction or other NAAT; end volume of RT‐qPCR reaction or other

NAAT. Figure 1 shows each of the volumes used for RNA isolation and RT‐qPCR or other NAAT for all

workflows for which results were reported. For those workflows for which extraction input, elution

and RT‐qPCR/other NAAT input volumes were reported the sample equivalent input volume in RT‐

qPCR/other NAAT reaction was calculated and plotted (Figure 1).

of 50

Other factors that may determine the performance of the workflows are the used kits, equipment

and/or separate enzymes used for extraction and amplification implemented in SARS‐CoV‐2

diagnostics for the Dutch population. Therefore for each workflow these details were inventoried.

Figure 2 shows the kits used for RNA/total NA isolation, Figure 3 shows the RNA isolation equipment,

Figure 4 shows the kits used for the RT‐PCR or other NAAT reaction, Figure 5 shows the separate

enzymes used for the in‐house RT‐PCR or other NAAT reaction and Figure 6 shows the equipment used

for the RT‐PCR or other NAAT reaction. In several occasions the kit used for extraction and for RT‐qPCR

or other NAAT has the same name because these are all‐in‐one kits.

Figure 1: Volumes used in the RT‐PCRs or other NAATs described for the workflows reported. Most commonly used volumes: 200 µl input volume; 50 µl elution volume; 10 µl elution input volume in reaction; 20 µl end volume of reaction; 20 µl specimen input equivalent.

of 50

Figure 2: The RNA isolation kits used by workflows testing for SARS‐CoV‐2 together with the number of workflows per kit (n=180)

of 50

Figure 3: The RNA isolation equipment used by workflows testing for SARS‐CoV‐2 together with the number of workflows per machine (n=180)

of 50

Figure 4: The RT‐qPCR or other NAAT kits used by workflows testing for SARS‐CoV‐2 together with the number of workflows per kit (n=130). Not all workflows use ‘ready to use’ kits for their RT‐qPCR or other NAAT, so the total N is not equal to the amount of workflows tested. For each kit the used target genes are listed. Workflows using separate enzymes and primers and probe are listed in Figure 5.

of 50

Figure 5: The enzymes used for performing RT‐PCR or other NAAT by workflows testing for SARS‐CoV‐2 together with the number of workflows per enzyme (n=50). Not all workflows use separate enzymes for their RT‐PCR or other NAAT, so the total N is not equal to the amount of workflows tested. In total 41/50 of the above mentioned workflows use 1 target gene to test for SARS‐CoV‐2 presence (E‐gene Sarbeco specific (n=33); N2‐gene (n=1); N‐gene (n=2); ORF1a/b (n=2); RdRP‐gene (n=3)). 9/50 workflows use 2 target genes to test for SARS‐CoV‐2 presence (E‐gene Sarbeco specific + N1‐gene (n=4); E‐gene Sarbeco specific + N‐gene (n=1); E‐gene Sarbeco specific + RdRP‐gene (n=2); E‐gene SARS‐CoV‐2 specific + N‐gene (n=1); E‐gene SARS‐CoV‐2 specific + RdRP‐gene (n=1)). Workflows using complete ‘ready to use’ kits for SARS‐CoV‐2 detection are listed in Figure 4.

Figure 6: The RT‐PCR or other NAAT equipment used by workflows testing for SARS‐CoV‐2 together with the number of workflows per machine (n=180)

of 50

3.2 Target genes used for RT‐PCR or other NAAT As the sensitivity of a workflow may also depend on the used gene or genes, for all workflows the

target genes used were inventoried. Some workflows used up to 4 target genes. From the 180

workflows a total of 87 workflows used 1 target gene, 78 workflows used 2 target genes, 11 workflows

used 3 target genes and 4 workflows used 4 target genes. In Figure 7, the target genes used in the

order reported for each workflow are shown. In Figure 8 the combinations of target genes used in the

workflows is displayed. Combinations of genes used by number of workflows are listed in Table 4.

Some workflows using more than one gene do not generate separate result for each independent

gene but rather a composite conclusion.

Figure 7: Target genes used in the workflows as reported in the questionnaire (n=180). Color coding genes is shown in the legend. Black color indicates workflows that do not contain a 2nd, 3rd or 4th gene target.

of 50

Figure 8: Combinations of target genes used in the workflows as reported in the questionnaire (n=180). Color coding genes is shown in the legend.

of 50

Table 4. Overview of number and type of target genes used per reported workflow.

№ target genes in workflow

Target gene(s) № workflows

1

E‐gene Sarbeco specific 39

E‐gene SARS‐CoV‐2 specific 18

N2‐gene 1

N‐gene 7

Nsp2‐gene 1

ORF1a/b 16

RdRP‐gene 5

2

E‐gene Sarbeco specific; N1‐gene 1

E‐gene Sarbeco specific; N2‐gene 4

E‐gene Sarbeco specific; N‐gene 30

E‐gene Sarbeco specific; ORF1a/b 5

E‐gene Sarbeco specific; RdRP‐gene 3

E‐gene Sarbeco specific; S‐gene 1

E‐gene SARS‐CoV‐2 specific; N‐gene 1

E‐gene SARS‐CoV‐2 specific; RdRP‐gene 2

N1‐gene; N2‐gene 7

N‐gene; ORF1a/b 13

N‐gene; RdRP‐gene 6

ORF8; RdRP‐gene 1

M‐gene; S‐gene 4

3

E‐gene Sarbeco specific; N‐gene; RdRP‐gene 3

E‐gene Sarbeco specific; N‐gene; S‐gene 1

E‐gene SARS‐CoV‐2 specific; N‐gene; RdRP‐gene 2

E‐gene SARS‐CoV‐2 specific; N‐gene; RdRP‐gene 1

ORF1a/b; N‐gene; S‐gene 4

4 E‐gene Sarbeco specific; N‐gene; RdRP‐gene; S‐gene 4

3.3 Performance of the workflows

All laboratories participating in testing of the LEQA were asked to score each of the 10 samples

contained in the panel for the presence of SARS‐CoV‐2 in any of these six categories: Positive,

Negative, Indeterminate, Equivocal, Inconclusive or Not tested. As said before, for some workflows

only a smaller subset of samples needed to be tested. These workflows had to test only LEQA2_CoV20‐

3, LEQA2_CoV20‐5, LEQA2_CoV20‐8 and LEQA2_CoV20‐9. Figure 9 – Figure 18 show the obtained

results for LEQA2_CoV20‐1 up to LEQA2_CoV20‐10 for all target genes tested, in the order how the

genes have been reported and are displayed in Figure 7 and a summary plot for the total workflow

results. Some workflows using more than one gene do not generate separate result for each

independent gene but rather show a composite conclusion. Due to this, some results obtained of

multiple target genes are combined into and shown as one target gene in Figure 9 – Figure 18. The

sample containing B.1.1.7/20B/501Y.V1 VOC‐202012/01 (UK) variant SARS‐CoV‐2 was detected by

176/177 (99.4%) workflows, indicating primary diagnostic tests perform well detecting this variant

(Table 2, Figure 17). A re‐test of the panel was performed on the one workflow for which the

B.1.1.7/20B/501Y.V1 VOC‐202012/01 (UK) variant SARS‐CoV‐2 containing specimen (specimen

of 50

number 9) was reported negative for SARS‐CoV‐2. In this second test, the workflow managed to detect

the SARS‐CoV‐2 VOC. During the initial test, this workflow found LEQA2 specimen number 10,

containing no virus, positive for SARS‐CoV‐2 presence, whereas it remained negative during the

second test. It is likely that specimen number 9 and 10 were switched during the first round of testing

the LEQA2 panel. When including the results obtained during the re‐test of this workflow, 100% of the

workflows included in this quality round could detect B.1.1.7/20B/501Y.V1 VOC‐202012/01 (UK)

variant SARS‐CoV‐2. No results on performance of variant specific assays were obtained.

Figure 9: Results obtained for LEQA2_CoV20‐1 containing SARS‐CoV‐1 combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. This sample was not deemed a core sample in the panel. Overall, of 161 workflows that reportedan overall conclusion 80/161 (49.7%) did correctly not detect SARS‐CoV‐2 in this sample. 11/161 (6.8%) workflows reported an indeterminate, equivocal or inconclusive result and 70/161 (43.5%) reported a positive result, highly likely because in the current epidemiological situation no other Sarbeco virus than SARS‐CoV‐2 would be expected.

of 50

Figure 10: Results obtained for LEQA2_CoV20‐2 containing SARS‐CoV‐2 (with 1.28*10^4 copies E target/ml sample (determined with dPCR) and 1.73*10^4 copies RdRP target/ml sample (determined with dPCR)) combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. Overall, all 160 workflows that reported an overall conclusion did correctly detect SARS‐CoV‐2 in this sample.

Figure 11: Results obtained for LEQA2_CoV20‐3 containing hCoV‐NL63 combined from all workflows. All laboratories participating in testing of theLEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. This sample was not deemed a core sample in the panel. Overall, of all 176 workflows that reported an overall conclusion 175/176 (99.4%) identified this sample as SARS‐CoV‐2 negative and 1/176 (0.4%) with an indeterminate, equivocal or inconclusive result.

of 50

Figure 12: Results obtained for LEQA2_CoV20‐4 containing hCoV‐229E combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. Overall, of all 160 workflows that reported an overall conclusion 160/160 (100%) workflows correctly did not detect SARS‐CoV‐2 in this sample.

Figure 13: Results obtained for LEQA2_CoV20‐5 containing SARS‐CoV‐2 (with 1.28*10^1 copies E target/ml sample (determined with dPCR) and 1.73*10^1 copies RdRP target/ml sample (determined with dPCR)) combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. This sample was not deemed a core sample in the panel. Overall, of all 179 workflows that reported an overall conclusion 98/179 (54.7%) correctly identified this sample as SARS‐CoV‐2 positive and 12/179 (6.7%) workflows gave an indeterminate, equivocal or inconclusive overall conclusion. 69/179 (38.5%) workflows reported incorrectly a negative result.

of 50

Figure 14: Results obtained for LEQA2_CoV20‐6 containing SARS‐CoV‐2 (with 1.28*10^2 copies E target/ml sample (determined with dPCR) and 1.73*10^2 copies RdRP target/ml sample (determined with dPCR)) combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. Overall, of all 162 workflows that reported an overall conclusion 147/162 (90.7%) correctlyidentified this sample as SARS‐CoV‐2 positive and 5/162 (3.1%) workflows gave an indeterminate, equivocal or inconclusive overall conclusion.10/162 (6.2%) workflows reported incorrectly a negative result.

Figure 15: Results obtained for LEQA2_CoV20‐7 containing hCoV‐OC43 combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. Overall, of all 160 workflows that reported an overall conclusion 160/160 (100%) workflows correctly did not detect SARS‐CoV‐2 in this sample.

of 50

Figure 16: Results obtained for LEQA2_CoV20‐8 containing SARS‐CoV‐2 (with 1.28*10^2 copies E target/ml sample (determined with dPCR) and 1.73*10^2 copies RdRP target/ml sample (determined with dPCR)) combined from all workflows. All laboratories participating in testing of the LEQAwere asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. Overall, of all 176 workflows that reported an overall conclusion 157/176 (89.2%) correctly identified this sample as SARS‐CoV‐2 positive and 7/162 (4.0%) workflows gave an indeterminate, equivocal or inconclusive overall conclusion. 12/176 (6.8%) workflows reported incorrectly a negative result.

Figure 17: Results obtained for LEQA2_CoV20‐09 containing SARS‐CoV‐2 (SARS‐CoV‐2 variant B.1.1.7; 20B/501Y.V1) (with 3.39*10^5 copies E target/ml sample (determined with dPCR) and 2.32*10^5 copies RdRP target/ml sample (determined with dPCR)) combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order inwhich they were reported) the percentage of results reported per scoring category are shown. Overall, of all 177 workflows that reported an overall conclusion 176/177 (99.4%) correctly identified this sample as SARS‐CoV‐2 positive and 1/177 (0.6%) workflows reported incorrectly a negative result.

of 50

As described before, all workflows were graded

using a point system from 0 (being the lowest

grade) up to 8 (highest grade). 148 workflows were

given an ‘8’, five workflows scored a ‘7.5’, 13

workflows scored a ‘7’, one workflow scored a ‘6.5’,

eight workflows scored a “6”, one workflow scored

a ‘5’, one workflow scored a ’3’ and three workflows

scored a ‘1’. It should be noted that all workflows

scoring less than 6 points did not test the full panel

or all samples of the reduced panel, thus these

workflows lost a lot of points due to samples being

scored ‘Not tested’. Figure 19 shows all grades

given to the reported workflows.

An overview containing the results obtained per target gene per panel sample for workflows reporting

Ct values is shown in Figure 20. In this figure for each of the target genes used (shown in the order in

which they were reported) the Ct values are shown for each of the tested samples. Also shown are

the number of tested samples using each of the target genes, the percentage of results showing a Ct

< 50, the number of reported negative results and the percentage of reported negative results (likely

due to implemented cut‐off values) with Ct < 50 .

The obtained scores per workflow are also coupled to the extraction kit or method used, the PCR or

other NAAT test performed and the number of target genes used in order to assess the effect of

different techniques on the performance of workflows. An overview of these factors on the grade is

shown in Figure 21. Unfortunately the specimen input equivalent volume in the PCR could only be

Figure 18: Results obtained for LEQA2_CoV20‐10 containing no virus (negative control) combined from all workflows. All laboratories participating in testing of the LEQA were asked to score this sample for the presence of SARS‐CoV‐2 in any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive or Not tested. For each of the target genes used (which were processed in the order in which they were reported) the percentage of results reported per scoring category are shown. Overall, of 160 workflows that reported an overall conclusion 176/177 (99.4%) identified this sample as SARS‐CoV‐2 negative and 1/176 (0.4%) with an indeterminate, equivocal or inconclusive result. 1/177 (0.6%) workflows reported incorrectly a positive result.

Figure 19: All grades obtained by the reported workflows out of the maximum of 8 points (n=180).

of 50

calculated for 90/180 workflows and therefore this factor is not included in Figure 21. For the 90

workflows for which it was calculated the specimen equivalent volume was median 20 µl (range 2 µl

– 400 µl). For the 148 workflows for which specimen input volume in extraction was reported the

median volume was 250 µl (range 50 µl – 1400 µl). The median reaction volume reported for 95

workflows was 20 µl (range 4 µl – 30 µl).

In total one laboratory reported data for 9 workflows, two laboratories reported data for 7 workflows,

one laboratory reported data for 6 workflows, four laboratories reported data for 5 workflows, eight

laboratories reported data for 4 workflows, ten laboratories reported data for 3 workflows, sixteen

laboratories reported data for 2 workflows and thirty‐seven laboratories reported data for 1 workflow.

There are six laboratories which only have scores of < 7 for all reported workflows. The obtained scores

per workflow are sorted (anonymously) per laboratory and shown in Figure 22. Of the 79 laboratories,

68 laboratories reported at least one workflow with fully correct results. When excluding all mPOCT

assays from the analysis, one laboratory reported data for 7 workflows, one laboratory reported data

for 6 workflows, one laboratory reported data for 5 workflows, two laboratories reported data for 4

workflows, nine laboratories reported data for 3 workflows, eighteen laboratories reported data for 2

workflows, forty‐three laboratories reported data for 1 workflow and four laboratories did not report

any other workflows than mPOCT assays. Four laboratories only have scores of < 7 for all reported

workflows. This is shown in Figure 23. It should be noted that all workflows (also including mPOCT

assays) scoring less than 6 points did not test the full panel or all samples of the reduced panel, thus

these workflows lost a lot of points due to samples being scored “Not tested”. Therefore for these

laboratories and workflows a true performance estimate cannot be given.

of 50

Figure 20: Results obtained per target gene per panel sample for workflows reporting Ct values. The numbers on the X‐axis indicate which target gene (in order in which they were reported) is used for the detection of each sample. Underneath these numbers the contents of the sample are shown. All negative values for which no Ct value was given by the reporters have been given an artificial Ct value of 50. Not all negative results have a Ct value of 50. Some results with Ct < 50 are deemed negative by laboratories, likely due to Ct cutoff values used in the interpretation of an obtained result. Above the graph the number of tests (N) and the percentage of Ct values below 50 is shown per sample per target gene. Above the graph the number (N) and the percentage of tests with a negative results reported with a Ct value below 50 is shown as well per sample per target gene. Samples deemed negative with a Ct value below 50 are indicated with a purple diamond inside the graph.

of 50

Figure 21: A flow diagram showing all workflows reported to have tested the LEQA panel with extraction method, PCR test, the number of target genes used and the final score achieved by each workflow. In the alluvial plot PCR tests using 1 target gene are depicted in blue, PCR tests using 2 target genes are shown in red, PCR tests using 3 target genes are shown in green and PCR tests using 4 target genes are shown in purple. For the target gene combinations used per kit, see Figure 5. Color of trails per workflow are based on the grade obtained for LEQA2 All workflows receiving grades below 6 are grouped in <6.

of 50

Figure 22: Grades obtained per workflow per laboratory (anonymized). For each of the laboratories the amount of reported workflows is shown on the X‐axis together with their accompanying grades. In total 79 laboratories sent in data of their workflows. There are six laboratories which only have scores of < 7 for all reported workflows. It should be noted that all workflows scoring less than 6 points did not test the full panel or all samples of the reduced panel, thus these workflows lost a lot of points due to samples being scored “Not tested”. The same numbering is maintained as in Figure 23, where all workflows excluding mPOCT assays are listed.

of 50

Figure 23: Grades obtained per workflow per laboratory excluding all mPOCT assays (anonymized). For each of the laboratories the amount of reported workflows is shown on the X‐axis together with their accompanying grades. In total 79 laboratories sent in data of their workflows. There are four laboratories which only have scores of < 7 for all reported workflows. Four laboratories do not report any other workflows than mPOCT assays. It should be noted that all workflows scoring less than 6 points did not test the full panel or all samples of the reduced panel, thus these workflows lost a lot of points due to samples being scored “Not tested”. The same numbering is maintained as in Figure 22, where all workflows including mPOCT assays are listed.

of 50

3.4 Comparing results LEQA2 to LEQA1 Whereas 65 laboratories reported data for 164 workflows for LEQA1, 79 laboratories reported data

for 180 workflows for LEQA2. This is an increase of 21.5% of reporting laboratories and an increase of

9.8% of reported workflows. The grades obtained by the workflows for both LEQA1 and LEQA2 are

listed in Table 5. It should be noted that 7/8 core samples were exactly the same in both panels. The

single different sample was LEQA1 influenza virus A(H3N2) replaced by LEQA2 B.1.1.7/20B/501Y.V1

VOC‐202012/01 (UK) variant SARS‐CoV‐2. Because the concentration of the variant virus in the sample

in LEQA2 was high the number of workflows detecting this variant was expected to be 100%, similar

to expected 100% of workflows not detecting A(H3N2) in panel LEQA1. Therefore, grades can directly

be compared between LEQA1 and LEQA2.

Table 5. Overview of grades obtained by the workflows having tested LEQA1 and LEQA2. All workflows receiving grades below 6 are grouped in <6

The shift in grades obtained by the workflows tested between LEQA1 and LEQA2 are shown in Table

6. Not all workflows were tested for both LEQA1 and LEQA2. These workflows are labelled ‘Not

Tested’ or ‘NT’ abbreviated. This data is also visualized in Figure 24.

Table 6. Overview of grades obtained by workflows that tested LEQA1 and how they scored testing LEQA2. Green cells indicate workflows performing better in LEQA2 than in LEQA1. Red cells indicate workflows performing worse in LEQA2 than in LEQA1. White cells indicate workflows performing equally in LEQA 2 and LEQA1. All workflows receiving grades below 6 are grouped in <6

Grade obtained for LEQA 1

<6 6 6.5 7 7.5 8 NT1 Subtotal

Grade obtained

for LEQA2

<6 1 4 5

6 4 4 8

6.5 1 1

7 2 2 5 4 13

7.5 2 3 5

8 1 3 4 3 71 66 148

NT2 4 5 5 1 50

Subtotal 6 8 0 11 7 132

180 164

1 Short for ‘Not Tested’. These workflows were not used to test LEQA1, but were used to test LEQA2 2 Short for ‘Not Tested’. These workflows were not used to test LEQA2, but were used to test LEQA1

Grade obtained

8 7.5 7 6.5 6 <6 Total workflows tested

LEQA1 132 (80.5%)

7 (4.3%)

11 (6.7%)

0 8 (4.9%)

6 (3.7%)

164

LEQA2 148 (82.2%)

5 (2.7%)

13 (7.2%)

1 (0.1%)

8 (4.4%)

5 (2.7%)

180

of 50

In total 65 workflows were not used to test LEQA2 but did test LEQA1. Twenty‐seven of these

workflows were mPOCT assays (41.5%), 22 workflows used ‘ready to use’ kits for their RT‐qPCR

(33.8%) and 16 workflows were in‐house assays (24.%). Of the 81 newly implemented workflows in

LEQA2 (which were not tested in LEQA1), 31 were mPOCT assays (38.3%), 30 workflows used ‘ready

to use’ kits for their RT‐qPCR (37.0%), 2 workflows were Loop‐mediated isothermal amplification

(LAMP) assays (2.5%) and 18 workflows were in‐house assays (22.2%).

In total 78/162 (48.2%) of the workflows tested in LEQA1 could detect the educational load of SARS‐

CoV‐2 (with 12.8 and 17.3 copies E gene and RdRP gene target/ml sample, respectively; dilution d4,

see Table 1) whereas 98/179 (54.7%) of the workflows tested in LEQA2 could detect SARS‐CoV‐2 in

this sample. For LEQA1 both LEQA1_CoV20‐5 and LEQA1_CoV20‐7 contained SARS‐CoV‐2 at a

concentration of 128 and 173 copies E gene and RdRP gene target/ml specimen, respectively (dilution

d3, see Table 1). LEQA1_CoV20‐5 was measured positive for SARS‐CoV‐2 by a total of 131/148 (88.5%)

of the workflows. For LEQA1_CoV20‐7 139/158 (88.0%) of the workflows tested these samples

positive for SARS‐CoV‐2. In LEQA2 identical samples with the same SARS‐CoV‐2 load but with different

labelling were put in the panel: LEQA2_CoV20‐6 and LEQA_CoV20‐8 respectively. LEQA2_CoV20‐6 was

scored positive for SARS‐CoV‐2 by 147/162 (90.7%) workflows, an increase with 2.7% compared to

LEQA1_CoV20‐5. LEQA2_CoV20‐8 was scored positive by 157/176 (89.2%) workflows, an increase with

1.2% compared to LEQA1_CoV20‐7. Another set of specimens with the same SARS‐CoV‐2 loads in both

LEQA panels were LEQA1_CoV20‐1 and LEQA2_CoV20‐2. These samples contained 1.28×104 copies E

gene and RdRP gene target/ml specimen, respectively (dilution d1, see Table 1). LEQA1_CoV20‐1 was

scored positive for SARS‐CoV‐2 by 161/162 (99.4%) workflows, whereas LEQA2_CoV20‐2 was scored

Figure 24: Overview of grades obtained by workflows that tested LEQA1 and how they scored testing LEQA2. ‘NT’ stands ‘Not Tested’. These workflows were not used to test LEQA1, but have tested LEQA2 or vice versa. Color of trails per workflow are based on the grade obtained for LEQA2. All workflows receiving grades below 6 are grouped in <6

of 50

positive by 160/160 (100%) workflows, an increase by 0.6%. So, at the highest SARS‐CoV‐2

concentration (d1), the amount of positive results increased from 99.4% to 100%. The percentage of

positive results for both specimens with the intermediate SARS‐CoV‐2 load (d3) increased from

88.0%/88.5% to 89.2%/90.7%. Lastly the specimens with the lowest SARS‐CoV‐2 load (d4) of both

panels show an increase in positive results from 48.2% to 54.7%. Taken together, for each of the

samples with different SARS‐CoV‐2 concentrations, the workflows included in LEQA2 showed an

increase in the percent of positive results compared to those included in LEQA1. This is an indication

that more‐sensitive workflows have been implemented since LEQA1 or that existing workflows have

been made more sensitive.

In total, 99 workflows used to test LEQA1 were also used to test LEQA2 (Table 6). A detailed description

of a comparison of the performance with these workflows is given below.

In total, 74/99 workflows did not show a change in grade between LEQA2 and LEQA1. One workflow

obtained a grade below 6/8, two workflows obtained a grade of 7/8 and 71 workflows scored a grade

of 8/8 for both LEQA rounds.

Of all workflows scoring a lower grade for LEQA2 than for LEQA1 (n=14/99), 11 (78.6%) previously

scored a grade of 8/8 and 3 (21.4%) scored a grade of 7.5/8 for LEQA1. For LEQA2 the obtained grades

were 6/8 by four (28.6%) workflows; 6.5/8 by one (7.1%) workflow; 7/8 by seven (50%) workflows;

7.5/8 by two (14.3%) workflows. For LEQA2 6 (42.9%) of the workflows reported one false negative

result for one of the lowest concentration SARS‐CoV‐2 containing core specimens (dilution d3, see

Table 1); 2 (14.3%) of the workflows reported a false negative result for both lowest concentration

SARS‐CoV‐2 containing core specimens (dilution d3, see Table 1); 1 (7.1%) of the workflows reported

a false indeterminate/inconclusive/equivocal result for one of lowest concentration SARS‐CoV‐2

containing core specimens (dilution d3, see Table 1); 2 (14.3%) of the workflows reported a false

indeterminate/inconclusive/equivocal result for both lowest concentration SARS‐CoV‐2 containing

core specimens (dilution d3, see Table 1); 1 (7.1%) of the workflows reported one false

indeterminate/inconclusive/equivocal and one false negative result for the lowest concentration

SARS‐CoV‐2 containing core specimens (dilution d3, see Table 1); 1 (7.1%) of the workflows reported

a false negative result for SARS‐CoV‐2 variant B.1.1.7; 20B/501Y.V1 VOC‐202012/01 (UK) and a false

positive result for no virus control; 1 (7.1%) of the workflows reported a false

indeterminate/inconclusive/equivocal result for the no virus control. Most of the incorrect results

have been obtained for the SARS‐CoV‐2 dilution d3 of the core samples (10‐fold higher concentration

than the educational load sample), suggesting that the concerned workflows lost some sensitivity

between both LEQA rounds. This data is summarized in Table 7.

Of all workflows scoring a higher grade for LEQA2 than for LEQA1 (n=11/99), 3 (27.3%) workflows

previously scored a 7.5/8; 4 (36.4%) workflows scored a 7/8; 3 (27.3%) workflows scored a 6/8; 1

workflow (9.1%) scored a 4/8. When testing LEQA2, all these workflows scored an 8/8. So, all core

samples were determined correctly using all the concerned workflows. For LEQA1 3 (27.3%) workflows

reported a false indeterminate/inconclusive/equivocal result for the no virus control; 2 (18.2%) of the

workflows reported one false negative result for the lowest concentration SARS‐CoV‐2 containing core

specimens (dilution d3, see Table 1); 2 (18.2%) workflows reported of the workflows reported a false

indeterminate/inconclusive/equivocal result for both lowest concentration SARS‐CoV‐2 containing

core specimens (dilution d3, see Table 1); 2 (18.2%) workflows reported false negative results for both

lowest concentration SARS‐CoV‐2 containing core specimens (dilution d3, see Table 1); 1 (9.1%)

workflow only tested part of the LEQA1 panel and did not test the reduced mPOCT panel. Most of the

improvements in LEQA2 compared to LEQA1 have been obtained for the SARS‐CoV‐2 dilution d3 of

the core samples (10‐fold higher concentration than the educational load sample), suggesting that the

of 50

concerned workflows gained some sensitivity between both LEQA rounds. These results are

summarized in Table 8.

Table 7. Overview of the reasons workflows obtained lower grades for LEQA2 than for LEQA1. For LEQA1 the previously obtained grades are also shown and how they were obtained.

1 na = not applicable. 2 These samples were likely switched during testing. Upon retesting by the involved laboratory after closure of

the reporting period and receiving the expected results, this workflow correctly determined all core samples of

LEQA2. 3 One of these workflows did find the educational SARS‐CoV‐2 containing specimen (dilution d4; see Table 1)

positive for SARS‐CoV‐2. This might be an indication that the workflow does have a high sensitivity concerning

SARS‐CoV‐2 detection.

Higher

Lower

LEQA1 1x false

indeterminate/inconclusive/equivocal

for SARS‐CoV‐2 dilution d3

(grade 7.5/8)

No errors (grade

8/8)

LEQA2

1x false

indeterminate/inconclusive/equivocal for no

virus control

(grade 7.5/8)

na1 n=1

1x false

indeterminate/inconclusive/equivocal for

SARS‐CoV‐2 dilution d3

(grade 7.5/8)

na n=1

1x false negative for SARS‐CoV‐2 dilution d3

(grade 7/8) na n=5

2x false



(grade 7.5/8)

n=2 na


AND 1x false



(grade 6.5/8)

n=1 na


(mPOCT panel; grade 6/8) na n=1


(grade 6/8) na n=23

1x false negative for SARS‐CoV‐2 variant

B.1.1.7; 20B/501Y.V1 VOC‐202012/01 (UK)

AND 1x false positive for no virus control

(grade 6/8)2

na n=1

of 50

Table 8. Overview of the reasons workflows obtained higher grades for LEQA2 than for LEQA1. For LEQA1 the previously obtained grades are also shown and how they were obtained.

On a laboratory level, when testing LEQA1, two laboratories had workflows with grades below 7/8

only. One of these laboratories (laboratory 61 in Figures 22 and 23) has since improved their workflow

and no longer scores a grade below 7 with the implemented workflow. The other laboratory

(laboratory 58 in Figures 22 and 23) unfortunately still scores a grade below 7/8. When testing LEQA2,

unfortunately two new labs have received grades of below 7/8 only, whereas they did not in LEQA1

(laboratories 45 and 71 in Figures 22 and 23). The workflow of laboratory 71 (in Figure 22 and 23)

scored a grade below 7/8 due to missing two core specimen containing SARS‐CoV‐2. This same

workflow however was able to detect SARS‐CoV‐2 in the educational specimen (dilution d4, see Table

1). This might indicate that this workflow does have a high sensitivity for SARS‐CoV‐2 detection. The

reason why the 10‐fold higher concentration has been missed remains unclear.

Also three laboratories who did not participate in LEQA1 have received grades below 7/8 for LEQA2

(laboratories 76, 78 and 79 in Figures 22 and 23). Two of these latter laboratories did not test the full

LEQA2 panel nor the reduced mPOCT panel, leading to a lower grade. And therefore the performance

of these laboratories can in fact not be evaluated using these results.

Higher Lower

LEQA2 No errors (grade 8/8)

LEQA1

1x false indeterminate/inconclusive/equivocal for SARS‐CoV‐2 dilution d3

(grade 7.5/8) n=3

1x false negative for SARS‐CoV‐2 dilution d3 (grade 7/8)

n=2

2x false indeterminate/inconclusive/equivocal for SARS‐CoV‐2 dilution d3

(grade 7/8) n=2

2x false negative for SARS‐CoV‐2 dilution d3 (grade 6/8)

n=3

1x Testing only part of LEQA panel, but not mPOCT panel (grade 4/8)

n=1

of 50

4. Discussion and conclusion

Out of 180 workflows reported, 148 scored a 100% correct score for all 8 core specimens (8 points)

and thus met all criteria set for reliable SARS‐CoV‐2 diagnostics, 18 scored between 7‐7.5, making it

likely that only minor adjustments need to be made to meet all criteria and 16 workflows scored a 6.5

or lower, indicating that a lot of improvements still need to be made for these workflows to be reliable

for SARS‐CoV‐2 diagnostics in clinical diagnostic settings and surveillance. All workflows scoring grades

below 6 (n=6) seemed to have obtained a low score due to testing only a small subset of the core

samples while not testing the reduced panel of 4 core samples that has been recommended for

mPOCT testing. Due to this, a lot of samples received a score of “Not tested” and thus decreased the

overall grade of the test. When the workflows on which only the full panel or reduced mPOCT panel

were tested are taken into account (n=174), none of the workflows submitted received a grade below

6.

When considering all workflows each laboratory has access to and have reported LEQA2 results for,

six laboratories have workflows which do not have a grade of 7 or above. Although it is not necessary

to use a workflow scoring a 7 or 7.5 out of 8 for SARS‐CoV‐2 diagnostics, due to slight reduced

sensitivity to detect SARS‐CoV‐2, during the current period with a prevalence of COVID‐19 around 10%

or higher, it is likely that only minor adaptions of the concerned workflows are needed to perform

within the desired criteria detecting all core samples correctly.

Only four laboratories have access to solely mPOCT based assays, limiting their maximal daily

throughput of clinical specimens.

Throughout the reported workflows lots of different target genes and combinations of them are

reported, but the E‐gene as target gene as either a Sarbeco specific or SARS‐CoV‐2 specific target is

most prevalent (115 out of 180 workflows). There was no significant difference in performance

between workflows using different single target genes or combinations of target genes.

False indeterminate/inconclusive/equivocal results have been obtained for samples containing hCoV‐

NL63 (1/176; 0.6%) and no virus control (1/160; 0.6%). Otherwise no such results have been reported

for non SARS‐CoV‐2 containing specimen. This shows that the implemented workflows show a high

specificity to SARS‐CoV‐2 and the RT‐PCRs or other NAATs rarely cross‐react to other human

coronaviruses or other non SARS‐CoV‐2 containing samples. One false positive result for

LEQA2_CoV20‐10 that does not contain virus has been reported by the laboratories. This was likely

due to accidental swapping of specimens LEQA2_CoV20‐9 and ‐10 by the technician. Additionally,

many workflows reported the SARS‐CoV‐1 containing sample (LEQA2_CoV20‐1) positive for

SARS‐CoV‐2 due to using Sarbeco‐specific RT‐PCR. Considering the fact that SARS‐CoV‐1 is not

circulating, this does not pose a problem for the accuracy of SARS‐CoV‐2 diagnostics performed in

clinical and surveillance settings. However, it is striking that for a quite some workflows SARS‐CoV‐2

specific E‐gene primers/probe are reported whilst it still detects SARS‐CoV‐1 (21.74%; Supplemental

Figure 1). It appears that for these workflows most likely the specificity of E‐gene primers/probe is

wrongly labelled as SARS‐CoV‐2 specific rather than Sarbeco specific. Or have a strong cross‐reaction

to SARS‐CoV‐1, which is less likely.

Whereas very little false positive results have been reported for the core samples included in the

panel, a number of workflows reported false negative results for some of the core samples containing

SARS‐CoV‐2. Apparently sensitivity is a bigger issue than specificity for the workflows used for SARS‐

CoV‐2 diagnostics.

Despite some workflows generated a Ct value using SARS‐CoV‐2 RT‐PCR, the sample was deemed

SARS‐CoV‐2 negative, possibly related to a used cut off value or other criterium, e.g. shape of the

amplification curve. Up to 1.5% of each SARS‐CoV‐2 positive sample (range: 0% – 1.5%) was reported

as SARS‐CoV‐2 negative despite one (or more) of the target genes against SARS‐CoV‐2 in the assay

of 50

giving a Ct value. For all LEQA samples tested with a workflow providing Ct values as an output, a

broad range of Ct values for the individual SARS‐CoV‐2 containing samples has been reported. The

biggest range of reported Ct values was found for LEQA_CoV20‐9. The average Ct value was 26.89 (SD

2.40; data not shown) where the lowest Ct value reported was 16.29 and the highest value was 41.60

(making a 25.31 difference in Ct value), indicating a wide spread of Ct values reported for the same

sample, generated by the large variety of workflows used. Despite this wide range of Ct values for the

same sample by different workflows, this did not affect the sensitivity of the workflows significantly.

This finding indicates clearly that comparing Ct values between workflows and laboratories should not

be done without calibration using a standard.

A wide array of varying in‐house and kit‐based SARS‐CoV‐2 workflows have been reported, that has

been changed and even broadened compared to LEQA1 [2]. Compared to the 2009 influenza

pandemic, the Dutch clinical diagnostic field for respiratory diagnostics shows a divergent pattern in

use of kits, reagents and equipment. [4] A more divergent use of kits, reagents and equipment can be

quite useful in a laboratory network as a shortage of any of these can be compensated by switching

to different equipment or when certain workflows are less capable of detecting new strains of SARS‐

CoV‐2. This is highly relevant with the rising level of infections with new variants of SARS‐CoV‐2.

Therefore it is encouraging that in the current LEQA2 variant of concern B.1.1.7; 20B/501Y.V1 that has

become the major strain (up to 82%) in circulation by week 9/2021 in The Netherlands [1] was

detected by 176/177 (99.4%) workflows that tested this sample.

Comparing this second round of quality assessment with the first round, 14 additional laboratories

reported their findings. The total number of reported workflows increased with 15 compared to the

earlier assessment. From the workflows initially scoring a grade <6 for LEQA1, 4/6 (66.7%) were not

retested with LEQA2. Similarly, from 5/8 (62.5%) workflows initially scoring a 6, 5/11 (45.5%)

workflows initially scoring a 7, 1/7 (14.3%) workflows initially scoring a 7.5 and 50/132 (37.9%)

workflows initially scoring an 8 were not retested with LEQA2. The relative amount of LEQA1

workflows not re‐tested in LEQA2 is bigger for the poorer scoring workflows (<7) compared to the

better scoring workflows (7‐8) in LEQA1. This data seems to indicate that laboratories use these quality

assessments as a way to improve their performance in SARS‐CoV‐2 diagnostics. The absolute amount

of workflows obtaining a score of <6 only decreased by 1 (from 6 to 5) from LEQA1 to LEQA2. This is

due to an influx of four new workflows compared to LEQA1 with which not the full LEQA2 panel nor

the reduced mPOCT panel was tested, leading to a lower grade. Two of these new workflows belong

to laboratories that did not test the LEQA1 panel. Nevertheless, there seems to be a trend showing a

disuse of lesser performing workflows. However, this data might be biased due to not all workflows

being used by a laboratory have been tested and reported in this second round by all laboratories.

Overall we can conclude that the workflows used for SARS‐CoV‐2 diagnostics for the Dutch population

perform very well. There are some workflows which need some work in order for them to perform as

desired, but all in all the Dutch SARS‐CoV‐2 diagnostics laboratory network appears to perform on a

very high level.

of 50

5. References 1. RIVM. Varianten van het coronavirus SARS‐CoV‐2. 2021 [cited 2021 31‐3‐2021]; Available from:

https://www.rivm.nl/coronavirus‐covid‐19/virus/varianten.

2. J. Sluimer et al., “External Quality Assessment of laboratories Performing SARS‐CoV‐2 Diagnostics

for the Dutch Population, November 2020”, RIVM report. Available from:

https://www.rivm.nl/sites/default/files/2021‐

02/EQA%2520of%2520Laboratories%2520Performing%2520SARS‐CoV‐

2%2520Diagnostics%2520for%2520the%2520Dutch%2520Population%2520November‐2020.pdf

3. RIVM. Aanvullende informatie diagnostiek COVID‐19. 2021 [cited 2021 26‐3‐2021]; 24: Available from: https://lci.rivm.nl/covid‐19/bijlage/aanvullend.

4. A. Meijer et al., “Preparing the outbreak assistance laboratory network in the Netherlands for the detection of the influenza virus A(H1N1) variant”, J. Clin. Virol., vol. 45, no. 3, pp. 179–184, 2009.

of 50

6. Supplemental material 6.1 Results obtained per target gene per sample Here all results (any of these six categories: Positive, Negative, Indeterminate, Equivocal, Inconclusive

or Not tested) obtained per target gene are shown in percentages per panel sample number. Some

workflows using more than one gene do not generate separate result for each independent gene but

rather a composite conclusion. In Supplemental Figure 1‐10 these are shown together as one target

gene.

Supplemental Figure 1: The percentages of the various scores obtained for LEQA2_CoV20‐1 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depicted in relation to SARS‐CoV‐2 presence in the sample

of 50

Supplemental Figure 2: The percentages of the various scores obtained for LEQA2_CoV20‐2 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depictedin relation to SARS‐CoV‐2 presence in the sample

of 50

Supplemental Figure 3: The percentages of the various scores obtained for LEQA2_CoV20‐3 using the various target genes implemented in the workflowsreported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depicted in relation to SARS‐CoV‐2 presence in the sample

of 50

Supplemental Figure 4: The percentages of the various scores obtained for LEQA2_CoV20‐4 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depicted in relation to SARS‐CoV‐2 presence in the sample. This sample was not deemed a core sample from the LEQA panel.

of 50


of 50

Supplemental Figure 6: The percentages of the various scores obtained for LEQA2_CoV20‐6 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depictedin relation to SARS‐CoV‐2 presence in the sample

of 50

Supplemental Figure 7: The percentages of the various scores obtained for LEQA2_CoV20‐7 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depicted in relation to SARS‐CoV‐2 presence in the sample.

of 50


of 50

Supplemental Figure 9: The percentages of the various scores obtained for LEQA2_CoV20‐9 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depicted in relation to SARS‐CoV‐2 presence in the sample.

of 50

Supplemental Figure 10: The percentages of the various scores obtained for LEQA2_CoV20‐10 using the various target genes implemented in the workflows reported. The percentages shown are combined from target genes reported as either target gene 1, 2, 3 or 4 in the various workflows. The results are depicted in relation to SARS‐CoV‐2 presence in the sample. This sample was not deemed a core sample from the LEQA panel.

of 50

6.2 Participating laboratories All participating laboratories are listed below. We would like to thank colleagues from these

laboratories for their participation in this round of LEQA for the Dutch SARS‐CoV‐2 diagnostics

laboratory network.

Laboratory

Admiraal de Ruyter Ziekenhuis

Alrijne Zorggroep

ArminLabs GmbH

Atalmedial

Brightlabs B.V.

Canisius Wilhelmina Ziekenhuis

Catharina Ziekenhuis

Certe

Deventer Ziekenhuis

Diagnostiek voor U

Diakonessenhuis Utrecht

Elisabeth Tweesteden Ziekenhuis

Erasmus Medisch Centrum

Eurofins Biomnis France

Eurofins Genomics Europe Applied Genomics GmbH

Eurofins NMDL‐HVL

Eurofins NMDL‐LCPL

Fenelab Consortium ‐ Mérieux Nutrisciences

Fenelab Consortium ‐ NofaLab B.V.

Fenelab Consortium ‐ Normec Biobeheer

Fenelab Consortium ‐ Nutreco Nederland B.V. ‐ MasterLab

Fenelab Consortium ‐ NutriControl

Fenelab Consortium ‐ Nutrilab B.V.

Fenelab Consortium ‐ SGS Nederland B.V.

Fenelab Consortium ‐ Triskelion

Franciscus Gasthuis & Vlietland

Gelre Ziekenhuizen Apeldoorn

GGD Amsterdam Streeklaboratorium

Groene Hart Ziekenhuis

Haaglanden Medisch Centrum

Hagaziekenhuis

IJssellandziekenhuis

Ikazia ziekenhuis

inBiome

Isala

Izore

Jeroen Bosch ziekenhuis

LabMicTA

Labor Dr Wisplinghoff

Laurentius ziekenhuis

of 50

Laboratory

Leiden University Medical Center

Maasstad ziekenhuis

MeanderMC

Medizinisches Versorgungszentrum Dr. Stein + Kollegen

Microbe & Lab

Microvida

Mozand B.V.

Noordwest Ziekenhuisgroep Alkmaar

Novogenia GmbH

OLVG Lab B.V.

Pro Health Medical

RadboudUMC

Reinier Haga MDC

Rijnstate

RLM Dordrecht‐Gorinchem

Royal GD

Saltro, locatie Hudsondreef

Saltro, locatie Mississippidreef

Sanquin, NSS

St. Antonius ziekenhuis

Star‐shl

Stichting PAMM

Streeklab Haarlem

SYNLAB Heppignies

Synlab MVZ Weiden

SYNLAB Jena Oncoscreen

SYNLAB Laboratoire Collard

SYNLAB MVZ Trier

Tergooi ‐ CBSL

TLR International Laboratories

TNO

Universitair Medisch Centrum Groningen

Universitair Medisch Centrum Utrecht

VieCuri MC

Wageningen Bioveterinary Research

Ziekenhuis Gelderse Vallei

Ziekenhuis Rivierenland Tiel

Ziekenhuis StJansdal

Zuyderland Medisch Centrum

external quality assessment of laboratories performing

Documents