cross-platform evaluation of sensitive immunoassays for ... · * il-2 elisa standards failed...

Cross-Platform Evaluation of Sensitive

Immunoassays for Protein Biomarkers Alvydas Mikulskis, PhD

Principal Investigator, Translational Sciences, Biogen

AAPS NBC June 8, 2015

2

Biomarker Measurements Often Require High

Sensitivity Assays

• Most biomarkers are low abundance often present in sub-

ng/mL or lower levels in biological fluids

• Low pg/mL or better assay sensitivity is often required

• Below the limit of quantitation (BLQ) results are common in

biomarker measurements

http://en.wikipedia.org/wiki/Reference_ranges_for_blood_tests



3

Study Objectives

• Evaluate highly sensitive immunoassay platforms

for protein biomarkers:

Perform unbiased platform assessments using vendor-

optimized assays run by vendor experts in most cases

Use the same freshly-prepared and aliquoted set of 40

serum samples for all evaluations across platforms

Compare sensitivity between platforms using selected

cytokines with reported pg/mL to sub-pg/mL levels in

human serum

Evaluate additional assay performance parameters for

platform comparison

• Share experience with bioanalytical community

4

Evaluated Technologies and Assays

Technology (Vendor) Descriptions IL-2 IL-17a IL-6 TNFα

SiMoA™ HD-1 Analyzer

(Quanterix)

Fully automated singleplex bead-based digital ELISA coupled

with single molecule counting on microarray of beads

Erenna®

(Singulex)

Manual singleplex digital Single Molecule Counting (SMC™)

technology coupled with microparticle-based immunoassays

Milliplex®

(EMD Millipore)

Manual bead-based multiplex assays using Luminex®

technology

V-PLEX (MSD) Manual plate-based multiplex assays with ECL detection

High Sensitivity ELISA

(eBioscience or R&D Systems)

Manual singleplex plate-based High Sensitivity ELISA with

tyramide signal amplification

Biochip Array Technology

(RANDOX) Manual multiplex assay using ceramic surface biochip array

Ella™

(ProteinSimple) Automated multi-analyte cartridge-integrated ELISA assays

AMMP™ ViBE®

(Bioscale)

Manual singleplex bead-based assay coupled with Acoustic

Membrane MicroParticle detection on the ViBE platform

Imperacer®

(Chimera Biotec GmbH)

Manual singleplex plate-based assay coupled to quantitative-

PCR read out

5

Are We Comparing Apples to Apples?

• If not just apples, let’s focus on fruit!

6

Evaluated Performance Parameters

Assay Precision (Endogenous cytokines

expressed in PBMCs)

Analytical

Sensitivity (Recombinant Protein

Spike in Buffer) Frequency of

Endogenous

Analyte Detection

Correlation

Analysis Across

Platforms

Parallelism

9 Technologies, Up To 4 Cytokine Assays 40 Individual

Serum Samples

Composite

Score

7

Assay Precision

Technology (Vendor) Intra-assay %CV Inter-assay %CV

IL-2 IL-17a IL-6 TNFα IL-2 IL-17a IL-6 TNFα

SiMoA™ HD-1 Analyzer (Quanterix) 10.1 10.2 3.4 10.0 13.9 10.2 14.2 10.7

Erenna® (Singulex)

Milliplex® (EMD Millipore) 8.8 12.3 16.5 5.5 11.0 23.2 18.3 5.5

V-PLEX (MSD) 5.8 10.1 13.7 11.1 21.2 24.5 16.4


(eBioscience or R&D Systems) 10.8 11.4 16.7 22.1 17.1 17.0

Biochip Array Technology (RANDOX) 17.8 6.6 17.1 15.7 22.9 6.6 17.1 17.6

Ella™ (ProteinSimple) 9.7 4.2 9.4 4.5 15.5 10.8 15.3 8.5

AMMP™ ViBE® (Bioscale)

Imperacer® (Chimera Biotec GmbH) 12.4 17.4

Intra- and inter-assay precision was evaluated using pooled human serum

samples spiked with supernatants from stimulated PBMCs

• Most evaluated assays had the intra- and inter-assay precision <20%

• SimoA and Ella showed best and most consistent precision

Assay precision was evaluated in 2 to 4 different runs by a single operator using at least 3 replicate sets of spiked samples on each run

8

Analytical Sensitivity

Technology (Vendor) IL-2 IL-17a IL-6 TNFα

SiMoA™ HD-1 Analyzer (Quanterix) 0.02 0.02 0.16 0.28

Erenna® (Singulex) 0.05 0.03 0.02

Milliplex® (EMD Millipore) 0.49 2.93 0.73 1.71

V-PLEX (MSD) 0.72 2.36 0.38 0.62


(eBioscience or R&D Systems) –* 0.46 0.32 0.5

Biochip Array Technology (RANDOX) 0.9 3.95 0.12 0.59

Ella™ (ProteinSimple) 1.04 1.04 0.42 1.04

AMMP™ ViBE® (Bioscale) 3.28 3.28

Imperacer® (Chimera Biotec GmbH) 0.46

Analytical sensitivity was estimated in pg/mL as the lowest recombinant standard

that consistently meets precision and accuracy requirement of ≤25%

• Shown sensitivity values are based on recombinant protein spiked in a buffer

• What really matters is the ability of an assay to quantitate specifically an

endogenous analyte in a matrix of interest

Numbers in the table are concentrations of spiked recombinant protein in pg/mL

* IL-2 ELISA standards failed %CV≤25% acceptance criteria

9

Frequency of Endogenous Analyte Detection In

Individual Serum Samples

• Frequency of endogenous analyte detection (FEAD) reflects assay’s

ability to quantify endogenous analyte in samples of interest

• FEAD was evaluated in 40 individual human serum samples


SiMoA™ HD-1 Analyzer (Quanterix) 95 100 97 100

Erenna® (Singulex) 100 90 100

Milliplex® (EMD Millipore) 98 93 88 100

V-PLEX (MSD) 0 0 60 95


(eBioscience or R&D Systems) – 20 88 100

Biochip Array Technology (RANDOX) 30 5 93 90

Ella™ (ProteinSimple) 0 43 53 100

AMMP™ ViBE® (Bioscale) 53 70

Imperacer® (Chimera Biotec GmbH) 98 Numbers in the table correspond to percentage of samples with quantifiable values

Passed if FEAD > 70%; Maybe if 30% < FEAD ≤ 70%; Failed if FEAD ≤ 30%; NA = Not Assessed

• FEAD measure is valuable but only if the measurements are

specific and matrix interference is negligible

Passed

Maybe

Failed

NA

10

Correlation Analysis Helps To Infer True and False

Endogenous Analyte Measurements

• Results from 40

individual human

serum samples

compared

• Significant correlation if

p<0.05

• Also must be positive

and consistent across

platforms

Assumption: correlation of measurements for a given assay among majority

of platforms are likely to represent true endogenous analyte results

High

Correlation

Good

Correlation

Moderate

Correlation

No Correlation

11

Correlation Analysis Combined With FEAD

Numbers in the table correspond to percentage of samples with quantifiable values

• For IL-2 and IL-17a assays, only SimoA and Erenna platforms had acceptable

correlations though none of the IL-2 correlations were very strong

• Acceptable correlations were observed in all IL-6 assays though correlations of

Milliplex IL-6 results to other IL-6 results were weaker

• TNFα assay correlations were acceptable for SiMoA, V-PLEX, and Ella platforms

Passed

Maybe

Failed

NA

Frequency of Endogenous Analyte Detection + Correlation

Passed if both FEAD and correlation passed; Failed if either FEAD or correlation failed; Maybe all other in between cases; NA = Not Assessed





V-PLEX (MSD) 0 0 60 95






Imperacer® (Chimera Biotec GmbH) 98





V-PLEX (MSD) 0 0 60 95







12

Parallelism – Further Insight Into Sensitivity

Parallelism assessments were performed in samples with mid to high levels

of a measured analyte in a given assay

Technology (Vendor) All Samples

IL-2 IL-17a IL-6 TNFα

SiMoA™ HD-1 Analyzer (Quanterix) 2/3 6/8 10/10 10/10

Erenna® (Singulex) 2/6 6/6 5/5

Milliplex® (EMD Millipore) 1/8 1/7 1/7 1/14

V-PLEX (MSD) 7/11 8/11


(eBioscience or R&D Systems) 5/8 0/5

Biochip Array Technology (RANDOX) 1/8 1/7

Ella™ (ProteinSimple) 7/7 3/3

AMMP™ ViBE® (Bioscale) 0/4 0/3

Imperacer® (Chimera Biotec GmbH) 0/13 Numbers in the table correspond to a number of samples that met parallelism acceptance criteria / total number of evaluated samples

Parallelism acceptance criteria = two or more sample dilutions with reproducible result within the 20% recovery range to a result at MRD

Passed if majority of samples met parallelism acceptance criteria; Failed otherwise; NA = Not Assessed

• Parallelism results were acceptable for IL-6 ELISA and in most

assays when using SiMoA, Erenna, V-PLEX, and Ella platforms

Passed

Failed

NA

13

Composite Score of All Analyses

• Overlaying all analysis results provided most stringent assessment of assay’s

ability to quantitatively measure endogenous analyte

• SimoA and Erenna platforms provided the most sensitive and reliable

measurements for evaluated assays

Numbers in the table correspond to percentage of samples with quantifiable values

Passed

Maybe

Failed

NA

Frequency of Endogenous Analyte Detection + Correlation + Parallelism

Passed if all assessments passed; Failed if at least one assessment failed; Maybe all other in between cases; NA = Not Assessed





V-PLEX (MSD) 0 0 60 95











V-PLEX (MSD) 0 0 60 95











V-PLEX (MSD) 0 0 60 95







14

Summary

Performed unbiased cross-platform and cross-assay evaluation using

9 technology platforms and up to 4 vendor-optimized cytokine assays

Sensitivity and precision-accuracy evaluations using spiked

recombinant proteins may provide only limited information about

assay performance

Frequency of endogenous analyte detection, correlation between

platforms, and parallelism were shown to be useful parameters for

assessing performance of biomarker assays

SimoA (Quanterix) and Erenna (Singulex) platforms provided the

most sensitive and reliable measurements for evaluated assays

Our findings apply to evaluated cytokine assays and may not be

generalized for other assays

Fit-for-purpose approach is recommended when selecting the

appropriate biomarker assay platform

15

Acknowledgement

Dave Yeung

Shawn Ciotti

Elham Gharakhani

Shobha Purushothama

Brian Schlain

Geoffrey Kuesters

Chase Shen

Doug Donaldson

Lakshmi Amaravadi

Meena Subramanyam

Technology Vendors

Serum samples procured by

16

There Are Better Ways to Compare Fruit

cross-platform evaluation of sensitive immunoassays for ... · * il-2 elisa standards failed...

Documents