![Page 1: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/1.jpg)
Biomarkers: the Good, the Bad and the AmbiguousEnrique F. Schisterman, PhD
Epidemiology Branch – DESPR – NICHD
1
![Page 2: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/2.jpg)
In Memory of My Friend Sholom Wacholder
2
“I never feel I truly understand something if I can’t explain it to someone else, orally or in writing,”
![Page 3: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/3.jpg)
Outline
Background
Limit of Detection
Pooling Biomarkers – Hybrid Design
Calibration Curves
Lipid Standardization (if we have time)
Conclusions
3
![Page 4: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/4.jpg)
Background
Biomarker: A specific physical trait used to measure or indicate the effects or progress of a disease or condition
Newly developed laboratory methods expand the number of biomarkers on a daily basis Cost Measurement Error Causal Link to Disease
4
![Page 5: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/5.jpg)
Motivation
Preliminary analysis of salivary concentrations of cortisol from the LIFE study
P=0.04Shipment 1 n Mean SDMichigan 85 0.40 0.19Texas 142 0.57 0.79
5
![Page 6: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/6.jpg)
Cortisol by Site & Plate6
02
46
8
Ship1 Ship2 Ship1 Ship2 Ship3
1 2 3 4 5 1 2 3 1 2 3 4 5 1 2 1 2 3 4 5 6 7 8
Michigan Texas
ug/
dL
Graphs by Site
![Page 7: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/7.jpg)
ROC curve—background
Before a marker is used its discriminating accuracy needs to be evaluated
Most commonly used tool - Receiver Operating Characteristic (ROC) curve.
We consider only continuous markers
![Page 8: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/8.jpg)
Distribution of marker in Healthy and Disease
c c c
![Page 9: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/9.jpg)
* ROC curve – graph of (1-p,q) for all c.
ROC(1-p)=1-FD(FH-1(p))
* ROC defined over all possible thresholds gives the entire range of possible sensitivities and specificities.
* Diagnostic accuracy evaluated using measures including the area under the curve (AUC AUC = Prob(XH<XD)) and Youden’s index (J)
![Page 10: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/10.jpg)
ROC curve example Complete Separation
q(c)
1 – p(c)0 1
1
Chanc
e lin
e
Df
c q(c)p(c)
q(c) = 1 and p(c) = 1 for some c
![Page 11: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/11.jpg)
ROC curve example Complete Overlap
q(c) = 1- p(c), for all c
q(c)
1 – p(c)0 1
1
Chanc
e lin
e
c q(c)p(c)
![Page 12: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/12.jpg)
ROC curve example Partial Separation
Sensitivity by 1- Specificity
P( True Pos.) by P( False Pos.) across all c
q(c)
1 – p(c)0 1
1
Chance
lineDf
c q(c)p(c)
![Page 13: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/13.jpg)
q(c)
1 – p(c)0 1
1
ROC: AUC
Area Under the Curve Overall discriminatory ability Area under the ROC curve
x
dxdyygxf
YXPAUC
)()(
)(
![Page 14: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/14.jpg)
Do Common Laboratory Practices Affect our Estimates of Risk?
Limit of Detection
Measurement Error
Calibration Curves
Lipid Standardization
14
![Page 15: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/15.jpg)
Outline
Background
Limit of Detection
Pooling Biomarkers – Hybrid Design
Calibration Curves
Lipid Standardization
Conclusions
15
![Page 16: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/16.jpg)
Reporting of Biomarker Data
ID Z1 3.12 1.53 8.44 0.85 5.46 3.27 2.08 5.89 13.410 2.511 1.912 6.1
• Reporting threshold is equal to 2.2
ID Z1 3.12 ND3 8.44 ND5 5.46 3.27 ND8 5.89 13.410 2.511 ND12 6.1
Report values < threshold as ‘not detected’
ID Z1 3.12 1.13 8.44 1.15 5.46 3.27 1.18 5.89 13.410 2.511 1.112 6.1
Report values < threshold as one half the value of the threshold
16
![Page 17: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/17.jpg)
Conventional Determination of the Limit of Detection (LOD)
6.15)3( blanksblanksLOD
BLANK SERIES• 10.0• 5.0• 8.1• 7.1• 4.0• 11.3• 12.0• 8.0• 7.7• 7.0Mean = 8.02Std Dev = 2.53
0 5 10 15 20
17
![Page 18: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/18.jpg)
Example of LOD left-censored data
0 5 10 15 20 25 30 35
Blanks
“True” biomarker
Better LOD?
18
![Page 19: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/19.jpg)
Example of LOD left-censored data
0 5 10 15 20 25 30 35
Blanks
“True” biomarkerObserved biomarker (samples)
19
![Page 20: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/20.jpg)
CONTROLS
CASES
Why is this a problem?Comparisons of PCBs in cases and controls
Controls—mean PCB Cases—mean PCB
Effect size
LOD
Blanks
20
![Page 21: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/21.jpg)
Approaches for LOD/ missing data
Simplest approach is substitution Under certain circumstances yield minimal bias Conventionally, values below the LOD are usually
1. replaced by zero, LOD, LOD/2, LOD/√22. excluded 3. retained
Model based approaches Likelihood models (Perkins et al., AJE 2007)
Multiple imputation (Harel et al., Epidemiology 2010)
Schisterman EF, Vexler A, Whitcomb BW, Liu A. AJE 2006
21
![Page 22: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/22.jpg)
CONTROLS
CASES
Why is this a problem?Comparisons of PCBs in cases and controls
LOD
Impute what? 0 LODLOD/2
22
![Page 23: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/23.jpg)
LOD Simulation
Purpose: To evaluate the effect of the handling of values below the LOD on risk estimates
Simulated data from a normal and log normal distribution and varied:
Effect size Variance of PCBs in the exposure group LOD level Measurement error mean and variance
23
![Page 24: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/24.jpg)
Effect of Handling of Values < LOD on %Bias
*LOD “low” indicates 1.6 SDs below the mean of controls, resulting in imputed values for a small number of data points. LOD “high” indicates 1 SD above the mean of the controls, resulting in imputed values for a large number of both controls and cases
24
Effect size = .25 SD
Method for values < LOD LOD high LOD low 1. Replace by a. Zero -59.0 -25.1 b. LOD -187.1 -40.8
c. LOD/2 -19.3 -18.1 d. LOD/2 -17.1 -15.9
2. Exclude (truncated) -314.2 -265.3 3. Retain (observed) -11.5 -11.7
![Page 25: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/25.jpg)
LOD—Conclusions
Choice of how to handle values below the LOD can result in a loss of accuracy in estimating risk
Retaining observed values below the LOD produces the least biased estimates
Substitution of LOD/√2 for values below the LOD produces not terribly biased estimates
25
![Page 26: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/26.jpg)
How the LOD Affects the ROC Curve?26
![Page 27: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/27.jpg)
LOD: ROC curve
Properly accounting for the mass yields a ROC curve bias toward the null that is identical for any choice of a<d.
Xf
a
ROC
* Schisterman, et al.(2005)
![Page 28: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/28.jpg)
MLE to correct for LOD
Normal Case
MLE’s of dist.parameters
Yk
jYYYYjY
Y
YYYYY
knz
kCzL
1
2
22
1)(log)()(
log)|,(log
YYY d /)( where
* Gupta(1952)
![Page 29: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/29.jpg)
Normal Case
MLE’s of dist.parameters
MLE’s of ROCmeasures
InvarianceProperty
22 ˆˆ
ˆˆˆ
YX
YXCUA
MLE to correct for LOD
![Page 30: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/30.jpg)
Normal Case
MLE’s of dist.parameters
MLE’s of ROCmeasures
CI’s for ROCmeasures
2nd Deriv. and Fisher Info.
22
22
1
YX
X
YX
X
AUC
22
22
1
YX
Y
YX
Y
AUC
NzCUA A
2
2
/
ˆ 1
22
2
1
)(1)ˆ(ˆ2
)(1)(
)(1)(
)()(
][)ˆ,ˆ()ˆ(
qp
q
qqp
ICovCov
MLE to correct for LOD
![Page 31: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/31.jpg)
Gamma Case
MLE’s of dist.parameters
where
,)(log)(log)1(
log)(log)|,(log
1 1
**
Y
YY
Y
YY
n
knjYYY
n
knjjYjYY
YYYYYY
Gknzz
kCzL
Y
jYjY
zz
*
Y
dyeyG y
yY
0
1
)(
1)(
YY
d
* Harter and Moore(1967)
MLE to correct for LOD
![Page 32: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/32.jpg)
dtttCUA XY
Q
YX
YX 1
0
1 1
ˆ
ˆ
ˆ )()ˆ()ˆ(
)ˆˆ(ˆ
Gamma Case
MLE’s of dist.parameters
MLE’s of ROCmeasures
InvarianceProperty
MLE to correct for LOD
![Page 33: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/33.jpg)
,)(
);1()ln(
)(1
)()1()(
)(
);1(21
)(
);1();1();1())(ln(
**
**
2112
*****
*
222
2
2***
2
2
11
ddq
q
dhdpII
q
dhddqdhd
dpI
q
ddd
d
dI
Gamma Case
MLE’s of dist.parameters
MLE’s of ROCmeasures
CI’s for ROCmeasures
2nd Deriv. and Fisher Info.
NzCUA A
2
2
/
ˆ
X
X
X AUC
AUC
AUC
Y
Y
Y AUC
AUC
AUC
1
2221
12111][)ˆ,ˆ()ˆ(
II
IIICovCov θ
MLE to correct for LOD
![Page 34: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/34.jpg)
LOD: ROC curve
Corrected!
Yf
Xf
a
ROC
* Schisterman, et al.(2005)
![Page 35: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/35.jpg)
Evaluation of MLE
0
1
10
Example
PCB114 levels of 28 cases 51 controls classifying women with and without endometriosis.
Empirical 0.701 (0.576, 0.827)
MLE 0.777 (0.655, 0.899)
*assuming gamma distributions and d=0.005
AUC
![Page 36: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/36.jpg)
Outline
Background
Limit of Detection
Pooling Biomarkers – Hybrid Design
Calibration Curves
Lipid Standardization
Conclusions
36
![Page 37: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/37.jpg)
What is pooling?
Physically combining several individual specimens to create a single mixed sample
Pooled samples are the average of the individual specimens
1
2 p
37
![Page 38: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/38.jpg)
Random Sample of Biospecimens
RANDOM SAMPLE
Randomly select 20 samples
FULL DATA
N = 40 Individual Biospecimens
38
![Page 39: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/39.jpg)
Pooling Biospecimens
POOLED DATA
40 samples in groups of 2
FULL DATA
N = 40 Individual Biospecimens
39
![Page 40: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/40.jpg)
Un-pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
Effect of Pooling on Markers
Pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
40
Un-pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
Pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
![Page 41: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/41.jpg)
0
20
40
60
80
100
120
0 2 4 6 8 10
# of Pooled Samples
# o
f A
ssa
ys o
f P
oo
led
Sa
mp
les
20
50
100
200
Number of assays required to reach equivalency on ROC Curves
![Page 42: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/42.jpg)
Limit of Detection and Pooling
/i i
ii
x if x LODZ
N A if x LOD
Unpooled Specimens
Pooled Specimens
( ) ( )( )
( )
, if LOD
N/A, if LOD
p pp i i
i pi
x xZ
x
![Page 43: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/43.jpg)
Un-pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
Effect of Pooling on Markers Affected by an LOD
Pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
43
Un-pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
Pooled
0
0.2
0.4
0.6
0.8
1
-4 -3 -2 -1 0 1 2 3 4
![Page 44: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/44.jpg)
Estimation Based on Pooling Using Likelihood Function
dXi
i
k
i
XfdXkNk
NL
:
)(Pr)!()!1(
!
0ˆ
);ˆ,ˆ(log,0
ˆ);ˆ,ˆ(log
LL
Differentiating with respect to
ˆ,ˆ
,
To obtain estimates of
Schisterman et al. Bioinformatics 2006
![Page 45: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/45.jpg)
Efficiency of the Mean and Variance
Variance of Estimated Mean
Variance of Estimated Variance
FULL DATA POOLED RANDOMFULL DATA POOLED RANDOM
45
LOD below Mean
LOD below Mean
LOD above Mean
LOD above Mean
![Page 46: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/46.jpg)
Pooling and Random Sampling
Pooling advantages Reduces the number of assays we need to test Efficiently estimates the mean Cost-effective
Random sampling advantages Reduces the number of assays we need to test Efficiently estimates the variance Cost-effective & easy to implement
46
![Page 47: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/47.jpg)
Hybrid Design: Pooled—Unpooled
Creates a sample of both pooled and unpooled samples Takes advantage of the strengths of both the pooling and
random sampling designs
Reduces number of tests to perform Cuts overall costs Gains efficiency (by using pooling technique) Accounts for different types of measurement error without
replications
– Pooling error– Random measurement error– LOD
47
![Page 48: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/48.jpg)
Unpooled: X1,…,X5
Pooled: Z1,…,Z15
Hybrid Sample S: X1,…,X5,Z1,…,Z15
Setup of Hybrid Design
Unpooled: X1,…,X[αn]
Pooled: Z1,…,Z[(1-α)n]
In GeneralHybrid Sample S: X1,…,X[αn],Z1,…,Z[(1-α)n]
α is the proportion of unpooled samples
48
![Page 49: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/49.jpg)
Maximum Likelihood Estimators
Random Sampling Pooling
In order to estimate the variance, α cannot be zero.
Schisterman EF et al, Stat Med 2010
pn
zxni
xi
p
2)1(
2
2 ˆ
)1(
)ˆ(
ˆ
22
22
ˆ)1(ˆ
ˆ)1(ˆˆ
xz
xzx nn
znxn
n
xni
xi
x
2
2
)ˆ(ˆ
2
)1(
2
2
2
2
)(
ln)1(2
)(ln),,(
z
nixi
zx
nixi
xpxx
z
nx
n
49
![Page 50: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/50.jpg)
Hybrid Design: Pooled—Unpooled
Create a sample of both pooled and un-pooled samples that takes advantage of the strengths of the pooled and random sample designs Reduce number of tests to perform Cut overall costs Gain efficiency (by using pooling technique) Accounts for different types of measurement
error without replicationsPooling errorRandom measurement error Limit of detection
![Page 51: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/51.jpg)
Mathematical Setup
),(~ 2xxi NX NiX i 1
),0(~ 2p
pi Ne
pi
ip
pijii eX
pZ
1)1(
1
2
2
,~ px
xi pNZ
Individual Samples
Pooled Samples
p is the pooling group size
Average of p individual samples pooled together
Pooling error
![Page 52: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/52.jpg)
Maximum Likelihood Estimators
22
(1 )
2 2
( )( )( , , ) ln (1 ) ln
2 2
i xi xi ni n
x x p x zx z
zxn n
Random Sampling Pooling
22
22
ˆ)1(ˆ
ˆ)1(ˆˆ
xz
xzx nn
znxn
n
xni
xi
x
2
2
)ˆ(ˆ
2
2(1 )2
ˆ( )ˆ
ˆ(1 )
i xi n x
p
z
n p
In order to estimate the variance, α cannot be zero.
![Page 53: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/53.jpg)
Add in Measurement Error
),0(~ 2M
Mi Ne
Individual Samples + ME
Pooled Samples + ME
p is the pooling group size
Measurement error
* 2 2~ ( , )i x x MX N
22
2* ,~ Mp
xxi p
NZ
![Page 54: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/54.jpg)
Hybrid Design Example: IL-6
Measured IL-6 on 40 MI cases and 40 controls
Biological specimens were randomly pooled in groups of 2, for the cases and controls separately, and remeasured
We want to evaluate the discriminating ability of this biomarker in terms of AUC
54
![Page 55: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/55.jpg)
Hybrid Design Example: IL-6
n αx αy AÛC Var(AÛC)
Empirical 40 1.00 1.00 0.640 0.0036
Hybrid design: Optimal α 20 0.40 0.35 0.621 0.0049
Random sample: α=1 20 1.00 1.00 0.641 0.0071
Hybrid design reduced the variability of Var(AÛC) by 32% as compared to taking only a random sample
55
![Page 56: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/56.jpg)
Summary—Hybrid Design
Hybrid design is a more efficient way to estimate the mean and variance of a population
Cost-effective
Yields estimate of measurement error without requiring repeated measurements
Here we focus on normally distributed data, but can be applied to other distributions as well
56
![Page 57: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/57.jpg)
Outline
Background
Limit of Detection
Pooling Biomarkers – Hybrid Design
Calibration Curves
Lipid Standardization
Conclusions
57
![Page 58: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/58.jpg)
Measurement of G-CSF
Chemiluminescence assays 96-well plate Antibody against the biomarker of interest Set of standards of known biomarker concentration
included in each batch Set of samples (concentration unknown) Light emitting molecule binds to bound biomarker
58
![Page 59: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/59.jpg)
59
![Page 60: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/60.jpg)
Measurement of Cytokines
Cytokines are not measured directly Antibodies against analyte(s) coat wells
60
![Page 61: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/61.jpg)
Measurement of Cytokines
Samples added, analyte binds to antibodies Unbound proteins are washed away
61
![Page 62: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/62.jpg)
Measurement of Cytokines
A ‘tag’ is added to the assay that binds to the protein – antibody complex that produces color
62
![Page 63: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/63.jpg)
Measurement of Cytokines
A ‘tag’ is added to the assay that binds to the protein – antibody complex that produces color
The intensity of the color is measured
63
![Page 64: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/64.jpg)
ELISA/Multiplex Layout
Step 1: prepare antibodies mixture and add to plate Step 2: prepare calibrators, add to plate Step 3: prepare unknowns, add to plate
64
![Page 65: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/65.jpg)
Use of Chemiluminescence Assays for Measuring Protein Concentrations
Use calibration to convert relative measures to the desired unit of concentration From optical density in relative fluorescence units (RFU)
to concentration in pg/mL
Current practice is per assay calibration Results in potentially large calibration datasets used only
minimally in current practice
65
![Page 66: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/66.jpg)
Calibrating the Assay: The Standard Curve
66
![Page 67: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/67.jpg)
Calibrating the Assay: The Standard Curve
The human G-CSF standard curve is provided only for demonstration
A standard curve must be generated each time an assay is run, utilizing values from the Standard Value Card included in the Base Kit
Potential variation in the relation between relative fluorescence and concentration Chromophore potentially affected by temperature, humidity,
etc.
67
![Page 68: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/68.jpg)
G-CSF and Miscarriage in the CPP
Case-control study nested in the Collaborative Perinatal Project study cohort 462 miscarriage cases 482 non-miscarriage controls
Serum biospecimens from early pregnancy, prior to miscarriage onset
For n = 944, 24 assays were used
68
![Page 69: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/69.jpg)
69
![Page 70: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/70.jpg)
Unadjusted model Adjusted model OR [95% CI] OR [95% CI] Factor G-CSF 0.84 [0.72, 0.99] 0.78 [0.64, 0.95]
This estimate is based on the conventional batch specific approach
70
![Page 71: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/71.jpg)
Objective
Question: Is the current practice of standard batch-specific calibration the best use of information?
To evaluate the effect of different approaches for calibration models on risk estimation
To assess bias associated with different approaches
71
![Page 72: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/72.jpg)
Data from the calibration experiments
24 batches, each with 7 known concentrations measured in replicate Batches varied by
Shape Location Agreement between replicates Presence of outliers
72
![Page 73: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/73.jpg)
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Batch 1 Calibration Curve – G-CSF
Standard 1 – undiluted (conc = 6000 pg/mL)
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
*All calibration data (in log10)
73
![Page 74: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/74.jpg)
Batch 1 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Standard 2 – 1/3rd dilution (conc = 2000 pg/mL)
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
74
![Page 75: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/75.jpg)
Batch 2 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
75
![Page 76: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/76.jpg)
Batch 3 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
76
![Page 77: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/77.jpg)
Batch 6 Calibration Curve – G-CSF
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
77
![Page 78: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/78.jpg)
Batch 9 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
78
![Page 79: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/79.jpg)
Batch 10 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
79
![Page 80: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/80.jpg)
Batch 21 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
80
![Page 81: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/81.jpg)
Batch 22 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
81
![Page 82: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/82.jpg)
Batch 24 Calibration Curve – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
82
![Page 83: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/83.jpg)
All Calibration Curves Collapsed – G-CSF
l ogod
1
2
3
4
l ogcon
0 1 2 3 4
s i mpl e l i near
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4
Mea
sure
d op
tical
den
sity
Fixed ‘known’ concentration
83
![Page 84: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/84.jpg)
Effect of Calibration Method on Logistic Regression Results
Calibration models As observed Outliers removed aOR 95%CI aOR 95%CI Forward Collapsed Linear 0.34 (0.13, 0.90) 0.27 (0.10, 0.73) Batch specific Linear 0.73 (0.46, 1.17) 0.60 (0.33, 1.10) Mixed model Linear 0.67 (0.39, 1.14) 0.56 (0.30, 1.06) Collapsed Curvilinear 0.21 (0.05, 0.84) 0.25 (0.09, 0.71) Batch specific Curvilinear 0.81 (0.57, 1.16) 0.98 (0.93, 1.02) Mixed model Curvilinear ~ ~ ~ ~ Reverse Collapsed Linear 0.37 (0.15, 0.91) 0.28 (0.11, 0.73) Batch specific Linear 0.63 (0.37, 1.11) 0.58 (0.32, 1.07) Mixed model Linear 0.43 (0.19, 0.94) 0.53 (0.27, 1.02) Collapsed Curvilinear 0.37 (0.15, 0.91) 0.29 (0.11, 0.74) Batch specific Curvilinear 0.86 (0.53, 1.41) 0.67 (0.38, 1.16) Mixed model Curvilinear 0.50 (0.27, 0.95) ~ ~
84
![Page 85: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/85.jpg)
Simulation Study
1. Generate dataset with:
True biomarker concentration True effect on risk Overall relation between concentration and RFU Batch variability Occasional outliers
2. Simulate calibration experiments to estimate RFU – concentration relation according to each approach
3. Assess bias and variance of estimators from risk models
85
![Page 86: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/86.jpg)
Simulation Study:The Biomarker
Biomarker: exp(X ~ N(5,1))
Miscarriage risk: OR = 1.05, 1.15 or 1.65 β={0.05, 0.14, 0.50}
Conc. and OD: OD determined througha single function
86
![Page 87: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/87.jpg)
Summary of simulation study resultsComparison of shape, model for β = 0.14
CollapsedMixedBatch-specific
Linear Curvilinear Linear CurvilinearFORWARDS REVERSE
β̂
0.14
Whitcomb et al, Epidemiology 2010
87
![Page 88: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/88.jpg)
Conclusions
Underestimation of effects due to calibration approach has broad implications
Use of conventional batch-specific approaches performed poorly Greatest bias to estimates in simulations Most prone to loss of data for batches with failure of
some calibration points
88
![Page 89: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/89.jpg)
Outline
Background
Limit of Detection
Pooling Biomarkers – Hybrid Design
Calibration Curves
Lipid Standardization (later if we have time)
Conclusions
89
![Page 90: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/90.jpg)
Outline
Background
Limit of Detection
Pooling Biomarkers – Hybrid Design
Calibration Curves
Lipid Standardization
Conclusions
90
![Page 91: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/91.jpg)
Do Common Laboratory Practices Affect our Estimates of Risk?
Limit of Detection Request the observed values Design away using hybrid methods and overcome cost, LOD and ME
Calibration Curves Study Design should include a calibration curve plan
Standardization Don’t do it!
YES!91
![Page 92: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/92.jpg)
Questions?
Thank you!
![Page 93: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/93.jpg)
Future Direction
Limit of detection: Linking the CV, AUC and calibration.
22
2
x
xR
x
xCV
CV
RAUC
x
X
x
X
2222
),(~ 2xXNX ),0(~ 2
NW
Reliability index could be applied such that
Coefficient of variation is used by labs during calibration
For normally distributed biomarkers, we can use these two summaries such that
93
![Page 94: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/94.jpg)
Limit of detection: Linking the CV, AUC and calibration.
22
2
x
xR
x
xCV
CV
RAUC
x
X
x
X
2222
),(~ 2xXNX ),0(~ 2
NW
Reliability index could be applied such that
Coefficient of variation is used by labs during calibration
For normally distributed biomarkers, we can use these two summaries such that
94
![Page 95: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/95.jpg)
We can investigate the relation between these factors easily because R and CV are relative measures.
95
![Page 96: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/96.jpg)
This is of particular importance because for certain types of measurement processes, the Limit of detection is based on the CV and a more appropriate cut-point might be better generated using ROC methods for differentiating between signal and noise.
96
![Page 97: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/97.jpg)
modeling calibration data
dependent/independent variables
Assay yields OD for observations in the main dataset
‘Forward’ (regression calibration) Estimate & use calibration equations in the same order, i.e.:
‘Reverse’ Estimate calibration equations, flip for calibration, i.e.:
)(ODfionConcentrat
)(ODfionConcentrat
)( ionconcentratfOD
97
![Page 98: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/98.jpg)
modeling calibration data
treatment of batch
Approach to batch, OD as independent variable
use calibration data to model:
conijk = α + βODijkCollapsed
conijk = αk + βkODijk Batch-specific effects
conijk = α* + β*ODijk + aj + bjODijkMixed effects
i = standard 1 – 7
j = replicate 1 – 2
k = batch 1 – 24
98
![Page 99: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/99.jpg)
modeling calibration data
treatment of batch
Approach to batch, concentration as dependent variable
use calibration data to model:
ODijk = α + βconijkCollapsed
ODijk = αk + βkconijk Batch-specific effects
ODijk = α* + β*conijk + ak + bkconijkMixed effects
i = standard 1 – 7
j = replicate 1 – 2
k = batch 1 – 24
99
![Page 100: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/100.jpg)
Data calibration
Parameters estimated for linear and curvilinear models
For concentration as dependent Unknown concentrations calibrated from estimated f(OD)
)(ˆˆ ODionconcentrat
)(ˆˆ ODionconcentrat
100
![Page 101: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/101.jpg)
Data calibration
Parameters estimated for linear and curvilinear models
For concentration as dependent Unknown concentrations calibrated from estimated f(OD)
221 )(ˆ)(ˆˆ ODODionconcentrat
221 )(ˆ)(ˆˆ ODODionconcentrat
101
![Page 102: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/102.jpg)
Data calibration
Parameters estimated for linear and curvilinear models
For OD as dependent Unknown concentrations calibrated as:
ˆˆ
OD
con
)(ˆˆ ionconcentratOD
102
![Page 103: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/103.jpg)
Data calibration
Parameters estimated for linear and curvilinear models
For OD as dependent Unknown concentrations calibrated as:
22
221
ˆ2
ˆ)ˆ(4ˆ
OD
con
221 )(ˆ)(ˆˆ conconOD
103
![Page 104: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/104.jpg)
Simulation study the calibration relation
7 known concentrations in replicate/batchMeasured OD for each true concentration
True overall relation by 4 parameter logistic
4
3
212
1
i
ix
y
34
213
2
01
at slope2)(at
)(lim
)(lim
xfx
xf
xf
x
x
34
213
2
01
at slope2)(at
)(lim
)(lim
xfx
xf
xf
x
x
CONCx
ODy
104
![Page 105: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/105.jpg)
Simulation study the calibration relation
7 known concentrations in replicate/batchMeasured OD for each true concentration
True overall relation by 4 parameter logistic
Random batch variability Random failures with frequency 0.01
4
3
212
1
ixOD
105
![Page 106: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/106.jpg)
Calibration approaches
Calibration models varied as regards:1. Dependent/independent variable
– Measured OD and fixed known amount
2. Shape (linear or quadratic in log)3. Treatment of batch
Disregard (collapsed) Model as fixed effect (batch-specific) Model as random effect
106
![Page 107: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/107.jpg)
Results
Calibration models Plots of cross-classification
Effects on risk estimation Simulation study
107
![Page 108: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/108.jpg)
s i mpl e l i near
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4
All batches combined – outliers in the calibration set
0 1 2 3
1
108
![Page 109: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/109.jpg)
s i mpl e l i near
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4
All batches combined – outliers in the calibration set
0 1 2 3
1
109
![Page 110: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/110.jpg)
Bat ch number of assay 1 2 3 4 5 6 7 89 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
A A A B B B C C C D D D E E E F F F G G G H H HI I I J J J K K K L L L M M M N N N O O O P P PQ Q Q R R R S S S T T T U U U V V V W W W
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4 5
AA
A
A
AA
AA
AA
AA
A
A
BB
B
B
BB
BB
BB
B
B
BB
CC
CC
CC
CC
CC
CC
CC
DD
DD
DD
DD
D
D
DD
D
D
E
E
EE
EE
EE
E
E
EE
EE
F
F
FF
FF
FF
FF
FF
FF
GG
GG
GG
GG
GG
GG
GG
HH
HH
HH
HH
HH
HH
HH
I
I II
II
II
II
I
I
II
JJ
JJ
JJ
JJ
J
JJJ
J
J
KK
K
K
KK
KK
K
K
K
K
KK
LL
LL
LL
LL
LL
LL
LL
MM
MM
MM
MM
MM
MM
MM
NN
NN
NN
NN
NN
NN
NN
OO
O
O
OO
OO
O
O
O
O
OO
Q
Q
RR
R
RR
R
R
RR
RR
RR
SS
SS
SS
SS
SS
SS
S
S
TT
TT
TT
TT
TT
TT
TT
UU
UU
U
U
U
U
UU
UU
U
U
VV
VV
VV
VV
VV
VV
V
V
WW
WW
WW
WW
WW
WW
WW
s i mpl e l i near
Bat ch number of assay 1 2 3 4 5 6 7 89 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
A A A B B B C C C D D D E E E F F F G G G H H HI I I J J J K K K L L L M M M N N N O O O P P PQ Q Q R R R S S S T T T U U U V V V W W W
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4
AA
A
A
AA
AA
AA
AA
A
A
BB
B
B
BB
BB
BB
B
B
BB
CC
CC
CC
CC
CC
CC
CC
DD
DD
DD
DD
D
D
DD
D
D
E
E
EE
EE
EE
E
E
EE
EE
FF
FF
FF
FF
FF
FF
FF
GG
GG
GG
GG
GG
GG
GG
HH
HH
HH
HH
HH
HH
HH
I
I II
II
II
II
I
I
II
JJ
JJ
JJ
JJ
J
JJJ
JJ
KK
K
K
KK
KK
K
K
K
K
KK
LL
LL
LL
LL
LL
LL
LL
MM
MM
MM
MM
MM
MM
MM
NN
NN
NN
NN
NN
NN
NN
OO
O
O
OO
OO
O
O
O
O
OO
Q
Q
RR
R
RR
R
R
RR
RR
RR
SS
SS
SS
SS
SS
SS
S
S
TT
TT
TT
TT
TT
TT
TT
UU
UU
U
U
U
U
UU
UU
U
U
VV
VV
VV
VV
VV
VV
V
V
WW
WW
WW
WW
WW
WW
WW
All calibration data (in log10), with batch-specific regression
All batches combined outliers in the calibration set
110
![Page 111: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/111.jpg)
Bat ch number of assay 1 2 3 4 5 6 7 89 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
A A A B B B C C C D D D E E E F F F G G G H H HI I I J J J K K K L L L M M M N N N O O O P P PQ Q Q R R R S S S T T T U U U V V V W W W
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4 5
AA
A
A
AA
AA
AA
AA
A
A
BB
B
B
BB
BB
BB
B
B
BB
CC
CC
CC
CC
CC
CC
CC
DD
DD
DD
DD
D
D
DD
D
D
E
E
EE
EE
EE
E
E
EE
EE
F
F
FF
FF
FF
FF
FF
FF
GG
GG
GG
GG
GG
GG
GG
HH
HH
HH
HH
HH
HH
HH
I
I II
II
II
II
I
I
II
JJ
JJ
JJ
JJ
J
JJJ
J
J
KK
K
K
KK
KK
K
K
K
K
KK
LL
LL
LL
LL
LL
LL
LL
MM
MM
MM
MM
MM
MM
MM
NN
NN
NN
NN
NN
NN
NN
OO
O
O
OO
OO
O
O
O
O
OO
Q
Q
RR
R
RR
R
R
RR
RR
RR
SS
SS
SS
SS
SS
SS
S
S
TT
TT
TT
TT
TT
TT
TT
UU
UU
U
U
U
U
UU
UU
U
U
VV
VV
VV
VV
VV
VV
V
V
WW
WW
WW
WW
WW
WW
WW
s i mpl e l i near
Bat ch number of assay 1 2 3 4 5 6 7 89 10 11 12 13 14 15 16
17 18 19 20 21 22 23 24
A A A B B B C C C D D D E E E F F F G G G H H HI I I J J J K K K L L L M M M N N N O O O P P PQ Q Q R R R S S S T T T U U U V V V W W W
l ogod
1
2
3
4
5
l ogcon
0 1 2 3 4
AA
A
A
AA
AA
AA
AA
A
A
BB
B
B
BB
BB
BB
B
B
BB
CC
CC
CC
CC
CC
CC
CC
DD
DD
DD
DD
D
D
DD
D
D
E
E
EE
EE
EE
E
E
EE
EE
FF
FF
FF
FF
FF
FF
FF
GG
GG
GG
GG
GG
GG
GG
HH
HH
HH
HH
HH
HH
HH
I
I II
II
II
II
I
I
II
JJ
JJ
JJ
JJ
J
JJJ
JJ
KK
K
K
KK
KK
K
K
K
K
KK
LL
LL
LL
LL
LL
LL
LL
MM
MM
MM
MM
MM
MM
MM
NN
NN
NN
NN
NN
NN
NN
OO
O
O
OO
OO
O
O
O
O
OO
Q
Q
RR
R
RR
R
R
RR
RR
RR
SS
SS
SS
SS
SS
SS
S
S
TT
TT
TT
TT
TT
TT
TT
UU
UU
U
U
U
UU
UU
U
U
VV
VV
VV
VV
VV
VV
V
WW
WW
WW
WW
WW
WW
WW
All calibration data (in log10), with batch-specific regression
All batches combined outliers in the calibration set
111
![Page 112: Biomarkers: the Good, the Bad and the Ambiguous Enrique F. Schisterman, PhD Epidemiology Branch – DESPR – NICHD 1](https://reader035.vdocuments.site/reader035/viewer/2022081401/5697bfa71a28abf838c990a1/html5/thumbnails/112.jpg)
Impact of calibration on risk estimationoutput from logistic regression Assay response data were calibrated to convert OD
to concentration according to each of the previously described methods
Logistic regression relating miscarriage risk to log10 levels of GCSF were performed
112