bias in clinical research: measurement bias measurement bias in descriptive studies measurement bias...

Bias in Clinical Research: Measurement Bias

• Measurement bias in descriptive studies

• Measurement bias in analytic studies

– Misclassification of dichotomous exposure & outcome variables

• non-differential misclassification

• differential misclassification

• magnitude and direction of bias

– Misclassification of interval scale variables

– Advanced topics (mention only)

• misclassification of multi-level categorical variables

• misclassification of confounding variables

• back-calculating to the truth

Measurement Bias

• Definition– bias that is caused when any measurement

collected about or from subjects is not completely valid (accurate)

• any type of variable: exposure, outcome, or confounder

– aka: misclassification bias; information bias (text); identification bias

• misclassification is the immediate result of an error in measurement

Misclassification of Dichotomous Variables: Terms Related to Measurement Validity

• Sensitivity

– the ability of a measurement to identify correctly those who

HAVE the characteristic (disease or exposure) of interest.

• Specificity

– the ability of a measurement to identify correctly those who

do NOT have the characteristic of interest

• Applies to any dichotomous variable, not just diagnoses

Gold Standard Present Absent

Your Present a b Measurement Absent c d

Sensitivity = a/(a+c) Specificity = d/(b+d)

Positive predictive value = a/(a+b)

Negative predictive value = d/(c+d)

Causes for Misclassification• Questionnaire problems

– inaccurate recall– socially desirable responses– ambiguous questions– under or overzealous interviewers

• Biological specimen collection– problems in specimen collection or processing or storage

• Biological specimen testing– inherent limits of detection– faulty instruments

• Data management problems in coding• Design or analytic problems

– incorrect time period assessed– lumping of variables (composite variables)

SOURCE POPULATION = CALIFORNIA

STUDY SAMPLE = PRE-ELECTION POLL(Field Poll)

Descriptive Study: Measurement Bias

Deukmejian

Bradley +7%

1982 California Governor Election

SOURCE POPULATION = CALIFORNIA

STUDY SAMPLE = PRE-ELECTION POLL(Field Poll)

Descriptive Study: Measurement Bias

“Bradley Effect” = Respondents who favored Deukmejian sought to avoid appearing racist and hence did not

state their true choice during polling

Deukmejian

Deukmejian 49% Bradley +7%

Bradley 48%

1982 California Governor Election

SOURCE POPULATION

STUDY SAMPLE

Contrast with Selection Bias

Uneven dispersion of arrows

e.g., Dewey backers were

over-represented

Diseased

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

Non-Differential Misclassification of Exposure: Imperfect Sensitivity

Problems with sensitivity in the measurement of exposure - independent of disease status

e.g., case-control study

exposure = alcohol abuse

Evenly shaded arrows =

non-differential

Non-differential Misclassification of Exposure

Truth: No misclassification (100% sensitivity/specificity)

Exposure Cases ControlsYes 50 20No 50 80

OR= (50/50)/(20/80) = 4.0

Presence of 70% sensitivity in exposure classification

Exposure Cases ControlsYes 50-15=35 20-6=14No 50+15=65 80+6=86

OR= (35/65)/(14/86) = 3.3

Effect of non-differential misclassification of dichotomous exposures: Bias the OR toward the null value of 1.0

Diseased

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

Non-Differential Misclassification of Exposure: Imperfect Specificity

Problems with specificity of exposure measurement - independent of disease status

e.g., exposure = self-reported second-hand smoke exposure

Non-differential Misclassification of Exposure

Truth: No misclassification (100% sensitivity/specificity)

Exposure Cases ControlsYes 50 20No 50 80

OR= (50/50)/(20/80) = 4.0

Presence of 70% specificity in exposure classification

Exposure Cases ControlsYes 50+15=65 20+24=44No 50-15=35 80-24=56

OR= (65/35)/(44/56) = 2.4

Effect of non-differential misclassification of dichotomous exposures: Bias the OR toward the null value of 1.0

Diseased

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

No misclassification


50

50

20

80 OR = 4.0

Diseased

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

Non-Differential Misclassification of Exposure: Imperfect Specificity


OR = 2.4

65

50 35

44

80 56

differences become blurred

Diseased

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

Non-Differential Misclassification of Exposure: Imperfect Specificity and Sensitivity

Problems with sensitivity - independent of disease status

Problems with specificity - independent of disease status

Non-Differential Misclassification of Exposure: Imperfect Sensitivity and Specificity

Exposure Cases ControlsYes 80 50No 20 50 True OR = (80/20) / (50/50) = 4.0

True Cases Controls Distribution exp unexp exp unexp (gold standard) 80 20 50 50

Study distribution: Cases ControlsExposed 56 6 62 35 15 50Unexposed 24 14 38 15 35 50

sensitivity 0.70 0.70 0.70 0.70 or specificity

Exposure Cases ControlsYes 62 50No 38 50 Observed OR = (62/38) / (50/50) = 1.6

SOURCE POPULATION

STUDYSAMPLE

Sensitivity = 0.7

Specificity = 0.7

Non-Differential Misclassification of Exposure: Imperfect Sensitivity and Specificity






SOURCE POPULATION

STUDYSAMPLE

Sensitivity = 0.9

Specificity = 0.8

Non-Differential Misclassification of Exposure: Imperfect Sensitivity & Specificity and Uncommon Exposure






SOURCE POPULATION

STUDYSAMPLE

e.g. radon exposure

Sensitivity = 0.9

Specificity = 0.8

Non-differential Misclassification of Exposure: Magnitude of Bias on the Odds Ratio

True OR=4.0

2.20.0770.900.90

2.80.200.900.90

3.00.3680.900.90

1.90.200.600.90

3.20.200.950.90

1.90.200.850.60

2.60.200.850.90

Observed ORPrev of Exp in controls

SpecificitySensitivity

Bias as a function of non-differential imperfect sensitivity and specificity of exposure measurement

0.9

0.7

0.5

Sensitivity of exposure measurement

Specificity of exposure measurement

Copeland et al. AJE 1977

True OR = 2.67

Prevalence of exposure in controls = 0.2

Ap

par

ent

Od

ds

Rat

io

2.8

2.5

2.2

1.9

1.6

1.3

1.0

.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.00

Bias as a function of non-differential imperfect sensitivity and specificity of exposure measurement

0.9

0.7

0.5

Sensitivity of exposure measurement

Specificity of exposure measurement


True OR = 2.67

Prevalence of exposure in controls = 0.2

Ap

par

ent

Od

ds

Rat

io

2.8

2.5

2.2

1.9

1.6

1.3

1.0

.50 .55 .60 .65 .70 .75 .80 .85 .90 .95 1.00

When does OR fall

below 2?

Non-Differential Misclassification of Exposure in a Cohort Study: Effect of Sensitivity, Specificity and Prevalence of Exposure

All RR < 8

If Pe >.25, ↑ Sn. influ.

Dependence upon Pe

Flegal et al. AJE 1986

True Risk Ratio = 10

App

aren

t Ris

k R

atio

U = sensitivity; V = specificity

Non-Differential Misclassification of Exposure: Rules of Thumb Regarding Sensitivity & Specificity


SOURCE POPULATION

Sens + Spec = 1 gives OR = 1 (no effect)

Sensitivity Specificity Observed OR

0.8 1.0 2.6

0.8 0.8 1.9

0.4 0.6 1.0

0.4 0.4 0.82

0 0 0.33

Sens + Spec >1 but <2 gives attenuated effect

Sens + Spec < 1 gives reversal of effect

Coding error

Diseased

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

Non-Differential Misclassification of Outcome

Problems with outcome sensitivity -independent of exposure status

Problems with outcome specificity - independent of

exposure status

Evenly shaded arrows =

non-differential

Bias as a function of non-differential imperfect sensitivity and specificity of outcome measurement in a cohort study

Sensitivity of outcome measurement0.9

0.7

0.5

Specificity of outcome measurementCopeland et al. AJE 1977

True risk ratio = 2.0

Cumulative incidence in unexposed = 0.05

Steep bias with change in

specificity

Relatively less influence

from sensitivity

App

aren

t Ris

k R

atio

Non-Differential Misclassification of Outcome: Effect of Incidence of Outcome


Specificity of outcome measurement

0.2 0.1

0.1 0.05

0.05 0.025

Cumulative incidence of outcome

Exposed Unexposed


Sensitivity of outcome measurement held fixed = 0.9

App

aren

t Ris

k R

atio

Special Situation In a Cohort or Cross-sectional Study

Misclassification of outcome• If specificity of outcome measurement is 100%• Any degree of imperfect sensitivity, if non-differential, will not

bias the risk ratio or prevalence ratio• e.g.,

• Risk difference, however, is changed by a factor of (1 minus sensitivity), in this example, 30% (truth=0.1; biased = 0.07)

DiseaseNoDisease

Exposed 20 80 100Unexposed 10 90 100

2.0

1001010020

ratio )prevalence (or Risk

DiseaseNoDisease

Exposed 20-6=14 80+6=86100Unexposed 10-3=7 90+3=93100

2.0

1007

10014

ratio )prevalence (or Risk

Truth

70% sensitivity

When specificity of outcome is 100% in a cohort or cross-sectional study

Sensitivity of outcome measurement0.9

0.7

0.5

Specificity of outcome measurementCopeland et al. AJE 1977


Cumulative incidence in unexposed = 0.05

App

aren

t Ris

k R

atio

When specificity of outcome measurement is 100% in a cohort or cross sectional study

• Worth knowing about when choosing outcomes, such as cutoffs for continuous variables on ROC curves

• Choosing most specific cutoff (or 100% cutoff) will lead to least biased ratio measures of association

0.0 0.1 0.2 0.3 0.4 0.5

0.5

0.6

0.7

0.8

0.9

1.0

1.0 0.9 0.8 0.7 0.6 0.5

Sen

sit

ivit

y

1 - Specificity

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

OD: 0.06 Specificity: 84 % Sensitivity: 100 %



0.0 0.1 0.2 0.3 0.4 0.5

0.5

0.6

0.7

0.8

0.9

1.0

1.0 0.9 0.8 0.7 0.6 0.5

Sen

sit

ivit

y

1 - Specificity

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8




OD: 0.06 Sensitivity = 100%OD: 0.06 Specificity = 84%



1.0 0.9 0.8 0.7 0.6 0.5

Specificity

Efficacy of a pertussis vaccine

• Outcome: Cough > 5 days– No. of events: 2672 (and apparently lots of “power”)– Result: No significant difference between groups

• Outcome: Cough + microbiologic pertussis confirmation– No. of events: 10– Result: rate ratio = 0.08 (92% vaccine efficacy) (95% CI = 0.01 to 0.68)

• Acellular vaccine vs. control (hepatitis A vaccine) for the prevention of pertussis in adults (Ward et al. NEJM 2005)

Group No. of subjects Person-years Pertussis vaccine 1391 2421 Control 1390 2444

Pervasiveness of Non-Differential Misclassification

• Direction of this bias is towards the null

• Therefore, called a “conservative” bias

• Goal, however, is to get the truth

• Consider how much underestimation of effects must be occurring in research

• How many “negative” studies are truly “positive”?

Differential Misclassification of ExposureWeinstock et al. AJE 1991• Nested case-control study in Nurses Health Study cohort

• Cases: women with new melanoma diagnoses

• Controls: women w/out melanoma - by incidence density sampling

• Measurement of exposure: questionnaire about self-reported

“tanning ability”; administered shortly after melanoma development

MelanomaNoMelanoma

No tan to light tan 15 77Med to dark tan 19 157

1.6

157771915

OR

• Question asked after diagnosis

• Question asked before diagnosis (NHS baseline)

MelanomaNoMelanoma


0.7

15579259

OR

MelanomaNoMelanoma


1.6

157771915

OR

Virtually unchanged

Substantially changed

Melanoma

Tanningability

+ -

No

Yes

SOURCE POPULATION

STUDY SAMPLE

“Tanning Ability” and Melanoma:

Differential Misclassification of Exposure

Imperfect specificity of exposure measurement - mostly in cases

Bias away from the null

leading to spurious

association

Congenital Malformation

Exposed

+ -

+

-

SOURCE POPULATION

STUDY SAMPLE

Differential Misclassification of Exposure:

Exposures During Pregnancy and Congenital Malformations

Cases more likely than controls to remember a variety of exposures

Cases might be more likely than controls to falsely state a

variety of exposures

Uneven shading of arrows =

differential

Differential Misclassification of Exposure: Magnitude of Bias on the Odds Ratio

True OR=3.9

Exposure Classification

Sensitivity Specificity

Cases Controls Cases Controls OR

0.90 0.60 1.0 1.0 5.79

0.60 0.90 1.0 1.0 2.22

1.0 1.0 0.9 0.70 1.00

1.0 1.0 0.7 0.90 4.43

Prevalence of Exposure in Controls = 0.1

Misclassification of Dichotomous Exposure or Outcome: Summary of Effects

Misclassification Effect on Ratio Measure of Association

Non-differential Exposure Towards null Outcome Towards null*

Differential

Exposure Away or towards null Outcome Away or towards null

*Exception: When specificity is 100%, no effect on risk ratio or prevalence ratio regardless of sensitivity

Relating Last Week to This Week:Relating Reproducibility/Validity of Individual

Measurements to Measurement Bias in Inferences in Analytic Studies

• Validity– How sensitivity and specificity of a measurement

results in measurement bias covered in prior slides

• How about reproducibility? – Recall that a measurement with imperfect

reproducibility will lack perfect validity --unless it is repeated many many times

Reproducibility and Validity of a Measurement

With only one shot at the measurement, most of the time you will be off the center of the target

GoodB-Ball

PoorB-Ball

>6 ft 10 30 40 +1 10 +3 30<6 ft 10 50 60 10 +1 50 +5

20 80 100 20 80

P

GoodB-Ball

PoorB-Ball

>6 ft 10 32 42<6 ft 10 48 58

20 80 100

Truth = Prevalence Ratio= (10/40) / (10/60) = 1.5

Observed = Prevalence Ratio = (10/42) / (10/58) = 1.38

10% Misclassification

Imperfect reproducibility leads

to 90% sensitivity and 90% specificity of

height measurement –non-differential with respect to outcome

Relating the Reproducibility and Validity of Measurements to Measurement Bias in Analytic Studies – Interval Scale Variables

Validity (Systematic error)• Result moves systematically up or down scale by given factor or absolute

difference• e.g., systematic error in an interval scale outcome variableMean Ratio of Means Difference in Means

Truth Exposed 100

Unexposed 50 2 50

Measurement off by factor of 10 Exposed 1000

Unexposed 500 2 500

Measurement off by difference of 10 Exposed 110

Unexposed 60 1.83 50

Bias depending

upon measure of association

Relating the Reproducibility and Validity of Measurements to Measurement Bias in Analytic Studies – Interval Scale Variables

Reproducibility (Random error)

e.g., random error in a predictor variableAssuming:

• Exposure is normally distributed with variance, 2True

• Random error is normally distributed with variance, 2E

• Then, the observed regression coefficient is equal to the true regression coefficient times:

• i.e., the greater the measurement error, the greater the attenuation (bias) towards the

null (e.g., if ICC is 0.5, the measure of association is halved)

22

2

ETrue

True

(i.e. reproducibility, the intraclass correlation coefficient)

Truth and Error

Truth

Advanced Topics• Misclassification of multi-level categorical variables

– some of the rules change regarding direction of bias

• Misclassification of confounding variables– net result is failure to fully control (adjust) for that variable (left with

“residual confounding”)– measures of association may be over or under-estimated

• Back-calculating to unbiased results (Quantitative bias analysis)– thus far, truth about relationships have been assumed– in practice, we just have observed results– when extent of classification errors (e.g., PPV, NPV, sensitivity &

specificity) are known, it is possible to back-calculate to truth– if exact classification errors are not known, it is possible to perform

sensitivity analyses to estimate a range of study results given a range of possible classification errors

Poor Reproducibility

Poor Validity

Good Reproducibility

Good Validity

Managing Measurement Bias

• Prevention and avoidance are critical– study design phase is critical; little to be done after study over

• Become an expert in the measurement of your primary variables

• For the other variables, seek out the advice of other experts

• Optimize the reproducibility/validity of your measurements!

bias in clinical research: measurement bias measurement bias in descriptive studies measurement bias...

Documents

measurement slide

measurement of exposure

nondifferential slide

truth slide

california study sample

casecontrol study exposure

dc d slide

over represented slide