critiquing for evidence-based practice: diagnostic and screening tests m8120 columbia university...
TRANSCRIPT
Critiquing for Evidence-based Practice: Diagnostic and Screening Tests
M8120Columbia University
Fall 2001
Suzanne Bakken, RN, DNSc
Overview
Probability fundamentalsClinical role of probability revision Characterizing new information (test
performance)Probability revision methods
– Bayes’ formula– Contingency table
In class exercises
Probability Fundamentals
Strength of beliefA number between 0 and1 that expresses an
opinion about the likelihood of an eventProbability of an event that is certain to occur
is 1Probability of an event that is certain to NOT
occur is 0
Summation Principle
Probability that an event will occur plus probability that it will not occur equals 1
Probability of all possible outcomes of a chance event is always equal to 1– Blood type: What is p[AB] given p[O]=0.46,
p[A]=.40, and p[B]=.10?– Fraternal triplets: What is the probability of at
least one boy and one girl?
Diagramming Probabilities
Type O (0.46)
Type A (0.40)
Type B (0.10)
Type AB (0.04)Chance node
Sum of probabilities at chance node = 1
Conditional Probability
Probability that event A occurs given that event B is known to occur
p[AlB]p[A,B]=p[AlB] X p[B]Examples in health care
Components of Probability Estimates
Personal experiencePublished experience - evidenceAttributes of the patient
Role of Probability Revision Techniques
Abnormal Finding
Diagnosis
BeforeFinding
AfterFinding
0 1Probability of Disease
Prior Probability Posterior Probability
Role of Probability Revision Techniques
Negative Finding
Diagnosis
AfterFinding
BeforeFinding
0 1Probability of Disease
Posterior Probability Prior Probability
Findings
SignsSymptomsDiagnostic testsProbabilistic relationships between
findings and disease basis of diagnostic decision support systems– Dxplain– QMR– Iliad
Definitions
Prior probability - the probability of an event before new information (finding) is acquired; pretest probability or risk
Posterior probability - the probability of an event after new information (finding) is acquired; posttest probability or risk
Probability revision - taking new information into account by converting prior probability to posterior probability
Review of Conditional Probability
p[AlB]
p[AlB]p[A and B]
p[B]=
Probability that an event is true, given that another event is true
What is the probability that someone has HIV antibody given a positive HIV test?What is the probability that someone has venothrombosis given a swollen calf?What is the probability that someone has dyspnea given the nurse says dyspnea is present?
Characterizing “Test” Performance
Compare test against gold standard (e.g., presence of disease; established test)
Ideal test - no values at which the distribution of those with the disease and without the disease overlap
Few tests ideal so …– TP– TN– FP– FN
Test Performance
True positive rate (TPR) p[+lD] = probability of an abnormal test result given that the disease is present; the number of persons WITH the disease who have an abnormal test result divided by the number of persons WITH the disease; sensitivity
Contingency Table View
Disease present = TPR + FNR = 1Disease absent = FPR + TNR = 1
TestResults
DiseasePresent
DiseaseAbsent
Total
Positive TP FP TP + FP
Negative FN TN FN + TN
TP + FN FP + TN
Test Performance
False positive rate (FPR) p[+lno D] = probability of an abnormal test result given that the disease is absent: the number of persons WITHOUT the disease who have an abnormal test result divided by the number of persons WITHOUT the disease
Contingency Table View
Disease present = TPR + FNR = 1Disease absent = FPR + TNR = 1
TestResults
DiseasePresent
DiseaseAbsent
Total
Positive TP FP TP + FP
Negative FN TN FN + TN
TP + FN FP + TN
Test Performance
True negative rate (TNR) p[-lno D] = probability of a normal test result given that the disease is absent; number of persons WITHOUT the disease who have a normal test result divided by number of persons WITHOUT the disease; 1 - FPR or specificity; 100% specificity = pathognomonic
Contingency Table View
Disease present = TPR + FNR = 1Disease absent = FPR + TNR = 1
TestResults
DiseasePresent
DiseaseAbsent
Total
Positive TP FP TP + FP
Negative FN TN FN + TN
TP + FN FP + TN
Test Performance
False negative rate (FNR) p[-l D] = probability of a normal test result given that the disease is present; number of persons WITH the disease who have a normal test result divided by number of persons WITH the disease; 1 - TPR
Contingency Table View
Disease present = TPR + FNR = 1Disease absent = FPR + TNR = 1
TestResults
DiseasePresent
DiseaseAbsent
Total
Positive TP FP TP + FP
Negative FN TN FN + TN
TP + FN FP + TN
Sensitivity vs. Specificity
Weighing sensitivity Vs. specificity in setting cutoff level for abnormality in a test
Consequences of FPR vs. FNR– Severity of disease– Availability of treatment– Risk of treatment
Sensitivity and specificity are characteristics of a test and a criterion for abnormality
Receiver Operating Characteristic (ROC) Curves
0 0.5 1.0
1.0
0.5
FPR (1 - specificity)
TPR
( s
en
siti
vit
y) Increased p[D]
Decreased p[D]
Contingency Table View
Disease present = TPR + FNR = 1Disease absent = FPR + TNR = 1
TestResults
DiseasePresent
DiseaseAbsent
Total
Positive TP FP TP + FP
Negative FN TN FN + TN
TP + FN FP + TN
Example of HIV
HIV TestResults
AntibodyPresent
AntibodyAbsent
Total
Positive 98 3 101
Negative 2 297 299
100 300 400
What is the TPR?What is the FPR?What is specificity?
Example of HIV
TPTPR =
TP + FN
TPR = sensitivity p[+lD]
Example of HIV
FPFPR =
FP + TN
FPR = p[+lno D]
Example of HIV
TNTNR =
TN + FP
TNR = specificity p[-lno D]
Nurse and Patient Rating of Symptoms
By definition, patient is gold standard for symptom rating
RN as “test” for presence or absence of symptom
Fatigue– RN yes/Pt yes = 50– RN yes/Pt no = 15– RN no/Pt yes = 20– RN no/Pt no = 15
What is sensitivity? What is specificity?
Nurse and Patient Ratings of Symptoms: Sensitivity
TPTPR =
TP + FN
Nurse and Patient Ratings of Symptoms: Specificity
TNTNR =
TN + FP
Moving from Test Characteristics to Predictive Value and Posterior Probabilities
Predictive valueForms of Bayes’
– Bayes’ formula– Contingency table view– Likelihood ratio
Prevalence
Frequency of disease in the population of interest at a given point in time
Predictive Value
Sensitivity, specificity, and their complements (FNR & FPR) focus on probability of findings given presence or absence of disease so not in a clinically useful form
Predictive value focuses on probability of disease given findings
Predictive value takes prevalence of disease in study population into account
Positive Predictive Value
number of persons with disease with abnormal findingPV+ = number of persons with abnormal finding
TPPV+ = TP + FP
The fraction of persons with an abnormal finding who have the disease
Negative Predictive Value
number of persons with normal finding WITHOUT diseasePV- = number of persons with normal finding
TNPV+ = TN + FN
The fraction of persons with an normal finding who DO NOT have the disease
Nurse and Patient Rating of Symptoms
By definition, patient is gold standard for symptom rating
RN as “test” for presence or absence of symptom
Shortness of Breath– RN yes/Pt yes = 25– RN yes/Pt no = 10– RN no/Pt yes = 15– RN no/Pt no = 50
What is PV+? What is PV-?
Nurse and Patient Ratings of Symptoms: PV+
TPPV+ =
TP + FP
Nurse and Patient Ratings of Symptoms: PV-
TNPV- =
TN + FN
Role of Probability Revision Techniques
Abnormal Finding
Diagnosis
BeforeFinding
AfterFinding
0 1Probability of Disease
Prior Probability Posterior Probability
Calculating Posterior Probability with Bayes’
What is the probability that someone has HIV antibody given a positive HIV test?
Can calculate with Bayes’ if you know:– Prior probability of the disease– Probability of an abnormal test (+) result
conditional upon the presence of the disease (TPR)
– Probability of an abnormal test (+) result conditional upon the absence of the disease (FPR)
Bayes’ Theorem
p[Dl+]p[D] x p[+lD]
{p[D] x p[+lD]} + {p[no D] x p[+lno D]}
=
Not a clinically useful form!
Deriving Bayes’ Theorem
Summation principle:
p[Dl+]
p[+]
p[+,D]
=
Given definition of conditional probability:
p[+] = p[+,D] + p[+,no D]
p[Dl+]
p[+,D] + p[+, no D]
p[+,D]
=
Thus:
1
2
3
p[+lD]
p[D]
p[+,D]
=
p[+lno D]
p[no D]
p[+,no D]
=
4
and
Given principle of conditional independence, rearrange expressions above:
p[+,D] = p[D] x p[+lD] p[+,no D] = p[no D] x p[+lnoD]5
p[Dl+]
p[D] x p[+lD] +p[no D] x p[+lnoD]
p[D] x p[+lD]
=
Substitute into 1:
6
Given definition of conditional probability
Calculating Posterior Probability with Bayes’
What is the probability that someone has HIV antibody given a positive HIV test?
Can calculate with Bayes’ if you know:– Prior probability of the disease– Probability of an abnormal (+) test result
conditional upon the presence of the disease (TPR=.98)
– Probability of an abnormal (+) test result conditional upon the absence of the disease (FPR=.01)
Bayes’ Theorem
p[Dl+]p[D] x p[+lD]
{p[D] x p[+lD]} + {p[no D] x p[+lno D]}
=
TPR
FPR1 - p[D]
When Abnormal Test Result is Present
p[Dl+]p[D] x TPR
{p[D] x TPR} + {1 - p[D] x FPR}
=
A somewhat more clinically useful form
When Normal Test Result is Present
p[Dl-]p[D] x 1 - TPR
{p[D] x 1 - TPR} + {1 - p[D] x 1 - FPR}
=
A somewhat more clinically useful form
FNR
TNR
Calculating Posterior Probability with Bayes’
What is the probability that someone has HIV antibody given a positive HIV test?
Can calculate with Bayes’ if you know:– Prior probability of the disease (can be prevalence
or other information)– Probability of an abnormal test result conditional
upon the presence of the disease (TPR=.98)– Probability of an abnormal test result conditional
upon the absence of the disease (FPR=.01)
The Role of Prior Probability
p[Dl+]
=
Prevalence of HIV Antibody in Homosexual Men in SF in mid1980s = .5
p[Dl+] =
The Role of Prior Probability
p[Dl+]p[.5] x .98
.49 + (.51 x .01) = .4951
=
Prevalence of HIV Antibody in Homosexual Men in SF in mid1980s = .5
p[Dl+] = .99
The Role of Prior Probability
p[Dl+]
=
Prevalence of HIV Antibody in Female Blood Donors = .0001
p[Dl+] =
The Role of Prior Probability
p[Dl+]p[.0001] x .98
.00098 + (.9999 x .01) = .010979
=
Prevalence of HIV Antibody in Female Blood Donors = .0001
p[Dl+] = . 089
Likelihood Ratios
Likelihood ratio =
FPR
An even more clinically useful form!
TPR
Nomogram for interpreting diagnostic test result
Diagramming Probabilities
Type O (0.46)
Type A (0.40)
Type B (0.10)
Type AB (0.04)Chance node
Sum of probabilities at chance node = 1
Path Probability
Operate
Do not operate
Disease present
Disease absent
Disease present
Disease absent
Survive
Operative death
Palliate
Operative deathOperative death
Survive
Survive
No cure
Cure
Cure
No Cure
No cure
Cure
p=.10
p=.90
p=.10
p=.90
p=.90
p=.10
p=.02
p=.98 p=.10
p=.90
p=.10
p=.90p=.90
p=.10
p=.01
p=.99
Try for the cure
Path probability of a sequence of chance events is the product of all probabilities along that sequence
Conditional Independence
Two findings are conditionally independent if TPR and FPR of one clinical finding do not depend upon the presence of the other finding
Assumption of conditional independence invoked when the same TPR (or FPR) is used in Bayes’ regardless of the prior probability of disease
Relevant in series of testsMay be invalid in some clinical situations
Interpreting Sequence of Tests
Posttest probability of first test used as pretest probability of second test
TPR and FPR of second test used in Bayes’ to calculate posttest probability following second test
Test Performance Biases Most significance source of error in measuring test performance is
due to differences between population in which test performance is measured and the population in which the test will be used
Spectrum bias - differences between populations in the spectrum of disease presentation and severity– Test population contains more sick persons than clinically relevant population– Test-referral bias - the composition of the population used to evaluate a diagnostic test
is altered when the test is a criterion for referring a patient for the definitive diagnostic procedure
TPR is usually higher in the study population than in the clinically relevant population due to few negatives (FN & TN) referred
FPR is usually higher in the study population than in the clinically relevant population due to few TN (remember FP + TN = 1)
Critically Analysis of Report of Diagnostic or Screening Test
Are the results of the study valid?
What are the results?Will the results help me in caring
for my patients?
Critically Analysis of Report of Diagnostic or Screening Test
Are the results of the study valid?– Was there an independent, blind comparison with
a reference (gold) standard?– Did the patient sample include an appropriate
spectrum of patients to whom the diagnostic test will be applied in clinical practice?
– Did the results of the test being evaluated influence the decision to perform the reference standard?
– Were the methods for performing the test described in sufficient detail to permit replication?
Critically Analysis of Report of Diagnostic or Screening Test
Are the results of the study valid?
What are the results?Will the results help me in caring
for my patients?
Critically Analysis of Report of Diagnostic or Screening Test
What are the results?– Are likelihood ratios for the test
result presented or data necessary for their calculation included?
Critically Analysis of Report of Diagnostic or Screening Test
Are the results of the study valid?
What are the results?Will the results help me in caring
for my patients?
Critically Analysis of Report of Diagnostic or Screening Test
Will the results help me in caring for my patients?– Will the reproducibility of the test
result and its interpretation be satisfactory in my setting?
– Are the results applicable to my patient?