royal college of surgeons in ireland coláiste ríoga na máinleá in Éirinn
DESCRIPTION
Royal College of Surgeons in Ireland Coláiste Ríoga na Máinleá in Éirinn. Indices of Performances of CPRs Nicola Motterlini, Borislav D Dimitrov, Tom Fahey. Contents. CPR derivation Analytical approach CPR validation Indices of discrimination Indices of calibration - PowerPoint PPT PresentationTRANSCRIPT
Division of Population Health Sciences
Royal College of Surgeons in IrelandColáiste Ríoga na Máinleá in Éirinn
Indices of Performances of CPRs
Nicola Motterlini, Borislav D Dimitrov, Tom Fahey
1
Division of Population Health Sciences
Contents
• CPR derivation– Analytical approach
• CPR validation– Indices of discrimination– Indices of calibration
• Performance of a CPR in several validation studies (systematic review with meta-analysis)
2
Division of Population Health Sciences
Figure 2. Development of a Clinical Decision Rule
Level of Evidence
4 3 2 1
Step 2. Validation
Evidence of reproducible accuracy
Narrow Validation Broad Validation
Application of a rule in a Application of rule in Similar clinical setting and multiple clinical settings Population as Step 1 and varying prevalence and outcomes of disease
Step 3. Impact Analysis
Evidence that rule change physician behaviour and improves patient outcomes and/or reduces costs
Step 1. Derivation
Identification of factors with predictive power
Stages of development of a CPR
3
Division of Population Health Sciences
Multivariable approach(e.g., logistic regression model)
Analytical approachto CPR(an example)
Target disorder(presence or absence)
4
Prediction LOW RISK %Total score ≤ 3
MODERATE RISK %4 ≤ Total score ≤ 7
HIGH RISK %Total score ≥ 8
Decision NO TREATMENT FURTHER DIAGNOSTIC TESTING
TREATMENT
“Weights”(e.g. odds
ratio)
1.9 (1.4-2.5)5.4 (4.1-7.1)16.2 (12.2-
21.6)
5.7 (4.3-7.5)
1.8 (1.5-2.1)
2.3 (1.8-3.1)
18.2 (14.2-23.6)
Points
123
2
1
2
3
Total score:3+2+1= 6
Independent risk factors
Age (years) 60-74 75-84 ≥85
Sex male
Diabetes mellitus
Heart failure
Current smoker
Division of Population Health Sciences
Thresholds approach to diagnosis
Diagnosis / test thresholdDiagnosis / test threshold
Test / reassurance thresholdTest / reassurance threshold
Probability of disease
100%HIGHRISK
0%LOWRISK
Treatment
Further diagnostic testing
No treatment
5
10987654321
Score
Division of Population Health Sciences
Figure 2. Development of a Clinical Decision Rule
Level of Evidence
4 3 2 1
Step 2. Validation
Evidence of reproducible accuracy
Narrow Validation Broad Validation
Application of a rule in a Application of rule in Similar clinical setting and multiple clinical settings Population as Step 1 and varying prevalence and outcomes of disease
Step 3. Impact Analysis
Evidence that rule change physician behaviour and improves patient outcomes and/or reduces costs
Step 1. Derivation
Identification of factors with predictive power
Stages of development of a CPR
6
Division of Population Health Sciences
Performance of a CPR can be assessed like a performance of a diagnostic test!
TP, number of true positives, FP, number of false positives, FN, number of false negatives, TN, number of true negatives, NT+, the number of patients with positive test results, NT-, the number of patients with negative test results, ND+, the number of patients with the disease, and ND−, number of patients without the disease
Disease+
Disease-
Total
Test +score ≥ Cut-off
TP FP NT+
Test -score < Cut-off
FN TN NT-
Total ND+ ND- N
7
Division of Population Health Sciences
Indices of performance
1. For discrimination (classification)– Ability to discriminate between diseased
individuals and non-diseased individuals (e.g., sensitivity, specificity, ROC curve, odds ratio, positive and negative predictive values, likelihood ratio etc.)
2. For calibration (comparison)– Agreement between predicted probabilities and
observed probabilities (e.g., risk ratio, Hosmer-Lemeshow goodness-of-fit test etc.)
8
Division of Population Health Sciences
Sensitivity = TP/ND+
Specificity = TN/ND-
Disease+
Disease-
Total
Test +score ≥ Cut-off
TP FP NT+
Test -score < Cut-off
FN TN NT-
Total ND+ ND- N
1. Discrimination – sensitivity and specificity
9
Division of Population Health Sciences
1 2 3 4 50
5
10
15
20
25
30
Disease
risk (%)
Predicted
Observed
Risk ratio ->
Low Moderate High
Score risk
2. Calibration – risk ratio
Predicted
Observed
<1 “under-prediction”=1 “accurate prediction” >1 “over-prediction”
10
Division of Population Health Sciences
Performance of a CPR in several validation studies(systematic review with meta-analysis)
11
Division of Population Health Sciences
Pooled analysis
• When the 95% CI includes 1 there is no significant statistical difference between the observed and predicted by CPR
• The smaller the interval, the more accuracy are the results
12
Division of Population Health Sciences
Index of heterogeneity
I2 value over 50% is a strong indicator for heterogeneity
13
Division of Population Health Sciences
Summary
• CPR derivation– Multivariable approach to develop a CPR
• CPR validation– Indices of discrimination (sensitivity,
specificity)– Indices of calibration (risk ratio)
• Systematic review with meta-analysis of validation studies of CPRs
14