Grading evidence and Grading evidence and recommendationsrecommendations
Holger Schünemann
Gunn Vist
Gordon Guyatt
for the GRADE Working Group
Introduction to GRADE Example of applying GRADE Demonstration of the GRADE profiler
sofware
Today’s talk
When to make a recommendation?When to make a recommendation? never
patient values differ guidelines should just lay out
benefits and risks
when evidence strong enough if very weak, too uncertain
clinicians need guidance• intense study demands decision
What type of recommendations?What type of recommendations?
strong recommendations– high quality methods – large precise effect – few down sides of therapy
weak recommendations– low quality methods– imprecise estimate– small overall effect– substantial down sides– people react differently to same
outcomes/circumstances
Should we grade Should we grade recommendations?recommendations?
People draw conclusions about the– quality of evidence– strength of recommendations
Systematic and explicit approaches can help– protect against errors– resolve disagreements– facilitate critical appraisal– communicate information
Why grade Why grade recommendations?recommendations?
Change practitioner behavior
Strong: apply uniformly – just do it
Weak: think about it– examine evidence yourself– consider patient circumstances very carefully– explore with the patient
However, there is wide variation in currently used approaches
Which grading system? Which grading system?
Evidence Recommendation
II-2 B C+ 1 Strong Strongly
recommended
Organization
USPSTF ACCP GCPS
Still not confused?Still not confused?
EvidenceRecommendation
B Class I C+ 1 IV C
Organization
AHA ACCP SIGN
Recommendation for use of oral anticoagulation in patients with atrial fibrillation and rheumatic mitral valve disease
What do you know about What do you know about GRADE?GRADE?
Have prepared a guidelineHave prepared a guideline Read the BMJ paper Read the BMJ paper
Have prepared a systematic review and a Have prepared a systematic review and a summary of findings tablesummary of findings table
Have attended a GRADE meeting, workshop or talkHave attended a GRADE meeting, workshop or talk
About GRADEAbout GRADE
o Began as informal working group in 2000Began as informal working group in 2000o Researchers/guideline developers with Researchers/guideline developers with
interest in methodologyinterest in methodologyo Aim: to develop a Aim: to develop a commoncommon system for system for
grading the quality of evidence and the grading the quality of evidence and the strength of recommendations that is sensible strength of recommendations that is sensible and to explore the range of interventions and and to explore the range of interventions and contexts for which it might be useful*contexts for which it might be useful*
o 13 meetings (~10 – 35 attendants)13 meetings (~10 – 35 attendants)o Evaluation of existing systems and reliability*Evaluation of existing systems and reliability*o Workshops at various places including Workshops at various places including
Cochrane Colloquia, WHO and GIN since 2000Cochrane Colloquia, WHO and GIN since 2000
*Grade Working Group. CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005
GRADE Working GroupGRADE Working GroupDavid Atkins, chief medical officerDavid Atkins, chief medical officeraa Dana Best, assistant professorDana Best, assistant professorbb Peter A Briss, chiefPeter A Briss, chiefcc Martin Eccles, professorMartin Eccles, professordd Yngve Falck-Ytter, associate directorYngve Falck-Ytter, associate directoree Signe Flottorp, researcherSigne Flottorp, researcherff Gordon H Guyatt, professorGordon H Guyatt, professorgg Robin T Harbour, Robin T Harbour, quality and information quality and information
directordirector h h Margaret C Haugh, methodologistMargaret C Haugh, methodologistii David Henry, professorDavid Henry, professorjj Suzanne Hill, senior lecturerSuzanne Hill, senior lecturer jj Roman Jaeschke, clinical professorRoman Jaeschke, clinical professorkk Gillian Leng, guidelines programme directorGillian Leng, guidelines programme director ll Alessandro Liberati, professorAlessandro Liberati, professormm Nicola Magrini, directorNicola Magrini, directornn
James Mason, professorJames Mason, professordd Philippa Middleton, honorary research fellowPhilippa Middleton, honorary research fellowoo Jacek Mrukowicz, executive directorJacek Mrukowicz, executive directorpp Dianne O’Connell, senior epidemiologistDianne O’Connell, senior epidemiologistqq Andrew D Oxman, directorAndrew D Oxman, directorff Bob Phillips, associate fellowBob Phillips, associate fellowrr Holger J Schünemann, associate professorHolger J Schünemann, associate professorg,sg,s Tessa Tan-Torres Edejer, medical Tessa Tan-Torres Edejer, medical
officer/scientistofficer/scientisttt Helena Varonen, associate editorHelena Varonen, associate editoruu Gunn E Vist, researcherGunn E Vist, researcherff John W Williams Jr, associate professorJohn W Williams Jr, associate professorvv Stephanie Zaza, Stephanie Zaza, project directorproject directorww
a)a) Agency for Healthcare Research and Quality, Agency for Healthcare Research and Quality, USAUSA b)b) Children's National Medical Center, Children's National Medical Center, USA USAc) Centers for Disease Control and Prevention, c) Centers for Disease Control and Prevention, USAUSAd) University of Newcastle upon Tyne, d) University of Newcastle upon Tyne, UKUKe) German Cochrane Centre, e) German Cochrane Centre, GermanyGermanyf) Norwegian Centre for Health Services, f) Norwegian Centre for Health Services, NorwayNorwayg) McMaster University, g) McMaster University, CanadaCanadah) Scottish Intercollegiate Guidelines Network, h) Scottish Intercollegiate Guidelines Network, UKUKi) Fédération Nationale des Centres de Lutte i) Fédération Nationale des Centres de Lutte Contre le Cancer, Contre le Cancer, FranceFrancej) University of Newcastle, j) University of Newcastle, AustraliaAustraliak) McMaster University, k) McMaster University, CanadaCanadal) National Institute for Clinical Excellence, l) National Institute for Clinical Excellence, UKUKm) m) Università di Modena e Reggio Emilia, Università di Modena e Reggio Emilia, ItalyItalyn)n) Centro per la Valutazione della Efficacia della Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Assistenza Sanitaria, ItalyItalyo) Australasian Cochrane Centre, o) Australasian Cochrane Centre, AustraliaAustralia p) Polish Institute for Evidence Based Medicine, p) Polish Institute for Evidence Based Medicine, PolandPolandq) The Cancer Council, q) The Cancer Council, AustraliaAustraliar) r) Centre for Evidence-based Medicine, Centre for Evidence-based Medicine, UKUKs)s) National Cancer Institute, National Cancer Institute, ItalyItalyt) World Health Organisation, t) World Health Organisation, SwitzerlandSwitzerland u) Finnish Medical Society Duodecim, u) Finnish Medical Society Duodecim, FinlandFinland v) Duke University Medical Center, v) Duke University Medical Center, USA USA w) w) Centers for Disease Control and Prevention, Centers for Disease Control and Prevention, USAUSA
How can we judge the How can we judge the extent of our confidence extent of our confidence
that adherence to that adherence to aa recommendation will do recommendation will do more good than harm?more good than harm?
Grading SystemGrading System
Strength of the recommendation• do it (or don’t do it)/recommend• probably do it (or probably don’t do it)/suggest
Quality of underlying evidence• high quality (well done RCT)• moderate (quasi-RCT)• low (well done observational)• very low (anything else)
Moving quality downMoving quality down• poor (RCT) design, implementation
→randomization, blinding, concealment, follow-up, intention to treat principle, early stopping for benefit
• inconsistency• Indirect evidence
→patients, interventions, outcomes→A vs B, but have A to C, B to C
• sparse or imprecise data• reporting bias
Reporting biasReporting bias
• high likelihood of reporting bias can lower quality
• reporting of outcomes
• reporting of studies• publication bias
Moving quality upMoving quality upObservational studies – high or moderate Observational studies – high or moderate
quality?quality?
Strong association→strong association: RR > 2 or RR < 0.5→very strong association: RR > 5 or RR < 0.2
Dose response relationship– bleeding risk associated with increasing INR (blood
thinning with warfarin)
Plausible confounders would have reduced the effect
For example, plausible explanatoryfactors that were not adjusted for in
studies comparing mortalityrates of for-profit and not-for-profit
hospitals would havereduced the observed effect. Thus, the evidence showing that for-profit
hospitals have a higher risk of mortality is more convincing
Quality assessment criteriaQuality assessment criteria
Quality of evidence Study design Lower if Higher if
High Randomised trial
Moderate
Low Observational
study
Very low Any other evidence
Study quality:
-1 Serious
limitations
-2 Very serious
limitations
-1 Important
inconsistency
Directness:
-1 Some
uncertainty
-2 Major
uncertainty
-1 Sparse or
imprecise data
-1 High probability
of reporting bias
Strong association:
+1 Strong, no
plausible
confounders
+2 Very strong,
no major
threats to
validity
+1 Evidence of a
Dose response
gradient
+1 All plausible
confounders
would have
reduced the
effect
Categories of qualityCategories of quality
High: Further research is very unlikely to change our confidence in the estimate of effect.
Moderate: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.
Very low: Any estimate of effect is very uncertain.
“The extent to which one can be confident that an estimate of effect or association is correct.”
Strength of recommendationStrength of recommendation
“The extent to which one can be confident that adherence to a recommendation will do more good than harm.”
quality of the evidence translation of the evidence into practice
in a specific setting uncertainty about baseline risk trade-offs (the relative value attached to
the expected benefits, harms and costs)
You are hiking.You are hiking.
Which of the following animals Which of the following animals would you prefer to would you prefer to
encounter?encounter?
You are buying an ice cream.You are buying an ice cream.
Which flavor do you prefer?Which flavor do you prefer?
Values and preferencesValues and preferences
underlying values and preferences always present
sometimes crucial
important to make explicit
Judgements about the balance Judgements about the balance between benefits and harmsbetween benefits and harms
Before considering cost and making a recommendation
Judgment: Benefits vs Risks/CostsJudgment: Benefits vs Risks/Costs
quality of evidence seriousness of outcome magnitude of effect precision of treatment effect risk of target event risk of adverse events cost of therapy values
Judgements about Judgements about recommendationsrecommendations
1. Benefit and downside evaluation
? ?
2. Recommendation (wording) STRONG WEAK WEAK STRONG Recommend suggest suggest recommend don’t do it probably don’t do it probably do it do it should not do it might not do it might do it should do it
Benefits<< Downsides
Benefits ?≤ Downsides
Benefits ?≥ Downsides
Benefits >> Downsides
Strong recommendation when Strong recommendation when evidence is weak?evidence is weak?
Balance of benefits and downsides clearly on one side
Not frequent if quality is low or very low
Comparison of GRADE and other Comparison of GRADE and other systemssystems
• Explicit definitions• Explicit, sequential judgements• Components of quality• Overall quality• Relative importance of outcomes• Balance between health benefits and harms• Balance between incremental health benefits
and costs• Consideration of equity• Evidence profiles• International collaboration• Software• Consistent judgements?• Communication?
Who is interested in GRADEWho is interested in GRADE
• American College of Chest Physicians (ACCP)
• WHO• American Endocrine Society• UpToDate• Clinical Evidence• American Society of Clinical Oncology
(ASCO)• American Thoracic Society (ATS)• Urologists worldwide (EBUro)• NICE
ConclusionConclusion Challenges in grading
– judgment always required
Must consider study design, execution, consistency, directness, reporting bias– magnitude, precision
Balance of benefits and risks/cost– magnitude of effects; precision of effects; values and preferences
Separation of recommendation from quality of the evidence
GRADE working group active in obtaining feedback and dissemination
The clinical questionThe clinical question
Population: In patients with chronic atrial fibrillation and no prior history
of stroke
Intervention: does oral anticoagulation (comparison) compared with no therapy
Outcome: reduce the risk for embolic stroke, hemorrhage
and death?
The evidenceThe evidence Systematic Review* 5 RCTs 2,313 Patients randomised Warfarin in all studies Studien 1.5 years mean follow-up Outcomes: Ischemic Stroke, hemorrhage
(major, including intracranial), death (vascular and all cause) and dependency
*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005, Issue 3.
All disabling or fatal stroke (isch. and hemorrh.)
Major hemorrhage (non IC) All cause mortality Minor bleeding (hematoma, prolonged
bleeding of minor wounds)
*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,
Issue 3.
Outcomes/endpointsOutcomes/endpoints
How important is the endpoint for decision making?
Judgment about the relative importance for each endpoint on a scale from 9 (most important) to 1 (least important): • 7 – 9: the endpoint is critical for decision making.• 4 – 6: the endpoint is important but not critical.• 1 – 3: the endpoint is not important.
Outcomes/endpointsOutcomes/endpoints
All disabling or fatal stroke (isch. and hemorrh.)
Major hemorrhage (non IC) All cause mortality Minor bleeding (hematoma, prolonged
bleeding of minor wounds)
*Systematic Review: Aguilar & Hart. Cochrane Database of Systematic Reviews 2005,
Issue 3.
Outcomes/endpointsOutcomes/endpoints
9
7
9
5
Quality assessment criteriaQuality assessment criteria
Quality of evidence Study design Lower if Higher if
High Randomised trial
Moderate
Low Observational
study
Very low Any other evidence
Study quality:
-1 Serious
limitations
-2 Very serious
limitations
-1 Important
inconsistency
Directness:
-1 Some
uncertainty
-2 Major
uncertainty
-1 Sparse or
imprecise data
-1 High probability
of reporting bias
Strong association:
+1 Strong, no
plausible
confounders
+2 Very strong,
no major
threats to
validity
+1 Evidence of a
Dose response
gradient
+1 All plausible
confounders
would have
reduced the
effect
Disabling or fatal Disabling or fatal strokestroke
Study design: 5 RCTs
Quality of evidence for this endpoint: High
Disabling or fatal Disabling or fatal strokestroke
Detailed design and execution Concealment Follow-up In two studies (CAFA; SPINAF) both
patients and outcome assessors were blinded; in the other studies only outcomes assessors.
Quality of evidence for this endpoint now:
High (or -1 Moderate)
Disabling or fatal Disabling or fatal strokestroke
Consistency: No inconsistency
Quality of evidence for this endpoint now:
High
Directness of evidenceDirectness of evidence
indirect treatment comparisons– interested in A versus B– have A versus C and B versus C
alendronate vs risedronate (biphosponates)– both versus placebo, no head-to-head
Directness - patientsDirectness - patients
patients meet trials’ eligibility criteria
not included, but no reason to question– slight age difference, comorbidity, race
some question, bottom line applicable– valvular atrial fibrillation
serious question about biology– heart failure trials applicability to aortic stenosis
Directness - interventionsDirectness - interventions
same drugs and doses– captopril 100 mg. tid in heart failure
similar drugs and doses– captopril in lower doses
same class and biology– other ACEI in heart failure
questionable class and biology– ARB in heart failure
Directness - outcomesDirectness - outcomes
same outcomes– alendronate over 3 years on fracture
similar but questionable– alendronate over long-term
serious question– surrogate outcomes
– bone density; arrhythmia suppression; cholesterol levels (clofibrate)
Disabling or fatal Disabling or fatal strokestroke
Directness of the evidence: Population, Intervention, Outcomes Direct
Quality of evidence for this endpoint now:
High
Disabling or fatal Disabling or fatal strokestroke
Imprecise or sparse data: Would few additional events or
larger studies likely alter the results?
Disabling or fatal Disabling or fatal strokestroke
Imprecise or sparse data: No imprecise or sparse data
Quality of evidence for this endpoint now:
High
Disabling or fatal Disabling or fatal strokestroke
Reporting bias: Not present
Quality of evidence for this endpoint now:
High
Disabling or fatal Disabling or fatal strokestroke
Strong association? present (RR = 0.46)
Quality of evidence for this endpoint now:
High [or +1 High (from moderate)]
strong, no plausible confounder, consistent
and direkt evidence
Endpoint: Major extracranial Endpoint: Major extracranial hemorhagehemorhage
• Study design: 4 RCTs → Quality: High• Study details and execution:
No serious limitations• No inconsistency and direct• Imprecise or sparse data?
Imprecise or sparse dataImprecise or sparse data
There is not an empirical basis for defining imprecise or sparse data. Two possible definitions are:
Data are sparse if the results include just a few events or observations and they are uninformative
Data are imprecise if the confidence intervals are sufficiently wide that an estimate is consistent with either important harms or important benefits.
These different definitions can result in different judgments.
Major extracranial hemorhageMajor extracranial hemorhage
• Study design: 4 RCTs → Quality: High• Study details and execution:
no serious limitations• No inconsistency and direct• Imprecise or sparse data?
• Imprecise data (wide confidence intervals benefit and harm uncertain)
GRADE Evidence Profile Author(s): Schunemann Date: 8/28/2005 Question: Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation? Patient or population: Patients with non-valvular atrial fibrillation Settings: Long term outpatient management Systematic review: Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: CD001927.pub2. DOI: 10.1002/14651858.CD001927.pub2.
Summary of findings Quality assessment
No of patients Effect
No of studies
Design Limitations Consistency Directness Other
considerations warfarin
placebo or no treatment
Relative (95% CI)
Absolute (95% CI)
Quality Importance
Disabling or Fatal Stroke (ischemic and hemorrhagic) (Neuroimaging or autopsy6 Follow up: Mean: 1.5 years)
5 Randomised trials
No limitation8,12
No important inconsistency
No uncertainty
Strong association (+1)13
18/1154 (1.6%)
39/1159 (3.4%)
RR 0.46 (0.27 to 0.81)
20/1 000 (30 fewer/1000 to 10 fewer/1000)
High
914
Extracranial hemorrhage (Transfusion or invasive procedure requirement1 Follow up: 1.5 years)
4 Randomised trials
No limitations 8
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)11
17/1154 (1.5%)
16/1159 (1.4%)
RR 1.06 (0.54 to 2.09)
0/1 000 (10 fewer/1000 to 10 more/1000)
Moderate
7
All cause mortality3,5 (Direct patient follow-up Follow up: 1.5 years)
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 69/1225 (5.6%)
99/1236 (8%)
RR 0.70 (0.52 to 0.95)
20/1 000 (40 fewer/1000 to 1 fewer/1000)
High
9
Footnotes:
1. required transfusion of two or more units of red blood cells, hospitalization, or invasive procedures to control bleeding and those that resulted in death or permanent functional impairment (e.g. blindness) were included.
2. These consisted of death due to stroke, heart disease, hemorrhage and sudden deaths of unknown cause.
3. All cause mortality: death from any cause (vascular and non- vascular) within 30 days from onset of stroke symptoms. For this outcome, results of published data, which included about 6% of patients with prior stroke or TIA, were used.
4. The diagnosis of MI was usually based upon electrocardiographic changes, elevation of enzymes or post-mortem examination.
5. From Fig 10
6. Follow-up for this outcome was less than 100%.
7. Ischemic stroke was an identified outcome in all trials, with the ischemic nature conrmed by neuroimaging or autopsy in the majority of cases.
8. In two studies (CAFA; SPINAF) patients and outcome assessors were blind to OAC administration, while in the remaining trials treatment was given open label with outcomes verified by those unaware of treatment assignment.
9. Methodological quality was not downgraded because the lack of blinding in some studies did not have important impact on the results
10. Only 5 events in the OAC group and 2 events in the control group, confidence intervals wide
11. Only 17 events in the OAC and 16 events in the control group
12. Loss to follow-up not reported in AFASAK I and CAFA, ranged from 0 to 3% in the other studies
13. Strong association present: RR 0.46
14. Importance is rated on a scale from 1 to 9. 1 represents least important (not important for decision making) and 9 most important (for decision making).
15. Lack of blinding in two trials of lesser concern
16. Strong association present: RR 0.36
Judgements about the overall Judgements about the overall quality of evidencequality of evidence
Most systems not explicit
Options:– strongest outcome– primary outcome– benefits– weighted– separate grades for benefits and harms– no overall grade– weakest outcome
Based on lowest of all the critical outcomes Beyond the scope of a systematic review
GRADE Evidence Profile Author(s): Schunemann Date: 8/28/2005 Question: Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation? Patient or population: Patients with non-valvular atrial fibrillation Settings: Long term outpatient management Systematic review: Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: CD001927.pub2. DOI: 10.1002/14651858.CD001927.pub2.
Summary of findings Quality assessment
No of patients Effect
No of studies
Design Limitations Consistency Directness Other
considerations warfarin
placebo or no treatment
Relative (95% CI)
Absolute (95% CI)
Quality Importance
Disabling or Fatal Stroke (ischemic and hemorrhagic) (Neuroimaging or autopsy6 Follow up: Mean: 1.5 years)
5 Randomised trials
No limitation8,12
No important inconsistency
No uncertainty
Strong association (+1)13
18/1154 (1.6%)
39/1159 (3.4%)
RR 0.46 (0.27 to 0.81)
20/1 000 (30 fewer/1000 to 10 fewer/1000)
High
914
Extracranial hemorrhage (Transfusion or invasive procedure requirement1 Follow up: 1.5 years)
4 Randomised trials
No limitations 8
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)11
17/1154 (1.5%)
16/1159 (1.4%)
RR 1.06 (0.54 to 2.09)
0/1 000 (10 fewer/1000 to 10 more/1000)
Moderate
7
All cause mortality3,5 (Direct patient follow-up Follow up: 1.5 years)
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 69/1225 (5.6%)
99/1236 (8%)
RR 0.70 (0.52 to 0.95)
20/1 000 (40 fewer/1000 to 1 fewer/1000)
High
9
Quality across all endpoints
Risk groupsRisk groupsRisk for cardio-embolic stroke: High (prior TIA or stroke*, > 75 yrs,
LVEF/CHF, HTN or DM): 10%/year
Moderate risk (65 to 75 years) or one risk factor: 3 to 4%/year
Low risk (< 65 years): 0.5%/year
GRADE Evidence Profile Author(s): Schunemann Date: 8/28/2005 Question: Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation? Patient or population: Patients with non-valvular atrial fibrillation Settings: Long term outpatient management Systematic review: Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: CD001927.pub2. DOI: 10.1002/14651858.CD001927.pub2.
Summary of findings Quality assessment
No of patients Effect
No of studies
Design Limitations Consistency Directness Other
considerations warfarin
placebo or no treatment
Relative (95% CI)
Absolute (95% CI)
Low risk (0.5%/year) Mean/moderate risk (3.5%/year)
High risk (10%/year)
Quality Importance
Disabling or Fatal Stroke (ischemic and hemorrhagic) (Neuroimaging or autopsy6 Follow up: Mean: 1.5 years)
5 Randomised trials
No limitation8,12
No important inconsistency
No uncertainty
Strong association (+1)13
18/1154 (1.6%)
39/1159 (3.4%)
RR 0.46 (0.27 to 0.81)
Low: 2 fewer/1000 (1 fewer/1000 to 3 fewer/1000)
Mean (moderate): 20 fewer/1 000 (30 fewer/1000 to 10 fewer/1000)
High: 50 fewer (20 fewer to 75 fewer)
High
914
Extracranial hemorrhage (Transfusion or invasive procedure requirement1 Follow up: 1.5 years)
4 Randomised trials
No limitations 8
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)11
17/1154 (1.5%)
16/1159 (1.4%)
RR 1.06 (0.54 to 2.09)
0/1 000 (10 fewer/1000 to 10 more/1000)
Moderate
7
All cause mortality3,5 (Direct patient follow-up Follow up: 1.5 years)
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 69/1225 (5.6%)
99/1236 (8%)
RR 0.70 (0.52 to 0.95)
20 fewer/1 000 (40 fewer/1000 to 1 fewer/1000)
High
9
Risk groupsRisk groupsRisk for cardio-embolic stroke: High (prior TIA or stroke*, > 75 yrs,
LVEF/CHF, HTN or DM): 10%/year– Benefits greater downsides: do it, high
Moderate risk (65 to 75 years) or one risk factor: 3 to 4%/year – Benefits greater downsides: do it, high
Low risk (< 65 years): 0.5%/year– Benefits smaller than downsides: values:
probably do not do it, high
Value and preference Value and preference statementsstatements
underlying values and preferences always present
sometimes crucial
important to make explicit
Observational studies – Observational studies – high or moderate quality?high or moderate quality?
Strong association
Dose response relationship– bleeding risk associated with increasing INR
(blood thinning with warfarin)
Plausible confounders would have reduced the effect
For example, plausible explanatoryfactors that were not adjusted for in
studies comparing mortalityrates of for-profit and not-for-profit
hospitals would havereduced the observed effect. Thus,
the evidence that for-profithospitals have a higher risk of mortality is more convincing
Guideline development processGuideline development process
Prioritise Problems, establish panelPrioritise Problems, establish panel
Systematic ReviewSystematic Review
Evidence ProfileEvidence Profile
Relative importance of outcomesRelative importance of outcomes
Overall quality of evidenceOverall quality of evidence
Benefit – downside evaluationBenefit – downside evaluation
Strength of recommendationStrength of recommendation
Implementation and evaluation of guidelinesImplementation and evaluation of guidelines
GRADE
Guideline development processGuideline development process
Prioritise Problems, establish panelPrioritise Problems, establish panel
Systematic ReviewSystematic Review
Evidence ProfileEvidence Profile
Relative importance of outcomesRelative importance of outcomes
Overall quality of evidenceOverall quality of evidence
Benefit – downside evaluationBenefit – downside evaluation
Strength of recommendationStrength of recommendation
Implementation and evaluation of guidelinesImplementation and evaluation of guidelines
GRADE
Summary of Findings
GRADEpro©GRADEpro©
Visual studio.net Windows based (Mac version
coming) Simple installation Help file Will be integrated with Revman
(trial) Free availability Beta version
Development of GRADE Development of GRADE profilesprofiles
GRADE Evidence Profile Author(s): Schunemann Date: 8/28/2005 Question: Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation? Patient or population: Patients with non-valvular atrial fibrillation Settings: Long term outpatient management Systematic review: Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: CD001927.pub2. DOI: 10.1002/14651858.CD001927.pub2.
Summary of findings Quality assessment
No of patients Effect
No of studies
Design Limitations Consistency Directness Other
considerations warfarin
placebo or no treatment
Relative (95% CI)
Absolute (95% CI)
Quality Importance
Disabling or Fatal Stroke (ischemic and hemorrhagic) (Neuroimaging or autopsy6 Follow up: Mean: 1.5 years)
5 Randomised trials
Serious limitations (-1)8,12
No important inconsistency
No uncertainty
Strong association (+1)13
18/1154 (1.6%)
39/1159 (3.4%)
RR 0.46 (0.27 to 0.81)
20/1 000 (30 fewer/1000 to 10 fewer/1000)
High
914
Intracranial Hemorrhage (Clinical diagnosis confirmed by CT or post-mortem Follow up: Mean follow-up: 1.5 years)
5 Randomised trials
No limitations
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)10
5/1244 (0.4%)
2/1159 (0.2%)
RR 1.75 (0.47 to 6.41)
3/1 000 (0 more/1000 to 10 more/1000)
Moderate
8
Extracranial hemorrhage (Transfusion or invasive procedure requirement1 Follow up: )
4 Randomised trials
Serious limitations (-1)8
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)11
17/1244 (1.4%)
16/1159 (1.4%)
RR 0.93 (0.48 to 1.79)
0/1 000 (10 fewer/1000 to 10 more/1000)
Low
7
All cause mortality3,5 (Direct patient follow-up Follow up: )
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 69/1225 (5.6%)
99/1236 (8%)
RR 0.70 (0.52 to 0.95)
20/1 000 (40 fewer/1000 to 1 fewer/1000)
High
9
Vascular death2 (Death due to stroke, heart disease, hemorrhage and sudden death Follow up: )
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 43/1244 (3.5%)
51/1159 (4.4%)
RR 0.79 (0.53 to 1.17)
1/1 000 (3 fewer/1000 to 1 more/1000)
High
9
All ischemic stroke (Neuroimaging or autopsy6,7 Follow up: 1.5 years)
5 Randomised trials
No limitations8,9
No important inconsistency
No uncertainty
None 22/1154 (1.9%)
69/1159 (6%)
RR 0.32 (0.20 to 0.51)
40/1 000 (60 fewer/1000 to 20 fewer/1000)
High
7
8. In two studies (CAFA; SPINAF) patients and outcome assessors were blind to OAC administration, while in the
remaining trials treatment was given open label with
outcomes verified by those unaware of treatment
assignment.
GRADE Evidence Profile Author(s): Schunemann Date: 8/28/2005 Question: Should warfarin vs placebo or no treatment be used for patients with non-valvular atrial fibrillation? Patient or population: Patients with non-valvular atrial fibrillation Settings: Long term outpatient management Systematic review: Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. The Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: CD001927.pub2. DOI: 10.1002/14651858.CD001927.pub2.
Summary of findings Quality assessment
No of patients Effect
No of studies
Design Limitations Consistency Directness Other
considerations warfarin
placebo or no treatment
Relative (95% CI)
Absolute (95% CI)
Quality Importance
Disabling or Fatal Stroke (ischemic and hemorrhagic) (Neuroimaging or autopsy6 Follow up: Mean: 1.5 years)
5 Randomised trials
Serious limitations (-1)8,12
No important inconsistency
No uncertainty
Strong association (+1)13
18/1154 (1.6%)
39/1159 (3.4%)
RR 0.46 (0.27 to 0.81)
20/1 000 (30 fewer/1000 to 10 fewer/1000)
High
914
Intracranial Hemorrhage (Clinical diagnosis confirmed by CT or post-mortem Follow up: Mean follow-up: 1.5 years)
5 Randomised trials
No limitations
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)10
5/1154 (0.4%)
2/1159 (0.2%)
RR 1.87 (0.51 to 6.82)
3/1 000 (0 more/1000 to 10 more/1000)
Moderate
8
Extracranial hemorrhage (Transfusion or invasive procedure requirement1 Follow up: 1.5 years)
4 Randomised trials
Serious limitations (-1)8
No important inconsistency
No uncertainty
Imprecise or sparse data (-1)11
17/1154 (1.5%)
16/1159 (1.4%)
RR 1.06 (0.54 to 2.09)
0/1 000 (10 fewer/1000 to 10 more/1000)
Low
7
All cause mortality3,5 (Direct patient follow-up Follow up: 1.5 years)
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 69/1225 (5.6%)
99/1236 (8%)
RR 0.70 (0.52 to 0.95)
20/1 000 (40 fewer/1000 to 1 fewer/1000)
High
9
Vascular death2 (Death due to stroke, heart disease, hemorrhage and sudden death Follow up: 1.5 years)
5 Randomised trials
No limitations15
No important inconsistency
No uncertainty
None 43/1154 (3.7%)
51/1159 (4.4%)
RR 0.85 (0.57 to 1.26)
1/1 000 (3 fewer/1000 to 1 more/1000)
High
9
All ischemic stroke (Neuroimaging or autopsy6,7 Follow up: 1.5 years)
5 Randomised trials
No limitations8,9
No important inconsistency
No uncertainty
Strong association (+1)16
22/1154 (1.9%)
69/1159 (6%)
RR 0.32 (0.20 to 0.51)
40/1 000 (60 fewer/1000 to 20 fewer/1000)
High
7
Footnotes:
1. required transfusion of two or more units of red blood cells, hospitalization, or invasive procedures to control bleeding and those that resulted in death or permanent functional impairment (e.g. blindness) were included.
2. These consisted of death due to stroke, heart disease, hemorrhage and sudden deaths of unknown cause.
3. All cause mortality: death from any cause (vascular and non- vascular) within 30 days from onset of stroke symptoms. For this outcome, results of published data, which included about 6% of patients with prior stroke or TIA, were used.
4. The diagnosis of MI was usually based upon electrocardiographic changes, elevation of enzymes or post-mortem examination.
5. From Fig 10
6. Follow-up for this outcome was less than 100%.
7. Ischemic stroke was an identified outcome in all trials, with the ischemic nature conrmed by neuroimaging or autopsy in the majority of cases.
8. In two studies (CAFA; SPINAF) patients and outcome assessors were blind to OAC administration, while in the remaining trials treatment was given open label with outcomes verified by those unaware of treatment assignment.
9. Methodological quality was not downgraded because the lack of blinding in some studies did not have important impact on the results
10. Only 5 events in the OAC group and 2 events in the control group, confidence intervals wide
11. Only 17 events in the OAC and 16 events in the control group
12. Loss to follow-up not reported in AFASAK I and CAFA, ranged from 0 to 3% in the other studies
13. Strong association present: RR 0.46
14. Importance is rated on a scale from 1 to 9. 1 represents least important (not important for decision making) and 9 most important (for decision making).
15. Lack of blinding in two trials of lesser concern
16. Strong association present: RR 0.36
GRADEproGRADEpro
Reproducible Transparent
– Footnotes– Judgments
GRADE profiles– Summary
Integration with Revman Real time
Judgements about Judgements about recommendationsrecommendations
• “We recommend”…”should” …“Do it”• “We suggest”…”may” … “Probably do it” • “We suggest not”… “may not” …“Probably
don’t do it”• “We recommend not”…”should not”… “Don’t
do it”
No recommendation
This could include considerations of costs; i.e. “Is the net gain (benefits-downsides) worth the costs?”
What are we concerned about?What are we concerned about?
methodological quality of evidence– likelihood of bias– high, moderate, low and very low
quality
strength of recommendations– must do to might do– strong recommendation or weak
Strong recommendation when Strong recommendation when evidence is weak?evidence is weak?
recommendations against– uncertainty of benefit– confidence in down sides
whole body CT or MRI screening– maybe benefit, maybe not– true positives some harm– false positive some harm
Strong recommendation when Strong recommendation when evidence is weak?evidence is weak?
known benefit, strong recommendation for one of two alternatives– CABG for left main stem disease– but venous or mammary artery graft
benefit: weak evidence suggests appreciable benefit for mammary artery
harm: strong evidence that little difference
Population: In patients with chronic atrial fibrillation and no prior history
of stroke
Intervention: does oral anticoagulation (comparison) compared with no therapy
Outcome: reduce the risk for embolic stroke, hemorrhage
and death?
Different risk groups:Low, moderate, high
Other outcomes:Inconvenience, quality of
life