Grading evidence and Grading evidence and recommendationsrecommendations

Workshop W-069Workshop W-069

Congress Hall ABEFCongress Hall ABEF

Oct 6 2004Oct 6 2004

Professional good intentions and Professional good intentions and plausibleplausible theories aretheories are insufficientinsufficient for selecting policies and for selecting policies and practices for protecting, practices for protecting, promoting and restoring healthpromoting and restoring health..

Iain Chalmers

How can we judge the How can we judge the extent of our confidence extent of our confidence

that adherence to that adherence to aa recommendation will do recommendation will do more good than harm?more good than harm?

GRADEGRADE

Grades of Recommendation Assessment, Development

and Evaluation

GRADE Working GroupGRADE Working GroupDavid Atkins, chief medical officerDavid Atkins, chief medical officeraa Dana Best, assistant professorDana Best, assistant professorbb Peter A Briss, chiefPeter A Briss, chiefcc Martin Eccles, professorMartin Eccles, professordd Yngve Falck-Ytter, associate directorYngve Falck-Ytter, associate directoree Signe Flottorp, researcherSigne Flottorp, researcherff Gordon H Guyatt, professorGordon H Guyatt, professorgg Robin T Harbour, Robin T Harbour, quality and information directorquality and information director h h Margaret C Haugh, methodologistMargaret C Haugh, methodologistii David Henry, professorDavid Henry, professorjj Suzanne Hill, senior lecturerSuzanne Hill, senior lecturerjj Roman Jaeschke, clinical professorRoman Jaeschke, clinical professorkk Gillian Leng, guidelines programme directorGillian Leng, guidelines programme directorll Alessandro Liberati, professorAlessandro Liberati, professormm Nicola Magrini, directorNicola Magrini, directornn

James Mason, professorJames Mason, professordd Philippa Middleton, honorary research fellowPhilippa Middleton, honorary research fellowoo Jacek Mrukowicz, executive directorJacek Mrukowicz, executive directorpp Dianne O’Connell, senior epidemiologistDianne O’Connell, senior epidemiologistqq Andrew D Oxman, directorAndrew D Oxman, directorff Bob Phillips, associate fellowBob Phillips, associate fellowrr Holger J Schünemann, associate professorgHolger J Schünemann, associate professorgg,sg,s Tessa Tan-Torres Edejer, medical officer/scientistTessa Tan-Torres Edejer, medical officer/scientisttt Helena Varonen, associate editorHelena Varonen, associate editoruu Gunn E Vist, researcherGunn E Vist, researcherff John W Williams Jr, associate professorJohn W Williams Jr, associate professorvv Stephanie Zaza, Stephanie Zaza, project directorproject directorww

a)a) Agency for Healthcare Research and Quality, Agency for Healthcare Research and Quality, USA USA b)b) Children's National Medical Center, Children's National Medical Center, USA USAc) Centers for Disease Control and Prevention, c) Centers for Disease Control and Prevention, USAUSAd) University of Newcastle upon Tyne, d) University of Newcastle upon Tyne, UKUKe) German Cochrane Centre, e) German Cochrane Centre, GermanyGermanyf) Norwegian Centre for Health Services, f) Norwegian Centre for Health Services, NorwayNorwayg) McMaster University, g) McMaster University, CanadaCanadah) Scottish Intercollegiate Guidelines Network, h) Scottish Intercollegiate Guidelines Network, UKUKi) Fédération Nationale des Centres de Lutte Contre le Cancer, i) Fédération Nationale des Centres de Lutte Contre le Cancer, FranceFrancej) University of Newcastle, j) University of Newcastle, AustraliaAustraliak) McMaster University, k) McMaster University, CanadaCanadal) National Institute for Clinical Excellence, l) National Institute for Clinical Excellence, UKUKm) m) Università di Modena e Reggio Emilia, Università di Modena e Reggio Emilia, ItalyItalyn)n) Centro per la Valutazione della Efficacia della Assistenza Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Sanitaria, ItalyItalyo) Australasian Cochrane Centre, o) Australasian Cochrane Centre, Australia Australia p) Polish Institute for Evidence Based Medicine, p) Polish Institute for Evidence Based Medicine, PolandPolandq) The Cancer Council, q) The Cancer Council, AustraliaAustraliar) r) Centre for Evidence-based Medicine, Centre for Evidence-based Medicine, UKUKs)s) University of Buffalo, University of Buffalo, USAUSAt) World Health Organisation, t) World Health Organisation, Switzerland Switzerland u) Finnish Medical Society Duodecim, u) Finnish Medical Society Duodecim, Finland Finland v) Duke University Medical Center, v) Duke University Medical Center, USA USA w) w) Centers for Disease Control and Prevention, Centers for Disease Control and Prevention, USAUSA

Opinions do not necessarily represent those of Opinions do not necessarily represent those of the institutions with which the members of the the institutions with which the members of the GRADE Working Group are affiliatedGRADE Working Group are affiliated..

What do you know about What do you know about GRADE?GRADE?

Have prepared a guidelineHave prepared a guideline Read the BMJ paper Read the BMJ paper

Have prepared a systematic review and a Have prepared a systematic review and a summary of findings tablesummary of findings table

Have attended a GRADE meeting, workshop or talkHave attended a GRADE meeting, workshop or talk

Why bother about grading?Why bother about grading?

People draw conclusions about thePeople draw conclusions about the– quality of evidencequality of evidence– strength of recommendationsstrength of recommendations

Systematic and explicit approaches can helpSystematic and explicit approaches can help– protect against errorsprotect against errors– resolve disagreementsresolve disagreements– facilitate critical appraisalfacilitate critical appraisal– communicate informationcommunicate information

However, there is wide variation in currently However, there is wide variation in currently used approachesused approaches

Who is confused? Who is confused?

Evidence Evidence RecommendationRecommendation

II-2II-2 BB C+ C+ 11 StrongStrong Strongly Strongly

recommendedrecommended

OrganizatioOrganizationn

USPSTFUSPSTF ACCPACCP GCPSGCPS

Still not confused?Still not confused?

EvidenceEvidenceRecommendationRecommendation

BB Class IClass I C+ C+ 11 IVIV CC

OrganizatioOrganizationn

AHAAHA ACCPACCP SIGNSIGN

Recommendation for use of oral Recommendation for use of oral anticoagulation in patients with atrial anticoagulation in patients with atrial fibrillation and rheumatic mitral valve fibrillation and rheumatic mitral valve diseasedisease

Guidelines development processGuidelines development process

Quality of evidenceQuality of evidence

The extent to which one can be confident that an estimate The extent to which one can be confident that an estimate of effect or association is correct. of effect or association is correct.

It depends on the:It depends on the:– study designstudy design (e.g. RCT, cohort study) (e.g. RCT, cohort study)– study quality/limitationsstudy quality/limitations (protection against bias; (protection against bias;

e.g. concealment of allocation, blinding, follow-up)e.g. concealment of allocation, blinding, follow-up)– consistency of resultsconsistency of results– directness of the evidencedirectness of the evidence including the including the

populationspopulations (those of interest versus similar; for (those of interest versus similar; for example, older, sicker or more co-morbidity)example, older, sicker or more co-morbidity)

interventionsinterventions (those of interest versus similar; for (those of interest versus similar; for example, drugs within the same class)example, drugs within the same class)

outcomesoutcomes (important versus surrogate outcomes) (important versus surrogate outcomes) comparisoncomparison (A - C versus A - B & C - B) (A - C versus A - B & C - B)

Quality of evidenceQuality of evidence

The quality of the evidence (i.e. our confidence) may also The quality of the evidence (i.e. our confidence) may also be REDUCEDbe REDUCED when there is: when there is:

Sparse or imprecise dataSparse or imprecise data Reporting biasReporting bias

The quality of the evidence (i.e. our confidence) may be The quality of the evidence (i.e. our confidence) may be INCREASEDINCREASED when there is: when there is:

A strong association A strong association A dose response relationshipA dose response relationship All plausible confounders would have reduced the All plausible confounders would have reduced the

observed effect observed effect All plausible biases would have increased the observed All plausible biases would have increased the observed

lack of effectlack of effect

Quality assessment criteriaQuality assessment criteria

Quality of evidence

Study design Lower if Higher if

High Randomised trial

Moderate

Low Observational study

Very low Any other evidence

Study quality: -1 Serious limitations -2 Very serious limitations -1 I mportant inconsistency Directness: -1 Some uncertainty -2 Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias

Strong association: +1 Strong, no plausible confounders +2 Very strong, no major threats to validity +1 Evidence of a Dose response gradient +1 All plausible confounders would have reduced the eff ect

Categories of qualityCategories of quality

HighHigh: Further research is very unlikely to : Further research is very unlikely to change our confidence in the estimate of change our confidence in the estimate of effect. effect.

ModerateModerate: Further research is likely to have : Further research is likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and may change the estimate of effect and may change the estimate.estimate.

LowLow: Further research is very likely to have : Further research is very likely to have an important impact on our confidence in the an important impact on our confidence in the estimate of effect and is likely to change the estimate of effect and is likely to change the estimate.estimate.

Very lowVery low: Any estimate of effect is very : Any estimate of effect is very uncertain.uncertain.

Judgements about the overall Judgements about the overall quality of evidencequality of evidence

most systems just use evidence about primary most systems just use evidence about primary benefit/outcomebenefit/outcome

but what about other outcomes (downsides)?but what about other outcomes (downsides)? options:options:

– ignore all but primary outcomeignore all but primary outcome– basing it on the evidence for benefitsbasing it on the evidence for benefits– some blended approachsome blended approach– having separate grades for benefits and having separate grades for benefits and

harmsharms– weakest of any outcomeweakest of any outcome

Based on lowest of all the Based on lowest of all the criticalcritical outcomes outcomes Beyond the scope of a systematic reviewBeyond the scope of a systematic review

Strength of recommendationStrength of recommendation

The extent to which one can be confident that The extent to which one can be confident that adherence to a recommendation will do more adherence to a recommendation will do more good than harm. good than harm.

trade-offstrade-offs (the relative value attached to (the relative value attached to the expected benefits, harms and costs)the expected benefits, harms and costs)

quality of the evidencequality of the evidence translation of the evidencetranslation of the evidence into practice into practice

in a specific settingin a specific setting uncertainty about baseline riskuncertainty about baseline risk

Judgements about the balance Judgements about the balance between benefits and harmsbetween benefits and harms

Before considering cost and making a Before considering cost and making a recommendationrecommendation

For a specified setting, taking into For a specified setting, taking into account issues of translation into account issues of translation into practicepractice

Clarity of the trade-offs Clarity of the trade-offs between benefits and the between benefits and the

harms harms the estimated size of the effect for the estimated size of the effect for

each main outcomeeach main outcome the precision of these estimatesthe precision of these estimates the relative value attached to the the relative value attached to the

expected benefits and harmsexpected benefits and harms important factors that could be important factors that could be

expected to modify the size of the expected to modify the size of the expected effects in specific settings; expected effects in specific settings; e.g. proximity to a hospitale.g. proximity to a hospital

Balance between benefits and Balance between benefits and harmharm

Net benefitsNet benefits:: The intervention does The intervention does more good than harm.more good than harm.

Trade-offsTrade-offs:: There are important There are important trade-offs between the benefits and trade-offs between the benefits and harms.harms.

Uncertain net benefitsUncertain net benefits:: It is not It is not clear whether the intervention does clear whether the intervention does more good than harm.more good than harm.

Not net benefitsNot net benefits:: The intervention The intervention does not do more good than harm.does not do more good than harm.

Judgements about Judgements about recommendationsrecommendations

This should include considerations of This should include considerations of costs; i.e. “Is the net gain (benefits-costs; i.e. “Is the net gain (benefits-harms) worth the costs?”harms) worth the costs?” Do itDo it Probably do it Probably do it No recommendationNo recommendation Probably don’t do itProbably don’t do it Don’t do itDon’t do it

Should healthy asymptomatic postmenopausal women Should healthy asymptomatic postmenopausal women have been given oestrogen + progestin for prevention in have been given oestrogen + progestin for prevention in

1992?1992? Quality of evidence across studies forQuality of evidence across studies for

– CHDCHD– Hip fractureHip fracture– Colorectal cancerColorectal cancer– Breast cancerBreast cancer– StrokeStroke– ThrombosisThrombosis– Gall bladder diseaseGall bladder disease

Quality of evidence across critical outcomesQuality of evidence across critical outcomes Balance between benefits and harmsBalance between benefits and harms RecommendationsRecommendations

Will GRADE lead to change

Evidence profile: Quality assessmentEvidence profile: Quality assessment Oestrogen + progestin for prevention before Oestrogen + progestin for prevention before

WHI and HERSWHI and HERS

Oestrogen + progestin versus usual care

Oestrogen + progestin for Oestrogen + progestin for prevention after WHI and HERSprevention after WHI and HERS

GRADE for diagnostic testsGRADE for diagnostic testsQuality of evidence Study design Lower if * High Cross-sectional (or cohort)

studies of patients with diagnostic uncertainty with direct comparison

Moderate Low Anything else Very low

Study limitations (including representativeness of population, choice of gold standard, incomplete performance of tests, independence of test interpretation) -1 Serious limitations -2 Very serious limitations -1 Important inconsistency Directness -1-Some uncertainty -2-Major uncertainty -1 Sparse or imprecise data -1 High probability of reporting bias

Challenges for GRADEChallenges for GRADE

Operationalise all stepsOperationalise all steps Dissemination/buy inDissemination/buy in

– simple to dosimple to do– easy to understand and useeasy to understand and use

Tool and manualTool and manual

GRADE profiler (GRADEpro)GRADE profiler (GRADEpro)

Separation by outcomesSeparation by outcomes

Work in groups of twoWork in groups of two

take a pencil (and paper)take a pencil (and paper) write down the most important write down the most important

issues/questions you have about issues/questions you have about GRADEGRADE

The 10 burning questions/issues The 10 burning questions/issues about GRADEabout GRADE

1.1. Different expertsDifferent experts2.2. Prospective studiesProspective studies3.3. Valuing benefits and harms – decide about Valuing benefits and harms – decide about

tradeoffstradeoffs4.4. Low quality evidence leading to strong rec’s Low quality evidence leading to strong rec’s 5.5. How can one introduce/disseminate one How can one introduce/disseminate one

single/uniform system single/uniform system 6.6. Empirical evidence for GRADE – how should we Empirical evidence for GRADE – how should we

obtain it obtain it 7.7. Mechanisms for balancing benefits and cost Mechanisms for balancing benefits and cost 8.8. Reliability? Reliability? 9.9. Other than RCT evidenceOther than RCT evidence10.10. Decisions about quality of evidence/limitations Decisions about quality of evidence/limitations

of study design, guidance about magnitude of of study design, guidance about magnitude of effect effect

The 10 burning questions/issues The 10 burning questions/issues about GRADEabout GRADE

1.1. Other than RCT evidenceOther than RCT evidence

2.2. What type of cost and resourcesWhat type of cost and resources

3.3. Who judges the importance of Who judges the importance of outcomesoutcomes

4.4. How can one evaluate whether all How can one evaluate whether all outcomes are reported?outcomes are reported?

5.5. Decisions about quality of Decisions about quality of evidence/limitations of study designevidence/limitations of study design

Small group sessionsSmall group sessions

find a groupfind a group select spokes personselect spokes person take 30 minutes to complete the tasktake 30 minutes to complete the task be prepared to criticisebe prepared to criticise

SummarySummaryWhat is good about GRADE?What is good about GRADE?

– – –

What is most challenging?What is most challenging?– Takes too longTakes too long– relative importance is difficult to work outrelative importance is difficult to work out– Difficult - much more time neededDifficult - much more time needed–

What do we need to do next?What do we need to do next?– more time more time – –

The 10 burning questions/issues The 10 burning questions/issues about GRADEabout GRADE

1.1.

2.2.

3.3.

4.4.

5.5.

6.6.

7.7.

8.8.

9.9.

10.10.

SummarySummaryWhat is good about GRADE?What is good about GRADE?

– – –

What is most challenging?What is most challenging?– – –

What do we need to do next?What do we need to do next?– – –