1

Grading evidence and formulating recommendations

The GRADE Working Group

Yngve Falck-YtterGerman Cochrane Centre

Freiburg, May 7, 2003

2

Background

Need to know: How much confidence in recommendation Many different grading schemes exists, e.g.

• Canadian Task Force on Periodic Health Examinations

• United States Preventive Services Task Force

• Oxford Center of Evidence Based Medicine

• Scottish Intercollegiate Guidelines Network

• American Heart Association The same grading scheme is often used differently No sufficient explanation how to approach the grading

Oxford Centre of Evidence Based Medicine; http://www.cebm.net

3

Example

Level of evidence

I

II

III

IV

V

Studies on therapy

RCTs

Cohort studies

Case-control-studies

Case-series

Expert opinion

A

Recom-mendation

B

C

D

4

Goal

Simple Sensible Explicit Reliable Address shortcoming of other schemes

(e.g explicitly include trade offs between benefit and risks)

5

Conceptual definitions

Quality of evidence: extent to which one can be confident that an estimate of effect is correct.

Grade (formulation) of a recommendation: extent to which one can be confident that adherence to the recommendation will do more good than harm

6

General Approach

5 step sequential judgment:1. Quality of evidence across studies for each important

outcome

2. Which outcomes are critical to a decision

3. The overall quality of evidence across these critical outcomes

4. The balance between benefits and harm

5. Recommendation

7

1. Quality of evidence of outcome

a) Study design• RCT vs observational design (not MA, SR, expert)

b) Study quality• Critical appraisal (outcome dependent)

c) Consistency• Similarity of effect across studies

d) Directness• People, intervention & outcome similar to those of interest

e) Other• Sparse data, high risk of reporting bias

8

Quality of evidence across outcomes

Observational studies Quality of evidence

Randomized trials

HighExtremely strong

association and no major threats to validity

No serious flaws in study quality

Strong, consistent association and no

plausible confounders

Serious flaws in design or execution or quasi-

randomized trials

Moderate

LowNo serious flaws in

study qualityVery serious flaws in design or execution

Very lowSerious flaws in design

and executionVery serious flaws and

at least one other serious threat to validity

9

...continued Additional factors that lower quality are:

• Important inconsistency of results• Uncertainty about directness

• Some uncertainty• Major uncertainty can lower the quality by two levels

• High probability of reporting bias• Sparse data

Additional factors that may increase quality are:• All plausible residual confounding, if present, would

already have reduced the observed effect• Evidence of a dose-response gradient

10

2. Outcomes critical to a decision

Outcomes may be important, but sometimes not critical to a decision

Only outcomes critical to a decision should provide a basis for recommendation

If information on harm is critical, it should be included even if uncertainty exists

11

3. Overall quality of evidence

The lowest quality of evidence for any critical outcome should provide the basis for grading

However, if evidence favors the same alternative and there is high quality for some but not all of those outcomes, overall quality should still be high

Weak evidence about implausible putative harms should not lower the overall grade of evidence

12

4. The balance between benefits and harm

Net benefits: Clearly does more good than harm Trade-offs: Important trade-offs between benefits

and harm Uncertain trade-offs: It is not clear whether the

intervention does more good than harm No net benefits: Clearly does not do more good than

harm

13

5. Grades of recommendation

„Do it“ or „Don‘t do it“• Indicating a judgment that a majority of well informed

people will make the same choice• Medical practice is expected to not to vary much

„Probably do it“ or „Probably don‘t do it“• Indicating a judgment that a majority of well informed

people will make the same choice, but a substantial minority will not

• Medical practice is expected to vary to some degree