Implementing the GRADE Method in Guideline Development: Real-

World Experiences

Contemplation Stage: To GRADE or Not to GRADE?

Sheila A. Agyeman, MHA

Director of Evidence Based Medicine

American Gastroenterological Association Institute

September 15, 2009

AGA Institute Guideline Development Process Prior to 2007

Searching for and appraising the evidence

•Authors performed literature searches

•Table below was provided to aide authors in organizing findings but not a requirement

•Evidence summarized in Technical Review document

•Technical review (intended for academics) summarized into Medical Position Statement (intended for practicing clinicians)

Reference Year Study Design/type

No. of Subjects

Length of Study

Intervention Outcome Measurement

Statistical Analysis

Conclusions Comments

AGA Institute Guideline Development Process Prior to 2007

Inconsistencies identified in Pre-2007 Process

•Technical review and medical position statement not adequately evidence based

•Disconnect between technical review and medical position statement

Not guiding readers in interpreting the data

Not giving clear clinical recommendations

•Concept of quality of evidence and strength of recommendations not well emphasized

AGA Institute Guideline Development Current Process

Modifications made to the process• Construction of specific clinical questions to be addressed

in technical review

• Implementation of U.S. Preventive Services Task Force (USPSTF) Ratings – Management of GERD guideline

• Composition of Medical Position Panel o Patient/patient advocateo Payero Gastroenterologist in community practiceo Gastroenterologist with knowledge in health services

research o Primary care physician, surgeon or pathologist (included

as needed based on topic)

AGA Institute Guideline Development Current Process

Challenges of current process

•USPSTF grading system primarily designed for preventive services

•USPSTF grading system changed after first guideline was already in the final stages (mid 2007)

•USPSTF grading system caused confusion amongst guideline authors (Dysplasia in IBD guideline) how to grade evidence and recommendations (e.g., diagnostic tests)

AGA Institute Guideline Development Current Process

What made the AGA Institute decide to consider switching from USPSTF to using GRADE?

•Invitations to collaborate with other GI and non-GI societies

•Establish uniformity across all GI societies

Further invitations for collaboration led to the discovery that more societies are using or also considering GRADE

Considered using GRADE to reduce the confusion in different grading schemes used in various guidelines

Joining effort to establish an international standard

AGA Institute Guideline Development Current Process

Benefits of using GRADE we considered:

•A very rigorous and formalized process that will ensure evidence-based guidelines necessary for value-based health care initiatives (e.g., P4P)

•Already closely resembles AGA Institute’s current process

•Provides increased transparency of judgments of quality of evidence and strength of recommendations

•Investment into GRADE evidence profiles in the long-term could be future asset at the time of guidelines update

•Availability of GRADE methodologists who could assist in the process

AGA Institute Guideline Development Current Process

Initial reservations about using GRADE

•Higher costs?

Need to hire full time methodologist? Answer: GRADE not necessarily more complex than other grading systems.

Increased training of staff and authors? Answer: Any grading system would require some form of training

•Requires more time and lengthens the development process?

• Answer: No. The most time consuming and costly phase of the guideline development process is the systematic review of the evidence. This is independent from the grading system used.

AGA Institute Guideline Development Current Process

Next Steps

•Develop an implementation plan

Train staff in GRADE process

Train committee members in GRADE process

Include GRADE experts on committee

•Include in budget cycle for 2010-2011