Systematic Review Module 7:Systematic Review Module 7:Rating the Quality of Individual Rating the Quality of Individual

StudiesStudies

Meera Viswanathan, PhD Meera Viswanathan, PhD RTI-UNC EPCRTI-UNC EPC

Learning ObjectivesLearning Objectives

Define the concept of quality Define the concept of quality assessmentassessment

What are the reasons for quality What are the reasons for quality assessment?assessment?

What are the stages in quality What are the stages in quality assessment?assessment?

How do we report quality assessment?How do we report quality assessment?

CER Process OverviewCER Process Overview

3

Prepare topic:

· Refine key questions

· Develop analytic frameworks

Search for and select

studies:

· Identify eligibility criteria

· Search for relevant studies

· Select evidence for inclusion

Abstract data:

· Extract evidence from studies

· Construct evidence tables

Analyze and synthesize data:

· Assess quality of studies

· Assess applicability of studies

· Apply qualitative methods

· Apply quantitative methods (meta-analyses)

· Rate the strength of a body of evidence

Present findings

Reasons for Quality Reasons for Quality AssessmentAssessment

Quality assessment is required for– Interpreting results

– Grading the body of evidence

Quality assessment may also be used for– Selecting studies for inclusion

– Selecting studies for pooling

4

What is Quality Assessment?What is Quality Assessment?

Quality can be defined as “the extent to which all aspects of a study’s design and conduct can be shown to protect against systematic bias, nonsystematic bias, and inferential error” (Lohr, 2004)

Considered synonymous with internal validity

Relevant for individual studies Distinct from assessment of risk of bias for

a body of evidence

5

Lohr KN. Rating the Strength of Scientific Evidence: Relevance for Quality Improvement Programs. International Journal for Quality in Health Care 2004; 16(1):9-18.

What are Components of What are Components of Quality Assessment?Quality Assessment?

Systematic errors include selection bias and confounding, in which values tend to be inaccurate in a particular direction

Nonsystematic errors are attributable to chance

Inferential errors result from problems in data analysis and interpretation, such as choice of the wrong statistical measure or wrongly rejecting the null hypothesis

6

Lohr KN, Carey TS. Assessing 'best evidence': issues in grading the quality of studies for systematic reviews. Joint Commission . Journal On Quality Improvement 1999, Sep; 25(9), 470-9.

Consider the Contribution of an Consider the Contribution of an Individual Study to a Body of EvidenceIndividual Study to a Body of Evidence

7

Internal validity Internal validity of resultsof results

Size of studySize of study(random error)(random error)

Direction and degree of resultsDirection and degree of results

Relevance of Relevance of resultsresults

(applicability)(applicability)

Type of studyType of study

Limitations in Limitations in study design and conductstudy design and conduct

Risk of BiasRisk of Bias

PrecisionPrecision

DirectnessDirectness

ConsistencyConsistency

ApplicabilityApplicability

What Are the Stages in What Are the Stages in Quality Assessment?Quality Assessment?

Classify the study design Apply predefined criteria for quality and

critical appraisal Arrive at a summary judgment of the

study’s quality

8

Questions to Consider When Questions to Consider When Classifying Study DesignClassifying Study Design

Is a control group present? Is there concurrent assessment of intervention or

exposure status? Do investigators have control over allocation and

timing? Do investigators randomly allocate interventions? Is there more than one group? Is there concurrent assessment of outcomes?

9

Apply Predefined CriteriaApply Predefined Criteria

Apply one of several available tools that consider Similarity of groups at baseline in terms of baseline

characteristics and prognostic factors Extent to which valid primary outcomes were

described Blinding of subjects and providers Blinded assessment of the outcome Intention-to-treat analysis Differential loss to followup between the compared

groups or overall high loss to followup Conflict of interest

10

Additional Criteria for TrialsAdditional Criteria for Trials

Methods used for randomization Allocation concealment

11

Additional Criteria for Additional Criteria for Observational StudiesObservational Studies

Sample size Methods for selecting participants

(inception cohort, methods to avoid selection bias)

Methods for measuring exposure variables Methods to deal with any design-specific

issues such as recall bias and interviewer bias

Analytical methods to control confounding

12

Arrive at a Universal Arrive at a Universal Judgment of QualityJudgment of Quality

Assign ratings of good, fair, or poor Ratings may vary across outcomes for

an individual study Ratings should be based on the

assessment of the impact of individual criteria on overall internal validity rather than on summary scores

13

Attributes of Good StudiesAttributes of Good Studies

A formal randomized controlled study Clear description of the population, setting,

interventions, and comparison groups Appropriate measurement of outcomes Appropriate statistical and analytic methods

and reporting No reporting errors Low dropout rate Clear reporting of dropouts

14

Attributes of Fair StudiesAttributes of Fair Studies

Fair studies do not meet all the criteria required for a rating of good quality, because they have some deficiencies

No flaw is likely to cause major bias Missing information often drives rating

15

Attributes of Poor StudiesAttributes of Poor Studies

Significant biases, including– Errors in design, analysis, or reporting

– Large amounts of missing information

– Discrepancies in reporting

16

Reporting Quality Reporting Quality AssessmentAssessment

Overall assessments of quality must be accompanied by a statement of– Flaws in design or execution of a study

– Assessment of the potential consequences of that flaw

Poor studies may be excluded or included– Decisions should be guided by gaps in

current evidence

– Selective inclusion of poor studies for subgroups should be justified

17

Key MessagesKey Messages

Transparency of process– Full reporting on all elements of quality for

each individual study

– Clear instructions on how abstractors scored quality

– Description of reconciliation process

Transparency of judgment– Explanation of final score

18

Key SourceKey Source

Draft AHRQ Methods Guide, Chapter 6, AHRQ, 2007 http://www.effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf.

1919