different types of review and applying the principles of...

MRC/CSO Social and Public Health Sciences Unit

Different types of review and applying the principles of systematic review

Matt Egan

MRC/CSO Social and Public Health Sciences Unit

MRC|CSO Social and Public Health Sciences Unit

Outline

• Different types of review & systematic review

• Challenges for systematic reviews

• Applying systematic review principles to non-systematic reviews


Different types of review

• Systematic reviews of

• Intervention outcomes

• Intervention process

• Epidemiology

• Qualitative research

• Methods

• Rapid Review

• Scoping review

• Traditional review

•Conceptual/theoretical review •Realist review •Critical review •Expert review •“State of the art” review •Review of reviews


Not clear cut definitions

• Conceptual/theoretical review •Review of ideas, models, debates

• Realist review • Review of studies to identify generalisable theories

• Critical review

• Assess a theory by a less formalised critical assessment of studies

• Expert review • Review by acknowledged expert(s)

• “State of the art” review • Recent literature to bring readers up to date about research on specific subject


Use of “systematic review” label

• „Traditional review‟ is a term used by systematic reviewers to describe the kind of reviews they do not do

• Use of multiple labels can be unnecessarily complicated

• Don‟t waste time getting confused by “clever” labels

• Essential items of systematic review are: • Clear question

• Record of search sources & strategy

• Clear inclusion/exclusion criteria

• Assessment of study quality

• Description of synthesis methods: meta-analysis or narrative

• Any review type can use above principles of systematic review, e.g. expert review may have clear question etc


Most “systematic reviews” aren‟t systematic

• The validity of 480 systematic review and meta-analyses using a simple scale was assessed

• Thorough search

• Appraisal of study quality

• Assessment of heterogeneity

• 26% met all three the criteria

Petticrew M, Song F, Wilson P, Wright K International Journal of Technology Assessment in Health Care 2000; 15(4)671-8.


Marks of a good systematic review

• Essential characteristics- transparent & replicable

• Clear & focussed review question

• Search strategy described

• Comprehensive searching of all included sources

• Two reviewers to select included studies

• Two reviewers to assess quality of included studies

• Justification for method of synthesis- meta-analysis or narrative

• Synthesis reflects quality of included studies- less weight given to studies with high levels of bias

• Results & characteristics of individual studies to accompany final conclusions of synthesis- tables, forest plots etc


Making non-systematic reviews more systematic

Steps & components of systematic review

• Define question

• Write protocol

• Develop & conduct searches

• Screen for eligible studies according to clear incl/excl criteria

• Assess study quality

• Extract findings

• Synthesise data taking into consideration study quality

• Clear links between data and conclusions- good tables

Including any of these

stages will make a review „more systematic‟:

The more the better!


Different types of questions & data for systematic reviews


Policy relevant questions

• How, why and when do Community Based Disaster Risk Management initiatives impact on social and economic costs of disasters? Too many questions

• Are interventions to reduce the impact of arsenic contamination of groundwater on human health and crop productivity in developing countries effective? Effective to achieve what...

• Does private involvement (including public private partnerships) in delivery of water, information and communication technology and electricity services lead to improved access and quality of service in developing countries? Intervention not defined- very broad

Does communication for development contribute to the diffusion of information that results in behaviour change around key social issues such as HIV, domestic violence etc? Intervention not defined- very broad


What kind of questions can a systematic review address?

• What works?

• How does it work?

• Will it do more harm than good?

• Is it worth the money?

• Acceptability and appropriateness to certain groups

• Is there an association between A & B?

• How strong it the association?

• What affects the association?

• How was intervention implemented?


What type of research evidence can be included?


Methodological aptness: choose data that can answer the question Research question Qualitative

Research Surveys Case control

studies Cohort Studies

RCTs Sys Reviews

Effectiveness Does this work?

+ ++

+++

Effectiveness of service delivery: How does it work ?

++

+

+++

Salience Does it matter ?

++

++

+++

Safety: Will it do more harm than good?

+ + + ++ +++

Acceptability Will children/parents want to use it?

++

+

+

+++

Cost-effectiveness: Is it worth buying?

++ +++

Satisfaction with service

++ ++ + + +

Appropriateness Is this the right service

++

++

++

(Muir Gray, 1997 & Petticrew, 2003)


Challenges for systematic reviews

• Large reviews can take a long time and are expensive to conduct

• This can be harmful...

• Requires considerable expertise

• This is a particular problem for reviews of complex social interventions

• Difficult to identify literature

• Requires broad search strategies

• Includes diverse study designs, interventions & outcomes etc

• Multiple relevant questions- not just what works?


Delaying the publication of findings can cause harm


Rapid reviews

• Large team of reviewers to conduct large review more quickly • Still costly

• Narrow focus of review to conduct review more quickly

• Limit review criteria

• Sources

• PICOCS

• Assessment of study quality • Advantage: quicker & cheaper

• Disadvantage: Limited review criteria

• Sometimes this is the most appropriate use of resources

– Can be useful to establish nature of available research for broad policy questions


Rapid reviews: maintain systematic principles

• Important to maintain transparent methods

• Keep record of selection of included studies with justification

• Consider study quality in the synthesis

• Is the included evidence strong enough to answer the questions?

• Consider how the limits of the review might affect your conclusions?

• E.g. Only included data from Sri Lanka, only included RCTs, only included since 2000 etc etc


Making non-systematic reviews more systematic: use principles of systematic review

Steps & components of systematic review

• Define question

• Write protocol

• Develop & conduct searches

• Screen for eligible studies according to clear incl/excl criteria

• Assess study quality

• Extract findings

• Synthesise data taking into consideration study quality

• Clear links between data and conclusions- good tables

Including any of these

stages will make a review „more systematic‟:

The more the better!



Example 2: theory review

Why do one?

• Bigger picture: whilst empirical research often focuses on single interventions or risk factors, theory reviews identify the underlying assumptions across a range of research.

• Understanding theories underpinning interventions can aid transferablity

Tips

• Manual searches likely to be particularly important.

• Defining a „theory‟ is tricky. Input mechanism outcome

Drawback

• No systematic way of critically appraising theories

• Not always obvious what to conclude or recommend


Example of theory review



Example 4: scoping reviews

Why do one?

• Quick (rapid review)

• Prepare the ground for a more systematic review

• If the field is under researched you can use a broad question to produce a review that is both systematic and scoping

Tips

• Not too broad: eg. Similar interventions, different outcomes;

• Focus on how to deal with heterogeneity: e.g. categorise findings.

• May need to restrict search to stop it becoming unmanageable.

Drawback

• Too big?

• Too heterogeneous?


Examples of scoping systematic reviews

New Roads

•Similar interventions •Different study designs •Different outcome measures (Egan et al, AJPH, 2003)

Modal Shift

•Different interventions •Different study designs •Similar outcomes measures (Ogilvie et al, BMJ, 2004)


Different types of review

• Systematic reviews of

• Intervention outcomes

• Intervention process

• Epidemiology

• Qualitative research

• Methods

• Rapid Review

• Scoping review

• Traditional review

•Conceptual/theoretical review •Realist review •Critical review •Expert review •“State of the art” review •Traditional review •Review of (systematic) reviews


Why do reviewers care about implementation process, mechanisms, context, etc?

• So we can learn how to improve intervention delivery

• To help us assess the extent to which findings are generalisable and transferable (relationship between context, implementation and outcomes).

• So we can distinguish between theory failure and implementation failure.


Including process, context and mechanisms in systematic reviews of complex interventions



Theme Does the study describe….. (yes/no)

1. Motivation

Why the management decided to do the intervention? Was concern about stress/health a factor?

2. Theory of change

A theory of change describing pathway from implementation to health outcome?

3. Implementation context

Political, economic, managerial or other contextual factors relevant to implementation?

4. Experience

Whether those implementing the intervention had appropriate experience/training to implement intervention?

5. Planning consultations

Consultation between managers, employees and any other relevant parties during planning stage?



Theme Does the study describe….. (yes/no)

6. Delivery collaborations

Consultation between managers, employees and any other relevant parties during delivery stage?

7. Manager support

Were on-site managers/supervisors supportive of the intervention?

8. Employee support

Were employees supportive of the intervention?

9. Resources

resources required in implementing the intervention (eg time, money, people, equipment)?

10. Differential effects Who benefited and who didn't?


Specific problems with our checklist

• Ignorance: No well established hierarchy for implementation quality. We didn‟t know what a well implemented workplace reorganisation intervention looked like.

• Uninformative: unclear if we are reviewing implementation or reporting of implementation.

• Simplistic: not clear which implementation issues are most important. So there‟s a danger of implying (falsely) that they are of equal weight

• There wasn’t much to appraise.

(average checklist score was 3 out of 10).


Problem not just about quantity of reporting – also quality

• General reliance on anecdotal evidence rather than a planned study of implementation.

• Sometimes appeared to privilege one view point – e.g. either managers or employees.

• Few studies discussed how implementation influenced outcomes.

• Quality of implementation reporting did not vary much by type or date of publication, or methodological quality, or health outcome.


Qualitative version of the checklist (no score)


This more qualitative approach produced more useful findings

• Context: Participation interventions don‟t work in downsizing organisations. Task restructuring interventions could be harmful when driven by financial rather than health considerations.

• Mechanism(?): Findings often supportive of psychosocial „demand, control, support‟ models explaining workplace health – especially control.

• Differential effects: Some evidence that the positive and negative health effects of organisational changes to the psychosocial work environment are felt more by men and lower socio-economic groups.


Conclusion Do you like your water clear or muddy?

A purist approach to systematic reviews leads to high quality, comprehensive reviews that reduce the risk of erroneous conclusions.

A less pure approach may have advantages related to speed, efficiency, pragmatism, innovation and ability to review a wider range of topics.

different types of review and applying the principles of...

Documents