developing systematic reviews karin hannes featuring mieke ... · 2003) review protocol anatomy •...
TRANSCRIPT
Developing systematic reviews
Karin Hannes
Featuring Mieke HeyvaertCentre for Methodology of Educational Research
Program
• 14.00: Introduction
• 14.30: Searching the literature
• 15.00: Critical appraisal
• 15.20: Break
• 15.30: Critical appraisal exercise
• 16.00: Data-analysis: a worked example (M. Heyvaert)
• 16.45: Anatomy of a systematic review
Introducing Cochrane,
Campbell and systematic
reviews
Outline
• Systematic reviews
– What is it and why do we need them?
– How do they look like?
– Who is producing them and where can we locate them?
– What is their (potential) impact?
– How can we start producing one ourselves?
What is it?
A systematic review is a „systematic identification, evaluation and synthesis of all relevant studies regarding a specific topic, based on an explicit and pre-defined methodology‟
If high quality research
studies exist...
Summarize them!
Half of what you learn in school will beshown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you whichhalf! (Dr. Sydney Burnwell)
Why do we need them?
• Evidence of „effectiveness‟: the extent to which an intervention, when used appropriately, achieves the intended effect.
• Evidence of „feasibility‟: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context.
• Evidence of „appropriateness‟ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given.
• Evidence of „meaningfulness‟: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.
Why do we need them?
“Over two million articles are published
annually in the biomedical literature in over
20,000 journals – literally a small mountain of
information … a stack … would rise 500
metres”
Cynthia Mulrow, in Systematic Reviews (BMJ
Publishing Group, 1995)
And within that stack, we would be
looking for a needle…
What is it?
• Lack rigor
• Methodology not
transparent
• Different reviewers reach
different
conclusions
• Become out of date
• Scientific rigor to minimise bias
• Explicit and reproducible
methodology
• Regularly updated
(Cochrane/Campbell)
Traditional
reviewSystematic
Review
What is it?
Predefined format:
To help review authors be systematic
To help people reading the reviews to find
information quickly
Transparancy
Reliability
Key characteristics of a SR
1. A clearly stated set of objectives with pre-defined
eligibility criteria for studies
2. An explicit, reproducible methodology
3. A well defined, systematic search that attempts to
identify all studies that would meet the eligibility criteria
4. An assessment of the methodological quality of the
findings of the included studies (assessment of risk of
bias )
5. A systematic extraction, synthesis, and presentation
of the characteristics and findings of the included
studies
PART 2: SYSTEMATIC REVIEWS OF
EFFECTIVENESS
HOW DO THEY LOOK LIKE?
What is a meta-analysis?
Optional part of a systematic review
Systematic reviews
Meta-analyses
• To identify a common
effect among a set of
studies
• To improve precision of
an estimate
• To investigate whether
the effect is constant
• To answer controversies
arising from conflicting
studies or to generate
new hypotheses
Meta-analysis
Gives a more precise estimate of effect than when
derived from the individual studies included within a
review
Intervention Comparisonno effect
Skills training versus usual curricula in the
prevention of drug use in school kids.
1. Summary statistic
+ CI for each study
individually (RR,
OR, RD…)
2. Pooled intervention
effect + CI is
calculated as a
“weighted average”
Meta-analysis
Facilitates investigations of the consistency of
evidence across studies, and the exploration of
differences across studies.
Intervention Standardno effect
Skills training versus usual curricula in the
prevention of drug use in school kids.
Heterogeneity!
CC & C2: meta-analyse
„Skill-based interventions‟ prove to be an effective
strategy to prevent from marihuana use in schoolkids.
Narrative summary
PART 3: SYSTEMATIC REVIEWS
WHO IS PRODUCING THEM?
“Combinations of data might
be better than attempts to
choose amongst them”
Astronomy, 17th century.
Karl Pearson, 1904.“Many of the groups are far too
small to allow of any definite
opinion being formed at all, having
regard to the probable error
involved”
Glass, 1976.
Archie Cochrane‟s challenge
“It is surely a great
criticism of our profession
that we have not organised a
critical summary, by
specialty or subspecialty,
adapted periodically, of all
relevant randomised
controlled trials.”
(1979)
Cochrane Collaboration: 1993
an international not-for-profit
organisation which aims to help
people make well-informed
decisions about healthcare by
preparing, maintaining and
promoting the accessibility of
systematic reviews of the effects of
health care interventions
Cochrane Collaboration
Collaborative
Review
Groups
Fields
The
Consumer
Network
Centres
Steering
Group Methods
Groups
Steering Group: Policy making body of the Cochrane
Collaboration.
Elected representatives of each of the Collaboration
entities
Fields: Represent a
population, group, or
type of care that
overlaps multiple
Review Group area.
Examples: Primary
Health Care, Health
Care of Older Adults,
Complementary
Medicine
Methods Groups:
Develop methods
and products integral
to internal functioning
of the Collaboration
Develop state of the
art methods for
systematic review
Examples: Statistical,
Economics, Placebo
Effects, Informatics,
Qualitative
Centres: Help organise and register
review groups
Facilitate collaboration among
reviewers
Provide training and consultation
Establish liaisons
Promote the Cochrane Collaboration
Provide unique contributionReview Groups: Focus on Treatment of disease or health problemsTo prepare and maintain systematic reviews
To develop a specialised trials register
Consumer Network:
Provides consumer input
Helps set priorities
Helps with dissemination
Campbell Collaboration (C2): 2000
an international organisation which
aims to help people make well-
informed decisions about public
policy (crime & justice, education,
social welfare) by preparing,
maintaining and promoting the
accessibility of systematic reviews of
the effects of social and
behavioural interventions
Campbell Collaboration
Coordinating
Groups
Users
Group
Partners
Steering
Group Methods
Groups
Users group:
Knowledge translation
to increase the impact of Campbell
reviews in policy and practice arenas,
and to make the information more
accessible
Methods Groups:
Improving the
methodology of
research synthesis,
and disseminating
guidelines for state-
of-the-art review
methods
Coordinating Groups:
Provide editorial
services and support to
authors of Campbell
reviews and build links
with users of systematic
reviews:
Crime and Justice
Social Welfare
Education
International Development
Steering group: strategic and policy
making body
Partners:
Institutes who have a
formal agreement with
Campbell and support the
same goals.
The Belgian Campbell Group
http://www.campbell-
collaboration.be/Belgian_Campb
ell_collaboration/Welcome.html
Examples of C2 Review Protocols and Reviews
are available on:
http://www.campbellcollaboration.org/library.php
CC & C2 Collaboration goals
• To ensure high quality, up-to-date systematic reviews
are available across a broad range of topics
• To promote access to systematic reviews
• To develop an efficient, transparent organisational
structure and management system for the Collaboration
• To achieve sustainability of the CollaborationPrinciples
Collaboration
Building on the enthusiasm of individuals
Avoiding duplication
Minimising bias
Keeping up to date
Striving for relevance
Promoting access
Ensuring quality
Continuity
PART 4: SYSTEMATIC REVIEWS
WHAT IS THEIR (POTENTIAL)
IMPACT?
Evaluate a solution
• Situation: 1 mj. people die each year due to traffic accidents
• Solution: Educate the drivers
• Systematic Review: 24 studies on education, no evidence that it reduces traffic accidents
• Recommendations: Stimulate teleconferences and the use of safe, public transport
“Driver education is big business – our results show that it is also a big con” (Prof. Roberts)
Identify gaps in „evidence‟
• Situation: anti-psychotics are subscribed for pregnant women with a psychotic disease
• Systematic Review: No literature on adverse effects of anti-psychotics on the mother or the development of the (unborn) baby
– RCT‟s with pregnant women are considered unethical
– Medicals‟ referals are based on habits and opinions
• Recommendation: Research on the effects of anti-psychotics in pregnant women is necessary
“The continued use of antipsychotic drugs in women during pregnancy and lactation without sound evidence raises serious clinical and ethical concerns.” (Webb)
Identify solutions that cause „harm‟
• Situation: Illinois law: Chicago Public Schools mandated to identify childrenat-risk for future criminal behaviour.
• Solution: Scaired straigth programs: take them on tours of adult prison facilities
• Systematic Review: Not only does a scared straight program fail to detercrime, it actually leads to more offending behaviour.
“Governments need to adopt rigorous evaluation to ensure that they are not causing more harm to the citizens they intend to protect” (Dr. Petrosino).
PART 5: SYSTEMATIC REVIEWS
HOW CAN WE CONTRIBUTE TO
THEM?
1. Select a topic
2. Make a review team
3. Develop/Register your title
4. Write a protocol
1. Select a topic
Systematic reviews:
• Commissioned
• Invited
• Unsolicited
Motivation to undertake a review:
• Resolve conflicting evidence
• Address questions when practice is uncertain
• Explore variations in practice
• Confirm appropriateness of current practice
• Highlight need for future research
• Learn to ask questions
(Treatment or Intervention):
• P population
• I intervention
• C comparison
• O outcome
1. Select a topic
Does providing information (I) at school have a preventive effect on the use of drugs (O) in high school students (P) compared to sanctions whensomeone is caught using drugs (C) ?
1. Select a topic
“Information campagnes in the
prevention of drug use”
“Does information provision at school
decreases the risk of drug use in high
school students (compared to a
standard approach)?” Scope of the question?
BROAD (LUMP) NARROW (SPLIT)
Advan
-tages
Comprehensive
Generalizibility
Effect modifiers (why different effect
between apples and oranges?)
Manageable for review team
Easy to read
Dis
advan
-tages
More resources
“mixing apples and oranges”
Evidence sparse?
Lack of generalizibility
Choice of scope “biased” by authors
Excercise 1: developing a question
2. The review team
Why more than 1 person?
• Detection of errors (selection of eligible studies, data extraction)
• Reduces risk of bias
• When more than one person / team is interested in topic
Who should be in the team?
• Expertise in topic area
• Expertise in systematic review methodology
• Incorporate view of “users” (teachers, consumers, therapists)
• Address questions that are important to people
• Take account of outcomes that are important to those affected
• Make it accessible to people making decisions
• Reflects variability in populations, settings…
3. Registering a title
Contact a Cochrane Review Group (CRG) or Campbell
Coordinating Group (CCG) and fill in a title form
Cochrane Review Group:
• > 52
• Specific condition (eg
incontinence, learning & developmental
disorder group)
• Group of diseases (eg ear nose
and throat disorders group)
• Organ (eg renal group, heart group)
• Patient (eg neonatal group, pregnancy
and childbirth group, pain and palliative
care)
Campbell Coordinating Group:
• 4
• Crime & Justice CG
• Education CG
• Social welfare CG
• International Development
There is a potential to
co-register reviews!
4. Write a protocol
• A priori statement of aims and methods of the review
• Research question(s), aims, methods are considered in
advance of identifying the relevant literature
– Conduct review with minimal bias
– Access to peer review
– Greater efficiency in review process
(Torgerson, 2003)
ADVANTAGES:
• Clear research question before the review
Avoid retrieving irrelevant papers
• A priori inclusion and exclusion criteria avoid changing criteria as review progresses or studies may be included on basis of their results
• If decisions are explicit it enables them to be justified
• Develop protocol as independently as possible from the literature avoid influence by one or two major studies, less bias (Torgerson, 2003)
Review Protocol Anatomy
• Title and authors
• Background
• Objectives
• Methods
– Searching for studies
– Selecting studies
– Data extraction (including assessment study quality)
– Data analysis
• Acknowledgements / conflicts of interests
• References
• Tables / Figures
Review Protocol Contents
BACKGROUND AND OBJECTIVES
• Describes the context of the review: why is it important?– Description of the problem
– Description of intervention
– How the intervention might work
– Description potential subgroups
• Clarify conceptual issues central to the review
• Precise statement of objective and research question, using PICO(S)– Which population?
– Which intervention(s)?
– Which comparison(s)?
– Which outcome(s)?
– (Which study design?)
A well formulated objective focuses the review
To assess the effects of
home-based programmes aimed at
improving developmental outcomes for
pre-school children from disadvantaged
families
Review Protocol Contents
METHODS
• How to select studies?
• How to search for studies?
• Assessment quality of included studies
• How to extract data?
• How to analyse results?
Review Protocol Contents
SELECTING STUDIES• Define inclusion and exclusion criteria
• Key features:
– Established a priori
– Explicit (explain all terms)
– Based on PICO
– Include study design, if applicable
• All retrieved studies are listed in the review either under included or
excluded together with a justification
Review Protocol Contents
Types of studies
Randomised controlled trials. The control group will either receive no intervention or standard
care. Studies comparing two different types of home based programme without a control group
will be excluded.
Types of participants:
Home based child development interventions for pre-school children from socially
disadvantaged families & Parents with children up to the age of school entry and who are
socially disadvantaged in respect of poverty, lone parenthood or ethnic minority status. Age of
school entry can vary between countries (4 to 7 years) and so the upper age range for this
review will be the school entry age for the country in which the trial took place.
Studies will be excluded if they aim to recruit particular clinical subgroups of parents.
Types of outcome measures
Primary outcomes: Cognitive development (including language development and attention) &
Socio-emotional development (including self regulation and behavioural development) +
Adverse outcomes: Parents feel disempowered.
Types of interventions
Home based interventions designed to improve child intellectual and socio-emotional
development through the provision of relevant knowledge and skills to the parent.
The intervention is delivered by trained lay or professional family visitors.
Review Protocol Contents
SEARCH FOR STUDIES
• Process needs to be reproducible!
• Describe search terms + search strings (BOOLEAN
LOGIC)
• Describe sources (electronic databases, other sources,
etc)
Review Protocol Contents
ASSESSMENT OF STUDY QUALITY
• Study quality may affect conclusions
• A systematic review always includes an assessment of
individual study quality
• Different tools available
• Describe how to use this information
– Description methodological quality
– Exclude studies of poor quality?
– To include in conclusions (strong versus weak evidence)
Review Protocol Contents
DATA EXTRACTION
• Process of reading through a study and extracting the
relevant information from each study, preferably by two
independent reviewers
• The reviewer fills out an data extraction sheet with the
appropriate information taken from that study
• Data extraction sheet may be on paper or electronically
Extraction form consists of two different parts
1. characteristics that apply to the entire study
2. results: studies typically have multiple outcomes
and therefore different effect sizes
Descriptive
Statistical
part
Summary
• Review protocol important first step when
undertaking systematic review
• Helps to focus and structure the review
• Limits the scope for bias
• In addition to how you plan to deal with the
different steps in a review, include information
on:– How many reviewers?
– How are reviewers trained?
– How to resolve disagreements?
Check the resource centre at
the Campbell Website for
specific guidance from
methods groups and
coordinating groups
http://www.campbellcollaboration.or
g/resources/research.php
References / more reading
• Egger M & Davey-Smith G. (2001). Principles of and procedures for systematic reviews. In M Egger G Smith & D Altman (Eds.), Systematic Reviews in Health Care: Meta-analysis in Context (2nd ed., pp. 23-42 ). London, UK: BMJ Books.
• Lipsey MW & Wilson DB. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.
• Miller S & Eakin A. Home based child development interventions for pre-school children from socially disadvantaged families (Protocol). The Campbell Library, 2011.
• Petticrew M & Roberts H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishing.
• Torgerson C. (2003). Systematic Reviews. London, UK: Continuum.
• Resources Campbell Collaboration: http://www.campbellcollaboration.org/
• education_articles/ECG_Resources_for_Reviewers.php
• Resources Cochrane Collaboration: http://www.cochrane.org/training/cochrane-handbook
Searching the literature
Outline
• The Information Retrieval Process– Search strategies: Decisions and challenges
• Searching the Main Databases– Selection and Types of databases
– Preparing a search strategy
– Implementing a search
– Saving & Managing the results
• Additional Databases & Retrieval Methods
• Wrap Up
• Resources & Additional Readings
Information Retrieval:
A Continuous Process
– Preliminary (scoping) Searches• Supports beginning steps: Definition of key concepts &
research question
• Use of standard reference tools and broad searches for review articles and key primary studies
– Main Searches• Identification of primary studies through searches of
databases, Web, branching, manual searches
• Most difficult given a number of challenges
Main Searches: Decisions
• Selection of Information Retrieval Tools
– Scope of search: Which disciplines or subject fields should be searched (including all related fields)?
– Availability of indexing tools & expertise: Which tools do we have access to at our institution? Are there others who can perform searches for us?
– Format of indexing tools: What format are they in (e.g. online, print, web-based)?
– Dates: How far back does the indexing go for each tool?
– Language: What is the language of the material that is indexed? How can we locate non-English material?
– Unpublished work: How can we access dissertations, reports, and other grey literature?
Exclusion criteria:
Selection of Databases
• Consult your academic library‟s website to learn what
databases are accessible from your institution!
– Education: ERIC, British Education Index, Australian Education Index, CBCA Education, Education: A SAGE Full-text Collection; Education Full text, Linguistics and Language Behavior Abstracts, …
– Psychology: PsycINFO, PubMed (Medline), Ageline, Psychology: A SAGE Full-Text Collection,Criminology: A SAGE Full-Text Collection, …
– Sociology: Sociological Abstracts, Contemporary Women‟s Issues. Sociology: A SAGE Full-text Collection, …
– Health Care: Medline, CINAHL, EMBASE, AMED, Cochrane Trial Register,…
– Multidisciplinary: Academic Search Premier, ProQuestDissertations and Theses, FRANCIS, Social Sciences Index, SCOPUS, Web of Science
Main Searches: Decisions– Preparation of Search
Strategies
• What are the key concepts to be searched?
• How are these represented in each discipline?
• What are their related terms?
• How are these key concepts represented in the controlled vocabulary within each database to be searched?
Approaches to Parent Involvement
for
Improving the Academic Performance
of
Elementary School Age Children
Chad Nye, Herb Turner, Jamie Schwartz
The purpose of this review is to determine the
effectiveness of parental involvement in
improving the academic performance of school
age children in grades K-6.
Using a Thesaurus
1. From the research question, determine the main concepts to be searched (usually there are three):
– Intervention: Parental involvement
– Outcome: Academic performance
– Population: Kindergarten or Elementary students
2. Consult the main database to be searched.
3. Look up each concept in the thesaurus for this database.
– A thesaurus is an alphabetical listing of the controlled vocabulary(or descriptors) used within a subject database
– A hierarchical arrangement is used so that Broader, Narrowerand Related headings may be discovered
Check definition of terms.
Keep those from which
the definition matches
your inclusion criteria.
Example: ERIC
• Selecting the ERIC Descriptors
– Descriptors: Parental Involvement See: Parent participation
– Related descriptors: Family involvement, Parent-school relationship, Parent role, Parents as teachers
– Related keywords: parent* involvement, parent* effectiveness, parent* support, family support
– Descriptors: Academic Performance See: Academic achievement
– Related descriptors: Science achievement, Reading achievement, Writing achievement, Achievement gains
– Descriptors: Elementary School Children See: Elementary school students
– Related descriptors: Elementary education, Primary education, Kindergarten
Intervention
Outcome
Population
Main Searches: Decisions
• Construction of the Search Statements
– What terms should be searched as descriptors or as “keywords”?
– What Boolean operators should be used? AND / OR / NOT / NEXT / NEAR?
– Where should truncation characters be used? (e.g. parent* will retrieve parent, parents, parental)
– What limiting features are available to narrow results? (e.g. use of Publication Type codes, time period, language)?
ERIC Example
Boolean Operators
AND: Both terms must be present in order for a record to be
retrieved. Used to combine different concepts.
e.g parent participation AND achievement
OR: Either term may be present in order for a record to be
retrieved. Used to search for related terms or synonyms.
e.g. parent OR family
NOT: Used between two terms to ensure that the second term
will not appear in any of the results.
e.g. literacy NOT adult
(Parental involvement OR parent participation) AND academic
achievement AND (elementary OR primary education)
Example: ERIC, cont‟d
• Combining Keywords/Descriptors using Boolean operators:
1. DE=(Parent participation OR Family involvement OR Parent role OR Parent-school relationship OR Parents as teachers)
2. “Parent* involvement” OR “Parent* effectiveness” OR “Parent* support” OR “family support”
3. #1 OR #2 = 28,958 records
4. DE=(Academic achievement OR Science achievement OR Reading achievement OR Achievement gains) –46,574 records
5. DE=(Elementary school students OR elementary education OR elementary schools OR primary education OR kindergarten) - 291997 recs
6. #3 AND #4 AND #5 = 1,669 records
Intervention
Outcome
Population
Limiting Your Results
Using the Limiting Commands:
• Limiting fields contain information that is common to a large number of records within a database. These include language, document type, publication year and so.
• Some limiting fields will vary across databases (e.g. Classification Code, Age Group)
• Decisions about whether you are going to restrict:
• The language of the document
• The search to a certain time period
• The results to empirical studies only
ERIC record
AUTHOR : Polovina,-Nada; Stanisic,-Jelena
TITLE: A Study on Family-School Cooperation Based on an Analysis of School Documentation
PUBLICATION YEAR: 2007
SOURCE: Online Submission. Journal of Educational Research (Belgrade), v39 n1 p115-133
ERIC DOCUMENT LINK: http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED499159
DOCUMENT TYPE: Journal-Articles; Reports-Research
LANGUAGE: English
DESCRIPTORS: Student-Behavior; Attendance-Patterns; Child-Development; Parents-; Family-School-Relationship; Parent-Teacher-Cooperation; Foreign-Countries; Elementary-Schools; Parent-Participation; Grades-Scholastic; Academic-Achievement; Parent-Influence; Parent-Child-Relationship
IDENTIFIERS: Serbia-
ABSTRACT: Family-school cooperation is a very complex process that can be studied at different levels in a number of different ways. This study has covered only some aspects of cooperation between parents and teachers, based on school documentation of a Belgrade elementary school. The study covered analyses of 60 Attendance Registers pertaining to 60 classes with 1289 students from Grade 1 through Grade 8 during an academic year. The unit of analysis included: parents attendance at PTA meetings and individual meetings between parents and teachers. In addition to the frequency of parents' visits to school, therelationship between such registered parents' visits and overall academic performance, grades in conduct, excused and unexcused absence from classes were also considered. The research findings indicated interference between development factors (attitude change in parent-child relationship and growing-up) and parents' informal "theory of critical grades" i.e. transitional processes in schooling. The findings confirmed that parents' individual visits to school were mainly meant to offer an excuse for the student's absence from school, while attendance at PTA meetings was linked to poor grades in conduct and missed classes (both excused and unexcused). The findings also showed that parents pursued visiting strategies which were pragmatic, less time-consuming and less emotionally draining ones. The closing part refers to discussions on practical use of the study and possible further research. (Contains 4 graphs.) [This article is the result of the project "Education for Knowledge-Based Society" No. 149001 (2006-2010), financially supported by the Ministry for Science and Environmental Protection of the Republic of Serbia.] (Author)
High quality search strategies deliver relevant results!
Next Steps
Repeat these steps for each database to be searched.
Additional Retrieval Methods
• We haven‟t talked about….
– The web and other resources…to locate grey literature
• Use Advanced Search screens on large engines (eg. Google,
Altavista, AlltheWeb, MSN Live)
• Consult specific sites
• Screen conference abstracts and databasis storing
dissertations
– Manual searches
• Browse the Table of Contents of key journals for current years
– Reference Searching
• Look for relevant references in key-articles retrieved and
conduct a forward reference search in Web of Science.
Managing Your Results
• Export the results– Save as a Text file
• Import into a bibliographic management software:– RefWorks,
– Reference Manager,
– EndNote
• Edit your inhouse database– Add Source code for each database searched (e.g ERIC1,
PsycINFO1…)
– Add notes to the records (e.g.includes vs excludes)
• Compile a Search History document listing the original search strategies
Information Retrieval: Wrap Up
• Importance of information retrieval process– Not a “one-shot”deal
– Requires expertise in the planning and implementation of
searches
– Consulting with the Trials Search Advisor or an Information
Specialist is highly recommended
• Use a bibliographic management software– Store, manage and organize results
• Must have ability to replicate review– Documentation of entire process, including search strategies
used for each database, decisions taken, etc.
Resources
C2. Education Coordinating Group. (2009). Information Retrieval Methods Group Systematic Review Checklist and Database Worksheet. Available: http://www.campbellcollaboration.org/education_articles/ECG_Resources_for_Reviewers.php
Designed to help both reviewers and those reviewing C2 protocols.
– The Checklist itemizes the recommended steps for information retrieval during the protocol and review stages. A good resource for new C2 reviewers as it provides an indication of the criteria that will be used to evaluate the information retrieval component of a protocol and a review;
– The Database worksheet provides a useable, expandable template for the documentation of database searches.
Lefebvre, C., Manheimer, E., & Glanville, J. (2008, Feb.). Chap. 6: Searching for studies. In J.P.T. Higgins & S. Green (Eds.). Cochrane handbook for systematic reviews of interventions. London: Wiley.
Provides a “how to” guide for Cochrane Trial Search Coordinators. Clearly written and easy to understand for those who have no experience with searching. Focus is on the retrieval of information in the health sciences.
Information Retrieval Methods Group (2009). Searching for studies. Campbell Collaboration.
This comprehensive document provides the background on the C2 policy related to information retrieval and will be useful to Trials Search Co-ordinators (or Advisors) who are new to their post, as well those who are conducting reviews. This document outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage references found during the search process and correctly document and report the search process. It is currently being revised and updated.
Readings
• Gomersall, A. (2007). Literature searching: Waste of time of essential skill? Evidence & Policy: A Journal of Research, Debate and Practice, 3(2), 8301-308. Available:
• Hopewell, S., Clarke, M., & Mallett, S. (2006). Grey literature and systematic reviews. In Dr. H.R. Rothstein, A.J. Sutton, & M. Borenstein (Eds.). Publication bias in meta-analysis (pp.49-72). New York: John Wiley & Sons.
• McGowan, J. & Sampson, M. (2005). Systematic reviews need systematic searchers. Journal of the Medical Library Association, 93(1), 74-80.
• Wade, A., Turner, H. M., Rothstein, H. R., & Lavenberg, J. (2006). Information retrieval and the role of the information specialist in producing high-quality systematic reviews in the social, behavioral, and education sciences. Evidence & Policy: A Journal of Research, Debate and Practice, 2(1), 89-108
Critical appraisal
Why is it so important?
• Meta-analysis aims to increase precision
• Meta-analysis of studies with bias in results gives very
precise but wrong results
• Garbage in, garbage out
Methodological rigor relates
directly to the interpretation
and generalization of findings
(Troia, 1999).
Bias versus imprecision
BIAS Ideal study
Bias:
A systematic error in the results or the inferences
Methodological flaw
Overestimation or underestimation
Bias versus imprecision
Ideal study IMPRECISION
Imprecision:
• A random error in the results
• Sample variation
• Direction of error is random
Risk of Bias versus Bias
BIAS + IMPRECISION
•Clear empirical evidence that particular flaws in study
design can lead to bias.
•Usually impossible to know to what extent biases have
affected the results.
•Key consideration = should the results be believed
http://www.sign.ac.
uk/methodology/
checklists.html
http://www.cochrane
.org/contact/method
s-groups
Online guideance:
Chapter Critical Appraisal
Domaine Description Judgement
Sequence generation: Did the study use a randomizedsequence of assignments (centralized)?
QUOTE: “patients were randomly allocated” COMMENT: probablydone, since earlier reports of this study describe use of random sequences.
YES (low risk of bias)
Allocation concealment:Did the study use anymechanism that shields those who enroll patients in a study from knowing the next assignment?
QUOTE: “We used sealed enveloppes to allocate the students to either the intervention or the control group”. COMMENT: Risk unclear since there is no mention of any safeguards used, such as opaque or sequentially numbered enveloppes.
YES (low risk)
NO (high risk)
UNCLEAR (uncertain)
Blinding: Participants, Providers, Outcome assessors
Lack of blinding could affect actual outcomes•Differential drop-out•Differential cross-over to alternative intervention•Differential administration of co-interventions
Lack of blinding could affect the outcome asssessment
YES
NO
UNCLEAR
Incomplete outcome data: Data available, butexcluded from analysis: Withdrawal, Do not attendfollow-up appointment, Failure to complete questionnaire / diaries; Cannot be located (lost to follow-up), Decision by investigator to cease follow-up, Data or records are lost
QUOTE: “Participants found to be ineligible after enrolment”.COMMENT: Justifiable, so not considered as leading to missing outcome data. VERSUS: Difference in proportion of incomplete data across groups and related to outcomes (e.g. adverse effectsin experimental group)
YES
NO
UNCLEAR
Selective outcome reporting: Selection of a subset of the variables recorded for inclusion in publication,
on the basis of the results
Omission of non-significant outcomesChoice of data for an outcome (e.g. competences)Choice of analysis (e.g. scores on a test for skills)Reporting of subsets of data (e.g. no attitudes, little correlation)Under-reporting of data (e.g. only “not significant”)
YES
NO
UNCLEAR
Cochrane Tool
“Risk of bias summary”
Downs and Black
instrument (J Epidemiol
Community Health
1998;52:377-84)
http://www.nccmt.ca/
registry/view/eng/9.html
Tool for Non-
randomized
studies
Focus on
confounding
issues in studies!
In summary
• Risk of bias assessment is a very important part of
your review
• Think about it when you write your protocol, describe
the methodology
• For RCT: use the Cochrane tool
• For NRS:
• Potential biases are likely to be greater (selection
bias & reporting bias)
• Use the appropriate tool to assess risk of bias
• Consider how potential confounders are
addressed
Excercise: critical appraisal
“Schrijven van reviews”:
Een praktijkvoorbeeld
Mieke Heyvaert
Ph. D. Fellow of the Research Foundation Flanders (FWO)
Methodology of Educational Sciences Research Group, Faculty of
Psychology and Educational Sciences, Katholieke Universiteit
Leuven, Belgium
“Schrijven van reviews” -
Een praktijkvoorbeeld:
A meta-analysis of intervention effects on
challenging behaviour (CB) among
persons with intellectual disabilities (ID)
Background
Interventions for CB among persons with ID
Examples of CB: Verbal and physical aggression,
Property damage and destructiveness,
Disruptive and antisocial behaviour,
Overactivity, Temper tantrums, Screaming,
Stereotyped and repetitive behaviour, General
delinquency, and Self-injurious behaviour (e.g.
head punching, self-biting, skin picking, and
hitting against hard objects or other body-parts)
Background
• High prevalence of CB among persons with ID
• Negative impact of CB for persons with ID and
their family/context
Intervention research
Bio–Psycho–Social intervention components
Study aim
• Systematically review biological,
psychotherapeutic, and contextual intervention
components applied to treat CB among persons
with ID, & analyse intervention effects and
moderating variables
• Method: Meta-analysis
Boost the total sample size and effect
precision by combining „sufficiently‟
homogeneous results across studies
Method – Literature search
• Idealiter:
– Systematic search of Electronic databases;
Grey literature; Journals; Reference lists;
Citation indexes; Contact authors & experts
• For this study:
– Systematic search of 4 electronic databases:
ERIC, PsycINFO, Web of Science, Medline
Method – Literature search
Systematic search of 4 electronic databases:
ERIC, PsycINFO, Web of Science, Medline
How would you conduct this systematic
search?
Method – Literature search
• Articles reporting on intervention effects for CB among
persons with ID by combining key terms describing the target
group and outcome variables
• Key terms Target group: intellectual disability, mental
retardation, learning difficulty, complex/high support needs
• Key terms Outcome variables: challenging behaviour,
problem behaviour, behavioural problems, aggression, self-
injury, self-injurious behaviour
• Terms in singular and plural, in US- and UK-terminology
Method – Literature search
• Included: English-language articles
• Included: Empirical studies
• Included: Published in peer-reviewed journals
• Included: Published between January 2000 - November 2008
• Excluded: Articles that only reported on diagnostic
instruments / only presented review material
• Excluded: Articles had to contain data or sufficient statistical
information to make quantitative analysis feasible
Method – Data-analysis
• Coding
• Meta-analysis
• Sensitivity analysis
• Subgroup analysis
• Meta-regression analysis
• Publication bias analysis
Intervention effects
Intervention characteristics
• Intervention type
• Intervention combination
Methodological features
• Study quality
• Study design
• Data collection
• Reliability
Study characteristics
• Research period
• Continent
Participant features
• Gender
• Age
• Degree of ID
Coding
Method – Meta-analysis
Statistical analyses:
Comprehensive Meta-Analysis Version 2.0
(Borenstein et al. 2009, http://www.meta-
analysis.com)
Method – Meta-analysis
Fixed-effect vs. Random-effects meta-analysis:
• Fixed-effect meta-analysis:
• We assume that there is one true effect size („fixed
effect‟) which underlies all the studies in the analysis
• The goal is to compute the common effect size for the
identified population, and NOT to generalize to other
populations
Fixed-effect meta-analysis: Rather rare !
Method – Meta-analysis
Fixed-effect vs. Random-effects meta-analysis:• Random-effects meta-analysis:
• We allow that the true effect could vary from study to study. There may be different effect sizes underlying different studies – E.g. the effect size might be higher in studies where a more intensive variant of an intervention is used, with older participants, with higher educated participants,…
• The goal is to generalize to a range of scenarios
Random-effects meta-analysis: Often done ! When accumulating data from studies written by different independent researchers, it would be unlikely that all the studies are functionally equivalent
Method – Meta-analysis
Random-effects meta-analysis:
+ Between-study variance into account
+ Study weights are more balanced: Less
relative weight to large studies
+ Inferential results referring to a universe of
more diverse studies than fixed-effect
analyses
+ Width of the confidence intervals: Generally
more „conservative‟ and „realistic‟
Method – Meta-analysis
Random-effects meta-analysis:
• Assuming that the studies in the analysis only
represent a random sample of effect sizes that
could have been observed
• The summary effect is the estimate of the mean
of these effects
• Studies weighted by study precision: Studies
that yield more precise estimates of the effect
size are assigned more weight
Method – Meta-analysis
• (1) Calculate effect sizes (standardised mean
difference; SMD) and variances for all included
studies‟ intervention effects
• (2) Calculate the summary effect (effect size
and variance) with a 95% confidence interval
(CI)
Results – Meta-analysis
(1) Calculate effect sizes (SMD) and variances for
all included studies‟ intervention effects:
• Combined effect sizes and their standard
errors computed for all 30 articles are
presented in Table 1
• The applied interventions made an
improvement in each study: All calculated
effect sizes are positive
Method – Meta-analysis
(2) Calculate the summary effect (effect size and
variance) with a 95% confidence interval (CI) –
Random effects model:
• SMD is 0.671 with a 95% CI of 0.570 to 0.771
• Medium (effect size around 0.5) to large (effect
size around 0.8) effect
• The null hypothesis that the mean of these
effects is zero, can be rejected, Z(N = 30) =
13.070, p < 0.001
Results – Meta-analysis
Comparison: For the Fixed-effect model:
• Standardized mean difference is 0.670
• 95% CI of 0.572 to 0.768
• The null hypothesis that the mean of these
effects is zero, can be rejected, Z(N = 30) =
13.452, p < 0.001
The results for the fixed- and random-effects
analysis are analogous
Method – Sensitivity analysis
= Determine the robustness of our results by
examining whether our conclusions might differ
substantially if a study was dropped
• Running the analysis 30 times, each time
removing one study, in order to show each
study‟s impact on the combined effect
Results – Sensitivity analysis
• Calculate for each study the overall
standardized mean difference when that study
was hypothetically removed from the meta-
analysis
• Since the 30 overall effect sizes formed by
omitting each study separately varied between
0.649 and 0.689 (see next Table), our results
look very robust
Method – Subgroup & meta-regression analysis
To assess the impact of moderating variables:
• Subgroup analysis: Compare treatment effects
across groups concerning categorical variables
• Meta-regression analysis: Explores the impact
of continuous moderators
Results – Subgroup analysis
• Analyses of variance to compare treatment
effects across groups for the categorical
variables: intervention type, intervention
combination, quality assessment, study design,
data collection, reliability, length of the research
period, continent, gender, age, degree of ID
• For all these variables the differences between
the groups are not significantly related to the
effect size
Results – Meta-regression analysis
• One continuous moderator: Publication year
• This variable is not significantly related to the
effect size
• Possible explanation of „not significantly
related to the effect size‟: Relatively small
number of studies included in the meta-
analysis
Method – Publication bias analysis
We refer to publication bias when studies
included in an analysis differ systematically
from all studies that could be included:
Particularly, studies with larger effects are more
likely to be published, and this can lead to an
upward bias in the summary effect
Method – Publication bias analysis
• (A) Funnel plot analysis: Plot the included
studies
• (B) Classic fail-safe N analysis: Do we need to
be concerned that the entire observed effect is
an artifact of bias?
• (C) Duval‟s and Tweedie‟s trim and fill-analysis:
How would the effect size shift if the apparent
bias were to be removed?
Results – (A) Funnel plot analysis
Results – (A) Funnel plot analysis
• In the Figure a measure of study size
(precision, the inverse of standard error) is
plotted on the vertical axis as a function of
effect size on the horizontal axis
• In the absence of publication bias, we would
expect the included studies (white circles) to be
distributed symmetrically around the combined
effect size - OK!
Results – (B) Classic fail-safe N analysis
Results – (B) Classic fail-safe N analysis
• Quantify the possible publication bias effect
• The fail-safe N is 1336 We would need to
locate & include 1336 'null' studies in order for
the combined two-tailed p-value to exceed 0.05
• Because this number is large, we can be
relatively confident that the treatment effect,
while possibly inflated by the exclusion of some
studies, is real
Results – (C) Trim and fill-analysis
Trim and fill plot of precision by standardised difference in means:
Plot with observed (white) and imputed (black circles) studies
Results – (C) Trim and fill-analysis
Left side of the mean effect:
• Duval and Tweedie's trim and fill analysis
suggests that four studies are missing
• Random-effects model: SMD & 95% CI for the
combined studies is 0.671 (0.570, 0.771)
• Using trim and fill: Imputed SMD & 95% CI is
0.634 (0.525, 0.744) - Little bit lower than the
original SMD
Results – (C) Trim and fill-analysis
Right side of the mean effect:
• Duval and Tweedie's trim and fill analysis
suggests that no studies are missing
Conclusions
• We found effect sizes ranging from 0.223 to
1.411 The effect sizes vary between a small
and a very large effect, all indicating positive
treatment effects
• The combined effect size over all studies is
0.671, with a 95% CI of 0.570 to 0.771, which is
a medium to large effect
• The implemented sensitivity analysis revealed
that this effect is robust
Conclusions
• Analyses of variance showed no significant different
treatment effects for biological, psychotherapeutic, and
contextual interventions
• Differences between unimodal and multimodal
treatments turned out to be not significant as well
• The calculated means and standard errors for the
moderators intervention type and intervention
combination show that there are only small differences
between the mean effects for biological,
psychotherapeutic, and contextual interventions, and for
unimodal and multimodal treatments
Conclusions
• Some authors claim that there is evidence for the effectiveness of pharmacological treatments, although many authors assert that they lack empirical effectiveness. Additionally, there are many concerns regarding their adverse effects
• Concerning psychotherapeutic and contextual interventions too, there are authors advocating their effectiveness, while some say that the evidence is rather limited
• Our meta-analysis shows that there is evidence for the effectiveness of pharmacological, psychotherapeutic and contextual interventions, used alone or in combination
• Furthermore, we found no indications for the superiority of one of the treatment approaches or combination types
Conclusions
• Treatment effects also did not vary much across groups for the categorical moderators included in our meta-analysis: quality assessment, study design, data collection, reliability, length of the research period, continent, and gender, age, & level of ID of participants
• We also did not find any effect of the continuous moderator publication year on the treatment effect
• The nonsignificance of moderator effects is not uncommon, due to low statistical power for detecting interaction between a moderator variable and the independent variable
• There exist only small differences between the mean effects for the tested categorical moderating variables, so even if we would increase the power, we would most likely not detect clinically important differences
Conclusions
• A funnel plot-, a fail-safe N-, and Duval‟s and
Tweedie‟s trim and fill-analysis demonstrate
that our meta-analysis does not seem to suffer
much from publication bias effects
Conclusions
• In summary, interventions for challenging behaviour among persons with ID described in the 30 included articles were effective, with only small differences between the mean effects for biological, psychotherapeutic, and contextual treatments, and for unimodal and multimodal interventions
• In contrast to claims in the literature that the evidence for one or another intervention is still rather limited, the effects in our meta-analysis were robust and convincing
Bibliography
• Borenstein M., Hedges L. V., Higgins J., & Rothstein H.
(2009). Introduction to meta-analysis. Wiley, Chichester.
• Comprehensive Meta-Analysis Version 2.0 (2009).
http://www.meta-analysis.com
• Heyvaert, M., Maes, B., & Onghena, P. (2010). A meta-
analysis of intervention effects on challenging behaviour
among persons with intellectual disabilities. Journal of
Intellectual Disability Research, 54, 634-649.
Data-analysis: narrative
approaches
Synthesis examples
Preliminary synthesis examples
– Textual summaries
– Tabulation
– Grouping and clustering
• Exploring relationships examples
– Idea webbing
– Synthesis tabulation
– Concept mapping
Preliminary synthesis
• Textualsummaryexample:
Young et al (1999) and Camit (2002) report on the effectiveness and implementation of a smoke alarm preventioncampaign in NSW Australia for Asianforeigners, including info sessions by skilledworkers. Qualitative data onbarriers to purchase were alsocollected in focus groups. The mainobservation is that….
• Tabulation example
Author/year
Location/Setting
Target population Method Mainfindings
Robertset al (2004)
London, UK, urban
58 adults and 41 children in qualstudy, 2145 householdsexposed to intervention
Focus groups and interviewsBefore and After studydesign
Problemswith smokealarmsidentified as major barrier
Camit(1998)
NSW, Australia, mixed
Chinese, vietnamese, arabic-speaking(numbers notgiven)
Focus groups
Implementationsuccesfulusinglanguageappropriateapproach.
Preliminary synthesis
• Grouping example
Grouping according to
Location Focus of report Population
UKDiGiuseppe et al (1999), Roberts et al (2004)
Broad, generalfactors affectingprogram: DiGiuseppe (1999), Camit (2002)
Ethnically mixed:Camit (2002), Young et al (1999), Roberts et al (2004)
AustraliaCamit et al (2002), Young et al (1999)
Individual factors affecting program:Roberts (2004), Young et al (1999)
Ethnically mixed & low income:DiGiuseppe (1999)
Preliminary synthesis: thematic
• Within case
• Cross-case
Study Themes
Study 1 Involvement of fire service
False alarm issues
Fear from installers
Testing of knowledge
Study 2 Lack of awareness of danger
Overestimation of cost
Frequent moving
Lack of knowledge on detectors
Barriers Facilitators
Barriers and facilitators to installation of smoke alarms
General Anxiety of property damage Landlord approval for installation
Specific to alarm
campaign
Lack of knowledge on installation
Unwillingness to install
Anxiety to letting installer in the house
Installation of alarm by skilled
worker from community
Barriers and facilitators to continued use of smoke alarms
Specific to alarm Problems with maintenance Project workers offer to maintain
Exploring relationships: Web of Ideas
Exploring relationships:
Synthesis Tabulation
Exploring relationships:
Concept mapping
http://www.lancs.ac.uk/shm/research/nssr/res
earch/dissemination/publications.php
Systematic Review Summary
• Establish objectives, selection criteria
• Search for eligible studies
• Apply selection criteria
• Assess study quality, extract data
• Analyse results using meta-analysis,
where appropriate and possible
• Perform sensitivity analyses
• Prepare report (keep it up-to-date!)
Acknowledgments:
Staff members of CMPO: Wim Van den Noortgate
Staff members of CEBAM: Bert Aertgeerts, Filip
Cools, Trudy Bekkering
Cochrane and Campbell Colleagues International
Contact:
Tel: 016/326220
ANNEX: Anatomy of a
systematic review
Anatomy of a Systematic Review
• Background/Introduction– Establish need
– (Distinguish from previous review efforts)
– State objectives and review questions
• Methods– Criteria for inclusion and exclusion
• Type of population
• Type of studies
• Type of intervention (+ comparison)
• Type of outcomes
Anatomy of a Systematic Review (cont.)
• Methods
– Locating studies: Consulted data sources (databases, grey literature, reference searches, expert consulting etc.,
– Search strategy (final)
Anatomy of a Systematic Review (cont.)
• Methods
– Proces of selecting
studies
• Include screening
instrument in annex
1.
Published between 2005 (jan) and 2008
(dec)
Already filtered out.
2.
Conducted within health care or a
health care context
Include:
Syntheses of qualitative with quantitative research) by
synthesis methods other than informal review.
Exclude:
Papers commenting on methodological issues but
without including details of the outcomes of the
synthesis.
Papers that do not explicitly describe or name a method
for synthesis.
Reviews on concepts/definitions used within health care
or research issues
3. Published in English language
4. Published in a peer-reviewed journal
Inclusion criteria review on QES in the literature (update Dixon-Woods)
Possible screening criteria
• Timespan
• Language restrictions
• Discipline / Scientific field
• INCLUSION and EXCLUSION
CRITERIA
Consider a
flowchart of
results from
screening
Anatomy of a Systematic Review (cont.)
• Methods:– Data Extraction
• Introduce coding form
• Describe and define coding categories
• Describe process of data extraction „at least two independent reviewers‟
Do not re-invent the fire!
„We used the EPOC guidance on data-extraction (reference)‟…
Anatomy of a Systematic Review (cont.)
• Results– Descriptive results – Inferential results (if applicable)
• Discussion• Conclusions
– Implications for practice– Implications for research
• References • Appendix: search strings, critical appraisal checklist,
list with excluded studies (usually a flow chart), coding/extraction sheets, outcomes of meta-synthesis exercise etc.
• Consider developing a user sheet (short summary avoiding scientific jargon)