What is a Rapid Evidence Assessment? What’s involved?Evidence Base Camp 2013
Levin WhellerPractice Development TeamResearch Analysis and Information Unit
This session
• Evidence reviews:
The what, why and how…
• Evidence reviews in practice:
Examples of recent Rapid Evidence Assessments
Evidence reviews:the what, why and how…
Why review evidence?
“The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”.
Davies, 2003
Methods for reviewing evidence…
Review type Systematic? Timeframe
Literature Review No 1-8 weeks
Scoping Review No 1-8 weeks
Rapid Evidence Assessment
Yes 2-6 months
Systematic Review Yes 8-12+ months
Multi-arm systematic review
Yes 12+ months
Review of Reviews Yes Variable
Literature reviews
Look! The breadcrumbs lead here, this MUST be the answer!
Literature reviews
AHAHAHAHA! I have tricked you into only
reviewing only *some* of the
available evidence!
Are we happy for professionals to only have some of the evidence when making decisions?
We need to look at all the evidence…
• Antman et al, 1992.
• Study comparing recommendations for treating heart attacks based on literature reviews with recommendations based on a systematic meta-analysis.
• Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures.
• In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.
Systematic Reviews
Systematic reviews
Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.
Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of
police.
Policy opinions of systematic reviews
• Of those that had heard of them, some had concerns:
– Timeliness
–Relevance
–Usefulness
• Some had difficulty distinguishing them from literature reviews, even when explained
Campbell S et al (2007) Analysis for policy: evidence-based policy in practice
Rapid Evidence Assessments
• Use systematic principles and the same process as a systematic review…
• …but make compromises given available time and resources
• Pragmatic and transparent approach
The process (in a nutshell)
1. Draft search terms
2. Draft sift criteria
3. Sift received abstracts
4. Request relevant papers
5. Read and ‘grade’ papers
6. Write it up (‘synthesis’)
Some key principles
• Demonstrate consistency in searching/ sifting
• Document search and sift process
– Process should be transparent and repeatable
• Specify required quality of evidence
– Systematic reviews only?
– Pre-post studies only?
– All ‘empirical’ papers?
• Be explicit/ transparent about the limitations of the approach
Some examplesWhat it is like in practice?
One practical example• Review of Police Leadership and
Training commissioned by the Home Secretary
• CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?”
• Three weeks to deliver an evidence review– Not a full REA, but…– More than literature or scoping
reviews– Used systematic principles– Due to time limit, search
restricted to evaluations and systematic reviews only
Searching (Training)
Search terms1. training OR learning OR
development 2. AND evaluat* OR assess*
OR what works OR impact3. AND systematic review
Limitations• Searched 11 databases
and 2 websites • English language only
Initial search identified 1,015 abstracts to sift
Sifting (Training)
• Q1: Is the study about adult training, learning, or development?
• Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review?
• Secondary sift to remove papers related to inappropriate populations and specific medical conditions
Original references
1,015
After first sift
Less duplicates
After second sift
Available from NPL/ BL in time
for inclusion
38
32
22
10
Read, Appraise and Synthesise papers –What works in training?
What works (Good practice)
Strong evidence in healthcare shows training that is integrated into routine practice is more effective at changing individual’s attitudes and behaviour than traditional classroom based approaches.
Strong evidence from education shows that collaborative continuous professional development is effective in improving pupil outcomes (learning and behaviour) and the practice, attitudes and beliefs of teachers
What doesn’t work (Bad practice)
There is systematic review evidence that classroom-based training alone is not necessarily an effective way to improve practitioner’s skills or to change their behaviour
What’s promising (Promising/noteworthy practice)
There is some evidence that simulation-based training may have some advantage over more traditional classroom methods
What’s unknown There is a lack of clear evidence on the effects of reflective practice; portfolio learning; problem based learning; and learning technologies and virtual learning.
Limitations
• Time
– Availability of papers (10/22)
– Unable to pilot search terms
• Available databases
– Only those available to the NPL
• English language only
• Literature focussed in different areas
– Almost nothing on policing
– Papers mostly from healthcare
Organisational change and business improvement
• Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings
• Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc.
• So – are these techniques the answer?
• Is that magic potion? Or is it snake oil?
Rapid Evidence AssessmentStep 1
Systematic search(11,960 abstracts)
Step 2Quality assess & critically appraise
(181 empirical studies)
Step 3Synthesise findings
(41 studies with useful findings)Draw conclusions
Example of the two searches in the organisational change and business improvement REA
Organisational change and business improvement
Question Approach Abstracts identified
Included papers
What works in delivering organisational change?
Rapid review of reviews
797 5
What works in organisational change and business improvement?
REA 11,163 36
Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included…
Why so many papers?• Better search terms needed?• Dodgy descriptions of papers in abstracts?• Problems with searching?
Organisational change – what works?
• Potential success factors for organisational change
Leadership (and having a clear strategy)
Resources (financial, personnel and training)
Organisational culture and structure
Staff feelings of active participation (and related employee empowerment and strong teamwork)
Communication
…and prior experience of implementing a quality improvement programme
Organisational change: potential success factors
Leadership
• stability of supervision throughout implementation • direct support from supervisors - ‘on-the job’ training • staff involvement in decisions • transformational leadership behaviour = reduced employee
cynicism
Engagement
• staff active participation in decision making & ‘room to experiment’
• degree to which staff understood rationale for change• communication found to influence self reports of job
performance
Lessons learned...• Piloting your searches is critical (we’ll do this tomorrow) to
give you an idea of the size of the job, and if there is much available material
• It’s important to focus on the end result and how will the findings be used – make it relevant…
• Recognise (and accept) there are limitations of the approach
• Make sure you are researching the right question
• Follow the key principles:– Demonstrate consistency in searching/ sifting– Document search and sift process– Specify required quality of evidence – Be explicit about the limitations of the approach– REAs should be replicable
The process (for Evidence Base Camp)
Stage Who When
Draft search terms Review teams EBC Day 2
Draft sift criteria Review teams EBC Day 3 (Nov)
Sift received abstracts
Review teams EBC Day 3 (Nov)
Request full papers NPL and researchers
After EBC Day 3 (Nov)
Read and grade full papers
Review teams EBC Day 4/5 (Mar)
Synthesise evidence
Review teams EBC Day 4/5 (Mar)
Terms Results
Anywhere in the article
“Domestic violence” 597,000
“Domestic abuse” 32,500
Title only
“Domestic violence" 23,700
“Domestic abuse” 1,220
Published since 2000 (title only)
“Domestic violence" 13,200
“Domestic abuse” 822
Adding ‘randomised controlled trial’
“Domestic violence" 0
“Domestic violence" 0
Adding ‘experiment’
“Domestic violence" 32
“Domestic abuse” 0
Adding ‘systematic review’
“Domestic violence" 7
“Domestic abuse” 0
• Search terms are hugely important.
• Tomorrow is all about developing search terms and seeing the impact of using different terms.
• We will be running live pilot searches in your groups with College librarians.
Useful links
Civil service REA toolkit:
http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment
Campbell Collaboration (social interventions, e.g. crime and justice)
http://www.campbellcollaboration.org/
Cochrane Collaboration (medical interventions)
http://www.cochrane.org/
EPPI centre (education)
http://eppi.ioe.ac.uk/cms/