what is a rapid evidence assessment? what’s involved? evidence base camp 2013 levin wheller...
TRANSCRIPT
![Page 1: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/1.jpg)
What is a Rapid Evidence Assessment? What’s involved?Evidence Base Camp 2013
Levin WhellerPractice Development TeamResearch Analysis and Information Unit
![Page 2: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/2.jpg)
This session
• Evidence reviews: The what, why and how…
• Evidence reviews in practice:Examples of recent Rapid Evidence Assessments
![Page 3: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/3.jpg)
Evidence reviews:the what, why and how…
Why review evidence?
“The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”.
Davies, 2003
![Page 4: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/4.jpg)
Methods for reviewing evidence…Review type Systematic? Timeframe
Literature Review No 1-8 weeks
Scoping Review No 1-8 weeks
Rapid Evidence Assessment
Yes 2-6 months
Systematic Review Yes 8-12+ months
Multi-arm systematic review
Yes 12+ months
Review of Reviews Yes Variable
![Page 5: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/5.jpg)
Literature reviews
Look! The breadcrumbs lead here, this MUST be the answer!
![Page 6: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/6.jpg)
Literature reviews
AHAHAHAHA! I have tricked you into only
reviewing only *some* of the
available evidence!
![Page 7: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/7.jpg)
Are we happy for professionals to only have some of the evidence when making decisions?
![Page 8: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/8.jpg)
We need to look at all the evidence…• Antman et al, 1992.• Study comparing recommendations for treating heart
attacks based on literature reviews with recommendations based on a systematic meta-analysis.
• Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures.
• In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.
![Page 9: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/9.jpg)
Systematic Reviews
![Page 10: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/10.jpg)
Systematic reviews
Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.
Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.
![Page 11: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/11.jpg)
Policy opinions of systematic reviews
• Of those that had heard of them, some had concerns:
– Timeliness
–Relevance
–Usefulness• Some had difficulty distinguishing them from
literature reviews, even when explained
Campbell S et al (2007) Analysis for policy: evidence-based policy in practice
![Page 12: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/12.jpg)
Rapid Evidence Assessments
• Use systematic principles and the same process as a systematic review…
• …but make compromises given available time and resources
• Pragmatic and transparent approach
![Page 13: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/13.jpg)
The process (in a nutshell)
1. Draft search terms
2. Draft sift criteria
3. Sift received abstracts
4. Request relevant papers
5. Read and ‘grade’ papers
6. Write it up (‘synthesis’)
![Page 14: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/14.jpg)
Some key principles
• Demonstrate consistency in searching/ sifting
• Document search and sift process– Process should be transparent and repeatable
• Specify required quality of evidence – Systematic reviews only?– Pre-post studies only?– All ‘empirical’ papers?
• Be explicit/ transparent about the limitations of the approach
![Page 15: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/15.jpg)
Some examplesWhat it is like in practice?
![Page 16: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/16.jpg)
One practical example• Review of Police Leadership and
Training commissioned by the Home Secretary
• CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?”
• Three weeks to deliver an evidence review– Not a full REA, but…– More than literature or scoping
reviews– Used systematic principles– Due to time limit, search
restricted to evaluations and systematic reviews only
![Page 17: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/17.jpg)
Searching (Training)Search terms1. training OR learning OR development 2. AND evaluat* OR assess* OR what
works OR impact3. AND systematic review
Limitations• Searched 11 databases and 2
websites • English language only System
atic review
Evaluation
Training
Initial search identified 1,015 abstracts to sift
![Page 18: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/18.jpg)
Sifting (Training)• Q1: Is the study about
adult training, learning, or development?
• Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review?
• Secondary sift to remove papers related to inappropriate populations and specific medical conditions
Original references
1,015
After first sift
Less duplicates
After second sift
Available from NPL/ BL in time
for inclusion
38
32
22
10
![Page 19: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/19.jpg)
Read, Appraise and Synthesise papers – What works in training?
What works (Good practice)
Strong evidence in healthcare shows training that is integrated into routine practice is more effective at changing individual’s attitudes and behaviour than traditional classroom based approaches.
Strong evidence from education shows that collaborative continuous professional development is effective in improving pupil outcomes (learning and behaviour) and the practice, attitudes and beliefs of teachers
What doesn’t work (Bad practice)
There is systematic review evidence that classroom-based training alone is not necessarily an effective way to improve practitioner’s skills or to change their behaviour
What’s promising (Promising/noteworthy practice)
There is some evidence that simulation-based training may have some advantage over more traditional classroom methods
What’s unknown There is a lack of clear evidence on the effects of reflective practice; portfolio learning; problem based learning; and learning technologies and virtual learning.
![Page 20: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/20.jpg)
Limitations• Time– Availability of papers (10/22)– Unable to pilot search terms
• Available databases– Only those available to the NPL
• English language only
• Literature focussed in different areas– Almost nothing on policing– Papers mostly from healthcare
![Page 21: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/21.jpg)
Organisational change and business improvement
• Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings
• Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc.
• So – are these techniques the answer?
• Is that magic potion? Or is it snake oil?
![Page 22: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/22.jpg)
Rapid Evidence AssessmentStep 1
Systematic search(11,960 abstracts)
Step 2Quality assess & critically appraise
(181 empirical studies)
Step 3Synthesise findings
(41 studies with useful findings)Draw conclusions
Example of the two searches in the organisational change and business improvement REA
![Page 23: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/23.jpg)
Organisational change and business improvement
Question Approach Abstracts identified
Included papers
What works in delivering organisational change?
Rapid review of reviews
797 5
What works in organisational change and business improvement?
REA 11,163 36
Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included…
Why so many papers?• Better search terms needed?• Dodgy descriptions of papers in abstracts?• Problems with searching?
![Page 24: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/24.jpg)
Organisational change – what works?
• Potential success factors for organisational change
Leadership (and having a clear strategy)
Resources (financial, personnel and training)
Organisational culture and structure
Staff feelings of active participation (and related employee empowerment and strong teamwork)
Communication
…and prior experience of implementing a quality improvement programme
![Page 25: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/25.jpg)
Organisational change: potential success factors Leadership
• stability of supervision throughout implementation • direct support from supervisors - ‘on-the job’ training • staff involvement in decisions • transformational leadership behaviour = reduced employee
cynicism
Engagement
• staff active participation in decision making & ‘room to experiment’
• degree to which staff understood rationale for change• communication found to influence self reports of job
performance
![Page 26: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/26.jpg)
Lessons learned...• Piloting your searches is critical (we’ll do this tomorrow) to give
you an idea of the size of the job, and if there is much available material
• It’s important to focus on the end result and how will the findings be used – make it relevant…
• Recognise (and accept) there are limitations of the approach
• Make sure you are researching the right question
• Follow the key principles:– Demonstrate consistency in searching/ sifting– Document search and sift process– Specify required quality of evidence – Be explicit about the limitations of the approach– REAs should be replicable
![Page 27: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/27.jpg)
The process (for Evidence Base Camp)
Stage Who When
Draft search terms Review teams EBC Day 2
Draft sift criteria Review teams EBC Day 3 (Nov)
Sift received abstracts
Review teams EBC Day 3 (Nov)
Request full papers NPL and researchers
After EBC Day 3 (Nov)
Read and grade full papers
Review teams EBC Day 4/5 (Mar)
Synthesise evidence
Review teams EBC Day 4/5 (Mar)
![Page 28: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/28.jpg)
Terms Results
Anywhere in the article
“Domestic violence” 597,000
“Domestic abuse” 32,500
Title only
“Domestic violence" 23,700
“Domestic abuse” 1,220
Published since 2000 (title only)
“Domestic violence" 13,200
“Domestic abuse” 822
Adding ‘randomised controlled trial’
“Domestic violence" 0
“Domestic violence" 0
Adding ‘experiment’
“Domestic violence" 32
“Domestic abuse” 0
Adding ‘systematic review’
“Domestic violence" 7
“Domestic abuse” 0
• Search terms are hugely important.
• Tomorrow is all about developing search terms and seeing the impact of using different terms.
• We will be running live pilot searches in your groups with College librarians.
![Page 29: What is a Rapid Evidence Assessment? What’s involved? Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information](https://reader036.vdocuments.site/reader036/viewer/2022081421/56649dde5503460f94ad6d24/html5/thumbnails/29.jpg)
Useful links
Civil service REA toolkit:http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment
Campbell Collaboration (social interventions, e.g. crime and justice)
http://www.campbellcollaboration.org/
Cochrane Collaboration (medical interventions)
http://www.cochrane.org/
EPPI centre (education)
http://eppi.ioe.ac.uk/cms/