innovative synthesis methods
DESCRIPTION
Learn key steps to synthesize numerous evaluation studies that have different target groups, interventions, evaluation designs, statistical techniques and geographical locations. Utilizing principles of the realist synthesis method and multiple case studies contributes to innovative alternatives to the meta-analysis technique.TRANSCRIPT
![Page 1: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/1.jpg)
Innovative Synthesis Methodologies
Donna Smith-Moncrieffe Canadian Evaluation Society Conference
Fairmont Royal York, TorontoJune 9-12, 2013
Innovative Synthesis Methodologies
Donna Smith-Moncrieffe Canadian Evaluation Society Conference
Fairmont Royal York, TorontoJune 9-12, 2013
![Page 2: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/2.jpg)
2
Summary of the PresentationSummary of the Presentation
NCPC Mandate
Government Context Evaluation considerations at the corporate level (Performance) Evaluation considerations at the project’s level (Knowledge products)
Overview of 3 Types of synthesis methods Key points Challenges
Case Study (Step by Step Process) Collating data from projects with different target groups, interventions and outcomes Synthesis tables Providing reporting options for summarizing various statistical results: percentages,
Effect Sizes, T tests and F coefficients
Conclusions
![Page 3: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/3.jpg)
3
![Page 4: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/4.jpg)
Government Context: Evaluation Approaches Government Context: Evaluation Approaches
4
![Page 5: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/5.jpg)
Why did we need to Synthesize the Data?Why did we need to Synthesize the Data?
● We needed to respond to the questions about, “what works in crime prevention”
● We needed to provide findings and conclusive statements about 13 programs (total of 36 project sites) with different:
Target groups;
Interventions;
Outcomes; and
Evaluation designs
● Single site repeated design
● Quasi-experiment design (sometimes with a matched comparison group)
5
![Page 6: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/6.jpg)
Types of Synthesis MethodsTypes of Synthesis Methods
6
![Page 7: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/7.jpg)
Synthesis Method Requirements for quality and RigorSynthesis Method Requirements for quality and Rigor
7
![Page 8: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/8.jpg)
What approach did we use to Synthesize 13 Evaluations?
What approach did we use to Synthesize 13 Evaluations?
8
![Page 9: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/9.jpg)
Step 1: Identify Key QuestionsStep 1: Identify Key Questions
What behaviour related changes were made (i.e. changes in violent offending, police contact and victimization?
What percentage of projects made favorable changes in relation to the criminal justice system (i.e. reduced violent offending, police contacts etc..)?
How much change was made?
What contextual and program related factors contributed to these changes (i.e. geographical location, partnerships, resources etc..)?
9
![Page 10: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/10.jpg)
Step 2: What are the inclusion criteria?Step 2: What are the inclusion criteria?
● In the Government context, we need to use as many evaluation studies as possible in the synthesis; even when they do not meet rigid evaluation standards.
● Evaluation studies that met the following minimum requirements were included in the synthesis:
Study uses quantitative data (descriptive or inferential statistics);
Measures that correspond with key crime related outcomes (i.e. association with anti-social peers, emotional regulation, substance abuse, offending, education, employment etc..);
Study used at least the basic evaluation design (single group pre-post test repeated measures design); and
At least pre and post test measures had been available
10
![Page 11: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/11.jpg)
Step 3: Enter the Data (Partial Sample)Step 3: Enter the Data (Partial Sample)
Name of Study
Description of Variable
Sample Size
Result P Level Type of Change
Level of Study
Project Intervention Toronto
Arrests E=76C=43
F=0.291 P=.823 No Change
Exemplary
Multi-Systemic Therapy
Youth not arrested for an offence
E=28C=12
E=89.3%C=66.7%
N/A Favorable Basic
Towards No Drugs
Weapon Carrying
E=847C=54
F=0.63 P<0.01 Favorable Notable
Velocity Police contacts
E=87 69%(reduction)
N/A Favorable Notable
Interpretation of Results
11
![Page 12: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/12.jpg)
Step 4: Analysis and InterpretationStep 4: Analysis and Interpretation
Synthesis for 9 evaluation studies measuring changes in criminal behaviours
Overall interpretation
Background Information•Number of evaluation studies: 9•Number of sites: 11•Common measures include: 1) Arrests; 2) Non-Violent Offending; 3) Violent offending; 4) Criminal Victimization; and 5)Weapon Carrying
Quantitative results•% of change by sub-measure: 50%•% of projects demonstrating favorable change (showing at least one stat. significant result at the project level: 75%•% of projects demonstrating unfavorable change:0%•% of projects demonstrating no change: 25%
Qualitative results•Provide qualitative results by model, site or project level to qualify the quantitative findings 12
![Page 13: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/13.jpg)
Step 5 : Reporting Aggregate DataStep 5 : Reporting Aggregate Data
13
![Page 14: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/14.jpg)
Step 5: Reporting Aggregate Data (Sample of Approach # 1)Step 5: Reporting Aggregate Data (Sample of Approach # 1)
14
![Page 15: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/15.jpg)
Step 5: Reporting (Sample of Approach #2 reporting on magnitude of change ) Step 5: Reporting (Sample of Approach #2 reporting on magnitude of change )
15
![Page 16: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/16.jpg)
ConclusionsConclusions
● What synthesis method works best when the evaluation studies have a variety of interventions, outcomes and statistical measures?
Elements of the Realist Synthesis review and Multiple-case Study Synthesis are recommended; and
NCPC utilized elements of both of these synthesis methods but could benefit from responding to the Why questions incorporated in the Multiple- Case Study Synthesis
● What reporting method works best to summarize non-standard data?
Approach #1 and # 2 are both useful; and
Assess your audience to determine the best reporting method
16
![Page 17: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/17.jpg)
Conclusions cont’dConclusions cont’d
● What planning activities would be required to utilize both Realist and Multiple Case study Methods?
Develop strategic direction (strategic plan to help focus the direction of the evaluation);
Prioritize key areas to minimize complexities;
Ensure that the information management system is designed to:
● Clarify the unit of analysis (by model, by project, evaluation questions, etc..);
● Respond to How and Why questions such as: How does the program produce these effects or How does the intervention contribute to change?; and
● Collect data about the context for each model, site or project.
17
![Page 18: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/18.jpg)
References for Recommended Synthesis MethodsReferences for Recommended Synthesis Methods
18
![Page 19: Innovative Synthesis Methods](https://reader033.vdocuments.site/reader033/viewer/2022061120/546bf738af79598a798b559f/html5/thumbnails/19.jpg)
19
Thank YouThank You
Contact:
Donna Smith-Moncrieffe, BSc. Crim Dip, MSc.
Senior Evaluation Advisor
Public Safety Canada, National Crime Prevention Centre
Policy, Research and Evaluation
Tel: 416-952-0423
Fax: 416-952-0483
Email: [email protected]
Web : www.PublicSafety.gc.ca | www.SecuritePublique.gc.ca