day 2, presentation in oslo ‘what have we learnt from the application phase?’ 17 february 2011
DESCRIPTION
Methodological Lessons Joint Evaluation of Conflict Prevention and Peace-Building in DRC, 2002-2010. Day 2, Presentation in Oslo ‘What Have We learnt From the Application Phase?’ 17 February 2011. The General Feedback On the Draft Working Guidance. - PowerPoint PPT PresentationTRANSCRIPT
Methodological Lessons
Joint Evaluation of Conflict Prevention and Peace-Building in
DRC, 2002-2010
Day 2, Presentation in Oslo‘What Have We learnt From the Application Phase?’
17 February 2011
The General Feedback On the Draft Working Guidance
1. Feedback received from the Congo evaluation team, but also from recent work with USAID-OTI, Ausaid, and training given by Channel on evaluation of CPPB.
2. The Guidance is useful introduction, especially for practitioners with limited experience of evaluation. Particularly useful are the overview of key steps and the development of the evaluation questions.
3. Intrigued by Theories of Change and importance of conflict analysis. Find Theories of Change better for project level.
4. However the Guidance does not tackle the “how to” questions: conflict analysis (which, when, how), testing theories of change, baselines.
Channel Research 2
The Evaluative Framework Used for the Congo Evaluation
Effects on the Conflict
1. Evaluation Questions & Criteria
1. Evaluation Questions & Criteria
2. Sample of Projects2. Sample of Projects
3. Conflict Analysis3. Conflict Analysis
4. Research Hypotheses and
Themes
4. Research Hypotheses and
ThemesOutcomes Outcomes
The Evaluative Framework Used for the Congo Evaluation
Effects on the Conflict
1. Evaluation Questions & Criteria
1. Evaluation Questions & Criteria
2. Sample of Projects2. Sample of Projects
3. Conflict Analysis3. Conflict Analysis
4. Research Hypotheses and
Themes
4. Research Hypotheses and
ThemesOutcomes Outcomes
Good to have simple questions but tricky to analyse by “results”.
The Evaluative Framework Used for the Congo Evaluation
Effects on the Conflict
1. Evaluation Questions & Criteria
1. Evaluation Questions & Criteria
2. Sample of Projects2. Sample of Projects
3. Conflict Analysis3. Conflict Analysis
4. Research Hypotheses and
Themes
4. Research Hypotheses and
ThemesOutcomes Outcomes
Issue N° 1:No Theories of ChangeConfused Strategies
Channel Research 6
Reasons for the Weakening of Theories of Change & Strategies as Reference
1. Seizing Opportunities
• Implementation requires a response to constraints and opportunities as they arise. •These cannot be planned for in advance so plans and objectives are not a good evaluation point of reference
1. Seizing Opportunities
• Implementation requires a response to constraints and opportunities as they arise. •These cannot be planned for in advance so plans and objectives are not a good evaluation point of reference
2. Compartmentalisation
• Too many levels and too many instruments to carry out an analysis•Severe lack of documentation, issues of disclosure, many “indirect” effects.
2. Compartmentalisation
• Too many levels and too many instruments to carry out an analysis•Severe lack of documentation, issues of disclosure, many “indirect” effects.
3. Problems of Reconstruction
• Theories of Change conceived as an elicitive tool in planning• Retrospective interpretation opens up real chances of contested findings, with no clear data sets
3. Problems of Reconstruction
• Theories of Change conceived as an elicitive tool in planning• Retrospective interpretation opens up real chances of contested findings, with no clear data sets
4. End State?
• Problems in defining terms such as peace and conflict. •Usually depends on the identification of an end-state. •This may not be explicit or shared by all stakeholders or those within a single programme
4. End State?
• Problems in defining terms such as peace and conflict. •Usually depends on the identification of an end-state. •This may not be explicit or shared by all stakeholders or those within a single programme
The Evaluative Framework Used for the Congo Evaluation
Effects on the Conflict
1. Evaluation Questions & Criteria
1. Evaluation Questions & Criteria
2. Sample of Projects2. Sample of Projects
3. Conflict Analysis3. Conflict Analysis
4. Research Hypotheses and
Themes
4. Research Hypotheses and
ThemesOutcomes Outcomes
Issue N° 2:No Conflict Analyses Used, and No Conflict Sensitivity Analysis
Our understanding of peace depends on where we are
The Evaluative Framework Used for the Congo Evaluation
Effects on the Conflict
1. Evaluation Questions & Criteria
1. Evaluation Questions & Criteria
2. Sample of Projects2. Sample of Projects
3. Conflict Analysis3. Conflict Analysis
4. Research Hypotheses and
Themes
4. Research Hypotheses and
ThemesOutcomes Outcomes
Issue N° 3:Finding the right balance in the sample is not easy
An excessive number of projects.A database selected before the evaluation and a great difficulty in accessing information for these projects.
Consequence:Time spent to obtain information: 80% (to avoid focus on + transparent)Sample may not represent the full range of interventionsStrategic dimensions were lost. Case study approach weakened.
The Question of Sampling
Are We on the Right Track?
Channel Research
Channel Research 12
Evaluation Communities Must Become More Closely Engaged
- -
Desk Bound Donors only
-Theories of
ChangeResults
reporting
0Simple Eval Questions
Good consultation
+
Many field visits
Regular reporting
++
Drivers based
Public workshops
Key factor A
Key factor B
Negative / Positive Change
A Strong Steering Committee willing to arbitrate between approaches and to track the evaluation team in country(ies)
A highly structured approach with detailed phasing and willingness to communicate
Internal & Static Conflict Analysis
Resource Conscious Steering Committee
- - ++
1
5Engagement
Public Uptake
Channel Research 13
Specific Recommendation for the Final Guidance: Recommend Care in 3 Key Stages and Outline Unique Nature of Conflict Interventions
Optimal scenario
Drawing Conclusions
Sequencing
Conflict AnalysisConflict Analysis
Case StudiesCase Studies
Outcome to FactorsOutcome to Factors
Channel Research 14
Specific Recommendation for the Final Guidance: Recommend Care in 3 Key Stages and Outline Unique Nature of Conflict Interventions
ToR Preparation Collection Analysis
Optimal scenario
Drawing Conclusions
Sequencing
Conflict AnalysisConflict Analysis
Case StudiesCase Studies
Outcome to FactorsOutcome to Factors
Conflict analysis must precede sampling
Conflict analysis must precede sampling