Measuring effects and impact: Analysis of practices in HI & partners projects

Impact and evaluation seminar - 2 to 6 December, Phnom Penh

Findings of the meta-analysis

Method and corpus

A meta-evaluation, or meta-analysis of evaluations, is a study and synthesis of several other evaluations based on a set of pre-determined criteria. The aim is to learn lessons about the quality and impact of interventions and obtain the information resources necessary for improving future actions.

18 external evaluations for 2012

Based on an analysis chart, the meta-analysis examined 5 OECD criteria (relevance, effectiveness, efficiency, impact and sustainability), and 5 other criteria (gender, partnership, beneficiary participation, coordination and innovation) corresponding to approaches promoted by HI.

Impact and evaluation seminar - 2 to 6 December, Phnom Penh

How is impact analysed in evaluations?

No specific methodology for measuring impact , but evaluators used participatory tools for measuring effects (e.g.: SWOT);

The evaluations did not allow any real conclusions to be drawn on the long-term impact of the projects, but more on the projects’ direct and immediate effects;

The lack of a baseline/endline was one of the main factors limiting the measurement of project impact;

Limitations of the indicators monitoring tools and process;

Some evaluations considered the positive and negative effects of the projects, as well as expected and unexpected effects.(e.g.: gender approach not taken into account).

Findings of the meta-analysis

Impact and evaluation seminar - 2 to 6 December, Phnom Penh

Outcomes of the analysis of effects from the evaluations:

The evaluation reports highlighted the projects’ effects on improving the lives of beneficiaries and their inclusion; The projects also had a positive effect on the stakeholders’ “sphere” in terms of inclusion and respect for the rights of vulnerable and marginalised people (recognition of rights by local authorities, change in attitudes within communities, etc.); Several evaluations mentioned positive effects that went beyond expected effects (e.g. in the Prison project of Madagascar programme); It is essential for HI to invest resources and introduce processes for measuring the effects of certain activities which are crucial to project strategy, such as the training and capacity-building of the services’ staff • .

Findings of the meta-analysis

Impact and evaluation seminar - 2 to 6 December, Phnom Penh

Measuring effects or impact: methods and tools used within the

projects

Policy makers and duty bearers

Users and/or beneficiaries, communities

Service providers

Tools/user satisfaction surveys, Pre and post-test training, Mapping tools, Mozambique/Kosovo survey…

Participation measurement tool DPOs (Vida Brasil), quality of life measurement tools (e.g. Asia), social participation survey (e.g. Togo), CAP surveys, Pre and Post Impact Assessment (Mines), Case studies…

Rights Monitoring Tools, PRDLI research, …

Impact and evaluation seminar - 2 to 6 December, Phnom Penh

Methods and tools: stakes

A diversity of effects monitoring and measurement tools but a rather irregular implementation → How to share and to better use them?

Few impact surveys (HIB Burundi/Colombia survey, Social participation survey Togo 2007-2009, Filariasis impact survey)→ Develop more surveys, with which objectives and which resources

Methodological limitations within effects and impact measurement: Most often, absence of baselineWhen baseline exists, no endline Irregular participative approach in methods and tools Balance between quantitative and qualitative toolsNeed for reinforcing Monitoring-Evaluation toolsBetter designing indicators of effects and impact data collection

Impact and evaluation seminar - 2 to 6 December, Phnom Penh