Annexe 1. TERMS OF REFERENCE for the External Evaluation of ACF’s Nutrition Programme

1.1. Key Evaluation Dates

Expected Start Date: 11th June 2012 End Date: 16th July 2012 Submission of Draft Report

7th July 2012 Submission of Final Report

16th July 2012

1.2. Language of the Evaluation

Language Requirements for the Evaluation:

English Language of the Report: English

1.3. Workplan & Timetable

Activities Workin

g

Days Briefing HQ (teleconference) 1 Travel to mission 1 Briefing mission, review of documents, preparation of fieldwork + evaluation framework.

Detailed methodology to be proposed to ACF prior to commencing fieldwork.

3

Field work 13 Collection of secondary information in capital 1 Data analysis and preparation of the draft report, common editing of the report with ACF

Pakistan

5

Debriefing and presentation of preliminary findings in-country on the basis of the draft

report

1

Travel back from mission 1 HQ debriefing with desk officer 1 Finalization of the report on the basis of Field, HQ and ACF-UK Feedback 3

Total 30

1.4. Budget Details/Conditions Payment will be done on the basis of the above mentioned time table, daily fees shall be

negotiated.

20% of the fees will be paid upon signature of contract, 40% upon reception of draft report and

60% will be paid after validation of the final report by ACF-UK. Travel, accommodation (guest house for in-country nights), food, will be provided at ACF

guesthouses at field level. The application of the visa is the responsibility of the consultant however these costs will be

reimbursed in full upon receipt of the expenses claim form and supporting receipts. Evaluation costs (in country transport, evaluation team, translator etc.) will be covered by ACF,

but the evaluator may be required to absorb the cost initially. Insurance costs will not be covered, and the evaluator shall manage their own insurance, and

provide the details of this cover to ACF-UK before departure.

2. DETAILS OF THE PROGRAMME

Name of the Programme: Emergency Nutrition intervention in Sindh Location:

Thatta and Badin districts, Sindh province,

Pakistan Nutrition interventions implemented under two different

programs: -Pakistan Emergency Food Security Alliance (PEFSA2):

Emergency food security and nutrition support to flood affected

populations in Pakistan (emergency response to 2010 floods) - Emergency nutrition, food security and livelihood support to

flood affected populations in Pakistan (emergency response to

2011 Monsoon)

ECHO/PAK/BUD/2011/91009

Start Date: 01/07/2011 End Date: 31/05/2012 ECHO/PAK/BUD/2011/91025

Start Date: 01/11/2011 End Date: 15/06/2011

2.1. Map of Programme Thatta and Badin districts:

-PEFSA 2: Thatta district, Union Councils (UCs): Bano, Bachal Gugo, Kinjhar, Goongani and

Ladium -Monsoon: Thatta district, UCs: Liakpur, Jokh Sharif, Mehar Shah and DK Suho; and Badin

district, UCs: Shaheed Fazal, Kario, Rahuki and Kadi Kazia

2.2. Programme Overview

ACF in Pakistan: A Brief History

ACF's first intervention in Pakistan was in 1979 in the context of the Afghan crisis and the

arrival of thousands of Afghan refugees. Since then, ACF has been active in responding

to the needs of Pakistani and Afghan people caught in disaster and conflict in Pakistan.

More recently, in October of 2005, ACF intervened in response to the massive

earthquake that struck Northern Pakistan, beginning operations only 24 hours after. Able

to draw on its expertise in the country and the region (with programs having been

implemented in Afghanistan, Nepal, and Tajikistan), ACF was able to provide effective

relief to communities through distributions of food aid, water, sanitation services, and food

security and nutrition programs in areas such as Jared, Kaghan Valley and Rashang,

Allai Valley, Balakot and Battagram. Following the floods caused by a cyclone in 2007,

ACF implemented relief operations in Dadu and Kamber Shahdadkot districts of

Sindh Province, assisting 36,000 flood victims with latrines, water trucking, drilling of wells,

hygiene promotion, and hygiene kits. Also in 2007, ACF conducted Food Security and

Nutrition surveys in these areas as well as Rawalpindi city, to document humanitarian needs

among the populations.

Since 2009, ACF has been implementing an integrated (FSL/WASH) project in Sindh

Province, Thatta District, and in 2010 an ECHO-funded integrated (WASH/FSL) operation

in KPK province so as to begin targeting the underlying causes of malnutrition. When

the 2010 flooding occurred, ACF responded in Thatta District with major WASH and

FSL responses, covering 400,000 people. ACF current strategy is to maximize the impact

of our presence by targeting those areas within our current operational areas. ACF will

respond to immediate needs but also recognizes the need to mitigate and minimize the

negative impact of recurrent floods. Hence ACF will also focus on activities that will

ensure protection of ongoing livelihood recovery. This experience has given ACF a detailed

institutional knowledge of the area.

The Present Nutrition Situation

In Sindh Province, child malnutrition rates have remained persistently high for a number of

years. National nutrition surveys reported the provincial prevalence of wasting to be equal

to 21.3% in 2001-02, and to be equal to 17.5% in 20111. Another nutrition survey2

conducted six months after the 2010 monsoon floods hit revealed critical levels of

malnutrition as well, showing i) Severe Acute Malnutrition (SAM) rates of 6.1% and 2.9%

in Northern and Southern Sindh respectively, and ii) a Global Acute Malnutrition (GAM)

rate of 23.1% in Northern Sindh and of 21.2% in Southern Sindh.

It has been more than a decade since malnutrition rates have been well above the World

Health Organization's (WHO) emergency threshold level (GAM>15%). At district level,

in 2010, Mirpur Bathoro Taluka in Thatta district was highly affected by the floods. In the

four most affected Union Councils of Mirpur Bathoro, ACF assessed the nutritional

status of 6-59 months children and confirmed the severity of the situation with above

emergency threshold level of GAM (GAM=19.5% and SAM=2.4%, based on weight-for-

height z-scores, and using WHO 2006 standards)3, and the

1National Nutrition Survey, Aga Khan University, Pakistan, Sep 2011

2Flood Affected Nutrition Survey (FANS) Dept of health Gov of Sindh Province, Unicef and ACF-Canada

3ACF Integrated Survey, Mirpur Bathoro, Thatta, Pakistan, December 2010

presence of aggravating factors such as high disease burden and poor immunization

coverage. One of the recommendations made was for ACF to start interventions in order

to treat malnourished children and to prevent those who were at risk of becoming

malnourished.

ACF Nutrition Programs

NUM CODE DONOR DEPARTMENT PROVINCE DISTRICT STARTING

DATE ENDING

DATE Status BUDGET COMMENTS

3 PKA1D ECHO-

PEFSA II FSL/NUT SINDH THATTA 1/7/2011 30/03/2012 2,000,000

EURO 30/5/2012NCE

4 PKA1F ECHO-

Monsoons FSL/NUT SINDH BADIN/THATTA 1/11/2011 1/4/2012 850,000EURO 15/6/2012NCE

The direct beneficiaries for the above ECHO-funded programs were flood-affected

households that were rendered acutely food insecure having lost access to food and income

and were still in need for external support. With a multi sectoral approach, the projects

aimed i) to assist the targeted households in regaining their capacity to generate income

and/or produce food while at the same time covering their immediate food needs; and ii) to

reduce mortality and morbidity by improving the general health and nutritional status of

children under five (U5) and Pregnant and Lactating Women (PLW) through a

Community-based Management of Acute Malnutrition (CMAM) approach.

Coordination

ACF coordinates with the Early Recovery Working Groups, individual UN Agencies

and other humanitarian organizations at district, provincial, and national levels in Pakistan

in order to avoidduplication and direct as much aid as possible to those in need.

2.3. General Objective

The humanitarian situation of vulnerable people affected by the food crisis in Southern Sindh

is improved.

2.4. Programme Activities

ACF Nutrition interventions included all components of the Community-based

Management of Acute Malnutrition (CMAM): identification (screening) of under 5

children and pregnant and lactating women (PLW), referral and treatment of those

suffering from acute malnutrition in the appropriate program component (supplementary

feeding programme, outpatient treatment programme or stabilisation centre).

Awareness raising sessions (Hygiene, Health, Breastfeeding and Nutrition best practices)

were conducted to raise awareness among beneficiaries and their caretakers. The programs

also aimed at training i) local health workers on the management of acute malnutrition,

and ii) community volunteers and outreach workers on screening and referral of

malnourished cases to the adequate program sites.ACF established decentralized Out-patient

Therapeutic Programs (OTPs, for the treatment of SAMcases) and Supplementary Feeding

Programs (SFPs, for the management of MAM cases) sites for

the delivery of CMAM services and conducted CMAM trainings to build the capacity of

the health workers and worked to collaborate with the Executive District Offices

(Health), District Coordination Office and other relevant government ministries and

departments to ensure that MoH will in the future handle cases of acute malnutrition in

non emergency situations at localhealth centers and basic health units using the national

CMAM guidelines.

3. AIM OF THE EVALUATION

3.1. Target User(s) of the Evaluation

ACF ELA Unit Implementing HQ Pakistan Nutrition Advisor, Desk Officer Field Level Nutrition team, Country Director, Deputy Country Director Other -

3.2. Overall Objective of the Evaluation

To evaluate the implementation of the ACF ECHO funded Nutrition programs in Thatta and

Badin to assess alternative/improved modes of operation and provide solid recommendations

for the future of the programme.

3.3. Specific Objectives of the Evaluation

1. Assess the capacity building of MoH/PPHI and provide recommendations for

future programming.

2. Assess the current approach to community mobilization using volunteers and

outreach workers such as Lady Health Workers (LHWs), and provide

recommendations for future programming.

3. Assess current community sensitization activities and provide recommendations for

future programming.

4. Assess the extent of the integration of Nutrition with Food Security& Livelihood

(FSL)

activities

5. Analyze implementation strategy and efficiency of intervention and identify best

practices

(activities within the context)

6. Assess relevance and methodology used for targeting and reaching of beneficiaries

on the field

3.4. Scope of the Evaluation

The evaluator should answer the following questions with the aim of making

specific recommendations on how ACF can improve the nutrition programme in the future.

Relevance/Appropriateness

How adequately did ACF assess the needs of the population?

Assess the appropriateness of the objectives and results pursued by the program in

relation

to the identified needs.

How appropriate is the use of community volunteers for case-finding?

Assess its effectiveness and how the community receive this approach.

Is ACF’s current strategy of integration of CMAM services in the MoH appropriate?

Assess the relevance of the chosen partnership with the MoH, integration of

CMAM vs. direct implementation and the relevance of an increased advocacy push.

Coherence

How coherent are ACF’s activities with national policy, ACF country strategy

and otherhumanitarian actors?

Did ACF adequately involve the community in each stage of the programme cycle?

(Needs assessment, programme design and implementation).

Sustainability

Were communities and authorities sufficiently informed of changes in programme

design? How have the current partnerships been developed with UNICEF, WFP and

how can it be improved for the coming period

Assess the level of local-ownership and financial sustainability of the programme.

Did the project decrease the vulnerabilities of the targeted population? Any

linking with recovery (LRRD)? Coverage

How appropriate was the coverage (selection of geographical area and target

group selection) of the CMAM programme?

In UCs where both Nutrition and FSL sectors operate, what was the extent

of the integration of activities and/or of the collaboration between ACF Nut

and FSL

departments?

Impact

How well were nutrition activities integrated with ACF’s WaSH and FSL activities?

To what extend has ACF taken into account principles of ‘Do-no-Harm’ and

mitigated

potential negative environmental impacts?

Effectiveness

Assess the effectiveness of ACF’s ability to deal with foreseen and unforeseen

challenges in the programmes

Sensitization: How effective were the sensitization and awareness sessions given

to the community?

Screening and Case-Finding: How effective was the training in screening methods

for each chosen group (LHWs, community volunteers etc.)?

Efficiency

How could the current monitoring system be strengthened in order to measure

impact more effectively in the future programming?

What level of emergency preparedness (scale-up capacity) is/should be considered

within the programme?

To what extend could program cost have been reduced/made more efficient

without sacrificing the quality of the result?

Cross cutting issues

What measures have ACF put in place in order to ensure gender equality and

sensitivity to other gender issues throughout the programme cycle?

What measures have ACF taken to ensure the security of its employees and other

stakeholders (including beneficiaries’ and non-beneficiaries) in the programme?

3.4. Evaluation Criteria ACF subscribes to the Development Assistance Committee (DAC) criteria for evaluation:

Impact, Sustainability, Coherence, Coverage, Relevance / Appropriateness, Effectiveness and

Efficiency. ACF also promotes systematic analysis of the monitoring system and cross cutting

issues (gender, HIV/AIDS etc). All external evaluations are expected to use DAC criteria in data

analysis and reporting. In particular, the evaluation must complete the following table and include

it as part of the final report. The evaluator will be expected to use the following table to rank the performance of the

overall intervention using the DAC criteria. The table should be included as an Annex to the

report.

Criteria Rating

(1 low, 5 high) Rationale

1 2 3 4 5 Impact Sustainability Coherence Coverage Relevance/Appropriateness Effectiveness Efficiency

3.5. Best Practices The evaluation is expected to provide one key example of Best Practice from the

project/programme. This example should relate to the technical area of intervention, either in

terms of processes or systems, and should be potentially applicable to other contexts where

ACFIN operates. This example of Best Practice should be presented in the Executive Summary

and/or the Main Body of the report.

3.6. Evaluation Outputs The result of this evaluation should be presented in a written report and through

several oral presentations:

One on the mission (to Head of Mission and relevant technical

staff) One at HQ (in person or via teleconference).

3.7. Methodology

3.7.1. Preparation

Review of project documents (proposal, logical framework, donor reports, activity and

monitoring reports, assessment reports, capitalisation documents, budget follow-up, etc.)

3.7.2. Briefing Prior to the evaluation taking place, the evaluator is expected to attend a briefing at HQ level,

and at field level with the Country Director and/or the relevant technical focal point. Briefings

by telephone must be agreed in advance. 3.7.3. Field activities

Consultants are expected to collect an appropriate range of data. This includes (but not

limited to):

Direct information: Primary data collection using standard and participatory

evaluation methods (e.g. semi structured interviews, FGD and observation)

Indirect information: Interviews with local representatives; interviews with project

staff expatriate and national staff; meeting with local and provincial authorities,

groups of beneficiaries, humanitarian agencies, donor representatives and other

stakeholders. For indirect data collection, standard and participatory evaluation methods

are expected to be used (HH interviews and FGDs with beneficiaries, non-beneficiaries,

key informants – health workers and leaders).

Secondary information analysis: including analysis of project monitoring data or of any

other relevant statistical data related to the nutrition implementation. 3.7.4. Report The report shall follow the following format.

Cover Page

Table of Contents

Executive Summary: must be a standalone summary, describing the programme, main

findings of the evaluation, and conclusions and recommendations. This will be no more

than 2 pages in length.

Main Body: The main body of the report shall elaborate the points listed in the

Executive Summary. It will include references to the methodology used for the evaluation

and the context of the action. In particular, for ACF key conclusion there

should be a corresponding recommendation. Recommendations should be as

realistic, operational and pragmatic as possible; that is, they should take careful

account of the circumstances currently prevailing in the context of the action, and of the

resources available to implement it both locally and in the commissioning HQ. Annexes:

Listed and correctly numbered. Format for the main body of the report is:

o Background Information o Methodology o Findings & Discussions o Conclusions Recommendations o Annex I (Best Practice) o Annex II (DAC-based Rating Table)

The report should be submitted in the language specified in the ToR. The report should not be

longer than 30 pages excluding annexes. The draft report should be submitted no later than 10

calendar days after departure from the field. The final report will be submitted no later than

the end date of the consultancy contract. Annexes to the report will be accepted in the working

language of the country and programme subject to the evaluation.

3.7.5. Debriefing & Learning Workshop The evaluator should facilitate a learning

workshop:

To present the draft report and the findings of the evaluation to the ACF Pakistan Team and other stakeholders.

To gather feedback on the findings and build consensus on recommendations.

To develop action-oriented workshop statements on lessons learned and proposed

improvements for the future. 3.7.6. Debriefing with ACF HQ

The evaluator should provide a debriefing with ACF HQ in New York on her/his draft report,

and on the main findings, conclusions and recommendations of the evaluation. Relevant

comments should be incorporated in the final report.

4. RESOURCES AND DOCUMENTATION

The following documentation will be made

available: Project proposals (ECHO; PEFSA,

Monsoon) Budgets

Quarterly reports

5. PROFILE OF THE EVALUATOR

Significant experience in the design and implementation of community-based nutrition programmes,

Experience in the evaluation of CMAM programmes,

Experience with nutrition programmes in Pakistan

desirable,

Significant field experience in the evaluation of humanitarian / development

projects, Relevant degree / equivalent experience related to the evaluation to be

undertake, Good communications skills and experience of workshop facilitation,

Ability to write clear and useful reports (will be required to produce examples of previous

work) Fluent in English,

A strong understanding of donor requirements,

Ability to manage the available time and resources and to work to tight

deadlines, Independence from the parties involved.

6. RIGHTS

The ownership of the draft and final documentation belong to the agency and the funding

donor exclusively. The document, or publication related to it, will not be shared with

anybody except ACF before the delivery by ACF of the final document to the donor. ACF is to be the main addressee of the evaluation and its results might impact on both

operational and technical strategies. This being said, ACF is likely to share the results

of the evaluation with the following groups:

Donor

Governmental partners

Various co-ordination bodies

Intellectual Property Rights

All documentation related to the Assignment (whether or not in the course of the

evaluator’s duties)

shall remain the sole and exclusive property of the Charity.

Annexe 2. Chronogram

Date Day Activities

20.6.12 Wednesday Preparation and Document Review + Briefing HQ

21.6.12 Thursday Flight Madrid-Islamabad

22.6.12 Friday Briefing in Islamabad. Evaluation framework

23.6.12 Saturday Review of documents

24.6.12 Sunday Preparation of fieldwork. Detailed methodology

25.6.12 Monday Discussions with nutrition coordinator, WASH and FSL

26.6.12 Tuesday Travel to the field. Briefings in Thatta base with programs coord.

27.6.12 Wednesday Travel to TMK to meet nutrition Teams. Worked local data bases

28.6.12 Thursday Interviews and FGD with beneficiaries in Badin.

29.6.12 Friday Interviews and FGD with beneficiaries in Badin

30.6.12 Saturday Interviews and FGD with beneficiaries in Thatta

1.7.12 Sunday Resting day

2.7.12 Monday Interviews and FGD with beneficiaries in Thatta district

3.7.12 Tuesday Interviews and FGD with beneficiaries in Thatta District

4.7.12 Wednesday Meetings with EDO, PPHI, Focal Nutrition and visit to SC

5.7.12 Thursday Interviews and FGD with beneficiaries in Thatta district, meetings

with LHW and MERLIN

6.7.12 Friday Failed to meet UNICEF and WFP in Karachi, Phone interviews

7.7.12 Saturday Flight to Islamabad

8.7.12 Sunday Data analysis and preparation of the draft report

9.7.12 Monday Collection of secondary information in ACF base

10.7.12 Tuesday Data analysis and preparation of the draft report

11.7.12 Wednesday Discussions with Nut Coordinator, Consultant for surveys & FSL

12.7.12 Thursday Meetings with UNICEF, OXFAM, MERLIN and SC in Islamabad

13.7.12 Friday Debriefing and presentation of preliminary findings

14.7.12 Saturday Report writing.

15.7.12 Sunday Sending draft report

16.7.12 Monday Flight Islamabad Madrid

Annex 3 Selected UC and villages

District Unit Council Village Comments

Badin Dubi Saeed Khan Chandio

Kario Ganhwar Not feasible due to time

constraints

Karyio Sohrab Khan

Nizamani Aprox 2000 inhabitants

Thatta Bachal Gogo Mohad Khan Samoo

Khamoon Malha

Jock Sharif Abdula Palijo Not feasible due to demonstration

on the road

Atal Sha

Mehar Shah Wali Sha

Qadir Dino

Kiinjhar Rahib Amro

Nodo Bara

Darro Darro

Leemo Malha

Annex 4 List of persons met for interviews

Place Name Organization Position

Madrid-NY

via skype

Charmaine Brett ACF Desk Officer

Cecile Basquin ACF Nutrition Advisor

Islamabad Shahid Fazal ACF Nutrition Coordinator

Paola Maria Valdettaro ACF FSL Coordinator

Eric ACF WASH Coord

Dewi Dwiyanti ACF Admin Coord

Onno Van Mannen ACF Country director

Sarma Mazish OXFAM PEFSA Coord

Alison Donnelly SC Nut Advisor

Silvia Koffman UNICEF Nut Chief

Dr Ijaz Habib MERLIN Nut Coord

Thatta John Batley ACF Field Coord

Jackeline ACF Nut PM

Milton Zhakata ACF FSL PM

Fayaz Ahemd CNV

Razia beneficiary

Salma beneficiary

Momal beneficiary

Asilam Khan beneficiary

Akhter Ali ACF FSL MEAL

Dr Arhum MD Dahro

Abdul Wahid CNV

Ghulam Haibar beneficiary FSL CFW

Gulzar beneficiary Grant

Ahmed beneficiary FSL CFW

Mohamed Haseem beneficiary FSL CFW

Suleiman beneficiary OTP

Ali Mohamed beneficiary FSL CFW

Karan Bux beneficiary OTP

Mohamed Mussa beneficiary SFP

Tallat beneficiary PLW

Mohamen Mussa beneficiary FSL CFW

Khalis beneficiary OTP

Ali Khan beneficiary Grant

Dr Nider Ahmed Mimon MD Nodo Baro

Dr Khalid EDO Nut focal point

Dr Ifthikar EDO Exec. District Off. Heal.

Manzo MERLIN

Dr Paras MD SC

Maqsood PPHI Distric Support Manag.

Jawaid Nurse SC

Zahida Nurse MERLIn

Dr Makbull MD LHW

Telephone Zacharias UNICEF Nut Specialist

TMK Bheru Lal ACF Depute Nut PM

Mohamad Ali ACF Nut data analyst

Mohammend Khan ACF Nut PM in Dadu

Abdul Shakar ACF Depute FSL PM

Badin Parez Ali CNV

M. Jumani CNV

M. Harif CNV

M Khan CNV

M Younis beneficiary

M Aslam beneficiary

Ali Imlan beneficiary

Rashid beneficiary

Abdul Hameed beneficiary

Musthaqua beneficiary

M Hasan beneficiary

Abdul Istif beneficiary

Sarfaz Khan Abro ACF Nut Team Supervisor

Misbrah Qambrani ACF Nurse

Salma Nizamani CNV Female teacher

Mehar-ul-Niza CNV

Bilgees Nizamani CNV

Sajan Das ACF Screener

Names of participant in Focus Group are not listed since it was considered not relevant

Annexe 5. qualitative data collection

Interviews to key informants Name; Position; Organisation;

Context 1. At national, health facility and donors level: current level of integration in

relation to nutrition policies/strategies within Ministries and related departments

Design 1. Main processes in relation to needs assessment and proposal design

2. In which way was the project strategy relevant and appropriate for the context

and for the target beneficiaries?

3. What could have been improved during the design phase?

Implementation Results achieved 1. What were the most effective activities of the Project?

2. What activities could have been improved?

3. Were there any delays in the delivery of project activities? If yes, why?

Staffing 1. Was the ACF staffing structure (e.g. number of staff, location, capacity, skills,

qualifications, management support) appropriate for building capacity of partners

and achieve Project implementation?

2.CNV are doing screening? defaulters tracing? follow ups?

M&E 2. Were appropriate indicators identified to measure outcomes and outputs of the

Project?

3. How was the Project monitoring conducted?

4. Were appropriate project revisions or timely actions taken based on the

monitoring system? How? (Examples)

Impact 1. In your opinion, in which way did the Project benefit to community volunteers,

communities, women and children and health system as a whole

2. How are the trained health workers participating in the project?

Collaboration &

Partnerships

1. What were the main partnerships challenges?

2. In which ways did the partnership strengthen the capacity of MoH and local

partners?

3. What recommendations will you have for future partnership models for Projects

the same or similar to this one?

Sustainability 1. What were the main processes for exit?

2. What results and activities appear to have a significant chance of continuing

now that the Project is completed?

3. Does it appear that local partners have the capacity to continue without funding

from ACF? If yes, how?

4. If no, why and what could have been done to ensure continuity?

Best Practices 1. Can you give an example of best practice from this Project?

Lessons Learned 1. Can you give an example of the main lesson learned in this Project?

Group Discussion: Community Leaders & Community Members

Permission and Information Greetings. Introduce evaluator and translator names. We are doing an evaluation of ACF

project and we really appreciate your participation. We want to ask about. This information

will help ACF and implementing partners to assess whether the Project was successful. Usually

this discussion takes time about 30 minutes. Any information that you give will be kept

confidential. This is a voluntary participation and you can freely decide not to answer. However,

we would appreciate your input. Do you have any questions about the discussion?

Topics to explore: Understanding of Project content

Level of participation (how; when)

Perceived benefit of activity (training; follow up; referral; counselling)

Most difficult or challenging activity (training; follow up; referral; counselling; collaboration

with health services)

Main perceived improvements or benefits (individual level, community level) – as a result of

this Project

Gender issues among CNV

Constraints and/or barriers to improvements in this particular community

Recommendations and/or suggestions for future ACF involvement

Group discussion: Community Volunteers

Topics to explore: Understanding of Project content

Organisation of training

Most interesting component/activity (training; follow up; referral; counselling)

Coverage, level od admissions among those sent to OTP

Most difficult or challenging component/activity (training; follow up; referral; counselling;

collaboration with health services)

Elements that receive particular emphasis during a follow up, home visit or counselling

session

Main perceived improvements or benefits (individual level, community level, health system

level) – as a result of this Project

Constraints and/or barriers faced during implementation of outreach activities

Exit plan, activities ongoing after leaving.

Activities that are/are not likely to continue after the Project ends. Why?

Recommendations and/or suggestions for future ACF involvement

Annexe 6 evaluation matrix

Evaluation

criteria

Key questions Hypothesis Sources of Information

Relevance/

appropriat

eness

How Adequately Did Acf Assess The Needs Of

The Population? appropriateness of the

objectives and results

How appropriate is the use of community

volunteers for case-finding ?

The design, scale and scope of the

Project were in line with the needs of

stakeholders & beneficiaries

CNV network is not well

implemented and they are not main

source of admissions

Proposal

Assessments

National surveys

Evaluation reports

Key Informant interviews

+

CNV FGD

Coherence How coherent are ACF’s activities with national

policy, ACF country strategy and other

humanitarian actors?

Did ACF adequately involve the community in

each stage of the programme cycle?

The Project was coherent and

complementary to actions of key

stakeholders and other ACF

interventions

Yes, ACF involve all the main actors

of the community

Key informant interviews

Direct observation

Narrative reports

Sustainabil

ity how can the partnerships been developed with

UNICEF, WFP be improved?

Assess the level of local-ownership and

financial sustainability of the programme

Did the project decrease the vulnerabilities of

the targeted population?

We doubt that the Project has

promoted the integration of

community health activities into the

health system

Yes the Project has decrease it

Key informant interviews

Focus Group Discussions

Narrative reports

Direct observation

Coverage How appropriate was the coverage of the

CMAM programme

The Project has reached all the

people equally within the designated

areas

Proposal

Assessments

National surveys

Evaluation reports

Key Informant interviews

Impact Integration with ACF’s WaSH and FSL

activities

Has ACF taken into account principles of ‘Do-

no-Harm’ and mitigated potential negative

environmental impacts?

Due to major constraints integration

is been difficult and limited

The Project produced long-term

benefits for the communities, the

final beneficiaries, the local

partners ... in accordance with ACF

principles

Survey reports

Key Informant Interviews

Focus Group Discussions

Direct observations

Effectiven

ness ACF’s ability to deal with foreseen and

unforeseen challenges

How effective were the sensitization and

awareness sessions given to the community

effective was the training in screening methods

for each chosen group (LHWs, community

volunteers etc.)?

The program has adapted to

challenges.

Identification of malnutrition and

knowledge of nutrition have

improved

They were reasonably satisfied with

the assistance that they got from the

Project.

M&E framework

Annual Reports

Survey reports

Focus Group Discussions

Key informant interview

Direct observations

Efficiency How could the current monitoring system be

strengthened in order to measure impact more

effectively?

What level of emergency preparedness should

be considered ?

To what extend could program cost have been

reduced?

Several indicators can be added to

the Information collected

ACF should be prepared for forssen

emergencies

The use of financial resources can be

optimized

Organogram

Job description

Key informant interviews

Budget

Financial reports/audits

M&E plan

Narrative reports

Cross

Cutting

issues

gender equality and sensitivity to other gender

issues

security of its employees and other stakeholders

Gender sensitivity should and can be

reinforced

ACF took effective steps to take

account of security

Project proposal

Key informant interviews

Focus group discussions

Direct observation

Annex 7. DAC-rating for project evaluation

Criteria Rating Rationale

1 2 3 4 5 Relevance X Good

Good assessment of needs from different point of views

Consultation with government, beneficiaries and NGOs

Good choice of activities

Insufficient detailed exit plan

Objectives too focused on CMAM and emergency. Not

addressing underlying causes

Coherence X Good

National protocol has been followed

good level of dialogue and communication with

government

Involvement of the community

Coverage X Moderate

Geographical distribution not well done

Mobile OTPS have greatly increased coverage and

reduce defaulters

Training coverage failed among health workers.

Beneficiary sessions too crowded. No refreshing

workshop for CNVs

No good integration with FSL

Sustainability X Very Poor

Only SC is operating

Exit plan is not being implemented

Health staff not trained. LHW not partcipated

Partnership with PPHI was not created

CNV stopped working once ACF was gone

not integrated with Health Structures

Impact X Moderate

OTP was greatly delayed

SC only opened in June 2012

Good number of admission in SFP and PLW

Overcrowding of session for beneficiaries

Failed to raise awareness

Few negative impacts

Efficiency X Moderate

concerning Health Structures have not been achieved

CMAM program was operational with the exception of

the SC

good at dealing with unforeseen challenges

Monitoring system can be improved but is on place

Efficacy X Moderate

trainings should be given when participation is assured

sensitazion sessions should be optimized while

beneficiaries are waiting and given to small numbers

even if shorter time

human resources should be well employed for their

defined tasks

avoid incentives but use them if necessary to implement

program

Annexe 8. Examples of best practices

Title of best practice 1. Mobile OTPs

Innovative Features &

Key Characteristics

CMAM is meant to be a community based management of

Acute Malnutrition but often outpatients services are

installed in health structures far away from beneficiary

households and the population hardly know about it.

When the PPHI decided no to allow ACF to work in the

Basic Health Unit a mobile approach was established. OTP

changed everyday and were never more than 10 km away

from beneficiaries. Tracking defaulters was easy since they

can be visited on the same they that are defaulting. Places

can be changed as the time goes by to meet beneficiary’s

needs.

Mobile OTPs require good planning and some extra

logistical challenges. For a start every team needs

transportation to different sites every day. Fuel must be

considered in the budget and every day the team must leave

the base with the entire equipment load in the vehicles: in

order no to loose valuable time all this equipment must be

loaded the day before.

Coordination is important not only with logistics, but

especially with security team since Nutrition workers are

moving around big areas and it is known that situation on

the field can change fast. Weekly updates about movement

and constant communication is vital.

It also requires a constant approach with local leaders and

communities to find out best places and days to installed

OTP sites. Take into account markets days, presence of a

school etc.

Practical/Specific

Recommendations for

Roll Out

Though it may seem a great community approach it has a

slightly negative side. Being away from Health Structures

there is no integration at all with existing Health facilities.

Once the program is finished sustainability is almost zero.

Therefore it should be combined with a classical static

approach, spending some days of the week within Health

Centres.

Teams must not forget other aspects of the community

approach now that OTPs are placed in the community:

community volunteer’s networks to identify malnourished

cases, sensitizing the community about malnutrition,

tracking defaulters etc. but these activities are easily

followed up since the teams are very close to patients and

households.

Mobile OTPs impose additional workload to teams. It is

advisable to let the teams rest in the base once per week,

conduct sessions and refreshment trainings to allow

different teams to meet and share experiences etc.