EVALUATION of HUMANITARIAN ACTION

- an initial discussion around PHT evaluations following future emergencies -

United Nations Office for the Coordination of Humanitarian Affairs

Pacific Humanitarian Team Day 4

OBJECTIVE OF THIS SESSION IS TO:

COMMON UNDERSTANDING OF EVALUATIONS AND EVALUATION CRITERIA

BASED ON EVALUATION CRITERIA, IN WORKING GROUPS (CLUSTERS, NDMO, DONOR) COME UP WITH QUESTIONS THAT WILL BE CONSIDERED IN FUTURE EVALUATION OF HUMANITARIAN ACTION IN THE PACIFIC.

DIFFERENCE BTWN MONITORING AND EVALUATION

DIFFERENCE BTWN MONITORING AND EVALUATION

MONITORINGROUTINE ASSESSMENT OF ONGOING

ACTIVITIES AND PROGRESS. SYSTEMATIC AND CONTINUOUS

ASSESSMENT OF THE PROGRESS OF A PIECE OF WORK OVER TIME.

TO HELP ALL THE PEOPLE INVOLVED MAKE APPROPRIATE AND TIMELY DECISIONS THAT WILL IMPROVE THE QUALITY OF THE WORK.

DIFFERENCE BTWN MONITORING AND EVALUATION

EVALUATIONEPISODIC ASSESSMENT OF OVERALL

ACHIEVEMENTS OF PROGRAMME OBJECTIVES.

FOCUSES ON MEASURING WHETHER PLANNED OUTCOMES AND IMPACTS HAVE BEEN REALIZED.

PURPOSE OF EVALUATION: ‘LESSON LEARNING’ AND/OR ‘ACCOUNTABILITY’?

PURPOSE OF EVALUATION: ‘LESSON LEARNING’ AND/OR ‘ACCOUNTABILITY’?

Characteristic Lesson-learning oriented

Accountability-oriented

Terms of reference Likely to be set by those directly involved in the programme

Likely to be set by those external to the programme

Team membership Internal team of employees, or mixed team of employees and independent members

Independent external team

Emphasis in approach

Process of reflection and reaching conclusions emphasised – more subjective

Methodology of data collection and analysis emphasised – more objective

Style of management More facilitative More directive

Circulation of report

Internal to organisation/ restricted In public domain

EVALUATION CRITERIA

EVALUATION CRITERIA

RELEVANCE AND APPROPRIATENESSRELEVANCE IS CONCERNED WITH ASSESSING

WHETHER THE PROJECT IS IN LINE WITH LOCAL NEEDS AND PRIORITIES (AS WELL AS DONOR POLICY).

APPROPRIATENESS IS THE TAILORING OF HUMANITARIAN ACTIVITIES TO LOCAL NEEDS, INCREASING OWNERSHIP, ACCOUNTABILITY AND COST-EFFECTIVENESS ACCORDINGLY

EVALUATION CRITERIA

RELEVANCE AND APPROPRIATENESSEFFECTIVENESSEFFECTIVENESS MEASURES THE EXTENT TO WHICH

AN ACTIVITY ACHIEVES ITS PURPOSE, OR WHETHER THIS CAN BE EXPECTED TO HAPPEN ON THE BASIS OF THE OUTPUTS

EVALUATION CRITERIA

RELEVANCE AND APPROPRIATENESSEFFECTIVENESSCONNECTEDNESS (SUSTAINABILITY)CONNECTEDNESS REFERS TO THE NEED TO ENSURE

THAT ACTIVITIES OF A SHORT-TERM EMERGENCY NATURE ARE CARRIED OUT IN A CONTEXT THAT TAKES LONGER-TERM AND INTERCONNECTED PROBLEMS INTO ACCOUNT (SUSTAINABILITY – THE IDEA THAT INTERVENTIONS SHOULD SUPPORT LONGER-TERM GOALS)

EVALUATION CRITERIA

RELEVANCE AND APPROPRIATENESSEFFECTIVENESSCONNECTEDNESS (SUSTAINABILITY)COVERAGECOVERAGE IS THE NEED TO REACH MAJOR

POPULATION GROUPS FACING LIFE THREATENING SUFFERING, WHEREVER THEY ARE.

EVALUATION CRITERIA

RELEVANCE AND APPROPRIATENESSEFFECTIVENESSCONNECTEDNESS (SUSTAINABILITY)COVERAGECOORDINATIONCOORDINATION: THE SYSTEMATIC USE OF POLICY

INSTRUMENTS TO DELIVER HUMANITARIAN ASSISTANCE IN A COHESIVE AND EFFECTIVE MANNER. SUCH INSTRUMENTS INCLUDE STRATEGIC PLANNING, GATHERING DATA AND MANAGING INFORMATION, MOBILISING RESOURCES, ETC…

ADDITIONAL CRITERIA INCLUDE…

EFFICIENCYIMPACTCOHERENCEPROPORTIONALITY

ADDITIONAL CRITERIA INCLUDE…

EFFICIENCYEFFICIENCY MEASURES THE OUTPUTS - QUALITATIVE

AND QUANTITATIVE - ACHIEVED AS A RESULT OF INPUTS. THIS GENERALLY REQUIRES COMPARING ALTERNATIVE APPROACHES TO ACHIEVING AN OUTPUT, TO SEE WHETHER THE MOST EFFICIENT APPROACH HAS BEEN USED

ADDITIONAL CRITERIA INCLUDE…

EFFICIENCYIMPACTIMPACT LOOKS AT THE WIDER EFFECTS OF THE

PROJECT - SOCIAL, ECONOMIC, TECHNICAL AND ENVIRONMENTAL – ON INDIVIDUALS, GENDER - AND AGE-GROUPS, COMMUNITIES AND INSTITUTIONS. IMPACTS CAN BE INTENDED AND UNINTENDED, POSITIVE AND NEGATIVE, MACRO (SECTOR) AND MICRO (HOUSEHOLD)

ADDITIONAL CRITERIA INCLUDE…

EFFICIENCYIMPACTCOHERENCECOHERENCE IS THE NEED TO ASSESS SECURITY,

DEVELOPMENTAL, TRADE AND MILITARY POLICIES AS WELL AS HUMANITARIAN POLICIES, TO ENSURE THAT THERE IS CONSISTENCY AND, IN PARTICULAR, THAT ALL POLICIES TAKE INTO ACCOUNT HUMANITARIAN AND HUMAN-RIGHTS CONSIDERATIONS

ADDITIONAL CRITERIA INCLUDE…

EFFICIENCYIMPACTCOHERENCEPROPORTIONALITYPROPORTIONALITY IS THE EXTENT TO WHICH A

PARTICULAR HUMANITARIAN RESPONSE IS IN PROPORTION TO THE NEEDS OF AFFECTED PEOPLE, AND TO OTHER SIMILAR EMERGENCY RESPONSES.

EVALUATION STAGES:1 - PLANNING: DRAFT CONCEPT NOTE, ESTABLISH

STEERING GROUP, DRAFT TOR. 2 - PREPARATION AND RESEARCH:

PROJECT/PROGRAMME RESEARCH, INTERVIEWS, PLANNING COUNTRY VISITS AND DISSEMINATION STRATEGY

3 - COUNTRY VISIT: EVALUATION RESEARCH, WORKSHOP(S)

4 - REPORTING: DRAFT REPORT AND EVALUATION SUMMARY, CIRCULATE REPORT FOR COMMENT

5 - DISSEMINATION: PUBLICATION AND DISTRIBUTION OF REPORT AND EVALUATION SUMMARY

EVALUATION STAGES:1 - PLANNING: DRAFT CONCEPT NOTE, ESTABLISH

STEERING GROUP, DRAFT TOR. 2 - PREPARATION AND RESEARCH:

PROJECT/PROGRAMME RESEARCH, INTERVIEWS, PLANNING COUNTRY VISITS AND DISSEMINATION STRATEGY

3 - COUNTRY VISIT: EVALUATION RESEARCH, WORKSHOP(S)

4 - REPORTING: DRAFT REPORT AND EVALUATION SUMMARY, CIRCULATE REPORT FOR COMMENT

5 - DISSEMINATION: PUBLICATION AND DISTRIBUTION OF REPORT AND EVALUATION SUMMARY

6 – FOLLOW-UP - ACTING ON RECOMMENDATIONS

Cluster An evaluation of a set of related activities, projects and/or programs. Country Programme

Evaluation of one or more of the agency‟s portfolio of interventions and the assistance strategy behind them, in a particular country

Synthesis A synthesis of the findings from a number of evaluations of individual projects or programmes.

Joint An evaluation to which different donor agencies and/or partners participate.

Impact Focuses on the impact of the aid, rather than on aid delivery. Usually, but not always, carried out some time after project completion.

Meta-evaluation The term is used for evaluations designed to aggregate findings from a series of evaluations. It can also be used to denote the evaluation of an evaluation to judge its quality and/or assess the performance of the evaluators

Mid-term evaluation

Evaluation performed towards the middle of the period of implementation of the intervention.

Participatory Evaluation method in which representatives of agencies and stakeholders (including beneficiaries) work together in designing, carrying out and interpreting an evaluation.

Process An evaluation of the internal dynamics of implementing organizations, their policy instruments, their service delivery mechanisms, their management practices, and the linkages among these.

Programme Evaluation of a set of interventions, marshaled to attain specific global, regional, country, or sector development objectives.

Project Evaluation of an individual intervention designed to achieve specific objectives within specified resources and implementation schedules, often within the framework of a broader programme.

Policy Examines the framework of understanding, beliefs and assumptions that make individual projects possible as well as desirable. Policy evaluations seek out the inherent tensions or contradictions in policy objectives through tools such as discourse analysis.

Real time Evaluation of an ongoing operation to evaluate events as they unfold.

Self-evaluation An evaluation by those who are entrusted with the design and delivery of an intervention.

Sector Evaluation of a cluster of interventions in a sector within one country or across countries, all of which contribute to the achievement of a specific development goal.

Thematic Evaluation of a selection of interventions, all of which address a specific priority that cuts across countries, regions, and sectors.

JOINT EVALUATIONS?

JOINT EVALUATIONS ARE OVERALL OF HIGHER QUALITY THAN SINGLE AGENCY EVALUATIONS, FOR EXAMPLE:(I) TOR ARE GENERALLY CLEARER(II) CONSULTATION WITH LOCAL POPULATIONS AND BENEFICIARIES IS STRONGER(III) MORE ATTENTION IS PAID TO INTERNATIONAL STANDARDS; AND(IV) THE EHA CRITERIA ARE MORE RIGOROUSLY USED.

DISCUSSION…

A CATEGORY 4 CYCLONE HAS JUST STRUCK PORT VILA AND THE OUTER ISLANDS OF VANUATU EACH WORKING GROUP REVIEW

EVALUATION CRITERIA AND DERIVE A SET OF QUESTIONS TO BE USED IN A FUTURE JOINT EVALUATION OF PHT RESPONSE.

TEAM 1-7: THE CLUSTERS TEAM 8: DONORS AND NDMOS (DONORS, NDMOS AND OCHA TO COMPRISE TEAM 8)

THANK YOU