monitoring & evaluation lfa presentation

Programme for Internal UNIFEM meeting Migration team for finalizing the Project Document of Phase III

26-27th August 2010

UNIFEM SARO officeDelhi, India

1

Planning, Monitoring and Evaluation in

UNIFEM

Session 1: 9.30-10.30Presentation on LFA from UNIFEM perspective

Session 2: 2.00-3.00Presentation on Key aspects of Monitoring and Evaluation

2

Agenda

To learn relevant planning techniques1. Stakeholder Identification2. Causal Analysis

Making a Problem Tree Making an Objective Tree

3. Project Selection4. Project Logic – Making a Logical Framework

To apply the skills to develop a Monitoring and Evaluation (ME) plan

3

Objectives

Session 1Planning techniques to develop a Logical Framework

4

5

1. Stakeholders Identification

UNIFEM would like to help to improve migrant women’s situation in Asia.

What are the problems of migrant women in Asia?

6

Stakeholder Identification Exercise

List on cards all individuals, groups and organizations related to migrant women in Asia.

7

Stakeholder Identification Exercise

8

For instance

Children

Women in rural areas in Thailand

UNIFEM

Community leader

For instance:

◦ Who are beneficiaries?◦ Any negatively affected groups?◦ Who are decision-makers?◦ Who are funding agencies?◦ Who are implementing?◦ Who will oppose? ◦ Who will support?

9

Categorizing Groups

Different groups have different needs Project may effect different groups in

different ways Whose problems should be solved? Women first!

10

Stakeholder Identification helps

11

2. Causal Analysis – Making a Problem Tree

Root causes is to keep looking. Ask “why”, get a direct cause. Keep asking “why”, get a cause that you

can act on.

12

Purpose of Causal Analysis- Making a Problem Tree

What are the problems in countries/areas you are working?

List the problems.

13

Causal Analysis Exercise – Making a Problem Tree

Indicate existing problems (not theoretical or potential)

One problem per card Describe in a sentence Try to avoid expression such as “No

(solution or resource) is available). Describe the condition resulting from the lack or absence of resources.

14

Principles of writing problem cards

15

For instance

There is no school.Children cannot receive

quality education.

AVOID PREFER

Decide the core problem Keep ask “why” to identify the causes

16

Causal Analysis Exercise – Making a Problem Tree

Causal Analysis (Problem Tree) Exercise: Causes to the Core Problem

17

Causes

Core Problem

Core Problem

Example: The Core Problem to Effects

18

Children drop out school.

Children are engaged in child labor.

Many children become

street children.

Children are exploited.

Many street children are involved in illegal acts.

Core Problem

Negative Effects

Causes Core Problem Negative Effect If we ask “why”, we know what the

problem is. If we know what the problem is, we

should know how to solve!

19

Summary of Causal Analysis (Problem Tree)

Causal Analysis (Objective Tree) Exercise: Solutions to Core Objective

20

Solutions

Core Problem

Core Objective

Example: Core Objective to Positive Ends

21

Children complete school.

Children can basic skills.

Children live in a protected environment.

.Children can

get an adequately paid job.

Children stay away from illegal acts.

Core Objective

Positive Ends

Solutions Core Objective Positive Ends

If we can act on causes, we can solve the core problem.

22

Summary of Causal Analysis (Objective Tree)

23

3. Project Selection

1. Use the Objective Tree2. Categorize related topics into a group3. Name the Group4. Refer the selection criteria5. Compare and examine the different Groups6. Select one group to be developed into a

project.

24

Project Selection Exercise

Target group Needs of people Priority Technical aspect Social risk factors Environment aspects Inputs Economic aspects Probability of achieving goals Relationship with other projects

25

Project Selection Criteria

Example

26

Core Objective

1. School infrastructure

Children complete school.

2. Training

3. Income generation

1. School infrastructure 2. Training 3. Income generation

27

Example: Project Options

Example: Project Selection Criteria

1. School infrastructure

2. Training 3. Income generation

Size of target group

+++ ++ +

Needs of people

+++ +++ +++

Priority + +++ +

28

Example: Project Selection Criteria

1. School infrastructure

2. Training 3. Income generation

Technical aspects (new technology)

+ + ++

Social risk factors

+++ + +++

Environment aspects

+++ + ++

29

Example: Project Selection Criteria

1. School infrastructure

2. Training 3.Income generation

Inputs + + ++

Economic aspects +++ + +++

Probability of achieving goals

+++ +++ +

Relationship with other projects

Duplicated with ODA projects

New approach

Duplicated with NGO project

30

1. Training2. School infrastructure3. Income generation

31

Project Selection Priority

Compare all approaches using the selection criteria

Discuss advantage and disadvantage of different approaches

Decide an approach

32

Summary of Project Selection

33

4. Project Logic- Making a Logical

Framework

34

Now you have

Objective TreeBranch: Project

Approach

35

Now we are going to

Branch:Project Approach

Pyramid:Project Logic

36

Project Logic is like Pyramid

Strategic Result

Key Result

Key Result

Key Result

Key Result

37

Remember your Problem Tree

Positive Ends

Core Objective

Solutions

38

Change to Project Design Pyramid

Positive Ends

Core Objectives

Solutions

Project Goal

Project Outcome

Outputs

Activities

Inputs

39

Project Design Pyramid

Project Goal

Project Outcome

Outputs

Activities

Inputs

Module 2-1 40

LOGIC MODELS

Outcomes

Impacts

Outputs

Activities

Inputs

Resources used to implement activities (financial, materials, human)

Inputs are what is needed for the Activities

Question: In Real Life, what may be the inputs needed to build a bridge?

Answer: Laborers, cement, steel beams, etc.

41

Inputs

Activities daily efforts needed for the outputs (and then objectives and goals).

Question: In Real Life, what may be the activities involved in building a bridge?

Answer: Identify plan, lay foundation, build structure, etc.

42

Activities

Outputs are the products needed to achieve the objective.

They are the deliverables.

Question: In Real Life, what may be the product you want in building a bridge?

Answer: completed bridge connecting two different towns.

43

Outputs

Outcomes are the major CHANGES needed to achieve the goal.

Question: In Real Life, what may be the primary outcome you want to achieve with the bridge?

Answer: Increase trade and between two towns.

44

Outcomes (Objective)

Simple, clear statement of the IMPACT or RESULT to achieve with by the project.

Change we hope to bring in the community.

Can be bigger than the project itself.

Question: In Real Life, what may be the Overall Reason you want to build a bridge between 2 communities?

Answer: Improve economy in two towns.

45

Goal

Narrative Summary

Project Goal

Project Outcome

Outputs

Activities

46

Project Logic Tool: Logical Framework

At this point, you should have your own “project logic”

47

Summary of project logic

48

What is the second column in the Logical Framework?

INDICATORS

Indicators

Project Goal

Project Outcome

Outputs

Activities

49

50

Indicators are like the dashboard of a car.

51

Indicator Dashboard

SpeedPetrol

Odometer 12400km

OilHow fast are we going?

How much did we do?

Are people happy with our work?

How many resources did we use to get there?

Not Enough Enough

Example: Training Project

Project Goal Ensure a protected environment for children.

Project Outcome Children complete school.

Outputs Teachers’ skills are updated.Education managers have appropriate planning and management skills.

Activities Conduct participatory teaching methods training for elementary school teachers.Conduct school management training for education managers in elementary schools.

52

Example: Training Project

Narrative Summary Indicators

Project Goal Ensure a protected environment for children.

Number of children stay with their care-givers.

Project Outcome Children complete school. Number of children completed school.

Outputs Teachers’ skills are updated.Education managers have appropriate planning and management skills.

# of teachers who receive a certificate.# of managers who updated management skills.

Activities Conduct participatory teaching methods training for elementary school teachers.Conduct school management training for education managers in elementary schools.

# of training conducted.

53

54

Standard Indicator Levels

Project Goal

Project Outcome

Outputs

Activities

Impact

Outcome

Output

Process

Input Input

Measures ultimate change desired in people or a condition (not always feasible or necessary).

Measures key changes necessary for goal

Measures the immediate product of an activity

Measure the activities (process), i.e. attendance at the activities

Measures the resources used for the activities.

Impact Indicator Outcome Indicator Output Indicator Process Indicator Input Indicator

55

What are the indicators for making a bridge?

Specific (Who & What?)Measurable (How much?)Area specific (Where?) - Achievable

Relevant – RealisticTime-Bound (When?)

56

SMART Indicators

Economic wellbeing of households restored and construction of new houses.

Compound Indicator!!!

57

What’s wrong with this Indicator?

Satisfaction in this workshop.

SMART: # participants in the Planning workshop (date) who report they would recommend the workshop to others.

58

Example: How can we make this Indicator SMARTer?

Quantitative - Can be directly counted and expressed as a number◦ % of……◦ # of ……◦ Frequency of……◦ Ration of ……◦ Amount of …….◦ Timeliness of ……..

Qualitative – Involve perception ( can be expressed quantitatively or as narrative)◦ Level of congruence with◦ Satisfaction with…..◦ Knowledge of…..◦ Ability to…. ◦ Appropriateness of ….◦ Importance of……

59

Indicator Examples

Objective Tree Branch (Project Selection) Pyramid (Project Logic)

Project Logic (Inputs, Activities, Outputs, Outcome, and Goal)

SMART Indicators for each Level.

60

Summary of Logical framework

Expected Results Statements Indicators Means of Measurement / Verification

Assumptions/Risks

Goal – Link to SP. The programme is expected to contribute towards the achievement of this result, collectively with other development partners.

Outcome 1 – Link to SP. The programme is expected to contribute towards the achievement of this result, collectively with other development partners.

Link to relevant SP indicators.

Output 1.1 – The programme will be directly held accountable for this result.

X – not needed

Activity 1.1.1 X – Not needed X

Activity 1.1.2 X X

Activity 1.1.3 X X

Module 2-1 61

UNIFEM LOGFRAME

62

Questions?

Session 2:

Apply the Logical Framework to develop a Monitoring and Evaluation (ME) Plan

63

To apply the skills to develop a Monitoring and Evaluation (ME) plan

1. Monitoring – What to monitor?2. Evaluation – What to evaluate?

64

Objectives of Session 2

65

COMPLEMENTARY ROLES OF MONITORING & EVALUATION

MONITORING

Clarifies program objectives

Links activities & resources to objectives

Translates objectives into performance indicators & sets targets

Routinely collects data, compares actual results with targets

Reports on progress & identifies problems

EVALUATION

Analyzes why intended results were or were not achieved

Assesses specific causal contribution of activities to results

Examines implementation process

Explores unintended results Provides lessons, highlights

significant accomplishment or program potential & offers recommendations for improvement

Rist & Kusek, 2004Ten Steps to a Results-Based M&E System

Module 2-1 66

LOGIC MODELS

Outcomes

Impacts

Outputs

Activities

Inputs

Monitoring

Evaluation

1. Monitoring – What to monitor?

67

Expected results

Indicators

Sources/Means of verification

Baseline Collection methods

Responsibilities

Goal

Outcome

Outputs

68

UNIFEM Monitoring Framework

A description (qualitative or quantitative) of the situation prior to the intervention against which progress can be assessed or comparison made

Used as a benchmark for assessing programme induced outcomes or impacts

Often the first data collected for an indicator is the baseline

Baseline data are gathered before or shortly after project implementation begins

69

Baseline

The starting point Data and information Related to results (outputs, outcome and

impact)

70

Baseline

2. Evaluation – What to evaluate?

71

Making a decision Enable managers and partners to make

decision and plan strategically Programme improvement, knowledge

generation and accountability

72

Why evaluation?

What you want to learn about the programme

What you need to know to make the programme decisions you need to make.

73

How to decide an evaluation type

Purpose of evaluation Audience Information to decide Sources How to collect the information When Available resources

74

Points to condider

Corporate & Decentralized EvaluationsCorporate & Decentralized Evaluations

Evaluation Sub-Types Evaluation Sub-Types Evaluation Modalities Evaluation Modalities

Scope Timing Unit of Analysis

Management Conduct

Output

Outcome

Impact

Impact

During: Evaluability assessment; Formative; Mid-term

After: Final; Summativ

e

Sometime after: Ex-

post

Institutional

evaluation

Policy, strategy, plan evaluationProject programme

Cluster thematic

Meta evaluation

Individual evaluation

Joint evaluation

External evaluation

Self evaluation

Peer evaluation

Terms Definition When used Questions

Output evaluation

Goods and services (results of inputs)

Useful when preceded by formative evaluation.

Deliveries in a timely manner?

Outcome evaluation

Changes (behavioral change) by intervention

Immediately following the end of an activity (short term influence)

Achieve results? Unintended results? Make a difference?

Impact evaluation

Broad, long-term impacts or effects

After a programme has concluded

Effect of programme outcome? How much better off?

76

By Scope

UNIFEM programm

e cycle

Terms Definition When to use

Questions

Formulation Ex-ante Predict the likelihood

Formulation Will the activities meet the needs?

Before or during implementation

Evaluability assessment

Meaningfully evaluated

Before any evaluation takes place

Gender equality identified? Objectives realistic?

77

By Timing

UNIFEM programme cycle

Terms Definition When to use

Questions

Evaluation phase

Formative evaluation

Early insights

During implementation

What is happening?

Mid-term evaluation

Half-way During implementation

Is it working well?

Closing phase

Final evaluation

End implementation

Immediately the end of an activity or intervention cycle

Achieving?

Closing phase

Summative and ex-post evaluation

Some time after the programme

Concluded or a year after

Making difference?

78

By Timing

Formulation

Appraisal

Approval

Implementation and Monitoring

Evaluation

Closing

79

Timing in UNIFEM Programme Cycle Ex-ante

Summative

and ex-post

Formative, mid-term,

outcome, impact

Before and during:

evaluability assessment

Terms Definition How to use

External evaluation External specialists Objective analysis

Self evaluation Periodic reviews to assess the achievements and constraints

By responsible for implementation, often used in smaller projects or as the first step

Peer evaluation Composed of external evaluators and programme staff

Formative, combine internal understanding with external expertise.

80

By Conduct

Mandatory evaluation Investment

A final evaluation during programme life cycle

Programme with budget over USD 1 million

A mid-term and final evaluation during programme life cycle

Programmes with budgets over USD 3 million

Mandatory evaluation

Need a quality Logical Framework Decide which level you monitor and

evaluate Use UNIFEM Monitoring Framework Think about baseline from the beginning Decide what to evaluate and which level of

the Logical Framework Evaluation is to make a decision about the

programme.

82

Summary of develop a ME plan

83

Questions?

84

Thanks!