6 monitoring & evaluation in project cycles
DESCRIPTION
Part 4 of an 8 module course in aid and development project cycle planning using the logical framework approach.Parts 5 and 6 deal with indicators and data Sources in more detail.Also forms part 6 of a series of 10 on managing teams in NGOs.TRANSCRIPT
![Page 1: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/1.jpg)
Development Project Planning 4
Monitoring and EvaluationIn the Project Cycle
![Page 2: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/2.jpg)
Ground Rules
Suggested Rules:
Come on time
No Phones
No come and go
No chat
![Page 3: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/3.jpg)
M&E and the Project Cycle
What is M&E Monitoring
o Types of monitoringo Working with Indicatorso Data sources
Evaluation Choosing Indicators
![Page 4: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/4.jpg)
Why ?
The purpose of monitoring and evaluation is to measure program effectiveness
![Page 5: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/5.jpg)
Uses Of M & E
Monitoring and evaluation helps: make decisions on implementation ensure the most effective and efficient use
of resources determine if the project is on track and make
any needed corrections evaluate the impact of the project
![Page 6: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/6.jpg)
What is Monitoring ?
Monitoring is collecting routine data to measure the progress of a project or activity. It is used to track project performance over time, to make informed decisions about the effectiveness of projects and the efficient use of resources.
Monitoring is also called process evaluation because it looks at the implementation process and asks:
o How well has the project been implemented?o How much does implementation vary from site to site?o Did the project benefit the intended people? At what
cost?
![Page 7: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/7.jpg)
What is Evaluation ?
Evaluation measures how well the program activities have achieved the objectives, andhow much the changes in outcomes are due to the project.
The difference in the outcome between having the project or not having the project is known as its “impact,” and measuring this difference is referred to as “impact evaluation.”
![Page 8: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/8.jpg)
Some Other Words
Impact: any effect caused by project activity including human health and safety, plants, animals, soil, air, water, climate, landscape, structures, cultural heritage or socio-economic conditions.
Metric: a unit of measure
Benchmark: a chosen level of an indicator
![Page 9: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/9.jpg)
Some Other Words
Objective: factual, real; can be measuredscientific, repeatable resulte.g. how many people have TB?
Subjective: from one person’s viewCan be different for different peoplee.g. are we a democratic country?
![Page 10: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/10.jpg)
M&E Plan
The project proposal must include an M&E plan M&E requires resources – time, staff, money The indicators must relate to the project aim The data must be reliable Managers must be willing to use and learn from
the results, and follow up Dissemination – share information and lessons
learned
![Page 11: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/11.jpg)
M&E Plans should include.. assumptions regarding context, activities, and goals relationships between activities, targets, and outcomes description of measures and operational definitions
(indicators and metrics), with baseline values, monitoring schedule, data sources, and M&E resource estimates
partnerships and collaborations required to achieve results
specific attention to periodic evaluation, with resources allocated at least midterm and at project end.
![Page 12: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/12.jpg)
M&E
M&E looks at progress against the INDICATORS in the logframe
So the first step for good M&E is choice of indicators
![Page 13: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/13.jpg)
What is an Indicator ?
a variable …that measures one aspect of a projectthat is directly related to the program’s objectives.
![Page 14: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/14.jpg)
What is an Indicator ?
An indicator is a variable whose value changes from the
baseline level (at the time the program began) to a new value after the program and its activities have made their impact felt.
Then the variable, or indicator, is calculated again.
![Page 15: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/15.jpg)
What is an Indicator ?
Secondly, an indicator is a measurement.
It measures the value of the change in meaningful units that can be compared to past and future units.
This is usually expressed as a percentage or a number.
![Page 16: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/16.jpg)
What is an Indicator ?
Finally, an indicator focuses on a single aspect of a program or project.
This aspect may be an input, an output or an overarching objective, but it should be narrowly defined in a way that captures this one aspect as precisely as possible.
![Page 17: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/17.jpg)
How many Indicators?
an appropriate set of indicators will include at least one for each significant element of the project (i.e. at least one per box in a logframe)
a reasonable guideline recommends one or two indicators per result, at least one indicator for each activity, but no more than 10-15 indicators per area of significant program focus.
![Page 18: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/18.jpg)
A good indicator should:
produce the same results when used repeatedly to measure the same condition or event;
measure only the condition or event it is intended to measure;
show changes in the state or condition over time;
have reasonable measurement costs; and be defined in clear and unambiguous terms.
![Page 19: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/19.jpg)
Good Indicators
Valid
Reliable
Precise
Independent
Timely
Comparable
![Page 20: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/20.jpg)
Good Indicators Validity
The indicator measures what it is intended to measure
![Page 21: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/21.jpg)
Good Indicators Reliability
The indicator minimises measurement errorrepeatable
Types of measurement error Sampling Error - caused by observing a
sample instead of the whole population Non-Sampling Error – all other errors Subjective Measurement - bias
![Page 22: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/22.jpg)
Good Indicators Precise Definition
Is defined in clear termso What you measure and how
![Page 23: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/23.jpg)
Good Indicators Independence
Not directional – can be positive OR negative
One dimensional – up and down a line
Describe a discrete result at a single point in time
![Page 24: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/24.jpg)
Good Indicators Timeliness
Provides a measurement over the periods of time that matter (e.g. the project life)
with data available for all appropriate intervals
![Page 25: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/25.jpg)
Good Indicators Comparability
Compares with other similar situations
Assists in understanding results across different population groups and project approaches
![Page 26: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/26.jpg)
Additional Factors InfluencingIndicator Selection
Data availability
Resources
Program needs
Donor requirements
![Page 27: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/27.jpg)
Take a break
![Page 28: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/28.jpg)
Types of Indicators
Quantitative - an actual number of some output
Qualitative - descriptive observations that can supplement the numbers and percentages provided by quantitative indicators. They add to quantitative indicators a richness of information about the context in which the program has been operating. Examples include “availability of a clear, strategic organisational mission statement” and “existence of a multi-year procurement plan for each product offered.”
“For a quantitative indicator you would collect numbers, and for a qualitative indicator you would collect facts or opinions.”
![Page 29: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/29.jpg)
Types of Indicators
Confusion exists in what are qualitative and quantitative indicators.
It is clear that quantitative indicators measure changes that can be counted. It is not clear what is a qualitative indicator.
some say qualitative indicators relate to the quality of the change being measured (DFID, 1995) - e.g. women’s political representation: a quantitative indicator could be the percentage of parliament seats occupied by women, while a qualitative indicator would describe the quality of women’s political participation.
some say qualitative indicators describe a subjective opinion on an issue or project impact.
![Page 30: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/30.jpg)
Qualitative is Quantitative?
Most do not define indicators as qualitative or quantitative - they assume that all indicators are by definition quantitative.
For example:A qualitative indicator could be “most village women feel
they have a voice at meetings”[ but how do you know this? ]
So this could be quantitative:“the percentage of women surveyed who say they have
a voice at meetings”
![Page 31: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/31.jpg)
Indicators of What ?Inputs Shows what goes in to providing a service - the resources
used, e.g.• Amount spent on travel per week• Home care supplies purchased per month• Wages, allowances and incentives paid• Production costs for brochures and posters
Activity orOutput
Shows what a service has done or provided, e.g.• Number of brochures produced• Number of condoms distributed• Number of home care visits• Number of clients counselled and tested
Utililisation Utilisation Shows if a service is being used, e.g.• Number of people attending a nutrition course• Number of people requesting VCT• Applications received for a training course
![Page 32: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/32.jpg)
Indicators of What ?Coverage Coverage Shows what proportion of people / groups in need
receive a service, e.g.• Proportion of all orphans receiving visits• Proportion of schools with an AIDS awareness club• Proportion of commercial farms with peer education programmes
Performance Performance Shows how well something was done, e.g.• Number of people reporting they are ‘satisfied’ with a training workshop• Number of reported cases of STIs• Proportion of VCT clients returning to collect their HIV test results• Number of orphans supported in the community
![Page 33: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/33.jpg)
Exercise
Water and Sanitation Project
1. Read the Concept Note
2. Using the Logframe, choose the Indicators
![Page 34: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/34.jpg)
Setting up an Indicator
You must identify exactly how a given concept or behaviour will be measured – the Metric.
The Metric is the precise calculation or formula on which the indicator is based. Calculation of the metric establishes the indicator’s objective value at a point in time. Even if the factor itself is subjective or qualitative, (eg attitudes of a target population), the indicator metric calculates its value at a given time objectively
This can be called “Operationalising” an Indicator
![Page 35: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/35.jpg)
Setting up an Indicator
You need to be careful about exactly how you define the metric
e.g. “the percentage of HIV+ mothers who have prepartum AZT therapy”
is it – ‘% of those births attended by the health care system’ or ‘% of all births’
is it – ‘% of recorded diagnosed HIV+ women’ or ‘% of all HIV+ women’
![Page 36: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/36.jpg)
Setting up an Indicator
In many cases, indicators need to have definitions of the terms used.
For instance, let’s look at the indicator: ‘number of antenatal care (ANC) providers trained’. If this indicator is used by a program, definitions need to be included.
Providers would need to be defined, e.g‘any worker providing direct clinical services to clients seeking ANC at a public health facility’. For this indicator then, providers would not include those working in private facilities.
Trained would also need to be defined, perhaps as ‘those staff who attended every day of a five-day training course and passed the final exam with a score of at least 85%’.
![Page 37: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/37.jpg)
Thankyou
Please keep papers for next session
![Page 38: 6 Monitoring & Evaluation in Project Cycles](https://reader033.vdocuments.site/reader033/viewer/2022061123/5472326fb4af9f311d8b458c/html5/thumbnails/38.jpg)
Produced by Tony HobbsHealth Unlimited, Ratanakiri, Cambodiawww.healthunlimited.org
With the support of Australian Volunteers Internationalwww.australianvolunteers.com
© 2009 HU. Use with Acknowledgement