1

MONITORING AND EVALUATION TRAINING MANUAL FOR CBOs AND NGOs

Overall Goal:

To give concepts, skills and knowledge on monitoring and evaluation for

development

Developed by: Institutional Capacity Building Team SHIELD Project International Rescue Committee - Thailand

2

Table of Contents TIME-TABLE ................................................................................................................ 3 monitoring and evaluation Training, Pre Test ........................................................... 4 Session I: Orientation ............................................................................................... 5 Session II: Warm-up .............................................................................................. 7

WORKSHEET 2 – 1, Key steps of project management ...................................... 10 WORKSHEET 2 – 2, CAUSAL HYPOTHESIS ..................................................... 11

Session- III: what is monitoring? .............................................................................. 12 WORKSHEET 3 – 1, MONITORING .................................................................... 13

Session- IV: what is evaluation? ............................................................................... 14 WORKSHEET 4 – 1, EVALUATION .................................................................... 16

Session V: Developing indicators ....................................................................... 18 WORKSHEET 5 – 1, INDICATORS ...................................................................... 22 WORKSHEET 5 – 2, DEVELOPING OUTPUT/EFFECT/IMPACT IMPACT INDICATORS ……………………………………………………………………….23 WOKSHEET-5-3DEVELOPING EFFICIENCY, EFFECTIVENESS AND IMPACT INDICATORS INDICATORS ..................................................... …… 24

Session VI: Planning for monitoring and evaluation ......................................... 26 WORKSHEET 6 – 1 ,Develop monitoring plan – 1 .............................................. 31 WORKSHEET 6 – 2, DEVELOP MONITORING PLAN – 2................................ 32 WORKSHEET 6 – 3, DIFFERENT TYPES OF EVALUATION ........................... 34 WORKSHEET 6 – 4, Case Study ............................................................................ 36 WORKSHEET 6 – 5, EVALUATION BASICS ...................................................... 39 WORKSHEET 6 – 6, TERMS OF REFERENCE .................................................... 40

Session VII: Gathering information for monitoring and evaluation .................... 43 WORKSHEET 7 – 1, Different social research tools............................................. 45

Session VIII: Data analysis ........................................................................................ 48 WORKSHEET 8 – 1, DATA ANALYSIS ............................................................... 51 .................................................................................................................................. 51

Session IX: Reporting ................................................................................................ 53 WORKSHEET 9 – 1, EVALUATION REPORT FORMAT ................................... 55

Monitoring and evaluation Training, Post-Test ....................................................... 56 References ................................................................................................................... 57

3

TIME-TABLE DAY 9:00 – 10:30

CO

FFEE BREA

K (15 M

INU

TES)

10:45 TO 12:00 AM

LUN

CH

BREA

K (O

NE H

OU

R)

1:00 TO 3:00 PM

CO

FFEE BREA

K (15 M

INU

TES)

3:15 TO 4:30 PM DAY – 1

• REGISTRATION (15 MIN)

• PRE-TEST (15 MIN) • ORIENTATION ( 1

HOUR)

• WARM UP (30 – 45 MIN)

• WHAT IS MONITORING? ( 30 MIN)

• WHAT IS EVALUATION? ( 30 MIN)

• DEVELOPING INDICATORS

• DEVELOPING INDICATORS

DAY – 2

• PLANNING FOR MONITORING AND EVALUATION

• PLANNING FOR MONITORING AND EVALUATION

• PLANNING FOR MONITORING AND EVALUATION

• PLANNING FOR MONITORING AND EVALUATION

DAY – 3

• DATA COLLECTION METHODS

• BASIC DATA ANALYSIS METHODS

• REPORTING • WRAP UP SESSION • POST TEST • TRAINING EVALUATION

4

MONITORING AND EVALUATION TRAINING PRE TEST

Please answer the following questions to know your existing knowledge and skills. 1) What is Monitoring? 2) Why do we do Monitoring? 3) Who does Monitoring? 4) When do we do Monitoring? 5) What are the important things to look at when you monitor a project? 6) What are indicators? Please give an example of indicator. 7) What is evaluation? 8) For what purposes do you normally do evaluation? 9) When do you normally evaluate? 10) Write down the monitoring and evaluation tools you know.

5

SESSION I: ORIENTATION

The Facilitator/s and participants will get an opportunity to introduce themselves, to exchange information about past experiences and to share workshop expectations Time: 1 hour Materials: A4 size paper (cut into four pieces), flipcharts, markers

Introduction Step 1: Ask the participants to divide into two groups equally: Group A

and Group B Step 2: Then cut A4 size paper into four pieces and distribute one piece

to each participant. Ask them to write down • their name • Mother organization • M&E related experience they have before

Step 3: When everyone finishes writing, ask each group to stand in line facing other group.

Step 4: Ask people from Group A to give their card to the one who is in

front of him/her in Group B introducing themselves. Then the one from Group B also introduces about himself/herself exchanging their cards.

Step 5: When all have finished introducing each other, ask each pair to introduce their friends to the whole class.

6

Forming groups Step 6: Ask all the participants to stand up in line again based on the

birth months starting from January. Based on the number of participants, ask them to count 1 to 5 (if you want to form 5 groups). Then ask all people with 1, 2, 3, 4, 5 number can sit together. Ask them to give a name for their group. Tell them that they have to be together for group work for three days.

Expectations Step 7: Tell the participants that this workshop will focus on

“Monitoring and Evaluation” of a project included in project cycle management. Ask them to write down their expectations and concerns related to this workshop on a flipchart. (5 – 10 minutes)

Step 8: When all the groups finish, ask them to read other groups’ expectations and concerns and if they find the same ones, they can write down group number in the bracket.

Step 9: Then share the workshop objectives and if what they expect

cannot be included in the workshop objectives, discuss with them. Related to the concerns, if the facilitators can deal with them, please tell them how and if the participants need to deal with, tell them to set ground rules and how they want to take action if somebody breaks the rule/s. e.g. phone calls, punctuality, dominate the class by one or two participants.

7

SESSION II: WARM-UP

This session revisits key steps of project management and terminology used in M&E to refresh the memory. Time: About 30 to 45 minutes Material: flipcharts, markers, worksheets

Step 1: The facilitator starts the session by asking the key steps of project

management. Then show the key steps of project management distributing Worksheet 2.1: project management.

8

Key steps of project management When a project is started to implement, it is necessary to monitor the project activities while implementing the project and at the end of the project, normally, the project is evaluated.

Step 2: The facilitator ask the participants to fill the blanks included in the

worksheet. The causal hypothesis is a concise statement of the logic behind the project. Normally, causal hypothesis is written as follows:

Assessing needs

Planning the project

Implementing and monitoring the project

Evaluating the project

Learning from the project and evolving

9

This set of --------- and ---------- will result in these products and services (--------). These will in turn lead to these changes in people’s behavior (------------), which will contribute to the desired ---------------. (activities, IMPACT, outputs, inputs, effects) Step 3: Then discuss with the participants about the terms used in causal

hypothesis and show example.

Below is the meaning of the terms used in causal hypothesis. Causal Pathway Framework

Design Direction

Implementation, Monitoring and Evaluation

IMPACT: A change in population's health, economic or social status. EFFECT: A change in the knowledge, attitudes, skills, intentions, or and behaviors needed of the population that contribute to the desired IMPACT. OUTPUTS: Products and services that must be in place before the EFFECTS can occur. ACTIVITIES: The technical and support tasks required to produce the OUTPUTS. INPUTS: Resources required to support your ACTIVITIES. Inputs refer to the set of resources (i.e. financial, policies, personnel, facilities, space, equipment and supplies, etc.) that are the basic materials of the project/program.

Inputs Activities Outputs Effect

Inputs Activities Outputs Effects Impact

M&E training Training follow-up

# of participants who attended M&E training # of participants who gets M&E skills

# of staff who apply monitoring and evaluation skills in their organizations # of staff who can monitor effectively and evaluate the project

Trainers Training materials Training equipments Training venue Training participants (staff from CBOs/NGOs)

Improved project quality with effective monitoring and evaluation system

10

WORKSHEET 2 – 1

Key steps of project management When a project is started to implement, it is necessary to monitor the project activities while implementing the project and at the end of the project, normally, the project is evaluated.

Assessing needs

Planning the project

Implementing and monitoring the project

Evaluating the project

Learning from the project and evolving

11

WORKSHEET 2 – 2

CAUSAL HYPOTHESIS The causal hypothesis is a concise statement of the logic behind the project. Normally, causal hypothesis is written as follows: This set of --------- and ---------- will result in these products and services (--------). These will in turn lead to these changes in people’s behavior (------------), which will contribute to the desired ---------------. (activities, IMPACT, outputs, inputs, effects) The terms used in causal hypothesis. Causal Pathway Framework

Design Direction

Implementation, Monitoring and Evaluation

IMPACT: A change in population's health, economic or social status. EFFECT: A change in the knowledge, attitudes, skills, intentions, or and behaviors needed of the population that contribute to the desired IMPACT. OUTPUTS: Products and services that must be in place before the EFFECTS can occur. ACTIVITIES: The technical and support tasks required to produce the OUTPUTS. INPUTS: Resources required to support your ACTIVITIES. Inputs refer to the set of resources (i.e. financial, policies, personnel, facilities, space, equipment and supplies, etc.) that are the basic materials of the project/program.

Inputs Activities Outputs Effect

Inputs Activities Outputs Effects Impact

M&E training Training follow-up

# of participants who attended M&E training # of participants who gets M&E skills

# of staff who apply monitoring and evaluation skills in their organizations # of staff who can monitor effectively and evaluate the project

Trainers Training materials Training equipments Training venue Training participants (staff from CBOs/NGOs)

Improved project quality with effective monitoring and evaluation system

12

SESSION- III: WHAT IS MONITORING?

After this session, the participants will understand the concept of monitoring and how to develop good indicators. Time: 30 – 45 minutes Material: flip chart, markers, worksheets

Step1: Eliciting (15 - 20 minutes)

Ask the following questions to the participants: - What is Monitoring? - Why do we do Monitoring? - Who does monitoring? - When do we do Monitoring?

Write down what the participants discuss on flipchart. Then distribute the Worksheet 3 – 1 : Monitoring and discuss for a while.

13

WORKSHEET 3 – 1

MONITORING What is Monitoring? Monitoring is the systematic collection of information on all aspects of the project while it is being implemented. It can be divided into internal monitoring (staff performance, planned expenditure for each activity versus actual expenses, procurement procedures etc) and external monitoring (planned versus actual activities, timely implementation of activities, targeted beneficiaries versus true beneficiaries, unintended effects on the community and unexpected problems etc). Both are important and both need to be monitored. Why Do We Do Monitoring? We do monitoring to analyze the current situation, identify problems and find solutions, discover trends and patterns, keep project activities on schedule, measure progress towards objectives, formulate/revise future goals and objectives, make decisions about human, financial, and material resources. Actually, it is a very useful tool for management and provides necessary information for evaluation. In other words, monitoring means checking how things are going on and comparing actual progress to what is planned. Who Does Monitoring? Monitoring is concerned both with project staff, implementation organization and donors. As mentioned above, it is useful for management and project manager or program coordinator conducts monitoring on all aspects of the project-budget/finance, materials, staff, activities, outputs/results etc. Respected project staff are also responsible for monitoring staff and tasks under them-for example, finance manager has to monitor the accountant and cashier as well as budget allocations. At the same time, representatives of donors also conduct monitoring to measure the progress towards objectives and goal/impact. When Do We Do Monitoring? According to the desired schedule of implementation of the projects…. e.g. monthly, bi-monthly, quarterly

14

SESSION- IV: WHAT IS EVALUATION? After this session, the participants will understand the concept of evaluation

Time: 30 – 45 minutes Material: flipcharts, markers, worksheet

Step 1: In plenary, ask the following questions to the participants: (15 - 20

minutes) • What is evaluation? • Why Do We Do Evaluation? • When Do We Evaluate? • Who Does Evaluation?

The facilitator can read through the worksheet 4 – 1: Evaluation together with the participants to understand more about evaluation. Step 2: Show the below project activity cycle to the participants.

15

Step 4: The facilitator can end the session with below explanation. M&E is an ongoing process in a project since the project is started.

It is important to recognize that monitoring and evaluation are not magic wands that can be waved to make problems disappear, or to cure them, or to miraculously make changes without a lot of hard work being put in by the project or organization. In themselves, they are not a solution, but they are valuable tools.

Monitoring and evaluation can: Help you identify problems and their causes; Suggest possible solutions to problems; Raise questions about assumptions and strategy; Push you to reflect on where you are going and how you are getting there; Provide you with information and insight; Encourage you to act on the information and insight; Increase the likelihood that you will make a positive development difference.

16

WORKSHEET 4 – 1

EVALUATION What is evaluation? Evaluation is a systematic and objective assessment of ongoing or completed project. It makes comparison of the outcomes of the project with planned ones. Why Do We Do Evaluation? The primary objective of evaluation is to ascertain whether the project has achieved its intended objectives. By drawing conclusions, evaluation intends to provide recommendations for the improvement on the future course of the project as well as lessons learned for other projects. Some big organizations use specific criteria when they do evaluation. When Do We Evaluate? Periodically, mid-term, at the end of the project (final evaluation) and years after the completion of the project (post-evaluation). Who Does Evaluation? Project manager or assigned project staff can conduct internal evaluation and donor/s or consultant/s can conduct external evaluation.

Project activity cycle

17

M&E is an ongoing process in a project since the project is started. It is important to recognize that monitoring and evaluation are not magic wands that can be waved to make problems disappear, or to cure them, or to miraculously make changes without a lot of hard work being put in by the project or organization. In themselves, they are not a solution, but they are valuable tools.

Monitoring and evaluation can: Help you identify problems and their causes; Suggest possible solutions to problems; Raise questions about assumptions and strategy;

Push you to reflect on where you are going and how you are getting there; Provide you with information and insight; Encourage you to act on the information and insight; Increase the likelihood that you will make a positive development difference.

18

SESSION V: DEVELOPING INDICATORS

At the end of this session, the participants will be able to develop indicators Time: About 2:30 hour Material: flipcharts, markers, worksheets

Step 1: The session can be started with the question, “What is indicator?”

After discussion for a while, distribute Worksheet 5 – 1: Indicators “Indicators are Units of measurement that tell us about IMPACT, EFFECT and OUTPUT to judge the effectiveness of the Project Indicators should be:

Ethical Useful Scientifically Robust Reliable Accessible

Good indicators should be 1) practical, 2) independent, 3) measurable and 4) targeted.

19

Give the below examples and ask them if they are good indicators or not. Yes No

Mortality rate x

Health education materials

x

# of clinics offering STD treatment x

% of clients satisfied with quality of STD treatment x

# of community health workers trained x

% of student who score at least 90% on final exam x

Sexual violence counseling x

Step 2: Ask the groups to read Worksheet 5 – 2: the causal pathway

hypothesis of Ah Lin Yaung. And develop indicators to measure outputs, effects and impact(s). (Group work 15 – 30 minutes and 5 minutes’ presentation for each group)

e.g. Impact indicators

Reduced HIV transmission rate among young people in targeted areas (compare with a baseline data) Effect indicators

Increased use of disposable needles among drug users (survey and compare with a baseline data)

Increased number of young people who understand how they can contract HIV/AIDS Output indicators # of health education sessions organized by peer educators # of trained peer educators in targeted community

Developing efficiency, effectiveness and impact indicators1

• Young people don’t have HIV/AIDS health education.

Step 4: Identify the problem situation you are trying to address. The

following might be problems for Ah Lin Yaung:

• Needle sharing among young people • People living with HIV/AIDS (PLHA) do not get access to

health care services.

1 Taken from CIVICUS Monitoring and Evaluation

20

• HIV transmission rate among young people is high. Step 5: Develop the vision (impact) of the project you want to achieve. This

will give you impact indicators. For Ah Lin Yaung, the end result (vision) can be: “Healthy young people in targeted areas”.

What will tell you that the vision has been achieved? What signs

will you see that you can measure that will “prove” that the vision has been achieved?

For example: if your vision was that young people in the

community would be healthy, then you can use health indicators to measure how well you are doing. Do fewer young people die of HIV/AIDS? Has the HIV/AIDS infection rate been reduced? If you can answer “yes” to these questions then progress is being made.

Step 6: Develop a process vision for how you want things to be achieved.

This will give you process indicators.

If, for example, you want success to be achieved through community efforts and participation, then your process vision might include things like peer educators from the community trained and providing health education, counseling session and other necessary healthcare services used by all; targeted community gets involved in all planned activities of the project, and so on.

Step 7: Develop indicators for effectiveness.

For example, if you believe that you can reduce HIV/AIDS transmission rate by using disposable needles and condoms among young people, then you need indicators that show you have been effective in promoting use of disposable needles and condoms e.g. evidence from a survey among targeted communities compared with a baseline survey.

Step 8: Develop indicators for your efficiency targets

Here you can set indicators such as: planned health education sessions are run within the stated timeframe, costs for needles, condoms and health sessions are kept to a maximum of US$ ___ per participant, no more than ____ hours in total of staff time to be spent on organizing a health education session, no complaints about health education session etc.

21

With this framework in place, you are in a position to monitor and evaluate efficiency, effectiveness and impact

22

WORKSHEET 5 – 1

INDICATORS

“Indicators are Units of measurement that tell us about IMPACT, EFFECT and OUTPUT to judge the effectiveness of the Project

Indicators should be:

Ethical Useful Scientifically Robust Reliable Accessible

Good indicators should be 1) practical, 2) independent, 3) measurable and 4) targeted.

Please give the below examples and ask them if they are good indicators or not. Yes No

Mortality rate

Health education materials

# of clinics offering STD treatment

% of clients satisfied with quality of STD treatment

# of community health workers trained

% of student who score at least 90% on final exam

Sexual violence counseling

23

WORKSHEET 5 – 2

DEVELOPING OUTPUT/EFFECT/IMPACT INDICATORS Causal hypothesis Ah Lin Yaung (ALY) will identify vulnerable groups among young population in targeted areas and train peer educators to give health education among vulnerable groups. At the same time, ALY will provide needles for drug-users and distribute condoms among high-risked groups. ALY will open counseling center for People Living with HIV/AIDS (PLHA) and mobile clinics to give treatments for opportunistic infections. These activities can change knowledge and attitudes of young people and PLHA to prevent getting HIV or spreading HIV to other people. Accessibility of needles and condoms can also change the practice and behaviors of young people for safer sex. These changes will then contribute to reduce HIV transmission rate among young people in targeted areas.

e.g.

Impact indicators Reduced HIV transmission rate among young people in targeted

areas (compare with a baseline data) Effect indicators

Increased use of disposable needles among drug users (survey and compare with a baseline data)

Increased number of young people who understand how they can contract HIV/AIDS Output indicators # of health education sessions organized by peer educators # of trained peer educators in targeted community

24

WORKSHEET 5 – 3

DEVELOPING EFFICIENCY, EFFECTIVENESS AND IMPACT INDICATORS2

1) Identify the problem situation you are trying to address. The following might be problems for Ah Lin Yaung:

• Young people don’t have HIV/AIDS health education. • Needle sharing among young people • People living with HIV/AIDS (PLHA) do not get access to

health care services. • HIV transmission rate among young people is high.

2) Develop the vision (impact) of the project you want to achieve. This will

give you impact indicators. For Ah Lin Yaung, the end result (vision) can be: “Healthy young people in targeted areas”.

What will tell you that the vision has been achieved? What signs will you see that you can measure that will “prove” that the vision has been achieved?

For example: if your vision was that young people in the

community would be healthy, then you can use health indicators to measure how well you are doing. Do fewer young people die of HIV/AIDS? Has the HIV/AIDS infection rate been reduced? If you can answer “yes” to these questions then progress is being made.

3) Develop a process vision for how you want things to be achieved. This

will give you process indicators.

If, for example, you want success to be achieved through community efforts and participation, then your process vision might include things like peer educators from the community trained and providing health education, counseling session and other necessary healthcare services used by all; targeted community gets involved in all planned activities of the project, and so on.

4) Develop indicators for effectiveness.

2 Taken from CIVICUS Monitoring and Evaluation

25

For example, if you believe that you can reduce HIV/AIDS transmission rate by using disposable needles and condoms among young people, then you need indicators that show you have been effective in promoting use of disposable needles and condoms e.g. evidence from a survey among targeted communities compared with a baseline survey.

5) Develop indicators for your efficiency targets

Here you can set indicators such as: planned health education sessions are run within the stated timeframe, costs for needles, condoms and health sessions are kept to a maximum of US$ ___ per participant, no more than ____ hours in total of staff time to be spent on organizing a health education session, no complaints about health education session etc.

With this framework in place, you are in a position to monitor and evaluate efficiency, effectiveness and impact

26

SESSION VI: PLANNING FOR MONITORING AND EVALUATION

At the end of this session, the participants will know monitoring and evaluation process and how to plan monitoring and evaluation for a project. Time: About 4 - 5 hours Material: flipcharts, markers, Worksheets

Designing a monitoring system Step 1: The facilitator can start the session as follows:

Monitoring and evaluation should be part of your planning process. It is very difficult to go back and set up monitoring and evaluation systems once things have begun to happen. You need to begin gathering information about performance and in relation to targets from the word go. The first information gathering should, in fact, take place when you do your needs assessment. This will give you the information you need against which to assess improvements over time.

Indicators developed in previous session can also provide the framework for your monitoring and evaluation system. They tell you what you want to know and the kinds of information it will be useful to collect.

27

Step 2: Revisit causal hypothesis, “Ah Lin Yaung”. Ah Lin Yaung will identify vulnerable groups among young population in targeted areas and train peer educators to give health education among vulnerable groups. At the same time, Ah Lin Yaung will provide needles for drug-users and distribute condoms among high-risked groups. Ah Lin Yaung will open counseling center for PLHA and mobile clinics to give treatments for opportunistic infections. These activities can change knowledge and attitudes of young people and PLHA to prevent getting HIV or spreading HIV to other people. Accessibility of needles and condoms can also change the practice and behaviors of young people for safer sex. These changes will then contribute to reduce HIV transmission rate among young people in targeted areas. Develop monitoring plan - 1 Step 3: Introduce the concepts of efficiency, effectiveness and impact

Efficiency: Whether the resources used for the activities are appropriate in terms of output or they are cost-effective. For example, training program that train 50 peer educators costs $10000. Is it efficient or cost-effective? Effectiveness: It is the measure of the extent of the achievement of the project against the target objectives. For example, we plan to improve the qualifications of all high school teachers in particular area, did we succeed after project completion? Impact: It measures whether or not project implementation has contributed to the change in the situation of the problem it was trying to address. For example, high HIV transmission rate among young people in the community was the main problem before and our project really made a difference to this situation that is reduction in HIV transmission rate among young people in the community.

Explain to the participants that a monitoring system needs to cover

all three. In the previous session, the groups have already developed a list of indicators for each of the three aspects.

Step 4: Ask the participants to develop a monitoring system of HIV/AIDs

project of “Ah Lin Yaung” covering all these three. ( 30 minutes’ discussion and 5 minutes’ presentation) Efficiency: e.g.

i. How much does it cost to organize a training workshop? ii. How many participants join?

28

iii. How much does it cost per participant? iv. Is it cost effective? v. Whether the resources used for the activities are appropriate

in terms of output or they are cost-effective.

All these information can be tracked easily in a tracking sheet. No. Date Training

topic No. participant

Duration Total training cost Remark

M F

Effectiveness: For example, we plan to improve the qualifications of all high school teachers in particular area, did we succeed after project completion? You can develop indicators to monitor the progress to achieve this target.

Impact: For example, you might have an indicator of impact which is that

“safer sex options are chosen” as an indicator that “young people are now making informed and mature lifestyle choices”. The variables that might affect the indicator include:

Age Gender Religion Urban/rural Economic category Family environment Length of exposure to your project’s initiative Number of workshops attended.

By keeping the right information you will be able to answer many questions such as:

Does age make a difference to the way our message is received? Does economic category i.e. do young people in richer areas

respond better or worse to the message or does it make no difference?

Does the number of workshops attended make a difference to the impact?

29

Answers to these kinds of questions enable a project or organization to make decisions about what they do and how they do it, to make informed changes to programmes, and to measure their impact and effectiveness. Answers to questions such as:

Do more people attend sessions that are organized well in

advance? Do more schools participate when there is no charge? Do more young people attend when sessions are over weekends

or in the evenings? Does it cost less to run a workshop in the community, or to

bring people to our training centre to run the workshop? Step 5: Decide how you will collect the information you need and where it

will be kept (on computer, in manual files). Step 6: Decide how often you will analyze the information – this means

putting it together and trying to answer the questions you think are important.

Develop monitoring plan - 2 Step 7: Or simply, distribute worksheet 6-2: Develop monitoring plan 2,

you can look at the table information needed and design monitoring plan.

Planning for Evaluation Step 8: The facilitator explains to the participants: Similar to monitoring plan, we should start evaluation plan right from the beginning. When we do needs assessment, we will collect data about economic, health, education and social situation of the community. They will become baseline data for us and to measure the impact or achievement of objectives, we can compare these data with those that we collect after or during the implementation of the project. Evaluation involves: Looking at what the project or organization intended to achieve – what

difference did it want to make? What impact did it want to make? Assessing its progress towards what it wanted to achieve, its impact

targets. Looking at the strategy of the project or organization. Did it have a

strategy? Was it effective in following its strategy? Did the strategy work? If not, why not?

30

Looking at how it worked. Was there an efficient use of resources? What were the opportunity costs of the way it chose to work? How sustainable is the way in which the project or organization works? What are the implications for the various stakeholders in the way the organization works.

In an evaluation, we also look at efficiency, effectiveness and impact. Step 9: Ask the participants to share with the class the evaluation types they

know. ( 5 – 10 minutes) Then distribute the names of different types of evaluation and definitions to match each other. Distribute Worksheet 6 – 3: different evaluation types to the participants. The facilitator can go through the table together with the participants and discuss about different types of evaluation.

Step 10: Distribute worksheet 6 – 4: Case study and ask the groups to discuss

among themselves first. ( 15 – 20 minutes) Step 11: Ask the groups to fill the form included in worksheet 6 – 5: Evaluation

Basics. ( 15 – 20 minutes) Step 12: Explain to the participants that normally, the first step in evaluation is

to develop Terms of Reference for evaluation. Share worksheet 6 – 6: format of ToR included and go through the format together explaining the points they want to clarify more.

Group work: Each group needs to develop Terms of Reference for evaluation of Youth’s Energy’s projects. (45 to 60 minutes)

31

WORKSHEET 6 – 1

DEVELOP MONITORING PLAN – 1 Efficiency: Whether the resources used for the activities are appropriate in terms of output or they are cost-effective. For example, training program that train 50 peer educators costs $10000. Is it efficient or cost-effective?

e.g. i. How much does it cost to organize a training workshop?

ii. How many participants join? iii. How much does it cost per participant? iv. Is it cost effective? v. Whether the resources used for the activities are appropriate in terms

of output or they are cost-effective. All these information can be tracked easily in a tracking sheet. No. Date Training

topic No. participant

Duration Total training cost Remark

M F Effectiveness: It is the measure of the extent of the achievement of the development project against the target objectives. For example, we plan to improve the qualifications of all high school teachers in particular area, did we succeed after project completion? We can monitor the quality of high school teachers regularly using different indicators. Impact: It measures whether or not project implementation has contributed to the change in the situation of the problem it was trying to address. For example, high HIV transmission rate among young people in the community was the main problem before and our project really made a difference to this situation that is reduction in HIV transmission rate among young people in the community. We can monitor the progress to achieve targeted impact comparing baseline data with current situation.

32

WORKSHEET 6 – 2

DEVELOP MONITORING PLAN – 2 What do we want to know? This includes looking at indicators for both internal issues and external issues. There are three basic types of monitoring commonly used: Management/administration staff/personnel vehicles supplies Finance project budget and expenditure staff salaries cash flow analysis Project activities project inputs – budget, equipment, key staff needed results of activities – project outputs/outcomes/impact (using indicators) the way the project is managed situation or context – policy environment, political situation The table below shows the information needed to collect for monitoring programs and project. INFORMATION FOR MONITORING PROGRAM OPERATIONS3

CATEGORIES OF INFORMATION

WHAT TO MONITOR

WHAT RECORDS TO KEEP

WHO COLLECTS DATA

WHO USES DATA

HOW TO USE INFORMATION

WHAT DECISIONS CAN BE MADE

1. Work plan

Activities

• Timing of activities • Availability of personnel, resources

•Monthly /quarterly work plans • Work schedules

• Project Manager •Supervisors

• Project Manager • Donor agency

• Ensure staff and other resources are available

• Reschedule activities and deployment of resources as needed

2.Costs and

Budgeted amounts,

• Ledger of expenditure

• Financial officer/

• Project Manager

• Ensure funds are

• Authorize expenditures

3 Source: The Family Planning Manager’s Handbook. Editors: James A. Wolff, Linda J. Suttenfield, Susanna C. Binzen. Management Sciences for Health, Boston.

33

Expenditure

funds on hand and expenditures • Balance in budget by approved cost categories

s by budget category • Receipts • Bank transactions • Reports to donor

accountant

• Auditor • Donor agency

available to execute activities • Ensure compliance w/funding regulations • If fee for service, determine fee structure

• Make budget and project revisions • Determine need for other funding sources

3.Staff and Supervision

• Knowledge, attitudes and skills of staff • Educational level of staff • Salaries and benefits • Job performance

• Performance reviews • Job descriptions • Resumes of staff • Feedback from training attended

• Supervisors • Personnel director • Trainers

•Supervisor • Project Manager • Personnel Director

• Motivate staff and resolve employment problems • Advise staff on career

• Placement • Training needs • Promotions • Disciplinary action

2. Commodities

• Stock • Ordering and shipment status • Procurement regulations

• Stock registers • Invoices • Field worker reports

• Logistics manager

• Project Manager • Donor agency

• Ensure availability of commodities in stock and distribution to field • Ensure good condition Commodities

• Quantity to order • When to order • Amount to keep in reserve for emergency

3. Results

• No. and type of services provided/ commodities dispensed • Characteristics of persons served/ educated

• Client cards/forms • Clinic registers • Field worker reports

• CBD Workers • Clinic nurse • Field supervisors

• Field supervisor • Project Manager • Donor agency

• Ensure objectives are realistic • Assess quality of services provided • Assess appropriateness of services

• Revise objectives • Retrain staff • Revise IEC strategy • Revise project strategy and approach

34

WORKSHEET 6 – 3

DIFFERENT TYPES OF EVALUATION Self-evaluation: This involves an organization or project holding up a

mirror to itself and assessing how it is doing, as a way of learning and improving practice. It takes a very self-reflective and honest organization to do this effectively, but it can be an important learning experience.

Participatory evaluation:

This is a form of internal evaluation. The intention is to involve as many people with a direct stake in the work as possible. This may mean project staff and beneficiaries working together on the evaluation. If an outsider is called in, it is to act as a facilitator of the process, not an evaluator.

Rapid Participatory Appraisal:

Originally used in rural areas, the same methodology can, in fact, be applied in most communities. This is a qualitative way of doing evaluations. It is semi-structured and carried out by an interdisciplinary team over a short time. It is used as a starting point for understanding a local situation and is a quick, cheap, useful way to gather information. It involves the use of secondary data review, direct observation, semi-structured interviews, key informants, group interviews, games, diagrams, maps and

35

calendars. In an evaluation context, it allows one to get valuable input from those who are supposed to be benefiting from the project. It is flexible and interactive.

External evaluation:

This is an evaluation done by a carefully chosen outsider or outsider team.

Interactive evaluation:

This involves a very active interaction between an outside evaluator or evaluation team and the organization or project being evaluated. Sometimes an insider may be included in the evaluation team.

36

WORKSHEET 6 - 4

CASE STUDY Introduction to “Youth’s Energy4

• Annual organization of a summer youth camp (sessions held over a period of 8 weeks each summer)

” The CBO “Youth’s Energy (Lu Nget Swan In)” is an organization formed with young volunteers that has operated in Mawlamyaing, Mon State since 2000. They started the organization with five young volunteers who were newly graduates at that time. They started giving free tuition to the tenth standard students in their ward. Most of the volunteers are the 10th standard passed students waiting to join universities. They receive all of their funding from donor grants, and since their budgets reached a peak in 2007, have begun to see a gradual decline in donor aid. They have a small staff of 6 full-time employees, and 2 part-time employees who work 15-20 hours per week. They have more than 20 volunteers from different areas of Mawlamyaing. Youth’s Energy provides recreational and educational activities to youth from the ages of approx. 10 – 18 years old. They make an effort to attract and welcome youth from all backgrounds and experiences. They currently have 4 main projects:

• Management of a Youth Center in the center of town, which includes an old gymnasium, 3 classroom-style all-purpose rooms, and a small room dedicated to computer use training, with 9 donated computers.

• Sponsorship of an adolescent reproductive health educational course module and peer education certification program that they coordinate and teach in cooperation with 4 local schools

• Organization of after-school sports and recreation activities (held at the youth center)

2009 has been a difficult year for the NGO. They have recently learned that their primary donor is cutting funding by 50% for 2010. While they were devastated to receive this news, they also recognized that they were fortunate that the donor did not withdraw completely, as has happened to other NGOs in the region. While they have begun to investigate various fundraising options, they realize that they are going to have to make some decisions about their program priorities and overall organization. Youth’s Energy director,Ko Thiha , has been speaking to the deputy director Ma Nwe about his concerns, and they decided to hold a meeting one evening after

4 Adapted from IRC Bosnia Evaluation training manual

37

work with all of the staff to clearly and openly explain their current and projected future financial status. They also wanted to use that time to let the other employees express their concerns, as they recognized that everyone was feeling very vulnerable and concerned about the future of the organization. The meeting was difficult, but worthwhile. Everyone knew that times were tough, but it was good to pull the whole team together to discuss it. A general consensus emerged that in order to survive, the organization would have to do two things. First, it would have to more actively work on alternative fundraising efforts in order to seek contributions to replace the lost donor funding. Second, they realized that they needed to look at their current projects, and determine which projects should be continued in the future, and which should be radically refocused or possibly even dropped entirely. Everyone knew that one project in particular needed attention: the Youth Center. The Youth Center had been a focal point of the organization for over 3 years, and has been a showcase of the Youth’s Energy initiative. Its prime location in the center of town made it accessible to many people, and also served as a great way to increase the organization’s visibility and reputation as a CBO that was active and involved. However, over the past 12 months several of the circumstances surrounding the center had changed, making it a drain on the CBO’s time and financial resources. For the first 2 years they were able to use the facility virtually free of charge, as the township authorities provided the space rent-free. Youth’s Energy only had to pay for the utilities and general maintenance and upkeep. However, one year ago the authorities asked the organization to begin paying a nominal rent for the facility, with a rent payment schedule that was tapered over a 4-year period so that it would gradually be rented out at standard market rates. The rent for the first year was a shock to their budget, but was reasonable and management. However, the increased rent for the next year will be a serious strain. Additionally, the facility is in need of some repairs that can not be put off much longer. Some are simple, such as fresh paint and a good cleaning, and could probably be done with help from volunteers in the community. However, others will necessitate professional expert support, such as a re-wiring of the electrical system, repair of a leaky roof, and purchase of new desks, chairs and tables that are more suitable to the needs of people using computers and other specialized equipment. Everyone recognized that, while long a favorite program of the staff, the Youth Center was beginning to drain resources form other initiatives, in particular, the adolescent reproductive health (ARH) training module, and related peer education certification program. Over the past few years it was easy to demonstrate action to donors through the Youth Center projects: they could

38

easily report on how many activities were held, how many kids participated, etc. It was tangible activity. Since the ARH training was not the same kind of tangible concrete activity, and was a lower profile project, it was often easily “lost in the shuffle” during budget development. When confronted with the new budget and resource situation, a few of the team members reflexively suggested that the Youth Center be maintained and the ARH module dropped, in order to shift and re-focus resources and energy. However, when Ma Nwe asked them to explain why they thought this was a good idea, no one in the room could provide an explanation based on facts or specific data. They suddenly realized that they did not know what their users or people in the community thought about either of these efforts, and that they didn’t know which project was more successful or had had a greater impact on the community. These were the questions that needed to be asked and answered before decision on the future direction of the organization could be made. Ko Thiha pointed out that it was time to begin an evaluation of the NGO’s projects, starting with the Youth Center and the ARH module. The organization has done basic evaluation work in the past, but always with guidance or staff support from the donor organization requesting the evaluation. They provided annual reports, but these were more of a summary of activities and events rather than a rigorous evaluation of these projects. Outside consultants had come in twice over the past 6 years to meet with the staff and assess their work, but while the organization had welcomed the consultants and provided them with information, they had not been closely involved in the evaluation process, and had been limited to simply providing information and answering questions. They realized that they needed to learn the basic of evaluation themselves, in order to conduct the project evaluations themselves. They began to gather evaluation training materials from some NGO resource centers, and began a dual effort: training of the staff in basic evaluation techniques, and applying these skills to the evaluation of the Youth Center and ARH projects.

39

WORKSHEET 6 – 5

EVALUATION BASICS 1. Read the case study carefully and break it down into its component parts. Fill

in the sample table, adding more space if needed.

Program Project Activity

2. What are some of the factors motivating “Youth’s Energy” to undertake an

evaluation? 3. Think about a case in which the organization had to make some difficult

decisions. Did you use data gathered from evaluations to make that decision? Why or why not?

4. Is there an internal process within the organization that you think could

benefit from evaluation? Why?

40

WORKSHEET 6 – 6

TERMS OF REFERENCE

Terms of Reference usually include: Background: This is background to the project or organization, something about the problem identified, what you do, how long you have existed, why you have decided to do an evaluation. Purpose: Here you would say what it is the organization or project wants the evaluation to achieve. The purpose of an evaluation is the reason why you are doing it. It goes beyond what you want to know to why you want to know it. It is usually a sentence or, at most, a paragraph. It has two parts:

• What you want evaluated; • To what end you want it done.

Examples of an evaluation purpose: To provide the organization with information needed to make decisions about the future of the project. To assess whether the organization/project is having the planned impact in order to decide whether or not to replicate the model elsewhere.

41

To assess the programme in terms of effectiveness, impact on the target group, efficiency and sustainability in order to improve its functioning. The purpose gives some focus to the broad evaluation process. Key evaluation questions: What the central questions are that the evaluation must address. The key evaluation questions are the central questions you want the evaluation process to answer. They are not simple questions. You can seldom answer “yes” or “no” them. A useful evaluation question is:

• Thought provoking • Challenges assumptions. • Focuses inquiry and reflection. • Raises many additional questions.

Some examples of key evaluation questions related to a project purpose: The purpose of the evaluation is to assess how efficient the project is in delivering benefits to the identified community in order to inform Board decisions about continuity and replicability. Key evaluation questions: Who is currently benefiting from the project and in what ways? Do the inputs (in money and time) justify the outputs and, if so/if not, on what basis is this claim justified? What would improve the efficiency, effectiveness and impact of the current project? What are the lessons that can be learned from this project in terms of replicability? Specific objectives: What specific areas, internal and/or external, you want the evaluation to address. So, for example, you might want the evaluation to include a review of finances, or to include certain specific programme sites. Methodology: should provide a broad framework for how the project or organization wants the work of the evaluation done. You might also suggest the kinds of techniques you would like the evaluation team to use. Most terms of reference will ask for some combination of these but they may also specify how they want the evaluation team to get input from stakeholder groups. Here too one would expect to find some indication of reporting formats: Will all reporting be written? Will the team report to management, or to all staff, or to staff and Board and beneficiaries? Will there be interim reports or only a final

42

report? What sort of evidence does the organization or project require to back up evaluator opinions? Who will be involved in analysis? Logistical issues: These would include timing, costing, requirements of team composition and so on.

43

SESSION VII: GATHERING INFORMATION FOR MONITORING AND EVALUATION

At the end of this session, the participants will understand how different information gathering methods can be used to get the information they want to get for M&E purposes Time: About 1:15 minutes Material: flipcharts, markers, Worksheets

Step 1: The facilitator can start the session by explaining to the participants as

follows: Data collection is possibly the most important part of monitoring and evaluation, as you must have reliable and concrete information available to you in order to assess needs, measure progress, or determine cause and effect. Detailed data collection and information-gathering techniques are included in “Social research tools training manual”. So the ones who have joined social research skills training will be familiar with many of these techniques.

Step 2: Distribute worksheet 7 – 1: “Different social research tools” and discuss together. (The facilitator needs to read social research tools training manual in advance if they are not so familiar with social research skills.)

Step 3: Ask the participants if anyone knows the words, “primary and

secondary sources”. Then explain to all the participants that

44

Primary sources are first-hand, original sources that provide raw, un-analyzed or un-interpreted facts and information. Examples of primary sources include: Training evaluation forms, Beneficiary satisfaction forms, Organizational assessment forms Secondary sources are exactly what they sound like- data from sources that refer to or interpret information gathered by another researcher. Examples of secondary sources include: A newspaper article on the problem of AIDS in Burma/Myanmar

EIU report on Burma/Myanmar Step 4: Ask each group to pick up one topic OR the facilitator can assign one

topic to each group and ask them how they will gather data on the following topics identifying primary and secondary sources. (30 minutes’ discussion and 5 minutes for each presentation)

• How the role of women in the community has changed in the past five

years • The process through which drug addicts successfully kick the habit • The most effective media to give health education among young people • The role of monasteries in providing education services to young people

during Burmese kingdoms • Reading habit among young people in urban areas

Step 5: After presentation, the facilitator can end the session by the below statement. They will probably find that they want or need to use more that one data collection technique. This is a standard practice, and is referred to as adopting a mixed method approach. A mixed method approach combines data gathered through different techniques, bringing all of the information together for the final review and analysis. The data is then analyzed and interpreted collectively. Bringing information together from a variety of sources is a good way to confirm or validate your data. This process is called data triangulation, and it can help to ensure the validity of the conclusions. If data gathered through interview process yields similar feedback as data gathered through a questionnaire distribution or through basic research and document review, then they can have more confidence in the results. If the data gathered through one method is very different than data gathered through another, then they should analyze your results with caution, possibly even obtaining additional data to try to determine the reasons for the discrepancy.

45

WORKSHEET 7 – 1

DIFFERENT SOCIAL RESEARCH TOOLS Tool Description Usefulness Disadvantages Interviews These can be structured,

semi-structured or unstructured. They involve asking specific questions aimed at getting information that will enable indicators to be measured. Questions can be open-ended or closed (yes/no answers). Can be a source of qualitative and quantitative information.

Can be used with almost anyone who has some involvement with the project. Can be done in person or on the telephone or even by e-mail. Very flexible.

Requires some skill in the interviewer.

Key informant interviews

These are interviews that are carried out with specialists in a topic or someone who may be able to shed a particular light on the process.

As these key informants often have little to do with the project or organization, they can be quite objective and offer useful insights. They can provide something of the “big picture” where people more involved may focus at the micro (small) level.

Needs a skilled interviewer with a good understanding of the topic. Be careful not to turn something into an absolute truth (cannot be challenged) because it has been said by a key informant.

Questionnaires These are written questions that are used to get written responses which, when analyzed, will enable indicators to be measured.

This tool can save lots of time if it is self-completing, enabling you to get to many people. Done in this way it gives people a feeling of anonymity and they may say things they would not say to an interviewer.

With people who do not read and write, someone has to go through the questionnaire with them which means no time is saved and the numbers one can reach are limited. With questionnaires, it is not possible to explore what people are saying any further. Questionnaires are also over-used and people get tired of completing them. Questionnaires must be piloted to ensure that questions can be understood and cannot be misunderstood. If the questionnaire is complex and will need computerized analysis, you need expert help in designing it.

Focus groups In a focus group, a group of about six to 12 people are interviewed together by a skilled interviewer/facilitator with a carefully

This can be a useful way of getting opinions from quite a large sample of people.

It is quite difficult to do random sampling for focus groups and this means findings may not be generalised. Sometimes people

46

structured interview schedule. Questions are usually focused around a specific topic or issue.

influence one another either to say something or to keep quiet about something. If possible, focus groups interviews should be recorded and then transcribed. This requires special equipment and can be very time-consuming.

Community meetings This involves a gathering of a fairly large group of beneficiaries to whom questions, problems, situations are put for input to help in measuring indicators.

Community meetings are useful for getting a broad response from many people on specific issues. It is also a way of involving beneficiaries directly in an evaluation process, giving them a sense of ownership of the process. They are useful to have at critical points in community projects.

Difficult to facilitate – requires a very experienced facilitator. May require breaking into small groups followed by plenary sessions when everyone comes together again.

Visual/audio stimuli These include pictures, movies, tapes, stories, role plays, photographs, used to illustrate problems or issues or past events or even future events.

Very useful to use together with other tools, particularly with people who cannot read or write.

You have to have appropriate stimuli and the facilitator needs to be skilled in using such stimuli.

Rating scales This technique makes use of a continuum, along which people are expected to place their own feelings, observations etc. People are usually asked to say whether they agree strongly, agree, don’t know, disagree, disagree strongly with a statement. You can use pictures and symbols in this technique if people cannot read and write.

It is useful to measure attitudes, opinions, perceptions.

You need to test the statements very carefully to make sure that there is no possibility of misunderstanding. A common problem is when two concepts are included in the statement and you cannot be sure whether an opinion is being given on one or the other or both.

Critical event/incident Analysis

This method is a way of focusing interviews with individuals or groups on particular events or incidents. The purpose of doing this is to get a very full picture of what actually happened.

Very useful when something problematic has occurred and people feel strongly about it. If all those involved are included, it should help the evaluation team to get a picture that is reasonably close to what actually happened and to be able to diagnose what went wrong.

The evaluation team can end up submerged in a vast amount of contradictory detail and lots of “he said/she said”. It can be difficult not to take sides and to remain objective.

Participant observation This involves direct observation of events, processes, relationships and behaviors. “Participant” here implies

It can be a useful way of confirming, or otherwise, information provided in other ways.

It is difficult to observe and participate. The process is very time-consuming.

47

that the observer gets involved in activities rather than maintaining a distance.

Self-drawings This involves getting participants to draw pictures, usually of how they feel or think about something.

Can be very useful, particularly with younger children.

Can be difficult to explain and interpret.

Primary sources are first-hand, original sources that provide raw, un-analyzed or un-interpreted facts and information. Examples of primary sources include: Training evaluation forms, Beneficiary satisfaction forms, Organizational assessment forms Secondary sources are exactly what they sound like- data from sources that refer to or interpret information gathered by another researcher. Examples of secondary sources include: A newspaper article on the problem of AIDS in Burma/Myanmar

EIU report on Burma/Myanmar Mixed method approach They will probably find that they want or need to use more that one data collection technique. This is a standard practice, and is referred to as adopting a mixed method approach. A mixed method approach combines data gathered through different techniques, bringing all of the information together for the final review and analysis. The data is then analyzed and interpreted collectively. Bringing information together from a variety of sources is a good way to confirm or validate your data. This process is called data triangulation, and it can help to ensure the validity of the conclusions. If data gathered through interview process yields similar feedback as data gathered through a questionnaire distribution or through basic research and document review, then they can have more confidence in the results. If the data gathered through one method is very different than data gathered through another, then they should analyze your results with caution, possibly even obtaining additional data to try to determine the reasons for the discrepancy.

48

SESSION VIII: DATA ANALYSIS

At the end of this session, the participants will understand some basic methods for data analysis, breaking analysis down into qualitative and quantitative data analysis methods Time: About 1:30 hours Material: flipcharts, markers, Worksheets

Step 1: Ask the participants to go around the class and ask them to

interview at least 10 friends the following questions. (15 – 20 minutes)

1) How old are you? (don’t need to ask exact age) (Under 20), (within 20 – 30 years old), (30 – 40 years old), (40 years above) 2) Do you like reading? If yes, continue the following questions. If

no, skip the questions 4 and 5. 3) How many hours a week do you spend time for reading? 4) What kind of books do you like to read? 5) What do you do if you have free time? 6) How many hours a day do you spend to do what you like?

Step 2: Ask the participants to review and analyze the responses like the

examples.

1) Age range

49

Among (10) respondents, most of them are within the age of 20 to 30. ( ) are above 30 and ( ) is under 20.

OR Among ( ) respondents, ( ) are within the age of 20 – 30 years old and ( ) are in mid-30. ( ) are above 40.

2) Among ( ) people interviewed, ( ) like reading. They spend

( ) to ( ) hours per week for reading. Out of ( ), ( ) like reading newspapers and journals and ( ) read biographies and ( ) like to read variety of books. Most of the people who like reading are within the age of - years old.

3) Among ( ) respondents, ( ) don’t like reading. Among them, (

) like listening music and spend ( ) hours a week for listening music. ( ) like watching movie and ( ) like -----------------------.

Step 3: Explain to the participants that above review can identify some

trends among targeted population. Whether working with qualitative or quantitative data, the most basic part of data analysis will consist in identifying trends and patterns. Consider the following examples of potential trends that they might identify through its data analysis: A questionnaire revealed that people within the age of 30 to 40 liked reading and they liked to read magazines and journals most. Respondents with the age of 20 to 30 preferred music or shopping to reading. A focus group indicated that while young people liked the new brand of candy, older people thought it was too sweet. The study showed that there was no difference in music tastes among children from poor and wealthy families. They may be able to clearly and immediately identify some trends, though others may be more difficult to identify, and may necessitate detailed review and reflection. Step 4: The facilitator tells the participants if they have some basic

understanding of trends and response frequencies, they can begin to interpret and analyze this information. • What does the trend mean? • What does the data suggest?

e.g. The data indicates that 80% of the children who completed the World Religions course module had a favorable opinion of people from different ethnic groups, while only 23% of

50

the child respondents who did not complete the course noted a similar favorable opinion. As the children were all from the same school, the results suggest that the course increases children’s respect for diversity.

91% of the respondents noted that they would not feel personally affected by the elimination of ARH program, suggesting that the program could be cut without any serious repercussions in the community. When asked about their satisfaction with the training program of IRC ICB team, 14% of respondents rated interpreter’s translation skills as “Excellent,” 56% as “Satisfactory,” and 25% as “Poor.” (5% responded “No Opinion”.) This suggests that ICB team might need to consider providing training on translation skills of interpreters. However, tell them to be careful to consider other explanations for cause and effect. Later, these trends will be turned into a summary of general findings and recommendations.

51

WORKSHEET 8 – 1

DATA ANALYSIS

1) Go around the class and ask the following questions to your friends. i. How old are you?

(Under 20), (within 20 – 30 years old), (30 – 40 years old), (40 years above)

ii. Do you like reading? If yes, continue the following questions. If no, skip the questions 4 and 5.

iii. How many hours a week do you spend time for reading? iv. What kind of books do you like to read? v. What do you do if you have free time?

vi. How many hours a day do you spend to do what you like? 2) Review and analyze the responses like the following examples.

1) Age range Among (10) respondents, most of them are within the age of 20 to 30. ( ) are above 30 and ( ) is under 20.

OR Among ( ) respondents, ( ) are within the age of 20 – 30 years old and ( ) are in mid-30. ( ) are above 40.

2) Among ( ) people interviewed, ( ) like reading. They spend

( ) to ( ) hours per week for reading. Out of ( ), ( ) like reading newspapers and journals and ( ) read biographies and ( ) like to read variety of books. Most of the people who like reading are within the age of - years old.

3) Among ( ) respondents, ( ) don’t like reading. Among them, ( ) like

listening music and spend ( ) hours a week for listening music. ( ) like watching movie and ( ) like -----------------------.

1) Whether working with qualitative or quantitative data, the most basic part of

data analysis will consist in identifying trends and patterns. Consider the following examples of potential trends that you might identify through its data analysis:

A questionnaire revealed that people within the age of 30 to 40 liked reading and they liked to read magazines and journals most. Respondents with the age of 20 to 30 preferred music or shopping to reading. A focus group indicated that while young people liked the new brand of candy, older people thought it was too sweet.

52

The study showed that there was no difference in music tastes among children from poor and wealthy families. You may be able to clearly and immediately identify some trends, though others may be more difficult to identify, and you may necessitate detailed review and reflection. 2) They can begin to interpret and analyze this information.

• What does the trend mean? • What does the data suggest?

e.g. The data indicates that 80% of the children who completed the World Religions course module had a favorable opinion of people from different ethnic groups, while only 23% of the child respondents who did not complete the course noted a similar favorable opinion. As the children were all from the same school, the results suggest that the course increases children’s respect for diversity.

91% of the respondents noted that they would not feel personally affected by the elimination of ARH program, suggesting that the program could be cut without any serious repercussions in the community. When asked about their satisfaction with the training program of IRC ICB team, 14% of respondents rated interpreter’s translation skills as “Excellent,” 56% as “Satisfactory,” and 25% as “Poor.” (5% responded “No Opinion”.) This suggests that ICB team might need to consider providing training on translation skills of interpreters. However, be careful to consider other explanations for cause and effect. Later, these trends will be turned into a summary of general findings and recommendations.

53

SESSION IX: REPORTING

At the end of this session, the participants will understand how to present all the information and findings in the report. Time: About 1 hour Material: flipcharts, markers, Worksheets

54

Step 1: Distribute “Evaluation report format” to the participants. Like other reporting, “evaluation report” needs to do SPR analysis first.

• Subject (What do you want to focus?) • Purpose (What is the purpose?) • Reader (Who will read?)

Step 2: The facilitator can explain to the participants that in general, your

evaluation should include the following elements.

1. Executive Summary: Provide a short (2-3 paragraph) summary of the main issues evaluated, the main findings of the evaluation, and recommendations

2. Introduction: Explain the issue being evaluated and the reason for and goals of the evaluation (needs assessment, impact analysis, process evaluation, or a combination of the 3)

3. Methodology: Explain how you developed your questions, your data sources (respondents), and how you gathered your data

4. Findings: Explain the results of your data analysis 5. Recommendations: Explain your recommendations based on

the results of the evaluation 6. Appendices: You would include Terms of Reference, list of

people interviewed, questionnaires used, possibly a map of the area and so on.)

The facilitator can conclude the training by sharing the following information.

If the report is for public consumption, they may want to (or be required to) prepare a short presentation for the donors in which they orally summarize the main results and recommendations, possibly using a PowerPoint slide presentation to draw attention to key results.

55

WORKSHEET 9 – 1

EVALUATION REPORT FORMAT SPR analysis

• Subject (What do you want to focus?) • Purpose (What is the purpose?) • Reader (Who will read?)

EVALUATION REPORT FORMAT

1. Executive Summary: Provide a short (2-3 paragraph) summary of the main issues evaluated, the main findings of the evaluation, and recommendations

2. Introduction: Explain the issue being evaluated and the reason for and goals

of the evaluation (needs assessment, impact analysis, process evaluation, or a combination of the 3)

3. Methodology: Explain how you developed your questions, your data sources

(respondents), and how you gathered your data 4. Findings: Explain the results of your data analysis 5. Recommendations: Explain your recommendations based on the results of

the evaluation 6. Appendices: You would include Terms of Reference, list of people interviewed,

questionnaires used, possibly a map of the area and so on.)

If the report is for public consumption, they may want to (or be required to) prepare a short presentation for the donors in which they orally summarize the main results and recommendations, possibly using a PowerPoint slide presentation to draw attention to key results.

56

MONITORING AND EVALUATION TRAINING POST-TEST

Please answer the following questions to know your existing knowledge and skills. 1)What is Monitoring? 2) Why do we do Monitoring? 3) Who does Monitoring? 4) When do we do Monitoring? 5) What are the important things to look at when you monitor a project? 6) What are indicators? Please give an example of indicator. 7) What is evaluation? 8) For what purposes do you normally do evaluation? 9) When do you normally evaluate? 10) Write down the monitoring and evaluation tools you know.

57

REFERENCES 1. Perry, Valery,Sarajevo, Bosnia and Herzegovina, “Did it Work? The Basics of

Evaluation and Assessment for NGOs”, Sponsored by: International Rescue Committee (IRC): NGO Development Program

2. CIVICUS toolkits: Monitoring and Evaluation

http://www.civicus.org/toolkits/managing-csos-toolkits 3. Gosling, Louisa with Mike Edwards,Toolkits: A practical guide to planning,

monitoring, evalution and impact assessment, Save the Children – UK 2003 Pulbications