slpfe.org · web view3.1 by 2030, reduce the global maternal mortality ratio to less than 70 per...

Annex 6

Training Modules on Evaluation

for evaluation capacity building in Sri Lanka

Published by Sri Lanka Parliamentarians Forum for Evaluation (SLPFE) and Parliamentarians Forum for Development

Evaluation – South Asia (PFDE-SA) in partnership with the Center for Evaluation – University of Sri Jayewardenepura,

National Secretariat for Non-Governmental Organizations and UNICEF Sri Lanka.

January 2018

EVALUATION TRAINING

1-DAY MODULE

ContentModule Description.....................................................................................................................................2Session 1......................................................................................................................................................3Introduction................................................................................................................................................3Activity 1.1: Welcome and introduction......................................................................................................3Activity 1.2: Baseline assessment and participants’ getting to know each other........................................3The Sustainable Development Goals (SDGs) – new global context for evaluation practice........................4Activity 1.3: Intro to the SDGs.....................................................................................................................4Activity 1.4: Linking project outcomes to the SDGs.....................................................................................5Session 2......................................................................................................................................................6Evaluation basics.........................................................................................................................................6Activity 2.1: Mini-lecture “Evaluation as a management instrument”........................................................6Activity 2.2: Lecture “Evaluation criteria and questions”............................................................................6Activity 2.3: Group exercise “Development of evaluation questions for the Rehabilitation of St. Sebastian South Canal project”...................................................................................................................................7Session 3:.....................................................................................................................................................8Evaluation use.............................................................................................................................................8Activity 3: Analysis of factors that facilitate use..........................................................................................8Session 4:.....................................................................................................................................................8Diversity of evaluation approaches and related terminology. Evaluation standards and ethics.................8Activity 4.1: Interactive Lecture “Diversity of approaches and terms”........................................................9Closing.......................................................................................................................................................10Activity 4.2: Final assessment....................................................................................................................10Activity 4.3: Concluding discussion............................................................................................................10Annexes.....................................................................................................................................................11Annex 1: Household rules..........................................................................................................................11Annex 2: Baseline survey...........................................................................................................................12Annex 3: Case: Rehabilitation of St. Sebastian South Canal......................................................................13Annex 4: Criteria matrix.............................................................................................................................16Annex 5: Evaluation Matrix.......................................................................................................................17Annex 6: Case: Salvaging Sri Lanka’s small and medium businesses: how an evaluation led to rapid change.......................................................................................................................................................18Annex 7: Different types of evaluation......................................................................................................20Annex 8: AEA Evaluator Competencies.....................................................................................................22Annex 9: Final survey.................................................................................................................................25Annex 10: List of the Sustainable Development Goals..............................................................................26

Module DescriptionTarget audience:Public officials who are already using or planning to use results of project/program evaluations to inform their decisions.

Overall objective:This training shall contribute to the implementation of the broad vision that every public official in Sri Lanka should understand importance of evaluation and basics of evaluation practice.

The specific objective of this one-day training module is to give public officials who are already using or planning to use results of project/program evaluations the skills and knowledge necessary to understand and appreciate evaluation as a way to get high quality evidence to inform decisions.

Expected number of participants: 15-30 people.

Recommended room set up:3-6 rounds tables to seat 5 participants each. Set a plate with a number (from 1 to 3-6) to each table.

Agenda overview

Session Topics Activities

Session 1 Introduction.

The SDGs – new global context for evaluation practice.

Activity 1.1: Welcome and introduction.

Activity 1.2: Baseline assessment and participants’ getting to know each other.

Activity 1.3: Intro to the SDGs.

Activity 1.4: Linking project outcomes to the SDGs.

Session 2 Brief introduction to evaluation. What is evaluation?

Discussion on types of decisions and relevant information.

How evaluators think about projects.

Activity 2.1: Mini-lecture “Evaluation as a management instrument”.

Activity 2.2: Lecture “Evaluation criteria and questions”.

Activity 2.3: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”.

Session 3 Use of evaluation results. Activity 3: Analysis of factors that facilitate use of evaluation results.

Session 4 Diversity of evaluation approaches and related terminology.

Evaluation standards and ethics.

Activity 4.1: Lecture “Diversity of approaches and terms”.

Activity 4.2: Final assessment.

Closing. Activity 4.3: Concluding discussion.

Short version of agenda

If it is not possible to have a full day workshop, one-day module can be reduced to three sessions.

Session Topics Activities


SDGs – new global context for evaluation practice.






Discussion on types of decisions and relevant information.

How evaluators think about projects.

Activity 2.1: Mini-lecture “Evaluation as a management instrument”.



Session 3 Use of evaluation results. Activity 3: Analysis of factors that facilitate use of evaluation results.


Activity 4.3: Concluding discussion.

Session 1IntroductionActivity 1.1: Welcome and introductionDuration: 15 minutes

Objectives: Give participants general understanding of the training objectives and agenda; Agree on housekeeping rules.

Materials: Presentation

Prepared flipchart sheet with the list of housekeeping rules. (Leave some space to add rules suggested by participants).

Process:1. Welcome participants.2. Make a brief overview of the training objectives and agenda using presentation.3. Present a list of household rules (Annex 1). Invite participants to offer additional rules as

necessary. Have them agree on the rules.

Activity 1.2: Baseline assessment and participants’ getting to know each otherDuration: 30 minutes

Objectives: Get baseline data for evaluation of the training. Have participants to get to know each other.

Materials: Baseline survey (Annex 2) Flipchart Markers

Process:1. Ask participants to fill the baseline survey (printed copies of blank survey should be on each

table).2. Ask each table to calculate cumulative baselines scores of participants’ knowledge and skills in

areas of project/program management, monitoring and evaluation.3. Ask each table to have one person to make a quick presentation about their group starting with

the cumulative scores. As people present, take a note of groups scores on a flipchart. Keep the flipchart – you will need it for a final assessment exercise in the end of the workshop.

4. Review the resulting table of scores.5. In the end of activity collect the surveys.

The Sustainable Development Goals (SDGs) – new global context for evaluation practiceLearning objectives:As a result of this session, participants shall:

Understand connection between the MDGs to the SDGs; Understand why the SDGs are relevant to Sri Lanka and person’s job; Obtain basic knowledge of the SDGs; Understand how to link existing interventions to the SDGs; Understand role of evaluation in the SDGs.

Activity 1.3: Intro to the SDGsDuration: 15 minutes

Objectives:

Make participants understand that evaluation practice in embedded in a broader societal context.

Introduce the SDGs.

Materials: Presentation (including video. Video is also available at https://www.youtube.com/watch?

time_continue=2&v=5_hLuEui6ww) Flipchart Markers

Process:1. Start by asking participants if they have heard about the SDGs. Encourage people to share what

they already know. Record their answers on a flipchart.2. Make a short presentation on connections between evaluation practice and its societal context.

Key points for presentation:

Evaluation practice is embedded in broader organizational and social contexts and is affected by these contexts.

Project evaluation practice started in the 1960s in the US in the context of the War on Poverty and the Great Society programs when the federal government was heavily investing in social and educational programs. Senator Robert Kennedy, concerned that federal money would be misspent and not used to help disadvantaged children included evaluation clause into the Elementary and Secondary Education Act (ESEA) passed in 1965. ESEA required submission of an evaluation plan by local education agencies, and summary reports by state agencies. As a result, evaluation requirements became part of every federal grant given under ESEA. Early expectations were that evaluation would illuminate the causes of social problems and the clear and specific means with which to fix such problems.

In September 2015 adoption of the Agenda 2030 created a new global reality. The Agenda 2030 calls for evidence-based review of the progress towards SDGs and positions country-led evaluations as one of the tools contributing to this review.

3. Introduce video “Transition from MDGs to SDGs” developed by UNDP. (Note that one of the purposes of the video is to promote UNDP, but it gives a good overview of MDGs and SDGs).

4. Play video.5. Ask participants what they have learned from the video. Record answers on the flipchart.

Activity 1.4: Linking project outcomes to the SDGsDuration: 30 minutes

Objectives: Give participants a better understanding of the SDGs and respective targets.


Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3). Copies of the SDGs (Annex 10).

Process:1. Summing up the discussion concluding the previous activity, note that distinctive features of the

SDGs are their complexity and interconnectedness.2. Make a short presentation on complexity and interconnectedness of the SDGs.


Complexity: o 5 pillars – People, Planet, Prosperity, Peace, Partnership;o 17 goals and 169 targets;o Targets are related to what should be achieved and how it should be done.

Interconnectedness:o Planet, Prosperity, Peace, Partnership are necessary conditions for People with No

one left behind. The SDGs will be implemented through various projects and programs contributing to specific

goals and targets. The SDGs are not something completely new. Many existing projects and programs are

already contributing to the SDGs.

3. Present the task for group work. First, divide 17 SDGs between tables (3 SDGs per table. If there are six tables. one table will work on two goals - Goals 16 and 17).

4. Then explain that participants should review the case of the Project “Rehabilitation of St. Sebastian South Canal” implemented as part of the beautification of Colombo efforts and see if it contributes towards specific SDGs and respective targets. Each participant should have a copy of the case and of the SDGs.

5. Give groups 15 minutes to complete the task.6. Ask groups to present results of their work explaining their choices. Each group can present one

target at a time. Record goal and targets on a Flipchart. Go through all tables, then repeat the round until all targets identified by groups are listed.

7. To complete activity, ask participants what they have learned from the exercise.

Session 2Evaluation basicsActivity 2.1: Mini-lecture “Evaluation as a management instrument”Duration: 15 minutes

Objectives: Position evaluation within the management cycle.

Explain that defining value (worth, merit) is part of decision making process. Explain that professional evaluation makes valuing process explicit.


Process:1. Make a presentation.


Evaluation is part of the management cycle. Defining value (worth, merit) is part of decision making process. Professional evaluation makes valuing process explicit

2. Invite participants to ask questions and make comments about content of the mini-lecture.

Activity 2.2: Lecture “Evaluation criteria and questions”Duration: 30 minutes

Objectives: Give participants general understanding of the origin and purpose of evaluation criteria; Give participants general understanding of how evaluation questions come from evaluation

criteria. Introduce Evaluation matrix as a useful evaluation design tool.




Evaluation emerged in response to donors’ need to know if the project implemented with their fund was “good”.

The definition of “good” that emerged eventually in 1970s was that a good project should be relevant, effective, efficient, have sustainable results and have positive societal impact.

Hence OECD-DAC evaluation criteria of relevance, effectiveness, efficiency, sustainability and impact.

Evaluation criteria of relevance, effectiveness, efficiency, sustainability and impact are logically connected.

Key evaluation questions are directly linked with evaluation criteria. Key evaluation question shall be developed by people who commission evaluation and will use its’ results. Sub-questions shall be jointly developed by evaluators and commissioners. Other decisions about evaluation design shall be done by evaluators themselves.

Evaluation Matrix. Evaluation sub-questions reflect the interpretation of corresponding evaluation criteria in the

context of evaluated intervention. Integration of SDGs criteria with DAC criteria.

2. Open the floor for questions and comments from participants.

Activity 2.3: Group exercise “Development of evaluation questions for the Rehabilitation of St. Sebastian South Canal project”Duration: 45 minutes

Objectives: Give participants practical experience with selecting and interpreting evaluation criteria for an

intervention evaluation. Give participants practical experience with developing evaluation questions.

Materials: Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3) Criteria matrix (Annex 4) Evaluation Matrix (Annex 5)

Process:1. Explain the task:

Each group will have to decide which criteria shall be used for the final evaluation of the Rehabilitation of St. Sebastian South Canal project using Criteria matrix.

Based on the selected criteria each group shall then develop key evaluation questions. Selected criteria and key questions have to be put on a flipchart in a format of

Evaluation matrix. This will create a short ToR.

2. Give groups 30 minutes to complete the task.3. Ask groups to present their results to plenary.4. Then have a plenary discussion about how people feel about the exercise that they have just

completed and what challenges did they have.5. In the end ask all teams to put flipchart with their ToRs on the walls so that all participants could

see them.

Session 3:Evaluation useLearning objectives:As a result of this session participants shall understand factors that facilitate use of evaluation results.

Activity 3: Analysis of factors that facilitate useDuration: 90 minutes

Objectives: Make participants understand factors that facilitate use of evaluation results.

Materials: Case “Salvaging Sri Lanka’s small and medium businesses: how an evaluation led to rapid

change” (Annex 6) Flipcharts Markers


Working in groups, participants shall read the case “Salvaging Sri Lanka’s small and medium businesses: how an evaluation led to rapid change”.

Then, on flipcharts, groups shall record the factors that facilitated successful use of evaluation results.

Groups will have about 30 minutes to complete the task. Resulting flipcharts shall be posted on the walls.

2. Give people 30-40 minutes to complete the task.3. Ask groups to post their maps on the wall.4. Give participants 10 minutes to go around and review each other’s work.5. Ask people to return to tables and take flipcharts along.6. Ask groups to name factors that facilitated use of evaluation results in the reviewed case – one

factor from a table a time. Ask each group to explain the factor. Divide a flipchart into two vertical columns. Records factors identified by participants in the left column. After taking all ideas from participants, you may suggest some factors yourself if necessary.

7. Once the list is complete, ask participants what factors in the list are within control of the commissioners. Mark them.

8. Ask how commissioners can use these factors to get useful evaluation results.9. Complete with the discussion about what participants learned as a result of this exercise. Ask:

How participants feel about the exercise? What insights did they get in the process? Are there any pending questions?

Session 4:Diversity of evaluation approaches and related terminology. Evaluation standards and ethics.Learning objectives:As a result of this session participants shall become aware of the diversity of evaluation approaches they may encounter when they start commissioning evaluations.

Activity 4.1: Interactive Lecture “Diversity of approaches and terms”Duration: 45 minutes

Objectives: Introduce participants to diversity of terminology used in the evaluation field.

Materials: Presentation Flipchart Markers

Process:1. Explain that evaluation field was evolving in response to emerging needs of decision-makers.

And terminology in the field was evolving along the way.2. Ask participants to name of types of project evaluation that they have heard of. Record them on

the flipchart grouping as follows.

Key points for presentation (see also Annex 7):

Terminology evolved to differentiate between:

Evaluations done at different points during project lifetime:o Ex ante and ex post evaluationo Midterm and final evaluationo Needs assessment

Evaluations done for different purposes:o Summativeo Formativeo Developmental

Theoretical approaches to evaluation:o Utilization-focused evaluationo Theory-based evaluation

Evaluations undergird by explicit value sets:o Equity-focusedo Gender-responsiveo Human rights based

Who commissions evaluation:o Joint evaluationo Country-led evaluationo Donor-led evaluation

Role of evaluator in the evaluation process:o Independento Collaborativeo Empowerment

3. Add types of evaluation that participants have not identified.4. Overview the resulting scheme providing short explanation about every type of evaluation.5. Conclude by saying that now participants should have understood that evaluation is a broad

field, and that evaluators have to have a lot of knowledge and skills. 6. Explain that evaluation profession does self-regulation by developing evaluation standards,

ethical codes and competencies list. Tell people that they have an examples of evaluation standards developed by the American Evaluation Association (Annex 8) in their workshop kits.

7. Open the floor for questions and comments.

ClosingActivity 4.2: Final assessment Duration: 15 minutes

Objectives: Get data for evaluation of the training. Make participants reflect on what they have learned.

Materials: Final survey (Annex 9) Flipchart Markers

Process:1. Ask participants to fill the survey (printed copies of blank survey should be on each table).2. Ask each table to calculate cumulative baselines and final scores on participants’ knowledge and

skills in areas of project/program management, monitoring and evaluation.3. Ask each table to present the cumulative scores. As people present, take a note of scores on a

flipchart.4. Review the resulting table of scores.5. Collect the surveys before moving to concluding discussion.

Activity 4.3: Concluding discussion Duration: 30 minutes

Objectives: Close the workshop.

Materials: Flipchart Markers

Process:1. Ask people if they have other pending questions. Provide answers.2. Invite participants to share their key insights and take-aways from the workshop. Record them

on a flipchart.3. Conclude with thanks to participants.

AnnexesAnnex 1: Household rules

Every voice and opinion matter Pace the process so that no one is left behind Comfortable learning space:

o No mobile phones inside the roomo If you need to step out, no need to ask for permission

Annex 2: Baseline surveyPlease, using a scale of 0 (minimum) to 10 (maximum) assess the level of your competencies in the following areas:

Area Score

Project/program management

Monitoring

Evaluation

TOTAL

What are the most important questions about project/program evaluation that you would like to have answered during this workshop?

Annex 3: Case: Rehabilitation of St. Sebastian South CanalThe Rehabilitation of St. Sebastian South Canal (RSSC) is a sub-project of the Metro Colombo Urban Development Project (MCUDP) which is one of the projects initiated with the vision of making Colombo the ‘Miracle of Asia’. RSSC was completed in 2016.

Metro Colombo Urban Development Project (MCUDP)

Flooding in and around the Colombo City has been occurring for many years and causes considerable economic disruption and social hardship to a large segment of the population. The urban poor, who mostly live in low-lying areas prone to flooding, are particularly hard hit by the problems caused by flooding. The Metro Colombo Urban Development Project (MCUDP) aims to improve the flood control and drainage infrastructure and management system of the Colombo water basin and enhance the competence of central and local governmental authorities to deliver and manage infrastructure and services in the Colombo Metropolitan area.

The MCUDP is jointly financed by the International Bank for Reconstruction and Development (IBRD) of the World Bank group and the Government of Sri Lanka. MCUDP has three main components. The first component focuses on flood and drainage management and addresses the urgent issue of urban flooding, which regularly paralyzes the economy of the Metro Colombo Region with high socioeconomic

costs. The second component focuses on urban development and infrastructure rehabilitation for Project Local Authorities (PLAs). The third component consists of implementation support.

Rehabilitation of St. Sebastian South Canal (RSSC) sub-project description1

Flooding problem in the area

Floods in Colombo are caused by heavy rainfall and a macro-drainage system with insufficient storage, conveyance and outflow capacity. In the last decade the storage capacity in the basin has reduced by about 30 percent, due to uncontrolled landfill and flood plain encroachments by illegal settlements. The conveyance capacities are limited by solid waste, floating debris, and bottlenecks in the canals. The outflow capacity of the system is too small, particularly in the Colombo North area via Mutwal Tunnel and the North Lock and South Lock in the St. Sebastian canal.

St. Sebastian South Canal in the Colombo North area drains out the storm water from a highly urbanized area of 2.4 km2. It starts from South Lock Gate at Maradana Technical College and runs in the North Easterly direction crossing Sri Sangaraja Mawatha at Maligawatta and Dr. Britto Babapulle Place at Grand Pass. The Canal terminates at the confluence of Dematagoda Canal where St. Sebastian North Canal takes off. The length of St. Sebastian South Canal is about 2.0 km and average width is 15 m.

The low elevation areas around St. Sebastian South Canal, particularly in Maligawatta and Grand Pass, get flooded during rainy seasons. Additionally, the water in this Canal is highly polluted due to raw sewer discharged into the canal from low income communities living along both banks. Rehabilitation of the St. Sebastian South Canal is vital to mitigate the floods in low elevation areas of its catchment as well as in the Colombo North area.

Sub-project objectives

The Rehabilitation of St. Sebastian South Canal sub-project aims to maximize canal conveyance capacity by strengthening and widening canal banks.

The sub-project will be implemented under Sub-Component 1.1 of the MCUDP, namely the Enhancement of Drainage Capacity in the Colombo Water Basin. The total cost of the sub-project is approximately LKR 315 million and the implementation period 18 months.

The Rehabilitation of St. Sebastian South Canal will be implemented by the Sri Lanka Land Reclamation and Development Corporation (SLLRDC), one of the key Project Implementing Agencies (PIAs) of the MCUDP. Overall monitoring and management will be carried out by the Project Management Unit (PMU) established in the Ministry of Defence and Urban Development (MoDUD).

The sub-project will benefit the community in terms of reduced flooding, improved water quality, healthy environment, and better housing for squatters living in hazardous, flood-prone canal banks.

Expected negative social impacts and their mitigation

A total of 107 squatter households will experience negative impacts. 98 households will lose their residential structure out of which six (6) households will additionally lose their commercial structure. Nine (9) households will lose their rental accommodation.

1 Based on Abbreviated Resettlement Action Plan, 2013.

Specifically, the sub-project will involve resettlement of 98 squatter households. The Government of Sri Lanka (GoSL) is implementing a pro-poor housing program under which it has commissioned construction of new multistoried buildings in several locations in the Colombo area to re-house people living in underserved settlements. The resettlement requirement of the MCUDP will be met by this housing program. These 98 households will each be offered a flat along with rehabilitation assistance.

This sub-project will result in loss of rental accommodation for 9 households. They will be provided with rental and transport allowance.

A total of 6 commercial non-titled squatters are expected to experience loss of structure due to the implementation of this sub-project. They have been included in the 98 households that are to be resettled. Apart from the new house and related rehabilitation assistance, these 6 households will be offered the opportunity to obtain an alternative shop available at the resettlement site or any other shopping scheme on a long-term lease basis, subject to their willingness to bear the cost of lease premium and maintenance / management charges. They will also be compensated for loss of livelihood and commercial structure.

A temple and mosque located along the canal bank may sustain minor damages during civil works. These structures will be reconstructed to original/better condition by the Contractor, under the supervision of the SLLRDC, and in consultation with the temple head priest and mosque management committee. Temporary impacts during construction such as hindrances noise and dust issues are expected and will be mitigated.

Expected positive social impacts

Improved Water Quality: Undertaking this sub-project will help prevent unauthorized waste or sewerage outlets to the St. Sebastian South Canal and thereby improve its water quality.

Reduced Flooding: The St. Sebastian South Canal is a very important canal for draining out the storm water of its own catchment under gravity flow, when the water level of the Kelani River is lower than that of the canal system. The rehabilitation of the Canal will help reduce flooding in the CMR and minimize disruption to daily livelihoods of the surrounding community.

Healthy and Safe Environment: The sub-project will have a long term positive impact on the community by way of a clean and healthy environment and by reducing local flood risk during rainy periods.

Better Housing for Relocated Community: Households located inside the 6.5 m reservation area of St. Sebastian South Canal reside in low-lying, flood-prone areas where regular flooding causes financial burden, disease, and disruption to children‘s schooling. They will be moved to permanent flats of 4002 ft and worth 2.5 million each, in newly built condominiums. Thus they will ultimately benefit in terms of improved housing and living conditions with minimal disruption to livelihoods and education as relocation will be within a 3-kilometre radius of their previous location.

Monitoring and Evaluation

Monitoring and Evaluation (M & E) and reporting will be carried out at two levels, at the PMU level and the SLLRDC level. The PMU has recruited a Social Specialist to undertake the M & E and reporting of social safeguards management to ensure that safeguards issues are sufficiently mitigated. The Social Officer at the SLLRDC and 02 Community Development Officers will monitor and report on safeguard

implementation at the field level. This includes paying close attention to the delivery of entitlements to people who will be resettled and the Contractor‘s social safeguard obligations. The PMU will submit quarterly social safeguard progress and quality monitoring reports with yearly safeguard reviews, which will be carried out by independent consultants. These will form the basis for any improvements to be brought about in the safeguard policy framework and implementation arrangements.

Annex 4: Criteria matrixNone

PeoplePlanet Peace Prosperity Partnership

Gender Equity

Relevance

Effectiveness

Efficiency

Sustainability

Impact

Annex 5: Evaluation MatrixCriteria Key

Evaluation Questions

Sub-questions

Necessary evidence

Sources of Data

Methods Assumptions

Annex 6: Case: Salvaging Sri Lanka’s small and medium businesses: how an evaluation led to rapid change Since the end of Sri Lanka’s long civil war in 2009, the nation’s economy has recovered at a startling rate. GDP growth now hovers between 6% and 8%. Peace has certainly brought dividends – tourism has rebounded, China has made large investments in infrastructure, and trains once again run the length of the pearl-shaped island in the Indian Ocean, just off the southern coast of India.

But it’s not all good news. As peace returned, small and medium enterprises (SMEs) found to their chagrin that China, India, and Bangladesh had all but cornered the world’s export markets in mass manufacturing and textiles – at their expense. To add to their woes, the national government had paid little attention to the sector over the years. For the hundreds of business owners renting land in the 18

government-owned industrial estates managed by the Industrial Development Board (IDB), the economic boom seemed to have all but passed them by.

The list of complaints was long and detailed. Rents were out of control, with some more than doubling year after year. And even when rents were paid, there seemed to be no correlation to the quality of infrastructure. Roads were more potholes than asphalt, electricity was unreliable, and coordinated marketing simply lacked focus. Although the government was investing heavily in national infrastructure, smaller businesses appeared to have been overlooked.

R.S. Balanathan, the managing director of handloom silk exporter Ko Lanka, says all the industries in the estates were undeveloped. ‘The roadways were bad, and the rents were very high,’ he says.

In 2013, the situation began to change. The Ministry of Traditional Industries and Small Enterprise Development commissioned a comprehensive evaluation of the sector in 2013. A team of independent evaluators led by Ajith Ekanayaka began travelling to each of the 18 estates across the country to examine the issues plaguing the sector and to suggest direct solutions. Ekanayaka made it clear that this evaluation would not be relegated to a filing cabinet but would be put to immediate use.

For five months, the evaluators gathered their material, forming focus groups for industrial associations and interviewing managers and district officers. The three broad problems that came up again and again were decaying infrastructure, exorbitant rents, and expensive loans. Some had special concerns. Metal industrialists, for instance, could not lay their hands on raw materials because almost all scrap metal was being exported.

The evaluation also found that some estates were being used to build housing by the owners of failed SMEs. The longterm weakness of the sector meant that employment for ethnic minorities and the rural poor – who provided much of the workforce in the estates – also suffered.

The evaluation report gave voice to longstanding grievances and recommended sweeping changes. What was needed, argued Ekanayaka, was direct intervention by the national government. The secretary of the small enterprise ministry, Velayuthan Sivagnanasothy, seized on the findings. He asserts:

“The evaluation showed that the IDB was a mere rent collector and not improving the estates. With an evaluation like this, change would be far easier to generate. The evaluation process was a platform serving to carry the voices of the voiceless to the highest level of policy makers”.

He took the evaluation to all relevant cabinet ministers to secure their support. With the cabinet on side, there was no way the issues raised by the evaluation could be ignored.

And so it came to pass. The ministry drafted a national policy on SME development, consulting widely and drawing on the expertise and advice of other ministries and agencies. But even before the policy could be adopted, change began to ripple outwards. The evaluation’s actionable recommendations had buy-in. Why?

Because policymakers had given frequent input through scoping sessions, and the evaluation was built on a broad, credible base of evidence and data as well as strong connections to stakeholders. According to Sivagnanasothy, a great deal has come from this single evaluation. He recalls: “When the rents went up, industrialists used to shout and cry How can we afford it? We were able to convince the central

government to cap rent increases at 7% annually. That’s a predictable environment, which is an enabling environment. Not only that, but the introduction of targeted loans with lower interest and longer grace periods meant that SMEs could now expand their operations more easily. In each of the estates, factory production is up, and with it, SME profits, leading to more jobs. It’s trickled down to the rural villages”.

A tangible shift in attitude amongst IDB management towards the SMEs was confirmed by Ratnamalala, who praised the evaluation for what it revealed. ‘We expanded our service delivery and created a shift in mindsets of our officers to be more development oriented.’ The shift for the IDB was significant, with the agency undergoing a restructure in order to shift from a reactive to a proactive stance. The IDB now focuses on training, access to raw materials (such as previously scarce scrap metal), and access to credit. Not only that, but the rental income earmarked for infrastructure improvement around the estates was increased tenfold, from as little as 2–3% percent to 20–30% of the rent paid.

To address the problem of access to credit, the government rolled out a special scheme with concessionary rates to boost access. Joint marketing schemes were introduced to give potential buyers a streamlined way of examining the range and quality of products. Ailing SMEs can now gain access to a ‘nursing programme’ to bring them back to health, while the rules for gaining entry have been made more stringent to cut down on residential use of industrial estates. And the metal industries were given a lifeline at the expense of the scrap merchants, with a ban on the export of scrap. The amount injected into infrastructure upgrades immediately following the evaluation was 305 million Sri Lankan rupees (US$2.3 million), with more on the way. For IDB director general Ratnamalala, another benefit of the evaluation is simply being heard. ‘We have direct access to central government,’ he says. ‘Now, with these improvements, we will see growth in our industrial estates.’

On the ground, results of these rapid evaluation-led changes have been impressive. The voices of small and medium industrialists have been heard, and the lines of communication with government are open.

For silk exporter Balanathan, the result has been more cash in his pocket – and that means expansion into his key markets in Europe and the Middle East. From 100 employees in 2013, he now employs 130. ‘We have strong competition with China and Bangladesh,’ he says, ‘but these changes make us more competitive. Now, we have higher silk production than they do. We can beat them!’

From Evaluations that make a difference

Annex 7: Different types of evaluationEvaluations done at different points during project lifetime:

Ex ante evaluation is a process that supports the preparation of proposals for new or renewed project.

Ex post evaluation is conducted after a certain period has passed since the completion of a project with emphasis on the effectiveness and sustainability of the project. This evaluation aims at deriving lessons and recommendations for the improvement of future projects.

Midterm evaluation is conducted in the middle of the project implementation, usually to see if any corrections are necessary.

Final evaluation in conducted right before of after the project completion.

Needs assessment is conducted before project development of assess to needs of target beneficiaries.

Evaluations done for different purposes:

Formative evaluation is used when programs or projects are still in their development or early implementation to provide information about how best to revise and modify for improvement. This type of evaluation often is helpful for pilot projects and new programs, but can be used for progress monitoring of ongoing programs.

Summative evaluation is done at the end of an operating cycle, and findings typically are used to help decide whether a program should be adopted, continued, or modified for improvement.

Developmental Evaluation is an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments.

Theoretical approaches to evaluation:

Utilization-focused evaluation approach, developed by Michael Quinn Patton, is based on the principle that an evaluation should be judged on its usefulness to its intended users. Utilization-focused evaluations should be planned and conducted in ways that enhance the likely utilization of both the findings and of the process itself to inform decisions and improve performance.

Theory-based approaches to evaluation use an explicit theory of change to draw conclusions about whether and how an intervention contributed to observed results.

Evaluations undergird by explicit value sets:

Equity-focused evaluation looks at a project though an equity lens. Gender-responsive evaluation has two essential elements: what the evaluation examines and

how it is undertaken. It assesses the degree to which gender and power relationships—including structural and other causes that give rise to inequities, discrimination and unfair power relations, change as a result of an intervention using a process that is inclusive, participatory and respectful of all stakeholders (rights holders and duty bearers).

Human rights based evaluation looks at the project thorough a human rights lens, using right holders and duty bearers’ perspective.

Who commissions evaluation:

Joint evaluation – jointly commissioned by donors and national government. Country-led evaluation – commissioned by national government. Donor-led evaluation – commissioned by donor organizations.

Role of evaluator in the evaluation process:

Independent – conducted by external evaluator. Collaborative – both an evaluator and project stakeholders are actively involved in the process

of evaluation design, data collection, analysis and interpretation. Empowerment – evaluator support project stakeholders who implement evaluation.

Annex 8: AEA Evaluator Competencies1.0 PROFESSIONAL PRACTICE DOMAIN—focuses on what makes evaluators distinct as practicing professionals.

Professional practice is grounded in the foundations of evaluation, including the Program Evaluation Standards, the AEA Guiding Principles, the AEA Statement on Cultural Competence, and the AEA Program Evaluator Competencies.

The competent evaluator . . .

1.1 Acts ethically as an evaluator through evaluation practice that demonstrates integrity and respects all people, cultural groups, and sovereign nations.

1.2 Knows and applies evaluation foundations that ground and guide evaluative thinking and professional practice.

1.3 Articulates the distinct attributes of program evaluation.

1.4 Knows and applies multiple evaluation approaches and theories.

1.5 Uses evidence and logic in making evaluative judgments.

1.6 Articulates personal evaluator competence and perspectives, areas for growth, and implications for professional practice.

1.7 Engages in ongoing professional development to extend personal learning, growth, and connections.

1.8 Considers social justice implications of evaluation practice.

1.9 Contributes to the public good through evaluation practice.

1.10 Articulates current trends and developments in the field.

1.11 Advocates for the field of evaluation and its value.

1.12 Fosters the use and influence of evaluation and its results.

2.0 METHODOLOGY DOMAIN—focuses on technical aspects of data-based, systematic inquiry for valued purposes.

Methodology includes quantitative, qualitative, and mixed designs for understanding, decision making, and judging.


2.1 Acts ethically in the conduct of evaluation inquiry.

2.2 Understands the range of methodologies that ground inquiry in evaluation practice.

2.3 Articulates the diverse values that underpin methodological choices.

2.4 Conducts reviews of the literature as appropriate.

2.5 Identifies evaluation purposes and needs.

2.6 Frames evaluation questions.

2.7 Designs sound, credible, and feasible studies that address evaluation purposes and questions.

2.8 Determines appropriate methods, including quantitative, qualitative, and mixed methods.

2.9 Identifies data sources and samples.

2.10 Collects data using sound, credible, and feasible procedures.

2.11 Analyzes data using sound, credible, and feasible procedures.

2.12 Interprets findings/results and draws conclusions by identifying possible meanings in context.

2.13 Justifies evaluation findings/results and conclusions, judging merit, worth, and value as appropriate.

2.14 Communicates evaluation findings/results, including strengths and limitations.

2.15 Articulates evidence-based judgments and recommendations for use.

2.16 Conducts informal and formal meta-evaluations of studies, identifying their strengths and limitations.

3.0 CONTEXT DOMAIN—focuses on understanding the unique circumstances, multiple perspectives, and changing settings of evaluations and their users/stakeholders.

Context involves site/location/environment, participants/stakeholders, organization/structure, culture/diversity, history/traditions, values/beliefs, politics/economics, power/privilege, and other characteristics.


3.1 Acts and interacts ethically within the evaluation context.

3.2 Respects and responds to the uniqueness of the context.

3.3 Addresses systems and complexity within the context.

3.4 Identifies and engages diverse users/stakeholders throughout the evaluation process.

3.5 Describes the program, including its basic purpose, components, and its functioning in its broader context.

3.6 Clarifies cultural assumptions, diverse perspectives, and political forces within the context.

3.7 Elicits program logic and program theory as appropriate.

3.8 Develops and facilitates shared understanding of the program and its evaluation in context.

3.9 Considers both specific and broader contexts of the program.

3.10 Attends to evaluation use and influence in context.

4.0 PLANNING AND MANAGEMENT DOMAIN—Focuses on determining and monitoring work plans, timelines, resources, and other components needed to complete and deliver an evaluation study.

Planning and management include networking, developing proposals, contracting, determining work assignments, monitoring progress, fostering use, etc.


4.1 Acts ethically in planning and managing evaluation.

4.2 Demonstrates cultural competence in planning and managing evaluation.

4.3 Obtains internal or external evaluation work.

4.4 Negotiates and manages a feasible evaluation plan, budget, resources, and timeline.

4.5 Plans for evaluation use and influence.

4.6 Documents evaluation processes and products.

4.7 Coordinates and supervises evaluation processes and products.

4.8 Monitors evaluation progress and quality and makes adjustments as necessary.

4.9 Uses appropriate technology and other tools to support and manage the evaluation.

4.10 Communicates evaluation processes and results in appropriate, timely, and effective ways.

5.0 INTERPERSONAL DOMAIN—focuses on human relations and social interactions that ground evaluator effectiveness for professional practice.

Interpersonal skills include cultural competence, communication, facilitation, negotiation, conflict resolution, etc.


5.1 Interacts ethically in interpersonal relations at all times.

5.2 Values and fosters constructive interpersonal relations foundational for professional practice and evaluation use.

5.3 Uses appropriate social skills to build trust and enhance interaction for evaluation practice.

5.4 Listens to understand, engage, and honor diverse perspectives.

5.5 Addresses issues of privilege and power dynamics in interpersonal relations.

5.6 Communicates in meaningful ways throughout the evaluation (written, verbal, visual, etc.).

5.7 Facilitates constructive and culturally responsive interaction throughout the evaluation.

5.8 Collaborates and engages in teamwork.

5.9 Negotiates decisions for evaluation practice.

5.10 Addresses conflicts constructively.

Annex 9: Final surveyPlease, using a scale of 0 (minimum) to 10 (maximum) assess the level of your competencies before and after this workshop in the following areas:

AreaScore

Before this workshop After this workshop


Monitoring

Evaluation

TOTAL

Annex 10: List of the Sustainable Development GoalsGoal 1. End poverty in all its forms everywhere

1.1 By 2030, eradicate extreme poverty for all people everywhere, currently measured as people living on less than $1.25 a day

1.2 By 2030, reduce at least by half the proportion of men, women and children of all ages living in poverty in all its dimensions according to national definitions

1.3 Implement nationally appropriate social protection systems and measures for all, including floors, and by 2030 achieve substantial coverage of the poor and the vulnerable

1.4 By 2030, ensure that all men and women, in particular the poor and the vulnerable, have equal rights to economic resources, as well as access to basic services, ownership and control over land and other forms of property, inheritance, natural resources, appropriate new technology and financial services, including microfinance

1.5 By 2030, build the resilience of the poor and those in vulnerable situations and reduce their exposure and vulnerability to climate-related extreme events and other economic, social and environmental shocks and disasters

1.a Ensure significant mobilization of resources from a variety of sources, including through enhanced development cooperation, in order to provide adequate and predictable means for developing countries, in particular least developed countries, to implement programmes and policies to end poverty in all its dimensions

1.b Create sound policy frameworks at the national, regional and international levels, based on pro-poor and gendersensitive development strategies, to support accelerated investment in poverty eradication actions

Goal 2. End hunger, achieve food security and improved nutrition and promote sustainable agriculture

2.1 By 2030, end hunger and ensure access by all people, in particular the poor and people in vulnerable situations, including infants, to safe, nutritious and sufficient food all year round

2.2 By 2030, end all forms of malnutrition, including achieving, by 2025, the internationally agreed targets on stunting and wasting in children under 5 years of age, and address the nutritional needs of adolescent girls, pregnant and lactating women and older persons

2.3 By 2030, double the agricultural productivity and incomes of small-scale food producers, in particular women, indigenous peoples, family farmers, pastoralists and fishers, including through secure and equal access to land, other productive resources and inputs, knowledge, financial services, markets and opportunities for value addition and non-farm employment

2.4 By 2030, ensure sustainable food production systems and implement resilient agricultural practices that increase productivity and production, that help maintain ecosystems, that strengthen capacity for adaptation to climate change, extreme weather, drought, flooding and other disasters and that progressively improve land and soil quality

2.5 By 2020, maintain the genetic diversity of seeds, cultivated plants and farmed and domesticated animals and their related wild species, including through soundly managed and diversified seed and plant banks at the national, regional and international levels, and ensure access to and fair and equitable sharing of benefits arising from the utilization of genetic resources and associated traditional knowledge, as internationally agreed

2.a Increase investment, including through enhanced international cooperation, in rural infrastructure, agricultural research and extension services, technology development and plant and livestock gene banks in order to enhance agricultural productive capacity in developing countries, in particular least developed countries

2.b Correct and prevent trade restrictions and distortions in world agricultural markets, including through the parallel elimination of all forms of agricultural export subsidies and all export measures with equivalent effect, in accordance with the mandate of the Doha Development Round

2.c Adopt measures to ensure the proper functioning of food commodity markets and their derivatives and facilitate timely access to market information, including on food reserves, in order to help limit extreme food price volatility

Goal 3. Ensure healthy lives and promote well-being for all at all ages

3.1 By 2030, reduce the global maternal mortality ratio to less than 70 per 100,000 live births

3.2 By 2030, end preventable deaths of newborns and children under 5 years of age, with all countries aiming to reduce neonatal mortality to at least as low as 12 per 1,000 live births and under-5 mortality to at least as low as 25 per 1,000 live births

3.3 By 2030, end the epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases and combat hepatitis, water-borne diseases and other communicable diseases

3.4 By 2030, reduce by one third premature mortality from non-communicable diseases through prevention and treatment and promote mental health and well-being

3.5 Strengthen the prevention and treatment of substance abuse, including narcotic drug abuse and harmful use of alcohol

3.6 By 2020, halve the number of global deaths and injuries from road traffic accidents

3.7 By 2030, ensure universal access to sexual and reproductive health-care services, including for family planning, information and education, and the integration of reproductive health into national strategies and programmes

3.8 Achieve universal health coverage, including financial risk protection, access to quality essential health-care services and access to safe, effective, quality and affordable essential medicines and vaccines for all

3.9 By 2030, substantially reduce the number of deaths and illnesses from hazardous chemicals and air, water and soil pollution and contamination

3.a Strengthen the implementation of the World Health Organization Framework Convention on Tobacco Control in all countries, as appropriate

3.b Support the research and development of vaccines and medicines for the communicable and non-communicable diseases that primarily affect developing countries, provide access to affordable essential medicines and vaccines, in accordance with the Doha Declaration on the TRIPS Agreement and Public Health, which affirms the right of developing countries to use to the full the provisions in the Agreement on Trade-Related Aspects of Intellectual Property Rights regarding flexibilities to protect public health, and, in particular, provide access to medicines for all

3.c Substantially increase health financing and the recruitment, development, training and retention of the health workforce in developing countries, especially in least developed countries and small island developing States 3.d Strengthen the capacity of all countries, in particular developing countries, for early warning, risk reduction and management of national and global health risks

Goal 4. Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all

4.1 By 2030, ensure that all girls and boys complete free, equitable and quality primary and secondary education leading to relevant and effective learning outcomes

4.2 By 2030, ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education

4.3 By 2030, ensure equal access for all women and men to affordable and quality technical, vocational and tertiary education, including university

4.4 By 2030, substantially increase the number of youth and adults who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship

4.5 By 2030, eliminate gender disparities in education and ensure equal access to all levels of education and vocational training for the vulnerable, including persons with disabilities, indigenous peoples and children in vulnerable situations

4.6 By 2030, ensure that all youth and a substantial proportion of adults, both men and women, achieve literacy and numeracy

4.7 By 2030, ensure that all learners acquire the knowledge and skills needed to promote sustainable development, including, among others, through education for sustainable development and sustainable lifestyles, human rights, gender equality, promotion of a culture of peace and non-violence, global citizenship and appreciation of cultural diversity and of culture’s contribution to sustainable development

4.a Build and upgrade education facilities that are child, disability and gender sensitive and provide safe, non-violent, inclusive and effective learning environments for all

4.b By 2020, substantially expand globally the number of scholarships available to developing countries, in particular least developed countries, small island developing States and African countries, for enrolment in higher education, including vocational training and information and communications technology, technical, engineering and scientific programmes, in developed countries and other developing countries

4.c By 2030, substantially increase the supply of qualified teachers, including through international cooperation for teacher training in developing countries, especially least developed countries and small island developing States

Goal 5. Achieve gender equality and empower all women and girls

5.1 End all forms of discrimination against all women and girls everywhere

5.2 Eliminate all forms of violence against all women and girls in the public and private spheres, including trafficking and sexual and other types of exploitation

5.3 Eliminate all harmful practices, such as child, early and forced marriage and female genital mutilation

5.4 Recognize and value unpaid care and domestic work through the provision of public services, infrastructure and social protection policies and the promotion of shared responsibility within the household and the family as nationally appropriate

5.5 Ensure women’s full and effective participation and equal opportunities for leadership at all levels of decisionmaking in political, economic and public life

5.6 Ensure universal access to sexual and reproductive health and reproductive rights as agreed in accordance with the Programme of Action of the International Conference on Population and Development and the Beijing Platform for Action and the outcome documents of their review conferences

5.a Undertake reforms to give women equal rights to economic resources, as well as access to ownership and control over land and other forms of property, financial services, inheritance and natural resources, in accordance with national laws

5.b Enhance the use of enabling technology, in particular information and communications technology, to promote the empowerment of women

5.c Adopt and strengthen sound policies and enforceable legislation for the promotion of gender equality and the empowerment of all women and girls at all levels

Goal 6. Ensure availability and sustainable management of water and sanitation for all

6.1 By 2030, achieve universal and equitable access to safe and affordable drinking water for all

6.2 By 2030, achieve access to adequate and equitable sanitation and hygiene for all and end open defecation, paying special attention to the needs of women and girls and those in vulnerable situations

6.3 By 2030, improve water quality by reducing pollution, eliminating dumping and minimizing release of hazardous chemicals and materials, halving the proportion of untreated wastewater and substantially increasing recycling and safe reuse globally

6.4 By 2030, substantially increase water-use efficiency across all sectors and ensure sustainable withdrawals and supply of freshwater to address water scarcity and substantially reduce the number of people suffering from water scarcity

6.5 By 2030, implement integrated water resources management at all levels, including through transboundary cooperation as appropriate

6.6 By 2020, protect and restore water-related ecosystems, including mountains, forests, wetlands, rivers, aquifers and lakes

6.a By 2030, expand international cooperation and capacity-building support to developing countries in water- and sanitation-related activities and programmes, including water harvesting, desalination, water efficiency, wastewater treatment, recycling and reuse technologies

6.b Support and strengthen the participation of local communities in improving water and sanitation management

Goal 7. Ensure access to affordable, reliable, sustainable and modern energy for all

7.1 By 2030, ensure universal access to affordable, reliable and modern energy services

7.2 By 2030, increase substantially the share of renewable energy in the global energy mix

7.3 By 2030, double the global rate of improvement in energy efficiency

7.a By 2030, enhance international cooperation to facilitate access to clean energy research and technology, including renewable energy, energy efficiency and advanced and cleaner fossil-fuel technology, and promote investment in energy infrastructure and clean energy technology

7.b By 2030, expand infrastructure and upgrade technology for supplying modern and sustainable energy services for all in developing countries, in particular least developed countries, and small island developing States

Goal 8. Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all

8.1 Sustain per capita economic growth in accordance with national circumstances and, in particular, at least 7 per cent gross domestic product growth per annum in the least developed countries

8.2 Achieve higher levels of economic productivity through diversification, technological upgrading and innovation, including through a focus on high-value added and labour-intensive sectors

8.3 Promote development-oriented policies that support productive activities, decent job creation, entrepreneurship, creativity and innovation, and encourage the formalization and growth of micro-, small- and medium-sized enterprises, including through access to financial services

8.4 Improve progressively, through 2030, global resource efficiency in consumption and production and endeavour to decouple economic growth from environmental degradation, in accordance with the 10-year framework of programmes on sustainable consumption and production, with developed countries taking the lead

8.5 By 2030, achieve full and productive employment and decent work for all women and men, including for young people and persons with disabilities, and equal pay for work of equal value

8.6 By 2020, substantially reduce the proportion of youth not in employment, education or training

8.7 Take immediate and effective measures to eradicate forced labour, end modern slavery and human trafficking and secure the prohibition and elimination of the worst forms of child labour, including recruitment and use of child soldiers, and by 2025 end child labour in all its forms

8.8 Protect labour rights and promote safe and secure working environments for all workers, including migrant workers, in particular women migrants, and those in precarious employment

8.9 By 2030, devise and implement policies to promote sustainable tourism that creates jobs and promotes local culture and products 8.10 Strengthen the capacity of domestic financial institutions to encourage and expand access to banking, insurance and financial services for all

8.a Increase Aid for Trade support for developing countries, in particular least developed countries, including through the Enhanced Integrated Framework for Trade-Related Technical Assistance to Least Developed Countries

8.b By 2020, develop and operationalize a global strategy for youth employment and implement the Global Jobs Pact of the International Labour Organization

Goal 9. Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation

9.1 Develop quality, reliable, sustainable and resilient infrastructure, including regional and transborder infrastructure, to support economic development and human well-being, with a focus on affordable and equitable access for all

9.2 Promote inclusive and sustainable industrialization and, by 2030, significantly raise industry’s share of employment and gross domestic product, in line with national circumstances, and double its share in least developed countries

9.3 Increase the access of small-scale industrial and other enterprises, in particular in developing countries, to financial services, including affordable credit, and their integration into value chains and markets

9.4 By 2030, upgrade infrastructure and retrofit industries to make them sustainable, with increased resource-use efficiency and greater adoption of clean and environmentally sound technologies and industrial processes, with all countries taking action in accordance with their respective capabilities

9.5 Enhance scientific research, upgrade the technological capabilities of industrial sectors in all countries, in particular developing countries, including, by 2030, encouraging innovation and substantially increasing the number of research and development workers per 1 million people and public and private research and development spending

9.a Facilitate sustainable and resilient infrastructure development in developing countries through enhanced financial, technological and technical support to African countries, least developed countries, landlocked developing countries and small island developing States

9.b Support domestic technology development, research and innovation in developing countries, including by ensuring a conducive policy environment for, inter alia, industrial diversification and value addition to commodities

9.c Significantly increase access to information and communications technology and strive to provide universal and affordable access to the Internet in least developed countries by 2020

Goal 10. Reduce inequality within and among countries

10.1 By 2030, progressively achieve and sustain income growth of the bottom 40 per cent of the population at a rate higher than the national average

10.2 By 2030, empower and promote the social, economic and political inclusion of all, irrespective of age, sex, disability, race, ethnicity, origin, religion or economic or other status

10.3 Ensure equal opportunity and reduce inequalities of outcome, including by eliminating discriminatory laws, policies and practices and promoting appropriate legislation, policies and action in this regard

10.4 Adopt policies, especially fiscal, wage and social protection policies, and progressively achieve greater equality

10.5 Improve the regulation and monitoring of global financial markets and institutions and strengthen the implementation of such regulations

10.6 Ensure enhanced representation and voice for developing countries in decision-making in global international economic and financial institutions in order to deliver more effective, credible, accountable and legitimate institutions

10.7 Facilitate orderly, safe, regular and responsible migration and mobility of people, including through the implementation of planned and well-managed migration policies

10.a Implement the principle of special and differential treatment for developing countries, in particular least developed countries, in accordance with World Trade Organization agreements

10.b Encourage official development assistance and financial flows, including foreign direct investment, to States where the need is greatest, in particular least developed countries, African countries, small island developing States and landlocked developing countries, in accordance with their national plans and programmes

10.c By 2030, reduce to less than 3 per cent the transaction costs of migrant remittances and eliminate remittance corridors with costs higher than 5 per cent

Goal 11. Make cities and human settlements inclusive, safe, resilient and sustainable

11.1 By 2030, ensure access for all to adequate, safe and affordable housing and basic services and upgrade slums

11.2 By 2030, provide access to safe, affordable, accessible and sustainable transport systems for all, improving road safety, notably by expanding public transport, with special attention to the needs of those in vulnerable situations, women, children, persons with disabilities and older persons

11.3 By 2030, enhance inclusive and sustainable urbanization and capacity for participatory, integrated and sustainable human settlement planning and management in all countries

11.4 Strengthen efforts to protect and safeguard the world’s cultural and natural heritage

11.5 By 2030, significantly reduce the number of deaths and the number of people affected and substantially decrease the direct economic losses relative to global gross domestic product caused by

disasters, including water-related disasters, with a focus on protecting the poor and people in vulnerable situations

11.6 By 2030, reduce the adverse per capita environmental impact of cities, including by paying special attention to air quality and municipal and other waste management

11.7 By 2030, provide universal access to safe, inclusive and accessible, green and public spaces, in particular for women and children, older persons and persons with disabilities

11.a Support positive economic, social and environmental links between urban, peri-urban and rural areas by strengthening national and regional development planning

11.b By 2020, substantially increase the number of cities and human settlements adopting and implementing integrated policies and plans towards inclusion, resource efficiency, mitigation and adaptation to climate change, resilience to disasters, and develop and implement, in line with the Sendai Framework for Disaster Risk Reduction 2015- 2030, holistic disaster risk management at all levels

11.c Support least developed countries, including through financial and technical assistance, in building sustainable and resilient buildings utilizing local materials

Goal 12. Ensure sustainable consumption and production patterns

12.1 Implement the 10-year framework of programmes on sustainable consumption and production, all countries taking action, with developed countries taking the lead, taking into account the development and capabilities of developing countries

12.2 By 2030, achieve the sustainable management and efficient use of natural resources

12.3 By 2030, halve per capita global food waste at the retail and consumer levels and reduce food losses along production and supply chains, including post-harvest losses

12.4 By 2020, achieve the environmentally sound management of chemicals and all wastes throughout their life cycle, in accordance with agreed international frameworks, and significantly reduce their release to air, water and soil in order to minimize their adverse impacts on human health and the environment

12.5 By 2030, substantially reduce waste generation through prevention, reduction, recycling and reuse

12.6 Encourage companies, especially large and transnational companies, to adopt sustainable practices and to integrate sustainability information into their reporting cycle

12.7 Promote public procurement practices that are sustainable, in accordance with national policies and priorities

12.8 By 2030, ensure that people everywhere have the relevant information and awareness for sustainable development and lifestyles in harmony with nature

12.a Support developing countries to strengthen their scientific and technological capacity to move towards more sustainable patterns of consumption and production

12.b Develop and implement tools to monitor sustainable development impacts for sustainable tourism that creates jobs and promotes local culture and products

12.c Rationalize inefficient fossil-fuel subsidies that encourage wasteful consumption by removing market distortions, in accordance with national circumstances, including by restructuring taxation and phasing out those harmful subsidies, where they exist, to reflect their environmental impacts, taking fully into account the specific needs and conditions of developing countries and minimizing the possible adverse impacts on their development in a manner that protects the poor and the affected communities

Goal 13. Take urgent action to combat climate change and its impacts*

13.1 Strengthen resilience and adaptive capacity to climate-related hazards and natural disasters in all countries

13.2 Integrate climate change measures into national policies, strategies and planning

13.3 Improve education, awareness-raising and human and institutional capacity on climate change mitigation, adaptation, impact reduction and early warning

13.a Implement the commitment undertaken by developed-country parties to the United Nations Framework Convention on Climate Change to a goal of mobilizing jointly $100 billion annually by 2020 from all sources to address the needs of developing countries in the context of meaningful mitigation actions and transparency on implementation and fully operationalize the Green Climate Fund through its capitalization as soon as possible

13.b Promote mechanisms for raising capacity for effective climate change-related planning and management in least developed countries, including focusing on women, youth and local and marginalized communities

* Acknowledging that the United Nations Framework Convention on Climate Change is the primary international, intergovernmental forum for negotiating the global response to climate change.

Goal 14. Conserve and sustainably use the oceans, seas and marine resources for sustainable development

14.1 By 2025, prevent and significantly reduce marine pollution of all kinds, in particular from land-based activities, including marine debris and nutrient pollution

14.2 By 2020, sustainably manage and protect marine and coastal ecosystems to avoid significant adverse impacts, including by strengthening their resilience, and take action for their restoration in order to achieve healthy and productive oceans

14.3 Minimize and address the impacts of ocean acidification, including through enhanced scientific cooperation at all levels

14.4 By 2020, effectively regulate harvesting and end overfishing, illegal, unreported and unregulated fishing and destructive fishing practices and implement science-based management plans, in order to restore fish stocks in the shortest time feasible, at least to levels that can produce maximum sustainable yield as determined by their biological characteristics

14.5 By 2020, conserve at least 10 per cent of coastal and marine areas, consistent with national and international law and based on the best available scientific information

14.6 By 2020, prohibit certain forms of fisheries subsidies which contribute to overcapacity and overfishing, eliminate subsidies that contribute to illegal, unreported and unregulated fishing and refrain from introducing new such subsidies, recognizing that appropriate and effective special and differential treatment for developing and least developed countries should be an integral part of the World Trade Organization fisheries subsidies negotiation2

14.7 By 2030, increase the economic benefits to Small Island developing States and least developed countries from the sustainable use of marine resources, including through sustainable management of fisheries, aquaculture and tourism

14.a Increase scientific knowledge, develop research capacity and transfer marine technology, taking into account the Intergovernmental Oceanographic Commission Criteria and Guidelines on the Transfer of Marine Technology, in order to improve ocean health and to enhance the contribution of marine biodiversity to the development of developing countries, in particular small island developing States and least developed countries

14.b Provide access for small-scale artisanal fishers to marine resources and markets

14.c Enhance the conservation and sustainable use of oceans and their resources by implementing international law as reflected in UNCLOS, which provides the legal framework for the conservation and sustainable use of oceans and their resources, as recalled in paragraph 158 of The Future We Want

Goal 15. Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss

15.1 By 2020, ensure the conservation, restoration and sustainable use of terrestrial and inland freshwater ecosystems and their services, in particular forests, wetlands, mountains and drylands, in line with obligations under international agreements

15.2 By 2020, promote the implementation of sustainable management of all types of forests, halt deforestation, restore degraded forests and substantially increase afforestation and reforestation globally

15.3 By 2030, combat desertification, restore degraded land and soil, including land affected by desertification, drought and floods, and strive to achieve a land degradation-neutral world

15.4 By 2030, ensure the conservation of mountain ecosystems, including their biodiversity, in order to enhance their capacity to provide benefits that are essential for sustainable development

15.5 Take urgent and significant action to reduce the degradation of natural habitats, halt the loss of biodiversity and, by 2020, protect and prevent the extinction of threatened species

15.6 Ensure fair and equitable sharing of the benefits arising from the utilization of genetic resources and promote appropriate access to such resources

15.7 Take urgent action to end poaching and trafficking of protected species of flora and fauna and address both demand and supply of illegal wildlife products

15.8 By 2020, introduce measures to prevent the introduction and significantly reduce the impact of invasive alien species on land and water ecosystems and control or eradicate the priority species

15.9 By 2020, integrate ecosystem and biodiversity values into national and local planning, development processes, poverty reduction strategies and accounts

15.a Mobilize and significantly increase financial resources from all sources to conserve and sustainably use biodiversity and ecosystems

15.b Mobilize significant resources from all sources and at all levels to finance sustainable forest management and provide adequate incentives to developing countries to advance such management, including for conservation and reforestation

15.c Enhance global support for efforts to combat poaching and trafficking of protected species, including by increasing the capacity of local communities to pursue sustainable livelihood opportunities

Goal 16. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels

16.1 Significantly reduce all forms of violence and related death rates everywhere

16.2 End abuse, exploitation, trafficking and all forms of violence against and torture of children

16.3 Promote the rule of law at the national and international levels and ensure equal access to justice for all

16.4 By 2030, significantly reduce illicit financial and arms flows, strengthen the recovery and return of stolen assets and combat all forms of organized crime

16.5 Substantially reduce corruption and bribery in all their forms

16.6 Develop effective, accountable and transparent institutions at all levels

16.7 Ensure responsive, inclusive, participatory and representative decision-making at all levels

16.8 Broaden and strengthen the participation of developing countries in the institutions of global governance

16.9 By 2030, provide legal identity for all, including birth registration

16.10 Ensure public access to information and protect fundamental freedoms, in accordance with national legislation and international agreements

16.a Strengthen relevant national institutions, including through international cooperation, for building capacity at all levels, in particular in developing countries, to prevent violence and combat terrorism and crime

16.b Promote and enforce non-discriminatory laws and policies for sustainable development

Goal 17. Strengthen the means of implementation and revitalize the global partnership for sustainable development

Finance

17.1 Strengthen domestic resource mobilization, including through international support to developing countries, to improve domestic capacity for tax and other revenue collection

17.2 Developed countries to implement fully their official development assistance commitments, including the commitment by many developed countries to achieve the target of 0.7 per cent of ODA/GNI to developing countries and 0.15 to 0.20 per cent of ODA/GNI to least developed countries; ODA providers are encouraged to consider setting a target to provide at least 0.20 per cent of ODA/GNI to least developed countries

17.3 Mobilize additional financial resources for developing countries from multiple sources

17.4 Assist developing countries in attaining long-term debt sustainability through coordinated policies aimed at fostering debt financing, debt relief and debt restructuring, as appropriate, and address the external debt of highly indebted poor countries to reduce debt distress

17.5 Adopt and implement investment promotion regimes for least developed countries

Technology

17.6 Enhance North-South, South-South and triangular regional and international cooperation on and access to science, technology and innovation and enhance knowledge sharing on mutually agreed terms, including through improved coordination among existing mechanisms, in particular at the United Nations level, and through a global technology facilitation mechanism

17.7 Promote the development, transfer, dissemination and diffusion of environmentally sound technologies to developing countries on favourable terms, including on concessional and preferential terms, as mutually agreed

17.8 Fully operationalize the technology bank and science, technology and innovation capacity-building mechanism for least developed countries by 2017 and enhance the use of enabling technology, in particular information and communications technology

Capacity-building

17.9 Enhance international support for implementing effective and targeted capacity-building in developing countries to support national plans to implement all the sustainable development goals, including through North-South, South-South and triangular cooperation

Trade

17.10 Promote a universal, rules-based, open, non-discriminatory and equitable multilateral trading system under the World Trade Organization, including through the conclusion of negotiations under its Doha Development Agenda

17.11 Significantly increase the exports of developing countries, in particular with a view to doubling the least developed countries’ share of global exports by 2020 23

17.12 Realize timely implementation of duty-free and quota-free market access on a lasting basis for all least developed countries, consistent with World Trade Organization decisions, including by ensuring that preferential rules of origin applicable to imports from least developed countries are transparent and simple, and contribute to facilitating market access

Systemic issues

Policy and institutional coherence

17.13 Enhance global macroeconomic stability, including through policy coordination and policy coherence

17.14 Enhance policy coherence for sustainable development

17.15 Respect each country’s policy space and leadership to establish and implement policies for poverty eradication and sustainable development

Multi-stakeholder partnerships

17.16 Enhance the global partnership for sustainable development, complemented by multi-stakeholder partnerships that mobilize and share knowledge, expertise, technology and financial resources, to support the achievement of the sustainable development goals in all countries, in particular developing countries

17.17 Encourage and promote effective public, public-private and civil society partnerships, building on the experience and resourcing strategies of partnerships

Data, monitoring and accountability

17.18 By 2020, enhance capacity-building support to developing countries, including for least developed countries and small island developing States, to increase significantly the availability of high-quality, timely and reliable data disaggregated by income, gender, age, race, ethnicity, migratory status, disability, geographic location and other characteristics relevant in national contexts

17.19 By 2030, build on existing initiatives to develop measurements of progress on sustainable development that complement gross domestic product, and support statistical capacity-building in developing countries

EVALUATION TRAINING

2-DAY MODULE

ContentModule Description.....................................................................................................................................3Session 1......................................................................................................................................................4Intro.............................................................................................................................................................4Activity 1.1: Welcome and introduction......................................................................................................4Activity 1.2: Baseline assessment and participants’ getting to know each other........................................4The Sustainable Development Goals (SDGs) – new global context for evaluation practice........................5Activity 1.3: Intro to the SDGs.....................................................................................................................5Activity 1.4: Linking project outcomes to the SDGs.....................................................................................6Session 2......................................................................................................................................................7Evaluation basics.........................................................................................................................................7Activity 2.1: Mini-lecture.............................................................................................................................7Activity 2.2: Mapping decisions and relevant information..........................................................................7Differences between evaluation and other ways of getting information for decision-making....................8Activity 2.3: Mini-lecture: project evaluation vs other types of getting information for decision making...8Session 3......................................................................................................................................................9How evaluators think about projects..........................................................................................................9Activity 3.1: Lecture “Logical frameworks, logical models and theories of change”....................................9Activity 3.2: Group exercise “Developing logic model for the Rehabilitation of St. Sebastian South Canal project”.....................................................................................................................................................10Session 4:...................................................................................................................................................10How evaluators think about projects (continued).....................................................................................10Activity 4.1: Lecture “Evaluation criteria and questions”..........................................................................10Activity 4.2: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”.................................................................................................................................11Session 5....................................................................................................................................................12Recap from Day 1......................................................................................................................................12Activity 5.1: Recap from Day 1...................................................................................................................12How evaluators collect and analyze data..................................................................................................13Activity 5.1: Lecture “Marching evaluation questions with data collection methods”..............................13Activity 5.2: Demonstration “Design of evaluation of the Rehabilitation of St. Sebastian South Canal project”.....................................................................................................................................................14Activity 5.3: Interactive Lecture “How evaluators analyze data to arrive at conclusions and recommendations”....................................................................................................................................14Session 6....................................................................................................................................................15Evaluation use...........................................................................................................................................15

Activity 6: Analysis of factors that facilitate use........................................................................................15Session 7....................................................................................................................................................16From ToR to Management Response and Follow-up plan.........................................................................16Activity 7: Biscuits evaluation and Management response.......................................................................16Session 8....................................................................................................................................................17Diversity of evaluation approaches and related terminology. Evaluation standards and ethics...............17Activity 8.1: Lecture “Diversity of approaches and terms”........................................................................17Closing.......................................................................................................................................................18Activity 8.2: Final assessment....................................................................................................................18Activity 8.3: Concluding discussion............................................................................................................18Annexes.....................................................................................................................................................20Annex 1: Household rules..........................................................................................................................20Annex 2: Baseline survey...........................................................................................................................21Annex 3: Case: Rehabilitation of St. Sebastian South Canal......................................................................22Annex 4: Criteria matrix.............................................................................................................................25Annex 5: Evaluation Matrix.......................................................................................................................26Annex 6: Results of an evaluation of a capacity building project..............................................................27Annex 7: Case: Salvaging Sri Lanka’s small and medium businesses: how an evaluation led to rapid change.......................................................................................................................................................29Annex 8: Biscuits survey............................................................................................................................31Annex 9: Biscuits data sheets....................................................................................................................32Annex 10: Different types of evaluation....................................................................................................33Annex 11: AEA Evaluator Competencies...................................................................................................35Annex 12: Final survey...............................................................................................................................38Annex 13: List of the Sustainable Development Goals..............................................................................39

Module DescriptionTarget audience:Public officials who are already using or planning to use project/program evaluation results to inform their decisions.

Training overall objective:This training shall contribute to the implementation of the broad vision that every public official in Sri Lanka should understand importance of evaluation and basics of evaluation practice.

The specific objective of this 2-day training module is to give public officials who are already using or planning to use project/program evaluation results the skills and knowledge necessary to understand and appreciate evaluation as a way to get evidence to inform their decisions.

Number of participants: about 30 people.

Recommended room set up:6 rounds tables to seat 5 participants each. Set a plate with a number (from 1 to 6) to each table.

Agenda overview

Day 1 Topics Activities


The SDGs – new global context for evaluation practice.






Discussion on types on decisions and relevant information.

Activity 2.1: Mini-lecture.

Activity 2.2: Mapping decisions and relevant information.

Activity 2.3: Mini-lecture: project evaluation vs other types of getting information for decision making.

Session 3 How evaluators think about projects. Activity 3.1: Lecture “Logical frameworks, logical models and theories of change”.

Activity 3.2: Group exercise “Developing logic model for the Rehabilitation of St. Sebastian South Canal project”.

Session 4 How evaluators think about projects – cont.



Day 2

Session 5 How evaluators collect and analyze data.

How evaluators arrive at conclusions and recommendations. Reporting and communication of evaluation findings.

Activity 5.1: Recap from Day 1.

Activity 5.1: Lecture “Marching evaluation questions with data collection methods”.

Activity 5.2: Demonstration “Design of evaluation of the Rehabilitation of St. Sebastian South Canal project”.

Activity 5.3: Interactive Lecture “How evaluators analyze data to arrive at conclusions and recommendations”.

Session 6 Use of evaluation results. Activity 6: Analysis of factors that facilitate use.

Session 7 From ToR to Management Response and Follow-up plan.

Activity 7: Biscuits evaluation and Management response.

Session 8 Diversity of evaluation approaches and related terminology.

Evaluation standards and ethics.

Closing.

Activity 8.1: Lecture “Diversity of approaches and terms”.


Activity 8.3: Concluding discussion.

Session 1IntroPreparation:In preparation to the session set plates with numbers and baskets of biscuits to all tables. Ideally tea should be served in the morning before and during Intro session. Place copies of baseline survey on each table.

Activity 1.1: Welcome and introductionDuration: 15 minutes


Materials: Presentation Flipchart sheet with the list of housekeeping rules. (Leave some space to add rules suggested by

participants).

Process:4. Welcome participants.5. Make a brief overview of training objectives and agenda using presentation.6. Present a list of household rules (Annex 1). Invite participants to offer additional rules as



Objectives: Get baseline data for training evaluation. Break participants into groups that have a balanced mix of people with more and less evaluation

experience. Have participants get to know each other.


Process:6. Ask participants to fill the baseline survey (printed copies of blank surveys should be on each

table).7. Ask each table to calculate cumulative baselines scores on participants’ knowledge and skills in

areas of project/program management, monitoring and evaluation.8. Ask each group to come up with a name for their group (optional).9. Ask each table to have one person to make a quick presentation about their group starting with

the cumulative scores. As people present, take a note of groups names and scores on a flipchart.10. Review the resulting table of scores.11. In the end of activity collect the surveys.

The Sustainable Development Goals (SDGs) – new global context for evaluation practiceLearning objectives:As a result of this session, participants shall:

Understand connection between the MDGs to the SDGs; Understand why the SDGs are relevant to Sri Lanka and person’s job; Obtain basic knowledge of the SDGs; Understand how to link existing interventions to the SDGs; Understand role of evaluation in the SDGs.


Objectives: Have participants understand that evaluation practice in embedded in a broader societal

context. Introduce participants to the SDGs.







Project evaluation practice started in the 1960s in the US in the context of War on Poverty and the Great Society programs when the federal government was heavily investing in social and educational programs. Senator Robert Kennedy, concerned that federal money would be misspent and not used to help disadvantaged children included evaluation clause into the Elementary and Secondary Education Act (ESEA) passed in 1965. ESEA required submission of an evaluation plan by local education agencies, and summary reports by state agencies. As a result, evaluation requirements became part of every federal grant. (Early expectations were that evaluation would illuminate the causes of social problems and the clear and specific means with which to fix such problems.)

Since September 2015 adoption of the Agenda 2030 created in a new global reality. The Agenda 2030 calls for evidence-based review of the progress towards SDGs and positions country-led evaluations as one of the tools contributing to this review.


9. Play video.10. Ask participants what are their key take way from the video. Record answers on the flipchart.



Materials: Presentation Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3). Copies of the SDGs (Annex 13).


SDGs are their complexity and interconnectedness.9. Make a short presentation on complexity and interconnectedness of the SDGs.


Complexity: o 5 pillars – People, Planet, Prosperity, Peace, Partnership;o 17 goals and 169 targets;o Targets of what and how.


one left behind. The SDGs will be implemented through various projects and programs contributing to specific

goals and targets. The SDGs are not something completely new. Many existing projects and programs are

already contributing to the SDGs.

10. Present the task for group work. First, divide 17 SDGs between tables (3 SDGs per table, so one of 17 SDGs will be given to two tables).

11. Then explain that participants should review the case of the “Rehabilitation of St. Sebastian South Canal” project implemented as part of the beautification of Colombo efforts and see if it contributes towards specific SDGs they were given and respective targets. Each participant should have a copy of the case and of the SDGs list.

12. Give groups 15 minutes to complete the task.

13. Ask groups to present results of their work explaining their choice. Each group can present one target at a time. Record goal and targets on a Flipchart. Go through all tables, then repeat the round until all targets identified by groups are listed.


Session 2Evaluation basicsLearning objectives:As a results of this session participants shall understand that evaluation is an instrument that provides decision-makers with information they need to make the pending decisions.

Activity 2.1: Mini-lecture Duration: 15 minutes

Objectives: Position evaluation within the management cycle. Explain that defining value (worth, merit) is part of decision making process. Explain that professional evaluation makes valuing process explicit.




Evaluation is part of the management cycle. Defining value (worth, merit) is part of decision making process. Professional evaluation makes valuing process explicit


Activity 2.2: Mapping decisions and relevant informationDuration: 45 minutes

Objectives: Build awareness about types of information used for different types of management decisions.

Materials: A4 paper, white or color (2 different colors) Flipcharts Markers White tape

Process:

1. Explain the task. Each group will have to identify 4-5 types of decisions that management of their agency makes. Each type of the decision has to be recorded on a separate A4 paper sheet. Then the group shall discuss what information is used to make these decisions and record it on separate sheets of A4 paper. (If you are using color paper, ask participants to use different colors for types of decisions and types of information needed for them).

2. Give participants 15-20 minutes to complete the task.3. Next step is to build a joint map of decisions/relevant information. You can put filled A4 sheets

of paper on a wall or on several pieces of flipchart paper put together. Each table can give you one decision/relevant information at a time:

Ask for the first decision/relevant information. Put A4 sheets in a left upper corner of a map.

Ask who has similar types of decision. Put all similar records near the first one. Then ask for a new type of decision (with relevant information). Put on a different place

on the map. Complete the process for this type of decision. Repeat the process as needed to complete the map. Once the map is complete, break for tea. During the tea review the map and decide

what decisions would require doing project/program (intervention) evaluation and where other ways of getting information for decision-making will be necessary.

After the tea break, present results of your analysis to participants.

Differences between evaluation and other ways of getting information for decision-makingLearning objectives:As a result of this session participants shall understand the difference between monitoring and evaluation and appreciate the value of evaluation.

Activity 2.3: Mini-lecture: project evaluation vs other types of getting information for decision makingDuration: 30 minutes

Objectives: Give participants experience with arriving at evidence-based conclusions and recommendations.

Materials: A4 sheets with names of the types of getting information for decision-making White tape Flipchart Markers

Process:1. Upon completing activity 2.3, say that now it’s time to convert your map into the map of types

of getting information for decision making.2. On a separate flipchart start building this map using A4 sheets with labels prepared in advance:

Put “Project/Program Evaluation” in the middle. Remind participants that this is what we are learning about in this workshop.

One by one add other titles on the map along with explanation about every type of getting information for decision making. Below is the tentative list of titles, but you can use additional ones based on your experience:

Monitoring Inspection Audit (financial and technical) Project proposal evaluation Personnel performance evaluation Product evaluation Financial evaluation (including cost-benefit analysis and Social Return on

Investment analysis) Research Organizational consulting

Once you have finished the map, explain that borders between these types of getting information for decision making are fuzzy, and that similar tools are used.

Ask participants which of these types of getting information for decision making are usually used to find and persecute wrong-doing by staff. Ask to explain why they think so. Mark these types with red marker.

Key points to be maintained in the discussion:

Project evaluation looks into how well a project model works, rather than how well project personnel works.

Project evaluation is not meant to discover wrong-doing on behalf of project personnel.

3. Conclude with Q&A.

Session 3How evaluators think about projectsActivity 3.1: Lecture “Logical frameworks, logical models and theories of change”Duration: 30 minutes

Objectives: Give participants general understanding of the language used in evaluation to describe different

levels of results; Give participants general understanding of logical frameworks, logical models and theories of

change.

Materials: Presentation (including video explaining difference between inputs, activities, outputs,

outcomes and impact. Video is also available at https://www.youtube.com/watch?v=lR8FalapJf0)



Explain the langue used by project planners and evaluators: inputs, activities, outputs, outcomes and impact.

Logical framework, logic model and ToC are complimentary ways of describing a project. Ideally logical framework, logic model and/or ToC should have been developed at the

planning stage. If not, evaluators have to reconstruct the project logic. Differences between logical framework, logic model and ToC. Building logical framework, logic model and ToC is both craft and art. Result largely depends

on the person who does them. Content and labels may somewhat differ.


Activity 3.2: Group exercise “Developing logic model for the Rehabilitation of St. Sebastian South Canal project”Duration: 60 minutes

Objectives: Give participants practical experience with development of logical models and theories of

change.

Materials: Flipchart Markers Sticky notes Copies of the case “Rehabilitation of St. Sebastian South Canal project” (Annex 3) Results from Activity 2.2: Linking project outcomes to the SDGs


Using sticky notes, each group first have to develop a logic model for Rehabilitation of St. Sebastian South Canal project on a piece of flipchart. Groups shall use results of Activity 2.2: Linking project outcomes to SDGs to inform their thinking about project impact.

Give people 30 minutes to complete the task. As groups work, go from table to table and see you participants are doing and provide assistance as necessary.

Then ask group to post the resulting posters on the wall. Tell people that there is no one right way of doing a logic model for the project, but it is

useful to learn from the way other people have done it. Instruct people to go and see logic models done by other groups.

Session 4:How evaluators think about projects (continued)Activity 4.1: Lecture “Evaluation criteria and questions”Duration: 30 minutes

Objectives:

Give participants general understanding of the origin and purpose of evaluation criteria; Give participants general understanding of how evaluation questions come from evaluation


Materials:

Presentation







Key evaluation questions are directly linked with evaluation criteria. Key evaluation question shall be developed by people who commission evaluation and will use its’ results. Sub-questions shall be jointly developed by evaluators and commissioners. Other decisions about evaluation design shall be done by evaluators themselves.

Evaluation Matrix. Evaluation sub-questions reflect the interpretation of corresponding evaluation criteria in the

context of evaluated intervention. Integration of SDGs criteria with DAC criteria. Concept of evaluative scales.


Activity 4.2: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”Duration: 60 minutes


intervention evaluation.

Give participants practical experience with developing evaluation questions.

Materials: Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal project” (Annex 3) Criteria matrix (Annex 4) Evaluation matrix (Annex 5) Flipchart showing pairs “commissioner” – “advisory body” (table numbers and team names)


Each group will have to decide which criteria shall be used for the final evaluation of the Rehabilitation of St. Sebastian South Canal sub-project using Criteria matrix.

Based on the selected criteria each group shall then develop key evaluation questions and corresponding sub-questions.

Selected criteria, key and sub-questions have to be put on a flipchart in a format of Evaluation matrix.

This will create a short ToR for the evaluation team. 7. Give groups 30 minutes to complete the task.8. Explain that now each table shall select a representative who will have to present the ToR they

developed to an “advisory body” made of participants from another table. Display a flipchart showing pairs “commissioner” – “advisory board”. For example:

Commissioner Advisory body

Table 1 Table 2

Table 2 Table 3

Table 3 Table 4

Table 4 Table 5

Table 5 Table 6

Table 6 Table 1

9. Ask one representative of a “commissioner” table go and present the ToR to a team at “advisory body” table.

10. Give 15 minutes for presentations and discussions with “advisory body”.11. Then ask presenters to go back to their tables and report back to their groups.12. Then have a plenary discussion about how people feel about the exercise that they have just

completed and what challenges did they have.13. In the end ask all teams to put flipchart with ToRs on the walls so that all participants could see

them.

Session 5Recap from Day 1Preparation:Review baseline survey filled by participants on Day 1. Come with the list of questions people want to get answers for in the workshop. Put them on sticky notes. In the morning of Day 2 put on the wall in the workshop room two flipcharts, one titled “Pending questions”, another – “Answered questions”.

Activity 5.1: Recap from Day 1Duration: 15 minutes

Objectives: Enhance key learning points from Day 1. Review questions that participants have regarding evaluation to see if some have been already

answered in Day 1.

Materials: Presentation Flipchart Markers White tape Sticky notes with questions that participants have

Process:1. Start by reminding people about yesterday’s survey. Present types of questions that emerged

from analysis of the answers, putting sticky note with questions on the flipchart titled “Pending questions”.

2. Make a short presentation covering key points from Day 1.3. Ask people if they have any questions regarding material covered on Day 1.4. Ask people if some of the questions they identified in the survey were answered. Move

answered ones to the flipchart titled “Answered questions”. Invite participant to add questions to the “Pending questions” if necessary using sticky notes.

How evaluators collect and analyze dataLearning objectives:As a result of this session participants shall have a basic understanding of how evaluators collect and analyze data.

Activity 5.1: Lecture “Marching evaluation questions with data collection methods”Duration: 30 minutes

Objectives: Give participants understanding of the sequence “Evaluation question – evidence necessary to

answer the questions – source of data to get the evidence – data collection method”.

Materials:

Presentation



Process of evaluation design should follow the sequence “Evaluation question – evidence necessary to answer the questions – source of data to get the evidence – data collection method”. This sequence is reflected in the Evaluation Matrix.

Overview of the most common evaluation methods: survey, interview, focus group, participatory methods, observation.

Triangulation is a key principle that undergirds evaluation. Types of triangulation:

o Triangulation of data sourceso Methodological triangulationo Implementer triangulationo Theoretical triangulationo Evaluation management triangulation (reference groups and steering committees)


Activity 5.2: Demonstration “Design of evaluation of the Rehabilitation of St. Sebastian South Canal project”Duration: 15 minutes

Objectives: Give participants basic understanding of going from evaluation question to necessary evidence

to data sources to data collection methods.

Preparation:Select two sub-questions from the ToRs prepared by participants under Activity 4.2: Group exercise “Development of evaluation questions for the Rehabilitation of St. Sebastian South Canal project” on Day 1. Complete the Evaluation Matrix for these sub-questions.

Materials: Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal project” (Annex 3) Two pieces of flipchart put together with Evaluation Matrix. Put two selected sub-question in

the matric in advance.

Process:1. Start by reminding participants that on Day 1 they selected criteria and developed evaluation

questions and sub-questions for evaluation of “Rehabilitation of St. Sebastian South Canal project”.

2. Explain that you have selected two sub-questions for this demonstration.

3. Complete the Evaluation Matrix by filling remaining empty columns for the first sub-question explaining the choices you made.

4. Do the same with the second sub-question.

Activity 5.3: Interactive Lecture “How evaluators analyze data to arrive at conclusions and recommendations”Duration: 30 minutes

Objectives: Give participants understanding of using evidence to answer evaluation questions and arrive to

conclusions. Give participants understanding of useful approaches of arriving at actionable

recommendations.

Materials: Presentation Copies of the case “Results of an evaluation of a capacity building project” (Annex 6) (You can

use your own case.)

Process:1. Make a short presentation covering all key points.

Key points for the lecture:

For every sub-question evaluators shall go through the following sequence: quantitative / qualitative analysis of data, comparing results to arrive to the answer.

Comparing answer for sub-questions under the same key evaluation questions to arrive at conclusion.

The best way to arrive at conclusion and use full recommendation is to go through the results of analysis together with commissioners and advisory body.

2. Invite participants to do the described steps for a case “Results of an evaluation of a capacity building project”.

3. Ask participants to review the case and quantitative and qualitative results for a sub-question “How useful was the training component?”

4. Ask participants how they would interpret these results.5. Ask participants what would be the conclusion for this question if this was mid-term evaluation

and if it was a final evaluation with no continuation for the evaluated project expected.6. Ask participants what recommendations they would make.

Session 6Evaluation useLearning objectives:As a result of this session participants shall understand factors that facilitate use of evaluation results.


Objectives: Help participants understand factors that facilitate use of evaluation results.





Then, on flipcharts, groups shall identify and record the factors that facilitated successful use of evaluation.

Groups will have about 30 minutes to complete the task. Resulting flipcharts shall be posted on walls.


factor from a table a time. Ask each group to explain the factor. Divide a flipchart into two vertical columns. Records factors identified by participants in the left column. After taking all ideas from participants, you may suggest some factors yourself if necessary.

16. Once the list is complete, what factors in the list are within control of the commissioners. Mark them.

17. Ask how commissioners can use these factors to get useful evaluation results.18. Complete with the discussion about what participants learned as a result of this exercise. Ask:

How participants feel about the exercise? What insights did they get in the process? Are there any pending questions?

Session 7From ToR to Management Response and Follow-up planActivity 7: Biscuits evaluation and Management responseDuration: 90 minutes

Objectives: Give participants opportunity to apply what they’ve learned so far into practice.

Materials: Copies of biscuits survey (Annex 8). Biscuits data sheets (Annex 9).

Flipchart Markers Flipchart with tables assignments.

Process:1. Explain the task. Each table will “wear” three hats in the course of this exercise: “sources of

data” – “evaluation team” – “commissioners”.2. Present the table showing how table shall interact:

Sources of information Evaluation team Commissioners

Table 1 Table 2 Table 3






3. Distribute biscuits surveys and ask people to fill them.4. Ask people at each table to put their survey sheets together and pass to assigned “Evaluation

team” table.5. Tell people that management of the training program is considering whether they should

continue the practice of giving biscuits on the first day of training. Teams at each table should answer the question “Is serving tea and biscuits in the beginning of a workshop a good practice and shall be continued?” using results of the survey (using the filled surveys they have) and information in the Biscuits data sheet. Conclusions and recommendations that they arrive at shall be put on in a left column of a flipchart divided into two columns. Explain, that the left column is for evaluation conclusions and recommendations and the right one is for the Management Response. Give participants 15 minutes to complete the task.

6. Ask “Evaluation teams” to send a representative to the corresponding “Commissioners” table to present results of evaluation. A person shall present the data, explain how the team interpreted it and what are their conclusions and recommendations.

7. Now “Commissioners” at each table have to come with a management response. For every conclusion and recommendation of a flipchart they got from the “Evaluation team” they will have to decide if they agree or disagree with the respective conclusion and recommendation and state what they are going to do if appropriate. Give groups 15 minutes to complete the task.

8. Have all groups present their Management responses.9. Continue with the reflection on the experience gained by participants:

How did the weight the information they had? How did the go from data to qualitative conclusion? How did the go from conclusion to recommendation? How did they decide how to respond to conclusions and recommendations? What challenges did they experience in the course of the exercise?

What have they learned from this exercise?

Session 8Diversity of evaluation approaches and related terminology. Evaluation standards and ethics.Learning objectives:As a result of this session participants shall become aware of the diversity of evaluation approaches they may encounter when they start commissioning evaluations.

Activity 8.1: Lecture “Diversity of approaches and terms”Duration: 45 minutes




And terminology in the field was evolving along the way.9. Ask participants to tell types of project evaluation they have heard of. Record the on the

flipchart.

Key points for presentation (use all Annex 10):


Evaluations done at different points during project lifetime:o ex ante and ex post evaluation, o midterm and final evaluation, o needs assessment




Theoretical approach to evaluation:o Utilization-focused evaluationo Theory-based evaluation



10. Conclude by saying that now participants should have understood that evaluation is a broad field, and that evaluators have to have a lot of knowledge and skills.

11. Explain that evaluation profession does self-regulation by developing evaluation standards, ethical codes and competencies list. Tell people that they have an examples of evaluation standards developed by the American Evaluation Association (Annex 11) in their workshop kits.


ClosingActivity 8.2: Final assessment Duration: 15 minutes

Objectives: Get data for training evaluation. Have participants reflects on what they have learned.






Objectives: Close the workshop.

Materials: Flipcharts “Pending questions” and “Answered questions” Flipchart Markers

Process:4. Start with reviews questions on the “Pending questions” flipchart and suggesting the ones that

might have already been answered. If participants agree, move relevant sticky notes to the “Answered questions” flipchart.

5. Provide answers to questions remaining on the “Pending questions” flipchart. If participants agree that answers were satisfactory, move relevant sticky notes to the “Answered questions” flipchart. If not, suggest where they can get more information on these questions.

6. Ask people if they have other pending questions. Provide answers.7. Invite participants to share their key insights and take-aways from the workshop. Record them






Area Score


Monitoring

Evaluation

TOTAL





The MCUDP is jointly financed by the International Bank for Reconstruction and Development (IBRD) of the World Bank group and the Government of Sri Lanka. MCUDP has three main components. The first component focuses on flood and drainage management and addresses the urgent issue of urban flooding, which regularly paralyzes the economy of the Metro Colombo Region with high socioeconomic costs. The second component focuses on urban development and infrastructure rehabilitation for Project Local Authorities (PLAs). The third component consists of implementation support.












Monitoring and Evaluation (M & E) and reporting will be carried out at two levels, at the PMU level and the SLLRDC level. The PMU has recruited a Social Specialist to undertake the M & E and reporting of social safeguards management to ensure that safeguards issues are sufficiently mitigated. The Social Officer at the SLLRDC and 02 Community Development Officers will monitor and report on safeguard implementation at the field level. This includes paying close attention to the delivery of entitlements to people who will be resettled and the Contractor‘s social safeguard obligations. The PMU will submit quarterly social safeguard progress and quality monitoring reports with yearly safeguard reviews, which will be carried out by independent consultants. These will form the basis for any improvements to be brought about in the safeguard policy framework and implementation arrangements.



Gender Equity

Relevance

Effectiveness

Efficiency

Sustainability

Impact



Sub-questions

Necessary evidence

Sources of Data

Methods Assumptions

Annex 6: Results of an evaluation of a capacity building project

Brief project description

The evaluated project aims to build capacity of small business owners. The project has two components: training and mentorship. Training included a series of eight interactive workshops. At each workshop a trainer was to present to participants materials on key aspects of business management and facilitate discussions on their experiences on these aspects.

Project was implemented in five sites (cities).

Evaluation

One of evaluation sub-question was “How useful was the training component?”

Evidence to answer this question was collected via participants’ survey, review of monitoring data, focus groups with participants and interviews with trainers.

Data collected in the course of evaluation

In the survey participants were asked to assess usefulness of the training component on a scale from 0 to 10. Results of each site and the project in general are presented below. Bars show the range of score from minimum to maximum as well as average scores (red boxes).

Graph below present the attendance of eight workshop in different sites – as percentage of participants at each site.

Themes that emerged in the course of interviews and focus groups with participants and trainers:

Trainers complained that they had to deliver to much material in the course of the workshop and did not have time for discussions among participants.

Participants felt that workshop did not allow for interactions and were more like lectures. Still many participants used ideas and instruments they learned in the workshops.

Questions to the group:

1. What would be your answer sub-question was “How useful was the training component?”2. What recommendation(s) could you make to the project management?


But it’s not all good news. As peace returned, small and medium enterprises (SMEs) found to their chagrin that China, India, and Bangladesh had all but cornered the world’s export markets in mass manufacturing and textiles – at their expense. To add to their woes, the national government had paid little attention to the sector over the years. For the hundreds of business owners renting land in the 18

government-owned industrial estates managed by the Industrial Development Board (IDB), the economic boom seemed to have all but passed them by.



In 2013, the situation began to change. The Ministry of Traditional Industries and Small Enterprise Development commissioned a comprehensive evaluation of the sector in 2013. A team of independent evaluators led by Ajith Ekanayaka began travelling to each of the 18 estates across the country to examine the issues plaguing the sector and to suggest direct solutions. Ekanayaka made it clear that this evaluation would not be relegated to a filing cabinet but would be put to immediate use.







Because policymakers had given frequent input through scoping sessions, and the evaluation was built on a broad, credible base of evidence and data as well as strong connections to stakeholders. According to Sivagnanasothy, a great deal has come from this single evaluation. He recalls: “When the rents went up, industrialists used to shout and cry How can we afford it? We were able to convince the central

government to cap rent increases at 7% annually. That’s a predictable environment, which is an enabling environment. Not only that, but the introduction of targeted loans with lower interest and longer grace periods meant that SMEs could now expand their operations more easily. In each of the estates, factory production is up, and with it, SME profits, leading to more jobs. It’s trickled down to the rural villages”.






Annex 8: Biscuits surveyManagement of this training program is constantly working to improve the quality of learning experiences we are providing to our participants. We would highly appreciate if you could answer this short survey to inform our efforts.

In the beginning of this workshop you were served biscuits.

Did you have some of these biscuits?

Yes No

How would you respond to the statement:

Serving biscuits in the beginning of a workshop is a nice way to welcome participants.

Strongly disagree DisagreeNeither agree, nor

disagreeAgree

Strongly agree

How long did it take you to get to this workshop from home?

Less than 30 minutes 30 to 60 minutes 1 to 2 hours Over 2 hours

Did you have breakfast before leaving?

Yes No

What is you gender?

Female Male

Annex 9: Biscuits data sheetsNutrition information

1 serving of biscuits (2 pieces/25 grams) contains:

Energy 125 kkal

Protein 1.3 g

Total fat 6 g

Saturated fat 5 g

Total carbohydrates 16 g

Sugars 7 g

Dietary fiber 0.5 g

Sodium 210 mg

Sugar and learning3

Glucose, a form of sugar, is the primary source of energy for every cell in the body. Because the brain is so rich in nerve cells, or neurons, it is the most energy-demanding organ, using one-half of all the sugar energy in the body.

Brain functions such as thinking, memory, and learning are closely linked to glucose levels and how efficiently the brain uses this fuel source. If there isn’t enough glucose in the brain, for example, neurotransmitters, the brain’s chemical messengers, are not produced and communication between neurons breaks down. In addition, hypoglycemia, a common complication of diabetes caused by low glucose levels in the blood, can lead to loss of energy for brain function and is linked to poor attention and cognitive function.

Although the brain needs glucose, too much of this energy source can be a bad thing. A 2012 study in animals by researchers at the University of California at Los Angeles indicated a positive relationship between the consumption of fructose, another form of sugar, and the aging of cells, while a 2009 study, also using an animal model, conducted by a team of scientists at the University of Montreal and Boston College, linked excess glucose consumption to memory and cognitive deficiencies.





Final evaluation in conducted right before of after the project completion. Needs assessment is conducted before project development of assess to needs of target

beneficiaries.

3 Sugar and the Brain. Retrieved from: http://neuro.hms.harvard.edu/harvard-mahoney-neuroscience-institute/brain-newsletter/and-brain-series/sugar-and-brain



AreaScore



Monitoring

Evaluation

TOTAL








1.b Create sound policy frameworks at the national, regional and international levels, based on pro-poor and gender sensitive development strategies, to support accelerated investment in poverty eradication actions





2.4 By 2030, ensure sustainable food production systems and implement resilient agricultural practices that increase productivity and production, that help maintain ecosystems, that strengthen capacity for

adaptation to climate change, extreme weather, drought, flooding and other disasters and that progressively improve land and soil quality


























11.5 By 2030, significantly reduce the number of deaths and the number of people affected and substantially decrease the direct economic losses relative to global gross domestic product caused by disasters, including water-related disasters, with a focus on protecting the poor and people in vulnerable situations
















14.6 By 2020, prohibit certain forms of fisheries subsidies which contribute to overcapacity and overfishing, eliminate subsidies that contribute to illegal, unreported and unregulated fishing and refrain from introducing new such subsidies, recognizing that appropriate and effective special and differential

treatment for developing and least developed countries should be an integral part of the World Trade Organization fisheries subsidies negotiation2















Finance






Technology




Capacity-building


Trade




Systemic issues











EVALUATION TRAINING

5-DAY MODULE

ContentModule Description.....................................................................................................................................4Session 1......................................................................................................................................................6Intro.............................................................................................................................................................6Activity 1.1: Welcome and introduction......................................................................................................6Activity 1.2: Baseline assessment and participants’ getting to know each other........................................6Session 2......................................................................................................................................................7Sustainable Development Goals (SDGs) – new global context for evaluation practice...............................7Activity 2.1: Intro to the SDGs.....................................................................................................................7Activity 2.2: Linking project outcomes to the SDGs.....................................................................................8Session 3:.....................................................................................................................................................9What is evaluation?.....................................................................................................................................9Activity 3.1: Biscuit evaluation.....................................................................................................................9Activity 3.2: Mini-lecture.............................................................................................................................9Activity 3.3: Mapping decisions and relevant information........................................................................10Session 4:...................................................................................................................................................11What is evaluation? (continued)...............................................................................................................11Activity 3.3: Mapping decisions and relevant information........................................................................11Activity 4.1: Mini-lecture: project evaluation vs other types of getting information for decision making.11Activity 4.2: Comparing monitoring and evaluation..................................................................................12Session 5:...................................................................................................................................................12Recap from Day 1......................................................................................................................................12Activity 5.1: Recap from Day 1...................................................................................................................13How evaluators think about projects........................................................................................................13Activity 5.2: Lecture “Logical frameworks, logical models and theories of change”..................................13Activity 5.2: Group exercise “Developing logic model for Rehabilitation of St. Sebastian South Canal project”.....................................................................................................................................................14Session 6....................................................................................................................................................14How evaluators think about projects (continued).....................................................................................14Activity 6.1: Lecture “Evaluation criteria and questions”..........................................................................14Activity 6.2: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”.................................................................................................................................15Sessions 7-8...............................................................................................................................................16How evaluators collect and analyze data..................................................................................................16Activity 7: EvalMethods Learning Lab........................................................................................................16Session 9....................................................................................................................................................17Recap from Day 2......................................................................................................................................17

Activity 9.1: Recap from Day 2...................................................................................................................17How evaluators collect and analyze data (continued)...............................................................................17Activity 9.2: Lecture “Matching evaluation questions with data collection methods”..............................17Activity 9.3: Group exercise “Design of evaluation of the Rehabilitation of St. Sebastian South Canal project”.....................................................................................................................................................18Session 10..................................................................................................................................................19How evaluators collect and analyze data (continued)...............................................................................19Activity 10: Sampling Learning Lab............................................................................................................19Sessions 11-12...........................................................................................................................................20How evaluators collect and analyze data (continued)...............................................................................20Activity 11: Evaluation of EvalMethods Learning Lab/ Sampling Learning Lab/ Biscuits Evaluation..........20Session 13..................................................................................................................................................21How evaluators collect and analyze data (continued)...............................................................................21Activity 13: Presentations of evaluation results........................................................................................21Session 14..................................................................................................................................................21Evaluation use...........................................................................................................................................21Activity 14: Analysis of factors that facilitate use......................................................................................21Session 15:.................................................................................................................................................22Evaluation process from management demand for information to use of evaluation results..................22Activity 15.1: Q&A.....................................................................................................................................22Activity 15.2: Lecture/Demonstration “Models of evaluation”.................................................................22Session 16..................................................................................................................................................24Comparison between small-scale internal, large-scale internal and external evaluation..........................24Activity 16: Debate - advantages and disadvantage for evaluation of different management levels........24Session 17..................................................................................................................................................25Activity 17.1: Recap from Day 4.................................................................................................................25Diversity of evaluation approaches...........................................................................................................25Activity 17.2: Lecture.................................................................................................................................25Session 18..................................................................................................................................................26Evaluator competences.............................................................................................................................26Activity 18.1: Self-assessment of evaluation competencies......................................................................26Activity 18.2: Paths for professional development....................................................................................27Session 19..................................................................................................................................................27Evaluation standards and ethics................................................................................................................27Activity 19.1: Develop your own standards...............................................................................................28Session 20..................................................................................................................................................29Closing.......................................................................................................................................................29Activity 20.1: Q&A.....................................................................................................................................29Activity 20.2: Final assessment..................................................................................................................29Activity 20.3: Concluding discussion..........................................................................................................30Annexes.....................................................................................................................................................31Annex 1: Household rules..........................................................................................................................31Annex 2: Baseline survey...........................................................................................................................32Annex 3: Case: Rehabilitation of St. Sebastian South Canal......................................................................33Annex 4: Biscuits survey............................................................................................................................36

Annex 5: Biscuits data sheets....................................................................................................................37Annex 6: Criteria matrix.............................................................................................................................38Annex 7: Evaluation Matrix.......................................................................................................................39Annex 8: EvalMethods Lab Information Pack............................................................................................40Annex 9: Case: Salvaging Sri Lanka’s small and medium businesses: how an evaluation led to rapid change.......................................................................................................................................................47Annex 10: Different types of evaluation....................................................................................................49Annex 11: AEA Evaluator Competencies...................................................................................................51Annex 12: Final survey...............................................................................................................................54Annex 13: List of the Sustainable Development Goals..............................................................................55

Module DescriptionTarget audience:Public officials and professionals who are already doing or planning to perform project/program evaluations.

Training overall objective:This training shall contribute to implementation of the broad vision that every public official in Sri Lanka should understand importance of evaluation and basics of evaluation practice.

The specific objective of this 5-day training module is to give public officials who are already doing or planning to perform project/program evaluations basic practical skills and knowledge to conduct high-quality evaluations.

Number of participants: about 30 people.

Recommended room set up:6 rounds tables to seat 5 participants each. Set a plate with a number (from 1 to 6) to each table.

Agenda overview

Day 1 Topics Activities

Session 1 Intro Activity 1.1: Welcome and introduction

Activity 1.2: Baseline assessment and participants’ getting to know each other

Session 2 SDGs Activity 2.1: Intro to the SDGs

Activity 2.2: Linking project outcomes to the SDGs


Discussion on types on decisions and relevant information

Activity 3.1: Biscuit evaluation

Activity 3.2: Mini-lecture

Activity 3.3: Mapping decisions and relevant information

Session 4 Differences between evaluation and other ways of getting information for decision-making

Complete activity 3.3

Activity 4.1: Mini-lecture: Project evaluation vs other types of getting information for decision making

Activity 4.2: Comparing monitoring and evaluation

Day 2

Session 5 How evaluators think about projects

Activity 5.1: Recap from Day 1

Activity 5.2: Lecture “Logical frameworks, logical models and theories of change”

Activity 5.2: Group exercise “Developing logic model for Rehabilitation of St. Sebastian South Canal project”

Session 6 Activity 6.1: Lecture “Evaluation criteria and questions”

Activity 6.2: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”

Session 7 How evaluators collect and analyze data

Activity 7: EvalMethods Learning Lab

Session 8

Day 3

Session 9 How evaluators collect and analyze data - continued


Activity 9.2: Lecture “Marching evaluation questions with data collection methods”

Activity 9.3: Group exercise “Design of evaluation of the Rehabilitation of St. Sebastian South Canal project”

Session 10 Activity 10: Sampling Learning Lab

Session 11 Activity 11: Evaluation of EvalMethods Learning Lab/ Sampling Learning Lab/ Biscuits evaluation

Session 12

Day 4

Session 13 Presentation of evaluation reports

Activity 13: Presentations of evaluation results

Session 14 Use of evaluation results Activity 14: Analysis of factors that facilitate use

Session 15 Evaluation process - recap Activity 15.1: Q&A

Activity 15.2: Lecture/Demonstration “Models of evaluation”

Session 16 Comparison between small-scale internal, large-scale internal and external evaluation

Activity 16: Debate - advantages and disadvantage for evaluation of different management levels

Day 5

Session 17 Diversity of evaluation approaches and related terminology


Activity 17.2: Lecture

Session 18 Required evaluator competencies and relevant professional development pathways for evaluators

Activity 18.1: Self-assessment of evaluation competencies

Activity 18.2: Paths for professional development

Session 19 Evaluation standards and ethics

Activity 19.1: Develop your own standards

Activity 19.2: Review of the UNEG standard on methodology

Session 20 Closing Activity 20.1: Q&A

Activity 20.2: Final assessment

Activity 20.3: Concluding discussion

Session 1IntroPreparation:In preparation to the session set plates with numbers (from 1 to 6) and baskets of biscuits at all tables. Ideally tea should be served in the morning before and during Intro session. Place copies on baseline survey on each table.

Activity 1.1: Welcome and introductionDuration: 15 minutes


Materials: Presentation Flipchart sheet with the list of housekeeping rules (Leave some space to add rules suggested by

participants).

Process:7. Welcome participants.8. Make a brief overview of training objectives and agenda using presentation.9. Present a list of household rules (Annex 1). Invite participants to offer additional rules as



Objectives: Get baseline data for training evaluation. Break participants into groups that have a balanced mix of people with more and less evaluation

experience. Have participants get to know each other.


Process:12. Ask participants to fill the baseline survey (printed copies of blank survey should be on each

table).13. Once people have completed the survey, mark a line across the room representing a scale from

0 to 10, and ask people to stand along this line according to the score they have given to their competencies level in the area of project/program evaluation in the survey.

14. Assign participants to tables. First give numbers from 1 to 6 to first six people starting from the higher end of the scale and ask them to go the respective tables. Then give numbers from 1 to 6 to first six people starting from the lower end of the scale and ask them to go the respective tables. Continue alternating the ends of the scale. This process should result in groups with approximately equal mix of participants in terms of evaluation experience.

15. Once participants settle on assigned tables, ask them to introduce themselves to other participants at the table and explain the scores they gave themselves in the baseline survey.

16. Ask each table to calculate cumulative baselines scores on participants’ knowledge and skills in areas of project/program management, monitoring and evaluation.

17. Ask each group to come up with a name for their group (optional).18. Ask each table to have one person to make a quick presentation about their group starting with

the cumulative scores. As people present, take a note of groups names and scores on a flipchart.19. Review the resulting table of scores.20. Before breaking for tea, collect the surveys.

Session 2Sustainable Development Goals (SDGs) – new global context for evaluation practiceLearning objectives:As a result of this session, participants shall:

Understand connection between the MDGs to the SDGs; Understand why the SDGs are relevant to Sri Lanka and person’s job; Obtain basic knowledge of the SDGs; Understand how to link existing interventions to the SDGs;

Understand role of evaluation in the SDGs.


Objectives: Have participants understand that evaluation practice in embedded in a broader societal

context. Introduce participants to SDGs.







Project evaluation practice started in the 1960s in the US in the context of War on Poverty and the Great Society programs when the federal government was heavily investing in social and educational programs. Senator Robert Kennedy, concerned that federal money would be misspent and not used to help disadvantaged children included evaluation clause into the Elementary and Secondary Education Act (ESEA) passed in 1965. ESEA required submission of an evaluation plan by local education agencies, and summary reports by state agencies. As a result, evaluation requirements became part of every federal grant. (Early expectations were that evaluation would illuminate the causes of social problems and the clear and specific means with which to fix such problems.)

Since September 2015 adoption of Agenda 2030 created in a new global reality. Agenda 2030 calls for evidence-based review of the progress towards SDGs and positions country-led evaluations as one of the tools contributing to this review.


14. Play video.15. Ask participants what are their key take way from the video. Record answers on the flipchart.



Materials: Presentation Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3). Copies of the SDGs (Annex 13).


SDGs are their complexity and interconnectedness.16. Make a short presentation on complexity and interconnectedness of SDGs.


Complexity: o 5 pillars – People, Planet, Prosperity, Peace, Partnership;o 17 goals and 169 targets;o Targets of what and how.


one left behind. SDGs will be implemented through various projects and programs contributing to specific

goals and targets. SDGs are not something completely new. Many existing projects and programs are already

contributing to SDGs.

17. Present the task for group work. Participants should review the case of “Rehabilitation of St. Sebastian South Canal project” implemented as part of the beautification of Colombo efforts and identify what specific SDGs and targets the project was contributing. Each participant should have a copy of the case and of the SDGs.

18. Give groups 20-30 minutes to complete the task.19. Ask groups to present the selected SDGs and targets explaining their choice. Each group can

present one target at a time. Record goal and targets on a Flipchart. Go through all tables, then repeat the round until all targets identified by groups are listed.


Session 3: What is evaluation?Learning objectives:As a result of this session participants shall understand that evaluation is an instrument that provides decision-makers with information they need to make the pending decision.

Activity 3.1: Biscuit evaluationDuration: 45 minutes


Materials: Copies of biscuits survey (Annex 4). Biscuits data sheets (Annex 5). Flipchart Markers

Process:10. Distribute biscuits surveys and ask people to fill them.11. Ask people at each table to put their survey sheets together and pass to a next table.12. Tell people that management of the training program is considering whether they should

continue the practice of giving biscuits on the first day of training. Teams at each table should answer the question “Is serving tea and biscuits in the beginning of a workshop a good practice and shall be continued?” using results of survey (using the filled surveys they have) and information in the Biscuits data sheets. Then each team will have to make a present and justify its’ recommendation to the management. Give participants 10-15 minutes to complete the task.

13. Ask each group to present results of their discussion: conclusion and recommendation along with explanation why they have arrived at them. Make sure that each group explicitly answer the question “Is serving tea and biscuits in the beginning of a workshop a good practice and shall be continued?”After the first group presents, ask if some group arrived at a different conclusion and let them present second. If all group come to same conclusions, still ask each group to contribute to a discussion by explaining their reasoning process.

14. Continue with the reflection on the experience gained by participants: How did the weight the information they had? How did the go from data to qualitative conclusion? How did the go from conclusion to recommendation? What challenges did they experience in the course of the exercise?

Activity 3.2: Mini-lecture Duration: 15 minutes

Objectives: Position evaluation within the management cycle. Explain that defining value (worth, merit) is part of decision making process. Explain that professional evaluation makes valuing process explicit.




Evaluation is part of the management cycle. Defining value (worth, merit) is part of decision making process. Professional evaluation makes valuing process explicit.


Activity 3.3: Mapping decisions and relevant informationDuration: 45 minutes

Objectives: Build awareness about types of information used for different types of management decisions.

Materials: A4 paper, white or color (2 different colors) Flipcharts Markers White tape

Process:4. Explain the task. Each group will have to identify 4-5 types of decisions that management of

their agency makes. Each type of the decision has to be recorded on a separate A4 paper sheet. Then the group shall discuss what information is used to make these decisions and record it on separate sheets of A4 paper. (If you are using color paper, ask participants to use different colors for types of decisions and types of information).

5. Give participants 15-20 minutes to complete the task.6. Next step is to build a joint map of decisions/relevant information. You can create this map on a

wall or use several pieces of flipchart paper put together. Each table can give you one decision/relevant information at a time:

Ask for the first decision/relevant information. Put A4 sheets in a left upper corner of a map.

Ask who has similar types of decision. Put all similar records near the first one. Then ask for a new type of decision (with relevant information). Put on a different place

on the map. Complete the process for this type of decision. Repeat the process as needed to complete the map. Once the map is complete, break for tea. During the tea review the map and decide

what decisions would require doing project/program (intervention) evaluation and where other ways of getting information for decision-making will be necessary.

After the tea break, present results of your analysis to participants.

Session 4: What is evaluation? (continued)Activity 3.3: Mapping decisions and relevant informationComplete activity 3.3 – 15 minutes

Activity 4.1: Mini-lecture: project evaluation vs other types of getting information for decision makingDuration: 30 minutes


Materials: A4 sheets with names of the types of getting information for decision-making White tape Flipchart Markers

Process:4. Upon completing activity 3.3, say that now it’s time to convert your map into the map of types

of getting information for decision making.5. On a separate flipchart start building this map using A4 sheets with labels prepared in advance:

Put “Project/Program Evaluation” in the middle. Remind participants that this is what we are learning about in this workshop.

One by one add other titles on the map along with explanation about every type of getting information for decision making. Below is the tentative list of titles, but you can use additional ones based on your experience:

Monitoring Inspection Audit (financial and technical) Project proposal evaluation Personnel performance evaluation Product evaluation Financial evaluation (including cost-benefit analysis and Social Return on

Investment analysis) Research Organizational consulting

Once you have finished the map, explain borders between these types of getting information for decision making are fuzzy, and that similar tools are used.

Ask participants which of these types of getting information for decision making are usually used to find and persecute wrong-doing by staff. Ask to explain why they think so. Mark these types with red marker.

Key points to be maintained in the discussion:

Project evaluation looks into how well a project model works, rather than how well project personnel works.

Project evaluation is not meant wrong-doing on behalf of project personnel.

6. Conclude with Q&A.

Activity 4.2: Comparing monitoring and evaluationDuration: 45 minutes

Objectives: Give participants clear understanding of differences and similarities between monitoring and

evaluation. Enhance their understanding of different roles of monitoring and evaluation in SDGs context.

Materials: Copies of the EVALSDGs Briefing “Counting critically: SDG ‘follow-up and review’ needs

interlinked indicators, monitoring and evaluation” – available at http://pubs.iied.org/17363IIED/ Flipchart Markers White tape

Process:1. Start by saying that because both monitoring and evaluation happen within a context of a

project, the dividing line between them is often very fuzzy. So it’s important to understand the differences between the two.

2. Explain the task. Working in groups at tables, participants shall review the Briefing and based on the text identify differences and similarities between monitoring and evaluation. Each group shall present results of the discussion on a flipchart using the following format:

Monitoring Evaluation

What they have in common?

What is specific only to M? What is specific only to E?

3. Give participants 15-20 minutes to complete the task.4. Ask groups to post resulting posters on the walls. Do the Gallery Walk: one person from each

team shall stay by the team poster, the rest of the group shall go for one poster to another and review their content. A person at the poster shall answer the questions and take note of suggestions. The trainer shall go with one of the groups to see results of groups work.

5. Bring participants back to tables. Conclude exercise with reflection: Did they learn something surprising when doing this activity? What are the main things that they learned?

Session 5:Recap from Day 1Preparation:

Review baseline survey filled by participants on Day 1. Come with the list of questions people want to get answers for in the workshop. Record them on sticky notes. In the morning of Day 2 put on the wall in the workshop room two flipcharts, one titled “Pending questions”, another – “Answered questions”.



answered in Day 1.

Materials: Presentation Flipchart Markers White tape Sticky notes with questions that participants have

Process:5. Start by reminding people about yesterday’s survey. Present types of questions that emerged

from analysis of their answers, putting sticky note with questions on the flipchart titled “Pending questions”.

6. Make a short presentation covering key points from Day 1.7. Ask people if they have any questions regarding material covered on Day 1.8. Ask people if some of the questions they identified in the survey were answered. Move

answered ones to the flipchart titled “Answered questions”. Invite participant to add questions to the “Pending questions” if necessary using sticky notes.

How evaluators think about projectsLearning objectives:As a result of studying this topic participants shall understand existing approaches to describing project logic.Activity 5.2: Lecture “Logical frameworks, logical models and theories of change”Duration: 30 minutes

Objectives: Give participants general understanding of the language used in evaluation to describe different

levels of results; Give participants general understanding of logical frameworks, logical models and theories of

change.

Materials: Presentation (including video explaining difference between inputs, activities, outputs,

outcomes and impact. Video is also available at https://www.youtube.com/watch?v=lR8FalapJf0)



Explain the langue used by project planners and evaluators: inputs, activities, outputs, outcomes and impact.

Logical framework, logic model and ToC are complimentary ways of describing the project. Ideally, they should have been developed at the planning stage. If not, evaluators have to reconstruct the project logic. Differences between logical framework, logic model and ToC. How one way of presenting the project can be converted into another. Building logical framework, logic model and ToC is both craft and art. Result large depends on

the person who does the. Content and labels may somewhat differ.


Activity 5.2: Group exercise “Developing logic model for Rehabilitation of St. Sebastian South Canal project”Duration: 45 minutes

Objectives: Give participants practical experience with development of logical models and theories of

change.

Materials: Flipchart Markers Sticky notes Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3) Results from Activity 2.2: Linking project outcomes to the SDGs


Using sticky notes, each group first have to develop a logic model for Rehabilitation of St. Sebastian South Canal project on a piece of flipchart. Groups shall use results of Activity 2.2: Linking project outcomes to SDGs on Day 1 to inform their thinking about project impact.

3. Give people 30 minutes to complete the task. As groups work, go from table to table and see you participants are doing and provide assistance as necessary.

4. Then ask group to post the resulting posters on the wall. 5. Tell people that there is no one right way of doing a logic model or the project, but it is useful to

learn from the way other people have done it. Instruct people to go and see logic models created by other groups.

Session 6How evaluators think about projects (continued)Activity 6.1: Lecture “Evaluation criteria and questions”Duration: 45 minutes

Objectives: Give participants general understanding of the origin and purpose of evaluation criteria; Give participants general understanding of how evaluation questions come from evaluation









Key evaluation questions are directly linked with evaluation criteria. Evaluation Matrix is a useful tool for evaluation design. Evaluation sub-questions reflect the interpretation of corresponding evaluation criteria in the

context of evaluated intervention. Integration of the SDGs criteria with the OECD-DAC criteria – Sri Lanka National Equity

Focused and Gender Responsive Evaluation Guidelines.


Activity 6.2: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”Duration: 45 minutes


intervention evaluation. Give participants practical experience with developing evaluation questions.


Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3) Criteria matrix (Annex 6) Evaluation matrix (Annex 7) Flipchart showing pairs “commissioner” – “implementer” (table numbers and team names)


Each group will have to decide which criteria shall be used for the final evaluation of the Rehabilitation of St. Sebastian South Canal sub-project using Criteria matrix.

Based on the selected criteria each group shall then develop key evaluation questions and corresponding sub-questions.

Selected criteria, key and sub-questions have to be put on a flipchart in a format of Evaluation matrix.

This will create a short ToR for the evaluation team that will later in this workshop develop an evaluation design.

15. Give groups 30 minutes to complete the task.16. Explain that each table will use the ToR they developed to “commission” evaluation from

another table. Display a flipchart showing pairs “commissioner” – “implementer”. For example:

Commissioner Implementer

Table 1 Table 2

Table 2 Table 3

Table 3 Table 4

Table 4 Table 5

Table 5 Table 6

Table 6 Table 1

17. Ask one representative of a “commissioner” table to go and present the ToR to a team at “implementer” table.

18. Have a plenary discussion about how people feel about the exercise that they have just completed and what challenges did they have.

19. In the end ask all team to put flipchart with ToRs on the walls so that all participants could see them during lunch break.

Sessions 7-8How evaluators collect and analyze dataLearning objectives:

Participants shall have understanding on how methodological decisions are made in the course of evaluation.

Activity 7: EvalMethods Learning LabDuration: 180 minutes

Objectives: Give participants general understanding of most commonly used data collection methods: desk

review, surveys, individual interviews, focus groups, participatory approaches.

Materials: Information pack on all types of methods to be examined in the lab (Annex 8) Flipcharts Markers Signs with method names to put on tables

Process:1. Introduce the task:

The EvalMethods Learning Lab will examine three common evaluation data collections methods: questionnaires/surveys, interviews, focus groups.

To do this, we will have three “research” teams, and each table shall delegate one-two people to each of these teams.

Each team will have to study one of the methods and prepare a presentation on a flipchart – like infographics – presenting advantages and limitations of the methods and some ‘how to do’ tips about collecting and analyzing data when using this method.

This work has to be complete by the next tea break. After the break teams will have to put their posters on the wall. And members of

“research” teams will have to return to other original groups. Now each group will have one to two experts in one of three methods.

Than groups will go around the room from poster to poster. At a relevant poster expert(s) on this method will have a make a 10-minute presentation to his/her group plus 5 minutes for Q&A. After 15 minutes, a group will have to move to the next poster.

2. Assign tables to “research” teams. Ask people to move accordingly.3. Start the exercise.4. After all presentations are completed, ask participants to return to their table. Ask how they feel

about this exercise. What have they learned? Are there any pending questions? Answer questions as necessary.

Session 9Recap from Day 2Preparation:Review “Pending questions”, note the ones that, in your opinion, can be moved to “Answered questions”.


Objectives: Enhance key learning points from Day 2.

Review questions that participants have regarding evaluation to see if some have been already answered in Day 2.

Materials: Presentation Flipchart Markers White tape

Process:1. Make a short presentation covering key points from Day 2.2. Ask people if they have any questions regarding material covered on Day 2.3. Go to “Pending questions” flipchart. Suggest the questions that might have already been

answered. If participants agree, move answered question to the flipchart titled “Answered questions”. Invite participant to add questions to the “Pending questions” if necessary using sticky notes.

How evaluators collect and analyze data (continued)Activity 9.2: Lecture “Matching evaluation questions with data collection methods”Duration: 30 minutes

Objectives: Give participants understanding of the sequence “Evaluation question – evidence necessary to

answer the questions – source of data to get the evidence – data collection method”.




Process of evaluation design should follow the sequence “Evaluation question – evidence necessary to answer the questions – source of data to get the evidence – data collection method”

Triangulation is one of the key principles that undergird evaluation. Types of triangulation:

o Triangulation of data sourceso Methodological triangulationo Implementer triangulationo Theoretical triangulationo Evaluation management triangulation (reference groups and steering committees)


Activity 9.3: Group exercise “Design of evaluation of the Rehabilitation of St. Sebastian South Canal project”Duration: 45 minutes

Objectives: Give participants practical experience with going from evaluation question to necessary

evidence to data sources to data collection methods.

Materials: Flipchart Markers Copies of case “Rehabilitation of St. Sebastian South Canal” (Annex 3) Flipchart showing pairs “commissioner” – “implementer” (table numbers and team names) Flipcharts produced as a result of Activity 6.2: Group exercise “Development of evaluation

questions for Rehabilitation of St. Sebastian South Canal project”


Each group will work with the criteria and evaluation questions developed by the relevant “commissioner” team as assigned in Activity 6.2: Group exercise “Development of evaluation questions for Rehabilitation of St. Sebastian South Canal project”.

Each group will have to complete Evaluation matrix by selecting appropriate evidence needs, data sources, data collections methods and assumption about availability of data.

Once the design is complete, one representative of the “implementer” table will have to present the design to the “commissioner” table.

2. Give participants 20-30 minutes to complete the task.3. Give 10 minutes for presentation of evaluation design to “commissioner” teams.4. Give 5 minutes for presented to report on “commissioner” feedback to their “implementer”

teams.

Session 10How evaluators collect and analyze data (continued)Activity 10: Sampling Learning LabDuration: 90 minutes

Objectives: Give participants general understanding of the commonly used ways of sampling.

Materials: Information pack on sampling types to be examined in the lab Flipcharts Markers Copies of the article “Sampling” by E. Taylor-Powell. Available at

http://learningstore.uwex.edu/assets/pdfs/G3658-3.PDF Plates for tables with topics (see below)

http://learningstore.uwex.edu/assets/pdfs/G3658-3.PDF


Each table shall delegate one person to one of five “research” teams. Research teams will work of the following topics:

i. Probability samplingii. Purposeful sampling – Extreme or deviant case sampling and Maximum

variation samplingiii. Purposeful sampling – Typical case sampling and Critical case samplingiv. Purposeful sampling – Snowball sampling and Opportunistic samplingv. Purposeful sampling – Criterion sampling and Theory-based sampling

Each group will have to review information on the assigned sampling approaches in the provided article, discuss how they could be applied in case of projects team members are familiar with and make a flipchart presentation with key information about this approach and examples. Teams will have 20 minutes to complete the task.

Then each team will have to make a 3-minute presentation of the approach in plenary.5. Set plates the topics on five tables. Allow people to move and join the “research” teams.6. Start the exercise.7. After all presentations are completed, ask participants to return to their original tables. Ask how

they feel about this exercise. What have they learned? Are there any pending questions? Answer questions as necessary.

Sessions 11-12How evaluators collect and analyze data (continued)Activity 11: Evaluation of EvalMethods Learning Lab/ Sampling Learning Lab/ Biscuits EvaluationDuration: 180 minutes

Objectives:

Give participants opportunity to apply what they’ve learned so far into practice.

Materials: A4 paper Printer


By the end of the day all teams will have to conduct an evaluation of one of the activities conducted in this workshop – going from design to conclusions and recommendations. “Implementers” will be able to use people at one of other tables as sources of information, but they are entitled to just 30 minutes of their time in total and will have to negotiate when people can contribute to data collection.

Present assignment of “implementers” and “sources of information”. For example,

Project to evaluate Implementer Sources of information

Biscuits evaluation Table 1 Table 3

Biscuits evaluation Table 2 Table 4

EvalMethods Learning Lab Table 3 Table 5

EvalMethods Learning Lab Table 4 Table 6

Sampling Learning Lab Table 5 Table 1

Sampling Learning Lab Table 6 Table 2

Explain the ToR. All evaluations should answer two key evaluation questions:- Was an exercise relevant?- Was an exercise effective?

Teams will have to develop logic models for an intervention, complete Evaluation matrix starting with sub-questions, develop data collection tool, collect and analyze the data, arrive and conclusions and recommendations.

Remind participants about the principle of triangulation. Ask teams to show you first their Evaluation matrix for review, and then their data

collection instrument before starting data collection. Tell that results of their evaluation will have to be presented tomorrow during the first

session.2. Start exercise.3. Review Evaluation matrices and data collection tools developed by the teams, recommend

improvements.4. Observe teams working, take notes of things that work well or should be improved for

tomorrow’s discussion.

Session 13How evaluators collect and analyze data (continued)Activity 13: Presentations of evaluation resultsDuration: 90 minutes

Objectives: Give participants opportunity to apply what they’ve learned so far into practice.

Materials: Flipcharts Markers

Process:1. Give each team 10 minutes to present results of their evaluations. Presentations shall cover:

Purpose of evaluation and evaluation questions; Evaluation design (sources of data and methodology); Evaluation findings; Conclusions;

Recommendations.2. Allow other participants to ask questions and make comments after each presentation.3. Record conclusions and recommendations from each team on a flipchart.4. Once all teams have completed their presentations, open the floor for general discussion. Ask

participants: How they feel about the exercise? What aspect of evaluation was most challenging? How did the overcome these challenges? How did not weigh data in the analysis stage? What have they learned?

Add your own comments that came from observation of the teams working on evaluations and review of Evaluation matrices and data collection instruments.

Session 14Evaluation useLearning objectives:As a result of this session participants shall understand factors that facilitate use of evaluation results.


Objectives: Help participants understand factors that facilitate use of evaluation results.





Then groups shall identify and record the factors that facilitated successful use of evaluation on flipcharts.

Groups will have about 30 minutes to complete the task. Resulting flipcharts shall be posted on walls.


factor from a table a time. Ask each group to explain the factor. Divide a flipchart into two

vertical columns. Records factors identified by participants in the left column. After taking all ideas from participants, you may suggest some factors yourself if necessary.

25. Once the list is complete, what factors in the list are within control of the evaluators. Mark them.

26. Then, taking one marked factor at a time, ask participants to suggest what they as evaluator can do to have this factor in place when doing evaluation.

27. Complete with the discussion about what participants learned as a result of this exercise. Ask: How participants feel about the exercise? What insights did they get in the process? Are there any pending questions?

Session 15:Evaluation process from management demand for information to use of evaluation resultsActivity 15.1: Q&ADuration: 45 minutes

Objectives: Clarify pending questions regarding evaluation process.

Materials: Flipcharts “Pending questions” and “Answered questions”



2. Ask people if they have other pending questions regarding execution of evaluation process.

Activity 15.2: Lecture/Demonstration “Models of evaluation”Duration: 45 minutes

Objectives: Review the material on evaluation process learned in previous sessions. Introduce an idea that ways of presenting evaluation results depend on the type of evaluation

commissioner. Prepare participants to Activity 16 in the next session.

Materials: Flipchart Markers White tape A4 pagers with necessary information (as per table below) Presentation

Process:1. Put together three flipchart sheets and divide them into five columns as per the table below.

Write column title.2. Start lecture/demonstration by reminding participants that project evaluation emerged in

response to the need of donor organizations to have an independent information about the projects they funded.

3. Put the label “Top management of donor organizations and parliament” under “Commissioner” in the table. Fill the first row of the table commenting along the way. Have detailed explanation about ways of presenting evaluation results using presentation slide.

Commissioner Information need

Evaluation process Points of interaction between

commissioner and evaluator

Ways of presenting evaluation results

Top management of donor organizations and parliament

Is project good?

Start with criteria – design – data collection – analysis – conclusions and recommendations – preparation of communication products

Focus/conceptua-lization session

Discussion about method and measurement options

Review of evaluation design and instruments

Data interpretation session

Full report:

Purpose Methodology Findings Conclusions Recommend

ations

Executive summary

Presentation

Brief

Top management of implementing organizations

Is project good?

Start with criteria – design – data collection – analysis – conclusions and recommendations – preparation of communication products


Discussion about method and measurement options



Full report:

Purpose Methodology Findings Conclusions Recommend

ations

Executive summary

Presentation

Brief

Project management

What’s wrong with a specific aspect of

Start with the issue – design – data collection – analysis – conclusions and


Discussion about

Presentation

the project?

recommendations – preparation of communication products

method and measurement options



4. Fill the other two rows in the table. Once you get to the issue of presenting evaluation findings for project management, ask participants if they think that all way of communication, including fell report, are still needed. Try to make the process of filling the table interactive by asking people to give their suggestions before filling the next cell in the table.

5. After filling the table, again highlight the idea that there are different ways of presenting evaluation findings and that they should be selected based on management needs.

6. Ask participants is they have pending questions and comments.

Session 16Comparison between small-scale internal, large-scale internal and external evaluationLearning objectives:As a result of this session participants shall understand advantages and disadvantages of external and internal evaluation.

Activity 16: Debate - advantages and disadvantage for evaluation of different management levelsDuration: 90 minutes

Objectives: Enhance participants’ understanding of internal vs external evaluation.

Materials: Flipchart with the table from Activity 15.2 Blank flipchart Markers Cards with debate topics

Process:1. Explain differences between approaches:

Small-scale internal evaluation

Project implementing team and project M&E officer (if there is one) can do a small-scale evaluation (collect and interpret data) to answer questions that project management may get in the course of project implementation, e.g. emerging from monitoring data.

Large-scale internal M&E department of an organization that implements a project

evaluation may conduct an evaluation of this project without hiring external evaluators.

External evaluation Evaluation conducted by external evaluator(s). Can be commissioned by management of the project, implementing or donor organization.

2. Introduce the task. Each table will have a topic to prepare for a debate:

1 Advantages of external evaluation

2 Disadvantages of external evaluation

3 Advantages of internal evaluation commissioned by agency management

4 Disadvantages of external evaluation commissioned by agency management

5 Advantages of small-scale internal evaluation commissioned by project management

6 Disadvantages of small-scale internal evaluation commissioned by project management

3. Invite representative of all tables to draw cards to get a topic to work on.4. Give teams 20 minutes to prepare.5. Prepare stage for debate, e.g. by putting several chairs in front of the room.6. In the debate give 20 minutes per topic. “Rival” team shall present one point at a time. As the

debate goes, record key point on a flipchart.7. After the debate, review key points that have emerged in the debate. Then invite participants to

share their comments and insights from the exercise.

Session 17Activity 17.1: Recap from Day 4Duration: 15 minutes


answered in Day 4.

Materials: Presentation Flipchart Markers White tape

Process:4. Make a short presentation covering key points from Day 4.5. Ask people if they have any questions regarding material covered on Day 4.

6. Go to “Pending questions” flipchart. Suggest the questions that might have already been answered. If participants agree, move answered question to the flipchart titled “Answered questions”. Invite participant to add questions to the “Pending questions” if necessary using sticky notes.

Diversity of evaluation approaches.Learning objectives:As a result of this session participants shall be informed about the diversity of evaluation approaches and related terminology.

Activity 17.2: LectureDuration: 75 minutes




And terminology in the field was evolving along the way.14. Ask participants to tell types of project evaluation they have heard of. Record the on the

flipchart.

Key points for presentation (see also Annex 10):


Evaluations done at different points during project lifetime:o ex ante and ex post evaluation, o midterm and final evaluation, o needs assessment




Theoretical approach to evaluation:o Utilization-focused evaluationo Theory-based evaluation



Session 18Evaluator competencesLearning objectives:As a result of this session participants shall understand what competencies evaluators shall have and how they can build these competencies.

Activity 18.1: Self-assessment of evaluation competenciesDuration: 45 minutes

Objectives: Stimulate participants’ self-reflection about evaluation competencies that they have and need.

Materials: Copies of the AEA Evaluator Competencies (Annex 11) Flipchart Markers Presentation

Process:1. Explain the concept of competency.2. Tell participants to read through the AEA Evaluator Competencies and mark competencies that

they already have (1) and ones that they are missing (0).3. Ask people to calculate their scores for separate domains and the total score.4. Ask people how they felt when reviewing the competencies list. What insights did they have?5. Then go separately over each of competency domains (use presentation with slides listing

competencies): Professional practice Methodology Context Planning and management Interpersonal

Ask participants to think critically about the list of competencies under each domain. Would the agree with the list? Would they add something to the list?

Activity 18.2: Paths for professional development Duration: 45 minutes

Objectives: Stimulate participants’ self-reflection about evaluation competencies that they have and need

and how they can get them

Materials: Copies of the AEA Evaluator Competencies from previous activity Flipchart Markers Presentation

Process:1. Ask people to have group discussions about how a person like them can develop necessary

evaluator competencies.2. Give groups 20 minutes for a discussion.3. Invite groups to share results of their discussion going over five domains of competencies one by

one. Record ideas on flipchart.4. Conclude by general discussion about what people have learned about competencies.

Session 19Evaluation standards and ethicsLearning objectives:As a result of this session participants shall understand the concept of evaluation standards and be aware of the internationally accepted evaluation standards and norms.

Activity 19.1: Develop your own standards Duration: 60 minutes

Objectives: Stimulate participants’ reflection about origin of evaluation standards


Process:1. Start with explaining that standards are a result of profession self-regulation – professionals

agree how the profession should be practiced. 2. Explain that standard is a rule or principle that should be adhered to.3. Now groups have to develop an evaluation standard of their own – based on their experience

before and during this workshop.4. Give people 20-30 minutes to complete the task.5. Then each group shall present their own standard in plenary.6. When groups finish their presentations let them know how their standards compare with UNEG

standards.

7. Ask participants how they felt about this exercise. What insights did they have in the course of working on their own standards? What have they learned?

Activity 19.2: Review of UNEG standard on methodology

Duration: 30 minutes

Objectives: Demonstrate that evaluation process that was presented to participants in the course of the

workshop is coherent with UNEG standard

Materials: Copies with UNEG Standard 4.5 Methodology Flipchart Markers Presentation

Process:1. Show participants the slide with the text of UNEG Standard 4.5 Methodology.

UNEG Standard 4.5 Methodology

Evaluation methodologies must be sufficiently rigorous such that the evaluation responds to the scope and objectives, is designed to answer evaluation questions and leads to a complete, fair and unbiased assessment.

63. Methodologies should be chosen with a clear intent to provide credible answers to the evaluation questions. The methodology should ensure that the information collected is valid, reliable and sufficient to meet the evaluation objectives and that the analysis is logically coherent and complete (and not speculative or opinion-based). Triangulation principles (utilizing multiple sources of data and methods) should be applied in order to validate findings.

64. Methodologies provide what information should be collected, from which source(s) it should be collected, for what purpose it should be collected and how the collected data will be analyzed in order to answer the evaluation questions. The methodology should not be confused with the data collection strategy. The methodology must also indicate, in analyzing data, what benchmarks will be used in making the assessment for each evaluation criteria or question.

2. Show the next slide with Evaluation matrix.3. Ask groups to have a discussion if designing evaluation using the Evaluation matrix will ensure

compliance with UNEG Standard 4.5 Methodology. Then each group will have to present and explain its conclusion. If participants decide, that using Evaluation matrix does ensure compliance with UNEG Standard 4.5 Methodology, they have to suggest measures to ensure this compliance.

4. Give participants 15 minutes to complete the task.5. Invite groups to present their conclusion and recommendations if any.6. Then ask participants how they felt about this exercise. What are their key insights and learning

points?

Session 20ClosingActivity 20.1: Q&ADuration: 45 minutes

Objectives: Clarify remaining questions.

Materials: Flipcharts “Pending questions” and “Answered questions”



2. Provide answers to questions remaining on the “Pending questions” flipchart. If participants agree that answers were satisfactory, move relevant sticky notes to the “Answered questions” flipchart. If not, suggest where they can get more information on these questions.

3. Ask people if they have other pending questions.

Activity 20.2: Final assessment Duration: 15 minutes

Objectives: Get data for training evaluation. Have participants reflects on what they have learned.






Objectives:

Close the workshop.

Materials: Tables of score from previous activity Flipchart Markers

Process:9. Invite participants to share their key insights and take-aways from the workshop. Record them






Area Score


Monitoring

Evaluation

TOTAL





The MCUDP is jointly financed by the International Bank for Reconstruction and Development (IBRD) of the World Bank group and the Government of Sri Lanka. MCUDP has three main components. The first component focuses on flood and drainage management and addresses the urgent issue of urban flooding, which regularly paralyzes the economy of the Metro Colombo Region with high socioeconomic costs. The second component focuses on urban development and infrastructure rehabilitation for Project Local Authorities (PLAs). The third component consists of implementation support.












Monitoring and Evaluation (M & E) and reporting will be carried out at two levels, at the PMU level and the SLLRDC level. The PMU has recruited a Social Specialist to undertake the M & E and reporting of social safeguards management to ensure that safeguards issues are sufficiently mitigated. The Social Officer at the SLLRDC and 02 Community Development Officers will monitor and report on safeguard implementation at the field level. This includes paying close attention to the delivery of entitlements to people who will be resettled and the Contractor‘s social safeguard obligations. The PMU will submit quarterly social safeguard progress and quality monitoring reports with yearly safeguard reviews, which will be carried out by independent consultants. These will form the basis for any improvements to be brought about in the safeguard policy framework and implementation arrangements.

Annex 4: Biscuits surveyManagement of this training program is constantly working to improve the quality of learning experiences we are providing to our participants. We would highly appreciate if you could answer this short survey to inform our efforts.

In the beginning of this workshop you were served biscuits.

Did you have some of these biscuits?

Yes No

How would you respond to the statement:

Serving biscuits in the beginning of a workshop is a nice way to welcome participants.

Strongly disagree DisagreeNeither agree, nor

disagreeAgree

Strongly agree

How long did it take you to get to this workshop from home?

Less than 30 minutes 30 to 60 minutes 1 to 2 hours Over 2 hours

Did you have breakfast before leaving?

Yes No

What is you gender?

Female Male

Annex 5: Biscuits data sheetsNutrition information

1 serving of biscuits (2 pieces/25 grams) contains:

Energy 125 kkal

Protein 1.3 g

Total fat 6 g

Saturated fat 5 g

Total carbohydrates 16 g

Sugars 7 g

Dietary fiber 0.5 g

Sodium 210 mg

Sugar and learning5

5 Sugar and the Brain. Retrieved from: http://neuro.hms.harvard.edu/harvard-mahoney-neuroscience-institute/brain-newsletter/and-brain-series/sugar-and-brain

Glucose, a form of sugar, is the primary source of energy for every cell in the body. Because the brain is so rich in nerve cells, or neurons, it is the most energy-demanding organ, using one-half of all the sugar energy in the body.

Brain functions such as thinking, memory, and learning are closely linked to glucose levels and how efficiently the brain uses this fuel source. If there isn’t enough glucose in the brain, for example, neurotransmitters, the brain’s chemical messengers, are not produced and communication between neurons breaks down. In addition, hypoglycemia, a common complication of diabetes caused by low glucose levels in the blood, can lead to loss of energy for brain function and is linked to poor attention and cognitive function.

Although the brain needs glucose, too much of this energy source can be a bad thing. A 2012 study in animals by researchers at the University of California at Los Angeles indicated a positive relationship between the consumption of fructose, another form of sugar, and the aging of cells, while a 2009 study, also using an animal model, conducted by a team of scientists at the University of Montreal and Boston College, linked excess glucose consumption to memory and cognitive deficiencies.



Gender Equity

Relevance

Effectiveness

Efficiency

Sustainability

Impact



Sub-questions

Necessary evidence

Sources of Data

Methods Assumptions

Annex 8: EvalMethods Lab Information PackSource: http://www.betterevaluation.org/evaluation-options/questionnaire

Questionnaires/ Surveys

A questionnaire is a specific set of written questions which aims to extract specific information from the chosen respondents. The questions and answers are designed in order to gather information about attitudes, preferences and factual information of respondents.

Questionnaires focus on the sampling of a smaller population statistically representative of the wider population in question. This sample, in turn, proves more manageable to study, reducing the investigator’s overall workload and costs while also making it easier to ensure homogeneity and quality within a smaller data-set. Three primary categories of surveying sampling include probability/random sampling (in which every unit of the population has a probability of being included in the sample) and non-probability/non-random sampling (in which certain elements of the population are intentionally excluded from the possibility of being part of the sample). Stratified samplking also involves the identification of sub-populations from the overall population, which are then sampled from randomly.

Once investigators have drawn their sample population, questions are administered to respondents. A number of data collection approaches can be followed to collect answers from the sample. Each approach has advantages and disadvantages, and may be more or less appropriate depending on the context, cost, coverage offered of the sample population, flexibility, accuracy and anticipated response rates. Note that approaches can be combined. Some of the most common include:

E-mail: distributing questionnaires online via email. Face to face: administering questionnaires in real time by a researcher reading the questions. Internet: collecting data via a form (with closed or open questions) on the web. Mobile: using mobile phones to distribute surveys, either by linking with an adapted internet-

based survey or through a specific survey app. Mail: posting hard copies to participants to be returned. Phone: administering questionnaires by telephone.

Questions asked in surveys are systematic – the same inquiries are asked of each and every respondent, in the same format and order - and are typically presented in the form of a questionnaire or structured interview. Questions can be either open or closed-ended (although open-ended questions are often coded into quantitative response scales during analysis). More specifically, questions can take one of several forms:

Dichotomous: respondents chose between two options for response List: respondents chose between a list of more than two items, of which any can be selected Category: respondent chooses between one of several mutually exclusive categories (such as

age ranges, i.e. 20-29, 30-39, etc). Ranking: respondents are asked to place several items into a ranked order Quantity: respondent is asked to provide an exact or approximate number Grid: respondents are presented with a table or grid in order to answer two or more questions

simultaneously Verbal: respondents are asked to provide verbal word, phrase, or comment responses

Rating scales: respondents are asked to rate items or phrases against pre-set criteria. The most common rating scale is the Likert scale which asks for the level of agreement on a symmetric agree-disagree scale for a statement.

Conducting a survey itself is a serious undertaking, one which demands careful preparation across three general stages: preparation and question design; administration of the survey, and analysis of results. Staff will need to be carefully trained and prepared for the task, and all relevant administrative and logistical preparations should be well-sorted before the survey’s start. Questionnaire design must be given particular attention – the scope, content and purpose of individual questions needs to be determined, a proper response format selected, and care given to the question’s specific wording, formatting, and ordering. If at all possible, questionnaires should always be pilot-tested before being sent out en-mass.

How to maximize response rates

One key aspect of surveying is how to maximise response rates among the population. There is a universe of specialised tips and techniques per approach, but some broad general tips to increase the likelihood of those contacted actually responding include:

Sending advance letters/calls/e-mails to respondents which presents the investigator’s research agenda and details of the project, when the respondent can expect to be contacted by call or by an in-person visit, and thanking the respondent for his or her potential time.

Construct ‘respondent-friendly’ survey questionnaires – as brief as possible (ideally single page), clearly-written, easy to complete, and non-offensive.

Survey staff should be careful trained and briefed on both the research project itself, as well as specialised training for the administration of interviews and group questionnaires, scheduling, and logistics.

Offer incentives, if appropriate – financial or non-monitory incentives given at the start or completion of a survey can help response rates, but should be carefully considered for any ethical complications this might raise.

Appealing to people emotionally – such as emphasizing the potential to make a positive impact on a social issue by taking part in a survey – can often persuade reticent or unmotivated respondents to participate. Likewise, appeals made from authority can have a similar effect, such as legal obligations to complete government census forms, or required participation in a survey as an official policy of an organization.

Surveys are major undertakings: before you start, be sure you have the resources, time, staff, expertise and time-management skills necessary, or you can easily become quickly overwhelmed.

Surveys are particularly useful for collection large amounts of data in a relatively short time, and are less expensive than many other data collection techniques. They can also be administered over long-distances by phone, computer, and mail, thus allowing the investigator to be based elsewhere if necessary. Finally, surveys are the most useful option for describing a large population’s characteristics with statistical significance.

On the other hand, surveys have several notable disadvantages. First, without careful survey construction and administrative oversight, even well-designed studies can quickly fall apart. Second, there will always be the potential for respondents to lie, suffer memory gaps, or feel obligated to answer in a certain way to impress the investigator – as such, their answers do not

always reflect reality. Third, response rates will always retain a potential for bias, because people who chose to respond can be different than those who chose not to.

In selecting the particular survey option, consider, first, whether the population of interest can be enumerated (that is, can you locate the people you wish to question through preexisting data, like phonebooks, registries, etc, or will you need to search them out yourself?), second, are there any language or technology-access issues which might make certain options inappropriate (for example, administering an online survey in an area of low computer usage), third, whether the population will be willing to cooperate with your research (for example, asking about anti-government sentiment in a repressive state), and fourth, whether your population of interest is so geographically dispersed as to make certain options (such as door-to-door in-person surveys) unrealistic.

Each data collection option has advantages and disadvantages. For several of the most common: Mail-based surveys are low cost, respondents are able to answer at their convenience, there is

no interviewer bias, and they can be far longer than other options. On the other hand, mail-based questionnaires are a poor format for asking any question that could require clarification from the respondent, and there are often significant delays before they are returned by post.

Telephone surveys often result in higher response rates (people are more willing to speak to another person directly and in the moment), but can also neglect large aspects of your population of interest in areas of low-phone usage. They are also incompatible with any questions that rely on visual prompts, such as the interpretation of pictures or graphs.

Internet surveys are low cost, offer immediate feedback, and given that respondents enter their answers directly, frees investigators from a large component of data entry. However, they are restricted to computer-literate respondents, and can be easily manipulated by single respondents who complete the survey multiple times under aliases (a problem that can be addressed to some degree through password-protection).

Survey design

Be sure to obtain any official approval needed to distribute the questionnaire. Carefully consider the timing of the survey stages, and set milestones to monitor progress. Establish a system in advance to handle paperwork and record information on responses, non-

responses, area coverage, etc. which will rapidly accumulate. Plan for follow-up surveying to be done on non-respondents, in order to maximise coverage. Consider the analysis stage early on, and how your survey design will impact the subsequent

analysis. This will help ensure you only collect data that will be used. Speak to any colleagues or peers who have experience in surveying your population of interest –

there are often numerous, idiosyncratic details of survey design which can only be learned from practical experience working with a particular population.

Link every question to a broader evaluation agenda, and ensure that there are no superfluous questions.

Consider your question’s content: can the respondents be realistically expected to know the answer? Will they be able to answer in the moment, or will they need to consult records to provide an answer?

Be sure to leave the respondent the option of responding with “I don’t know” or “Not sure” to an answer.

Use jargon-free, technical-free language, unless your population of interest is made up of experts.

Specific questions are better than vague ones – for example, “Did you walk to town this weekend” rather than “Did you walk to town?”.

The order of the questions is extremely important – ask easy questions at the beginning, and avoid more sensitive or demanding questions for later in the questionnaire.

Requisite background information which should always be included at the start of a questionnaire includes your research affiliation and purpose, the return address and date, a clear statement of the confidentiality and anonymity you are offering, as well as the voluntary nature of the survey (if it is not a legal requirement), and a note of thanks for the respondent’s time. Most importantly, you must include careful instructions for filling out the questionnaire, including exact details and, ideally, examples.

All questionnaires should have a designated number including information on the date and location administered.

Ensure the questionnaire has an attractive layout and formatting, which will aid in both response rates and in your subsequent analysis.

Leave enough time to plan, design, and pilot-test first drafts of a questionnaire for revisions, as well as enough time to print the required materials (if paper-based).

Avoid any potentially offensive phrasing. Be aware and sensitive of the local context – cultural, religious, educational level, etc.

Interviews

Interviewing is a fundamental methodology for both quantitative and qualitative social research and evaluation. Interviews are conversations between an investigator (interviewer) and a respondent (‘interviewees’, ‘informants’ or ‘sources’) in which questions are asked in order to obtain information. Interviews seek to collect data and narrative information in order to better understand the respondent’s unique perspectives, opinions, and world-views.

Types of interviews

There are many different types of interview approaches and techniques, Generally speaking, all interviews fall into one of three categories: structured, semi-structured, and depth/unstructured interviews.

Structured interviews are most typically used in quantitative investigations, including survey research. In structured interviews, the interviewer presents the interviewee with a standardized set of questions, often in questionnaire form. These questions usually have pre-set answers from which the interviewee selects, rather than ‘open-ended’ questions. Each individual interview features the same set of questions, asked in a fixed order. All questions included in the research design are asked in each interview session. Structured interview questions are the most common type used in surveying interviewing.

Semi-structured interviews center around a mixed framework of general themes and pre-established questions, which can be adapted in the context of individual sessions. The interviewers is thus free to leave certain questions out, mix the order of questions, or ask certain standard questions in different ways depending on context. Semi-structured interviews also rely on a combination of both open and closed questions.

Unstructured interviews – also known as ‘informal’ or ‘conversational’ interviews – are wholly qualitative, and include only topic areas and themes rather than standard questions. Unstructured interviews take the form of natural conversation between two or more people, and allow the interviewer to pursue follow-up questions or new lines of discussion as they see fit. Closed questions are avoided, and the interviewee is often asked to identify the information they feel is most important for the discussion.

In practice, these three approaches are routinely combined. Qualitative exploratory interviewing, for instance, can prove a good compliment to more structured interviewing using closed questions later in an evaluation.

Interviewing process

Whatever approach the investigator selects, the interviewing processes itself follows several general stages:

First, investigators design and plan the study, determining both generalized approach (structured, semi-structured, unstructured), specific technique, the research questions to be asked, and any practical, conceptual and ethical external factors to consider. Interviewers also undertake any specialized training/preparations required.

Second, the interviews themselves are conducted, and their results are subsequently transcribed. (Be warned that the transcription process can take as long, if not longer, than the interviewing itself.)

Third, the results of the interviews are analyses and interpreted using the investigator’s chosen option of analysis. During this stage, verification of the data and findings collected from the interviews is also required.

Fourth, results are reported.

Conducting the interview itself is as much ‘art’ as ‘science’, and requires practice. That said, the following steps provide a useful guide:

Opening up the interview: Begin with informal introductions and small talk, then properly introduce the investigation, interview format and structure to the participants, allowing the opportunity for them to withdraw their consent prior to its start.

Administer questions: Ask the questions of the interviewee, while recording their answers by hand, audio recorder, or video tape. Pay close attention to both the answers being recorded and, especially for semi- and unstructured interviews, the emerging themes, perspectives, opinions, and underlying logic which is being elicited in the respondent’s answers – all of these should be noted and if appropriate, probed further. Inconsistencies and diversionary answers should also be watched out for. Respondents should always be given the space during discussions to form their own answers. The investigator should also keep an eye on the time throughout.

Closing down the interview: After all questions have been asked, the investigator should ask the respondent their feelings on the interview, and whether they have anything further to add. It is particularly useful to summarise the key points of the discussion with the respondent while they are still there. End by thanking the respondent for their time. If possible, review notes immediately after and expand on any annotations made during the interview while it is still ‘fresh’ in your mind.

Ethics: Finally, it is the investigator’s duty to ensure that they adhere to proper ethical interviewing standards. First, interviews should only be conducted with informed consent – permission should be sought in a transparent way from interviewees, and granted, prior to the interview. This step should be conducted alongside risk assessments to identify potential risks to interviews and address them in the content process. Second, rights to confidentiality and anonymity must be offered where appropriate – keep in mind that many governments have strict laws about both confidentiality and circumstances where important/dangerous information cannot be kept confidential (for example, in instances of severe domestic or child abuse, etc.) Third, fair return for assistance should be considered, whether in terms of financial or material compensation. (compensation is itself a tricky ethical issue, and policies may differ by organization and context, but should be addressed and discussed prior to the start of the interview process.) Finally, be aware that interviewing vulnerable groups – including children, disabled people, elderly people, victims of violence, etc – bring with it a range of unique ethical considerations. An example of ethical guidelines for social research prepared by professional research associations to govern their professions may serve as a useful starting point.

In general, interviews have the following strengths: Depth and context of data – interviews elicit insights and perspective’s into people’s though-

processes and rationales behind a particular issue, and are able to articulate their ‘story’ in their own words.

Access – Interviews, particularly one-to-one interviews, can permit the investigator to access individuals unwilling or unable to participate in focus groups or surveys.

Interview weaknesses are: Interviews are time-consuming – in terms of identifying subjects, negotiating access, logistics,

direct interviewing time, and lengthy transcriptions. Less-structured interview data is difficult to analyze. ‘Interviewer effect’ – in which the interviewer’s presence and behavior bias the interviewee’s

response – is a consistent challenge to avoid, as are unintended transgressions of individual’s comfort-zones and levels of privacy.

People’s narrative explanations do not always conform to the reality of a situation, which requires consideration of reliability and triangulation.

Keep in mind that a translator may be required to assist in both the interview and transcription if the investigator does not speak the same language as the interviewee – keep in mind that this requires additional coordination between the translator and interviewer to ensure all questions are delivered as intended. Interviewers should also be involved in the post-interview review and analysis.

While most interviews outside group settings include only one interviewer, it can often be useful to consider including a second interviewer. Ideally, this second interviewer can also serve as the translator if needed, and can focus on note-taking while the first interviewer focuses on questions. Can also provide feedback and perspective after the interview is complete, and interviews can often benefit from two people with different perspectives asking different types of questions.

Interviewing is a unique and somewhat intuitive skill which requires practice. The following are a few, but by no means all, guidance on how to ask good questions.

Question design: Good questions should be clear, specific, unambiguous, and directly related to the overarching

research question. They should ideally concern one issue or topic at a time, and be worded in clear, jargon and technical-free language. They should ask only subjects respondents can be reasonably expected to possess information on.

Questions should be neutral in tone. Investigators must always remain aware of the danger of ‘leading questions’. Leading questions

are those which encourage the respondent to answer in a particular way, thus biasing your resulting answer and potentially discrediting your research. (wouldn’t you agree that this project should be recognized as a success?) To avoid asking leading questions – which can often be difficult to spot yourself – ask a colleague to review your questions for you.

Similarly, investigators should avoid questions which presume information (i.e. asking ‘what age did you graduate primary school’ presumes the respondent attended primary school), questions which contain subtle or explicit value judgments (i.e. asking ‘at what age were you forced to marry?’ to someone who was not forced to marry), and of course, any questions which may be considered rude, offensive, or insensitive (determining which will always depend on context, culture, and the individual).

Place questions in a logical order – for instance, beginning with an interesting and nonthreatening question which all respondents may be expected to answer confidently. (Conversely, do not open your interviewing with your most challenging or potentially embarrassing questions – rather, sensitively introduce them later in the interview after rapport has been established.)

If at all possible, pilot-test your questions on a small group from your final sample to identify potential issues, and expect to redraft them multiple times before finalizing the list.

Interviewing: Time-management and scheduling is critical. Arrive at every interview with an understanding of

the timing involved in the entire interview and each individual question. Keep a discrete eye on the time to ensure you have enough time remaining. If possible, leave extra time to begin and end (in case of late arrivals and protracted ending small-talk). If possible, remind respondents a day or two before the scheduled interview.

Pay close attention to your own body-language: maintain appropriate levels of eye contact. Also notice non-verbal signs in your respondents, and note them. Bring all relevant materials with you – typically, a notebook, writing tools, and any recording equipment (audio or video) needed. Ensure that your recording option (note-taking, taping) is workable in the context in which you will be doing the interview. Especially for long interviews get some water and maybe even food/ candies for you and your subject.

Wait to establish rapport, when the interviewee appears most comfortable, before asking sensitive questions.

Remain aware for potentially evasive, confused, or intentionally misleading/distracting response, and probe (gently) for clarification if necessary.

It is important to allow interviewees the space to express themselves the way they feel most comfortable, in order to elicit the deepest and most honest responses. Be very careful to give these interviewees enough time to respond, and learn to be comfortable with pauses and silence.

Always thank the respondent at the end of the interview.

Immediately following the interview, take time to retire to a quiet place and expand or complete any notes you made during the session – your conversation will never be fresher in your mind.

Continue to practice and refine your interviewing techniques.

Focus Groups

Focus groups is type of group interview designed to explore peoples’ attitudes. It can be used to find out what issues are of most concern for a community or group when little or no information is available. They are a very common technique but are can be poorly executed unless well planned and facilitated.

Focus groups aim to discover the key issues of concern for selected groups. Discovering these issues can help determine which of a number of options is the preferred way forward, or to determine what are the concerns that would prevent a proposal going ahead. The focus group may also be undertaken to discover preliminary issues that are of concern to a group or community, and on which to base further research or consultation.

Focus groups should deliver detailed knowledge of the issues that concern a specific demographic or community.

Use also Focus group resource file. Available at: http://pdf.usaid.gov/pdf_docs/pnadw110.pdf


But it’s not all good news. As peace returned, small and medium enterprises (SMEs) found to their chagrin that China, India, and Bangladesh had all but cornered the world’s export markets in mass manufacturing and textiles – at their expense. To add to their woes, the national government had paid little attention to the sector over the years. For the hundreds of business owners renting land in the 18 government-owned industrial estates managed by the Industrial Development Board (IDB), the economic boom seemed to have all but passed them by.



In 2013, the situation began to change. The Ministry of Traditional Industries and Small Enterprise Development commissioned a comprehensive evaluation of the sector in 2013. A team of independent

evaluators led by Ajith Ekanayaka began travelling to each of the 18 estates across the country to examine the issues plaguing the sector and to suggest direct solutions. Ekanayaka made it clear that this evaluation would not be relegated to a filing cabinet but would be put to immediate use.







Because policymakers had given frequent input through scoping sessions, and the evaluation was built on a broad, credible base of evidence and data as well as strong connections to stakeholders. According to Sivagnanasothy, a great deal has come from this single evaluation. He recalls: “When the rents went up, industrialists used to shout and cry How can we afford it? We were able to convince the central government to cap rent increases at 7% annually. That’s a predictable environment, which is an enabling environment. Not only that, but the introduction of targeted loans with lower interest and longer grace periods meant that SMEs could now expand their operations more easily. In each of the estates, factory production is up, and with it, SME profits, leading to more jobs. It’s trickled down to the rural villages”.










Final evaluation in conducted right before of after the project completion. Needs assessment is conducted before project development of assess to needs of target

beneficiaries.







AreaScore



Monitoring

Evaluation

TOTAL








1.b Create sound policy frameworks at the national, regional and international levels, based on pro-poor and gender sensitive development strategies, to support accelerated investment in poverty eradication actions





2.4 By 2030, ensure sustainable food production systems and implement resilient agricultural practices that increase productivity and production, that help maintain ecosystems, that strengthen capacity for adaptation to climate change, extreme weather, drought, flooding and other disasters and that progressively improve land and soil quality






11.5 By 2030, significantly reduce the number of deaths and the number of people affected and substantially decrease the direct economic losses relative to global gross domestic product caused by disasters, including water-related disasters, with a focus on protecting the poor and people in vulnerable situations

















14.6 By 2020, prohibit certain forms of fisheries subsidies which contribute to overcapacity and overfishing, eliminate subsidies that contribute to illegal, unreported and unregulated fishing and refrain from introducing new such subsidies, recognizing that appropriate and effective special and differential treatment for developing and least developed countries should be an integral part of the World Trade Organization fisheries subsidies negotiation2













Finance






Technology




Capacity-building


Trade




Systemic issues









