module: introduction topic: getting started welcome€¦ · module: introduction topic: getting...

Monitoring & Evaluating a Value Chain Project USAID Office of Microenterprise Development

1

Module: Introduction Topic: Getting Started

Welcome

Video: Welcome to "Monitoring and Evaluating a Value Chain Project." This is the last of four courses in The Value Chain Approach on‐line series developed by USAID's Microenterprise Development office. In this course, you will learn how to apply Monitoring and Evaluation concepts and practices to value chain programming. Evaluation of USAID projects and activities should always follow the USAID Evaluation Policy in addition to the basic principles and guidelines laid out in this module.

Course Objectives Audio: Specifically, with USAID staff and our development partners in mind, this course is designed to:

Highlight the iterative nature of monitoring and evaluation throughout the value chain project cycle

Review basic ME concepts as applied to value chain projects and activities , and

Identify specific implications and potential solutions when monitoring and evaluating value chain programming

In this course, we assume that you already have a basic understanding of ME concepts. If you do not, or would like a refresher, you may want to take some time before starting, to review one or more of the courses or online resources listed in the Resources link in the lower left side of the navigation pane.

Throughout this course, we will provide you with links to explanations of key concepts. We encourage you to click these links if a particular concept is unclear or if you would like to learn more.

Bullets: Bullet 1:

This course is designed to:

Bullet 2:

Highlight the iterative nature of monitoring and evaluation throughout the value chain project cycle.

Bullet 3:

Review basic ME concepts as applied to value chain projects and activities. Bullet 4:

Identify specific implications and potential solutions when monitoring and evaluating value chain programming.


2

Course Description

Audio: Before you begin, let's discuss what you can expect from this course. The course is structured as a series of modules, which can be completed either in a guided, linear fashion or in a self‐guided manner. For those of you who are familiar with the content and wish to skip ahead, the Table of Contents can help you navigate to particular topics of interest. Use the "+" to open the sections, and just click on the title to get there.

The online setting provides on‐demand support. Click Help for a brief tutorial on course functionality and the learning interface. The Resources link in the lower left of the course menu provides access to useful reference materials and web sites. A Glossary is available for checking terms and references highlighted in the course.

Throughout the course, you will be presented with periodic Knowledge Check and Think About It Questions. Both types of questions are designed as an interactive "self‐ check" to ensure you understand the most critical concepts. There is also a quiz at the end of the course to check your understanding and as a final review of what you have learned. Your score on this, too, is for your reference only.

The course will keep track of your scores on the Knowledge Check questions throughout, for your reference only. You can check your score at any time in the My Score panel at the top right of the screen.

This course will take you approximately an hour to complete. If you need to leave the course at any point, your progress will be retained and you can return to complete it later. Now, take a moment to familiarize yourself with our course features and functions. When you are ready, click Next to continue.

Images and Captions: Image 1.1 Description: (Screen shot highlighting navigational menu at left of main course screen, showing modular structure)


3

Caption 1:

This course features:

A Series of Modules

Image 2.1 Description: (Screen shot highlighting selectable Help button at top/right of course window)


4

Caption 2:


A Series of Modules

On‐demand Support

Image 3.1 Description:

(Screen shot of self‐check quiz item)

Caption 3:


A Series of Modules

On‐demand Course Support

Opportunities to Reinforce and Test Your Knowledge


(Image highlighting "My Score" tracking button)


5

Caption 4:


A Series of Modules



Score Tracking

Image 5.1 Description: (Image reflecting "24/7" course access)


6

Caption 5:


A Series of Modules



Score Tracking

Flexible Timing

Topic: M&E – The Project Cycle & Learning Frameworks

USAID's Program Cycle

Audio: USAID's Agency‐wide program cycle aims to determine strategic priorities, opportunities and tradeoffs; evaluate strategies and projects; and feed knowledge back into strategic planning, programming and future policy development. For more information about USAID's program cycle, as well as implications for program monitoring and evaluation, go to USAID's internal online resource, ProgramNet, or the USAID Learning Lab, a USAID community resource found at http://usaidlearninglab.org/.

Images and Captions: Image 1.1 Description: (USAID Program Cycle graphic)


7

Caption 1:

For more information about USAID's program cycle, as well as implications for program monitoring and evaluation, go to USAID's internal online resource, ProgramNet [programnet.usaid.gov] or the USAID Learning Lab, a USAID community resource found at http://usaidlearninglab.org/ .

Value Chain Project Cycle and ME Audio: As outlined in further detail in the Learning Value Chain Basics course, the value chain project cycle can be seen as a focused and technically specialized view of the program cycle, through which the value chain approach and key elements and principles are consistently applied throughout the design, implementation, monitoring and evaluation of the project or activity. Moreover, although the figure on this screen shows monitoring and evaluation as the last step, in practice, it is a continual process that begins early in the project design and continues through to the conclusion of the project. For value chain projects, ME involves the systematic collection, analysis, and use of information about the project, its performance, and changes in the market system in which the project operates.

Images and Captions: Image 1.1 Description: (Image of five‐step Value Chain Project Cycle flowchart ‐‐ Value Chain Selection; Value Chain Analysis; Competitiveness Strategy; Design & Implementation; Monitoring & Evaluation. M&E step is highlighted)


8

Caption 1:

The value chain project cycle can be seen as a focused and technically specialized view of the program cycle, through which the value chain approach, key elements and principles are consistently applied throughout the design, implementation, monitoring and evaluation of the project or activity.

A Learning Framework Video: (Animation showing flow of process from Theory of Change to Causal Model, then to Learning Process, with feedback then returning to both Theory of Change and Causal Model.)

The ME process should be couched within a larger learning framework. Generally this framework consists of an overarching theory of change which is then further articulated through a causal model and verified or corrected through a learning process that occurs during implementation.

The theory of change posits the underlying causes of a development problem and provides an evidence‐based overview of the systemic change required to achieve the desired results. The causal model then lays out our best guess (again, based on the evidence that we have) as to what that change process will look like to achieve the desired outcome.

The learning process enables us to track the progress of that change in real time so that we can adapt the causal model and/or theory of change as needed. This is how our M&E will feed into a practice of collaborating, learning, and adapting, or CLA. This learning framework should exist simultaneously at multiple levels within USAID's programming – at a higher, strategic level in the CDCS; at the Development Objective level in our project designs; and at the activity level in the structure of individual


9

mechanisms. At all levels, this learning framework helps us to deploy ME processes and tools more effectively.

The Strategic Vision: Theory of Change

Video: (Animation graphically depicting the elements described in audio)

Let's take a closer look at the learning framework as applied to a value chain case study. First we need to start with the theory of change.

In this simplified example, the desired result is inclusive agricultural growth and improved food security. Current evidence supports that improving the incomes of rural households is a way to achieve that result since lack of sufficient household income is a key underlying cause of food insecurity and stagnate growth. The hypothesis posits that increasing the productivity and competitiveness of the livestock value chain is a systemic change required to improve rural household income.

Bear in mind that this is the overarching theory of change, and in reality the progression of results will play out in a messier way. The monitoring and evaluation plan will track expected results and provide further evidence during implementation to help us re‐evaluate this theory of change.

Activity Design: the Causal Model Video: (Animation graphically depicting the elements described in audio)

Now let's think about the activity level. At this level we often use a causal model as a tool to articulate the path, or paths, to results within the theory of change. This causal model was developed as part of a design to address increasing productivity and competitiveness of the livestock value chain. The causal model adds more detail as to how the theory of change results will be achieved and the assumptions that are made. While inputs and outputs are under the direct control (or manageable interest) of the project implementer and USAID, outcomes, impact, and goal depend on a range of other factors – choices made by independent decision makers (such as farmers and product buyers) as well as other forces outside the implementer's or USAID's control (such as the impact of weather or changes in the market).

Even though implementation is rarely linear or simple, it is important to make sure the logic is clearly defined at the outset. A clear causal model helps to ensure that activities are consistent with the overarching theory of change. It also serves as a reference point against which implementation can be measured.

The Performance Management Plan will build upon and link to this causal model, adding indicators that will enable the activity managers to verify the validity of the causal model. Keep in mind that the causal model needs to be continually updated. Monitoring activities will inform the model, and the learning process about the results of the inputs is likely to lead to adjustments.


10

See the Value Chain project design course for more information on how to develop an activity design.

Knowledge Check: Causal Model Audio: Now it's your turn to construct a causal model. Drag each item to the appropriate column and row. When you are ready, click Submit.

Quiz: Question: Drag each item to the appropriate column and row. Note that the Activity/Outcome pairing is important and that the Activities should be placed in numerical order. Image 1 Description:

Eight elements are listed; each should be matched to where it best fits in one of four categories. The Categories are Activities (should include 3 elements), Outcomes (3 elements), Impact (1 element), and Goal (1 element). The eight elements listed are:

1. Development of GAP training modules facilitated 2. Utilization of irrigation increased 3. Economic growth that reduces poverty 4. Growth and increased competitiveness in horticulture 5. Number of GAP certified smallholders increased 6. Year‐round supply of higher quality products to lead firms provided 7. Lead firm linkages to smallholder producers promoted 8. Market for small‐scale irrigation developed


11

Correct Result:

Item: Activity 1 Item Match: Development of GAP training modules facilitated

Item: Outcome 3 Item Match: Utilization of irrigation increased

Item: Goal 1 Item Match: Economic growth that reduces poverty

Item: Impact 1 Item Match: Growth and increased competitiveness in horticulture

Item: Outcome 1 Item Match: Number of GAP certified smallholders increased

Item: Outcome 2 Item Match: Year‐round supply of higher quality products to lead firms provided

Item: Activity 2 Item Match: Lead firm linkages to smallholder producers promoted

Item: Activity 3 Item Match: Market for small‐scale irrigation developed

Knowledge Check: Learning Process Audio: Select the best answer or answers, and then click Submit.

Quiz: Question: What activities constitute a learning process? Select all that apply. Answer A: Adapting Answer B: Monitoring Answer C: Collaborating Answer D: Evaluating Correct Result:

[All answers are correct] Yes! All of these activities, when combined, constitute a learning process.


12

Module: Basic M&E Principles and Approaches Topic: Purpose and Types of M&E

Why ME? Audio: Now that we understand how ME fits into the program and project cycles, including during design, let's discuss some basic ME principles and approaches.

Let's start with the basic purpose of Monitoring and Evaluation. In short, we invest in monitoring and evaluation for two key reasons: The first is Accountability, or the ability to ensure that a particular investment is achieving intended results. The second purpose of ME is Learning, the means of understanding how and why (or why not) these results are being achieved.

Both accountability and learning are achieved through ME processes that facilitate proving that results are happening and explaining why they are happening, as well as using ME findings to improve both current programs and future program designs.

Bullets: Bullet 1:

There are two purposes for ME: Bullet 2:

Accountability– to ensure that an investment is achieving intended results

Bullet 3:

Learning– to understand how and why these results are being achieved

Images: Image 1 Description:


13

Image 2 Description:


14

How is ME Accomplished? Audio: Now we can consider the types of ME. We learned that the fundamental why of ME is to reach the objectives of accountability and learning. Let's briefly address the basic question as to how USAID achieves these overlapping objectives. USAID uses three primary types of ME that serve different but complementary purposes: Monitoring tracks the progress of anticipated results as compared to the results that are planned.

Performance evaluation gauges how well a project or activity is achieving the planned results, given the management of the project. And Impact evaluation serves to prove causal relationships between the inputs or interventions of a project or activity, and the outcomes, impacts, or goal achieved.

If you would like review each of the three types of ME‐‐monitoring, performance evaluation, and impact evaluation‐in greater detail, click on each type. If you are already comfortable with these three types, please click Next to move forward.

Bullets: Bullet 1:

USAID uses three primary types of ME:

Image 1 Description: (Clickable bullet text segments are shown on screen. Clicking on a bullet reveals additional text and audio information on that subject) Bullet 2:

Monitoring: tracks the progress of anticipated vs. planned results

Add'l Text for Bullet 2: Continuous learning through monitoring reveals whether desired results are occurring and whether implementation is on track.

Bullet Audio for Bullet 2: Monitoring involves continuous measurement of progress on indicators during the lifetime of the project. Monitoring focuses on outputs and early outcomes. It is often managed and reported by the project implementer. The implementer generally collects and reports on monitoring data, and USAID and the implementer together use this information to monitor progress. This is an important part of accountability; however, monitoring alone is not sufficient.

Continuous learning occurs when monitoring information is regularly integrated into project management. Projects should have a clear vision and a path to achieve that vision at the outset.

Moreover, it is essential to adapt project designs along the way‐‐in response both to changes in markets and to unexpected behaviors of value chain actors in reaction to both opportunities and threats in the market system. Effective monitoring and use of the results is critical to this process in that it shows you not only what is happening but what is not happening.


15

Image 2 Description:

Bullet 3:

Performance evaluation: gauges achievement of the planned results, given the management of the project

Add'l Text for Bullet 3:

Performance evaluations include questions such as:

What has the activity achieved?

Was the activity well‐managed to enable results?

Did it use resources well and apply best practices?

Bullet Audio for Bullet 3: Performance evaluation looks at key outcomes during the life of the project (for short‐ and intermediate‐term outcomes) or after the project has ended (for long‐term outcomes).

Performance evaluation may be conducted by the project implementer or more commonly by an independent evaluator. Performance evaluation refers to an evaluation of implementation, as measured against the activity's scope of work, deliverables, and so forth. These evaluations can produce useful findings on activity operations, and if designed and timed appropriately, can be used to guide management decision‐making, improving results.


16

Performance evaluation tends to emphasize project activities and the organizational structures, policies, systems, and day‐to‐day operations that drive results. It assesses whether the activity was implemented as intended and adapted as indicated, the resources were used well, and if best practices were applied, including the procedures undertaken, the decisions made, and the services delivered. By documenting the activity's development and operation, the evaluation uncovers reasons for successful or unsuccessful performance, and provides information for potential replication.

These evaluations are useful both for accountability in terms verifying good usage of USAID resources as well as learning if results are not being achieved so that approaches can be adapted. Image 3 Description:

Bullet 4:

Impact evaluation: serves to prove causal relationships between the inputs and outcomes

Add'l Text for Bullet 4: The purpose of impact evaluations is to prove the impact of project activities on target beneficiaries and on the enabling environment.

Bullet Audio for Bullet 4: Impact evaluation looks at intermediate‐ and long‐term outcomes, but with the distinction that it attributes change to the project. It does this by comparing the observed outcomes to a counterfactual that represents what would have happened in the absence of the project.


17

The counterfactual is needed to justify claims that the observed outcomes were caused by (or attributable to) the project, which supports both accountability and learning. A common way of establishing a counterfactual is by comparing a group of people who participated in the program (called a treatment group) with a group of people who did not participate in the program (called a control group).

Impact evaluation helps to verify or repudiate the causal linkages within a causal model or results framework. Having independent evaluators conduct impact evaluations is not only good practice in order to reduce bias, it is required by USAID policy. Image 4 Description:

The Role of Learning Audio: Often the accountability purpose of ME gets more attention due to USAID's requirements to report back to Congress on the usage of funds. However, there is a renewed emphasis on the importance of learning at USAID. We have already discussed the value of couching ME in a larger learning framework, but let's look at learning more specifically against the types of ME.

Monitoring allows for ongoing and continuous learning which can be acted upon in real time through adaptive management techniques. If output targets aren't being achieved, the project may need to adjust its approach, or determine if those outputs are really the best indicators.


18

Performance evaluations and other performance management tools should also reflect learning. Work plans and PMPs should adjust to reflect what is learned during implementation. Performance evaluations at the end of a project should document lessons learned to be incorporated into future projects.

Impact evaluations should be driven by a clear learning agenda at all levels‐‐strategy, development objective, project, and activity. The results of impact evaluations should be fed forward into the overall program cycle as part of the growing evidence base that informs our theories of change. It is critical that the value of learning, particularly for value chain approaches and techniques, is shared both within USAID and throughout our partner community to ensure that our evolving understanding of what works or doesn't work is incorporated into current implementation as well as the next generation of activities.

The third course in this series, Managing a Value Chain Project, talks more about incorporating ME into a project's knowledge management system.

Images and Captions: Image 1.1 Description:

Caption 1:

There is a renewed emphasis on the importance of learning at USAID. Image 2.1 Description:


19

Caption 2:

Monitoring– ongoing, continuous and then acted upon through adaptive management techniques.


module: introduction topic: getting started welcome€¦ · module: introduction topic: getting...

Documents