tools for innovation programming · usaid’s legacy of developing and implementing innovative...

S T E P 6 : E VA L U AT E P R O G R A M 1

TOOLS FORINNOVATIONPROGRAMMING

T O O L K I T SL I F E C Y C L E S T E P S

Step 6: Evaluate Program

U S A I D TO O L S F O R I N N O VAT I O N P R O G R A M M I N G2

About the USAID U.S. Global Development LabUSAID’s legacy of developing and implementing innovative breakthroughs—from the seeds of the green revolution, to microfinance and oral rehydration therapy—has saved lives, created economic opportunity, and advanced human development. For the first time in history, we have the scientific and technological tools to put an end to extreme poverty and its most devastating consequences within the next two decades.

Building on the belief that science, technology, innovation and partnership can accelerate development impact faster, cheaper, and more sustainably, USAID established the U.S. Global Development Lab (The Lab) in April 2014. The Lab is designed to experiment and test new ideas, models, interventions, and approaches and to accelerate the ones that work across the Agency and in Missions around the world.

The Lab’s mission is twofold: • To produce breakthrough development innovations by sourcing, testing, and scaling proven solutions to reach

hundreds of millions of people. • To accelerate the transformation of the development enterprise by opening development to people

everywhere with good ideas, promoting new and deepening existing partnerships, bringing data and evidence to bear, and harnessing scientific and technological advances.

To learn more about The Lab, visit: www.usaid.gov/GlobalDevLab

ACKNOWLEDGEMENTS

The USAID Tools for Innovation Programming were written and developed by Rebecca Askin, of Development Alternatives Inc. (DAI). The tools offer insights, processes, and guidance based on USAID’s experiences designing and implementing open innovation programs in The Lab and technical bureaus in USAID/Washington.

This collection of tools was informed by in-depth interviews and discussions with the champions and the managers of the following USAID programs: Securing Water for Food: A Grand Challenge for Development; All Children Reading: A Grand Challenge for Development; Saving Lives at Birth: A Grand Challenge for Development; Powering Agriculture: A Grand Challenge for Development; Higher Education Solutions Network (HESN); and Development Innovation Ventures (DIV).

Recognition and thanks the following individuals in The Lab who gave ideas, contributions, and comments during the drafting process: Dave Ferguson, Lanakila McMahan, Seema Patel, Lynlee Tanner Stapleton, Maurice Kent, Jarah Meador, Grace Hoerner, Chad Dear, Wes Day, Tony Bloome, Karen Clune, Marissa Leffler, Wendy Taylor, Ticora Jones, Michelle Jones, Jeffrey Haeni, Jeremy Foster, Jami Rodger, Sara Cardelle, Matt Corso, and Alexis Bonnell.

Special thanks to DAI’s Meredith Perry, primary author of the Prize Toolkit and Platform Toolkit; Nathan Wyeth, primary author of the Acceleration Toolkit; and Lorin Kavanaugh-Ulku, primary author of the Communications Toolkit.

Thanks also to Kristi Ragan, Nora Brown, Nick Brown, Lauren Yang, Carol Chanick, Pierce McManus, Bryan Gerhart, Tamara Stanton, and Ilirjana Dana-Tahmazi of the DAI team, all of whom read versions of these tools and made helpful contributions to their development.


Innovation is an imprecise term, definitions differ and expectations about whether innovation as a process or a thing can do vary. For USAID’s U.S. Global Development Lab (The Lab), innovation is defined as a means to modernize U.S. development in ways that will lead to impact by:

• Increasing the number of innovations that address development challenges by attracting new innovators and reaching out to local innovators.

• Facilitating growth and sustainability of promising innovations by building evidence of efficacy, enhancing business skills, and connecting opportunities for follow-on funding and adoption.

• Increasing the impact of innovations by identifying and expediting a few proven high potential innovations to sustainably scale and improve the lives of hundreds of millions of people.

USAID’S THEORY OF CHANGE

Innovation is built into the Lab’s theory of change, specifically, that by utilizing science, technology, innovation, and partnerships in international development approaches and priorities, development impact can be achieved faster, cheaper and more sustainably. Because innovation is central to this theory of change, both as a process and as a product, it is important for USAID to measure innovation programming to:

• Test and validate the theory of change that using innovations will have a better impact;

• Understand which innovations have the potential for scale to achieve the intended impact; and

• Make informed decisions on which ones to move through the pipeline and allocate the required time and resources.

L I F E C Y C L ES T E P

K E Y L E A R N I N G S

Start your M&E planning at the design phase of your program. Not

only does M&E help you define what success looks like, but trying to retrofit

frameworks and indicators is time-consuming for you, your partners, and

your innovators.

Measuring innovative programming entails a twofold approach: 1)

Measuring the innovation process – How the innovation progresses through stages will inform decision making on whether a program should continue support to help an innovation reach

scale or to stop support altogether. 2) Measuring the impact of the innovation

– How the innovation’s progress is achieving the desired impact, incrementally or substantially.

Keep the number of indicators manageable and meaningful. Teams should identify indicators that are essential for monitoring innovators’ performance and progress without

overburdening innovators or the USAID program team.

Step 6: Evaluate Program


Measuring InnovationMeasuring innovation is inherently difficult because much of what matters about it is intangible, such as learning about what works and what doesn’t, and the conditions or activities that enable or hinder it. As such, it is important to include in the M&E Plan, metrics for both tangible and intangible outcomes. Metrics for the intangibles may have to be contextual, for example: proxy indicators, anecdotal evidence, story-telling about the innovation; but they must be included because measuring what is easy instead of what is meaningful will not yield data or analysis about the theory of change. Both the objective and the intended results must be properly defined in order to develop meaningful performance measures. It is best to pick a few metrics that capture specific actions and behaviors, not generalized ones. When it comes to measuring innovation, it is critical to remember that not all that is measured counts and not everything that counts can be measured.

A list of definitions for key monitoring and evaluation terms is provided in the Resources section.

R E S O U R C E S &

R E F E R E N C E S

THE VALUE OF M&E FOR DECISION-MAKING & LEARNING

The monitoring and evaluation (M&E) process is the foundation for informed management decisions about whether, when, and how to support an innovation to achieve impact and scale; and M&E systems enable program teams to track and capture data that can inform program design, management and implementation. M&E frameworks and systems also support learning and objectives that seek to track whether an innovation progresses between stages, understand how it progresses, and capture and use lessons learned in the process.

M&E is essential to experimental innovation programming because it allows USAID to assess a program’s impact on three levels:

• Innovation Level: You are building evidence to show which innovations in a program portfolio are achieving success in moving through stages and achieving development impact that is relevant to beneficiaries’ needs as defined by your problem statement.

• Program Level: You are testing and validating the assumptions that the program model as a whole is effectively sourcing and supporting innovations to achieve the intended impact.

• Model Level: You have gathered the evidence and are determining if your theory of change is sound, specifically that innovation programming can achieve development impact faster, cheaper and more sustainably than the traditional development approach. Based on this determination, you make a decision about whether and when to integrate these approaches into USAID’s business processes.


TWO-FOLD MEASUREMENT APPROACH: PROCESS & IMPACT

The overall success of the innovation programming is captured through a two-fold approach:

1. Measuring the innovation process – How the innovation progresses through stages will inform decision making on whether a program should continue support to help an innovation reach scale or to stop support altogether.

2. Measuring the impact of the innovation – How the innovation’s progress is achieving the desired impact, incrementally or substantially.

The correlation between process and impact is critical to innovation programming. An innovation is an input that evolves over time and this directly effects whether the intended outcomes and impact are achieved. As an input in progress, assumptions about an innovation’s viability and reliability need to be tested iteratively in order to determine if it creates impact for beneficiaries. Testing this assumption requires a developmental evaluation of innovations as they evolve from concept and ideas to concrete products or approaches. This relationship is presented in Figure 1: Correlation of Process & Impact.

F I G U R E 1 : C O R R E L AT I O N O F P R O C E S S & I M PA C T

Impa

ct M

easu

rem

ent

Process Measurement

INPUT

OUTPUT

OUTCOME

IMPACT

Innovation’s progression along its pathway


HOW MEASURING INNOVATION PROGRAMMING DIFFERS FROM TRADITIONAL PROGRAMMING

Measuring the process of an innovation’s progress and measuring an innovation’s impact both involve unpredictability, failures and successes, and pivots. As such, you need a measurement approach that allows you to anticipate the unknowns by putting in place the right monitoring and evaluation (M&E) system. If you are willing to measure in more frequent intervals, you can mitigate the risks and unknowns of innovation and do this in a timely fashion at relatively low cost. This is best done using a “developmental evaluation” approach.

Figure 2: Innovation versus Traditional Framework shows the linear approach of a traditional Logical Framework to achieve results, whereas the innovation framework shows the complexity of simultaneously measuring process of an evolving innovation and its impact. In traditional programming, the input, or activity, is fixed, whereas in innovation programming the input is in progress and has not yet been proven. Innovation M&E makes an effort to establish a learning framework that emphasizes tracking and analyzing a process that is inherently iterative and adaptive in a continually changing context.

F I G U R E 2 : I N N O VAT I O N V E R S U S T R A D I T I O N A L F R A M E W O R K

I N N O VAT I O N M & E F R A M E W O R K T R A D I T I O N A L M & E F R A M E W O R K

Impa

ct M

easu

rem

ent

Process Measurement

INPUT

OUTPUT

OUTCOME

IMPACT

Innovation’s progression along its pathway

If the OUTCOMES occur; Then this should contribute to the overall GOAL

Project Description, Indicators, Means of Verification, Assumptions

If the OUTPUTS are produced;Then the OUTCOMES can occur.

If the ACTIVITIES are conducted;Then OUTPUTS can be produced.

If adequate RESOURCES / INPUTS are provided; Then the ACTIVITIES can be conducted.

GOALS

OUTCOME(S)

OUTPUTS

ACTIVITIES

Stages of InnovationUSAID’s innovation programs use varying definitions for the stages of innovation. For example, the Lab is currently using five stages of innovation to define the innovation’s progression along its pathway (see Figure 3: The Lab’s Stages of Innovation). However, as long as the program teams consistently measure the process against its set stages, then there is the potential to yield a continuous flow of information that should be used to track success and failure, to update initial assumptions about the innovation’s implementation plan, and to make choices about how a program team allocates additional time and resources for acceleration support.


F I G U R E 3 : T H E L A B ’ S S TA G E S O F I N N O VAT I O N

A SNAPHOT: APPLYING AN INNOVATION M&E FRAMEWORK

What follows is a practical example of how the Grand Challenges for Development (GCD) have applied an innovation M&E framework to their implementation. The GCD programs have defined a basic goal that represents the theory of change: to source, incubate, and accelerate high-potential solutions that overcome critical barriers to development through science, technology and engineering. The Grand Challenges for Development Monitoring and Evaluation (M&E) Operational Plan was created by GCD champions in the Lab to provide a framework and detailed guidance for current and new program teams (presented in Figure 4: GCD Results Framework).

PIP

EL

INE

STA

GE

STA

GE

DE

SCR

IPT

ION

Stage 1: Development

Stage 2: Initial Piloting

Stage 3: Early Adoption

Stage 4: Transitionto Scale

Stage 5: Global Adoption

Early ideation and prototyping to test technical efficacy of a product or approach

Small-scale field testing to validate viability/effectiveness in real-world setting

Implementation beyond the initial pilot of impact and build pathways to scale

Further support to high-potential innovatins, leading to adoption in new markets

Adoption of most promising innovations in multiple countries and/or regions

10% of Pipeline 1% of Pipeline

F I G U R E 4 : G R A N D C H A L L E N G E S F O R D E V E L O P M E N T R E S U LT S F R A M E W O R K

Source, incubate and accelerate high-potential solutions that overcome critical barriers to development through science, technology and engineering.

SO I : DEFINE

Data and evidence used to define the specific barriers

to development

SO 2: MOBILIZE

Partners and solvers mobilized

IR 2.1Partnerships built to

advance program

IR 3.1Solver capacity to scale

up improved

IR 2.2Innovations sourced from

diverse solvers

IR 3.2High potential solutions

are adopted

SO 3: MOBILIZE

Viable science and technology innovations

scaled

SO 4: IMPLEMENT

Solvers help to overcome critical barriers to

development


The Operational Plan also presents the 12 performance measures that were designed to evaluate if and how GCD programs are sourcing and accelerating high-potential innovations and to evaluate the efficacy of the GCD model as a way to deliver development results. These performance measures emphasize:

• The direct relationship between the program’s success and having a well-defined and validated problem with meaningful performance indicators that capture the progression of innovations through the stages.

• The importance of running an effective campaign in order to reach and encourage the best and brightest solvers to apply as well as promote their successes.

• The importance of leveraging and working with key stakeholders and partners to run a successful program that helps innovations to achieve impact.

• The combination of USAID award funding and non-financial support, as well as outside investment in those innovations with the proven ability to reach people at significant scale.

At the winner selection and award negotiation stage, the program team should develop performance measures of both the process of innovations and the expected outcomes and impact throughout the stages. These should be written into award terms so that innovators are held accountable for reporting against these metrics and to ensure that the data is captured in the M&E system.

T I P

The Grand Challenges for Development Monitoring and Evaluation (M&E) Operational Plan document covers the following topics in great detail: Results Framework and Key Indicators, Roles and Responsibilities and Flow of M&E Data Collection, Indicator guidance for tailored technical indicators, M&E Data Entry Platform, Plan for Performance, and Suggested M&E Coordination. This document included in its entirety in the Resources section.

USAID’s GCD champions developed 12 performance indicators that should be used by GCD programs and can be used by other innovation program teams. The Performance Indicators Reference Sheets (PIRS) are provided in the Resources section of this document.

An overview of the DevResults User Manual is provided in the Resources section. This User Manual was created as a tool to instruct awardees on how to use the DevResults platform when reporting performance information.

R E S O U R C E S &

R E F E R E N C E S

R E S O U R C E S &

R E F E R E N C E S

The Lab’s innovation programs, including the GCDs, are using DevResults as the monitoring system and tool for data collection and analysis within and across programs. This tool will be used by program awardees and by M&E managers as a repository of data and information. Program Managers use this information to analyze and make decisions about the progression and impact of each innovation in achieving their expected results.


Resources• Definitions of Terms• Performance Indicator Reference Sheets (PIRS)• Grand Challenges for Development Monitoring and Evaluation (M&E) Operational Plan• Overview of DevResults User Manual

Resources & References



Definition of Termsanalytics: the discovery and communication of meaningful patterns in data

approach: a policy, framework, strategy, process, practice, system, or protocol

assessment: a forward-looking process that may be designed to examine country or sector context, to inform technology design, or an informal review of projects and approaches (ADS Glossary)

capacity building (capacity development): the changing, transformation, or improvement of performance at the individual, organizational, sector, or broader system level (ADS Glossary)

data quality: The character of data as it relates to credibility for decision making. High data quality meets USAID’s five

standards of data quality: validity, integrity, precision, reliability, and timeliness. (ADS 203.3.11.1)

developmental evaluation: an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments.

disruptive innovation: radically changes something– a game-changing innovation. Profound innovation at a higher level that affects organizations, companies and even entire industries and has a profound impact on product, service, process, culture or community.

evaluation: A systematic and objective assessment of an on-going or completed project, program or policy. Evaluations are undertaken to (a) improve the performance of existing interventions or policies, (b) asses their effects and impacts, and (c) inform decisions about future programming. Evaluations are formal analytical endeavors involvingsystematic collection and analysis of qualitative and quantitative information. (F Glossary of Evaluation Terms)

evidence: the factual basis for programmatic and strategic decision making in the program cycle. Evidence can be derived from assessments, analyses, performance monitoring and evaluations. It can be sourced from within USAID or externally and should result from systematic and analytic methodologies or from observations that are shared and analyzed (ADS Glossary)

high potential: “potential” refers to the higher order effects, generally medium and long-term, produced by a project, program, technology, or approach. A “high potential” technology and approach should making a significant contribution toward achieving a Bureau, Mission, or Independent Office’s development objective, the most ambitious result that a USAID operating unit along with its partners, can materially affect, and for which it is willing to be held accountable. (ADS Glossary / GCD)

Innovation aggregators: individuals or groups that source and connect solver communities

learning: a continuous process of analyzing a wide variety of information sources and knowledge, leading to iterative adaptation of behavior (ADS Glossary)

scaling: Increasing the use of a technology or approach while maintaining or improving effectiveness and affordability

solvers: someone that focuses on the problem as stated and tries to synthesize information and knowledge to achieve a solution. In the context of GCDs, solvers and innovators are synonymous, and solvers are receiving resources from GCDs.



R E S O U R C E S

technology: a tangible scientific or industrial invention (HESN/GCD)

transformative innovation: an innovation that involves more significant change and may include elements that combine to provide a more important differentiation involving product, service, process, culture or community


Performance Indicator Reference Sheets (PIRS)


R E S O U R C E S


Grand Challenges for Development Monitoring and Evaluation (M&E) Operational PlanMonitoring and evaluation (M&E) is a critical step in the Grand Challenges for Development (GCD) program lifecycle. M&E gives the GCD team and partners the ability to assess a program’s impact on three dimensions: the impact of the innovations in the program’s portfolio, the impact of the GCD program on addressing the objective(s) defined by the Call for Innovations, and the efficacy of the GCD model as a whole in achieving impacting on the development challenge. M&E is key to enabling USAID to tell a story across these levels: the awardee/innovator level, the Grand Challenge program level, and across all Grand Challenges (the “meta-level”).

This document will also provide a discussion on key M&E indicators and the measures USAID will use to collect and report on program activities at three levels/layers. In addition to these standard indicators, each GCD will determine indicators that are specific to the technical areas in which they are working. Indicators measure outputs and outcomes related to the main GCD program objectives and correspond to specific activities under each of these objectives. Several indicators were designed to evaluate if/how the GCD initiative is accelerating the sourcing and adoption of high-potential solutions in general, as well as evaluating the efficacy of the GCD model as a way to deliver development results.

The document is organized as follows:

• Results Framework and Key Indicators, describing the GCD meta-level Results Framework and Sub-Objectives (SOs) that cut across all GCDs. Within the meta-level Framework, GCD teams should have a complementary Results Framework focused on technically-oriented SOs or Intermediate Results (IRs) (for example, from the Powering Ag GCD, “increased farmer and agribusiness access to clean energy solutions to enhance their operations” is a technically oriented IR).

• Roles and Responsibilities and Flow of M&E Data Collection, describing the various M&E roles at the meta- and GCD level.

• Indicator guidance for tailored technical indicators, covering resources and tools available to GCD teams in the creation of technically-oriented program and/or innovator indicators.

• M&E Data Entry Platform, covering M&E Platform data inputs for awardees, for individual GCDs, and for meta-level, as well as baseline surveys.

• Plan for Performance

• Suggested M&E Coordination, covering how the GCD teams interact with the Lab for the reporting of M&E data across the GCD programs.

RESULTS FRAMEWORK AND KEY INDICATORS

Figure 1: GCD Results Framework identifies four SOs in which M&E activities take place. The overarching objective of the GCD is to achieve solutions to development problems through science and innovation. Within the SOs, GCD team may identify additional technical IRs that are indicative of innovator or program performance in reaching a technical


objective. These kinds of findings can inform the ways in which the GCD team refines its M&E plan and defines success within the SO. For example, the Securing Water for Food GCD identified an IR oriented toward “improved resilience and water availability due to climatic and seasonal variability” within SO 4, at the meta-level, Thus, the team will use indicators related to increases in installed water storage capacity, as well as the number of activities that directly address adaptation to modeled climate changes, within the IR.

G C D R E S U LT S F R A M E W O R K

K E Y I N D I C ATO R S R E L AT E D TO E A C H S U B - O B J E C T I V E ( S O ) & I N T E R M E D I AT E R E S U LT ( I R )

Source, incubate and accelerate high-potential solutions that overcome critical barriers to development through science, technology and engineering.

SO I : DEFINE

Data and evidence used to define the specific barriers

to development

SO 2: MOBILIZE

Partners and solvers mobilized

IR 2.1Partnerships built to

advance program

IR 3.1Solver capacity to scale

up improved

IR 2.2Innovations sourced from

diverse solvers

IR 3.2High potential solutions

are adopted

SO 3: MOBILIZE

Viable science and technology innovations

scaled

SO 4: IMPLEMENT

Solvers help to overcome critical barriers to

development

The table below presents the key indicators related to each SO and IR, as well as the level at which indicator data will be collected. To avoid double counting, data should be collected only at one level whenever possible.

INDICATOR COLLECTIONLEVEL

ANALYSISLEVEL

DATA SOURCE/COLLECTION

METHOD

WHEN COLLECTED/ANALYZED

Gin1: Return on investment

Gin2: Ratio of total value of outside (non-USAID) resources utilized, to the dollar value of USAID investments

Awardee Level

GCD level; Awardee level

GCD level, Meta-level


Individual GCD Teams will review data sources and determine based on project records such as financial, budget and project records.

Budget inputs, Gift letters, MOUs

• Inputs by Awardees Semi-annually including data verification/validation by GCD/meta M&E officer

• GCD inputs occur every 2 years and at end of GCD program; Meta-GCD inputs occur every 5 years

• All inputs analyzed semi-annually

• Inputs ongoing; Analyzed semi-annually



ANALYSISLEVEL


METHOD


Gin3: Number of viable, high potential innovations sourced

Gin4: Number of viable, high potential, science & technology approaches progressing through the innovation pipeline

Gin5: Evidence-based problem statements created for each GCD

Gin6: Number of partnerships leveraged to address the specific critical barriers of the grand challenge

Gin7: Number of outreach initiatives undertaken by each GCD

Gin8: Number of innovators reached

Gin9: Number of applicants applying to each Grand Challenge, Prize and Launch Program

Gin7: Number of outreach initiatives undertaken by each GCD

GCD level

Awardee level

GCD level

Awardee Level

Awardee level; GCD level; Meta-level

GCD level; Meta-level

GCD level

GCD level



Meta-level

Meta-level

GCD, Meta

GCD, Meta

GCD

GCD, Meta

GCD team will review number of innovations awarded funding during time period; Application Review Process

Project records - GCD Performance and Impact Evaluations, Outcome Harvesting

GCD technical expert interviews and communications (email, phone, internet); panels

Project records - Budget inputs, Gift letters, MOUs

Project records - social media analytics, web analytics, network mapping.

Project records - social media analytics, web analytics, network mapping)

GCD Application Platform Analytics

Project records - social media analytics, web analytics, network mapping.

• Inputs by awardees Semi-annually including data verification/validation by GCD/meta M&E officer

• GCD inputs occur at each call for applications


• Inputs by awardees Semi-annually including data verification/validation by GCD/meta M&E officer

• GCD inputs occur every 2 years and at end of GCD program; Meta-GCD inputs occur every 5 years; (data collected and analyzed by M&E specialist inside and outside of the agency)




• Inputs ongoing; Analyzed quarterly


• Inputs ongoing; Analyzed after new program element (Prize, Call for proposals, etc.) intake is closed


Sub-Objective 1: Data and evidence used to define specific barriers to development

Sub-Objective 2: Partners and Solvers Mobilized

IR 2.1: Partnerships Built to Advance Program

IR 2.1: Partnerships Built to Advance Program



ANALYSISLEVEL


METHOD


Gin10: Percentage of awardees scoring at least 75% on the Solver Scale-Up score

Gin11: Number of consumers/ households/ other beneficiaries adopting GCD-supported solutions

Gin12: Percentage of awardees that had measurable positive change in results addressing the specific development challenge

Awardee level; GCD level

Awardee Level

Awardee level; GCD Level

GCD

GCD, Meta

GCD, Meta

Project records - online awardee survey, GCD Performance and Impact Evaluations, Outcome Harvesting

Project records - GCD Performance and Impact Evaluations, Outcome Harvesting

Project records - online awardee survey, GCD Performance and Impact Evaluations, Outcome Harvesting










Sub-Objective 3: Viable Science and Technology Innovations Scaled

IR 3.1: Solver Capacity to Scale Up Improved

IR 3.2: High Potential Solutions Are Adopted

Sub-Objective 4: Solvers Help to Overcome Critical Barriers to Development

GCD M&E at a GlanceThe activities in planning a comprehensive M&E system are detailed in this document. An overview of M&E at the program, across-GCD-programs, and meta- levels is provided below.

• GCD Teams are responsible for developing and implementing M&E systems specific to their GCD program which will also contribute to the meta level M&E system.

• Teams are responsible for creating indicators to assess their GCD program.

R E S O U R C E S


• M&E should begin at the procurement stage of the GCD process. Some indicator questions are most relevant at this stage in the process. Also, clearly explaining indicators to partners at the outset will improve the quality of the data, including improved reliability.

• The GCD team should include all 12 GCD indicators (see Indicators listed in Table 1, Gin1-Gin12) as part of their own M&E Plan.

• Any GCD indicators (Gin1-Gin12) not currently captured need to be incorporated into the next round of funding.

• GCD indicators that align to meta level indicators should be mapped and included in the GCD M&E Frameworks for transparency and ease in reporting at the meta level.

• M&E Frameworks should remain as stable as possible as new calls for innovation are issued so that comparable results can be made over time. Frequent changes to the M&E Framework will result in incomparable data sets over time, thus reducing the value in data.

• At the meta level, a very limited number of indicators (no more than 12) should be aggregated. Refrain from collecting too much data. Wide ranging interventions can yield a great diversity of data, but the collection of unnecessary data can weaken the validity and reliability of data.

• One platform should be used to collect GCD data. GCD Teams have the option of submitting data via a platform created for their innovators by the Meta-level GCD Team, or the GCD teams can input the data themselves onto the platform.

• A random audit of selected reported results should periodically be undertaken at the meta level to ensure GCD teams are reporting valid, reliable data.

• All teams are responsible for reading and understanding the GCD M&E Operational Plan.

ROLES & RESPONSIBILITIES & FLOW OF M&E DATA COLLECTION

For an initiative as complex as a GCD—with a wide variety of activities at multiple levels—it is essential that all GCD team members and awardees understand the requirements of the GCD M&E Plan and their respective roles in contributing to implementation of the Plan. Table 2: GCD M&E Roles and Responsibilities describes the functions and primary roles/responsibilities for each level.


As outlined above, data will be collected at three levels, depending on where the data originates, to maximize attribution and timeliness. As a result, USAID will be able to aggregate or “roll up” indicators across the individual GCDs for decision-making and reporting. Use of common indicators (with agreed-upon definitions), sources, and methods of data collection increases the likelihood that the GCD initiative will have access to comparable data over time. All teams will use the Performance Indicator Reference Sheets.

While your GCD team will be primarily responsible for designing and implementing an M&E system for your GCD program, it is important to recognize the extent to which the GCD team participates in M&E activities across the cohort and at the meta-level. Though a USAID staff member, the Lab M&E Advisor, leads the meta-level M&E analysis, for example, the GCD team will be called upon at different points in the meta-level analysis for data and discussion. An overview of the meta-level M&E process, roles and responsibilities, follows.

TA B L E 2 : G C D M & E R O L E S & R E S P O N S I B I L I T I E S

GCD(GCD M&E ADVISOR)

• Provide accurate, verifiable project information semi-annually

• Keep accurate, up-to-date records of all reported data for audit purposes

• Ensure that contact information is constantly kept up-to-date (name, phone, email, Skype, etc.)

• Participate in M&E training activities

META(LAB M&E ADVISOR)

• Serve as M&E Advisor to all GCD teams

• Develop schedule for reporting and ensure that each GCD team completes reports on a

timely basis

• Meet on a quarterly basis with each GCD M&E team to review actual results against targets

• Verify that GCD and awardee data is being reported accurately

• Provide semi-annual reports to LAB leadership on GCD performance in all indicator

categories

• Develop plan for mid-program performance evaluation

• Manage Meta- and GCD-level evaluation processes

• Work with GCD teams to develop baselines, where applicable

• Assists in the development of training guides (online, offline) for awardees

• Disseminate findings and facilitates use of findings to USAID and broader development

community

GCD(GCD M&E ADVISOR)

• Work with Meta-level M&E Advisor to develop GCD-specific M&E Plan at the start of the

GCD

• Develop technical indicators for SO4

• Work with Meta-level M&E Advisor to develop targets, if applicable, for select indicators

• Ensure that all applicable indicators are included in the competition (grant, prize) solicitation

• Verify awardee reporting (during implementation) and supply necessary supporting

documentation

• Develop training guides (online, offline) for awardees

• Disseminate findings and facilitates use of findings to awardees and GCD stakeholders

R E S O U R C E S


The Lab M&E Advisor will be responsible for merging all data, where appropriate, and triangulating it to maximize program coverage across the GCD portfolio without duplication. Merging data is made possible because each dataset will be associated with a unique awardee or source ID. As data is entered into the online platform, data will be associated with that specific awardee or source. When analyses are run for each indicator, only one source (e.g., awardee, cohort) will be used. Thus, it is important for your GCD team to create and maintain an M&E system that complies with the USAID model.

Data from your GCD Program will be collected by the Meta-level Team on a semi-annual basis. All GCDs will report at the same time to maximize the ability of USAID to conduct meta-analysis. Once data has been collected, the Lab M&E Advisor will analyze results and look for trends at the goal, SO, IR, and awardee levels. The Advisor will share findings with M&E representatives/teams from each GCD to determine whether the GCD is on course to achieve its objectives. Analysis and other information gained from these meetings will then enable your team to determine whether additional support is needed for awardees (i.e. acceleration support, capacity support), improve outreach efforts, and inform future program design/calls).

Indicator Guidance for Tailored Technical IndicatorsEach GCD team will develop technical indicators that are specific to the development outcome they seek and barriers they seek to overcome. These indicators will be tracked at the awardee level and should be developed at the competition stage and included in the solicitation/call. GCD teams should develop two types of technical indicators:

• Indicators that map to existing Agency indicators • GCD-specific custom indicators

First, to improve overall coordination and reporting, GCD teams should develop one or two technical indicators that map to existing indicators at the Bureau level. Some GCDs will likely want to include indicators that map to multiple Bureaus or Offices (for example, Feed the Future + Health, or Health + Water). As many awardees as possible should be able to map their own results to these indicators.

Second, GCD teams should establish GCD-specific custom indicators that are specific to the barrier they seek to overcome. For example, if the Securing Water for Food (SWFF) GCD has a barrier related to water inefficiency, the team might develop an indicator encouraging technologies that reduce the percentage of water lost by leakage and overflows up to the point of consumer metering. Teams may need to develop multiple barrier-specific indicators, and the Lab M&E Advisor will work with teams to develop these. Some of these indicators will be awardee specific, but it is envisioned that several awardees in a portfolio would be able to contribute to a single barrier-specific indicator.

Indicators should be included at the competition stage and awardees should be expected to conduct an early self-assessment. Including indicators early in the process helps in several ways:

• Enables more consistent evaluation across applications• Allows USAID to be more precise in sourcing solutions that overcome a specific, defined barrier• Allows USAID to offer tailored assistance based on awardee self-assessment• Establishes a baseline

We recognize that some GCD teams may weigh these indicators differently and that different weights may be applied based on the type of call developed. For example, an RFA seeking solutions for seed grant funding might weigh innovation and positive social returns most heavily, with less emphasis on the soundness of the financial model and the accuracy of the commercial viability projections. For an RFA seeking solutions for transition grant funding, more weight


might be placed on indicators related to commercial/market viability. GCD teams are encouraged to work with the Lab M&E Advisor to determine which indicators are most important to them in choosing solutions/technologies for funding.

Establishing highly specific indicators that correspond to a solver/solution’s ability to overcome a specific barrier will mean that teams should conduct a thorough Barrier Analysis prior to releasing a “call” for solutions.

M&E DATA ENTRY PLATFORM

The GCD M&E platform DevResults contains an awardee, GCD and Lab interface. It will allow for all awardees across GCDs to answer questions and it will allow specific GCD-level questions. In addition, the awardee user interface permits multiple response, ranked, and ordinal questions, as well as open-ended qualitative questions.

Input on M&E Platform for awardeesThe GCD M&E team will work with each awardee to develop a tailored M&E plan that includes targets for all indicators. Through DevResults, awardees will be able to enter M&E data directly or the GCD may choose to enter the data on the platform. Regardless, data must be entered semi-annually. Each field will have a brief explanation of the required information.

Fields include:

• Date• Name of awardee• Contact Information• Person at organization collecting/entering data• Grant #• Amount of grant• Reporting period

In most cases, awardees will be asked a series of yes/no questions, with follow-up open-ended questions for more thorough responses, and an opportunity to provide supporting data. For example:

• Have you sought and received additional outside capital to support advancement of your solution?• If so, from which sources?• How much have you received?

In order to be able to illustrate project results in multiple formats, some basic GPS data may be requested of Awardees.

Input on M&E Platform for individual GCDsYour GCD M&E team will be responsible for data verification/spot-checking Awardee information, as well as reporting on a number of indicators directly. Similar to Awardees, your GCD M&E team will enter information into the M&E platform so that the Lab M&E Advisor can aggregate across the GCDs.

R E S O U R C E S


Input on M&E Platform for meta-levelThe LAB M&E Advisor will be responsible for data verification/spot-checking GCD information as well as compiling and aggregating all data for reporting purposes.

Baseline surveyNone of the meta-level indicators require baseline figures and so a baseline assessment is not required for reporting on meta-level indicators. Technical indicators that are GCD specific may require baseline data. If you find that your GCD program requires baseline data, your GCD teams is responsible for data collection.

PLAN FOR MONITORING PERFORMANCE

InnovatorsThe GCD M&E model operates on the “trust but verify” principle, meaning that while USAID and its partners believe that Innovators are motivated to show their successes to the world and will thus provide credible evidence, it is the Agency’s responsibility to ensure that any information provided to the GCD team is accurate. Innovators will submit information to the online platform semi-annually. The GCD team will then follow-up data submission for further elaboration/explanation on the information entered into the online platform.

GCD-levelGCD level information will be provided semi-annually, and will be discussed within and among different GCD programs to affect program direction. In addition, GCD M&E advisors will periodically perform site visits to a subset of the awardees to gain more detailed and immediate knowledge of the awardees performance. There may be opportunities in some GCDs for awardees to share their thoughts in a group setting in a participatory performance evaluation (e.g., Duke Accelerator program). Such an opportunity would ideally occur at the midpoint of the GCD (roughly after 2 years).

During semi-annual evaluations, innovators/participants should be asked to evaluate:

• The quality of the support they’ve received from the GCD program and other support needs• Strengths/weaknesses of their technical solution• Successes and failures of their business and scale-up models

Finally, each GCD will undergo an end-of-program evaluation that will be led by an external evaluator, but will include the relevant GCD-level M&E officer and the LAB M&E officer. This evaluation may be structured in many ways, but at least one part of the evaluation should take a random sample from the subset of awardees/prize winners/others receiving GCD funding in order to determine the impact of the program on aspects such as:

• What is the magnitude of the effect that GCD has on the (intended outcome or result indicator)?• How much of the measured change in (outcome or result indicator) the intervention was designed to

affect can in fact be attributed to the intervention? That is, what portion of the result is not explained by the intervention examined by the impact evaluation?

• To what extent did the impact evaluation find differences between the planned intervention and what was actually delivered?

• Were solutions sourced through the GCDs adopted and utilized at scale?


In addition, a performance evaluation of the GCD team is suggested. Questions to include in a performance evaluation include:

• Did we take appropriate steps to source high-risk, high reward solutions?• Did we take appropriate steps to source high potential solutions?• Were any of these external to the GCD?• What were some of the external barriers faced by the Innovators that were not addressed by the GCD

program?• What were some questions that should have been asked during the program that were not?

The performance evaluation is based on the USAID’s outcome harvesting method, Here, “outcome” refers to a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution. Thus, the method seeks to assess change on a variety of dimensions. Open-ended questions such as, “What were some of the unintended outcomes (positive and negative) from the GCD?” capture information not collected by the M&E Framework and meet the objectives of the OH method. Other questions include:

• What were some of the unintended outcomes (positive and negative) from the GCD? Were any of these external to the GCD?

• What were some of the external barriers faced by the Innovators that were not addressed by the GCD program?

• What were some questions that should have been asked during the program that were not?

It is important to reinforce the notion that that both failures and successes need to be captured in the impact evaluation in order for other GCD programs, USAID, and GCD partners to use that knowledge for growth.

Meta-levelThe GCD Program (as a whole) should undergo a performance evaluation after 5 years to address the following questions:

• Did we take appropriate steps to source high-risk, high reward solutions? • Were solutions sourced through the GCDs adopted and utilized at scale?• Was the GCD model effective in making progress toward overcoming critical barriers to development

problems? What was the development impact?

This evaluation will be led by an external evaluator, but will include the Lab M&E officer and a representative from Policy, Planning and Learning (PPL). For purposes of this document, it is important to note that at least one part of this evaluation will survey a random sample of winners. Thus, your GCD team should coordinate with the Meta-level team to ensure that innovators/participants from your GCD program may be selected for survey, if the situation arises.

SUGGESTED M&E COORDINATION

The M&E effort will require a great deal of work by the awardees and M&E advisors, including those in the GCD team, its partners, and other stakeholder groups. Innovators will need to submit information/data in a timely manner for review. M&E advisors should review that information and share their findings regularly. Because analysis occurs at three levels, coordination between innovators, GCD teams, and others to collect and analyze data is critical.

R E S O U R C E S


Data collection at the meta-level not only guides Agency/Lab direction, but it also informs the course for GCD programs. Thus, while the activities in a meta-level analysis are outside the scope of duty for most users of this document, all users can benefit from a review of meta-level activities and clear understanding of when their participation is necessary.

As outlined above, all GCDs will report their data at the same time, semi-annually (e.g., twice a year). Once data has been collected, the Lab M&E Advisor will analyze results and look for trends at the goal, SO, IR, and awardee levels. The Lab M&E Advisor will then convene a quarterly meeting with the M&E representatives/teams from each GCD as well as the GCD Team Lead and the Director of the Center for Development Innovation to review findings and determine whether each GCD is on course to achieve its objectives. The analysis will enable your team to determine whether additional support is needed for its awardees (i.e. acceleration support, capacity support), improve outreach efforts, and inform future program design/calls.

Your GCD team also has the opportunity to share the progress of its GCD program with other program teams at the last quarterly meeting of the year. During that meeting, the GCD owners are invited to present findings on the state of the GCD program and join a broader discussion on whether the GCD initiative as a whole is on course to achieve its objectives.


Overview of Devresults User ManualThe Lab is using a monitoring system called DevResults as a data-collection platform for the innovation programs. This can be accessed by USAID staff and program innovators at www.lab.devresults.com.

The Lab uses this tool to capture and view information as follows:

• An interactive map to view all activities• Metrics for funds obligated and awarded • Results reporting and data visualizations• Upcoming task and event notifications for program managers• Breakdown of programs by sector, status and by mechanism

R E S O U R C E S

tools for innovation programming · usaid’s legacy of developing and implementing innovative...

Documents