julia compton siac phase1 evaluation

SIAC Phase 1Evaluation

**Some** **preliminary** findings and recommendations for discussion

ISPC, 15 Sept 2016

Julia Compton (independent), Tim Dalton (Kansas State University) and Sophie Zimm (IEA)

SIAC: Strengthening Impact Assessment in the CGIAR

Initially a three year project (2013-16) extended to 2017 Funders: BMGF, and Window 1 (mostly DFID top-up funding)

Two top-level indicators of success:

a) An expansion of the available set of impact studies, providing usefuland credible information to guide future investments in the CGIAR.

b) CRPs and Centers of the CGIAR have institutionalized impact assessment such that ex post impact assessment is regarded as an essential part of prudent research management for accountability purposes and as an input to ex ante strategic planning.

PresenterPresentation NotesAgreement of Fund Council to W1 funding in 2013 .Ambitious for a three year project

Fund Council minutes, 9th Fund Council Meeting, Apr 25-26, 2013, New Delhi, India, pp 18-20 http://library.cgiar.org/bitstream/handle/10947/2967/FC9%20SUMMARY%2c%20Final.pdf?sequence=1

SIAC has dramatically increased central resources for IA

0.00.51.01.52.02.53.03.54.04.5

2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Annu

al S

PIA

budg

et (U

S$ m

illio

n)

Year

SIAC W1SIAC BMGFDIIVA studyISPC

SIAC Phase 1

Source: SPIA data, with thanks to Ira Vater

PresenterPresentation Notes$12 M total SIAC Phase 1But 3 M a year is still not huge.

Compare 3iE at $26M a year, with single thematic windows ranging from about a milion to 2.5 million a year in 2014 and 2015 http://www.3ieimpact.org/media/filer_public/2016/04/29/3ie-annual-report-2015-1.pdf

Leveraging the CRP/Center investment in IA is a good idea (we are still working out but estimated at at least XX a year)

Context: CRP Phase 2 IA budgets

Source: CRP proposals, compiled by SPIA

0.5%1.7%1.5%

1.1%2.3%3.9%

3.0%3.2%3.1%

2.5%4.4%

5.3%

0 1 2 3 4 5 6

CCAFS Climate ChangeLivestock

Water Land EcosystemsLegumes/Dryland

WheatFish

MaizeFTA Forests Agroforestry

GRiSP RiceSIAC

Roots Tubers BananasA4NH Nutrition Health

PIM Policies Inst Markets

Average annual budget (US$ M) for Impact Assessment (labels = % of CRP total budget)

PresenterPresentation NotesNot simple to interpret due to varying definitions of IA and what gets included. Also the cost of IA may vary a lot household surveys vs modelling or qualitative approaches, and type of data collected.

Four main SIAC activity areas1 (Methods): Pilot and verify innovative methods for diffusion data

2 (Outcomes): Institutionalize the collection of diffusion data

3 (Impacts): Assess the full range of impacts from CGIAR research

4 (Building a community of practice): for epIAwithin the CGIAR and between the CGIAR and the development community more broadly

ISPC

SPIA SIAC

Funded projects CGIAR centers External incl.

MSU)

1Independent experts For external

reviews, consultancies

2IA focal points (IAFPs) one per CGIAR

center

3

PSCSPIA chair (chair)

SPIA Secretariat (1 staff)BMGF, DFID

External expert (UC Davis)CO and IEA member (observer)

CGIAR Fund Council EIAC

FAO

BMGF

Administrative reporting

Fund Use agreement

Legendintellectual exchangeFunding Project and financial reporting

W1 money

SIAC Organigram (2013- June 2016, before latest reforms)

ISPC Chair and Exec Director CGIAR

CO

Introduction to this SIAC evaluation

Financed by SIAC and managed for (independence) by IEA

Objectives: Accountability to SIAC funders (BMGF, DFID, System Council) Learning wider lessons for SPIA, and improving the design of SIAC Phase II

(SIAC Phase II proposal going to System Council in 2017)

Scope of evaluation: SIAC, not SPIA. IEA evaluation of ISPC/SPIA planned 2017(However, overlap since SIAC covers whole SPIA budget and workplan.)

Methods

Limited time/ timing (most data collection in August).

Analysis of project and other documentation

Semi-structured interviews /group discussions with >60 people SPIA, ISPC, Fund Office, System Office 11 CGIAR research leaders (CRP leaders/DGs/DDG-Rs) 21 CGIAR Impact Assessment Focal Points 5 donors - BMGF, DFID, USAID, ACIAR and Germany-GIZ External IA experts who know SPIA/SIAC CLEAR and 3iE

PresenterPresentation NotesSome of you were missed! If you want to contribute individually, please let us know. No structured way to inform people of upcoming opportunities to input into evaluations

Questions addressed (in the order of this presentation)

Effectiveness (1) Outputs: Has SIAC produced its planned outputs?

Quality of Science: Is SIAC promoting high quality work and cutting edge methods?

Effectiveness (2) Outcomes: Is SIAC on track for planned outcomes?

Relevance : Does SIAC respond to need, demand and SPIAs mandate?

SIAC management and governance: Does this promote efficiency, accountability and transparency?

Effectiveness (1): SIAC Outputs

- A very ambitious work programme, with many subactivities- Most outputs (under project control) achieved.- Evidence of reflection, learning and adjustments.

KEYCompleted

On track

Some setbacks

Discontinued

New or modified

Too early to tell

1: Develop, pilot and verify innovative methods for diffusion data1.1 Methods for crop varieties1.2 Protocols for diffusion of NRM technologies1.3 New institutional approaches for diffusion data1.4 Disseminate best practices learned

2. Institutionalize the collection of the diffusion data needed for IA2.1 Institutionalise collection of adoption data2.2 Collect and validate NRM claimsOrganise (and institutionalise) POR resultsInstitutionalise collection of adoption data

PresenterPresentation Notes3 technical staff - FTE 2.25

Outputs continued

KEYCompleted

On track

Some setbacks

Discontinued

New or modified

Too early to tell

3. Assess the full range of impacts from CGIAR research3.0 IA on nutrition and health3.1 Long term/large scale epIAs3.2 Experimental/quasi exp studies3.3. Under-evaluated areas3.4 Meta analyses at system level

4. Support Community of Practice for epIA with CGIAR/partners4.1 Small grants4.2 Capdev of IA in the CGIAR4.3 Biennial conference on epIA4.4 Quality star rating for CGIAR IA studiesex-4.5 Support RROs and NARs4.5 SPIA website (but not yet one stop shop for CGIAR IA)4.7 Support and capdev to ConsortiumAt least three studies by post docs

Some key outputsMethods: DNA fingerprinting for varieties;

policy workshop with PIM; NRM pilots

Adoption data: central database of 12 crops and 17 countries with 130 crop x sub-region combinations

CGIAR policy outcome database

Capacity development: Center/Unipartnerships, training, IAFP meetings

Evidence of CGIAR impact: Reviews of water management, livestock. Many other studies under way.

Survey enumerators practice taking leaf samples for genotyping (Photo: SPIA)

Percentage of cost of SIAC studies by area of work

Varietal adoption43%

Crop/Natural Resource

Management34%

Policy6%

Extension/ Information

4%

Nutrition/health9%

Livestock4%

Source: evaluation team analysis of approved research proposals

Quality of science SIAC process (for most): competitive calls, workshops, peer review of

proposals, approval by Project Steering Committee We analysed processes and (accepted) proposals Findings:

Most proposals are good-to-high quality Cutting edge methods in some areas (not always the objective) Some proposals do not clearly specify testable hypotheses One or two cases of poor research design: e.g. sample too small to test hypothesis Few spell out plans to assess distribution of adoption/benefits by gender or by social

groups (e.g. wealth/ethnic/caste) Few contain human subjects statements (ethics) Most calls now specify open data, but not all proposals are clear on plans Calls managed by different people consistency in stated requirements is increasing

PresenterPresentation NotesToo early for outputs only one paper produced to date.

Effectiveness (2): SIAC OutcomesSome progress, but could be stronger Methods have promise for broad uptake, eg DNA fingerprinting Slowly building a picture of CGIAR impacts, but could be more focused Institutionalisation is more challenging (e.g. of adoption studies, or of

links to prioritisation).

Two main areas for improvement identified:a) Organisation of SIAC research in Phase 1b) Theory of change of SIAC

PresenterPresentation NotesMethods have promise for broad uptake: e.g. varietal fingerprinting, joint methods workshops (e.g. on policy oriented research with PIM) Institutionalisation (eg of adoption studies, harmonisation of methods etc) is more challenging Challenging institutional environment, with many bilateral projects pulling IA people in Centers / CRPs in different directions

Organisation of Phase 1 has favoured productivity and highacademic quality with limited resources

Outsourcing management of key areas to trusted academics

Competitive calls. Most calls broad in topic with proposals chosen for high academic quality; opportunities to leverage other initiatives

- Experimental studies- NRM impacts- Nutritional impacts- Testing large scale outcome claims by CGIAR

Lack of systematic gap analysis and linkages to key indicators (IDOs)

Result: Many high-quality studies on the way, but also fragmentation and potential loss of opportunities for institutionalisation and learning

PresenterPresentation Notes40% of project managed by Michigan State University (varietal methods and adoption work). Other universities have managed particular calls. In selecting proposals, I was primarily motivated by what research would give most high quality papers for high impact journals (Research leader) 84% of studies judged moderate-high leverage, n=32Example of fragmentation (not unique) is the nutrition call: studies not only on different agricultural systems (high iron bean varieties in Rwanda; NERICA rice in Sierra Leone ; Crop diversification in maize cropping systems in Malawi and Ethiopia; Smallholder dairy hubs in Tanzania ) but also on different overarching research questionsc) OK for phase 1 Proof of concept of SIAC upscaling IA, leveraging limited funds, trying various avenues and introducing academic rigor into previously-patchy IA d) For Phase 2, more focus especially on key decision questions, institutional learning and institutionalisation

For example: Michigan State varietal methods work, varietal adoption work; Wageningen nutrition studies; Paris experimental studies

Outputs after 5 years Outcomes Impacts

Systematic collection of adoption data institutionalized

Greater range of types of research and types of impact covered by impact assessments

Greater rigor and scale of CGIAR adoption estimates and impact assessments

Greater transparency about CGIAR impacts

Greater confidence of donors in CGIAR system

More evidence-based decision-making within the CGIAR

CGIAR system becomes more effective at reaching its System-Level Outcomes

Increased number of impact assessments of CGIAR research SIAC LOGIC

MODEL

Summary logic model for the project

CGIAR system becomes more effective at reaching its System-Level OutcomesSystematic collection of adoption data institutionalizedGreater transparency about CGIAR impactsGreater confidence of donors in CGIAR systemOutputs after 5 yearsOutcomesImpacts

Increased number of impact assessments of CGIAR researchMore evidence-based decision-making within the CGIAR



Outputs after 5 years Outcomes Impacts

Systematic collection of adoption data institutionalized



Greater transparency about CGIAR impacts

Greater confidence of donors in CGIAR system

More evidence-based decision-making within the CGIAR

CGIAR system becomes more effective at reaching its System-Level Outcomes

Increased number of impact assessments of CGIAR research

Selection, design and management of SIAC activities

Does not specify activities, outputs, links, assumptions and risks explicitly and in a testable way

PresenterPresentation NotesWhy does this matter? Because more thought and investment is needed to meet the longer-term objectives, especially institutionalisationSIAC (like CRPs!) needs to work closely with others to achieve outcomes, to test its assumptions and to change course as needed. This is done intuitively, but more explicit analysis would (we believe) help indicate entry points, changes to make. SIAC did have indicators of success for the BMGF funding, but not all were well linked to the ToC and they were not checked on regular basis (except the minority used for reporting)

Summary logic model for the project

CGIAR system becomes more effective at reaching its System-Level OutcomesSystematic collection of adoption data institutionalizedGreater transparency about CGIAR impactsGreater confidence of donors in CGIAR systemOutputs after 5 yearsOutcomesImpacts

Increased number of impact assessments of CGIAR researchMore evidence-based decision-making within the CGIAR



Relevance

SPIAs mandate (reflects demand) :

To (1) provide CGIAR members with timely, objective and credible information on the impacts at the system level...

To (2) provide support to and (3) complement the Centers in their ex post impact assessment activities...

To (4) provide feedback to CGIAR priority setting and ... links to ex ante assessment...

Source: SPIA website http://impact.cgiar.org/about 7/9/16

PresenterPresentation NotesAll these aspects are relevant to demand from donors, CGIAR Centers and others, according to our interviews. Donors want to see large and credible numbers of people reached, more distributional analysis (rich/poor, farming types etc), and a variety of indicators (gender, nutrition, environment) not just economic CBA. They also want to see Centers/CRPs with high-quality IA that is used in research prioritisation and management. Centers. At system level, they want to see IA feed into ex-ante work and prioritisation more broadly, and also have ex-ante work feed into IA prioritisation.

http://impact.cgiar.org/about

SPIA Mandate SIAC Relevance1. Timely and credible information on the impacts at the system level...

Broadly relevant, but could be better focused. More focus on SLOs/IDOs More gap analysis More consultation on priorities

2. Complement the Centers in their epIA activities

SIAC has moved beyond epIA - potential overlap with Center research needs to be clearer on its role

Increase consultation on priorities with Centers/CRPs IPSC/SPIA evaluation should address wider issue of who does what

3. Support to the Centers in epIA

Broadly relevant, but could be better focused- needs further analysis of comparative advantage Demand for cross-cutting methods (working with expert CRPs), collection

and centralisation of data, (guidance, quality assurance) Concerns about narrow SPIA disciplinary expertise

4. Feedback to CGIAR priority setting and ... links to ex ante assessment...

Potentially well placed in ISPC, but no structured feedback loops to ex-ante and priority setting (?)

PresenterPresentation NotesMost of these issues are not new and have been raised in previous reviews of SPIA dating back to 2000. Already coveredIA term often used for (e.g.) efficacy/proof of concept research and operations research

Management efficient, within constraints

Understaffed for ambitious workplan - leading to outsourcing major parts of the work, and employing consultants

Efficient management, learning and improving

Routing part of funding through FAO - a major efficiency constraint

PresenterPresentation Notes2.25 FTE technical experts in SPIA secretariatAnother constraint is approval of all calls and proposals by PSC - can take months (committed but busy people) and not all qualified for this technical role

Governance - reconsider for Phase 2PSC composition: SPIA, core donors, IA expert, Consortium Office

(observer) and IEA (observer) Approves forward work plans and research proposals No adequate consultation mechanism for System Council and

Centers/CRPs Fund Council did not give clear guidance

Potential opportunities: a) annual meeting of CGIAR research leaders b) new Committee in newly constituted System Council

PresenterPresentation Notesa) SPIA Chair (chair) + 1 from SPIA secretariat, BMGF and DFID (project donors), CO (financial specialist) and IEA (observer)b) Project Steering Committee (PSC) Composition and ToR need revisiting sits uneasily between a technical advisory committee (but many members unqualified) and a governance committee (but inadequate representation)c) Committed and conscientious but very busy people can cause delays if asked for detailed decisions eg on research proposals (7-13 weeks to approve after peer review)FC Evaluation and Impact Assessment Committee potential route, but not operationalIA Focal Points informed but not empowered

Preliminary recommendations (partial)1. Revisit the SIAC Theory of Change...2. More systematic use of reviews and evidence gap maps (of key indicators e.g.

IDOs) to identify key IA topics and research questions.3. Systematic consultation of CGIAR research leaders on IA needs, priorities and

responsibilities4. Improve two-way feedback loops from IA to ex-ante modelling (would

especially welcome discussion and comments from the ISPC meeting on this point)

5. Invest more strategically in supporting CGIAR to institutionalise and learn from IA. Suggestions include: Regular reviews of IA in the CGIAR Revisit design of Community of Practice for IA Develop a (two-way) communications strategy

6. (for IEA) Upcoming ISPC / SPIA evaluation should analyse IA and related activities across the CGIAR and consider roles and responsibilities.

PresenterPresentation NotesChecklists for good practice, for different types of studies, building on QA checklist already developed? More detailed discussion and agreement with IAFPs. More detailed guidance could be developed bit by bit to respond to particular FAQs (when and how should I do a baseline study?).

Thank you

SIAC Phase 1EvaluationSIAC: Strengthening Impact Assessment in the CGIARSIAC has dramatically increased central resources for IAContext: CRP Phase 2 IA budgetsFour main SIAC activity areasSlide Number 6Introduction to this SIAC evaluationMethodsQuestions addressed (in the order of this presentation)Effectiveness (1): SIAC Outputs - A very ambitious work programme, with many subactivities - Most outputs (under project control) achieved. - Evidence of reflection, learning and adjustments. Outputs continuedSome key outputsPercentage of cost of SIAC studies by area of workQuality of scienceEffectiveness (2): SIAC OutcomesSome progress, but could be strongerOrganisation of Phase 1 has favoured productivity and high academic quality with limited resourcesSlide Number 17Slide Number 18RelevanceSlide Number 20Management efficient, within constraintsGovernance - reconsider for Phase 2Preliminary recommendations (partial)Slide Number 24