THE EFFECTIVENESS OF PUBLIC GOVERNANCE MULTI-STAKEHOLDER

INITIATIVES:

Reviewing the Current Evidence

Jonathan Foxfox@american.edu

Brandon Brockmyerbbrockmyer@gmail.com

Goals of the Synthesis Study

To assess the current state of the evidence for MSI effectiveness and impact

To identify remaining research gaps

Still to come: To identify relevant lessons, insights, and trends and make recommendations to MSIs, participating governments, donors, and CSO good governance practitioners

Conceptual Issue 1:

Defining “public governance multi-stakeholder initiatives”

Global reach: participants on most continents

Voluntary: not encoded in international law

Multi-stakeholder membership: governments, CSOs, multilateral organizations, and multinational corporations

Multi-stakeholder governance: formal power-sharing arrangement

Public governance-oriented: addresses policy and decision-making by national governments

T/A/P focus: develops/improves processes or standards for transparency, accountability, and/or public participation

TMSIs

Public GovernanceT/A/P

Transnational public governance MSIs are a small subset of a much larger universe

Participatory Budgeting in Brazil

Integrated Health Services Initiative in Zambia

Sanitation scores for CA restaurants

Kimberly Process

GAVI Vaccine Alliance

International Union for the Conservation of Nature

Transnational Public Governance MSIs

EITI

OCP

CoST

GIFT

OGP

Transnational public governance MSIs have important differences in scope

Conceptual Issue 2:

Unpacking MSI progress Outputs

Process-Oriented; Compliance

Effectiveness Medium-term strategic outcomes; Influenced by a

mix of internal processes and external social and material factors

Impact Long-term outcomes; Heavily influenced by

external material and social factors

This framework draws on: Gutner & Thompson’s “The Politics of IO Performance” published in The Review of International Organizations (2010)

Definitions will depend on each MSI’s Theory of Change

Definitions will also depend on the Level of Analysis MSIs have international and national goalsOne stakeholder’s “output” can be another

stakeholder’s “input”

Conceptual Issue 2:

Unpacking MSI progress

What types of evidence are out there?

Single-country case studies (collected but not reviewed)○ OGP-IRM reports, EITI validation reports; CoST assurance team reports; OGP-

Support “Inspiring Stories”; “EITI stories”; OGP-Hub Civil Society progress reports

MSI strategy documents○ OGP research agenda; EITI Report from the Working Group on Theory of Change;

OCP Guide for Practitioners

Multi-country studies ○ EITI Progress Report 2014; CoST report on information disclosure and assurance

team findings; OCP Winter 2014 Update; Scanteam evaluation of EITI; “Eye on EITI”

Large-N quantitative studies○ OGP-IRM technical report; BIC survey of civil society participation in EITI

Cross-initiative studies○ Khagram Task Force; OKF “Joined-up data”; “The possible shape of a land

transparency initiative: Lessons from other transparency initiatives”

Documents collected split evenly between single-country focus and

broader MSI focus

# Documents Collected0

50

100

150

200

250

300

350

154

171

MSI FocusSingle-Country Focus

Half of all reviewed documents focus on EITI

# Docum

ents Review

ed

0 5 10 15 20 25 30 35 40

36

13

9

8

8

OCPGIFTCoSTOGPEITI

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

EITI

CoST

OGP

GIFT

OCP

Most MSIs have only been active for a few years

The number of documents identified is imperfectly related to MSI age

EITI documents vary by scope and content

Series10

5

10

15

20

25

30

35

40

45Single-country case study

Multi-country case study

Large-N quantitative study

MSI Strategy Doc-uments

Theory and/or Practice Literature Review

Cross-initiative studies

# Docum

ents

EITI documents reviewed come from a variety of sources

Series10

2

4

6

8

10

12

14

16

MSI Reports

Commissioned Evalua-tions

External (NGO, Government, Indus-try)

Academic Articles & Books

Blog Posts

# Docum

ents

MSI S

ecre

taria

t

MSI B

oard

MSI F

unde

r

ING

O

Count

ry-L

evel

Stake

holde

rs0

1

2

3

4

5

F

M

Document analysis has been augmented by stakeholder interviews

(with more to come)

EITI: What does the evidence tell us?

Only EITI has been operating long enough to generate serious reviews of medium and long-term effectiveness

EITI has made progress in improving the transparency of extractive industry payments to national governments

These bounded reforms have not translated into widespread accountability demands by citizens or improvements in resource distribution

Drawing attention to problems in the extractives industry

Bringing together diverse international stakeholders

Increasing uptake by governments and private interests

Creating new opportunities for dialogue between government, civil society, and the private sector at the national level

EITI has made progress on a number of process-oriented goals:

32 countries are compliant with EITI requirements EI payment information is being released (often for the first time) Countries are innovating

EITI implementation has led to some government policy changes Nigeria EITI Act 2007; Liberia EITI Act 2009; Ghana Petroleum &

Management Bill 2011

Revenue disclosure has uncovered discrepancies in extractive revenue payments e.g., Nigeria, DRC 

EITI has made progress implementing its agenda in participating countries:

Beyond identifying payment discrepancies, there are few examples of tangible benefits to participating countries:

NEITI audits have identified US $9.8 billion owed to the Federal Government, of which $2 billion has been recovered  

DRC found that $88 million in tax collections were missing in 2010, but no funds have been recovered, despite a long investigation by the auditor general’s office

Without a systematic review of EITI countries, there is no way to tell how widespread national-level progress has been

Country OGP EITI CoST GIFT OCP TotalAzerbaijan 5 4 9Brazil 3 2 5Croatia 5 5Ghana 1 3 1 5Guatemala 1 4 5Indonesia 4 1 5Liberia 7 7Mexico 5 5Nigeria 26 1 1 28Peru 3 2 5Philippines 3 3 4 10Tanzania 4 1 5UK 4 2 6

Countries for which five or more documents were collected across all MSIs

Findings regarding national-level MSI progress seem to be based on a relatively small subset of country cases

EITI participation has a statistically positive relationship with FDI, GDP per capita, and perceptions about business climate, rule of law, and government capacity Correlation studies alone are not able to determine the

direction of the causal arrow No relationship between EITI and perceptions of

democracy, political stability, or corruption “Laggards” are washing out  the successes of “leaders”

EITI has shown mixed results with regard to macro-level social impacts:

EITI’s transparency standard was too narrow Disclosure should include the whole contracting process

as well as expenditure data; Information should be disaggregated by project

Transparency gains will not improve accountability in a vacuum Requires building additional capacity within civil society

EITI needs an explicit theory of change To help inform overall strategy for medium and long-term

goals; Establish benchmarks for evaluation

Most assessments conclude that EITI’s agenda was too narrow to have broad social impacts

EITI multi-stakeholder groups (MSGs) in each country are required to set their own implementation objectives in a work plan

EITI Reports must contain basic contextual information about the extractive sector (disclosure of licenses, revenue allocation by region)

New disclosure requirements (including disaggregated reporting, subnational transfers, and social expenditures by companies)

Annual activity reports for all countries (not just compliant countries) Validation will now be procured and managed by the International

Secretariat rather than by implementing countries  Countries revalidated every three years as opposed to every five years Machine-readable data (encouraged)  

Recommendations that were not included: Expenditures disclosure Contracts disclosure (encouraged) Beneficial ownership (encouraged) EITI validation gradient (i.e., fail, meets the minimum requirement, high pass)

2013 Changes to the EITI Standard

From EITI Report from the Working Group on Theory of Change (WGTOC) (2012)

Like EITI, both CoST and OGP have helped to shape debates on public governance at both the international and national level

Like EITI, both CoST and OGP can point to some national-level policy changes

Unlike EITI, neither CoST, nor OGP have been subjected to an assessment of medium or long term effectiveness CoST’s secretariat host organization (EAP) is currently

being reviewed OGP’s first independent evaluation will be completed in

2016

CoST & OGP: What does the evidence tell us?

Bringing together diverse international stakeholders

Creating some new opportunities for dialogue between government and civil society (and, in the case of CoST, the private sector) OGP-IRM data suggests this is an area for improvement

Significant uptake by national governments (particularly OGP: 65 countries)

CoST & OGP show some progress on process-oriented goals:

OGP: 37 countries have completed a full action plan cycle; some policy changes at the national level

○ e.g., FOI laws in Brazil, Sierra Leone, UK (gains can’t necessarily be attributed to OGP)

CoST: 8 countries have disclosed public works data for some projects; some projects redesigned/canceled

○ e.g., inefficient contractor suspended for 2 years by Ethiopian Roads Authority; public works contract for the reconstruction of the Belize Bridge in Guatemala City was annulled

Evidence of progress limited to single-country compliance reports and “success story” briefs

CoST & OGP show some progress implementing their agendas in participating countries:

It is difficult to compare results across countries for CoST and OGP because each participating country sets its own goals

Requirement for Country Participation

Minimum international standard for scope and content

National discretion on scope and/or content

MSI EITI CoST; OGP*; GIFT; OCP

*Do OGP “starred” commitments improve cross-country comparability?

GIFT & OCP: What does the evidence tell us?

GIFT and OCP have only recently formalized their international governance structures

Both initiatives have completed high-level principles

Both initiatives are still designing a strategy for action at the national level

GIFT and OCP show progress promoting their agendas at the international level: GIFT

Harmonizing fiscal transparency standards across the IMF, IBP, and others

UN resolution adopted GIFT’s high-level principles of fiscal transparency

OCP UN Global Compact encourages Open Contracting commitments

Open Government Partnership promotes both OC and FT National action plans Working groups

GIFT and OCP principles are being implemented by national-level actors: GIFT

GIFT consults on fiscal transparency in Brazil, Mexico, Philippines, and South Africa and is arranging peer-learning events in Latin America and Asia

OCP Open contracting principles have been implemented in

some projects in over 15 countries via the World Bank Seven countries have expressed interest in piloting the

Open Contracting Data Standard (OCDS)

Sources of evidence for effectiveness and impact are still few and far between MSIs are still fairly new Existing documents focus on governance structure,

incentives for participation and membership, and procedures to ensure fairness and compliance

Stakeholders report that at the moment, “success” is still a fluid, negotiated concept Politics (and funding) matter

The state of the evidence

Other MSIs have learned from EITI evaluations EITI needs an explicit theory of change

Stakeholders report GIFT and OCP are working on TOC OGP and CoST already have them

The EITI standard should cover more of the EI value chain CoST considers the whole construction project value chain OCP considers all steps in the contracting process

EITI demonstrated no macro-level social impacts, despite ambitious claims to poverty reduction and anti-corruption Neither CoST nor OCP focus on macro-level metrics of success, they

focus on “value for the money” EITI needs to encourage linkages between national MSGs and other

national and international reform movements Stakeholders report linkages to the broader enabling environment are a

major strategy for OCP, GIFT, and OGP

Evaluation of effectiveness remains a challenge All MSIs struggle to fund a robust internal mechanism for

M&E Stakeholders report instances where plans for more comprehensive evaluations were cut

due to budget constraints

All MSIs struggle with how to measure medium and long term effects Often, “leaders” and “laggards” cancel each other out in the aggregate, limiting the

usefulness of large-N probability modeling; Comparing across case studies is difficult, time consuming, and expensive

There is not a shared understanding of the terminology used to evaluate MSIs (i.e., “impact”, “outcome”, “output”) e.g., “Impact” of EITI = Increasing membership? Increasing validation? National reforms?

Increases in FDI?

All MSI’s struggle with the “but for” question The complexity of global challenges precludes attribution of any positive effects to a

single entity, so MSIs must make the case (to funders and national citizenry) for their contribution to desired outcomes

Theory of Change: Tool for strategy or just a reporting device for donors? Helpful for strategic planning

Essential for identifying performance metrics

Articulates a hypothesized causal chain that can be tested as part of single or comparative case analysis Showcases the contribution of various MSI activities to the

causal chain

Increasingly important to funders

GIFT has a broad global TOC, but is still working on their national-level approach

From “Towards Stronger Incentives for Increased Fiscal Transparency, Participation and Accountability” Discussion Paper (2012)

CoST’s TOC distinguishes between intermediate outcomes, outcomes, and impact

From the CoST Public Website (“Objectives”) (2013)

OCP’s TOC for implementing OC principles shows the important role played by the enabling environment and highlights the cyclical nature of reform

From Open Contracting: A Guide for Practitioners (2013)

OGP’s national-level TOC links the action plan cycle to the work of key actors at three levels

From OGP Four Year Strategy 2015-2018 (2014)

From OGP Four Year Strategy 2015-2018 (2014)

OGP’s international TOC is used to drive strategic planning in 4 key areas

OGP uses their TOC to identify outcome metrics for each of the four key strategic areas….

…and for medium-term national outcomes as well

From OGP Four Year Strategy 2015-2018 (2014)

OGP also divides the overall scope of assessment into international and national-level effects and notes where responsibilities for evaluation are internal or external

From OGP Technical Report 1 (2014)

EITI incorporates the consideration of assumptions and evidence into their work on TOC

From EITI Report from the Working Group on Theory of Change (WGTOC) (2012)

…but this can get out of hand...

From EITI Report from the Working Group on Theory of Change (WGTOC) (2012)

Research Gaps Civil society engagement and empowerment

Civil society capacity and interest

Government champions and the political cycle

Private sector participation

Research Gaps Civil society engagement and empowerment

Stakeholders report that CSOs participating in national MSGs are representative of only some segments of civil society, nor do they always consult with the citizens most affected by MSG decisions

○ Who defines “participation” at the national level? ○ Are there ways to determine if and when CSOs face cooption due to their commitment to MSI

processes (i.e. their goal becomes “success of the MSI” rather than social impact)?○ Are there ways to make MSI outputs more “demand driven”?○ What effect does civil society participation have on MSI outcomes?○ Are there tangible benefits to broader civil society participation?

Stakeholders report concerns that civic space is shrinking in some participating countries

○ Are there better ways to measure the health of civic space?○ Is there any evidence that a “big tent” approach is preferable to imposing stricter rules and

sanctions? ○ If MSIs rely on civic participation, should we expect poorer outcomes in these countries? If we don’t

see poorer performance, what does that mean for the legitimacy of the MSI?

Research Gaps Civil society capacity and interest

Stakeholders report that the information being released to the public is too technical for civil society to use

○ Who is responsible for turning data into actionable information? ○ How is disclosed information being used?○ Are strategies for building capacity (e.g., learning sessions) working?

Stakeholders report concerns that a lack of information use by

citizens risks increasing apathy towards governance reform

○ Are MSIs really changing the relationship between government and civil society?

○ How committed are people to the MSI process?

Research Gaps Government champions and the political cycle

Stakeholders in all MSIs report numerous setbacks resulting from changing domestic political realities

○ Where do potential reformers “sit” within government? How do we create more of them (e.g., peer to peer learning)?

○ Which ministries are most successful at implementing reforms and why?

○ What are some useful strategies for preserving momentum across an election cycle or other shift in governmental priorities?

○ What is the effect of MSIs on national government policy, relative to other actors, advocacy efforts, and events?

○ Does MSI participation influence whether and how national governments support or repress civil society?  

Research Gaps Private sector participation

Stakeholders report that some MSIs have been far more successful at private sector outreach than others

○ What are the advantages and disadvantages of private sector involvement?

○ Is there evidence that private sector participation makes a difference in MSI outcomes?

○ What are some strategies for successful outreach?

Addressing “MSI fatigue”

Stakeholders report significant opportunity costs, especially for national-level participants

MSIs require time, energy and resources that cannot be invested elsewhere

○ Stakeholders want more “hard numbers” on national and international MSI outcomes (e.g., dollars saved, quality of roads and schools, citizen satisfaction, perceptions of success) but other MSI outcomes may be more intangible (e.g., linking pro-reform actors, preserving spaces for civil society and government interaction, building trust among coalition actors)

○ Mid-sized comparative case studies can provide a sense for whether and

how MSIs work across countries in the aggregate (single country case studies lack external validity; large-N probability modeling lacks causation)

How long should stakeholders realistically expect wait to see impacts before deciding to try something else?

Propositions for Discussion

I. Public governance MSIs are still operating within the early links of the results chain (i.e., inputs outputs outcomes)

The link between transparency and accountability is increasing understood as a virtuous cycle rather than a linear relationship and relies on assumptions that are still being tested

It is too soon to expect meaningful evaluations of “impact”

II. Points of engagement with the results chain will vary by stakeholder (i.e., one stakeholder’s input is another stakeholder’s output)

III. Institutional learning is occurring both within and between MSIs

Propositions for Discussion

IV. There are opportunities for both transnational and national-level synergies between MSIs

OGP^ EITI CoST* GIFT OCPFord X XG20 X X

Hewlett X XIBP X X

Integrity Action X XNRGI X X

OECD X XOmidyar X X X

One X XTAI X X

TI X X X

World Bank X X X X XWRI X X

^ OGP has working relationships with EITI, GIFT, and OCP

* CoST sits on the OCP board

Propositions for Discussion

IV. There are opportunities for both transnational and national-level synergies between MSIs

Country OGP EITI CoST GIFT OCP

Colombia * *     X^

Guatemala X X X   *

Honduras * *    

Philippines X * X X X

Tanzania X X X    

Ukraine X * *    

UK X * X   *^

USA X *   X  

X = Compliant with MSI; * = Commitment to MSI; ^ = OCDS pilot

I. Public governance MSIs are still operating within the early links of the results chain

II. Points of engagement with the results chain will vary by stakeholder

III. Institutional learning is occurring both within and between MSIs

IV. There are opportunities for both transnational and national-level synergies between MSIs

This research is ongoing and we welcome your input!

Propositions for Discussion