WHAT ARE INDICATORS OF

THE STRENGTH OF PUBLIC

MANAGEMENT SYSTEMS?

ISPMS BACKGROUND,

PURPOSE AND APPROACH

Initial motivation behind ISPMS 2

Failure of

prior

indicator

development

efforts Better set of

metrics would

improve

project

effectiveness

New thinking

shifted

debate from

form to

function

Desire to

provide

overall view

of what

government

looks like in a

country

Lots of $ spent without

much results

Ambitious/costly

initiatives not sustained

World Bank PSM

Approach 2011

World Bank IEG

2008 report

Many current indicators

focus on de jure aspects

Gaps exist in coverage of

current indicators across

themes, countries and time

Main purposes of ISPMS

Provide a comprehensive and comprehensible set of indicators that:

Strengthen government ownership of reforms

Help governments monitor and evaluate the results of reforms (see an example)

Support donors in meeting Busan commitments

Enable donor decisions to use country systems by providing information about their strengths and

weaknesses (see an example)

Improve project effectiveness

Help set time-bound targets for progress, which is particularly important for results-based lending

approaches (see an example)

Strengthen donor accountability by helping to track impact (see an example)

Improve targeting of reforms

Enable research to build the evidence base on which institutions matter and in which contexts they are

feasible (see an example)

Track regional differences and changes over time

Help identify outliers (countries and systems that deserve a closer look)

3

ISPMS Approach

Define a set of systems

Define a set of criteria

Apply criteria to available indicators

Are there gaps? What stories can

we tell? Build consensus

Identify strategy to measure

conceptually difficult areas

Overarching principle: Draw on existing efforts and avoid duplication

4

Setting the scope: Defining a set of public

management systems and cross-cutting themes 5

The focus is on upstream institutions at the center of government.

Public

Financial

Management

Procurement Tax Admin. Public Admin.

and Civil

Service

Public

Information

Transparency, Accountability & Participation

*The scope of the project may be expanded to

cover more public management systems.

ISPMS Criteria

ISPMS should meet 4 “utility” criteria…

Criterion Definition

1. Action-worthy We know (or strongly believe) that they contribute to results

2. Actionable They are amenable to government action and project interventions

3. Behavioral Focus on function, not form, and performance, not design

4. Replicable Generated transparently and can be reproduced by others

…and a “feasibility” criterion.

Criterion Definition

Sustainability Should be amenable to cost-effective and regular collection

6

The process: how the criteria were applied

1. Identify potential datasets

(see datasets)

2. Check if datasets are feasible,

replicable and actionable

3. Apply “action-worthy” criteria to

indicators

(see details)

4. Apply “actionable” criteria

to indicators

(see details)

5. Apply “behavioral”

criteria to indicators

(see details)

6. TEGs review results and refine

(see details)

Over 750 indicators

50% aren’t

action-worthy

5% aren’t

actionable

55% aren’t

behavioral

About 100 indicators meet all criteria

7

Existing indicators that meet criteria

0 5 10 15 20 25

Public Information Systems*

Procurement

Tax

Public Adminsitration and Civil Service

PFM

Number of indicators that meet ISPMS criteria

Don’t yet meet

feasibility criteria

Shortlist of most “action-worthy”

* Public Information Systems are a subset of Public Accountability Mechanisms.

8

Building consensus: ISPMS/AGI Governance

Technical Expert Groups (World Bank staff, donors, data collectors)

Procurement Tax Administration Public Financial Management

Public Administration and Civil Services Systems

AGI/ Public Accountability Mechanism

Global Steering

Group

(Donors, researchers, international organizations,

international NGOs)

Secretariat (World Bank)

Steering Group Members

Organization

German Ministry for Economic Cooperation and Development (BMZ)

UK Department for International Development (DFID)

Australian Government Overseas Aid Programme (AUSAID)

Swedish International Development Cooperation Agency (SIDA)

Canadian International Development Agency

Asian Development Bank

European Commission

Global Integrity

International Budget Partnership

Transparency International

Organization for Economic Cooperation and Development

Government of Bangladesh

World Bank

10

Status and Next steps

Launched Secretariat

Developed dataset of existing

indicators that met the criteria

Download the full list of

indicators and data at:

http://go.worldbank.org/99F3L

CSFR0

Financed by the German Ministry

for Economic Cooperation and

Development (BMZ)

Proposal with the following

elements is under preparation:

sourcing new indicators and

improving coverage through a

“marketplace”

incubating indicators in areas

that are conceptually difficult to

measure

exploring the feasibility of

administrative data sources

Phase 1: 2012-2013 Phase 2: 2014-2015

11

• Examples of the uses of ISPMS

• Details on the process used to identify

existing indicators that meet the criteria

Additional details 12

Purpose of ISPMS Strengthen government ownership of reform: example of PEFA

Mackie and

Caprio

(2011)

surveyed 11

countries

during 2009

and 2010.

9 out of 11 countries surveyed use PEFA

to take control of their PFM reform

agendas

LICs use to benchmark status of PFM systems,

becoming a centerpiece of dialogue with

Budget Support donors

MICs use to inform national-level PFM reform

processes

13

Purpose of ISPMS Support donors in meeting Busan commitments

Ongoing efforts to revise indicators monitoring the strength

and use of country financial and procurement systems.

ISPMS work on PFM and procurement will feed into this process

Research on action-worthiness can help deepen learning on

determinants of success

14

Purpose of ISPMS Improve project effectiveness: setting time-bound targets

What are reasonable

expectations for how

much a wage bill can

be decreased over a 5

year period?

-In most countries, the

wage bill didn’t change

much or even increased

slightly. The top 20% of

countries experienced a

decrease of about 10%.

-Donors and partner

countries can use this

information to set realistic

targets for project

outcomes.

15

Upper

qui

ntile

Purpose of ISPMS Improve project effectiveness: hold donors accountable

16

Corporate Scorecard Tier II Indicators : Country Results Supported by the World Bank Success rate is derived by looking at ISPMS in projects

Institutions and Governance

Civil Service

No of Countries with

WB Projects 44

Success Rate 55%

Tax Administration

No of Countries with

WB Projects 12

Success Rate 67%

Public Financial

Management

No of Countries with

WB Projects 26

Success Rate 73%

Purpose of ISPMS Improve targeting of reforms: providing empirical evidence to test

assumptions

Do countries with

higher

transparency of

taxpayer

obligations collect

more taxes [as %

of GDP]?

17

The Process Step 1-2: Identify potential datasets and check replicability,

feasibility and actionability

Afro/Arab/Asian barometer

Bertelsman Transformation

Index

Country Policy and Institutional

Assessments (CPIA)

Human Resource Management AGI

IAMTAX

Medium-Term Expenditure Framework

Dataset

Public Investment Management

Public Accountability

Mechanism

World Bank GAC Diagnostic Surveys

ILO Public Sector Employment Data

BEEPS

Doing Business

Enterprise Surveys

Global Integrity Indicators

Institutional Profiles Database

International Budget Practices and Procedures

Database

Methodology for Assessing

Procurement Systems (MAPS)

Open Budget Survey

Regional tax administration

datasets

Transparency International

Global Corruption Barometer

OECD Comparative

Information Series (Tax)

Evans and Rauch

Wage and Bill Pay Compression

Quality of Government

Public Expenditures and

Financial Accountability

IMF Government Financial Statistics/

fiscal decentralization

Replicability: Does the dataset have an institutionalized

review/cross-checking mechanisms? (100%)

Feasibility: Have data been collected twice for at least 20

countries? (orange datasets don’t meet feasibility criterion)

Actionable: Are disaggregated data available by country (i.e.

not just composites)? (yellow don’t meet criterion)

18

The Process Step 3: Apply “action-worthy” criterion

1. At the first stage the ISPMS Secretariat mapped to

CPIA standards

About 50% of indicators didn’t map to CPIA standards

2. Technical expert group leaders reviewed list and

compared to widely accepted conceptual

framework (e.g. PEFA for PFM, MAPS for

procurement), theory and empirical evidence

19

The Process Step 4: Apply “actionable” criterion

An indicator is “actionable” if it meets any of the

following standards:

It has been used in World Bank or other donor projects

before (and is therefore amenable to reform efforts)

It captures a well specified concept/is specifically

worded

About 5% of indicators didn’t meet actionability

criterion

20

The Process Step 5: Apply “behavioral” criterion

Two steps:

1. Does the indicator measure de facto changes?

De jure indicators measure the contents of government documents (laws,

regulations, etc.), such as the Public Financial Management Law or

Recruitment Regulations (e.g. policy requires an e-procurement portal)

2. Does the indicator measure performance rather than design features?

Design = a system is in place (e.g. IT systems and interface for e-

procurement portal established)

Performance = to what extent is the system being used (e.g. % of

transactions conducted through portal OR (further down the results chain)

cost savings resulting from use of e-portal)

55% of indicators didn’t meet the behavioral

criterion

21

The Process Step 6. Technical Expert Groups (TEGS) review and refine

• Action-worthiness reviewed based on evidence, theory, PEFA framework, used to develop shortlist

• List of existing indicators proposed with new indicators to fill gaps PFM TEG

• Action-worthiness reviewed based on theory; consensus being built via TADAT (work in progress)

• List of existing indicators proposed, new indicators likely to be developed by TADAT Tax TEG

• Framework proposed based on intrinsic value, currently soliciting comments

• Limited existing indicators meet criteria, expanded set to those that don’t yet meet feasibility criterion PACS TEG

• Currently drafting a discussion paper proposing a framework to determine action-worthiness and the application of criteria to develop a list of existing indicators

Procurement TEG

•Suggested expanding indicator set to focus on: Transparency, Rule of Law, Non-executive accountability institutions, and costs of corruption

•April 2013 in-person workshop to discuss methodologies for measuring difficult concepts

•Analytical paper drafted with discussion of concepts, trade-offs, and action steps for measuring/assessing governance

AGI/PAM TEG

22