1

2

MEASURING INSTITUTIONAL PERFORMANCE:

GOVERNANCE INDICATORS

May 7-8, 2012Jordan, AMMAN

Dr. Ohan S. BalianBy

3

Outline

Definition of governance What do we measure? Whose views we rely on? 5 best practice principles

4

Definition of governance

Definition 1: Governance is the manner in which power is exercised in the management of a country’s economic, human and social resources for development

Definition 2: The manner in which public officials and institutions acquire and exercise the authority to shape public policy and provide public goods and services

Note that both are broad and general Definition depends a lot on the specific situation of

a country Most definitions agree on the importance of a

capable state operating under the rule of law.

5

What do we measure?

Rules-based vs Outcome-based indicators Rules-based: Either a legislation exists or does

not exist on-the-books The presence or absence of a legislation on say

corruption or financial disclosure of public officials. Either the legislation exists or it does not (dummy variables)

Outcome-based: We look at the effectiveness of indicators on the ground

In reality, rules and outcome-based indicators are complementary.

6

What do we measure? (contd…)Taxonomy of Existing Governance Indicators

Whose Opinion About what  Rules Outcomes  Broad Specific Broad Specific

Experts        

Lawyers   DB     Commercial Risk Rating Agencies     DRI, EIU, PRS  

NGOs   GII HER, RSF,CIR,FRH GII, OBI

Governments and Multilaterals     CPIA PEFA Academics DPI, PIV   DPI, PIV  

         

Survey Respondents        

Firms       ICA, GCS, WCY Individuals     AFR, LBO, GWP  

         

Aggregate Indicators (Combining Respondents)

    TI, WGI, MOI  

7

What do we measure? (contd…)Legend

Code Name Countries Frequency

AFR Afrobarometer 18 Every 3 yrs

CIR Cingranelli-Richards Human Rights Dataset 192 Annual

CPIA Country Policy and Institutional Assessment 136 Annual

DB Doing Business 175 Annual

DPI Database of Political Institutions 178 Annual

DRI Global Insight 117 Quarterly

EIU Economist Intelligence Unit 120 Quarterly

FRH Freedom House 192 Annual

GCS Global Competitiveness Survey 117 Annual

GII Global Integrity Index 41 Every 3 yrs

GWP Gallup World Poll 131 Annual

HER Heritage Foundation 161 Annual

ICA Investment Climate Surveys 94 Irregular

LBO Latinobarometro 17 Annual

MOI Ibrahim Index of African Governance 48 Every 3 yrs

OBI Open Budget Index 59 Annual

PEFA Public Expenditure and Fiscal Accountability 42 Irregular

PIV Polity IV 161 Annual

PRS Political Risk Services 140 Monthly

RSF Reporters Without Borders 165 Annual

WCY World Competitiveness Yearbook 47 Annual

8

What do we measure? (contd…)Rules-based indicators (Advantages) Main advantage is their clarity Easy to tell if a country has a legally

independent anti-corruption commission Or how many distinct legal steps are

needed to register a business or fire a worker

It is relatively easy to measure progress in such indicators

This clarity is favored by most donors

9

What do we measure? (contd…)Rules-based indicators (Disadvantages)

Value judgments even in “objective” indicators: rules are made by lawyers and legislators who have their own subjective opinions in designing rules

Complexities and lack of knowledge regarding the links from rules to outcomes of interest: the links from rules to outcomes takes time and are not very well understood complicating the interpretation of rules-based indicators

Gap between rules on the books and their implementation on the ground: there are major gaps between laws on the books and their implementation in practice on the ground (Difficulties in enforcement)

These disadvantages require that rules-based indicators must be complemented by outcome-based ones

10

What do we measure? (contd…)Outcome-based indicators (Advantages) Majority of indicators fall in this class They can be specific or general Main advantage of outcome-based indicators is

that they capture directly the views of relevant stakeholders who in turn take actions based on these views

Governments, researchers, analysts and decision makers do care about public views on the prevalence of corruption, the quality of service delivery, and many other government outcomes

Outcome-based indicators provide direct information on the outcome of how the rules are implemented in practice

In other words, they help bridge the gap between rules on-the-books and action on-the-ground.

11

What do we measure? (contd…)Outcome-based indicators (Disadvantages) General outcome-based indicators can be

difficult to link back to specific policy interventions that might influence these governance outcomes

Outcome-based indicators may be too close to ultimate development outcomes of interest and therefore less useful as a tool of research and analysis (GDP, CPI, growth, etc.)

Outcome-based indicators are often based on arbitrary scales – e.g. “agree” vs “strongly agree” on a scale of 1-5 on say the “quality” of public services in surveys.

12

Whose views we rely on?

“Experts” vs firms & individuals Many rating organizations rely on their

network of experts (correspondents) such as the EIU, Doing Business, Global Integrity Index, Open Budget Index, etc…

Others conduct cross-country surveys of firms and individuals such as the Investment Climate Assessment Survey (WB), Enterprise Performance Survey (WB), the Executive Opinion Survey (WEF), the World Competitiveness Yearbook, etc…

13

Whose views we rely on? (contd…)Experts (Advantages) Cheaper to conduct Expert assessments can more readily be

tailored towards cross-country comparability because many rating organizations have elaborate benchmarking systems to ensure that scores are comparable across countries

For certain aspects of governance, experts are sometimes the only natural respondents for the type of information needed

For example, the Open Budget Index’s detailed questionnaire requires some level of expertise to complete

14

Whose views we rely on? (contd…)Experts (Disadvantages) Different experts may have different opinions

about similar aspects of governance The country ratings assigned by different groups

of experts may be too closely correlated – this is because experts rely on each other’s assessments

Expert assessments can be subject to biases For example, many expert opinions are biased in

favor of the business community but businesses and the public may have different opinions of what constitutes good governance

Expert assessments can be “colored” by the ideological orientation of the organization providing the ratings (especially NGOs).

15

Whose views we rely on? (contd…)Firms and Individuals (Advantages)

The fundamental advantage is that such surveys elicit the views of the ultimate beneficiaries of good governance (its consumers and producers – its citizens)

For example, governments can (and often do) dismiss external expert ratings as politically motivated but they cannot do so domestically

Survey-based data can be a powerful rationale for reform

The opinions of domestic citizens about the various dimensions of governance have provided a powerful input for action to reformist policy makers.

16

Whose views we rely on? (contd…)Firms and Individuals (Disadvantages)

Survey questions on governance can be vague or open to interpretation which will lead to serious measurement errors

There may be serious cultural biases – what constitutes corruption in one country may not be defined as corruption in another

It is sometimes useful to construct composite (aggregate) indicators of governance

There will always be measurement errors However, composite indicators will take into

account the complementarities between the different types of indicators.

17

5 best practice principles

1. Avoid false dichotomies and focus on the complementarities

2. Use indicators that are appropriate for the task at hand

3. Transparency is essential for the credibility of indicators

4. Acknowledge margins of error of all governance indicators

5. Exploit (gradually) the wealth of current governance indicators.