Appraisal of Public Investment in Practice–

Methodological Approaches and Decision Rules in Advanced Systems

Robert Taliercio

Lead Economist

Public Sector Performance – Governance Global Practice

World Bank Group

October 31, 2014

Motivation • To offer guidance to low and middle income countries on strengthening

their capital project appraisal systems in East Asia, while recognizing that their needs are different – MICs, such as Thailand and Malaysia, have systems that are in need of

strengthening and updating – LICs, such as Myanmar, Lao PDR, and Timor-Leste, have less developed

systems yet urgent need to appraise capital spending due to natural resource revenue windfalls

• Ultimately, systems in developing countries need to be user-friendly yet robust

• Approach: Look at selected OECD country systems, draw out lessons, and prepare capacity-specific guidance

• Support: Governance Partnership Facility grant resources from the governments of Australia, UK, Netherlands, Norway

• Drawn from R. Taliercio, “Appraisal of Public Investment in Practice–Methodological Approaches and Decision Rules in Advanced Systems,” manuscript, forthcoming.

2

Some recent examples of developing country concerns in East Asia

• Should we adjust our discount rate downward? If so, by how much? How should we make this determination?

• Who should prepare the project appraisal, the line ministry or the MOF?

• We have to rank proposed development projects for the next national five-year plan. We have nine months. How should we do it?

• Should our multi-criteria analysis include benefits from import substitution and employment generation?

• What’s the minimum size of projects we should subject to a cost-benefit analysis?

3

Five Core Characteristics of the Appraisal Function

1. Type of evaluation used and under which conditions applied

2. Project proposal, appraisal, and review processes

3. Appraisal methodology

4. Transparency (project data and analysis)

5. Capacity development approaches

Two core enabling factors

– Legal framework

– Organizational structure and staffing

4

• Features of each characteristic • Country examples • Issues arising

Key Issues and Design Questions arising from Divergent Practices in OECD

1. Strong central oversight function vs. autonomous spending units – Who should appraise projects, sponsoring spending ministries (Chile, UK)

or a central unit (Korea)? – Who should prepare sectoral methodologies, central unit (Chile, Korea) or

spending units (UK, Ireland)? – How robust should the review function be?

2. Methodological choices – Which methodology – cost-benefit analysis (Chile, UK, Ireland) or multi-

criteria analysis (Korea) – is most appropriate? If CBA, which methodological approach? If MCA, which criteria should be used and how should they be weighted?

– How far should social (economic) pricing go? How far should sectoral methodologies be adopted, in terms of breadth and depth?

3. Political vs. technical: accommodation vs. purity in system design – To what extent should conduits for political influence be built into the

appraisal process? Different designs: “protect the core” (Chile) vs. “coopt the politics” (Korea) vs. “guidance on the margin” (UK, Ireland)?

5

1. Type of Evaluation and its Application

• Types – Cost-Benefit Analysis (CBA) – Cost-Effectiveness Analysis (CEA) – Multi-criteria analysis (MCA) – Simplified methodologies

• Phases of evaluation – Pre-feasibility (in some systems, options appraisal) – Feasibility – Re-appraisal (in some systems) – Ex post

• Conditions for application – Threshold values (that trigger evaluation) – Exempted sectors/areas – Treatment of sub-national governments – Treatment of state-owned enterprises – Assessment of PPPs

6

Types of Evaluation and Applications – Country Examples: Thresholds

• Thresholds – Appraisal system applies to projects whose cost is > (approx., USD)

– Chile: US$ 150,000

– Korea: US$ 100 million (with central subsidy ≥ $ 50 mln) to take effect soon • Recently increased from US$ 50 mln (with central subsidy ≥ 30

mln) due to economic growth

– Norway: US$ 126 million

– Ireland: US$ <0.6 mln, simple assessment; 0.6 to 6.3 mln, single appraisal; 6.3 to 25.5 mln, MCA; > 25.5 mln, CBA • CBA threshold recently reduced from USD 38.2 to 25.3 mln due to

decreased capital spending envelope

7

Country Examples: Thresholds • Thresholds, even normalized by GDP, show a

huge range – Concept of proportionality in appraisal

operationalized very differently

• System coverage of public investment – In Chile approx. 75% (due to exclusion of large items

like housing subsidies and municipal investment)

8

2013 GDP (USD, bln) Threshold (USD, mln) Threshold/GDP Ratio Normalized thresholds relative to Chile

Chile 277 0.15 0.0005 1

Korea 1305 100 0.0766 142

Ireland 218 25.3 0.1161 214

Norway 513 126 0.2456 454

Source: WDI.

Types of Evaluation and Applications – Country Examples: Other aspects

• SOEs – partially covered in both Chile and Korea – In Korea, if over threshold, covered but with exceptions; in

Chile, coverage of SOE sectors is increasing over time (e.g., port authorities brought into system 2 years ago)

• Sub-national governments partially covered in both Chile and Korea – In Chile municipalities are exempted, but others are covered if

over threshold (in some sectors, sectoral ministry must provide technical endorsement as well, e.g. education); in Korea SNGs covered if subsidy exceeds US$ 30 million

• Exempted sectors in Chile and Korea – Defense – Natural disaster-related emergencies and reconstruction

• Phases – mostly applied in pre-feasibility phase

9

Types of Evaluation and Applications – Features and Issues: Political influence • “Settled politics?” – Some sectors are treated differently for

political reasons – In Chile education is a constitutional right, so it is not subject to

CBA (only CEA is applied); potable water has been contentious: moved from CBA to CEA and back to CBA (and back again?)

– In Korea sewerage services are legally required and so exempted (potable water services are subject to PFS, though previously were subject to a form of CEA)

• High level discretion – Presidential Priority (PP) projects in Chile (President also has

veto over appraisal results) – One recent presidential administration in Korea announced a

couple dozen politically important infrastructure projects that did not pass through the PFS system

10

Types of Evaluation and Applications – Features and Issues: Other aspects

• RSF (Re-assessment Study of Feasibility) in Korea

– Part of Total Project Cost Mgmt. system in which reassessment is triggered if cost overruns of 20% (for certain types of projects);

– When RSFs done, 40% (as a share of total project cost) led to cancellation (2003-2013)

• Support for SNGs – In Chile MOE has a TA line for SNGs to prepare education projects

• Treatment of PPPs?

11

2. Project Proposals and Appraisals

Dimensions of Appraisal • Policy relevance • Economic rationale • Environmental impact • Financial arrangements and

affordability • Technical design/engineering • Achievability (project management) • Commercial/partnering arrangements • Regional impact • Equality • Regulatory impact • Legal context • Information management • Health and safety • Risks

Appraising Options

• Varying scale and/or timing

• Refurbishments vs. new assets?

• Rent, build, purchase?

• Varying output (design) and/or service quality levels

• Varying combination of recurrent and capital inputs

• Outsourcing project services?

• Changing locations/sites

• Regulatory issues

12

2. Project Proposal, Appraisal, and Review Process

• Key responsibilities at pre-feasibility: proposing, appraising, and reviewing

• The UK and Ireland offer a contrast to Korea and Chile in that the former play a less robust gate-keeper function – The role of the central unit is more advisory and less regulatory

– E.g., in Ireland the central unit: does not do appraisals; only reviews CBA appraisals for the largest projects (before they go to cabinet); and has less control over methodological framework

13

Propose Appraise Review Recommendation

Chile Line ministry Line ministry Central evaluation unit (MDS)

MDS decision stands (unless overruled by President)

Korea Line ministry + PFM Comm. (VM of MoSF, KDI, others)

Central evaluation unit (Public and Private Infrastructure Investment Center, PIMAC/KDI)

MoSF* * If review is favorable, line ministry goes on to prepare feasibility study.

MoSF

Project Appraisal and Review – Key Issues

• Ensuring the integrity of the technical appraisal: Managing information asymmetry is a critical issue for success – In Korea PFS “destroyed” asymmetry between spending units and

central agencies – In Chile use of required methodologies and decision rules

(+capacity) is successfully used

• Performance indicators of the health of the appraisal function – Key indicator: Percentage of projects evaluated as feasible?

• In Korea 61.6% (1999-2011), so over 1/3 of proposals rejected • In many developing countries, rejection rate is nil or unknown

• Acquiring technical appraisal capacity in low-capacity public sectors – Role of consultants: pros and cons

• Thailand (de facto, consultants largely do the appraisal for line ministry clients, but possible conflicts of interest and lack of incentives to develop capacity in-house)

• Norway (external consultants provide quality assurance for MOF before approval by cabinet)

14

3. Appraisal Methodology

Four Core Features: a. Methodological Foundations b. National Economic Parameters c. Sector-specific Applications d. Sensitivity/Risk Analysis e. Distributional Analysis

• Some innovations

15

3.a. Appraisal Methodology – Foundations

• Chile: Cost-Benefit Analysis according to Arnold Harberger (U. Chicago) to Ernesto Fontaine (U. Católica) (Metodología General de Preparación y Evaluación de Proyectos)

• Korea: Multi-Criteria Analysis according to the Analytic Hierarchy Process (General Guidelines for Preliminary Feasibility Study, 2008) – Includes CBA as a core factor (weighted at 40%-50%)

• UK: Green Book (HM Treasury, 2003 edition – being updated). – Default approach is CBA but MCA (weighting and scoring) is

available as a technique for comparing unvalued costs and benefits (new version of GB will, however, only allow MCA under very circumscribed circumstances to avoid “silly stuff”)

16

3. b. Appraisal Methodology – National Economic Parameters

• Chile – 13 shadow prices estimated and updated annually – social

discount rate, shadow price of forex, social value of time (urban, cargo, aerial, sports), labor, inputs (fuel), social price of carbon, value of human life, etc. • Some done in-house, others by consultants

• Frequency of updating varies considerably

• Korea – Some parameters set by KDI with MoSF approval (e.g.,

social discount rate); some are updated regularly by KDI when standard guidelines are updated or for annual price index adjustments (e.g., time savings, depreciation rate, value of a life saved, etc.) • Some set by other institutions (e.g., unit price of carbon dioxide

emissions)

17

3. b. i. Approaches to discount rates

UK, Korea – Consumption-based discounting

• Social time preference rate = 𝜌 + 𝜇𝑔

𝜌 is a function of pure time preference (rate individuals discount future over present consumption) and catastrophe risk

𝜇 is the elasticity of the marginal utility of consumption

𝑔 is the growth rate in per capita consumption

Chile, USA – Social Opportunity Cost

• Social discount rate =

𝛽𝑖 𝑟𝑖+ 𝜃𝑗 𝑝𝑗+𝛼𝑓

𝛽 is the proportion of funds from increased savings (by group) and 𝑟 is the after-tax rate of return to savings

𝜃 is the proportion of funds from displaced investment (by sector) and 𝑝 is the marginal rate of capital productivity

𝛼 is the proportion of funds from foreign funding and 𝑓 is the marginal cost of those funds

18

UK = 3.5% Korea = 5.5%

Chile = 6% USA (OMB) = 7% (and test 3%)

Example – Chile: Current Social Prices, 2014

19

3. c. Appraisal Methodology – Sector-specific Methodologies

• Chile: approx. 20 sector-specific methodologies (water, transport, energy, communications, education, health, justice, sports, public buildings, etc.)

– E.g., transport (CBA): economic benefits are time savings of users, operational cost savings, reduction in accidents, and reduction in noise

– E.g., education (CEA): 3 levels (construction, expansion, replacement), where decision is based on analysis of need in terms of the network, population analysis, and transport analysis, taking into account sectoral norms (space per student, e.g.)

• Korea: approx. 10 sector-specific methodologies (airports, ports, IT, roads/railways, social welfare, health, industrial complexes, etc.)

• Key issue is who prepares sectoral methodologies

– In Chile and Korea they are prepared by the central unit

– In the UK and Ireland they are prepared by spending units, consistent with centrally provided guidance (e.g., the Green Book); central units (e.g., HMT) then provide advice and guidance as needed

• Risk that sectoral methodologies may not provide consistent analytic treatment across the public sector

20

Example: Ireland Transport

• An MCA approach, with CBA required above national threshold – CBA assesses net transport user benefits, safety benefits,

environmental emissions impact, noise impact benefits – Multi-modal demand model provides inputs for CBA

• MCA has 5 assessment criteria – Economy (3 sub-criteria) – Safety (no sub-criteria) – Environment (7 sub-criteria) – Accessibility and Social Inclusion (2 sub-criteria; e.g., impact on non-

car owners) – Integration (4 sub-criteria: e.g., impact on “deprived geographic

areas”)

• Issues identified: Appraisal framework is not enforced and quality control is weak; sector guidance is older than central guidance, so some inconsistencies; most parameter values have not been updated since 2004; guidance and methods need to be updated with recent best practice

21

3. d./e. Appraisal Methodology – Sensitivity/Risk and Distributional Analysis

d. Sensitivity/Risk Analysis (required by guidelines in UK, Ireland, Korea, and Chile)

• Sensitivity and risk analysis (e.g., adjusting key parameters by specified ranges, Monte Carlo simulations, etc.)

• Adjustment for “optimism bias” recommended by the UK (e.g., for capital costs, adjust for bias using factors provided in the Green Book for buildings, civil engineering, equipment, etc.)

e. Distributional Analysis?

• UK and New Zealand require distributional analysis, in which, at a minimum, appraisers identify how costs and benefits of a project accrue to different socio-economic groups. Further, distributional weights (e.g., so benefits for low income groups receive a relatively higher weight) should be used where possible.

• Distributional analysis where distributional weights are used is not widely accepted

22

Korea’s “Analytic Hierarchy Process” • Korea’s version of MCA: Gold standard in MCA? • Project score = f(Econ. Analysis (40-50%), Policy Analysis

(25-35%), Balanced Regional Development Analysis (25-30%))

• Economic analysis uses CBA or CEA, sensitivity analysis, etc. (benchmarked to UK)

• Policy analysis looks at consistency with higher level plans and policies, qualitative project risk, etc.

• Regional Development looks at regional backwardness (index), regional economic impacts

– Combines qualitative and quantitative dimensions into a single quantitative score/rank (e.g., project evaluated as feasible if AHP score is 0.5 or greater out of 1.0 total)

– AHP weights set to reflect government policy directions – Team of eight experts involved in analysis (including Team Lead

from KDI, academic (e.g., for demand analysis), engineer (e.g., for cost analysis), with advisory committee (staff of PIMAC + peer reviewers, who can be external)

23

CBA vs. MCA? Technical Issues

• CBA – Has a longer academic pedigree than MCA and CBA is

based on economic theory – Does not explicitly allow for political considerations

(pro or con?) – Cannot always quantify all relevant costs and benefits

• MCA – Can incorporate any dimension deemed important

and can bring together quantitative as well as qualitative factors

– Can be manipulated (e.g., by weights) or design of decision function (country examples) • How to decide which factors are important and justifiable? (E.g., import substitution, employment generation, multiplier effects, regional development?)

24

CBA vs. MCA? Political Issues

• Chile and Korea both try to strike a balance between technocratic insulations on the one hand and allowance of channels for political influence on the other

• Example: Territorial development – Korea: explicitly incorporated into MCA as an important political issue – Chile: not currently explicitly analyzed but hotly contested

• Under a previous administration, new rules introduced for territorial development: if indicators on territorial isolation and/or poverty (e.g., municipal revenue < 50% of national average) met, CEA could be used instead of CBA; later reversed

• So it’s harder to justify investments in less densely populated areas (e.g., because demand/benefits are lower), though investments per capita tend to be higher

• Is either system better at managing politics? Or does it just depend on context? – Korea’s system tries to allow for politics by incorporating political factors into

the policy analysis and balanced regional development categories via the MCA method (“coopt the politics”)

– Chile’s system empowers and protects a technical unit to do CBA but leaves political space via the presidential veto and presidential priority projects categories; also precedent of presidential decision to recategorize sectors from CBA to CEA (“protect the core”)

– Both systems also use law to require basic services, which are then exempt from a “yes/no” appraisal decision and instead subject to a “how” analysis

– These systems evolve over time 25

Appraisal Methodology – Issues

• Need for constant innovation – Recent innovations: Chile (transport safety and value of life; analysis of

environmental impact in terms of emissions/contaminants; shadow price of carbon; evaluation of public parks using hedonic prices); Korea (expansion of AHP to health and industrial complexes); UK (monetization of health damage from noise) and UK/NZ (Better Business Case approach – more in-depth, rigorous analysis of project design and alternative options)

• Management of sectoral methodologies – In Chile must be developed and endorsed by MDS, though usually

developed with sectoral participation (MDS is the regulator that decides whether methodologies are mandatory)

– In UK and Ireland, sectors can develop own methodologies as long as consistent with central guidance

• Challenges – Evaluating intangibles in CBA – Incorporating environmental effects – Evaluation of networks (e.g., in transport) 26

4. Transparency

• Collection, utilization, and publication of investment project information – Critical for analysis and feedback to strengthen system

• Korea’s National Finance Act revised (Jan. 2014) to require disclosure of demand forecast data by KDI; PFS publications are disclosed on KDI website

• Projects Databases – Chile’s Integrated online system with different user profiles

• Public access (not password protected), searchable by region, sector, project code, etc.

• Official access, for each user in the system according to their responsibilities

• Management view, for generating (standard and customized) reports

– Colombia’s projects database (also online)

27

5. Capacity Development – Chile

• A few key features of Chile’s experience – Training done since 1975 til present – Currently MDS offers 3 core courses

(basic/intermediate/diploma in social project evaluation) and several more specialized courses

– MDS finances training (training budget approx. US$ 800,000/year) and participating agencies co-finance ancillary expenses (travel, etc.)

– Nearly 700 officials trained each year through 27 programs – Courses are theoretical and practical (e.g., actual project

analysis) – Certificates/diplomas are perceived in the public sector to have

value – MDS has training coordinators in each region

28

Capacity Development – Issues

• Importance of incentives for staff to want to acquire and use new skills on the job – Chile: requirements for hiring and promotion – Wales: training requirements to access some lines of funding

• Partnerships with academic institutions – Chile: partnerships with Universidad Católica and Universidad de Chile – Korea: KDI’s role as a quasi-think tank, though no official training

provided by MoSF or KDI

• Providing training in-house versus outsourcing – UK BBC: Treasury designed methodology, firms designed curriculum

and now manage training accreditation program

• Strategy for approaching CD in low capacity/LICs/FCS? – Role of outsourcing of some functions (but issue of balance and longer

term consequences) – Use of consultants for core functions?

29

Five Nascent Conclusions and Questions 1. Clear, well-supported appraisal methodology is a sine qua non of a good

PIM system – But what are the implications for sequencing in specific country contexts?

2. Robust methodological management is required in developing country systems (includes clear methodology, with national and sectoral guidelines) – Importance of a robust methodology management/research unit, but where

should it be housed (e.g., MOF, MOP, affiliated think tank)?

3. Technical purity doesn’t exist: Political influence exists in advanced systems, but the systems are designed and negotiated to attempt to channel politics transparently and in a structured manner – A result is that appraisal system coverage is always partial – What design features are useful for helping channel political influence

constructively?

4. Serious capacity development is an expensive, long-term undertaking – So what are the short-term measures that can be used, esp. in relation to

natural resource management?

5. All advanced appraisal systems evolve and adapt over time in some ways (and some deteriorate) – There is a need for constant improvement and vigilance to keep systems

responsive and current 30