promoting public investment efficiency - world...
TRANSCRIPT
1
Promoting Public Investment Efficiency Global Lessons and Resources for
Strengthening World Bank Support for Client Countries
DRAFT BACKGROUND PAPER*
Promoting Public Investment Efficiency A Synthesis of Country Experiences
[Chapter 3]
Murray Petrie
Preliminary First Draft, Not for Citation
Preparatory Workshop
Washington, DC
Wednesday, July 7th, 2010, 11:00 am – 2:00 pm
* The views and opinions expressed in this paper are not those of the World Bank. All remaining errors are those of the authors.
ii
Table of Contents Overview ............................................................................................................................................................................. iv
1. Introduction ................................................................................................................................................................ 1
2. The Dataset and Methodology ................................................................................................................................ 2
3. PIM System Typologies ............................................................................................................................................ 8
A) Advanced PIM systems ........................................................................................................................................... 9
B) Donor-dependent states ......................................................................................................................................... 10
C) Natural resource-dependent states ....................................................................................................................... 13
D) Post-conflict/fragile states .................................................................................................................................... 15
E) New EU members and prospective members ................................................................................................... 16
4. 4 Summary of Cases by the Eight Must-Have Features of PIM ...................................................................... 18
1. Strategic guidance ..................................................................................................................................................... 18
2. Appraisal .................................................................................................................................................................... 21
3 Independent review of appraisal ............................................................................................................................. 25
4. Project selection and budgeting ............................................................................................................................. 28
5. Project implementation ........................................................................................................................................... 30
6. Project adjustment ................................................................................................................................................... 34
7. Facility operation ...................................................................................................................................................... 34
8. Post-Project Review ................................................................................................................................................. 36
5. Some Lessons for PIM System Reform and Areas for Further Work ............................................................ 38
PIM Reform in General .............................................................................................................................................. 39
PIM Reforms in the Country Case Studies .............................................................................................................. 43
Boxes, Figures and Tables
Box 1: Common Features of Advanced PIM Systems ............................................................................................... 10 Box 2: Common Features of PIM in Donor-Dependent Countries ........................................................................ 12 Box 3: Common Features of PIM in Natural Resource-Dependent States ............................................................ 15 Box 4: Common Features of PIM in Post-Conflict and Fragile States .................................................................... 16 Box 5:The “EU-effect:” Common Features of PIM in New and Prospective Member States ........................... 18 Box 6: Stylized Approaches to PIM System Reform .................................................................................................. 46 Box 7: The Main Drivers of Procurement Reforms ................................................................................................... 52
Table 1:: Country Case Studies by Classification and Region ...................................................................................... 7 Table 2: A Simple Typology of PIM-System Performance ........................................................................................ 41 Table 3: Table 3: Main PIM System Challenges by System Typology...................................................................... 48 Table 4: Notable PIM Reforms ...................................................................................................................................... 50
iii
iv
Overview
1. The main objective of this first draft is to present and get feedback on an approach to synthesizing the material in the case studies. In particular, Section 3 of the draft identifies five main PIM system typologies, based on distinctive patterns of PIM functionality across countries. Section 4 then summarizes the findings of a number of the cases by each of the eight must-have features of a sound PIM system. The final section discusses reform of PIM systems, identifies some stylized approaches to reform evident from the cases, and identifies some areas for future work.
2. This first draft is very much a work-in-progress. At this stage preparation of a number of the country case studies has only recently been initiated, while in other cases early drafts of chapters on public investment in PERs, or shorter aide-memoirs are available that do not provide as much depth or consistency of approach as will eventually be evident in the completed case studies. Not all of these partial studies have been incorporated in the first draft of the synthesis. The final synthesis will therefore draw on a considerably larger and richer set of case studies than is available at present.
3. The synthesis chapter will also draw on the thematic chapters on appraisal, procurement, and PIM in natural resource dependent states – although some material has been incorporated in this first draft from the first draft of the chapter on procurement.
4. At the moment the synthesis is based only on the available case study materials. To the extent possible future drafts will incorporate relevant secondary material and perspectives.
1
1. Introduction
1. In the last two years the World Bank has conducted a large number of studies of PIM, in a
diverse range of country settings. The case studies reflect a common focus on assessing how
national PIM systems are actually functioning. The studies also take a systemic approach, assessing
the whole public investment cycle rather than one or two stages only. Most of the studies are
specifically based on applying the framework in Rajaram et al, in some cases augmented in specific
dimensions.1 The remaining studies, which pre-dated the development of that framework, used an
approach to assessing the functionality of PIM systems that is consistent with and in many respects
overlaps with the Rajaram et al framework.2
2. Taken together, therefore, this set of over two dozen studies provides an extensive and
diverse dataset of country experiences on how PIM systems are actually functioning – in terms of
inputs, processes, and outputs from the PIM system. As noted in the Introduction, the approach is
based on the assumption that improvements to the features covered by the diagnostic framework
are likely to be associated with better projects having greater development impact. Due to lack of
data, little attempt has been made in the cases to describe actual rates of return to public investment.
It is not possible, therefore, to test the assumption that the framework captures the essential features
of a well-functioning PIM system, or to assess whether a sub-set of elements might be particularly
important in terms of the ultimate developmental impact of public investment. This is an area for
future research.
3. This chapter attempts to summarize and synthesize the main findings that emerge from the
studies. The next section sets out the rationale, methodology, and coverage of the country case
studies. Section 3 then identifies and describes typologies of PIM systems that capture distinctive
patterns of PIM functionality. This is followed by a high-level summary of the country cases,
organized vertically by each of the eight “must-have” features of a sound PIM system. Section 5
concludes with a discussion of approaches to PIM system reform, and areas for further research and
analysis.
1 The first draft of Rajaram et al was circulated within the Bank in 2008; while this draft was subsequently augmented in some respects, the basic must-have features remained largely in place throughout the period in which the most of the country analysis and case studies were completed. 2 These are the studies undertaken in 2007-2008 of public investment in transport infrastructure in new and old members of the EU, published in Laursen and Myers 2009.
2
2. The Dataset and Methodology
4. The initial core set of case studies was commissioned by the PREM Anchor in the World
Bank starting in 2008, with support of the Korean Trust Fund. The objective was to assess the
actual functioning of PIM systems in a diverse range of countries at different levels of development
using a consistent diagnostic framework. The motivation was to help answer, in a pragmatic and
objective manner, the question: will spending on public investment in any particular country translate into
commensurable productive assets? The core case studies involved a gap analysis against the institutional
features and diagnostic indicators as set out in Rajaram et al 2009 (see chapter 2). The initial eight
countries selected were Belarus, China, Chile, China, Ireland, Niger Ekiti State, South Korea, and
Vietnam. 3
5. A recent wave of additional cases arose from demand from Bank teams conducting Public
Expenditure Reviews and other country work. To date this has generated case studies, completed
PER chapters, draft PER chapters, or aide-memoirs on Brazil, Democratic Republic of Congo],
Republic of Congo (PER Chapter), Equatorial Guinea (PER Chapter), Mongolia (TA and Analytical
Note), Niger (PER Chapter), Peru (forthcoming), Rwanda (Aide Memoire), Sierra Leone, Uganda,
and Zambia.
6. Within these core country cases, there is a broad distinction between first and second
generation studies. The second generation studies go into greater detail in one or two areas (e.g. on
facility operation and on post-project review); discuss what the binding constraints on PIM system
performance are at the margin; and attempt to explore the trajectory of PIM system functionality
and reform over the past decade or so and the drivers of reform.
7. Closely related to the above, a parallel initiative, as described in Chapter 6, used the Rajaram
et al framework to drill down into the specific technical and political economy issues impacting on
PIM in states dependent on natural resources. The framework for PIM in resource-rich states
combines more detailed technical analysis of how the PIM system is performing - including an initial
attempt to rate levels of performance against a set of PEFA-style ordinal indicators - with political 3 The study of China, and the sub-national government studies, are not intended to be incorporated in this synthesis. Square brackets around a country indicate that the case study, PER chapter or extended aide-memoir has not yet been written, or was not available in time to be incorporated in this draft, or was not of sufficient depth for inclusion at this stage e.g. for inclusion of country examples in section 4. For example, a review of Equatorial Guinea was conducted as part of a PER but is not of sufficient depth for inclusion at this stage. In one or two other cases a study is available but has yet to be incorporated in section 4 e.g. Belarus.
3
economy analysis of why the system is performing as it is. The draft framework was piloted in and
has produced case studies of PIM in Angola and East Timor.
8. Finally, two separate cross-country initiatives in the Bank have produced case studies in a
further thirteen countries. First, a 2009 study of six Western Balkans countries assessed PIM system
performance against the eight must-have features (which were fleshed out in more detail in some
areas). This study produced summaries of PIM system performance in Albania, Bosnia, Kosovo,
Macedonia, Montenegro, and Serbia, and a synthesis of shared challenges and positive experiences.4
9. Secondly, a study of public investment in transport infrastructure in seven new and older
members of the EU was completed in 2008. The objective was to identify some of the key issues
and challenges faced by new EU member states, and to discuss good practice examples and
persistent challenges faced by older member states with longer histories of development under
democratic institutions of public management. In contrast to all the previous studies, this one
focused just on one sector. However, the project cycle framework used in the study overlapped with
and was consistent with the eight must-have features, although the focus was on project planning,
appraisal, and selection. This study produced individual country reports on Ireland, Latvia, Poland,
Slovakia, Slovenia, Spain, and UK (England). A synthesis report, drawing on the individual cases,
was published in 2009 (Laursen and Myers, 2009).
10. In general, the studies that are synthesized in this Volume were conducted by World Bank
staff, or teams involving Bank staff and consultants. The methodology relied on in-country
discussions with a range of country officials, review of published and unpublished material, and – to
the extent feasible - some analysis of statistics on public investment. Discussions were held, inter
alia, with central agencies and line ministries, and in some cases with members of the legislature, the
business community, and civil society organizations. Discussions were also held with donor
representatives in countries where external assistance plays a significant role in PIM. All the studies
attempted to go beyond diagnosis to providing recommendations on PIM system reform.
11. There was, however, inevitably some variation in methodology across these cases, reflecting
their somewhat varying origin and objectives. First, as noted there is a broad difference between the
4 This was followed by a conference focusing on the most significant challenges being faced and the most important areas for external support.
4
first and second generation core studies. Secondly, in one set of studies – the EU transport sector
studies – three or four actual projects in each country were tracked through the whole project cycle
to provide further insights into the functioning of the system. Those studies were also conducted by
independent consultants under the guidance of the World Bank and the UK Chartered Institute for
Public Finance Accountancy, with the latter producing the case study synthesis. Thirdly, for the
Balkans study, a standard survey questionnaire was designed that, within the framework of the eight
must-have features, contained some additional diagnostic questions to those in Rajaram et al (see
chapter 10). Finally, in one case (Ekiti State, Nigeria) a consultant conducted a three day workshop
with senior government officials as part of the diagnostic process.
12. Countries were selected for inclusion in the study based in part on their willingness to
participate, given the need to work with the authorities to understand and diagnose how the PIM
system is functioning. As noted, in a number of instances the cases arose out of demand from Bank
teams working in-country on public expenditure reform more generally. In some instances countries
were selected in order to achieve balance across regions and country types. This approach has
resulted in a reasonably representative sample of countries across regions and income levels, but one
lacking examples from the middle east and north Africa, while having an over-representation of
countries from Europe and central Asia.
13. With only three exceptions, the cases are of PIM by the central government, reflecting the
focus of the Rajaram et al framework on central government. This is not to deny the key role that
sub-national governments play in public investment in many countries. While outside the scope of
this initial set of studies, in some countries it may be necessary to drill down to sub-national
government level to properly diagnose how the national PIM system is functioning. For similar
reasons, investment by state-owned enterprises, while also a significant component of public
investment in many countries, was also outside the scope of this initial project, but is likely to be the
subject of future work.
14. Some limitations on the methodology should be noted. First, the definition of public
investment varies quite widely across countries. More advanced countries tend to base the definition
on the economic classification of expenditure, while in some countries the definition of investment
5
includes some items of maintenance.5 In countries that are recipients of aid, or at lower levels of
development, dual budgeting is typical, with a separate development budget for spending on projects
typically associated with donors. The development budget is generally a mixture of capital and
current spending. The case studies took a pragmatic approach, focusing on the processes for
development and implementation of long-lived physical infrastructure projects whether within a
unified or dual budget environment, but indicating in each case how capital or development
spending is defined.
15. Secondly, in most of the case study countries, detailed data on the portfolio of public
investment projects was not available, due to weaknesses in the project management information
systems. In countries with weaker systems there is typically a lack of data on the total cost of each
currently approved project, how this is distributed across fiscal years, and current accumulated rates
of physical and financial completion compared to the approved plan and budget. In one or two
instances where such detailed project databases are maintained, the data was not made available to
the researchers. This lack of data on the public investment portfolio severely restricted the ability to
generate quantitative indicators of PIM system performance, and enforced reliance on more
subjective judgments of institutional design, roles, incentives, and processes. Finally, it is also
difficult to ascertain, from discussion with officials, what information is actually used by decision
makers to form a judgment on, or to influence an investment decision.
16. In all cases, while any existing evidence on appraisal quality was drawn on, there was no
attempt to directly assess in any depth the quality of project appraisal. This is a gap that future
research could usefully help to fill [– see chapter 4 on appraisal]. In addition, the focus was on the
functional steps in the PIM system, rather than on how these functions are allocated across different
MDAs. While the presence or absence of a Ministry of Planning is recorded in Table 5 in the Annex,
the important issue of whether public investment should be the responsibility of a separate Planning
Ministry, under the full control of the MoF, or some other arrangement, is not an issue that is
explored in the case studies or this synthesis. 6
5 For example, some of the Western Balkans countries define investment to include “heavy repairs” and “investment maintenance,” which combine rehabilitation or improvement of assets (investment) with maintenance (current expenditure). 6 Note that consideration is being given to commissioning a separate chapter for this Volume on institutional arrangements for PIM.
6
17. Table 1 presents the full list of [intended] country cases, categorized by region (as defined by
the World Bank), by country level of development, and by whether the country is natural resource-
dependent, aid-dependent, and/or a post-conflict state. 7
18. The country level of development is based on the country classification in the IMF’s World
Economic Outlook (WEO), which defines two major groupings: advanced economies, and
emerging and developing economies.8
19. In this study, all natural-resource dependent, aid-dependent, and post-conflict/fragile states
are emerging and developing economies. The countries shown in the second row, emerging and
developing economies, are those countries in that category that do not also belong to one of the
categories in rows 3-5.
7 Note that at this stage Table 1 shows the full list of cases on which the synthesis will draw, but this is wider than the set of cases for which there will be separate chapters in the Volume. 8 The WEO indicates that this classification “…is not based on strict criteria, economic or otherwise, and it has evolved over time. The objective is to facilitate analysis by providing a reasonably objective method for organizing data.” WEO 2010, Statistical Appendix, p. 147.
7
Table 1:: Country Case Studies by Classification and Region Country classification
Africa East Asia and Pacific
Europe and central Asia
South America
1. Advanced economies
South Korea Ireland Slovakia Slovenia Spain UK (England)
2. Emerging and developing economies
Vietnam Albania Belarus Latvia Macedonia Montenegro Poland Serbia
Brazil Chile
3. Natural resource- dependent states
Angola D. R. Congo Republic of Congo Equatorial Guinea Sierra Leone
East Timor Mongolia
Peru
4. Aid-dependent states
D. R. Congo Republic of Congo Rwanda Sierra Leone Uganda Zambia
Bosnia Kosovo
5. Post-conflict/fragile states
Angola D. R. Congo Republic of Congo Sierra Leone
East Timor
Bosnia Kosovo
20. There is considerable overlap between the donor-dependent, resource-dependent, and post-
conflict/fragile states in the sample. This study uses the World Bank’s definition of “fragile
situations” to include countries with a low CPIA score, and those with a UN or regional peace-
keeping or peace-building mission.9 Natural resource dependence is defined as countries that have
an average share of hydrocarbon and/or mineral fiscal revenues in total fiscal revenue of at least 25
percent during a representative period. While the IMF defines natural resource dependence as either
fiscal dependence or an average share of hydrocarbon and/or mineral export proceeds in total
9 The World Bank defines a country as a Fragile State if it is a low income country or territory, IDA eligible, with a CPIA score of 3.2 or below. The CPIA is used to assess the quality of country policies. IDA defines “post-conflict countries” as (i) a country that has suffered from a severe and long-lasting conflict, which has led to inactivity of the borrower for an extended period of, or at least a substantial decline in the level of external assistance, including from IDA; (ii) a country that has experienced a short, but highly intensive, conflict leading to a disruption of IDA involvement; and (iii) a newly sovereign state that has emerged through the violent break-up of a former sovereign entity.
8
export proceeds of at least 25 percent (IMF 2007), for the purposes of this study fiscal dependence
would seem to be the relevant indicator in terms of the impact on PIM.10
21. Using these definitions, there are 8 natural resource-dependent states, 8 donor-dependent
states, and 7 fragile states. Of these, five are both fragile and resource-dependent (Angola, Congo
Republic, Democratic Republic of Congo, East Timor and Sierra Leone); five are fragile and aid-
dependent (Bosnia, D.R. Congo, Kosovo, Republic of Congo, and Sierra Leone); and two –
Democratic Republic of Congo – are in all three categories. The evidence suggests that the direction
of causality runs from resource-dependence to conflict to aid dependence (Collier and Hoeffler
2007).
22. As discussed in the next Section, distinct typologies of PIM systems overlap to a
considerable extent with the country categories in Table 1. The differences are that the set of
countries with advanced PIM systems is not the same as the set of advanced economies; and
countries that are new or aspiring members of the EU are identified in Section 3 as a separate PIM
typology.
3. PIM System Typologies
23. This section identifies and describes distinctive patterns of PIM system functionality across
the [29] countries in the sample. While there is some heterogeneity within each of the typologies, in
general countries within each typology share a pattern of key PIM features that is somewhat distinct
from that in other groupings.
24. The typologies are:
Advanced PIM systems.
Donor-dependent states.
Post-conflict and fragile states.
Natural resource-dependent states.
New and aspiring EU members.
10 Aid dependence has not yet been defined, and the categorization of countries as aid-dependent in Table 1 may need to be revised. A possible definition would be based on the proportion of total central government capital spending that is financed by concessional donor financing exceeding a threshold, but consistency will be considered with any DAC definition.
9
25. For the purposes of summarizing the case studies in section 4, a further country category is
used: Other advanced economies and emerging and developing countries. The PIM systems across
this grouping are too diverse to constitute a separate typology, and the grouping is used only as a
device to help summarize the findings from the case studies. A number of countries in this grouping
resemble to some extent one or more of the other typologies, but further research and analysis is
required to identify possible additional PIM typologies within this group e.g. upper middle income
economies.
A) Advanced PIM systems
26. The most effective PIM systems amongst the case study countries are in Chile, Ireland,
South Korea, and the UK (England). While there is considerable variation in the institutional
arrangements, modalities, and histories across these countries, in general they exhibit all of the eight
must-have features, and go considerably beyond that basic standard in all respects.
27. It is noteworthy that there is only partial overlap between this group of countries with
advanced PIM systems, and the advanced economies in the dataset. Chile is classified by the IMF as
an emerging and developing economy (albeit one that was admitted to membership of the OECD in
2010), but it has for some time been regarded as having an outstanding PIM system. On the other
hand, Slovakia, Slovenia, and Spain are classified as advanced economies, but the case studies
(admittedly of transport infrastructure only) find their PIM systems to have some important gaps.
Box 1 summarizes the features of advanced PIM systems.
10
Box 1: Common Features of Advanced PIM Systems
The countries with advanced PIM systems tend to exhibit the following features: Strategic guidance and preliminary screening: national and/or sector strategy documents are specific enough, and have sufficient coherence and authority to actually guide public investment, and are used systematically to screen new projects (with at least some projects dropped at the preliminary screening stage). Sector strategies are fully costed, and are closely integrated and consistent with medium term budgets. Appraisal: project development follows a standard and well-defined set of procedures, and projects are appraised using the full range of techniques as appropriate. There are comprehensive central guidelines on project appraisal, including specific detailed guidance on the appraisal of PPPs. Major efforts have been put into building capacity for project appraisal in all four countries, but especially in Chile and Ireland. Independent review of appraisal: this is a key feature of all four of the advanced countries (although arrangements vary markedly). In Ireland and the UK projects are subject to independent review by the CFA. In Ireland, major infrastructure projects are also subject to a public hearing before the end of the appraisal stage. In Chile, project appraisal is conducted by the planning ministry rather than by the sponsoring ministry. To subject these appraisals to independent review, a separate unit was created within MIDEPLAN. In Korea, the CPIM was established in 1999 in KDI, a semi-autonomous agency, to create an arms-length review of project appraisal. Project selection and budgeting: in general, only projects that have been subject to thorough appraisal, and have been independently reviewed, are selected for funding in the budget. Multi-year budget authority supports effective project implementation. In Chile, there is a pipeline of appraised and approved projects that are eligible for budget funding. In Ireland, Korea and the UK, medium term public investment envelopes are in place. Project implementation: there is a strong focus on managing the total project costs over the life of each project. Clear roles and responsibilities are in place for project implementation, with regular reporting on financial and non-financial progress and close monitoring by the CFA. Sound procurement systems are in place and are consistently implemented, with advanced techniques for allocating risks between government and contractors. Project adjustment: specific mechanisms are in place to trigger a review of a project’s continued justification if there are material changes to project costs, schedule, or expected benefits. Facility operation: comprehensive and reliable asset registers are maintained and are subject to external audit. In the UK, full accrual balance sheets are in place across the central government, and the Gateway process focuses specifically on readiness for service (Gateway 4). In Chile, there is systematic recording and checking of completed capital assets, and a register records the name of the responsible official for each asset. Project evaluation: all four countries put significant effort into ex post review. However, in Chile and Korea the executive’s roe is still largely confined in practice to reviewing output delivery against project plans. In Ireland and the UK, efforts are made, in addition, to selectively evaluate the impacts of investment projects on outcomes. In all four countries investment projects are subject to audit by the Supreme Audit Institution, including VFM audits.
28. Key features of the PIM systems in these countries, by each of the 8 must-have features, are
summarized in Section 4 and shown in Table 5 in the Annex.
B) Donordependent states
29. There are a number of PIM features and practices common to the countries in this study
that are aid-dependent: Bosnia, Democratic Republic of Congo, Republic of Congo, Kosovo, Sierra
11
Leone, Uganda and Zambia. In addition, East Timor was highly aid-dependent until 2005 when oil
and gas revenues came on stream, and exhibits some of these features.
30. Aid-dependence impacts on PIM across the whole investment cycle, and results in a highly
distinctive pattern of PIM functionality, as summarized in Box 2.
31. A distinctive feature of donor-dependent states is weak appraisal capacity and reliance on
donors to design good projects. It is certainly the case, however, that weaknesses in project appraisal
are of considerably less concern in a country where major donors generally conduct in-depth project
appraisal. Where public investment is fully-financed domestically, weaknesses in government
capacity to appraise projects create an immediate risk that low quality projects will be selected into
the budget and implemented. Given the scarcity of capital and of implementation capacity, the
selection of a low quality project imposes a high opportunity cost on any society: the loss of the
benefits of the well-appraised project that could potentially have been implemented instead. The
involvement of the major multilateral donors providing or contracting for in-depth appraisal of
projects that they finance reduces this risk considerably.
32. However, a good PIM system in a donor-dependent state nevertheless requires independent
review by the government of donor-appraised projects, for a number of reasons. Independent
review of individual projects by the government is needed to ensure that full account is taken of
domestic conditions and capacity constraints, local impacts, and prospective developments of which
a donor may not be fully aware. The current and future implications of development projects for the
government’s recurrent budget are often not properly assessed by donors. There may also be inter-
actions between projects, and between sectors and sub-sectors, that require some adjustment to
project design, such as sequencing or phasing of implementation. Furthermore, donors are most
focused up to the point of physical completion and handover of a project, whereas the government
has to operate and maintain the assets so as to deliver public services for many years more.
12
Box 2: Common Features of PIM in Donor-Dependent Countries
PIM systems in donor-dependent settings tend to exhibit the following distinctive features: Strategic guidance and preliminary screening: government strategy documents, such as a PRSP, tend to be directed towards the donors, rather than covering both external and domestic investment in an integrated and coherent manner. They are at a level of generality that limits the extent to which they can provide a basis for preliminary screening of projects, and are often not supported by effective sector strategies. Successive iterations of a PRSP often address these weaknesses, although in some cases initially promising national strategy processes have stalled e.g. Kosovo and Bosnia. Appraisal: there is a reliance on donors to conduct appraisal, with a serious lack of appraisal capacity within government; and a lack of guidance on defining the project preparation process and on how to appraise domestically-financed projects. Donor capacity-building on appraisal tends to be agency-specific, with little or no domestic training capacity. Attempts are being made in some countries to strengthen appraisal. Independent review of appraisal: reflecting reliance on donors, there is a lack of capacity for independent review, either of donor projects or domestically-financed projects. Project selection and budgeting: the budget is divided into a recurrent and a development budget, with weak integration between them and substantial off-budget aid. Aid-coordinating units manage the relationship with donors, but this is more a process management function than a strategy or investment priority setting function. The use of PIPs remains quite common, but these can be poorly connected to fiscal policy and the budget. In practice, a PIP tends to be more a coordination tool than a tool to manage the project portfolio strategically or to help enforce review of individual project proposals before they can be considered for budget funding. Agreement by a donor to finance a project is tantamount to the project being included in the budget – subject to basic screening for consistency with a PRSP and any required counterpart financing being affordable. CFAs are not functioning as gate-keepers. Project implementation: unpredictability of donor funding (especially program aid) interrupts project implementation due to lack of alternative financing. Weak project management capacity induces donors to set up multiple Project Implementation Units (PIUs) within implementing agencies that initially help to speed implementation and compliance with fiduciary standards, but which cut across and impact negatively on in-line capacities and accounting and reporting systems. Some rationalization of PIUs is taking place in a number of countries. Procurement is undertaken by PIUs or donors to donor standards rather than national procurement standards, although modernization of procurement laws has recently been put in place across most counties and work is underway to strengthen procurement practices. Project adjustment: reliance on donors to trigger review of any projects that are off-track. No similar mechanism for domestically-financed projects. Facility operation: formal hand-over procedures on completion of donor projects, but inadequate asset registration systems; and inadequate funding for operations and maintenance, in part due to weak integration of recurrent costs of donor projects in fiscal policy and budgets. Project evaluation: reliance on donors to undertake reviews and evaluations of their projects. Otherwise, little or no systematic basic post-project review, let alone evaluation, and little systematic use made of findings from donor evaluations to improve future project design and implementation.
33. Finally, a review of the donor’s project appraisal is also desirable to help counter the clear
empirical tendency to “optimism bias” amongst those preparing projects i.e., the tendency for
project proposals to systematically over-estimate project benefits and to under-estimate costs.
13
34. Recipient governments should also be analyzing whether the overall portfolio of individual
projects constitutes a well-designed program. That is, does the package of projects reflect the
government’s priorities, is it designed to make maximum development impact, and does it impose
future burdens on the recurrent budget that are affordable? When the individual projects are
aggregated across donors, the overall “program” may reveal concerns about the sectoral allocation,
regional distribution, macroeconomic impacts, a proliferation of small projects, overall impact on
vulnerable groups, or affordability and sustainability. While major donors are now trying to
coordinate more closely at a strategic level and to align their project portfolios with the recipient’s
priorities, this is less true of some other donors, and there is often significant activity by NGOs
outside the donor coordination framework.
35. In addition, some donors are willing to finance development projects, but do not themselves
develop and appraise the projects or finance project appraisal. In these instances, the lack of capacity
in a recipient government to develop projects at least to feasibility stage may result in the country
missing out on additional financing from these sources.
36. Finally, weaknesses in project appraisal create serious risks for the quality of domestically-
financed investment. In donor-dependent states, MDA budget submissions for domestically-
financed investment typically lack well-specified project proposals, and may arrive after the deadline
for submission, leaving little time for any assessment by the CFA. MDAs are often not indicating the
future recurrent cost impacts of development projects on their budgets.
37. [ cross-link the above discussion on appraisal to the appraisal chapter]
C) Natural resourcedependent states
38. Countries rich in oil, gas, and mining resources face the fundamental challenge of translating
prospective wealth beneath the ground into productive assets above ground (see chapter 6).
Effective public investment represents a key link in realizing the potential developmental
contribution of extractive industries to broad-based growth and improved social welfare. However,
owing to capacity and political constraints, many of these countries consistently fall short in terms of
the quantity and quality of their capital spending. Consequently, “investing in the capacity to invest”
could potentially yield a high development return.
14
39. Resource revenue-dependent governments face a number of generic as well as specific
challenges to improving public investment. These include revenue volatility; the fact that public
investment often takes place through a variety of modalities (such as Natural Resource Companies,
or resources for infrastructure deals) in addition to traditional budget financing and government
construction; the scale of public investment in relation to the size of the economy (which creates
congestion and crowding effects); and the rapid scaling up and cutting back of capital spending that
often takes place in resource-dependent settings. Furthermore, public investment decisions are
relatively discretionary compared to those on current spending, which creates additional challenges
from a political economy perspective in natural resource dependent states in which the executive is
often over-dominant.
40. There are a number of PIM features and practices common to the countries in this study
that are dependent on natural resources: Angola, Democratic Republic of Congo, Republic of
Congo, Equatorial Guinea, Mongolia, Peru, and East Timor.11 However, this group of countries also
exhibits considerable diversity. For instance, East Timor and Mongolia have only very recently
become resource-dependent; Mongolia is a transition economy; and Angola and East Timor are
post-conflict societies. These factors also have specific impacts on PIM, with the result that there is
considerable variation in the PIM systems across these countries. Nevertheless, some common
stylized patterns can be discerned, as summarized in Box 3.
11 Chile, Sierra Leone, and Vietnam are also defined by the IMF as resource-dependent states, but are not considered further in this section. Neither Chile nor Sierra Leone was dependent on natural resources for fiscal revenues in the period 2000-2005, and the impact of natural resource revenue volatility is not discussed in the case studies (the impact of aid volatility in Sierra Leone is). Vietnam was fiscally dependent on natural resources (in the period 2000-2005) but the impact of this on PIM is not discussed in the case study, and at this stage at least, Vietnam is discussed in section 4 in the category “other advanced, and emerging and developing economies”, rather than as a resource-dependent economy.
15
Box 3: Common Features of PIM in Natural Resource-Dependent States
PIM systems in resource-dependent states tend to exhibit the following features: Strategic guidance and preliminary screening: government strategy documents may not apply to all public investment because of the important role that semi-autonomous state-owned National Resource Companies play in financing investment. They typically do not guide actual investment decisions, which are taken in a non-transparent manner. Appraisal: there is typically a lack of capacity to conduct sound project appraisal, particularly when resource revenues and public investment spending are increasing rapidly. Abundant revenues weaken incentives on MDAs to prioritize and carefully appraise projects.12 The lack of checks and balances on executive power typical in these countries (except Mongolia) results in a lack of demand for project appraisal, and the politicization of public investment decision-making. Non-standard modes of investment are not appraised against standard public investment. Independent review of appraisal: there is a lack of capacity, and a lack of demand from decision-makers for independent review of projects prior to decision-making. Project selection and budgeting: separate development and recurrent budgets are common, with a lack of integration of recurrent costs of projects in fiscal policy and the budget, and a low level of transparency of investment projects (East Timor aside). Where NRFs are in place, their functionality ranges from effective insulation of the budget from resource revenue volatility (East Timor) to no insulation of the budget because of a lack of reserves in the Fund (Angola). The use of PIPs is common, but these can be poorly connected to fiscal policy and the budget, and comprise a large number of projects with a lack of national or sector-specific strategies. Project implementation: Weak implementation capacity, including procurement and project management (coupled with poor planning) results in chronic under-spending of the investment budget. Except in countries where revenue stabilization funds are effective, revenue volatility can create short term pressures to cut investment spending and result in large expenditure arrears e.g. Angola. Multiple modes of project implementation are common, including variously, public infrastructure investment by a National Resource Company, infrastructure spending by international resource companies, and bundled “resources for infrastructure” projects. Project adjustment: absence of processes to trigger review of projects that are off-track, in part due to dominant executive and relative lack of resource constraint, but also due to weak project monitoring. Facility operation: inadequate asset registration systems; and inadequate funding for operations and maintenance in part due to weak integration of recurrent costs of projects in fiscal policy and budget. Project evaluation: little or no systematic basic post-project review or evaluation (lack of demand from decision makers and lack of capacity).
D) Postconflict/fragile states
41. Post-conflict/fragile states exhibit a distinctive pattern of PIM system functionality, as
summarized in Box 4. In the immediate post-conflict period, this typology is in some respects an
12 Collier refers to the impact of revenue abundance generally on PIM in the following terms: “During a price boom government ministries, scenting the money available, put in outrageous bids for more spending. In Kenya one ministry raised its proposed budget thirteenfold and refused to prioritize. Probably it reckoned that other ministries were likely to do the same, so behaving responsibly was likely to leave it at the back of the line. With this sort of behavior, rational public investment is liable to go out the window.” Collier, 2007, p. 40
16
extreme form of donor-dependence. After the initial years following the conflict, this typology
comes to more closely resemble the typical donor-dependent typology as donors help to rebuild
PIM functionality – except where the country is also resource-dependent, in which case features of
that typology will also be present.
Box 4: Common Features of PIM in Post-Conflict and Fragile States
In the immediate post-conflict period, PIM systems in post-conflict/fragile states tend to exhibit the following features: Strategic guidance and preliminary screening: difficulty in forging a new national consensus, and a plethora of international agencies and NGOs, hamper effective strategy setting. Donors play an unusually large role using instruments such as Multi-Donor Needs Assessments and Multi-Donor Trust Funds. Appraisal: very weak capacity, and almost total reliance on donors, who are financing a very high proportion of all public investment. Independent review of appraisal: effectively no capacity for this. Project selection and budgeting: a lot of aid is off-budget, and the recurrent cost impacts of donor projects are poorly integrated in the budget. Periodic large unplanned-for increases in demands on current budget as donor funding winds down from the peak. Project implementation: plethora of PIUs; very weak procurement capacity in relation to the needs for reconstruction; weak monitoring of project implementation. Project adjustment: complete reliance on donor systems to trigger a review when projects are off-track. Facility operation: lack of asset registers; ineffective project hand-over arrangements; evidence of new assets not fit for purpose; insufficient operations and maintenance funding. Project evaluation: no basic post-project review.
E) New EU members and prospective members
42. It is clear that EU membership and prospective membership have a major impact on
country PIM systems. The transport infrastructure study identified new member states of the EU
(Latvia, Poland, Slovakia and Slovenia) as sharing a distinctive pattern of PIM system functionality
that differed from the older member states (Ireland, the UK, and Spain). This functionality was
clearly influenced by specific features of the EU’s Structural and Cohesion Funds which finance
infrastructure in new EU member states to the order of 2-3% of GDP per year. The Western
Balkans study also identified specific mechanisms that heavily influence the PIM systems of
17
countries that have been acknowledged by the EU as candidates for membership or that aspire to
EU membership.13
43. To some extent this “EU-effect” resembles the influence of aid-donors more generally.
However, there are important differences between the mechanisms for aid delivery, and those for
EU funding to member states. For instance, while a donor may temporarily require a recipient to
implement specific projects to donor standards (e.g. procurement standards), EU members and
accession candidates must revise a number of laws to full harmonization with (i.e. make them
identical to) relevant EU law. This represents a more or less permanent change in the legal
framework regulating PIM. It also avoids the problem of multiple donors each imposing a different
national standard for implementation of their separate projects. On the other hand, while donors
supposedly focus on public investment requirements within each recipient country, the EU’s
Structural and Cohesion Funds focus also on investment needs that span member states, such as
regional/trans-continental transport networks – reflecting the unique nature of the EU as an
international grouping of states based on shared jurisdiction, in which externalities between member
states are to some extent internalized within the EU. The Structural and Cohesion Funds heavily
influence the sectoral and sub-sectoral allocation of public investment towards EU (regional)
priorities rather than national priorities.
44. The “EU-effect” is a strong version of a “regional effect”, evident in one or two other
Regional Integration Agreements with respect to procurement. For instance, the UEMOA and
CEMAC regional agreements in west and central Africa involve harmonization of procurement laws
amongst the member states (see chapter 5).
Box 5 summarizes the distinctive features of the “EU-effect.”
13 While the older member states are also eligible for EU funds, the transfers are smaller relative to their domestic investment spending and their impact is more marginal. For instance, in Ireland earlier concerns that CBA was conducted by MDAs only to comply with EU requirements lead to creation of a unit in DoF to monitor compliance with DoF requirements. This perhaps reflects Ireland’s transition from a new to an old member state.
18
Box 5:The “EU-effect:” Common Features of PIM in New and Prospective Member States PIM systems in states that are new members of the EU, or are prospective or aspiring members, tend to exhibit the following features: Strategic guidance and preliminary screening: the EU requires all EU candidate countries to have a National Development Plan in order to be eligible for EU pre-accession funding. For member states (which must also have an NDP), the focus of national and sector strategy documents is on maximizing the country’s share of projects financed by the EU’s Structural and Cohesion Fund. Planning documents cover a fixed 7-8 year period aligned to EU budgetary cycles. There is a lack of coherent overall national strategies applying to both EU and domestically-financed investment e.g. in Latvia the National Strategy is devoted exclusively to projects financed by the EU. Appraisal: use of EU-issued appraisal methodologies in sectors where EU funding is important.14 Appraisal for non-EU financed investment is underdeveloped. Independent review of appraisal: the quality of CBA in the new member states is not independently reviewed within each country. Project selection and budgeting: alignment of planning periods helps to reduce uncertainty of external funding. MTBFs are in place, but in effect the selection of investment projects is still done in the annual budget round. Project implementation: special project monitoring and control structures are put in place, and a joint Monitoring Committee provides coordination with the EU. Full harmonization with relevant elements of the EU legal framework, including procurement laws, regulations, and guidelines, internal audit, and reporting. Compliance with the EU’s procurement framework is more in terms of form than substance e.g. there is little evidence of the use of modern techniques of sharing risks with contractors. Internal audit is at an early stage of functionality, and in some cases (e.g. Slovakia) is almost exclusively focused on EU-financed projects.
4. 4 Summary of Cases by the Eight MustHave Features of PIM
45. In this section, the findings from the case studies with respect to each of the eight must-have
features are summarized. The findings are organized by country category, using the PIM system
typologies described in Section 3, with the addition of a category of “Other advanced economies
and emerging and developing countries.” This category contains only those emerging and
developing countries that are not donor-dependent, resource-dependent, or post-conflict/fragile.
Table 5 in the Annex presents key findings of each case study by the eight must-have features.
1. Strategic guidance
46. Advanced PIM systems: a feature of these countries is strategies that are comprehensive,
specific, have authority across government, and actually guide public investment. They are typically
supported by detailed sector strategies that are fully-costed, and closely integrated and consistent
14 See for instance Guidance on the Methodology for Carrying Out Cost-Benefit Analysis, Working Document No. 4, European Commission, DG Regional Policy, August 2006.
19
with medium term budget frameworks. New project proposals are systematically screened for
compliance with the strategies. This is not a mechanistic compliance exercise: some new project
proposals are rejected at this stage. For example, in Chile, where development planning is carried out
on a sectoral and regional basis, 5-8% of project proposals are rejected at initial screening for
strategic alignment.
47. In Chile, public investments are screened against mandatory sectoral and regional objectives
established by the political leadership. The rejection rate of projects at preliminary screening is 5-8%
(see chapter 8). In Ireland the National Development Strategy is subject to external review by the
Economic and Social Research Institute, which has recommended changes that at times have been
accepted.15
48. Other advanced and emerging and developing economies:16 this is one of the weaker links in the PIM
cycle in these countries. Most of this group has a national strategy document or documents, and
some have sector strategies, but for a variety of reasons they do not guide line ministries in the
development of new proposals, and are of limited use in screening new projects. For instance, in
Vietnam the list of projects in the various strategy documents is too long to help with prioritization;
in Brazil the four yearly Multi-Year Plan (the PPA) is too detailed to constitute a strategic plan, and
there is no formal preliminary screening of projects against it; in Latvia, Poland, Slovakia, Slovenia
and Spain strategy documents contain policy goals defined only in very broad terms; while in
Macedonia, Montenegro and Serbia there is a lack of credible operational strategies, and important
gaps in sector strategies. Government officials consulted in the Western Balkans study frequently
complained about the absence of credible and operational strategic guidance from the political level.
49. In this group, Albania is a country that has put considerable effort in recent years to develop
a strategic framework for PIM. It has recently put in place a coherent, realistic, and authoritative
National Strategy for Development and Integration, supported by sector strategies linked to the
budget. New project proposals are required to be screened against the NSDI – although this in only
being done in a few cases to date, and there is a need to deepen and widen the sector strategies.
Albania’s reforms received considerable external support, but there was strong demand from a
15 The Korea case study does not discuss strategic guidance. 16 These emerging and developing economies exclude those that are donor-dependent , natural resource-dependent, or post-conflict/fragile states.
20
reform-oriented government. Macedonia has also put significant effort into upgrading strategic
planning, but there has been a lack of political buy-in, and the results so far are disappointing.
50. Aid-dependent states: in these countries government strategy documents, such as a Poverty
Reduction and Strategy Paper (PRSP), tend to be directed towards the donors, rather than covering
both external and domestic investment. They are often at a level of generality that limits the extent
to which they can provide a basis for preliminary screening of projects. There is variation across
these countries, however, depending in part on how many iterations of the PRSP there have been.
51. For example, Bosnia’s national strategy is driven largely by external pressures, it is not
costed, and it is not used to screen new projects. Uganda has a well developed national development
plan and a process to review whether project proposals are in line with national and sector
objectives. In practice, however, this preliminary screening has little impact. In Sierra Leone, the
second PRSP, completed in 2008, provides improved high-level strategic guidance for public
investment, setting out four priority areas: transport (especially roads, including feeder roads); energy
(especially electricity); agriculture, and human development. Under a new Aid Policy, introduced in
2009, donor-funded projects are being screened for consistency with the PRSP – although this is not
a very effective screen, because there are no strategies in some important sectors e.g. infrastructure
and agriculture, and for other sectors the strategies need deepening and better costing.
Domestically-financed investment also needs to be subject to better assessment of strategic
alignment.
52. Natural resource-dependent states: in these countries government strategy documents may not
apply to all public investment because of the important role that semi-autonomous state-owned
National Resource Companies (NRCs) play in financing investment. The fact that the executive is
often over-dominant in these countries also results in a tendency for strategy documents to be
under-developed and/or to have little impact. The spatial impacts of local infrastructure needs
associated with resource exploitation create the need for an interface between central and sub-
national government strategic planning.
53. Post-conflict/fragile states: the state of strategic planning in these states depends to a large extent
on how much time has elapsed since the conflict. In the immediate post-conflict period, state
capability to formulate strategic direction has typically collapsed, and a multiplicity of donors is
21
working to try to re-build basic infrastructure destroyed in the conflict. There are also typically
specific forms of poverty resulting from conflict – such as internally-displaced persons, and
impoverished regions where conflict was concentrated – that may impact on public investment
planning.17 In the initial post-conflict years, the approach to national strategy formulation is of
necessity constrained by the need to re-build social consensus shattered by the conflict, and it
generally takes a number of years to develop coherent and authoritative national strategies that
effectively guide first donor-financed, and then domestically-financed investment. While national
strategies are still an early stage of development, efforts at donor-coordination assume more than
usual importance to PIM, including coordination between different agencies within the same donor
government.
54. In some situations early post-conflict attempts by donors to facilitate a PRSP may not be
advisable where international engagement could legitimize a warring group(s) rather than facilitate
resolution of conflict. In these situations instruments such as Multi-donor Needs Assessments may
be preferable to a PRSP as means of coordinating immediate donor assistance.
55. Over time, a PRSP will go through further iterations that may see it deepened and supported
by strategies in key sectors, with the result that it starts to provide strategic guidance for PIM. In
these situations, strategic guidance for PIM resembles that in aid-dependent countries more
generally. This is the case in Sierra Leone (as noted above). In East Timor, where it is also around
ten years since the end of major conflict, the change of government in 2007 - following the outbreak
of further civil violence - has seen the previous National Strategic Plan lapse, and as of mid-2010 no
new national strategy has been put in place.
2. Appraisal
56. Advanced PIM systems: project development follows a standard and well-defined set of
procedures in all four of these countries, with explicit approval required for projects to advance
through various stages in the project cycle. Projects are appraised using the full range of techniques
as appropriate (including CBA, CEA, and business case analysis). England has a long experience
with sound project appraisal, while Chile, Ireland, and South Korea have all put major and sustained
17 This section draws on Dudwick, N and Adam Melsson, A Stocktaking of PRSPs in Fragile States, PREM notes Number 127, Poverty Reduction and Economic Management Network, World Bank, November 2008.
22
efforts into successfully strengthening project appraisal – Chile since the 1980s, Ireland since joining
the EC in 1973, and South Korea since the 1997 economic crisis.
57. Institutional arrangements vary considerably across this group, but in all four countries there
are comprehensive central guidelines on project appraisal, supplemented in some cases by sector-
specific guidelines developed by line ministries and approved by the CFA e.g. in the transport sector
in the UK. The guidelines include specific detailed guidance on the appraisal of PPPs, which are
appraised as just one of a number of possible procurement options. In the UK, the Treasury’s
“Green Book” on appraisal has evolved to include risk adjustment factors to off-set optimism bias,
distributional issues, and a greater focus on proposed project management arrangements. Similarly,
In Ireland DoF’s guidance on appraisal has evolved to include advice on project management and
implementation (including procurement). In both these cases, the evolution of CFA guidance
reflects the fact that the CFAs are moved to place greater emphasis, in their review of appraisal, on
project implementation arrangements.
58. In Chile, the sponsoring entity formulates the project, but it is appraised by the Ministry of
Planning (MIDEPLAN). There is a well-defined set of stages in the project cycle (profile, pre-
feasibility, feasibility, design, execution). In Ireland and the UK the project is initially appraised by
the sponsoring ministry. This is formally the practice also in South Korea, but in practice appraisals
of large projects put forward by MDAs are conducted independently by the KDI, a semi-
autonomous entity under the supervision of MOSF (see discussion of independent review of
appraisal).
59. Major efforts have been put into building capacity for project appraisal in all four countries,
but especially in Chile and Ireland (see chapter 7 on Korea and chapter 8 on Chile for further detail
and references) .
60. Other advanced and emerging and developing economies:18 there is a wide range of circumstances
amongst this group, ranging from some in which donors still play a significant role (e.g. Vietnam), a
number of smaller countries where recent or potential EU membership has a substantial impact on
PIM, and a major emerging market (Brazil). In all these cases, however, appraisal is relatively weak,
although it is being strengthened in some cases.
18 These emerging and developing economies are exclusive of those that are donor-dependent , natural resource-dependent, or post-conflict/fragile states.
23
61. In Vietnam, appraisal of domestically-financed projects is essentially qualitative in nature,
although some large projects in priority sectors are subject to more in-depth financial and economic
appraisal, as are PPPs. For externally-financed projects, there is a reliance on donor appraisals
(donors finance around 20% of total public investment).
62. In Brazil, the great majority of projects are not subject to formal appraisal. Instead,
bottlenecks are seen as being so obvious that no formal appraisal is required, and benefit statements
are often implicit. Priority projects in the large PAC portfolio are exempted from the requirement
for a feasibility study, and this has interrupted abruptly a previous attempt to institutionalize better
project appraisal.
63. In the new member states of the EU (Latvia, Poland, Slovakia and Slovenia) while there is
some CBA of projects associated with EU funding requirements, in general appraisal is more a
matter of form than substance. Project selection is subject to wide political discretion.
64. Amongst the western Balkan countries, Albania has recently implemented regulations clearly
defining the steps in project identification and appraisal, and has put some central guidelines in
place. Implementation of the framework is weak as yet, especially for domestically-financed
investment. Many politically-motivated projects that have not been appraised enter the budget at the
last minute. Serbia also has a framework regulating project appraisal, but implementation is weak,
and there is little economic analysis of project proposals.
65. Aid-dependent states: there is typically a reliance on donors to conduct appraisal, with a serious
lack of appraisal capacity within government. Central guidelines on project preparation and appraisal
are either not in place, or, where they are, are weak in defining the project preparation process, how
to appraise domestically-financed projects, and how to review donor-financed projects. Donor
capacity-building on appraisal tends to be agency-specific, with little or no domestic training
capacity.
66. Sierra Leone illustrates well some of the challenges faced by these countries. There is very
little appraisal of projects by the government, and limited capacity in Sierra Leone for good project
appraisal. Agreement by a donor to finance a project is tantamount to the project being included in
the budget – subject to a simple screening for strategic alignment and any required counterpart
financing being affordable. The decline in project appraisal capacity has occurred over a number of
24
years, as staff in MDA Planning Units (originally allocated there from the Ministry of Planning and
Economic Development), have joined PIUs, attracted by the much higher salaries. Some important
line ministries, such as the Ministry of Public Works, Housing and Infrastructure, currently have no
Planning Unit or planning staff. There are no centralized guidelines on appraisal in place.
67. Kosovo has introduced clearly defined steps in the project preparation process, including a
step where a formal decision is taken focusing mainly on value for money, although in practice there
is little analysis of VFM. In Bosnia, Kosovo and Sierra Leone there are no guidelines or policy
framework covering PPPs.
68. Natural resource-dependent states: in these countries there is typically a lack of capacity to
conduct sound project appraisal, particularly when resource revenues and public investment
spending are rising rapidly. The lack of checks and balances on executive power typical in these
countries (except Mongolia) results in a lack of demand for project appraisal, and the politicization
of public investment decision-making. Non-standard modes of investment, such as bundled
resources for infrastructure contracts, are not compared against standard public investment.
69. In East Timor, the single largest public investment project (a power station) was subject to
limited appraisal, and ran into immediate implementation difficulties due to poor project design and
appraisal. The lack of demand for project appraisal in Timor is in part illustrated by the under-
utilization of a large team of aid-financed project appraisal experts in a key line ministry. Mongolia
illustrates a very different set of circumstances. The Parliament has power over the selection of
public investment projects, many of which have not been appraised prior to their insertion in the
budget presented to the legislature.
70. Post-conflict/fragile states: again, the length of time since conflict ceased it an important factor.
In Bosnia and Kosovo, appraisal is somewhat more advanced than in Sierra Leone and East Timor –
although there is an EU-accession effect at play here as well. Bosnia has some guidelines on
appraisal, but implementation is patchy. In Kosovo, a few MDAs (e.g. Transport, Education) apply
good procedures for identification, preliminary screening, and appraising projects, but most MDAs
do not. In Sierra Leone, it is 8 years since hostilities ceased, and the government is at the point of
considering how it can reduce its reliance on donor appraisal and introduce a basic framework for
project appraisal, which involves a change to the legal framework, organizational restructuring within
25
the Ministry of Economy and Finance, and major capacity building (see Chapter x). In East Timor,
extreme reliance on donor-funding was replaced by rapidly growing domestic revenues from oil and
gas after 2005, and extreme reliance on resource revenues to finance public investment. Renewed
hostilities in 2007 interrupted political and institutional development, and only limited efforts have
been made to build the government’s appraisal capacity.
3 Independent review of appraisal
71. Advanced systems: this is a key feature of all four of the advanced countries, although
arrangements vary markedly. Ireland and the UK illustrate the most common arrangement
internationally, in which line ministry projects are appraised by the sponsoring entity, and then
subject to independent review by the CFA (the Treasury in the UK, the Department of Finance in
Ireland). In these two countries, the CFA has clear authority, capacity, and political support to
subject projects to independent reviews (which on occasions are contracted to outside entities). They
have issued comprehensive guidelines on what project proposals must contain. In addition, a sector
ministry, such as the MoT, will have subjected a project developed by an agency under its
supervision to review prior to the project being submitted to the CFA.
72. In the UK, there is also a Gateway process managed by the Office of Government
Commerce (an independent office of the Treasury), a staged external “peer review” of large projects
at six stages of the project cycle. At each stage, a project has to be formally approved before moving
to the next stage. The 6 Gateway stages are: 0 is strategic assessment; 1 is business justification; 2 is
delivery strategy; 3 is investment decision; 4 is readiness for service; 5 is operations review and
benefits realization. Typically the process involves a panel of experts, who have no other
connection with the project, working with the sponsoring agency to verify each stage approval.
73. In Ireland, major infrastructure projects are subject to a public hearing before the end of the
appraisal stage, in which the business case for the project must be clearly set out. A recently
established Central Expenditure Evaluation Unit in DoF is tasked with promoting best practice in
appraisal.
74. Korea and Chile each has quite different arrangements. In Chile, project appraisal is
conducted by the Technical and Economic Analysis Unit in the planning ministry and its regional
agencies (rather than by the sponsoring ministry), according to a standardized scoring system. The
26
results of the review are entered into the Integrated Project Bank (the BIP), and the appraisal must
be completed within a specified time period. The BIP is used as a management and communication
tool. To subject these appraisals to independent review, a separate unit in MIDEPLAN conducts
periodic checks on the quality of appraisal.
75. In Korea, the MoF established PIMAC within the KDI in 1999, to create an arms-length
review function. PIMAC conducts a preliminary feasibility study (PFS) on major infrastructure
projects referred to it by MOSF (including mandatory review of all unsolicited PPPs). The PFS
comprises three main elements: economic analysis; policy analysis; and balanced regional
development analysis, which are synthesized using a multi-criteria decision making approach. The
PFS was supposed to be relatively brief, and, where its findings were positive, was to be followed by
an in-depth feasibility study carried out by a line ministry. In practice, the PFS is an in-depth
analysis, and line ministries tend to accept its findings and do not conduct their own feasibility
studies. For the period 1999-2007, PIMAC conducted a PFS on 335 projects. It found 56.2% were
feasible, and 43.8% were infeasible (see chapter 7 for more details).
76. Other advanced, and emerging and developing economies: this step in the PIM cycle is weak across all
the countries in this group. In a number of them, the CFA or Planning Ministry lacks the formal
authority and/or the political support to effectively subject MDA proposals to independent review.
This is the case in the new member states of the EU (Latvia, Poland, Slovakia, and Slovenia).
77. In Brazil, there is virtually no independent review of the merit of proposed projects. While
there is a Committee for Monitoring and Assessment which is supposed to review projects
submitted for the budget, the exemption of PAC projects has emasculated this.
78. In Vietnam, the Ministry of Planning and Investment has tended to play a decreasing role in
appraisal, confined to organizing the appraisal process with appraisal left to MDAs. Its review of
donor-financed projects is limited to the social and environmental aspects. On occasions the Prime
Minister has organized a temporary appraisal council for a specific project. The government is
pursuing donor harmonization of appraisal (e.g. a common feasibility study) but this has yet to bear
fruit.
79. In two of the western Balkan countries, recent efforts have been made to introduce an
independent review function. In Albania, the Department of Public Investment Management in
27
MoF has since 2008 reviewed project proposals against a check-list of basic screening questions, and
has in fact rejected around 20% of projects put forward in the last two years. While there are still
many poorly prepared projects that are approved, MoF does not want to further test the limits of its
political support. It also lacks the technical capacity to challenge the quality of economic and
financial appraisal. In Serbia, ad hoc Revision Commissions comprising independent external
experts review feasibility studies, and have authority to make mandatory recommendations on
technical engineering issues to the sponsoring ministry.
80. Aid-dependent states: reflecting the reliance on donors, there is a lack of capacity in these
countries for independent review of projects, either of donor projects or domestically-financed
projects. Attempts by the CFA to enforce discipline are at times undermined by political pressures.
81. In Kosovo a PIP Manual is in place, supported by a manual on project preparation for
MDAs. While screening for entry to the PIP is weak, and it has been estimated that half the projects
in the PIP are deficient, nevertheless the recent strengthening of the PIP process is considered to
have made a significant improvement to the project preparation phase. Sierra Leone is currently
considering introducing a PIP in an attempt to improve project preparation and selection (see
Chapter 18).
82. Natural resource-dependent states: there is generally a lack of capacity in the CFA for independent
review of appraisal, again reflecting in part a lack of demand from decision-makers for project
appraisal. This is true, for instance, in East Timor and Mongolia.
83. Fragile states: again, the length of time since hostilities ceased is important. In Sierra Leone the
Ministry of Planning was merged with the MoF in 2008 (for the third time in recent decades), but
little progress has been made in integrating the former planning staff of MoPED structurally and
functionally, and they have become increasingly marginalized from where development projects are
planned, implemented and monitored in practice. The staff have relevant qualifications, but lack
resources and motivation, due in part to the very large gap between their remuneration and that of
staff paid under donor-funded arrangements.
28
4. Project selection and budgeting
84. Advanced economies: as a general rule, in these four countries only projects that have been
subject to thorough appraisal, and have been independently reviewed, are selected for funding in the
budget. Multi-year budget authority supports effective project implementation, including some
flexibility to adjust budgets between years (although the degree of flexibility varies across countries).
85. In Chile, the Ministry of Planning maintains a pipeline of appraised and approved projects
that are eligible for budget funding. The BIP is a rolling three-year framework. MDAs can enter
projects and project data into the BIP, and the database is publicly available. Since 1980 it has been a
requirement that each project requiring budget financing has to be appraised in accordance with the
basic methodologies and has to demonstrate positive indicators. MIDEPLAN and MoF work
closely together to ensure that the recurrent cost impacts of investment projects are incorporated in
the annual budget and MTBF.
86. In Ireland and the UK, sector strategies are closely integrated with the MTBF. Medium term
public investment envelopes are negotiated between the CFA and line ministries. For example, in
Ireland DoF provides 5 year capital envelopes; in the UK ministries negotiate three year budgets,
and must produce Departmental Investment Strategies that cover new and existing assets and their
maintenance. Also in the UK, PPPs in which private contractors will be repaid from public spending
are treated the same as standard public investment, in terms of their impact on departmental
budgets, removing the fiscal accounting bias towards PPPs.
87. In Korea, an MTEF was introduced in 2005 and represented a significant change from
detailed project level control by MOSF and annual bottom up budgeting, to a top-down medium
term (five year) approach. MOSF’s budget call circular now transmits spending ceilings for sectors
and programs to line ministries. The PPP program, however, remains beyond the direct control of
the budget authority.
88. Other advanced, and emerging and developing economies: again, there is a wide variety of
circumstances. In Brazil, there is little effective gate-keeping to prevent un-appraised or poorly
planned projects from being selected for budget funding. In recent years the Congress has cut
current spending from the budget submitted by the executive. It has also introduced new projects
29
that have not been appraised (there is no pipeline of appraised projects), representing around 20%
of discretionary spending– although in many cases these amendments are not implemented.
89. In Vietnam there is weak coordination between MPI and MoF (e.g. with respect to
integration of capital and current spending) in the context of a fixed five year PIP, and the PIP has
not functioned as a screening mechanism to filter out poorly prepared projects. In the new member
states of the EU, three year MTBFs are in place, but much prioritization of public investment is still
done in the annual budget process, and project appraisal has limited influence on project selection.
90. In some of the western Balkans (Macedonia, Montenegro, Serbia) the CFAs are not credible
gatekeepers, and project selection is politicized. Albania aside, there is a misalignment between the
budget and medium term fiscal frameworks, and in all of these countries there is a need for better
integration of capital and current spending. Many projects selected for the budget are not “ready to
go”, and the supplementary budget during the year constitutes an alternative entry point for new
investment projects, undermining the annual budget process. Albania is introducing program
budgeting, which is intended to integrate capital projects alongside current spending associated with
the same program.
91. Aid-dependent states: the budget is divided into a recurrent and a development budget. Aid-
coordinating units manage the relationship with donors, but this is more a process management
function than a strategy or investment priority setting function. Substantial volumes of aid are often
off-budget, and there is typically a lack of integration of the recurrent costs of donor projects in
fiscal analysis and the budget. A number of these countries operate Public Investment Plans (PIPs),
but these can be poorly connected to fiscal policy and the budget. In practice, the PIP tends to be
more a coordination tool than a tool to manage the project portfolio strategically or to help enforce
review of individual project proposals before they can be considered for budget funding.
92. For instance, in Bosnia there is a significant number of projects off-budget. These include
donor-financed projects, projects financed by extra-budgetary funds, and projects financed from
privatization proceeds.
93. Natural resource-dependent states: in East Timor, the Budget Call Circular does not contain a
ceiling for each MDA for their development budget spending. There is also a lack of time in the
30
budget calendar for the CFA to review new project proposals, especially given that many MDAs are
late with their budget submissions.
94. In Mongolia, as noted, parliament plays a key role in adding and changing projects with little
regard for economic feasibility or implementation capacity. For instance, in 2007 parliament
expanded the Cabinet’s proposed public investment program by 77% on projects most of which
lacked basic feasibility studies. The electoral system provides incentives for MPs to channel “pork
barrel” investments to their constituents, and existing legislation gives strong powers to parliament
in shaping the size and composition of capital spending.
95. In Angola there is a very large number of projects in the PIP. MoF has a PIP database, and
in theory can interrogate it, but in practice this is not possible. MOFs database of public investment
projects is not compatible with the Planning Ministry’s database, and projects do not have a
common project number across the two agencies.
96. Fragile states: arrangements here are similar to aid-dependent states, but in the early post-
conflict years aid coordination is a major challenge and aid-coordination units may not be in place.
Donors of necessity attempt to conduct more coordination through instruments such as multi-
donor trust funds.
5. Project implementation
97. Advanced PIM systems: there is a strong focus on managing the total project costs over the life
of each project. Clear roles and responsibilities are in place for project implementation, accounting
systems record and report total and annual project costs, and there is close monitoring by the CFA.
Sound procurement systems are in place, with advanced techniques for sharing risks with tenderers.
98. In Korea a Total Project Cost Management (TPCM) system controls all costs over the life of
a project. The TPCM system was introduced in 1994, and MOSF has been revising the “Guidelines
for Total Project Cost Management” on an annual basis. TPCM includes all costs accrued over the
life of the project, regardless of the source of funding, and applies to a series of defined stages viz.
project conception; PFS and feasibility study; draft design phase; blueprint design phase; contracting
phase; and construction phase. The TPCM mechanism has helped to reduce the number of projects
31
that go significantly over budget. In 1995 9.2% of TPCM projects experienced cost increases of
more than 20%; in 2004 this was only 3.7%.
99. In Chile, executing agencies are responsible for keeping information up to date on
the BIP on the technical and financial implementation of each project. A recent review of
implementation experience on a significant portion of the public investment portfolio
found that, while timelines where overrun in 28% of projects, actual project costs were 5%
lower than the originally approved cost.
100. In Ireland, detailed guidelines on project implementation have been issued by DoF,
and a new evaluation unit has been set up within DoF to oversee VFM and policy reviews
that are required of implementing agencies, and to conduct spot checks of compliance. Line
ministries on occasions engage external audits of compliance with DoF guidelines and of
project progress by agencies under their supervision.
101. Other advanced, and emerging and developing economies: while project implementation varies
considerably across these countries, in general the elements of a sound framework have been put in
place. Gaps remain, however, and implementation of the framework requires considerable
strengthening. In most cases procurement laws have been modernized (e.g. in all 6 Balkans
countries), and allow multi-year contracting, but weaknesses in project planning, capacity limitations,
and weak governance environments result in frequent instances of poor practice. Monitoring of
project implementation tends to be passive and to focus on spending against the annual project
appropriations. Modern treasury systems are in place in some countries, but are not always being
used to potential for project management. There are frequent contract variations, project accounting
and reporting need to be strengthened e.g. in the new member states of the EU there is no overall
accounting for the whole cost of the project and therefore no basis for monitoring and controlling
total project costs. The internal audit function is at an early stage of development and focuses on
transaction compliance rather than systemic risk management.
102. In Brazil, major efforts have been put into monitoring the implementation of a portfolio of
priority projects in order to speed execution and attract private investment. Since 1996 successive
priority portfolios of projects have been selected and submitted to enhanced project management
methodologies and intensive monitoring, along with committed funding. The current version is the
32
PAC, which comprises over 2,200 projects. Monitoring is lead by the Chief of Staff Office within
the President’s Office, and also involves the ministries of Planning and Finance, and line ministries.
Significant efforts have also been put into procurement reform. The SAI in Brazil considers the
main causes of unfinished projects to be lack of prioritization, excessive politicization, and
unpredictability of budget funding.
103. For instance, in Sierra Leone there are a number of serious weaknesses in project
implementation. Cash shortages during budget implementation are a major problem, due to the
unpredictability both of donor financing (especially budget support) and domestic revenues (see
Chapter 18 for details).19 There is concern that incentive effects (such as retention of salaries that are
many multiples higher than could be obtained in other employment) may result in unnecessary delay
by PIUs in completing projects, or the continuation of a “project” in the development budget even
after a project has been physically completed. The number of PIUs is being rationalized, and some
donors have decided not to establish any new ones. Different donors have their own project
implementation manuals, impose their own procurement standards and regulations, require technical
supervision and sign-off by foreign consultants rather than the Ministry of Works, and require
reporting in their own formats. The cumulative impact of this is there are no national (or MDA-
specific) project implementation manuals, no standardized reporting templates, and the ability of the
National Public Procurement Authority (NPPA) and the Ministry of Works to fulfill their respective
procurement and technical functions is compromised. A Procurement Law was implemented in
2004, and the NPPA (established in 2007) is helping to build capacity in MDAs, but much remains
to be done.
104. Fragile/post-conflict states: in many respects, circumstances in these states resemble those in aid-
dependent states, but there is considerable variation, in part depending on whether the state has
natural resource revenues and how these have been managed.
105. For instance, East Timor has recently built up a sizeable Petroleum Fund that has smoothed
revenue flows to the budget. Timor therefore has sizeable and reliable domestic revenues, and
project implementation has not been interrupted by revenue volatility or unexpected within-year
cash shortages. Angola, on the other hand, despite burgeoning oil and gas revenues and a Petroleum
19 Eifert and Gelb 2005 found, in a cross-country study, that budget support is the most volatile component of donor financing.
33
Fund, has experienced major interruptions to project implementation due to the low balance in the
Petroleum Fund. Large arrears have been run up on the capital budget.
106. In Bosnia and Kosovo, reasonable progress has been made on procurement reform. In
Bosnia, a new law is in place, an independent Public Procurement Authority has been established,
and a training program for procurement officers is underway. Kosovo has also modernized its
procurement law, and around 77% of procurements are by open tender. Implementation is a
problem, however, partly due to weak project planning, and to corruption.
107. Natural resource-dependent states: Weak implementation capacity, including procurement and
project management (coupled with poor planning) results in systematic under-spending of the
investment budget. Except where a revenue stabilization fund is effective (in East Timor), there are
short term pressures to scale up or cut back investment spending (the latter leading to large
expenditure arrears in Angola). Multiple modes of project implementation are the norm, including
variously public infrastructure investment by a National Resource Company, infrastructure spending
by international resource companies, and bundled “resources for infrastructure” projects involving
non-traditional donors.
108. In East Timor, in sharp contrast to many resource-dependent states, the national budget is
the main mechanism through which PIM is effected, capturing the great majority of spending on
public investment in the country. All government capital spending is appropriated in the central
government’s budget and spent by line ministries or autonomous agencies; from 2009, most donor-
funded projects (aside from in-kind assistance) in which the government is a partner are shown in
the government’s budget; the Petroleum Fund is fully integrated into the Budget; there is no
National Oil Company to conduct off-budget public investment; government commercial activities
remain un-corporatized and are included in the government’s budget; and there are no bundled
“resources for infrastructure” projects or PPPs.
109. Under-execution of the investment budget is a common problem in these countries,
particularly when public investment is being scaled up. For instance, in Mongolia the development
budget execution rate is less than 50%. In an effort to speed up project implementation,
procurement regulations have been relaxed in Mongolia and East Timor, but this has introduced the
risk of further irregularities in procurement.
34
6. Project adjustment
110. Advanced PIM systems: a highly distinctive feature of this group of countries is that specific
mechanisms are in place to trigger a review of a project’s continued justification if there are material
changes to project costs or expected benefits.
111. For example, in Chile, where the lowest tender for a project is 10% or more above the price
estimated at the time of project approval, the project is subjected to a reappraisal. If the government
wants to change the design during implementation in a way that increases costs, the project has to be
reappraised. In Korea, there are two mechanisms to trigger a reappraisal of a project’s continued
viability. First is a Re-assessment Study of Feasibility (RSF), which is conducted inter alia when total
project cost has increased by more than 20% (excluding inflationary effects and increase in land
acquisition costs) over the cost confirmed at the previous phase of the project. The focus of the RSF
is to find alternatives to cut down the size and cost of a project, with termination a possibility.
Between 2003 and 2007 five out of sixty one projects subjected to RSF were terminated. Secondly, a
Re-assessment of Demand Forecast (RDF) is conducted whenever substantial changes have
occurred in demand forecasts.
112. In Ireland, termination of a project is rare, but systems are in place to do so if necessary. In
the UK, the Gateway Process requires that the case for project continuation has to be made at
successive stages throughout the life of the project, including at the stage of readiness for service. In
addition, the internal audit role includes bringing projected cost increases to the attention of
Ministers.
113. All other countries: without exception, these countries lack any formal mechanism or trigger to
bring about a review of a project’s continued rationale or justification when costs or timelines have
escalated significantly, or when expected benefits have fallen.
7. Facility operation
114. This is a difficult feature to evaluate because the researchers treated it somewhat differently
across the case studies – reflecting the fact that, in the original formulation the framework focused
this step in the cycle on the existence and quality of asset registers (or in more advanced settings,
whether there are accrual balance sheets). [Other features that are now in the Rajaram et al
framework – such as whether the completed assets are “fit for purpose,” and the adequacy of
35
operations and maintenance funding – have only been discussed in a small number of the case
studies.]
115. Advanced systems: in the UK, Gateway 4 is a review of the readiness for service.
Comprehensive and reliable asset registers are maintained and are subject to external audit. Similarly
in Chile, there is systematic recording and checking of completed capital assets, and a register
records the name of the responsible official for each asset.
116. Other advanced, and emerging and developing economies: in these countries there is a wide range of
experiences with respect to the maintenance of asset registers. In Brazil, while the quality and
coverage of asset registers is improving, there is still some way to go before these are complete and
reliable. In the western Balkans, asset registers range from non-existent (Albania and Montenegro)
to partial coverage (Macedonia and Serbia). In Vietnam, the Budget regulation requires that vehicles,
equipment, buildings and other assets are subject to registration and monitoring, but implementation
of this is limited.
117. As noted, few of the case studies touched on other elements of facility operation. One of the
few that did was the Vietnam assessment, which found that operations and maintenance was under-
funded, and that completed projects were not always fit for purpose due to ancillary investments not
being included within the scope of the project e.g. a new bridge is not useable because the necessary
access roads have not been built. There is also evidence of a number of “white elephant” projects.
118. Aid-dependent states: formal hand-over procedures on completion of donor projects are
typically in place, but asset registration systems are inadequate. Operations and maintenance is
under-funded, in part due to weak integration of recurrent costs of donor projects in fiscal forecasts
and the budget.
119. In Sierra Leone, there is anecdotal evidence of numerous abandoned projects. Reasons given
in explanation include changing political priorities of a new Minister, interruptions to funding, and
corruption. This suggests serious weaknesses in project reporting and monitoring, and a lack of
mechanisms to adjust project funding during implementation. In Kosovo, project completion
reports check work against project specifications. Maintenance of asset registers varies considerably
across MDAs.
36
8. PostProject Review
120. Evaluation, as generally defined technically, refers to assessing the impacts of a project’s
outputs on outcomes. While the technical feasibility of this varies from sector to sector, in general
this is extremely demanding in countries outside those with the most advanced public management
systems. From the PIM case studies, even two of the outstanding performers, Chile and Korea, fall
short against such a standard. In this study, therefore, evaluation (or post project review) is defined
as comprising two dimensions:
1) Basic post-project review. This would comprise, for all projects, a review by the responsible MDA sometime after project completion of whether the project was completed within the original (and amended) budget and time frame, and whether the outputs were delivered as specified.20 As a supplement to this basic element, whether investment/development projects are subject to compliance and VFM audit by the Supreme Audit Institution should also be assessed.
2) Project evaluation, to be completed 2-3 years or more after project completion, which would attempt to identify what changes in outcomes are attributable to the project, and estimate CBA ex post. This should be completed on a highly selective basis. As a supplement, whether investment/development projects are subject to VFM audit by the Supreme Audit Institution should also be assessed.
121. Advanced systems: a distinctive feature of the four countries with the most advanced systems
is that they put significant effort into ex post review. However, in Chile and Korea, as noted, at this
stage this is still largely confined to reviewing output delivery against project plans. In Ireland and
the UK, however, significant efforts are made, in addition, to selective evaluation of the impacts of
investment projects on outcomes.
122. In Chile, a sample of projects completed over the previous two years is subjected to basic
evaluation of outputs, and the results are sent to the National Congress. The sample consists of
about 8-10% of all projects financed by the Regional Development Fund, and enables conclusions
to be drawn with a 90 to 95% probability. A second stage review is also undertaken on selected
projects where data is available, at least 5 years after completion, to review projected benefits. In
Korea, in principle each MDA is responsible for a 3-tier system of project review. In practice,
however, MDAs do not support the system, and the ability to implement it is also hampered by
20 The time period after completion and before post-project review should be long enough for all expenditures to be booked, perhaps including any final contract retentions. Such a basic review might also include identification of the main causes of deviation from plan, and any general lessons for project preparation and implementation.
37
weak specification of performance targets in the original project documents. In the UK and Ireland,
special reviews are commissioned to identify systemic factors affecting project cost or quality.
123. In Ireland there is a framework for evaluation of National Development Plan programs,
including a mid-term review of the NDP. There is also an explicit expectation that project managers
will have incorporated lessons learned from previous projects (both successes and failures). In the
UK, ex post reviews have been built into the Gateway process as a means to promote lesson
learning. These are still relatively new and to date have tended to focus on the economic justification
and assumptions for the project. The National Audit Office undertakes selective VFM analysis of
projects, and its reports are considered by the Public Accounts Committee of Parliament (which by
convention is chaired by a member of the opposition). The PAC at times calls heads of departments
before it to respond to questions and points raised by the NAO. There is generally a high degree of
transparency in the UK and Ireland about project costs and procurement which strengthens
accountability and promotes systemic error correction.
124. Other advanced, and emerging and developing economies: outside the countries with the most
advanced PIM systems, this is one of the weakest links in the PIM cycle – perhaps the weakest.
None of these countries has a systematic approach to basic post-project review, let alone selective
impact evaluation.
125. For instance, in Brazil, there is no proper post-project review to compare costs or benefits
against plan. However, the SAI has a specialized unit that audits a sample of public works every year
to identify risks. It also conducts some VFM audits. Vietnam has pursued cooperation with donors
on project evaluation, but this has yet to bear fruit. The SAI has, however, conducted compliance
audits on public investment projects.
126. In the new member states of the EU post-project reviews are generally not done, and none
of them publish ex post reviews of the original project cost estimates against final total project costs.
Amongst the western Balkans countries, ex post review has not been considered a priority, with the
focus of effort being on improving project preparation. Albania is a partial exception, in that the SAI
audits some projects for basic compliance (but is not yet attempting value for money audits). None
of the other SAIs in the western Balkans is as yet playing a role in auditing investment projects. In
38
the new member states of the EU, the external auditors provide only basic financial oversight of
implementing agencies and projects, not on the effectiveness of project management or VFM.
127. Aid-dependent states: amongst these countries there is typically a reliance on donors to
undertake reviews and evaluations of their projects. Otherwise, there is little or no systematic basic
post-project review, let alone evaluation, and limited use is made of the findings from donor
evaluations to improve future project design and implementation.
128. In Sierra Leone, the government at this stage pays little attention to post-project review,
relying on donor project review mechanisms. While many units in government have the title
“Monitoring and Evaluation” there is in fact virtually no domestic evaluation taking place. In
Kosovo there are no standard procedures for project evaluation and auditing, although the SAI has
audited capital spending in some MDAs, and has a mandate to conduct performance audits but does
not as yet have the capacity to do so.
129. Fragile/post-conflict states: the situation is typically at least as constrained in these countries
as in donor-dependent states. They may even lack a functioning SAI to conduct basic compliance
audits of investment projects.
130. For instance, in East Timor there is little or no basic post-project review, although a recently
established expenditure review division in the MOF may start to do some analysis of investment
project outputs. Timor does not have an SAI, the independent audit function being carried out by an
Australian audit company that makes short visits to Dili. This severely limits the capacity to conduct
compliance audits of investment projects.
131. Natural resource-dependent states: there is typically little or no systematic basic post-project
review or evaluation in these countries, due both to lack of capacity, and perhaps a lack of demand
from decision makers for feedback to improve PIM system performance. A limited exception is
Mongolia, where the SAI publishes reports on its audits of investment projects.
5. Some Lessons for PIM System Reform and Areas for Further Work
132. This concluding section discusses approaches to the reform of PIM systems. The first part
contains some general comments on the nature of PIM system reform. This is followed by a
39
discussion of some of the approaches to reform that are evident from the experience of case study
countries, and promising points of entry for external engagement in PIM reform. The section
concludes with suggested areas for further research.
PIM Reform in General
133. The way in which the PIM system functions, and the prospects for reform, will be
conditioned by the broader public management environment in which the system is nested. For
instance, the role and influence of the ministry of finance within government varies widely across
countries, affecting its ability to act as a “gate-keeper” for the quality of new project entering the
budget, and its ability to prompt remedial action during project implementation. The culture of civil
servant/elected official relationships influences the ability of civil servants to engage in direct
discussion with political leaders on the technical merits of individual projects. The broader civil
service culture, such as whether there is a disregard for compliance with the law, or a legalistic
culture that encourages formal adherence to rules at the expense of results, also impacts on the PIM
system (see discussion of procurement in chapter 5). The civil service remuneration system will
influence the ability to attract and retain specialized skills required for PIM, or the prospects for PIU
reform in donor-dependent settings. Finally, the wider PFM system in which the PIM system is
nested also influences the feasibility of PIM reforms. The general quality of budgeting, cash
planning, accounting and reporting, treasury systems, the internal control environment and internal
and external audit capacity, all have a major impact on the functioning of the PIM system.
134. PIM reforms face three particular challenges. First, public investment tends to be highly
politicized. Compared to recurrent spending, decisions on individual projects are discrete and
relatively discretionary. This, together with the location-specific and long-lived nature of projects,
makes public investment an attractive means for politicians to visibly and credibly claim that they
have delivered benefits to particular constituents. Any consideration of PIM system reform must
therefore recognize that the current pattern of investments, and the way in which the PIM system
functions is likely, at least to some extent, to reflect the interests of key decision makers. It is
therefore important to assess the extent to which political leaders are motivated to improve the
functioning of the PIM system, where a push for reform might come from, and which elements of
reform are more likely to be supported, tolerated or vigorously opposed.
40
135. Secondly, public investment is an area of high corruption risk.21 The opportunities for
private financial gain can distort investment choices and project implementation decisions. Reforms
are likely to upset the current distribution of illicit gains, although some reforms (e.g. procurement)
are likely to do so more than others.
136. Thirdly, PIM is an extremely demanding area of public management. Even the basic “must-
have” functionality requires governments to possess an array of specialized technical, financial, and
public policy analytical capacity; multi-year budgeting, financial management and project
management systems; staff capacity in project management, procurement, and audit; soundly
designed institutions and effective inter-agency coordination and cooperation (within central
government, within the public sector, with the legislature, and with sub-national governments); and
sufficient political coherence to sustain support for projects over the whole project cycle and beyond
through the subsequent period of service delivery.
137. It is important therefore to ensure that PIM system reforms are:
Incentive compatible i.e. that advice recognizes the political incentives facing elected officials, and does not counsel a politically unrealistic policy approach.
Based on a sound understanding of, and tailored to fit individual country trajectories, circumstances and practices.
Technically feasible. To that end the yardstick in the technical analysis is “good enough” practice, not good (or best) international practice for more developed countries.
Carefully designed and sequenced in recognition of implementation capacity and so that to the extent feasible early gains are achieved that help to build support for the reforms.
138. The PIM system comprises two broad sub-systems: project preparation, and project
implementation. A simple typology of PIM system performance, suggested by Brumby, is shown in
Table 2.22 The objective is to move to cell A - in which well-designed projects are well implemented
- in the most cost-effective manner.
21 In Transparency International’s 2008 Bribe Payers Survey, public works contracts and construction was the sector ranked most likely, out of nineteen sectors, to engage in practices of “state capture” (the degree to which firms in a sector use political contributions to achieve undue influence on government policies, laws or regulations). Of relevance to resource-dependent states, the oil and gas sector, and the mining sector, ranked second and third respectively. See Transparency International 2008, pp. 10-11. 22 J. Brumby, Concept Review of the Mongolia Policy Note on Public Investment Planning, November 19, 2009.
41
Table 2: A Simple Typology of PIM-System Performance
Well executed Poorly executed
Good projects A C
Poor projects B D
139. Reducing the number of poor projects selected for funding often appeals as an obvious
starting point in PIM reform, but involves a time lag before the benefits are felt while new projects
move through the PIM system to service delivery. Improving project execution may generate faster
benefits – although only if the projects are not poor projects that generate negative net benefits. A
relevant consideration here is the size of the current stock of projects compared to the annual flow
of new projects entering the PIM system. When spending on public investment is increasing rapidly,
there may be large potential gains from strengthening project planning. Where new spending is low
compared to that on a large stock of already-committed projects, the larger gains may be from
improved implementation.
140. It must be remembered, however, that PIM is a system, and there are inter-dependencies
across the different stages of the investment cycle. Lessons from improvements in project
implementation can feed back into better planning of new projects. For instance, closer monitoring
of project implementation, and/or completion of basic post-project reviews, could potentially
generate useful information on the sources of weakness in project planning. The gains from a one-
off (or regular) basic post-project review of a portfolio of recently completed projects could be a
very cost-effective and rapid way of identifying needed improvements in project planning. This is
part of the PIM reform strategy recommended in the case studies of Brazil and East Timor. On the
other hand, stronger capacity for project appraisal could improve project implementation, by helping
to identify, from amongst the projects that are significantly over-budget or behind schedule, those
that should be terminated because their expected net benefit is now negative (although actually
stopping a project during implementation is acknowledged to be very difficult in any country).
141. The key strategic issue in a PIM assessment is, in fact, to identify which components of the
system are constraining performance at the margin. In any system, some components may be infra-
42
marginal: they are performing at a sub-optimal level – compared to what is realistically achievable -
but they are not impacting on system performance because of more serious weaknesses elsewhere in
the system. The objective should be to identify what are the binding constraints on system
performance now and in the short-to-medium term. For instance, while in many countries project
preparation is weak, it is also the case that in some countries key decision makers are satisfied with
their current informal, non-transparent approach to project selection and there is currently no
effective demand for better project appraisal. Therefore, while a partial analysis may suggest the
need to strengthen project preparation, a system level analysis may in this instance suggest the need
to focus elsewhere in the system (such as improving project execution, or attempting to increase the
demand for better appraisal).
142. Put slightly differently, PIM system reform may be approached by identifying a small
number of fairly substantive changes aimed at making good quality PIM an enduring and sustainable
activity, rather than having a plethora of recommendations across all areas. This was the approach
adopted in the Brazil case study (chapter 9), and that is recommended with respect to procurement
reform (chapter 5).
143. Finally, transparency is likely to be an important element of improved PIM in many
countries, and especially in resource-dependent states where the executive is typically over-
dominant. The case for transparency is supported by analytical perspectives from political
economy, including principal-agent theory, collective action problems, and the literature on credible
commitments. A body of empirical evidence is also accumulating that transparency leads to better
economic and political outcomes.23
144. However, it is important to define the information to be disclosed with specificity, and to
take a broad approach to transparency that incorporates clarity of roles, assurances of integrity, and
opportunities for public participation in monitoring public investment. It is also necessary to pursue
transparency at all stages of the investment cycle if rent seeking and corruption are not simply to
shift elsewhere. Requiring a government to take concrete steps up front, such as instituting regular
publication of ex ante and ex post information on public investment, or establishing independent
23 Humphreys, Sachs and Stiglitz, 2007, pp. 330-331.
43
institutions or empowering civil society, might in some countries be an indication of a government’s
commitment to PIM system reform - although there will be other states where reneging on such
obligations will be foreseen to entail minimal domestic political cost, and where such initiatives
cannot therefore function as credible commitments.
PIM Reforms in the Country Case Studies
145. What do the country studies suggest with respect to reform of PIM systems? It is helpful
again to disaggregate by the different PIM system typologies.
146. Approaches to reform in advanced PIM systems: there has been a trend across these four countries
towards strengthening appraisal, through establishing clear expectations of what is expected of new
project proposals, building capacity across government in the techniques of project appraisal, and
implementing institutional reforms to strengthen the independent review of appraisal. Appraisal
techniques have broadened and deepened over time, with increased attention to proposed project
management arrangements, risk management strategies, and procurement strategies.
147. At the same time, however, attention has been paid to other key components of the PIM
cycle, such as defining clear stages in the project cycle and establishing decision points before
projects can proceed to the next stage; strategic alignment and preliminary screening; fully costed
sector strategies integrated with MTEFs; , multi-annual budgeting and authority to spend;
accounting and reporting to enable a focus on managing the total costs of projects over their life;
and putting sound policy and management frameworks in place to regulate PPPs.
148. It also seems apparent from these four countries that evaluation is perhaps the last step in
the cycle to receive sustained attention. Basic post-project review has been in place in all four
countries for some time, but impact evaluation is still at a relatively early stage even in Chile and
South Korea despite the length of time that PIM reforms have been underway and the extent of
progress achieved in those countries.
149. Approaches to PIM reform in other advanced economies, and emerging and developing economies: again
there is a wide variety of experiences here. Vietnam has modernized much of its legislation relating
to public investment, introducing a new Procurement Law in 2005 and regulations on project
appraisal and monitoring and evaluation (although there are still some gaps and overlaps between
44
the different instruments, and detailed guidelines on implementing them are not in place). There has
been some rationalization of the large number of projects in the PIP. For large-scale projects the
government has on occasion set up an inter-agency appraisal panel with input from non-government
interests. Significant efforts have gone into procurement reform, including publication of a
Procurement Bulletin and setting up a procurement information gateway. Brazil has also put
considerable effort into procurement reform, including the use of electronic tendering. It has also
invested top-level political commitment into speeding the implementation of a large portfolio of
priority projects (the PAC).
150. Amongst the 6 western Balkan countries priority has been given to strengthening project
planning through improved strategic guidance, definition of steps and decision points in project
preparation, better appraisal, and more effective independent review of appraisal. The rationale was
that it is difficult to improve implementation and evaluation of projects that have not been well-
designed in the first place.
151. Approaches to PIM reform in donor-dependent settings: PIM reforms tend to focus on strengthening
project implementation through:
Procurement reform: modernizing procurement laws, establishing new procurement authorities, training procurement officers, and strengthening internal and external audit.24
High-level political impetus to monitor progress in implementing individual projects. This tends to start with an initiative to set up new parallel systems under top-level political direction due to dissatisfaction with current monitoring arrangements, and to focus on execution of the annual budget (rather than total project cost and non-financial indicators of progress).
Rationalizing and reducing the number of PIUs. In some countries decisions have been taken, in conjunction with donors, not to establish new PIUs; not to allocate any new projects to existing PIUs; to combine/rationalize existing ones within a single institution or sector, and sometimes across donors; to attempt to report project spending implemented by PIUs through the national treasury system; to transfer PIU staff to in-line planning departments; and to terminate PIUs and mainstream the functions (although this is problematic given the much higher salaries PIU staff are typically paid in comparison to regular civil servants carrying out the same functions in in-line positions).
152. In addition, a number of donor-dependent countries have put effort into improving the
quality of PRSPs. There is also a trend to integrating responsibilities for public investment planning
and budgeting inside the ministry of finance e.g. all six western Balkan countries, Sierra Leone. The
PIP as a tool has been sidelined with the development of MTFFs in many of these countries, and in
24 These are the four standard recommendations in Country Procurement Assessment Reports – see chapter 5.
45
one or two cases e.g. Albania the PIP has been terminated. In other cases, however, the PIP remains
an important tool (e.g. Uganda), or one that is being reintroduced (e.g. Sierra Leone).
153. Approaches to PIM reform in fragile settings – in these situations donors are dominant in the early
post-conflict years, and the focus of effort is on re-building a consensus around national priorities
and developing strategies for public investment (e.g. PRSPs in Sierra Leone and Timor); establishing
mechanisms within the government to coordinate donor financing (all countries); and procurement
reform. In those fragile states that are natural resource dependent, attention then tends to turn to
speeding up project implementation through closer monitoring of projects e.g. Angola, East Timor.
154. Approaches to PIM reform in new and aspiring EU members: in these countries effort has been put
into a number of different parts of the project cycle, such as an emphasis on national strategy
documents, the application of CBA, procurement reform, and internal and external audit. However,
there is a clear tendency for all these efforts to be focused on projects implemented with EU
funding, and for domestically-financed investment to lag considerably behind.
155. From these experiences it is possible to draw out some common stylized approaches to PIM
system reform, as described in Box 6.
46
Box 6: Stylized Approaches to PIM System Reform
“Implementation first”: the focus here is on improving chronic poor project execution rates, on the presumption that poor project planning is not the main problem, or ex ante appraisal is limited in what it can achieve, or project selection is highly politicized, or that the fastest gains can be made at the implementation stage of the cycle. Brazil exemplifies this approach – described in the case study as one of “more projects, less appraisal” - but it is also a feature of countries where there is a lack of demand from decision makers for better project appraisal (e.g. resource-dependent states such as Angola and East Timor), and in donor-dependent states where there is a reliance on donors to conduct appraisal, and domestically-financed investment is small or escapes attention. “Better planning first”: the key to better projects is seen as improved strategic planning and preliminary screening, and higher quality appraisal, on the basis that it is hard to improve or stop a poor project once it has been approved. Chile and Korea exemplify this approach, but it is also evident in the Western Balkans. Korea in particular exemplifies the strategy of preventing low quality projects at the pre-feasibility study stage before they are launched, with a very high rejection rate of 44% (chapter 7). “Center of excellence”: a strategy of building analytical capacity on project appraisal in one center in government, usually the MoF or the Planning Ministry but possibly a key sector ministry. The center of excellence strengthens appraisal through setting clear expectations, issuing guidelines on implementing them, and then reviewing the quality of projects submitted. It also involves helping to build capacity in MDAs through staff training. Examples include Chile, Ireland, and South Korea. “Center of power”: dissatisfaction with chronic under-execution of the investment budget results in leaders of government setting up a project monitoring function reporting directly to them. Examples include the intensive monitoring under the Chief of Staff Office in the President’s Office in Brazil, and the Strategic Policy Unit in the Office of the President in Sierra Leone. In Vietnam the Prime Minister on occasions sets up a temporary Appraisal Council to appraise a specific project. “Legal change first”: a strategy of changing the law to introduce new requirements, and then working on implementing them (e.g. the Bank’s recommended approach in Sierra Leone to strengthen project appraisal, see chapter 18) – rather than working to improve system functioning and amending the legal framework in parallel (as pursued in procurement reform in Belize, see chapter 5). “Contracting out”: a strategy of selectively contracting for the services of non-government suppliers in project appraisal and/or project management in order to supplement capacity, but sometimes also to introduce an element of independence and credibility. There are numerous examples in the case study countries. From outside these countries, Algeria provides an example of large scale contracting out of the project appraisal function to an international consulting firm. “By-pass the system”: there are public investment modalities that effectively by-pass the whole mainstream PIM system. These include PPPs, and resources for infrastructure contracts. The motivation may be to seek efficiency gains from private sector involvement or to create some contestability for a chronically weak PIM system. On the other hand, the objective may be to escape the usual scrutiny, shift spending off-budget, and/or take advantage of corruption opportunities. This “by-pass” strategy seems to be growing rapidly, with PPPs being implemented or investigated in virtually all the study countries – despite the common lack of a policy or management framework - and growing use of resources for infrastructure deals in resource-dependent states (typified by the Democratic Republic of Congo). “Transparency”: In Chile the PIP is available to the public on-line, and the public can query the website on the full list of current and past projects. In Ireland many projects must go through a public hearing prior to approval. In East Timor, there is a high degree of transparency around flows of oil and gas revenues into the Petroleum Fund, and the method of calculation of annual withdrawals from the Fund to finance public spending, with quite detailed information on investment projects in the annual budget, and NGOs are actively monitoring selected projects; NGOs are actively monitoring the appraisal and implementation of selected projects. In Brazil 75% of procurement contracts by value are let by electronic tendering; and civil society concern over unfinished projects lead to the establishment of a specialized unit in the SAI to audit projects. In Vietnam local communities participate in discussions on prioritization, implementation supervision, and post-completion operation of smaller-scale projects.
47
156. It is also instructive to identify the key challenges in PIM system reform across the different
country typologies, as set out in Table 3.
48
Table 3: Table 3: Main PIM System Challenges by System Typology Advanced PIM systems Total quality management over the complete PIM cycle, with review at
specific decision points. Efficient allocation of risk between government and contractors. Rigorous appraisal of PPPs against standard modes. Strengthening impact evaluation.
Donor-dependent settings Preliminary screening of projects against comprehensive, authoritative and costed national and sector strategies that are integrated with the budget.
Building capacity for project appraisal, and defining steps in project preparation.
Consistency between development and recurrent budgets. Strengthening CFA gate-keeping (including project database). Better insulation of investment spending from donor funding volatility. More effective project accounting, reporting and monitoring. Strengthening implementation of procurement. Rationalizing and mainstreaming PIUs. Basic post-project review, and effective compliance audits of projects by
SAI. Resource-dependent settings Comprehensive national and sector strategies applying across all public
investment modalities. Building demand for project appraisal. Building capacity for project appraisal, and defining steps in project
preparation. Consistency between development and recurrent budgets. Strengthening CFA gate-keeping (including project database). Better insulation of investment spending from revenue volatility. Sound frameworks for appraisal of non-standard modes of investment. Strengthening implementation of procurement. Basic post-project review, and effective compliance audits of projects by
SAI. Transparency of public investment across PIM cycle.
Post-conflict/Fragile settings Re-establishing national consensus over public investment priorities. Donor-coordination, comprehensive project database. Building capacity for project appraisal. Consistency between development and recurrent budgets. Basic project accounting, reporting and monitoring. Procurement reform. Basic post-project review. Transparency of public investment across PIM cycle.
EU-accession settings National and sector strategies that reflect both domestic and EU-financing.
More effective appraisal and independent review of non-EU funded projects.
More effective oversight of implementation of non-EU funded projects. More cost-effective procurement.
157. It is difficult, however, to gain a sense of the trajectory of reforms from the descriptions
across country typology, or from the stylized characterizations of generic approaches to or
challenges in PIM reform. What is needed is a focus at the individual country level. Table 4
49
summarizes the key elements of some of the notable reform experiences from the case study
countries.
50
Table 4: Notable PIM Reforms Country Distinctive Elements of Reform
Chile Effective preliminary screening of projects for strategic alignment. Long period of broad capacity-building in project appraisal across government. Comprehensive framework for PPPs. Mechanisms to trigger a review of projects’ continued justification during
implementation; South Korea Total Project Cost Management (TPCM) system introduced in 1994.
Planning and Budget Ministries merged in 1999. In 1999 Cross-Ministerial Task Force on PIM designs Action Plan to strengthen PIM:
o Preliminary feasibility study introduced, conducted by new unit in KDI, a semi-autonomous institution;
o TPCM system strengthened by addition of RSF mechanism; o Ministry of Land Transport introduces ex post performance evaluation system.
In 2006 further strengthening of TPCM by addition of RDF, and legal framework for PIM strengthened in National Finance Act.
Ireland Post EU-accession, development of comprehensive national and sector strategies. Medium term investment envelopes by MDA. Strengthening of CFA for both project appraisal and monitoring of implementation. Capacity building across government in appraisal. Use of independent institutions, transparency, and external contracting. Selective impact evaluation.
UK-England Continual strengthening of an already advanced system through: o More operationally effective sector strategies. o Medium term investment envelopes by MDA. o Introduction of Gateway Review Process, managed by new independent entity
within Treasury. o Comprehensive framework for PPPs. o Selective impact evaluation.
Albania Coherent, realistic and authoritative national strategy, supported by sector strategies, linked to the budget.
Introduction of systematic PIM procedures linking the project cycle to the MTEF/annual budget process.
Strengthening of appraisal including defining steps in project preparation. MOF playing a more active gate-keeping role.
Brazil Succession of high-level political initiatives to institutionalize regular and routine monitoring and active intervention to speed up implementation of portfolio of high priority projects.
Vietnam [to come – need to draw on the WB PIM project documents, rather than the case study]
Uganda Over the last decade introduced: o National and sector plans. o MTEF. o Consistency between development and current budgets. o IFMIS. o Internal control and internal audit.[do the last two relate to PIM, or to PFM more
generally?] East Timor Advanced level of transparency introduced with respect to:
o The relationship between the Petroleum Fund and the annual budget. o No off-budget public investment (e.g. by a National Oil Company, or resources
for infrastructure projects). o Most donor-funded projects in which the government is a partner are shown in
the budget.
51
158. What is generally lacking in Table 4 is a sense of the dynamics of reform, the packaging,
sequencing and interactions between different elements across the PIM cycle, and the motivation for
and drivers of reforms. Unfortunately the first generation case studies did not provide much material
on reform dynamics or political economy, although the TOR for the second generation case studies
did introduce these elements.
159. One partial exception is the South Korea case study, which provided some description and
analysis of the PIM reforms over time (chapter 7). In Korea the 1997 crisis prompted reforms to
address the fundamental causes of the recession. In the public and fiscal sectors this entailed a more
market-oriented approach and managerial strategies to increase efficiency and transparency. To
strengthen PIM, the government organized a Cross-ministerial Task Force to develop an action
plan. The task force was jointly headed by the Ministry of Planning and Budget (formed in 1999 by
the merger of the Board of Planning and Budget and the Office of National Budget) and the
Ministry of Construction and Transport. In July 1999 the Task Force issued a Comprehensive Plan
to Enhance Efficiency of Public Investment. A key element of this was the strengthening of project
appraisal, due to distrust of the quality of feasibility studies being prepared by line ministries. The
MOSF initially attempted to take over responsibility for feasibility studies, but the line ministries
resisted. The compromise was to introduce a new system of preliminary feasibility studies under the
control of MOSF, but conducted by PIMA in KDI, a semi-autonomous entity. The MOSF played a
leading role in strengthening appraisal and evaluation, the TPCM system, and the MTEF.
160. The TPCM system was introduced in 1994, and MOSF has been revising the “Guidelines for
Total Project Cost Management” on an annual basis. TPCM includes all costs accrued over the life
of the project, regardless of the source of funding, and applies to a series of defined stages viz.
project conception; PFS and feasibility study; draft design phase; blueprint design phase; contracting
phase; and construction phase. There is also a Reassessment Study of Feasibility (RFS) intended to
prevent unnecessary cost increases by focusing a decision on whether or not to continue a project,
and in 2006 a Re-Assessment of Demand (RDF) was introduced to verify that the demand forecasts
for a project’s outputs remain valid.
161. [add findings here on reform dynamics from the second generation studies as available]
52
162. In addition, chapter 5 on procurement contains an extended discussion of the drivers of
procurement reform, both general drivers and country-level drivers. These are summarized in Box 7.
Box 7: The Main Drivers of Procurement Reforms
Sanchez identifies a number of important general developments over the last twenty years that have motivated
widespread reform of government procurement systems, including:
The creation of the EU that stimulated alignment of national procurement systems.
The 1994 UNCITRAL Model Law on Procurement .
The emergence of new tools for electronic procurement in the early 1990s.
Debt relief, HIPC, and the advent of PRSPs in the late 1990s, combined with IFI leverage, stimulated procurement reforms.
Standardized tools to assess country procurement systems e.g. CPAR, OECD/DAC Benchmarking
At the individual country level, Sanchez identifies the following drivers of reform:
The collapse of the old system e.g. Ghana.
Post-conflict reconstruction e.g. Mozambique
External pressure from IFIs e.g. Uganda
International economic integration e.g. harmonization of procurement laws in common markets in west and central Africa; Chile’s procurement reforms to facilitate Free Trade Agreements.
A general shift to greater transparency and accountability in the transition from authoritarian rule e.g. Chile and Peru
Demand for good governance and reduced corruption e.g. from Transparency International
163. Possible entry points for external engagement.
The following points in the investment cycle appear to offer promise as general points of entry for
external engagement with PIM system reform – although individual country circumstances will
always trump general considerations such as these.
Strategic alignment and preliminary screening: devising more operationally relevant national strategies supported by costed sector strategies (integrated with MTFFs where they exist) could help to prioritize the project portfolio and screen low value projects. It is generally considered less difficult to halt a poorly conceived project early in its life, before much political or bureaucratic capital is tied up in it.
53
Strengthening appraisal and gate-keeping: defining clear steps in the project development process, linking key steps to clear decision points, setting out clear expectations for project appraisal with guidelines for implementation, and building appraisal capacity across key institutions, offer promise as means to introduce more discipline to project selection. A “center of excellence” strategy may be appropriate in countries where initial capacity is weak, involving strengthening the CFA and/or ministry of planning as a gate-keeper, supplier of independent review of appraisal, and leader of capacity-building in key ministries.25
Setting up a comprehensive public investment project database as a management tool to generate basic information on the performance of individual projects and the overall portfolio.
Strengthening procurement by moving beyond first generation reforms - that focused on inputs such as regulations, institutions, and training - to focus on transforming culture and behavior and creating an enabling environment for all branches of government, the private sector and civil society to increase the demand for better procurement. This requires developing new tools to measure outcomes such as the efficiency and effectiveness of procurement systems (see chapter 5).
Independent physical quality inspections and audits of construction in infrastructure, combined with “social audits” by civil society.
The widespread interest in PPPs, and Resources-for-Infrastructure deals, combined with the common lack of a policy and management framework for them, provides an opportunity to engage, although there may be a lack of interest on the part of decision makers.
Project accounting and reporting – the ability to account for total expenditures on a project over the life of the project is a basic pre-condition for managing the total costs of projects. In a number of the case study countries this basic functionality is not present, for example because the accounting system only records expenditure against separate contracts issued, but does not link expenditures on different contracts (e.g. for the design of a project, for its construction, for procuring equipment) under the common project to which they relate.
Introducing basic post-project review: the case studies found that even basic post-project review, completed on a systematic basis, is absent in nearly all countries. Given that such reviews are not technically demanding, this could be a key entry point for PIM system reform in many countries. This is the approach adopted in Uganda – although admittedly there has been little response to date to the reports generated by the MoF. It is also the approach recommended in the Brazil and Timor case studies.
Similarly, a one-off compliance audit of a sample of investment projects by the SAI, or a regular review, might be a useful entry point for PIM reform in many countries. In countries where external audit capacity is somewhat more advanced, selected VFM audits of procurement, and risk identification and mitigation, could be useful general approaches.
Assessment of service delivery, including the extent to which completed assets are fit for purpose, and the adequacy of operations and maintenance funding.
164. There are also some general political economy situations that offer promise as potential
points of entry:
25 Alternatively, where the CFA is weak or lacks motivation, a key line ministry could be strengthened as an initial centre of excellence.
54
A post-conflict situation, when the political situation is unusually fluid. Collier (2007) identifies this as an opportunity for the international community to provide very large but time-limited TA inputs.
Early in the natural resource cycle in a resource dependent state, when the allocation of rents has not yet ossified. East Timor provides an example of the sort of advanced transparency around natural resource exploitation and fiscal policy that is possible early in the resource cycle.
A change of leader or government.
The presence of an influential PIM reform champion within the government (or a line minister who wants to strengthen project appraisal of projects put forward by her ministry).
Large scaling up of public investment, when decision makers may be focused on getting better value from public spending.
Dissatisfaction amongst key stakeholders with the current performance of the PIM system, particularly around chronic under-execution of the investment budget.
Active interest from the legislature and/or civil society interests with knowledge of PIM, concerns about current system performance, and a favorable enabling environment.
Suggested Issues for Future Research
165. The analytical framework, case studies, and other inputs that form the basis of this Volume
represent an initial package of efforts intended to support better management of public investment
in the Bank’s member countries. Of necessity these efforts have focused more on some aspects of
PIM than on others e.g. they have focused on central government, rather than on general
government or the wider public sector. They have taken some assumptions as starting points that
merit further testing – for example, that the quality of the processes that constitute the PIM system,
as set out in the Rajaram et al framework, are systematically associated with the quality of the actual
public investments that result.
166. It is anticipated that this initial PIM work program will be supplemented by substantial
further efforts by the Bank to work with individual countries, to conduct research and analysis, and
to further develop the framework in specific areas. This chapter therefore concludes with a short list
of the potential areas for further work on PIM reform.
1. How good a proxy for the quality of public investment (in terms of economic rates of return and other criteria), are assessments of the quality of the public investment management system of the kind set out in this Volume? Is there a sub-set(s) of elements that is particularly important in different country settings in terms of the ultimate developmental impact of public investment? How does this sub-set vary across country type and country circumstances?
2. What is the impact on PIM system functioning and PIM reforms of the broader environment in which PIM systems operate, such as the quality of public financial management, civil service traditions, or the complexity of inter-governmental fiscal relations? Are there minimum PFM capabilities that need to be in place to support viable PIM reforms at different parts of the PIM cycle?
55
3. How should the Rajaram et al framework be adapted to incorporate public investment by state-owned enterprises, and by sub-national governments?
4. What does the evidence suggest about the effectiveness of alternative high-level PIM reform strategies in different settings? What have the motivations for PIM reform been in practice, what are the drivers of reform? How do the motivations map onto the differential success of PIM reforms?
5. What has the experience been with different types of institutional reform as part of PIM reform, across different country settings?
6. What appraisal techniques are being used by countries at different levels of development and in different circumstances? What is the quality of project appraisal across these countries? How does the quality of appraisal relate to evidence on the ex post quality of projects actually implemented and completed?
7. What patterns are there in practice in country efforts to strengthen appraisal, and what factors appear to be associated with successful and unsuccessful reforms? What is the political economy of attempts to strengthen appraisal, including the areas where or circumstances under which demand from key decision makers for better project appraisal is more likely, and may overlap with what is technically feasible?
8. What has been the impact of country procurement reforms on the efficiency of procurement systems? Do wider PFM and/or civil service reform or legislature/civil society empowerment contribute to successful PIM reform?
9. What has been the impact on PIM systems of attempts to either cut back public investment spending or rapidly expand it in the context of the current international financial crisis? What factors appear to be associated with successful and unsuccessful responses of the PIM system to such demands?
10. What potential is there for regional approaches to strengthening PIM outside the EU “zone’’? 11. What are the most effective checks and balances on public investment in different country
settings? What has been the experience with attempts to build demand for higher quality PIM at different points of the PIM cycle?
1
Annex: Table 5: Summary of Key Features from Case Studies by PIM Functionality
Country Guidance/ screening
Appraisal Independent review
Selection/ budgeting
Implementation Adjustment Operation Evaluation
Mongoliai (2009)
Too many strategy docs. Weak strategic alignment New national planning body recently set up (NDIC)
Very weak for domestic; reliance on donors. Some failed PPPs (no law)
Very little capacity in MoF
Cabinet selects; Parliament. inserts many projects. Project aid off-budget and outside PIP; weak PIP and MTBF.
Corruption in procurement, and regulations weakened to speed implementation. Budget execution rate < 50%. Weak inspection, reporting and monitoring
No explicit provisions
Maintenance under-funded
Recent SAI report
Vietnam (2008)
Strategic docs, but list of projects too large, and do not guide actual project selection.
Weak for domestic, except some large projects; reliance on donors; PPPs appraised
MPI lacks authority
Weak coordination MoF/MPI; fixed 5 year PIP
Procurement problems (corruption). Lack of guidelines. Weak reporting on progress. Decentralization but lack of capacity.
Completed assets not always fit for purpose due to lack of ancillary investments. Inadequate O&M
Korea (2008)
CBA/ policy appraisal for most large projects by PIMAC, independent of project sponsor.
Yes. 1999-2007 44% of projects put forward by MDAs rejected.
Only projects technically appraised and accepted can be funded.
Total Project Cost Management System (TPCM) controls all costs over life of project.
RSF and RDF triggered by changes in TPCM or demand. ii
Each MDA responsible for 3-tier system of review. Weak (MDAs do not support , poor specification of performance targets).
Country Guidance/ screening
Appraisal Independent review
Selection/ budgeting
Implementation Adjustment Operation Evaluation
Brazil (2009)
Room to strengthen strategic guidance; no
No formal appraisal for vast majority of projects;
Virtually none.
No effective gate-keeping; no pipeline of approved
Intensive monitoring of portfolio of priority projects, led by President’s Office, with
Lack of formal procedures or trigger to review project
Quality and coverage of asset registers improving, but
No proper ex post evaluation to compare
2
formal preliminary screening.
priority projects exempted need for pre-feasibility study.
projects; Parliament cuts current spending and adds significant new projects.
active intervention. Slow implementation, reflecting problems in procurement, poor technical specification.
rationale. long way still to go.
costs/ benefits against plan.
Balkans (2009)iii
One of weakest links in PIM systems (Albania aside). Lack of credible/ operational national strategies, important gaps in sector strategies.
Albania and Kosovo aside, some systematic project preparation procedures but quality poor, focus on engineering v economic. Binding constraint is demand side.
None, aside from Serbia’s Revision Com-missions.
MoFs not credible
gatekeepers. Politicized
project selection. Albania aside,
misalignment of budget, MTEF
(and in Bosnia PIP).
Weak capital/ current
integration. No “ready-to-go”
checks. Excessive virement/
supplementary budgets.
All countries have progressively harmonized procurement laws with EU, majority of proc. now open and competitive. But corruption a problem, lack of trained proc. officers. Monitoring is infrequent, passive and only of financials. Frequent collusion between contracted project supervisors and contractors.
With one exception, monitoring is weak and there is no trigger to review project rationale. In Kosovo there is at least active monitoring of spending against annual budget.
Asset registers range from non-existent (Albania and Montenegro), through partial (Macedonia, Serbia, Kosovo), to a centralized register reconciled annually against physical inventory by MDAs (Bosnia).
Very weak; but not priority as need better planning first. Focus should be total cost cf budget. None of the SAIs playing role in PIM, but targeted one-off project audits could be useful.
Country Guidance/screening
Appraisal Independent review
Selection/ budgeting
Implementation Adjustment Operation Evaluation
Chile (2008)
National and sectoral plans, and active screening (rejection rate 5-8%)
Sponsoring entity formulates project, MIDEPLAN appraises (CEA and CBA)
Quality review by independent unit within MIDEPLAN
Pipeline of appraised projects; but projects added within year. MIDEPLAN/ MOF analyze recurrent costs.
Tight conditions for tendering. Review found timelines overrun in 28% of projects, but actual costs 5% lower.
Where tenders are 10% + over estimate, project subject to reappraisal
Systematic recording and checking of assets and responsible official.
Sample of projects Subject to basic evaluation of outputs. No impact evaluations
Timor Lack of Limited MoF does Weak integration Limited planning; No formal Patchy asset Little or no
3
Lesteiv (2009)
national/ sector strategies, screening.
appraisal in some MDAs.
very little review; lack of time in budget calendar
capital and current; no ceiling for capital to MDAs; lack of time in budget calendar
frequent contract variations; weak procurement and internal control; commitment controls, predictable funding, some monitoring
rules registers. Probably lack of O&M in some sectors
basic post-project review, no evaluation
Serbiav (2009)
Weak. New high level Council for Nat. Invest.
Rules ok, but weak in practice, neglects eco. analysis
Some, but limited scope
MoF lacks authority to prevent un-appraised projects entering budget. Weak analysis of future re- current costs
Procurement law ok, but weaknesses in implementation Frequent changes to project design. Weak monitoring (except for some PIUs)
No provision for project review
Incomplete asset registers
Very weak (including external audit)
To be completed
1
References
Brumby, J, 2009, Concept Review of the Mongolia Policy Note on Public Investment Planning,
November 19, 2009, PREM, World Bank, Washington DC.
Collier, Paul, 2007, The Bottom Billion: Why the Poorest Countries are Failing and What Can Be Done
About It, Oxford University Press, Oxford.
Dudwick, N and Adam Melsson, A Stocktaking of PRSPs in Fragile States, PREM notes Number 127, Poverty Reduction and Economic Management Network, World Bank, November 2008. Eifert, Benn, and Alan Gelb, 2005, Coping with Aid Volatility, Finance and Development, International Monetary Fund, September 2005.
European Commission,2006, Guidance on the Methodology for Carrying Out Cost-Benefit Analysis, Working Document No. 4, DG Regional Policy, August 2006.
International Monetary Fund, 2007, Guide on Resource Revenue Transparency, Fiscal Affairs
Department, Washington DC.
International Monetary Fund, 2010, World Economic Outlook 2010, Statistical Appendix,
Washington DC.
Laursen, Thomas and Bernard Myers, 2009, Public Investment Management in the New EU
Member States, World Bank Working Paper N0. 161, The World Bank, Washington DC.
Rajaram, A, Le, T, Biletska, N, and J. Brumby, 2010. A Diagnostic Framework for Assessing Public Investment Management. Public Sector and Governance Unit, Poverty Reduction and Economic Management Network. World Bank, Washington DC.
Transparency International, 2008. Bribe Payers Index 2008. Document available at www.transparency.org
[Links to the full list of case studies: to come]
i Case study combines PIM technical analysis with political economy analysis of PIM system functioning and reform prospects. ii RSF is Re‐assessment of Feasibility; RDF is reassessment of demand forecast. iii Synthesis of a six country study covering Albania, Bosnia, Kosovo, Macedonia, Montenegro, and Serbia.
2
iv Case study combines PIM technical analysis with political economy analysis of PIM system functioning and reform prospects. Technical analysis trialled a PEFA-style ranking of the stages of PIM functionality, together with a hazard warning system to indicate critical weaknesses/binding constraints on PIM system performance at the margin. v Part of Balkans six country PIM study.