developing tangible measures of government procurement...

___________________________________________________________________________

2008/SOM3/GPEG/SEM/005

Developing Tangible Measures of Government

Procurement Systems Across the APEC Region

Submitted by: Centre for International Economics

Seminar on Implementation of the Value for Money Non Binding Principle in Government

Procurement Lima, Peru

13-14 August 2008

Prepared By

Centre for International Economics

Developing tangible measures of government procurement systems across the APEC region

Prepared By Centre for International Economics July 2008

The Centre for International Economics is a private economic research agency that provides professional, independent and timely analysis of international and domestic events and policies.

The CIE’s professional staff arrange, undertake and publish commissioned economic research and analysis for industry, corporations, governments, international agencies and individuals. Its focus is on international events and policies that affect us all.

The CIE is fully self-supporting and is funded by its commissioned studies, economic consultations provided and sales of publications.

The CIE is based in Canberra and has an office in Sydney.

Canberra

Centre for International Economics Ian Potter House, Cnr Marcus Clarke Street & Edinburgh Avenue Canberra ACT 2601

GPO Box 2203 Canberra ACT Australia 2601

Telephone +61 2 6245 7800 Facsimile +61 2 6245 7888 Email [email protected] Website www.TheCIE.com.au

Disclaimer

While the CIE endeavours to provide reliable analysis and believes the material it presents is accurate, it will not be liable for any party acting on such information.

© Commonwealth of Australia 2008

ISBN 1 921182 47 4

Department of Finance and Deregulation This work is copyright. Apart from any use as permitted under the Copyright Act 1968, no part may be reproduced by any process without prior written permission from the Commonwealth.

Requests and inquiries concerning reproduction and rights should be addressed to the Commonwealth Copyright Administration, Attorney General’s Department, Robert Garran Offices, National Circuit, Barton ACT 2600 or posted at http://www.ag.gov.au/cca

DEVELOPING TANGIBLE MEASURES OF GOVERNMENT PROCUREMENT SYSTEMS ARCOSS THE APEC REGION 3

Abbreviations

APEC Asia Pacific Economic Co-operation

CPS Corruption Perceptions Score

FCI Freedom from Corruption Index

FPI Freedom of the Press index

GDP gross domestic product

GPEG Government Procurement Expert’s Group

IMF International Monetary Fund

NBP non-binding principles

TBS Time with Bureaucracy Score

DEVELOPING TANGIBLE MEASURES OF GOVERNMENT PROCUREMENT SYSTEMS ARCOSS THE APEC REGION

Contents

Executive summary 6

1 Introduction 8

2 The government procurement process and its conceptual underpinnings 11 Achieving value-for-money 12 The process of government procurement 13 Inter-relationship between principles 19 The business provider 20 The linkages between the principles and value-for-money 22

3 Developing indicators of doing business with government 24 Moving from principles to indicators 27 Principle: transparency 29 Principle: competition 29 Principle: accountability 30 Principle: fair dealing 32 Principle: efficiency 33

4 How would indicators be implemented? 35 Moving towards an index for overall government procurement 36

5 Conclusions and next steps 45

References 47

APPENDIX 49

A APEC member economy Government Procurement Index Tables 2007 51

Boxes, charts and tables 1 Better government procurement processes across APEC is associated with higher income 7 1.1 Government procurement as a percentage of GDP for 2006 9 2.1 A typical procurement timeline 11 2.2 Components in achieving value-for-money 13 2.3 APEC’s government procurement principles 14 2.4 The principles of good process 15 2.5 Optimal relationship between transparency and value-for-money 16


2.6 The principles are interrelated with each other 20 2.7 The business provider and value-for-money 21 2.8 The relationship between good process principles and price, quality and risk 23 3.1 Measuring the procurement process 26 3.2 Mapping the principles to measurement indicators 28 3.3 Principles to indicators: transparency 29 3.4 Principles to indicators: competition 30 3.5 Principles to indicators: accountability 31 3.6 Accountability — institutional checklist 32 3.7 Principles to indicators: fair dealing 32 3.8 Ethics quality checklist 33 3.9 Principles to indicators: efficiency 34 3.10 Examples of hypothetical contracts to allow consistent data collection 34 4.1 Principles to indicators: transparency 37 4.2 Principles to indicators: competition 37 4.3 Principles to indicators: accountability 38 4.4 Principles to indicators: fair dealing 39 4.5 A few economies have made significant gain in government procurement: illustrative

example from proxy data 40 4.6 Most economies have made some improvements in certain aspects of the government

procurement process: illustrative example from proxy data 41 4.7 The top five reformers have made most gains in transparency and accountability but

no improvement in efficiency: illustrative example from proxy data 42 4.8 Most economies have made improvements in some procurement facets; overall the

result is little change: illustrative example from proxy data 43 4.9 Better government procurement processes across APEC is associated with higher

income 44


Executive summary

Improving the efficiency of government procurement is important because the purchase of goods and services by government is a large part of economic activity, so even small gains can boost overall economic efficiency and growth.

It is difficult to improve the efficiency of government procurement if it is not quantified. Without objective measures it is not possible to tell whether progress is being made or not. As the saying goes, ‘you cannot manage what you do not measure’.

The key objective of government procurement is to achieve value-for-money. This goal has three dimensions: one is price, another is quality, and the third is risk. Trade-offs may be made between these three dimensions.

There is no reported statistic for ‘value-for-money’. Due to of the trade-offs between price, quality and risk, the only way to measure and ensure value-for-money by governments is through good procurement processes.

‘Good process’ in procurement is achieved by following APEC’s agreed principles for government procurement. These are:

– Transparency;

– open and effective competition;

– fair dealing;

– accountability, due process; and

– non-discrimination.

So the task of measuring whether value-for-money has been achieved in government procurement comes down to measuring how well each of the above five principles is followed when tenders for goods and services are filled.

There are no existing specific measures of what constitutes good process in government procurement and the achievement of value-for-money. Specific measures have to be devised and collected. What these measures might be and how they might work is suggested in this report.

Meanwhile, there are some other indicators that can serve as proxies for the five government procurement principles. These serve as a guide to what the measures of the five principles, if developed and refined, might show and how the results might be used. These indicators come from sources such as the World Bank’s Doing Business report which measures the ease of doing business around the world.

Using these proxy measures, an illustrative indicator of the effectiveness of government procurement has been produced for APEC economies. The report shows that, if


comprehensive measures of the five agreed principles were made, there is likely to be large differences between economies in their performance of government procurement. Those APEC economies with better government procurement processes tend to be more prosperous, as indicated by chart 1 below, cementing the notion of the importance of government procurement to overall economic performance.

1 Better government procurement processes across APEC is associated with higher income

0

10

20

30

40

50

60

70

80

90

100

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Income - per capita GDP in 2007

Gove

rnme

nt P

rocu

reme

nt In

dex ..

Least

Best

Data source: TheCIE.

The results also show that some economies had improved procurement practices over the last five years. Identifying what actions have been taken in different economies that has led to these improvements, and why some economies are doing better than others at government procurement, provides valuable lessons that can be shared across the APEC community. Doing so gives all economies the opportunity to improve their procurement practices and therefore economic efficiency and growth.

If such measurement is seen as worthwhile, the next step is to put the proposed methodology into practice by trialing the methodology in say four or five APEC economies that span from developed to developing economies. That experience will help refine the methodology by showing up pitfalls and difficulties. It will also give some indication of the likely effort required and potential benefits to be gained, so a more informed decision can be made on how to improve government procurement across APEC economies.


1 Introduction

Government procurement is a large source of government expenditure for APEC member economies, in 2006 alone government expenditure, of which government procurement forms a significant proportion, was worth a staggering US$2.7 trillion.1 Consequently, even a small improvement in the government procurement process has the potential to deliver large benefits. This is particularly salient for developing economies, where the need and the potential for economic gains are the greatest.

The relative value of government procurement to APEC economies varies, but is a significant component of GDP for all economies. Chart 1.1 presents the breakdown of government procurement for each APEC member economy. Most economies lie in the range of 10 to 20 per cent of GDP. At such a large proportion of GDP, it is important that a process of distributing public resources provides the economy with value-for-money from the goods and services procured as a 10 per cent improvement in government procurement could deliver an extra 1 per cent to GDP growth.

Emphasis is placed upon ensuring that value-for-money is delivered to the economy as a whole and not just to government, since an onerous government procurement policy may benefit the government, but at the cost of efficiency, which hurts the economy. A process of government procurement that is efficient, open and transparent has the potential to deliver wide-reaching benefits for the whole economy. It does this through increasing the competition between suppliers and reducing the cost; increasing the number of suppliers, which should improve the quality of the goods or services procured; providing better policy predictability, and by building credibility in the government’s policy and governance framework.

In recognition of the importance of government procurement to pursuing APEC’s objectives of enhancing economic growth and prosperity, the Government Procurement Experts’ Group (GPEG) was established in 1995. GPEG was established for the purpose of considering and promoting ways to increase the transparency and liberalisation of APEC economies’ government procurement markets.

Outside APEC, work on government procurement is being undertaken by the Organisation for Economic Co-operation and Development (OECD) and the World

1 Measured by general government final consumption expenditure in 1990 constant prices


1.1 Government procurement as a percentage of GDP for 2006

0 5 10 15 20

Australia

Brunei Darussalam

Canada

Chile

China, (People's Republic of)

Hong Kong, China

Indonesia

Japan

Korea, (Republic of)

Malaysia

Mexico

New Zealand

Papua New Guinea

Peru

Philippines

Russia

Singapore

Chinese Taipei

Thailand

United States

Viet Nam

Government expenditure (% of GDP)

Not available

Data source: Penn World Tables, version 6.2.

Trade Organisation (WTO), both attempting to progress government procurement policy towards more liberal government procurement markets through initiatives such as:

studies on transparency in government procurement at the WTO, although work is on hold;

a ‘global’ agreement, and a subsequent re-negotiation of that agreement , on liberalisation of government procurement through the WTO;

the establishment of a negotiation mandate on procurement in services;

a procurement harmonisation framework for nations and institutions; and

development of a government procurement assessment framework.


Much of the work to date has focused upon developing good government procurement policy. But the implementation of this, and the follow through to practical outcomes, has been limited. This lack of progress reflects that it is difficult, if not impossible to improve the process of government procurement if no assessment is made. That is, it is not possible to improve the process of government procurement without measuring one’s performance. Measures of performance are also needed to appraise where the returns from reform are greatest so that effort and limited resources are allocated to those areas of highest payoff. In short, you cannot manage well what you do not measure.

The aim of this report is to develop a way to measure performance in government procurement. It does this by providing a framework for good government procurement policy, built upon accepted core principles. The major innovation of this study is the development of key indicators to assess performance against the implementation. By measuring our performance in implementing a good government procurement process, progress can be made towards improving it.

This study provides a first attempt at constructing indicators of good government procurement process. This is not a simple task, which is why it has not been done before. The emphasis is conceptual and methodological. A second stage report could build on this study and address the practical issue of implementing the measurement of these performance indicators across APEC economies, providing case studies to illustrate how it is done.


2 The government procurement process and its conceptual underpinnings

This chapter presents the framework for government procurement, which is based upon the core principles, economic theory and a literature review of the current practices in government procurement.

Procurement covers the whole process of government acquiring goods and services from third parties. Around the APEC region government business in goods and services is an important component of economic activity. As previously illustrated the amount of public money spent on procurement is quite substantial. Therefore the action of ‘procurement’ needs to be effective to achieve the maximum benefit to the government and the public from its spending.

The government can achieve the most efficient and effective spend of public money by having a workable outline by which to procure goods and services from potential suppliers. Chart 2.1 illustrates a typical procurement timeline.

2.1 A typical procurement timeline

Identify procurement

need

Manage termination / transition / disposal

Manage contract /

relationship

Determine suitable

procurement process

Conduct suitably scaled

procurement process

Source: TheCIE.

The procurement process spans the lifecycle, from initial concept and identification of needs to the end of the useful life of an asset or the end of a service contract. Along the timeline, however, a procurement framework should encapsulate a number of decisions in the lifecycle and allow a procuring government agency to justify spending requirements, outcomes and apply a standardised strategy for procurement.

Successful procurement depends on all phases of the procurement timeline to be efficient and implemented effectively. In doing so, it ensures government is able to access markets and interact with potential suppliers to fulfil its requirements in a timely, efficient and uncomplicated way. There is no advantage in developing a complex process which satisfies


a variety of quality control mechanisms and standards if it fails to deliver in a timely and efficient way.

The objective of an efficient framework of government procurement is to provide value-for-money.

Value-for-money is an important objective and one that government’s should strive for. There are many benefits to achieving value-for-money in government procurement. From an efficiency perspective, it allows for a better allocation of scarce resources. This has obvious welfare effects. A stronger focus on efficiency will ensure more resources are dedicated to enhanced capacity and contribute more effectively to economic growth, which is of course in the best interests of government and the country as a whole.

Value-for-money has three dimensions to it that are fully described later. One obvious dimension is price, but value-for-money is not about achieving the lowest price as higher quality goods and services for the same or even higher price can give value-for-money. So ‘quality’ is a second dimension. The third dimension is ‘risk’ since the goods and services may not meet the specified quality standard; or the contractor may fail to deliver. Less risk for a given price and quality of procurement gives better value-for-money; it is defined as the optimum combination of price, quality and risk. When assessing value-for-money the procuring entity will weigh up the benefits of the purchase against the cost of the purchase.

Given these benefits, it is easy to understand why governments should be aiming for value-for-money in their procurement processes. But how do you achieve value-for-money?

Achieving value-for-money

The concept of value-for-money from processing goods and services goes beyond the initial selection process and issues at contract phase. Value-for-money also relates to the entire project life cycle through such areas as service provision, project implementation capability, contract management and even disposal value and cost. These are important areas for government to consider in any value-for-money procurement decision. However, the scope of this report is just on the procurement process. It is also necessary to note that value-for-money is a relative concept, which requires comparison of the potential or actual gains against alternative procurement options. And as markets change, develop and mature, what may constitute value-for-money now may need to be reassessed in the future.

To undertake a well-managed public acquisition of a good or service, it is necessary to consider upfront what the key drivers of value-for-money will be and how it is achieved.

As illustrated by chart 2.2, value-for-money is achieved through two main channels: first, through good process and, second, by selecting the most appropriate provider.


2.2 Components in achieving value-for-money

Process

OBJECTIVE

Value for money

Business provider

How good this process is dictates achieving value-for-money

Source: TheCIE.

Good process is essential in achieving value-for-money. It needs to be well planned, transparent and consistently implemented. In doing so, transaction costs will be reduced to the bidder and the provider, respectively, ensuring that there is lower entry costs, leading to greater competition and improved offerings from potential suppliers.

The process of government procurement

GPEG has established and endorsed a set of non-binding principles (NBP) on government procurement in 1999. There have been some changes since and the current NBP are:

value-for-money;

open and effective competition;

fair dealing;

accountability, due process; and

non-discrimination.

The original NBP (box 2.3) also included the concept of transparency, but this principle has been subsumed into the area-specific APEC Transparency Standards on government procurement.2 However, the principle of transparency remains an important component of the procurement process and given the overlap between transparency and other principles, are retained here for measurement purposes.

Of these principles, for the reasons stated earlier, value-for-money is the over-arching principle that governments should be targeting. It is more an objective than a principle. Value-for-money encapsulates the other principles in that it cannot be achieved without good process, which in itself involves open and effective competition, transparency, fair

2 APEC’s Government Procurement Experts Group (GPEG) http://www.apec.org/apec/

apec_groups/committee_on_trade/government_procurement.html


dealing, accountability and non-discrimination. In other words, the other principles are the means of achieving better value-for-money.

Better value-for-money is achieved when the process of government procurement exhibits openness and efficiency. By openness we mean that the process by which government obtains goods and services is freely accessible to all potential suppliers and exhibits the following attributes, which are closely aligned with APEC’s NBPs.

2.3 APEC’s government procurement principles

APEC’s Government Procurement Experts Group (SPEG) aims to:

develop a common understanding of government procurement policies and systems; and

achieve liberalisation of government procurement markets throughout the Asia-Pacific region.

GPEG completed the development of non-binding government principles in 1999. The principles identified elements of good governance for APEC member countries. APEC’s non-binding principles are:

transparency — aims to ensure that sufficient and relevant information is made available to all interested parties in a timely manner through readily accessible and widely available medium at no or reasonable cost;

value-for-money — comparison of relevant benefits and costs on a whole of life basis including encouragement of competition among supplies commensurate with anticipated value-for-money benefits from that competition;

open and effective competition — government procurement should be transparent and readily accessible to the public and in particular to potential suppliers. Processes should be designed to encourage levels of competition among suppliers, commensurate with anticipated value-for-money benefits from that competition;

fair dealing — ensuring that procurement activities are conducted in a reasonable and equitable manner, and with integrity;

accountability and due process — government procuring agencies and individual procuring personnel should be accountable to their governments, the end users, the public and suppliers for the efficient, cost-effective and fair conduct of their procurement; and

non-discrimination — procurement laws, regulations, policies, administrative guidelines, procedures and practices should not be prepared, adopted or applied that afford protection, favour or preference to suppliers of any particular economy.

Since these principles were adopted the principle of transparency has been subsumed into the area-specific APEC Transparency Standards on Government Procurement.


An open process should:

be transparent;

encourage competition between suppliers;

promote fair dealing towards those same suppliers; and

ensure all parties, both government and supplier are accountable.

Efficiency of process, although not outlined initially as a guiding principle in the NBPs, is just as important as an open process. Efficiency should be considered a constraint on the process, therefore to ensure that we achieve a good process, we do not want to construct a process that is neither too onerous on the supplier nor too costly to administer from a governments’ perspective.

Why these principles?

Each of these principles is important at creating a process that facilitates achieving value-for-money from the business provider. Chart 2.4 provides a list of the NBPs and how they relate to the concept of value-for-money.

In addition, each principle is described below with a discussion on how they contribute to achieving value-for-money.

2.4 The principles of good process

OBJECTIVE

Value for money

GOOD PROCESS

Openness • Transparency • Competition • Accountability • Fair Dealing

Efficiency

Good process is key to achieving value for money through openness and efficiency

Source: TheCIE.


Openness

Transparency

Transparency is defined as all relevant information regarding the procurement process, contract details and all rights and obligations of the parties involved are readily and easily accessible, in a timely fashion and at a minimum cost to all interested parties. It requires the disclosure of all relevant information at critical points in the procurement process. This is why transparency is important. If the process does not allow for disclosure of all relevant information, then it will impede the ability of firms to bid for contracts.

Decreasing the uncertainty (and therefore risk) associated with the tendering process is another aspect of transparency. By publicly advertising what is to be supplied, at what quantity and what quality, any potential supplier knows what demands they are facing (i.e. what is required of them) and they can accurately price their services accordingly. If this was not the case, then suppliers would incur a fixed cost to obtain this information. This cost would be passed on through the pricing mechanism to the procuring government thereby reducing the value-for-money from the procurement.

Of course there is a fine balance between transparency and value-for-money. Transparency does not come for free and at some point the additional value-for-money achieved from greater transparency will be less than the extra costs incurred. In other words there is an optimal level of rule setting to induce the most efficient use of public money. This point is represented graphically in chart 2.5.

2.5 Optimal relationship between transparency and value-for-money

Cost of transparency

Value for money OPTIMAL POINT AT

WHICH TRANSPARENCY INDUCES BEST

VALUE FOR MONEY

Source: TheCIE.


By formally setting out rules, regulations and obligations of all parties, savings can be made, but it is likely to increase administrative costs of implementing such a framework. At this point providing ever increasing levels of rules, regulations, obligations and documentation may systematically increase the level of transparency of the procurement, but it is also likely to overburden potential suppliers with bureaucratic ‘red tape’ and therefore increase the cost associated with tendering.

Competition

A competitive procurement process is one of the ways in which government can achieve value-for-money in its procurement needs. The procurement process should allow easy entry of any relevant suppliers and encourage market based competition between those suppliers. Robust competition requires a number of well-qualified bidders who have expressed strong interests in bidding for the good or service.

Potential suppliers within a commercial environment have different economic advantages and therefore differing costs. Better value-for-money is achieved through a competitive procurement process by comparing a larger number of different bids from suppliers with different cost structures. By allowing all potential suppliers to bid for government business, government can take advantage of these differing cost structures between suppliers to obtain a lower price and thereby reduce their budget expenditure.

Additionally, a competitive procurement process is further enhanced by outlining a procurement framework that exhibits non-discrimination. That is, the same opportunities to compete for government business should exist for domestic and foreign suppliers. This has the effect of increasing the pool of suppliers available to the government.

Although much economic theory suggests that in relation to competition, more is good, this fails to take into account costs in administrating a competitive platform. Like transparency, where there is an optimal level of rule setting to obtain value-for-money, it is generally believed that the level of competition should accurately reflect the market circumstances commensurate with the benefits received.

This, however, should not be taken as an argument in favour of artificially reducing competition. It merely highlights the point that, like administrating a transparent process, the benefits of increased competition need to be considered along with the costs of implementing and administrating competition rules.

Accountability

Accountability provides suppliers with the confidence to participate in government markets by reducing the risk of ‘self interest’ behaviour such as fraud and corruption. These types of actions can significantly increase the cost of doing business with government, which is passed on to government in the price charged for the good or service. This represents decreased value-for-money as monetary resources are diverted away from other potential uses.


The purchasing organisation, the supplier, their respective employees and the product quality, should all be accountable to the government and the public through independent oversight. It also requires that the procuring agency be able to publicly account for its decisions and take responsibility for the achievements of the procurement outcomes.

A good procurement process with elements of sound accountability should include the following:

A clearly defined hierarchy of responsibility for procurement decisions. An example of this would be the availability of a delegation matrix3;

An adequate audit trail of procurement records to be maintained for internal and external scrutiny of procurement decisions;

Publicly available contract award details;

Policies and procedures in place to help government officials through the procurement decision making process; and

Providing feedback to unsuccessful tenderers.

Fair dealing

Given the fact that expenditure of public funds is involved, fair dealing is especially important in government procurement activities and thus should be subject to public scrutiny. In addition, fair dealing has many advantages. It promotes public confidence in the procurement process and mutual trust and respect between the buyers and suppliers. It includes a clear understanding by all parties to a contract in regard of their obligations and expectations. This encourages participation in the government procurement process and therefore contributes to achieving the best value-for-money in the acquisition of goods and services through increased competition. Unethical behaviour in the first instance will damage the reputation of the procuring agency or government. Initially this may not add costs to and/or reduce the quality of the goods and services procured, but potentially in future contractual dealings the risk of unethical and/or inequitable behaviour will be factored in through higher prices or reduced product offerings.

Efficiency

Efficiency is important to ensuring that the process is conducted at least cost. This element is important to ensuring that the government procurement process achieves value-for-money, not just from the government’s perspective, but that of the economy. Bidding for government contracts is costly, so any inefficiency in the process will waste resources and increase the cost passed on to the purchasing agency in one form or another. That means that the cost burden on business of undergoing the procurement process is taken into consideration.

3 A delegation matrix refers to a flow diagram that clearly defines the hierarchy of an organization,

who reports to whom, who is responsible for what.


There are two elements to consider when assessing the efficiency of a procurement process.

Timeliness: Does the process allow all available information valid and are decisions relating to the contract award?

Complexity: How onerous is the process of procurement? (For example, the number of decision steps involved.)

Although this framework is somewhat detached from the rest of the economy, government procurement does not operate within a vacuum, with significant feedback linkages to the rest of economy. It is therefore important that, while this report is focused solely on the procurement process, for an effective and efficient public procurement system to operate, other basic economic conditions are required. Examples of this are: a robust legal institutional framework to ensure the enforcement of the contractual process, a stable macro environment, political stability and predictability of rules and lack of government corruption. Without these elements transaction costs to potential suppliers will be higher than need be.

Inter-relationship between principles

The above principles are interrelated, which means that incorporating all of them into the government procurement process will lead to a better outcome. The ‘whole’ is greater than the ‘sum of its parts’.

Graphically, we can represent the interrelatedness of the principles by chart 2.6. Not only are these principles linked but they reinforce each other. However, if we were to choose a single principle that is key to achieving our objective of value-for-money, it would be transparency. Hence, the reason transparency is placed at the top of the diagram with linkages to the other principles.

Transparency is essential for accountability. The government is held to account through the parliament and the parliament in turn is held accountable through a robust and open media. Transparency facilitates greater competition in the process by providing potential suppliers all relevant information thereby allowing them to price the bid effectively. Transparency and accountability allow for greater fair dealing by providing fewer opportunities for personnel gain through the procurement process.

Finally efficiency is also presented in the chart surrounding the principles. In this way it represents the idea that it supports the principles of the procurement process while at the same time, if the process is not onerous, efficiency can also act as a constraint.


2.6 The principles are interrelated with each other

Transparency

Accountability Competition Fair dealing

Efficiency

Source: TheCIE.

The business provider

As noted in the previous section, one of the main channels through which value-for-money is obtained through the public procurement process is by selecting the best and most appropriate provider of the good or service. But what constitutes the most appropriate provider?

In assessing a business provider for government procurement, it was established earlier that there are three factors that should be considered when deciding who the provider should be. They are:

price

quality

risk.

The ability of the potential supplier to provide the best quality good or service for the lowest risk and cost provides the best value-for-money that the government can obtain. Chart 2.7 illustrates the business provider and value-for-money. The most efficient allocation of resources will occur when the optimal combination of price, quality and risk is achieved.


2.7 The business provider and value-for-money

OBJECTIVE

Value for money

GOOD PROCESS

Openness • Transparency • Competition • Accountability • Fair dealing

Efficiency

For a given quality, the business provider provides further value for money through price and risk transfer

BUSINESS PROVIDER

Quality

Price

Risk

Source: TheCIE.

Price

Price is certainly an important feature in determining the most appropriate provider. Not only do we want to get the most affordable option, but price also acts as an indicator for the other factors. Price acts as an indicator of quality and any risks associated with the provider and even the procurement process itself can be reflected in increased prices charged to the government.

Quality

Quality can be considered a non-price factor that relates to the usefulness of the good or service that is being procured. Quality relates to the product requirements of the procuring agency and their evaluation of how well those requirements can be met by the potential supplier. Usually there are detailed specifications of the good or service to be provided, but not every aspect can always be written down without exhaustive documentation. Reputation and ‘brand name’ therefore, can be important indicators of quality.


Risk

The identification and management of risks in procurement suppliers is one of the key factors that drive value-for-money. Some of the risks which need to be considered in assessing the merits of different suppliers are:

corporate viability: is the supplier a going concern and likely to be so during the timeframe of the contract or project; and

technical capability: does the supplier have the technical ability and personnel expertise to complete the project?

These risks can be made more apparent through good, transparent process. In doing so the risk and effect of the risk can be taken into account in the decision making process.

The factors of price, quality and risk are all interrelated at a conceptual level. Price is certainly an indicator for quality. A higher price as often as not, reflects a better quality product. Alternatively, the lowest price regardless of quality may not be the best value-for-money decision. A lower price may be the best option from a monetary stand point, but it could also signal greater risks attached to the service provision, constituting a reduced value-for-money proposition.

Additionally, increased price may also reflect greater uncertainty. That is, any perceived risks in the procurement process (be it through non-transparent procedures etc.) will lead to an increase in the price of the tender. In this way price also acts as an indicator of risk transfer, as its the price at which a business provider is willing to accept any perceived risks associated with the procurement. This in itself is not an issue, but if a procuring government is willing to undertake risk minimising actions, for example, by increasing transparency in the procurement decision making process, it potentially benefits from also minimising the price for a good or service. This, of course has logical flow on benefits. In maximising the benefits of procurement, a more efficient allocation of monetary resources can be achieved.

The linkages between the principles and value-for-money

This section outlines why these principles are important, and in particular, how they relate to achieving higher quality, reduced risk and lower prices.

As spenders of taxation revenue, governments are under an obligation to provide the most efficient and effective use of public money. As previously outlined this is done through two channels, firstly by implementing and maintaining a good procurement process and secondly by selecting the most appropriate supplier subject to specific requirements. It is therefore important to outline how good process affects those qualities of value-for-money that are associated with the business provider.

Chart 2.8 illustrates some of the important effects of the procurement process, the principles involved and summarises the relationship between good process principles, price, quality and risk.


2.8 The relationship between good process principles and price, quality and risk

PROCESS CONTRACTOR

Openness

Efficiency

Quality

Risk

Price

The best value for money contractor is achieved through optimising the combination of the factors quality, risk and price.

To contribute to achieving value for money, the process of government procurement must be open and efficient. And it is the optimal combination of these principles that contributes to the maximum value for money.

GOOD PROCESS

Openness

Efficiency

BUSINESS PROVIDER

Quality

Risk

Price

Source: TheCIE.

The previously mentioned principles of government procurement in many respects complement each other. For example, transparency underpins three other principles of government procurement. First, it is a necessary condition in enabling enhanced competition by providing all relevant information to suppliers. This in turn decreases the uncertainty in the procurement process and allows providers to accurately price the good or service provided.

Second, it subjects governments, agencies and their employees to greater scrutiny and accountability thereby decreasing the opportunities for corruption and bribery. This is of course advantageous to business providers as it lowers the cost and associated risk of doing business with government.

While much might be said about providing unequal access to government procurement markets (as a lever for other policy initiatives), this may actually lessen the possibility of achieving value-for-money. Fairness, based on open competition provides better value-for-money because any provider chosen due to preferential treatment does not necessarily represent the best value-for-money. By opening the process up to all competitors with differing cost structures, it is more likely that suppliers offering better terms or who are better able to meet the tender specifications are included, thereby reflecting increased value-for-money.

Finally, efficiency in the process will be reflected through price. Any inefficiency is likely to add to the cost of the provider, which will be passed on to the procuring government through a higher price charged.


3 Developing indicators of doing business with government

Even though the NBPs on government procurement were agreed by member nations as far back as 1999, little progress has been achieved in the liberalisation of public sector procurement markets between member nations. One of the factors that have potentially delayed movement towards free government procurement markets is the lack of a suite of measurable variables or ‘measurement indicators’.

Indicators are important for measuring progress towards a better government procurement process, as measurement is the key to improving the process of government procurement. Or put another way, we can’t fix what we don’t measure.

Measurement indicators not only allow member nations to benchmark themselves against world best practice, they also allow for continuous self-appraisal over time. Further, good measurement indicators allow for the identification of bottlenecks in the reform process and can help in tailoring policy to address these areas, which ultimately leads to better value-for-money from the government procurement process.

This proposed framework for outlining potential indicators provides a platform for member economies to initially implement a system of measureable baseline indices that will eventually:

provide a baseline level for internal comparison;

increase the stock of information on national performance;

contribute to the effective monitoring of institutional and legislative reform;

increase quality in the government procurement market; and

contribute to a more effective regional reform agenda.

These measurement indicators are designed to help APEC member economies with the implementation of strategies that will facilitate the uptake of the NBPs as set out in 1999. It is hoped that these indicators will provide a benchmark that may help other public sector agencies within APEC member economies.

What are measurement indicators?

Measurement indicators are voluntary measures that APEC member economies may use to monitor internal and multi-lateral performance and progress. In the following sections we


present a set of indicators designed to help member economies and authorities improve their procurement practices, thereby achieving better value-for-money.

Key facets of a performance indicator: what makes a good measurement instrument?

Selecting a good indicator

There are several aspects to a good or ‘effective’ indicator. A good indicator provides intelligence or insight into a policy or outcome by providing a reporting mechanism to highlight and prioritise potential risk areas for action or improvement. Most importantly, a good indicator should enhance the value of policy by providing insight and comparability.

Ideally, an effective or good indicator would be:

measureable without greatly increasing administrative costs;

useful for authorities in assessing performance;

unambiguous in their meaning;

simple and practical; and

consistent across time.

In addition to the above five characteristics, a good indicator should initially be able to provide a baseline for the purposes of future benchmarking of performance. However, baselines only provide a point of comparison, a good indicator should also provide the flexibility to measure and monitor changes in performance across time. It is only through the monitoring of government procurement performance that reform measures to government procurement markets can be properly assessed.

Types of indicators

There are several different types of indicators, all with the same purpose of quantifying performance. The different types of measurement indicators are described below. Each type has its strengths and weakness. In developing a measurement system for government procurement, the framework should contain a combination of them.

Binary indicator: A binary indicator is a simple ‘yes’ or ‘no’ answer that reflects whether a certain state exists. For example, ‘Does an independent Auditor-General exist?’ These measures provide a simple means of determining whether an event exists. However, these measures can be quite limited in that they fail to describe, in quantitative terms, any relevant information that may be pertinent, for example the strength of the variable.

Leading indicator: A leading indicator can either be qualitative or quantitative in nature and measures performance during the process. For example, the percentage of tenders publicly advertised is one component of transparency. The major advantage is that corrective action can be taken before deficiencies show up through reduced performance.


Lagging indicator: Like leading indicators, lagging indicators can either be qualitative or quantitative in nature. However, unlike leading indicators, lagging indicators measure outputs or outcomes after the process. An example would be surveying suppliers’ opinions about procurement steps, outlines and procedures after the tender process.

Real time indicator: As the title suggests, these are indicators that provide access to information on the procurement process in real time. Since they provide information now, they can allow process performance to be measured quickly. However, they can be quite costly to implement.

What should the measurement indicators cover?

The action of procurement covers a number of steps along a defined timeline. Chart 3.1 provides a summary. The action of procurement covers many steps from the decision to obtain goods and services through to the completion of a contract and/or discharge of any residuals assets. From a theoretical standpoint, a suite of measurement indicators could measure any part of the procurement decision process, for example, contract relationship can be measured through ex-post supplier surveys. However, as informative as this might be, measuring procurement after the fact may not be the best way to improve the efficiency of each dollar spent. This is not to say that a post review process is a waste of time. To the contrary, useful information can be obtained. However, implementing changes after the fact means that there is a greater time lag involved in indentifying areas of improvement and implementing positive changes.

3.1 Measuring the procurement process

Identify procurement

need

Manage termination / transition / disposal

Manage contract /

relationship

Conduct suitably scaled

procurement process

THE PROCUREMENT PROCESS IS WHAT WE

ARE MEASURING

Determine suitable

procurement process

Source: TheCIE.

As previously stated, the overarching goal of government procurement is to provide the best and most efficient spend of public money, to attain value-for-money. There are two key factors that attribute to attaining efficiency in public procurement, the process of government procurement and the business provider.

There seems little point in ‘measuring’ the business provider, since the government, except for legislative powers and its choice of provider, has very little direct impact on the


supplier. That is the government has little influence on the price a supplier wishes to charge, the quality of the product the supplier wishes to deliver or the inherent risks associated with choosing one particular supplier over another.

However, the process by which the government purchases goods and services from suppliers is an area that government policy and regulations can affect and it is through this process that the government can indirectly affect prices and quality of goods and services it receives. Hence, the purpose of this report is to provide a measurement framework by which member economies of APEC can measure and benchmark their own procurement process.

Moving from principles to indicators The rationale for moving from principles to indicators is to provide a logical path as to how to construct the indicators, but more importantly it shows how the indicators are indicative of the underlying principles of the government procurement process. As the principles are the key to achieving value-for-money, by showing how the indicators relate back to the principles, it is possible for government to then identify areas of the process that need improving, thereby facilitating better value-for-money.

The sequential nature of moving from each of the NBPs to a clear suite of measurement indicators is illustrated in chart 3.2. This step is best illustrated by an example. The example here is the openness of the process. Openness can be broken down into a number of different sub-areas such as transparency, competition, accountability and fair dealing. From here each sub area can be broken down into various indicators.

In the example provided in chart 3.2, openness is broken down into competition, from here the sub-area is further broken down into entry barriers, local requirements and non-discrimination. Indicators are then assigned to each. In doing so, it provides a logical framework that can be aggregated back to the defining principles of the procurement process thereby providing a systematic way of quantifying the process of public procurement.

It should be noted here that the indicators for this particular example are not an exhaustive list nor are they the only indicators that can explain the sub-areas. What works for one member economy may not work elsewhere or the data may not be available due to different institutional frameworks or cultural arrangements.


3.2 Mapping the principles to measurement indicators

CATEGORY SUB-DIMENSION

2 Competition

Sub-

dimen

sion

SUB-DIMENSION HEADINGS

2 Competition

Sub-

dimen

sion

Head

ings

1. Entry Requirement 2. Local Content 3. Non-discrimination

OPENNESS

1. Transparency

2. Competition

3. Accountability

4. Fair Dealing

HEADINGS INDICATORS

• Entry Requirement MI: Payment to submit tender

DATA: % of tenders requiring a submission payment

• Local Content

MI: Local work inputs required DATA: % of tenders requiring any local or regional

content such as materials and/or labour • Non-discrimination

MI: Nationality Restriction DATA: % of tenders that dis-allow foreign bids

Source: TheCIE.

The indicators

A set of indicators are outlined below. They have been set out in the following format, a brief definition of the principle, the sub headings and finally a breakdown, with more detailed information, into the indicators.

The indicators are designed to help APEC member economies in both monitoring their government procurement process and identifying areas for reform. Development of these measurement indicators is not an easy task. This is only meant as a first step to outline possible options in constructing an overall indicator of procurement performance.


Principle: transparency

Transparency means all relevant information regarding the procurement process is readily accessible, in a timely fashion and at minimum cost to all interested parties.

For transparency we have outlined three areas and possible indicators for each (table 3.3). The first area is ‘opportunities for access’, which refers to the ease of accessibility of any relevant documentation that relates to the tender. This may take the form of posting draft contracts or tender requests on the internet or publicly advertising them through other local media and as such, the percentage of contracts or tenders advertised in public media or online is the indicator of choice to reflect opportunities to access.

‘Rules’ are the second area that has been identified as important to transparency and are easily measureable. This area doesn’t refer to the existence of rules. More so, it means that if rules do exist they are clearly defined and available. Making the rule clear and available provides potential suppliers with the information required to make a fully informed decision. Therefore, the indicator chosen to reflect Rules is the percentage of tenders that are advertised with draft contracts and/or statement of requirements attached.

Finally, ‘criteria availability’ refers to the number of criteria that the tender is judged on and whether these criteria are made available. As stated previously, price is not the only component to contribute to value-for-money and, therefore, should probably not be the only criterion by which a tender is judged.

3.3 Principles to indicators: transparency

Areas Indicator Data

Opportunities for access Public Advertisement of Tenders Per cent of contracts publicly advertised in the local media/online

Rules Draft Contract and Statements of requirement

Per cent of tenders advertised with draft contracts and statement requirements

Criteria availability Number of Criteria in a Tender Average number of bid criteria over and above that of price

Source: TheCIE.

Principle: competition

Competition means the procurement process allows easy entry of potential suppliers and encourages competition between suppliers.

Competition is important in providing value-for-money in public money expenditure. Any impediments are likely to decrease the number of potential suppliers that a procuring government may choose from, thereby depriving them of potentially better quality goods and services at a lower price. Hence, we have identified three areas that provide indicators on the level of competitiveness the process exhibits (table 3.4).


First, ‘entry barriers’ are key to competition. If restrictions are placed on suppliers, the government is effectively reducing the pool of potential suppliers it may call upon, thereby reducing the likelihood that it will receive the best value-for-money. As an indication of entry barriers we have identified the percentage of tenders that require a submission payment. Payments are a barrier to entry into the procurement process as it provides a disincentive for suppliers to enter.

‘Local requirement’ provides another key area for increased competition. By placing input restrictions on suppliers, a government faces the potential situation of suppliers not tendering for work if they are not allowed to use their own best-practice solutions to provide the good or service. We have therefore identified this as an area that can affect competition in the process, with it being quantified by the percentage of tenders that require any local or regional content.

Finally ‘non discrimination’ refers to placing restrictions on firms entering bids based on their home country or region. Once again this reduces the pool of potential suppliers into the procurement process, reducing the possibility of achieving the best value-for-money outcome. As an indicator, we have identified the percentage of tenders that do not allow bids from foreign firms.

3.4 Principles to indicators: competition


Entry barriers Payment to submit tender document

Per cent of tenders requiring a submission payment

Local requirement Local work inputs required Per cent of tenders that require any local or regional requirement such as labour and materials

Non discrimination Nationality Restrictions Per cent of tenders that disallow foreign bids

Source: TheCIE.

Principle: accountability

‘Accountability’ refers to the government’s responsibility to stakeholders for their procurement activities. Given the large quantities of public money that is expended through government procurement, governments must be accountable for their procurement processes and activities. Accountability means that officials are responsible for the actions and decisions that they take in relation to procurement and for the resulting outcomes. Officials are answerable for such activity through established lines of independent review and legal requirements.

Accountability is important for the government procurement process as it provides a level of insurance (or certainty) for potential suppliers, that rules, regulations and processes set out are in fact being engaged and implemented. To ensure this is the case, it requires the existence of external independent bodies and robust legal frameworks that can, after the


fact, review the methods used and to legally enforce any changes or lapses of process. As such, accountability can be broken down into five broad categories (table 3.5), which are outlined below.

Oversight relates to the existence and effective utilisation of a single entity that handles all government procurement contracts. In doing so one standard set of rules and regulations can be implemented across all procurement needs. This has the advantage of providing certainty to suppliers. Suppliers know who handles the initial procurement requests, what the rules are and that those rules will be implemented across the board.

Independent Review refers to the existence of independent bodies that are trusted with the task of providing impartial review of processes and procedures. Examples include the existence of an Audit Office, Auditor General and parliamentary review procedures. In providing a mechanism by which officials, employees and process are held responsible for their actions.

Contract enforcement is fundamental to accountability. If legally binding agreements can be broken at any time by any party, for any reason without due recourse, then uncertainty will prevail about contracts entered into. This would be reflected in an increased price to government to reflect the increased risk to suppliers. For effective contract enforcement, robust legal institutions are crucial.

If, for any reason, questions and queries arise about the procurement decision taken, an appeals process can provide another level of assurance that process and procedure was followed. In doing so, the appeal process provides certainty to suppliers.

3.5 Principles to indicators: accountability


Oversight Single, visible procurement entity

Independent review Independent budgetary and legislative review bodies

Contract enforcement Strength of Legal Institutions Heritage Foundation Property Rights score

Appeals process Formal Dispute resolution process

Predictability Public announcement of contract awards

Source: TheCIE.


Finally, predictability refers to the consistent implementation of rules and processes surrounding the government procurement decision. Again, this provides a level of assurance to potential suppliers that rules are being followed, reducing uncertainty in the decision making process.

Unlike the previously outlined principles, many of the indicators for accountability are binary indicators. That is, you either have them or you don’t. To develop an indicator of accountability a checklist could be implemented, with a percentage score attained from it. An illustrative example is provided in table 3.6.

3.6 Accountability — institutional checklist

Objective Yes No

Existence of a single procurement oversight organisation ×

Existence of an Auditor General and/or Internal Audit office ×

Formal Dispute resolution commission ×

Public announcement of awarded contracts ×

Score ¾ = 75

Source: TheCIE.

Principle: fair dealing

‘Fair dealing’ means all potential suppliers are treated by government and government employees in an even manner. No policies, procedure or practices should be adopted to afford an advantage to specific suppliers.

3.7 Principles to indicators: fair dealing


Corruption Level of Corruption Corruption Perceptions Score

Freedom from Corruption Score

Ethics Ethics Quality Checklist Per cent score obtained from a predetermined checklist

Source: TheCIE.

As part of the indicator for fair dealing, we have identified two main areas that can provide an indication to the quality of fair dealing. Corruption is a major detriment to achieving value-for-money. It diverts resources away from other areas of need and ultimately drives up the cost of procurement as suppliers will build the cost of corruption into any contract tender. As indicators of corruption, Transparency International produces a score (out of 10) of the corruption perceptions for each country. The rationale for a score of corruption perceptions is encompassed by this quote ‘perceptions are realities’. In other words, if there is a perception of corruption in an economy, then regardless of its fact, a premium will be attached to the price of the good or service supplied. Similarly, the Heritage Foundation produces a score (out of 100) of corruption perceptions for each country in the world.


Ethics enhances value-for-money by providing a framework by which employees act in good faith. It provides confidence in the system, which reduces the risk element in the process. This reduction is reflected in the price to government.

To provide an indication of ethics we have constructed an Ethics quality checklist. This is by no means an exhaustive list. A percentage score of the checklist would be used in this case. Table 3.8 provides an example of the checklist and how it could be used to score the issue of ethics within a government’s procurement process.

3.8 Ethics quality checklist

Objective Yes No

Policy/laws on privacy and confidentiality with tender information ×

Treat all submissions in confidence ×

Secure (electronic) audit trail ×

Ensure that all tenders are treated equally, with no one tender given an advantage not afforded to all other

×

Score ¾ = 75

Source: TheCIE.

Principle: efficiency

‘Efficiency’ means the procurement process is achieved in the most cost effective manner for both government and suppliers.

Efficiency of the process is an area not originally identified by the GPEG NBP, however, we believe it to be important. An inefficient process can add unnecessary costs to a supplier by overburdening them with procedures.

‘Complexity’ and ‘timeliness’ of the process have been identified as the two areas for measuring efficiency of the process (table 3.9). Complexity is measured by the number of steps in the process to awarding a contract. We recognise that there are an optimum number of steps that should be in place. Having no steps provides uncertainty to the process as no clear decision path is followed. However, a large number of steps can make the process cumbersome and costly.

‘Timeliness’ refers to the time period for the tender decision making process to be completed. Once again, there is an optimum level here, which is also dependent upon the type of contract being evaluated. The process should allocate enough time to properly consider all tender facts and options. However, you do not want procrastination to prevail either as it increases the uncertainty in the process.


3.9 Principles to indicators: efficiency


Complexity Steps in tender process Number of steps in the process to award a contract

Timeliness Time to award contract Number of days to award contract

Source: TheCIE.

Collection of the data

As with any measurement, there are problems with data. The way things are done and recorded in one country does not necessarily mean that is the way they are done in another country. Although using existing data can be quite informative within a country, it provides little help in providing a meaningful within country comparison across time.

A basis of comparison needs to be established. If a proper internal comparison across time is to take place, the data being used for the comparison needs to be standardised. Therefore, the data required to construct these indicators should be based upon the construction of a hypothetical government procurement case. The suggestion here is that both a good and a service be assessed to ensure consistency across countries (and time) and to minimise the cost of collection.

It is important to emphasise that the value of the exercise would be comparisons over time for each APEC member. The nature of the measurement of the FPIs means they should not be directly compared across APEC economies. What would be important over time however is which APEC economies are making the most improvement towards better government procurement and what are the lessons that could be shared across APEC in that progress

The hypothetical contracts assessed, would need to be a standard good or service that is found in every member economy with little variation between member nations. Examples of goods and services that could provide reasonable hypothetical cases are presented below in table 3.10

3.10 Examples of hypothetical contracts to allow consistent data collection

A good A service

Supply of IT hardware (e.g. Desktops) Auditing services

Office supplies IT desktop support

Purchase of car fleet Legal services

Source: TheCIE.


4 How would indicators be implemented?

Value-for-money has been the major theme so far throughout this report, it is the objective that government should strive for within public spending. The process of procurement is one of the conduits by which this can be achieved. However, moving towards a better use of public money is unrealistic if the process is not quantitatively measured. If the performance of an activity is not measured, it cannot be effectively managed nor can continuous and sustainable improvements be made.

This chapter moves one step further in measuring the procurement process by taking the framework just presented forward and applying it to member economies to show how government procurement processes could be measured. In other words, we begin to demonstrate how the quantification of the procurement process might work.

Previously, a number of indicators and data for those indicators for the five principles of the government procurement process were outlined, and these remain our preferred indicators for process measurement. However, data pertaining to many of the indicators previously outlined are not readily available. There is, however, a large amount of existing data from other independent sources that can be used to give APEC member economies an indication of their current government procurement process.

There are a number of external sources of data provided by multinational organisations that relate to a number of the principles of good procurement and could be used initially as a proxy to measure and cross reference performance, before a more comprehensive study is undertaken. An advantage that this existing data has is that much of it has been collected for many years and thereby provides a time series of data allowing for internal comparison across time.

Examples of what data is currently publicly available to proxy for the indicators include:

World Bank, Doing Business

World Bank, World Investment Climate Survey

Heritage Foundation, Index of Economic Freedom

Fraser Institute, Economic Freedom of the World

Transparency International

Freedom House Organisation.

As an illustrative example, some of the data has been used in this paper as a first step in developing tangible quantitative indicators of the process of government procurement. It


provides an illustrated example of how the measurement indicators would be used. It also provides member economies with a useful tool for internal benchmarking over time.

Finally, it should be noted that this illustrative use of proxy indicators should only been seen as a first step in casting some light on the process of government procurement. The indicators chosen here are by no means the final product. The reason for using these proxies is that data for our preferred indicators are not readily available. As a first step the proxies provide us with an illustration of the usefulness of such an exercise and what lessons and insights might be gained. As a future step, it would be essential for governmental agencies across the APEC region to begin collecting data pertaining to our preferred indicators. In doing so, a more nuanced approach and understanding can be achieved.

Moving towards an index for overall government procurement

This section provides an illustration of how the proxy measures could be used to develop an overall indicator of government procurement for each member across the APEC region. Each principle is presented along with the proxy indicators used, with a brief discussion on each. For completeness, indicator tables are presented in the appendix at the end of the report. They provide an easy to use summary of the overall proxy measures of government procurement indicators and the five principles adopted by APEC member nations.

Transparency

Recall that transparency can be broken down into three areas: opportunities for access, rules and availability of criteria. The indicators and the data presented are not readily available, although these should be attainable under a more comprehensive in-country review. Therefore to measure transparency and compare across the APEC region, we use existing data as a proxy for the three areas.

Opportunities for access and criteria availability are proxied by the use of the Freedom House’s Freedom of the Press Index (FPI). Our rationale for using this score is that a free press allows for easy advertisement of tenders and accessibility of tender criteria by all potential suppliers. The FPI is a score out of 100 with a low score representing greater press freedom.4 Administrative requirements, which is a component of the Fraser Institute’s Index ‘Regulation of Credit, Labour and Business’ is used as a proxy for ‘rules’. The index measures the extent of rules and regulation that surround business.

Table 4.1 is provided as a summary of the principle, the areas underlying it and the proxy measures used.

4 For a detailed discussion of the Freedom House Freedom of the Press index, please consult their

website: http://www.freedomhouse.org/


4.1 Principles to indicators: transparency

Areas Proxy Indicator

Opportunities for access Freedom of the Press Index

Rules Administrative requirements Index

Criteria availability Freedom of the Press Index

Note: Freedom of the Press index provided by Freedom House. Administrative requirement index provided by Fraser Institute. Source: TheCIE.

Competition

Competition was broken down into the following important areas, Entry Barriers, Local Requirement and Non-discrimination. As a proxy for the principle of competition, a composite index of several data sources was used. Table 4.2 below summarises the data used.

Although not a complete proxy index, by trying to capture how easy it is to do business within and across countries, some overall ‘competition’ qualities of an economy will be encapsulated within it. In other words, if an economy is willing to engage and encourage competitive behaviour within and outsides its borders, it is much more likely to engage in competitive practices in its government procurement policy. Again, whilst a specific indicator could and should be developed for ‘competition’, the best proxy available in the absence of an ideal indicator is likely to be some composite of the data indicated in table 4.2.

4.2 Principles to indicators: competition

Proxy Indicator Source

Ease of starting a business rank World Bank; Doing Business

Ease of closing a business rank World Bank; Doing Business

Trading Across borders Rank World Bank; Doing Business

Trade Freedom Index Heritage Foundation

Business Freedom Index Heritage Foundation

Source: TheCIE.

Accountability

Accountability relates to the responsibility government and stakeholders have in regards to the expenditure of public money to the public at large. In providing a framework for accountability, a robust set of institutions need to be in place. As such, to measure the principle of accountability a number of proxy indicators from different independent sources are used. These indicators relate to the strength of legal institutions and contract enforcement in APEC member economies. A list is provided in table 4.3


4.3 Principles to indicators: accountability

Proxy indicator Source

Property Rights Score Heritage Foundation

Enforcing Contracts World Bank: Doing Business

Judiciary Independence Fraser Institute

Impartial Courts Fraser Institute

Source: TheCIE.

Fair dealing

Corruption is harmful to value-for-money in procurement because it does not automatically reward the best bid, but rather the contractor with the most ‘influence’. As such it causes uncertainty which is reflected in costs and it deters better bids by alternative suppliers. It wastes resources that could be better used elsewhere.

A suggested measurement for the level of corruption is the rank of the arithmetic average of two exiting indices, namely Transparency International’s Corruption Perceptions Score (CPS) and The Heritage Foundation’s Freedom of Corruption Index (FCI) — table 4.4.

These indices in table 4.4 include other areas of an economy other than government procurement. But, as government procurement provides an opportunity for personal gain, if corruption exits in other areas of government and the economy, it is most likely to exist in the government procurement process too. In the absence of specific indicators of corruption in government procurement, the proxy indicator is likely to be positively correlated with the ideal measure.

Taking these two indicators provides an illustrated example of how the principle of fair dealing might be scored across three hypothetical APEC economies. It is a hypothetical case only, its purpose being to highlight the methodology and provide a ‘how to’ for member economies wanting to measure the principle of fair dealing. The example illustrates how well the principle of fair dealing is performed in these economies.


4.4 Principles to indicators: fair dealing


Corruption Perceptions Index Transparency International

Freedom of corruption index Heritage Foundation

Source: TheCIE.

Efficiency

Efficiency of the procurement process is important because if procurement is cumbersome, regulations and requirements onerous and/or time costly for suppliers, then the process can unnecessarily burden potential suppliers.

Therefore we have used the Fraser Institute’s Time with Government Bureaucracy Score (TGB), which is a component of their Economic Freedom of the World Reports as a proxy for efficiency. Effectively it measures the time taken by senior management in dealing with government bureaucracies. It is scored out of 10.5

4.5 Principles to indicators: efficiency


Time with Government Bureaucracy Score Fraser Institute, Economic Freedom of the World Reports

Source: TheCIE.

Quantifying the process of government procurement: an illustrative example

Now that a framework for measuring the principles of government procurement has been outlined, the next step is to start quantifying the process of government procurement. Following that, an overall index of government procurement is constructed.

Each APEC economy’s score for each of the five principles is calculated and ranked. Measuring the principles in their own right can be informative, however, it may not provide an indication of how the entire process is operating. But, the advantage of scoring the principles individually is that it provides a way of ‘unpicking’ the effectiveness of the government procurement process to its component parts to test where best progress might be made and lessons learned for other APEC economies.

By adding together the ranks of all the principles for each APEC economy, and ranking that score, an indicator of each member economy’s government procurement process can be made. Effectively, what has been calculated is a ‘rank of rank’ score. The illustrative results for 2007, including the breakdown by individual principle, are presented in the Appendix.

5 For a detailed discussion of the Government Bureaucracy score, in conjunction with the Economic

Freedom of the world scores, please consult their website: http://www.freetheworld.com/reports.html


There are many caveats with such a process, but the methodology is akin to that developed by the World Bank in its Doing Business report.

A ‘point in time’ analysis, although providing useful information, is not the most powerful use of the data. A better indication of progress in improving government procurement is obtained by analysing the changes within an economy over time. Although not presented, an illustrative proxy index for government procurement in 2004 was also calculated to provide variation across time and a means by which to better understand changes in government procurement.

Below are a set of graphs, by no means an exhaustive list, which utilises this data and casts a first light on the issue of government procurement performance. What are the main messages from this type of analysis? One message that can be obtained is how each APEC economy has gained in government procurement.

Chart 4.6 provides an illustration of the number of APEC member economies that have changed their government procurement practices. The graph shows the changes in the overall government procurement index from 2004 to 2007 based on the proxy data. Overall, the graph highlights that a number of APEC member economies have made improvements in their government procurement processes, but only a few have made significant improvements.

4.6 A few economies have made significant gain in government procurement: illustrative example from proxy data

0 1 2 3 4 5 6 7 8 9

Number of beneficial changes made in government procurement practices

21 A

PEC

Memb

er ec

onom

ies

a Changes made in government procurement performance relative to other APEC economies. Data source: CIE calculations from multiple data sources.


We now know that few economies have made significant gains in their government procurement processes, but it is also important to know what lessons can be learnt. That is, where are the improvements in the process being made? The advantage of constructing the data in this way is that it can be broken down into the principles. By drilling further down into the data a more nuanced understanding can be obtained where most of the changes were made.

Chart 4.7 provides a further breakdown of the data, by the APEC guiding principles on government procurement, and illustrates where the most changes were made. The chart indicates that many of the APEC economies have made at least some improvement in the different facets of the government procurement process, with only a few making no real improvement. This is true across all the guiding principles except for efficiency.

At this point it is worth noting that although the graph indicates that a number of economies ‘seemed’ to have gone backwards, it is important that the interpretation of this be clarified. The negative results are very much an artefact of the ranking system. When ranking within a defined range, if one economy makes positive progress, everything else being equal, then another must slip down the index. That is not to say the economy has gone backwards, it may be a case that the economy has made positive changes but relative to other economies, has made less positive changes. Another interpretation may be that an economy has made no improvement at all and has effectively ‘marked time’, but not regressed.

4.7 Most economies have made some improvements in certain aspects of the government procurement process: illustrative example from proxy data

Improvements made

No improvement

Transparency Competition Accountability Fair dealing Efficiency

18 14 14 18 8

3 7 7 3 13

a The number of APEC economies that have seen improvements in the government procurement process. Source: The CIE.

Another advantage of this quantification is that it can provide much more insight into the changes that the top reformers are making and what lessons can be learnt from them. Chart


4.8 illustrates where progress has been made by the top five reformers in the government procurement process.

4.8 The top five reformers have made most gains in transparency and accountability but no improvement in efficiency: illustrative example from proxy data

Transparency Competition Accountability Fair dealing Efficiency

2004

2007

No significant

improvement

70% Improvement

8% Improvement

61% Improvement

15% Improvement

a Where the improvements were made: Guiding principles and the average gains by the top5 procurement performers. Source: The CIE.

Of the top five reformers in government procurement, two areas stand out. They are transparency and accountability where 70 per cent and 61 per cent of improvements were made between 2004 and 2007. The main messages that can be taken from the graph are that reforming the areas of transparency and accountability in the government procurement process can have beneficial outcomes. Another interesting message that the graph highlights is that, although progress was made in transparency and accountability, less progress was made in the area of efficiency. This remains a potential area of further reform .

Although a majority of economies are making some positive changes, what is the overall result of those changes? As a relative measure, the data can be used to highlight the changes compared to other APEC member economies. Chart 4.9 below provides an illustration of this. For example, two economies have made significant positive progress in their overall government procurement practice between 2004 and 2007 as measured by the proxy indicators. Similarly, four economies have made moderate progress, three have made slight improvements while the rest have made little improvement. The slight improvement by twelve economies is not necessarily a bad thing. If these twelve economies are already adhering to world best practice then any gains in the procurement process are incremental compared to those economies that are starting from a lower base.


4.9 Most economies have made improvements in some procurement facets; overall the result is little change: illustrative example from proxy data

0 2 4 6 8 10 12 14

Significant

Moderate

Slight

Little

Number of economies that have made relative changes

a Relative measure of the number of APEC economies that have made changes. Note: ‘Little’ includes those economies that have made zero changes. Data source: Results are illustrative only and are based on proxy data from various sources.

Finally, by introducing new data from other organisations, more lessons in government procurement can be obtained. Chart 4.10 presents the results of the overall government procurement index against the per capita income of each of the member economies based on data from the IMF.6 The interpretation of this scatter diagram is straightforward. On the vertical axis we have the government procurement index for 2007, which is plotted against income as measured by per capita GDP. This graph, and the upward sloping trend line, illustrates a significant point. There is a strong positive correlation between good government procurement practices and national prosperity. In other words, those economies that score highly on the government procurement index are more likely to be high income economies. This reinforces the notion that good practices in government expenditure can have significant positive impacts on economic outcomes.

6 The Per Capita GDP data is for 2007 (PPP)


4.10 Better government procurement processes across APEC is associated with higher income

0

10

20

30

40

50

60

70

80

90

100

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Income - per capita GDP in 2007

Gove

rnme

nt P

rocu

reme

nt In

dex ..

Least

Best

a Based on 20 APEC economies due to data availability.

Data source: CIE calculation based on various data sources; GDP data sourced from the IMF.

To summarise, although only an illustrative exercise, there are many advantages to constructing a measurable index of government procurement because ‘you can’t manage what you don’t measure’. By comparing procurement results across time, lessons can be learnt on where to improve, how to improve and where the biggest returns from reform lie in improving the process of government procurement. Also, such an exercise provides the flexibility of augmentation with other data to provide even deeper insights into the process of government procurement and where to look for best practice guidelines.


5 Conclusions and next steps

This report has illustrated what the key principles of the government procurement process are and how measurement of these principles can be established to improve the value-for-money that government receives from the public spend.

It is possible to come up with a measure of value-for-money in government procurement. Such a measure has been developed from selectively using proxy indicators from other sources such as the World Bank’s Doing Business report or selective use of some of the indicators used by the Fraser Institute to develop its Economic Freedom Index

Using these proxy indicators the report shows how the information on value-for-money can be used in two ways. First, it can be used in a cross-sectional way to highlight where best practice might provide valuable lessons for other APEC economies to consider when making improvements to government procurement.

The second way the information can help is to examine the changes within an economy over time. Often it is not the cross-economy ranking that is important, but the changes within an economy over time. The aim is to improve the efficiency of government procurement — of getting more from less or value-for-money. Changes over time for each APEC economy will show which economy has made the most improvement in what area. Drilling down into the reasons for the improvement, how hard it was and the possible benefits can provide valuable lessons for other APEC economies to consider. It is this shared experience that can be most valuable for APEC economies.

Over time, a third more subtle lesson can be gained. If measurement shows an economy can do better in achieving value-for-money, it must be that either:

they are not aware of the scope for improvement, perhaps because it has not been measured before;

they face some political obstacle to implement desired changes; or

they may be using government procurement to pursue some other goals besides value-for-money.

On the last point, sometimes government procurement has been used to pursue other policy objectives in addition to procuring a good or service, such as:

the development of infant local industries

providing regional assistance to less-developed areas

equality for smaller firms that struggle to compete with their larger counterparts


promoting domestic producers

promoting wider national interests including security considerations

other social policy objectives.

Measurement of government procurement processes will help clarify just what government policies are for what purpose. It may well be that these other objectives, such as developing infant industries, could be better pursued through more direct policies than by directing government procurement policies to that end. Either way, more information will allow more informed and better decisions to be made.

Drawing on this framework for measuring public sector procurement, we believe that the approach outlined can be tailored and targeted to meet the needs of procurement performance measurement. It will provide an overview of all individual economies in the APEC region and will allow for effective benchmarking across economies, whilst being flexible enough to be tailored to the individual needs of each member economy.

It is, however, important to avoid measurement overload. The correct balance between easy and informative measures and timely collection needs to be struck, remembering that measurement has a cost.

With these caveats it is recommended that a standard measurement system be developed to provide measurements. However, to fully utilise the framework presented in this report, it’s necessary to build on the foundations presented here. The best way to refine and develop the measures suggested in this report is to put them into practice. This would best be done for say four or five APEC nations that span the range from developed to developing economies where the measurement issues may be more difficult. Attempting such measurement will show up difficulties and suggest refinements so better, easier indicators can be developed.

Once a comprehensive set of measures have been trialed in four or five APEC economies, the overall messages can be cross-correlated with the proxy measures used in this study to see if the messages are consistent with prior knowledge about government procurement practices. That information and experience will also help form a judgment about the resources required to conduct a comprehensive study across APEC and the likely benefits to be gained. Armed with this information a more informed decision could then be made about the value of improving government procurement across APEC economies. The choice is really one about the extent of the measurement of procurement processes since some objective measurement will be required if APEC economies want to improve the efficiency of government procurement. It is not possible to effectively manage something like improving procurement if it is not measured. The study has proposed a way forward to develop a set of measures or indicators that will bring some objectivity to improving government procurement across APEC economies.


References

APEC’s Government Procurement Experts Group (GPEG), http://www.apec.org/apec/ apec_groups/committee_on_trade/government_procurement.html

Fraser Institute 2007, Freedom of the World Report, http://www.freetheworld.com/ reports.html.

Freedom House 2007, Freedom of the Press Index, http://www.freedomhouse.org/.

Heritage Foundation 2007, Index of Economic Freedom, http://www.heritage.org/Index/.

Transparency International 2007, Corruption Perception Index, http://www.transparency. org/.

The World Bank 2004, Doing Business 2004: How to reform, comparing regulation in 175 economies, Washington DC.

The World Bank 2007, Doing Business 2007: How to reform, Comparing regulation in 175 economies, Washington DC.

A p p e n d i x

ERROR! NO TEXT OF SPECIFIED STYLE IN DOCUMENT. 51

www.TheCIE.com.au

A APEC member economy Government Procurement Index Tables 2007

Australia

Government Procurement Index 3

Transparency (rank) 4 Fair Dealing (rank) 5

Freedom of the Press index 18 Corruption Perceptions Score 8.5

Administrative Regulations 3 Freedom from Corruption Index 88

Competition (rank) 5 Accountability (rank) 1

Ease of setting up a business 2 Property Rights Score 90

Ease of closing a business 12 Enforcing Contracts Score 7

Trading across borders 7 Judiciary Independence Rank 9

Trade Freedom Index 89 Impartial Courts Rank 8

Business Freedom Index 84

Efficiency (rank) 10

Time Spent with Bureaucracy Index 5

Brunei Darussalam

Government Procurement Index NA

Transparency (rank) NA Fair Dealing (rank) NA

Freedom of the Press index Corruption Perceptions Score

Administrative Regulations Freedom from Corruption Index

Competition (rank) NA Accountability (rank) NA

Ease of setting up a business Property Rights Score

Ease of closing a business Enforcing Contracts Score

Trading across borders Judiciary Independence Rank

Trade Freedom Index Impartial Courts Rank

Business Freedom Index

Efficiency (rank) NA

Time Spent with Bureaucracy Index Note: NA = No available data

52 ERROR! NO TEXT OF SPECIFIED STYLE IN DOCUMENT.

www.TheCIE.com.au

Canada











Efficiency (rank) 3

Time Spent with Bureaucracy Index 6.4

Chile



Freedom of the Press index 24 Corruption Perceptions Score 7








Efficiency (rank) 7



www.TheCIE.com.au

China, (People's Republic of)













Chinese Taipei











Efficiency (rank) 2



www.TheCIE.com.au

Hong Kong, China



Freedom of the Press index Corruption Perceptions Score 8.3










Indonesia











Efficiency (rank) 8



www.TheCIE.com.au

Japan











Efficiency (rank) 1


Korea, (Republic of)











Efficiency (rank) 5



www.TheCIE.com.au

Malaysia













Mexico














www.TheCIE.com.au

New Zealand



Freedom of the Press index 12 Corruption Perceptions Score 9

Administrative Regulations 4.3 Freedom from Corruption Index 96







Efficiency (rank) 6


Papua New Guinea



Freedom of the Press index Corruption Perceptions Score

Administrative Regulations Freedom from Corruption Index


Ease of setting up a business 69 Property Rights Score


Trading across borders 52 Judiciary Independence Rank

Trade Freedom Index Impartial Courts Rank 3.9

Business Freedom Index


Time Spent with Bureaucracy Index


www.TheCIE.com.au

Peru













Philippines














www.TheCIE.com.au

Russia













Singapore











Efficiency (rank) 3



www.TheCIE.com.au

Thailand













United States

Government Procurement Index (rank) 5



Administrative Regulations 4.5 Freedom from Corruption Index 76







Efficiency (rank) 5



www.TheCIE.com.au

Viet Nam

Government Procurement Index (Rank) 18












developing tangible measures of government procurement...

Documents