independent evaluation group (miga) 2006 annual report€¦ · independence, which are also...

Independent Evaluation Group (MIGA) 2006 Annual Report

May 24, 2006

Document of the Independent Evaluation Group (MIGA) (IEG-MIGA)

INDEPENDENT EVALUATION GROUP

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

wb350881

Typewritten Text

53813

ii

Abbreviations and Acronyms AR Annual report AMS Agribusiness, manufacturing and services CDP Community development plan CODE Committee on Development Effectiveness EA Environmental assessment EAP Environmental action plan EHS Environmental, health & safety EI Extractive industries EMP Environmental management plan EMS Environmental management system ERR Economic rate of return EVP Executive Vice President FDI Foreign direct investment FIAS Foreign Investment Advisory Service FRR Financial rate of return FY Fiscal year IDA International Development Agency IDS Information dissemination services IEG-WB Independent Evaluation Group (WB) IEG-IFC Independent Evaluation Group (IFC) IEG-MIGA Independent Evaluation Group (MIGA) IFC International Finance Corporation IPA Investment promotion agency IPI Investment promotion intermediary IPP Indigenous peoples plan MATR Management Action Track Record MIC Middle income country MIGA Multilateral Investment Guarantee Agency OED Operations Evaluation Department (former name for IEG-WB) OEG Operations Evaluation Group (former name for IEG-IFC) OEU Operations Evaluation Unit (former name for IEG-MIGA) PAP Project affected people PSD Private sector development RP Resettlement plan RDE Review of Development Effectiveness in MIGA (OEU, 2004) RRU Rapid Response Unit (The World Bank, Private Sector Development Vice Presidency) SOE State owned enterprise TA Technical assistance TRS Time recording system UNCTAD United Nations Conference on Trade and Development UNIDO United Nations Industrial Development Organization WACC Weighted average cost of capital WBG World Bank Group (IBRD/IDA, IFC and MIGA)

iii

Acknowledgments This report was written by Ethel I. Tarazona, Stephan Wegner, and Aurora Medina Siy under the general supervision of Aysegul Akin-Karasapan. Roger Batstone and Ian Webb contributed to the analysis of this report. Carlos Nuñez and Bryan Towe provided inputs and research assistance. The authors would like to thank MIGA staff who provided valuable feedback on IEG’s individual project evaluations. Director-General, Evaluation: Mr. Vinod Thomas Director, Independent Evaluation Group (MIGA): Ms. Aysegul Akin-Karasapan Task Manager: Ms. Ethel I. Tarazona

iv

Table of Contents

Page

1. PURPOSE, SCOPE, AND APPROACH....................................................................................1

A. Background............................................................................................................................................1 B. Ex-Post Guarantee Evaluations.............................................................................................................1 C. Evaluation of Information Dissemination Services...............................................................................3 D. Status of Previous IEG Recommendations ...........................................................................................3

2. DEVELOPMENT OUTCOME ................................................................................................ 5

A. Business Performance............................................................................................................................6 B. Contribution to Economic Sustainability ..............................................................................................9 C. Private Sector Development Impact .................................................................................................... 12 D. Environmental and Social Effects ....................................................................................................... 13

3. MIGA’s EFFECTIVENESS .....................................................................................................22

A. Consistency with Strategy ....................................................................................................................23 B. MIGA’s Role and Contribution............................................................................................................23 C. Adequacy of Project Assessment, Underwriting, and Monitoring ......................................................24 D. Project Development Outcome and MIGA’s Effectiveness................................................................27 E. Improvements in MIGA’s Quality At Entry ........................................................................................28

4. CONTRIBUTION TO MIGA’S PROFITABILITY ...............................................................31

A. Background and Objective. ................................................................................................................. 31 B. Findings ...............................................................................................................................................34

5. EVALUATION OF MIGA’S INFORMATION DISSEMINATION SERVICES (IDS) .......36

A. Background..........................................................................................................................................36 B. Evaluation Methodology......................................................................................................................38 C. Evaluation Findings.............................................................................................................................38

6. STATUS OF IMPLEMENTATION OF PREVIOUS IEG-MIGA RECOMMENDATIONS ...............................................................................................................45

7. RECOMMENDATIONS .........................................................................................................47

v

FIGURES Figure 1. Distribution of Development Outcome Ratings ..................................................................6 Figure 2. Average Development Outcome Ratings...............................................................................6 Figure 3. Average Development Outcome Ratings in MIC and IDA Countries .............................6 Figure 4. Average Development Outcome Ratings, by Region...........................................................6 Figure 5. Business Performance Ratings .................................................................................................7 Figure 6. Contribution to Economic Sustainability Ratings...............................................................10 Figure 7. Business Performance and Contribution to Economic Sustainability .............................11 Figure 8. Private Sector Development Impact Ratings.......................................................................12 Figure 9. Environmental and Social Effects Ratings (Eighteen Projects*) ..........................................14 Figure 10. Summary Safeguard Performance Criteria at Approval (18 projects)...........................16 Figure 11. Summary Safeguard Performance Criteria at Evaluation (18 projects) .........................17 Figure 12. Comparison of Compliance Ratings at Approval and Evaluation (Category A

and B projects) ........................................................................................................................19 Figure 13. Selected Safeguard Performance Criteria at Approval and Evaluation .........................20 Figure 14. Overall MIGA Effectiveness Ratings .................................................................................22 Figure 15. Average MIGA Effectiveness Ratings................................................................................22 Figure 16. Consistency with Strategy Ratings.......................................................................................23 Figure 17. MIGA’s Role and Contribution Ratings ............................................................................24 Figure 18. Adequacy of Project Assessment, Underwriting, and Monitoring Ratings...................25 Figure 19. Development Outcome and MIGA’s Effectiveness ........................................................27 Figure 20. Development Outcome and MIGA’s Assessment, Underwriting, and

Monitoring...............................................................................................................................27 Figure 21. Comparison of Aspects of Quality at Entry Development Impact Assessments........28 Figure 22. MIGA Quality Management Steps.....................................................................................30 Figure 23. Development Outcome and Contribution to MIGA’s Profitability ..............................34 Figure 24. MIGA’s Exposure and Contribution to Profitability .......................................................35 Figure 25. Development Outcome and the Size of MIGA’s Exposure...........................................35 Figure 26. MIGA’s IDS and Other FDI-related Online Services .....................................................40 Figure 27. Page Requests by Service 2003-2005 (By Quarter) ...........................................................41 TABLES Table 1. Summary View of MIGA’s Information Dissemination Services (IDS) ..........................37 BOXES Box 1. Investment Disputes in Evaluated Projects ...............................................................................9 Box 2. How does IEG Assess Safeguard Consistency?......................................................................14 Box 3. Evolution of MIGA’s Quality Management Processes and Systems ...................................30 Box 4. Methodology for Assessing Contribution to MIGA’s Profitability......................................33

vi

ANNEXES Annex 1. IEG-MIGA Methodology for Evaluating Guarantees Project Ratings Guidelines .....51 Annex 2. IEG-MIGA Methodology for Evaluating Guarantees MIGA Safeguard Policies.

Criteria for Consistency .........................................................................................................55 Annex 3. IEG-MIGA Ex-Post Guarantee Project Sample Composition........................................60 Annex 4. FY96-FY2002 Evaluated MIGA Projects Composition of Evaluated Projects ............61 Annex 5. FY96-FY02 Evaluated MIGA Projects Summary of Evaluation Ratings.......................62 Annex 6. Safeguard Policy Consistency Ratings of MIGA Projects at Approval...........................63 Annex 7. Safeguard Policy Consistency Ratings of MIGA Projects at Evaluation ........................64 Annex 8. Development Outcome Results for Projects Evaluated During FY04-05 (Twelve

Evaluated Projects).................................................................................................................65 Annex 9. MIGA’s Effectiveness Results for Projects Evaluated During FY04-05 (Twelve

Evaluated Projects).................................................................................................................66 Annex 10. Potential and Actual Risks Affecting Evaluated MIGA Projects at

Underwriting and at Evaluation ...........................................................................................67 Annex 11. Methodological Note for Evaluating MIGA’s Information Dissemination

Services.....................................................................................................................................69 Annex 12. Survey Administered to IDS Users.....................................................................................77 Annex 13. Profiles and Findings for each Information Dissemination Service..............................83 Annex 14. Survey Results on IDS Impact ..........................................................................................101

1

1. PURPOSE, SCOPE, AND APPROACH

The objectives of the 2006 IEG-MIGA annual report are to inform CODE and the Board on MIGA’s development effectiveness, to provide feedback on evaluation findings to MIGA management and staff, and to inform the Board discussion of MIGA’s FY07 Business Plan and Administrative Budget. The report presents evaluation findings on MIGA’s guarantees and information dissemination services, and provides an update on the implementation of previous IEG recommendations.

A. BACKGROUND

1.1 Since 2003, the Independent Evaluation Group (MIGA) (IEG-MIGA,1 formerly OEU) has prepared annual reports for discussion by the Committee on Development Effectiveness (CODE) and the Board of Directors. The first report in 2003 focused on the status of quality assurance and evaluation systems in MIGA,2 and the second (2004) on the development effectiveness of MIGA guarantees and technical assistance.3 The 2005 Annual Report focused on the status of MIGA’s quality assurance, monitoring, and evaluation systems and processes relative to the achievement of its development mandate.4 It reported on the changes implemented by MIGA in the first half of FY05 and identified areas MIGA needed to address as part of its new strategy and business model.

1.2 The 2006 Annual Report (2006 AR) has three main components: (1) results of ex-post evaluations of a sample of MIGA guarantee projects; (2) findings from evaluation of MIGA’s online information dissemination services; and (3) the status of the implementation of previous IEG recommendations.

B. EX-POST GUARANTEE EVALUATIONS

1.3 The 2006 AR synthesizes the results of evaluations of MIGA guarantee projects undertaken by IEG-MIGA in fiscal years (FY) 2003 to 05. It presents aggregate results of 21 ex-post evaluations of MIGA guarantee projects underwritten between FY96-FY02, drawing emerging lessons and highlighting areas requiring attention by MIGA management. Given the relatively small number of ex-post evaluations covered, the report does not draw conclusions on overall portfolio performance. Instead, it focuses on common issues and themes emerging from

1 This is the first annual report since the Board approved changes in the unit’s Terms of Reference to strengthen its independence, which are also reflected in the name change for the three evaluation units of the World Bank Group to Independent Evaluation Group (IEG), highlighting evaluation independence and synergies across the World Bank, IFC, and MIGA. IEG-MIGA will be referred to in this report as “IEG”. 2 2003 Report on Operations Evaluation in MIGA 3 2004 Review of Development Effectiveness in MIGA 4 Operations Evaluation Unit: 2005 Annual Report

2

evaluated projects, also drawing from evaluation findings presented in previous IEG reports.5 The 2006 report addresses the extent to which the evaluated projects and activities have contributed to the achievement of MIGA’s development mandate.

1.4 Project Selection and Characteristics. As required by IEG’s methodology,6 all projects are randomly selected from a population of the remaining active and mature guarantee projects from the specified cohort years. The 21 projects covered in this report were underwritten in FY96-FY02 and evaluated in FY03-FY05.7 (See Annex 4 for project characteristics.)

1.5 Aggregate sample. Since projects are drawn by IEG each year from overlapping populations,8 this report presents findings and identifies themes for all 21 evaluated guarantee projects. Nine of these projects were discussed in the 2004 Review of Development Effectiveness (2004 RDE) and 12 project evaluations are presented for the first time in this 2006 report. A summary of the results for the group of 12 projects not previously presented to CODE and the Board are highlighted in Annex 5. Unless otherwise noted, analyses in this report refer to the aggregate sample of 21 guarantee project evaluations.

1.6 How does IEG evaluate the development effectiveness of MIGA guarantee projects? The IEG methodology endorsed by CODE calls for evaluation of projects in three dimensions: Development Outcome, MIGA’s Effectiveness, and Contribution to MIGA’s Profitability. Until now, IEG has only evaluated the first two dimensions, given data limitations for evaluating the contribution of individual guarantee operations to MIGA’s profitability. The report introduces a preliminary approach to assess this third dimension.

5 In particular, findings from Extractive Industries and Sustainable Development. An Evaluation of World Bank Group Experience are reflected in the discussion of Environmental and Social Effects. 6 IEG-MIGA’s methodology for ex-post evaluation of guarantee projects (endorsed by CODE) is consistent with IEG-IFC’s methodology, dimensions, and benchmarks for the evaluation of IFC investment operations. 7 Evaluated projects are from four regions (Latin America and the Caribbean, Africa, Europe and Central Asia, and Asia and Pacific) and cover three sectors: agribusiness, manufacturing and services (10 projects); infrastructure (8 projects); and financial services (3 projects). Two-thirds are in middle income countries and one-third in IDA and blend countries. 8 The approved methodology required IEG to select projects from active projects underwritten in FY96-98 (FY03 evaluations), FY96-99 (FY04 evaluations), and FY96-00 (FY05 evaluations). IEG draws random samples from eligible populations of active and mature projects from these cohorts, excluding those evaluated in earlier years. For FY03, the population of eligible projects (all remaining active projects underwritten in FY96-98) included 27 projects, of which IEG evaluated nine. For FY04, the eligible population was 35, and IEG evaluated eight. For FY05, the population comprised 35 eligible projects from FY96-00, of which IEG evaluated three. One additional project guaranteed in FY02 was included in this report. On average over these three years, IEG evaluated 23 percent of mature projects that were active at the beginning of each FY of evaluation. In relation to MIGA’s entire portfolio of guarantee projects from FY96-00 (comprising 171 projects, many of which were cancelled by the guarantee holder before reaching maturity for evaluation), IEG evaluations covered 12 percent. (See Annex 3.) It should be noted that in 2005 CODE approved a change in IEG-MIGA’s project selection criteria, shortening the period for a project to become eligible for evaluation from five to three years after issuing the guarantee.

3

1.7 Development Outcome (Section 2) refers to actual project results (on the ground), based on independent reviews of the projects (including financial, economic, environmental, social, and other aspects). Each evaluation includes a site visit by IEG, interviews with investors, project staff, government officials, and other stakeholders to gather updated project information on MIGA projects. IEG independently calculates financial and economic rates of return and evaluates consistency with safeguard policies and environmental guidelines. To complement this view, IEG evaluates MIGA’s Effectiveness (Section 3) by assessing MIGA’s own work in selecting, assessing, underwriting, and monitoring projects. This includes consistency with strategies, the Agency’s role and contribution, and the quality of assessment, underwriting, and monitoring. (For a complete description of IEG’s benchmarks, indicators, and rating scales, please see Annex 1 and Annex 2.) The third dimension, Contribution to MIGA’s Profitability (Section 4), is presented for the first time and on a preliminary basis. It is an attempt to identify important project characteristics that influence profitability at the project level and relies on estimated measures for costs and revenues. Given the limitations of data availability, particularly on costs, the approach provides a ranking of evaluated projects by their relative profitability within MIGA’s overall guarantee portfolio, rather than an absolute measure of profitability.

C. EVALUATION OF INFORMATION DISSEMINATION SERVICES

1.8 The report also presents the findings from an in-depth IEG-MIGA evaluation of MIGA’s online Information Dissemination Services (IDS), completed in FY06, in Section 5. Since 1995, MIGA has offered a suite of Web-based services disseminating information related to FDI. These can be distinguished by the type of content they offer and the users they target. IPAnet, PrivatizationLink, and FDIXchange provide information on the business environment and investment opportunities targeting potential investors, investment advisors, and financial institutions. The FDI Promotion Center is a learning toolkit aimed at investment promotion practitioners, and the website tool IPAWorks is meant to facilitate website development for investment promotion agencies.

1.9 Approach. For this first evaluation of Web-based services, IEG-MIGA developed a new approach jointly with the Independent Evaluation Group (World Bank) (IEG-WB). The evaluation assesses whether the services remain relevant to MIGA’s mandate in a changed business environment. It also evaluates outcomes and impacts of each service, and examines efficiency, sustainability, and MIGA’s work quality. The evaluation consists of a review of MIGA files; interviews with current and former MIGA staff, and staff from investment promotion intermediaries; an analysis of site activity, content, site usability; assessment of comparator websites; and a user survey (see Annex 11.)

D. STATUS OF PREVIOUS IEG RECOMMENDATIONS

1.10 As in past annual reports, the 2006 AR tracks MIGA’s progress in addressing previous IEG recommendations (Section 6). In addition, the Management Action Track Record (MATR) was updated with input from MIGA management.

4

1.11 New recommendations for the 2006 Annual Report to MIGA management and IEG are listed in Section 7.

5

2. DEVELOPMENT OUTCOME

Evaluations of a sample of 21 MIGA projects guaranteed in fiscal 1996 to 2002 show that about half (48 percent) of the projects attained satisfactory or better development outcomes, contributing positively to the achievement of MIGA’s development mandate. These projects provided extensive benefits to the host countries and the communities. The projects’ business performance was an important factor for their contribution to economic sustainability. Projects in middle income countries had generally better business performance and economic outcomes than projects in IDA countries. Projects in Africa had higher impacts on private sector development but had relatively poor financial results. More than half of projects (56 percent) were rated satisfactory or higher for environmental and social effects; the lower ratings of the remaining projects reflect some shortcomings in their consistency with environmental and social safeguard policies and guidelines.

2.1 MIGA’s “commitment to promoting projects with the greatest development impact that are economically, environmentally and sustainable” is evaluated by IEG in four aspects: their business performance, contribution to economic sustainability, Private Sector Development impact and environmental and social effects in host countries. The overall ratings for development outcome from the 21 evaluated projects are presented in Figure 1 below. About 48 percent of evaluated projects (10 out of 21) had satisfactory or better development outcomes.9

2.2 The four categories underlying development outcome ratings, were also analyzed across 21 projects. Average ratings in Figure 2 show an even picture across the four categories at slightly above partially unsatisfactory. While average ratings are lower, when the rating for each category is considered separately, the results are stronger. Out of the 21 projects, 11 were rated satisfactory or better for their business performance; 13 for their contribution to economic sustainability; 12 for their high impact on private sector development (PSD), and ten (out of 18 were this rating applied) for their Environmental and Social effects. Two projects were rated satisfactory or higher on all four development outcome categories while all other projects had at least one category which had a lower rating.

9 IEG-MIGA and IEG-IFC use a common method in evaluating the development outcome of projects adjusting for the differences in the scope of their business. By comparison, despite the differences in sample sizes, 59 percent of evaluated IFC investment operations have high development outcome ratings. See IEG Annual Review of FY2005 Evaluation Findings in IFC [CHECK WHETHER PUBLICly AVAILABLE]. Forty-four percent of these evaluated IFC projects had high ratings for Business Success; 63 percent for Economic Sustainability; 69 percent for Environmental, Social, Health & Safety Impacts; and 70 percent for Private Sector Development.

6

Figure 1. Distribution of Development Outcome Ratings

(Twenty-one Projects)

1

5

21

5

7

0

2

4

6

8

10

12

Unsatisfactory PartiallyUnsatisfactory

Satisfactory Excellent

Num

ber o

f Pro

ject

s

9 projects in 2004 RDE 12 projects in 2006 AR

Figure 2. Average Development Outcome Ratings


01234

BusinessPerformance

EconomicSustainability

Environmental andSocial Effects

PSD Impact

2.3 Evaluated projects in middle income countries had better development outcomes than those in IDA (including blend) countries (Figure 3). In particular, these projects as a group outperformed those in IDA countries in terms of business performance and economic contribution. However, projects in IDA countries on average had higher ratings for Environmental and Social Effects and PSD impact than projects in middle income countries.

Figure 3. Average Development Outcome Ratings in MIC and IDA Countries

(Twenty-One Projects)

Business Performance



PSD Impact

IDA / Blend Countries Middle Income Countries

U PU S E

Figure 4. Average Development Outcome Ratings, by Region


Business Performance



PSD Impact

Africa Asia-Pacific ECA LAC

U PU S E

A. BUSINESS PERFORMANCE

2.4 The Business Performance dimension assesses a project’s financial viability and its impact on the investor. Business performance is vital to development outcome, particularly to the project’s economic sustainability, since various development impacts depend on its long-

7

term financial viability. Sufficient financial returns are also necessary to attract private investors.10

Figure 5. Business Performance Ratings (Twenty-One Projects)

48%

52%

LOW

HIGH

Source: Annex 5: FY96-FY02 Evaluated MIGA Projects Summary of Evaluation Results.

2.5 Fifty-two percent (11 out of 21) of the evaluated projects were rated satisfactory or better for their business performance (Figure 5). These eleven projects made profits for the investors and had rates of return that reflected their financial viability due to characteristics that enabled these projects to cope better with the riskier business environment and exogenous changes such as macroeconomic crisis, market restructuring, or economic slowdown. The ten projects that were rated low11 for their business performance had problems with evolving market conditions and eight of the ten evaluated projects experienced project-related difficulties.

2.6 Exogenous factors affecting the financial viability of the evaluated projects included sector policy changes and macroeconomic and political crises, or a combination (Annex 10). All ten projects that were rated unsatisfactory or partially unsatisfactory for their business performance were unable to cope with evolving market conditions such as increased competition and changes in sector policies.12 Most of the projects affected by sector and market changes were in manufacturing. A financial sector project and a tourism project were also affected by more intense competition arising from sectoral or economic changes. Among these

10 When rating Business Performance, IEG uses an after tax, real financial rate of return (FRR), when feasible, and comparing it to a hurdle rate of 10 percent. This rate is equivalent to the discount rate used to adjust payment flows over time. For comparison, IFC uses the weighted average cost of capital (WACC) as a benchmark for its projects. The average WACC for IFC projects evaluated in 1999-2002 was 9.8 percent in real terms. In some projects a meaningful FRR can not be calculated due to the lack of adequate data or non-applicability for the financial sector projects. In such cases, IEG evaluates to what extent a project has met its business and profit objectives, also considering other financial indicators such as net present value, return on assets, and debt service ratios. 11 High ratings are Excellent and Satisfactory, whereas Low ratings are Partially Unsatisfactory and Unsatisfactory. 12 Results of the nonparametric correlation test (Spearman’s rho) indicated that there is a significant negative correlation between Business Performance and evolving market conditions among the 21 evaluated projects. The Spearman’s rho correlation coefficient of -0.432 is statistically significant at the 0.05 level (2-tailed test). Refer to Annex 10 for the projects affected and the definition of evolving market conditions or sectoral changes.

8

ten projects, macroeconomic or political crisis was a factor in four projects13 but there were also inherent weaknesses in all of these projects which increased their vulnerability. In one case, drought worsened the effects of an economic crisis.

2.7 Commercial and other factors inherent in the projects were also major contributors to low business performance. Eight out of the ten evaluated projects experiencing financial difficulties were affected by project related factors14 such as increased production cost, construction delays, weak business models, unrealistic forecasts, and eroding competitive advantage due to unsuitable or outdated technology. For example, a manufacturing project was experiencing losses because the product was becoming outdated and demand declined. Another manufacturing project built a state of the art facility but failed to plan and allocate resources for marketing the product and had difficulty in understanding consumer preferences.15

2.8 On average, evaluated projects in middle income countries had better overall business performance than those in IDA--including blend--countries. Eight of the fourteen (57 percent) projects located in Middle Income Countries had satisfactory or better ratings for financial viability due to a combination of project characteristics such as strong competitiveness, market niche advantage, and higher than anticipated demand. Among the seven evaluated projects in IDA countries, four suffered from lower than anticipated demand, unsuitable technology, or lack of competitiveness. Although the sample size by region is too small to be statistically significant, evaluated projects in the Asia and Pacific region had higher average ratings for business performance, while projects in Africa had the lowest average ratings (Figure 4).16

13 There is a negative correlation between Business Performance and macroeconomic and political crises, but the correlation is not statistically significant between these two variables among the 21 evaluated projects, based on the results of the nonparametric Spearman’s rho correlation test. Refer to Annex 10 for the projects affected by macroeconomic and political crisis. 14 Results of the nonparametric correlation test (Spearman’s rho) indicated that there is a significant negative correlation between Business Performance and project related factors among the 21 evaluated projects. The Spearman’s rho correlation coefficient of -0.636 is statistically significant at the 0.01 level (2-tailed test). Refer to Annex 10 for the definition of Project Specific Risk and the projects affected. 15 In some of these cases, the financial viability of these projects had not been assessed adequately by MIGA at the time of underwriting. 16 See Annex 4 for the regional distribution of the 21 evaluated projects.

9

Box 1. Investment Disputes in Evaluated Projects

Six of the 21 evaluated projects experienced some form of investor-host government dispute between the date of MIGA guarantee issuance and IEG’s ex-post evaluation. MIGA’s Legal Affairs and Claims Group was involved in resolving disputes in four projects; only one of these developed into a claim situation. All six projects are governed by concession agreements, power purchase agreements, or similar contracts between the investor and a host government entity.17

Factors contributing to concession contract disputes. The causes of the disputes included changes in political priorities and policies, tariff/charges disputes, unrealistic forecasts, and sectoral reforms that rendered the initial provisions of the concession or power purchase agreement inconsistent with the reforms. In five out of the six projects involving disputes with the government, there had been no competitive bidding for the concession, or the contracting procedures had not been transparent.

In the six projects involving disputes, the sustainability of contract provisions, affordability of the tariffs or other charges, and the contingent liability of the government in these projects had not been addressed adequately in the underwriting documents. More generally, in most of the projects with concessions, when MIGA assessed the risk factors of the projects at underwriting, it relied on host government approval and investor representations as mitigating factors, which may be insufficient in preventing disputes after the issuance of the contract of guarantee.

In two of the four projects with concession or similar contracts that did not have disputes with the host government, sector restructuring and policy changes occurred before the contracts were signed. In the other two cases, the projects were exempted from sector policy and concession contract changes due to their small market share.

In eight of the ten evaluated projects that involved concessions or similar agreements with a host government entity, the financial viability and sustainability of the projects had not been established clearly in MIGA’s underwriting documents. An example is a project that has been the subject of intermittent dispute between the host government, the investor and the lenders from a year after commercial operations started (and one year after the guarantee was issued) in the late 1990s and until the present. There were issues about the project’s financial viability and sustainability when IEG reviewed the project documents.

B. CONTRIBUTION TO ECONOMIC SUSTAINABILITY

2.9 This dimension assesses the net effect of the project on the economy, living standards, quality of life, and the project’s social impact. In evaluating a project’s contribution to economic sustainability, IEG considers its economic rate of return (ERR) as an indicator of economic costs and benefits to the society as a whole, and it also assesses the non-quantifiable impacts, both positive and negative.

2.10 Sixty-two percent of all evaluated projects had high ratings for their contribution to economic sustainability and development. These projects provided extensive benefits to the host countries such as employment, efficiency gains, ancillary infrastructure for nearby communities, host government revenues, and linkages with the rest of the economy and the 17 Ten out of the 21 evaluated projects were governed by contracts between the investors or project enterprise and a host government entity. Six of these had disputes with the host government while four did not.

10

global markets (Figure 6). However, 38 percent of the projects had low ratings for their contribution to economic sustainability because of poor business performance, existence of trade distortions, and/or changes in economic environment that adversely affected the projects.

Figure 6. Contribution to Economic Sustainability Ratings (Twenty-One Projects)

38%

62%

LOW

HIGH


2.11 Economic contribution tends to track financial viability. Among the 13 projects that were rated satisfactory or higher for their contribution to economic sustainability, nine were also rated highly for their business performance. A profitable project that does not benefit from economic distortions is better able to provide economic and social benefits for the project beneficiaries, as in the case of the 43 percent of the projects in the upper far right quadrant of Figure 7. However, 10 percent of the projects with high business performance were rated partially unsatisfactory for their contribution to economic sustainability because monopolistic pricing and consumer welfare loss had limited their economic benefits. For example, an infrastructure project had satisfactory business performance at the time of IEG evaluation but the loss in consumer welfare and the monopolistic nature of the concession had resulted in a less than satisfactory economic contribution. The biggest beneficiary of this project was the host government because of the substantial fees it received from the concession to the detriment of the users and service providers of the project.

2.12 Moreover, a project enterprise that is facing financial problems is likely to reduce or eliminate social programs or is unable to achieve the anticipated benefits. For example, an innovative program to provide slum dwellers with subsidized electricity rates and other incentives to reduce illegal power connections and integrate them into the legal economy had to be discontinued because the company was experiencing financial losses, and this social program was central to MIGA’s support for the project.

2.13 Yet 19 percent (4 out of 21) of the projects with low business performance at the time of IEG evaluation still generated significant benefits to society and were rated satisfactory for their contribution to economic sustainability. Despite poor performance at the time of evaluation, these projects continued to generate employment in countries with high unemployment, and also considerable additional economic activity. In the case of a financial sector project, the financial institution leveraged the MIGA-guaranteed funds to on-lend to several medium-sized and large private companies in key sectors of the economy although at the time of evaluation, the financial institution was experiencing losses in its other activities due to changing sector dynamics.

11

Figure 7. Business Performance and Contribution to Economic Sustainability (Twenty-One Projects)

HIGHLOWLO

WH

IGH 10%

High Business PerformanceLow EconomicSustainability

19%Low Business Performance

High EconomicSustainability


Low Economic Sustainability

43%High Business Performance

High Economic Sustainability

Bus

ines

s Pe

rfor

man

ce

Economic Sustainability

(=38%) (=62%)

(=52%)

(=48%)

HIGHLOWLO

WH

IGH 10%

High Business PerformanceLow EconomicSustainability


High EconomicSustainability


Low Economic Sustainability

43%High Business Performance

High Economic Sustainability

Bus

ines

s Pe

rfor

man

ce

Economic Sustainability

(=38%) (=62%)

(=52%)

(=48%)

2.14 Crises and Contribution to Economic Sustainability. At evaluation, seven out of the 21 projects were in countries affected by macroeconomic crises during the mid-to late 1990s, the period when the majority of the guarantee contracts for these evaluated projects was issued (Annex 10). Four of these projects were rated unsatisfactory for their business performance and for their contribution to economic sustainability. However, these four projects also faced problems such as loss of competitiveness even before the crises, and involved over-optimistic initial expectations by investors.18 Furthermore, the financial viability of these four projects was not clearly established by MIGA at the time of underwriting.

2.15 Beneficiaries. Thirteen out of the 21 projects have generated sizeable benefits to the host countries such as employment, efficiency gains to the economy, ancillary infrastructure for nearby communities, revenues to the government, and linkages with the rest of the economy and the global markets. Examples of broad-based benefits include a port service project with satisfactory business performance which is benefiting shipping, delivery and processing companies that handle and deliver grains from the port to the rest of sub-region. The project resulted in extensive efficiency gains for the sector and consumers. A manufacturing project rated highly for its economic contribution has provided the government and communities adjacent to the project site and beyond with revenues and benefits above what was expected. In addition to the considerable tax revenues paid to the government, this project also provided housing for employees and supported schools, clinics, livelihood programs, and other community projects that served the needs of nearby low income communities. The ancillary infrastructure (port, access road and bridges, wastewater treatment plant, and potable water) 18 Nonparametric statistical test results (Spearman’s rho) indicated that there is a statistically significant negative correlation between Contribution to Economic Sustainability and the two variables, evolving market conditions (significant at 0.05 level) and project-specific weaknesses (significant at 0.01 level) among the 21 evaluated projects. Using the same test, the results showed that contribution to economic sustainability and macroeconomic crisis are negatively correlated but the association is not statistically significant among the 21 evaluated projects. Refer to Annex 10 for the definitions of evolving market conditions or sectoral changes, and project specific risks, as well as projects affected.

12

built by the project has also benefited communities farther from the project area. An infrastructure project is supporting the host country’s national apprenticeship program by accepting students from vocational and technical schools for on-the-job training. With the scarcity of technical skills and high unemployment in the country, several of the students trained by the project have been hired either by the company or by other firms.

2.16 In other cases, however, the anticipated benefits did not materialize. Reduced demand caused by economic slowdown in a host country meant that the potential beneficiaries (including the government) were not able to reap the expected benefits from three infrastructure projects. In another case, inbound and outbound airport passengers were not able to benefit from the better baggage handling and catering service when the government changed the project location and limited the scope of the project.

C. PRIVATE SECTOR DEVELOPMENT IMPACT

2.17 The Private Sector Development (PSD) impact measures the effects of the project on the development of private enterprise beyond the project participants. This includes demonstration effects, effects on local suppliers of goods and services, technology and know-how, employee training, contribution to domestic capital market development, project governance and reputation, the extent to which a project leads to more private ownership and stronger local entrepreneurship, greater competition or competitiveness, as well as impacts on competitors, new market entrants, and producers of complementary goods. Twelve out of the 21 projects (57 percent) received satisfactory or better ratings for PSD impacts as a result of their high demonstration effects and large upstream and downstream effects on the host countries (Figure 8).

Figure 8. Private Sector Development Impact Ratings (Twenty-One Projects)

43%

57%

LOW

HIGH


2.18 Evaluated projects in Africa had the highest PSD impact. The PSD impact was the strongest among the evaluated projects in Africa and also IDA countries as a whole (refer to Figure 3 and Figure 4 for average PSD ratings by region and country classification). Evaluated projects in the Asia and Pacific, Europe and Central Asia, and Latin America and the Caribbean regions also generated positive PSD impacts but their marginal contribution to PSD

13

was, on average, not as high as for Africa.19 The evaluated projects in these three regions (mostly in host countries classified as middle income countries) operated in environments where foreign and domestic private sector participation was already relatively robust, adoption of new technology fairly wide-spread and skills levels comparatively high.

2.19 All four projects evaluated in Africa had satisfactory or better ratings for PSD impact. Their PSD impacts ranged from the introduction of new technology, transfer of skills, strong economic linkages with local suppliers, and positive demonstration effects in countries that were just emerging from conflict or whose private sectors are still nascent and characterized by a large informal economy. The beneficiaries included, in addition to company employees and direct suppliers, upstream enterprises and communities beyond the local project areas.

2.20 An example of a project with high PSD impact is a manufacturing enterprise established in a post-conflict country in Africa. The project’s pioneering status meant that the company had to ensure the reliability of its supplies by developing local capacity. It did so by initiating a program in partnership with a regional development agency to identify and fund the training and development of its local suppliers. The project also encouraged the host government to improve its administrative procedures and processes and, as a result, helped the host country attract new investments. In an infrastructure project in a middle income country, while the project was operating successfully and made positive contributions to economic sustainability, its demonstration effect was limited due to deficiencies in the regulatory environment and government-imposed restrictions on private sector involvement in the sector.

D. ENVIRONMENTAL AND SOCIAL EFFECTS

2.21 The environmental and social effects of guarantee projects are evaluated relative to their consistency with applicable MIGA environmental and social safeguard and environmental guidelines and policies (approved by the Board in 1999 and 2002, see Box 2); and the adequacy of measures to mitigate any adverse environmental and social impacts.

2.22 The evaluated guarantee projects consisted of two category A20 projects and 16 category B21 projects, while three financial sector projects were classified as category C. According to MIGA’s Environmental Assessment Policy, beyond screening, no further environmental assessment action is required for category C projects. Thus, this section is based on findings from 18 category A and B projects. 19 Because of the small sample size for each region, IEG cautions against drawing general conclusions from regional breakdowns. 20 A proposed investment will be classified as Category A if the project is likely to have significant adverse environmental impacts that are sensitive, diverse, or unprecedented. These impacts may affect an area broader than the sites or facilities subject to physical works. 21 A proposed investment will be classified as Category B if the project's potential adverse environmental impacts on human populations or environmentally important areas (including wetlands, forests, grasslands, and other natural habitats) are less adverse than those of Category A projects. These impacts are site-specific; few, if any, of the impacts are irreversible; and, in most cases, mitigating measures can be designed more readily than for Category A projects.

14

Box 2. How does IEG Assess Safeguard Consistency?

The objective of the safeguards review is to assess the extent to which evaluated projects were (and are) consistent with MIGA’s safeguard policies and environmental, health and safety (EHS)22 guidelines, IEG focuses on the consistency with environmental and social safeguards and EHS guidelines (collectively described below as “safeguards”) at two phases in the project cycle:

• Consistency with Safeguards at Board Approval: To what extent did a guaranteed investment comply or conform to the requirements of current MIGA safeguard policies and guidelines at the time of Board approval? (Any significant differences between the guidelines applicable at approval and current guidelines are highlighted in the evaluation of each project.)

• Consistency with Safeguards at Evaluation: To what extent did a project fulfill or conform to the conditions and requirements of the safeguard policies and guidelines during project implementation (and at evaluation) and adequately implement the safeguard management/action plans that had been identified at approval?

What criteria does IEG use? IEG uses a set of criteria (summarized in Annex 2) that reflect key safeguard policy requirements and the necessary steps involved in meeting them. The criteria are based on MIGA’s own environmental assessment and disclosure policies and procedures, as well as on MIGA’s interim issue-specific safeguards, as approved by MIGA’s Board of Directors in 1999 and 2002, respectively. It is necessary to assess MIGA’s safeguard consistency in the context of MIGA’s procedures which, in some specifics, differ from those of the IFC and the World Bank. Consistency with host country environmental and health and safety standards is also assessed.

What criteria does IEG apply for projects guaranteed before 2002? Current MIGA policies and guidelines are used as the basis for evaluation for all projects in the sample (i.e., MIGA’s 1999 environmental assessment and disclosure policies and procedures and 2002 interim issue-specific safeguards, as approved by MIGA’s Board). The application of the safeguard policy framework has evolved considerably in MIGA since issuing the first guarantee in 1990. Prior to adopting its own policies and guidelines, MIGA applied World Bank environmental and social policies23 and guidelines to its projects.24 Before establishing an in-house environmental unit in FY98, MIGA used specialized IFC staff to “ensure that [its projects] conform to the environmental standards adopted by other members of the World Bank Group”. MIGA’s 2002 interim safeguard policies adapted World Bank safeguards to the private sector and to MIGA’s business model. This involved some simplifications and clarifications but not a tightening of World Bank safeguards as they existed in 2002. Therefore, all projects covered in IEG evaluations were subject to the WB policies as they existed at the time of their Board approval (i.e., before 2002).

IEG developed this methodology in FY03 in collaboration with IEG-World Bank for the joint evaluation of World Bank Group experience in extractive industries. Thus, IEG first used this methodology in FY03 to review 11 extractive industries projects guaranteed by MIGA. The same methodology was subsequently applied at the 2004 Review of Development Effectiveness in MIGA for an evaluation of nine ex-post guarantee projects.

Figure 9. Environmental and Social Effects Ratings (Eighteen Projects*)

22 See the World Bank Group’s Pollution Prevention and Abatement Handbook and IFC’s supplementary guidelines as listed in: http://www.ifc.org/ifcext/enviro.nsf/Content/EnvironmentalGuidelines. [Public?] 23 The World Bank has 10 safeguard policies, of which nine are covered in the present review: Environmental Assessment; Involuntary Resettlement; Indigenous Peoples; Natural Habitats; Safety of Dams; Cultural Property; Projects on International Waterways; Pest Management; and Forestry. Projects in Disputed Areas was not applicable. 24 MIGA Environmental Assessment and Disclosure Policy and Environmental and Social Review Procedures: Implementation Report, p. 1.

15

44%

56%

LOW

HIGH

* Excluding three financial sector projects.

Source: Annex 4: FY96-FY02 Evaluated MIGA Projects Summary of Evaluation Results

2.23 Overall Ratings of Environmental and Social Effects. Figure 9 shows the ratings for the 18 evaluated category A and B projects. Ten out of 18 had a satisfactory or better rating for their environmental and social effects (taking into consideration safeguards compliance and measures to mitigate adverse impacts). The remaining eight projects were rated unsatisfactory or partially unsatisfactory mainly due to shortcomings in their consistency with MIGA’s current environmental and social safeguard policies and guidelines.25

D.1. Overall Consistency with Safeguard Policies

2.24 Findings on overall safeguard consistency. For the evaluated projects there was a significant difference in consistency with safeguards requirements at the time of Board approval26 and at IEG evaluation. For the eighteen evaluated projects, IEG found that at approval, 83 percent substantially27 met the requirements of MIGA’s environmental and social safeguards (Figure 10). However, only 61 percent of these projects substantially met the requirements for safeguard policies consistency at evaluation (Figure 11). This indicates a significant decline (on average) in consistency with safeguard and guidelines during the actual implementation of the project. Six of the seven projects with negligible or modest consistency with MIGA’s safeguard policies or guidelines at evaluation were category B projects (Annex 6 and Annex 7).

25 Three of the projects that were rated unsatisfactory were for a single investor in the infrastructure sector in one middle income country. The three projects faced similar issues. 26 Contracts of guarantee are offered to the applicant following Board concurrence and receipt of Host Country Approval. For simplicity, throughout the evaluation, IEG uses the term “Board Approval” to denote the phase of the guarantee project cycle subsequent to underwriting and before contract issuance. 27 The method of assessing safeguard consistency at approval and evaluation is to consider the rating in terms of “high”, “substantial”, “modest” or “negligible’, which was the approach used in the “Extractive Industries and Sustainable Development - An Evaluation of World Bank Group Experience”, Volume IV: MIGA’s Experience, Washington, 2003. In this report, “high” was defined as the set of requirements were fully met, or expected to be fully met, with no shortcomings; “substantial” as the set of requirements generally were met, or expected to be met, but with only minor shortcomings; “modest” as the set of requirements were met, or expected to be met, but with significant shortcomings; and “negligible” as the set of requirements were not met, or expected to be met, due to major shortcomings.

16

D.2. Key Findings on Safeguard Consistency at Board Approval and at Evaluation

2.25 Several factors are driving the difference in consistency with safeguard policies at approval and at evaluation for the evaluated projects (see Figure 10 and Figure 11). While there are specific requirements for the two different stages of the project cycle (Box 2 on methodology in page 14), there are also some common factors for both stages, which include: addressing issue-specific safeguards, meeting environmental guidelines, and setting up and implementing Environmental Action Plans and Environmental Management Systems.

Figure 10. Summary Safeguard Performance Criteria at Approval (18 projects)

CriterionApplicable to

(No. of Projects)

Comprehensive E&S baseline survey 4 100%

Public disclosure/consultation addressed 2 100%

Comprehensive dam safety measures proposed 1 100%

Pest Management provisions proposed 1 100%

Adequate EAP proposed 18 94%

EHS Guidelines or host country regulations comprehensively addressed 18 89%

Adequate analysis of feasible alternatives 18 72%

Comprehensive Environmental Assessment 18 67%

Project investor’s EMS adequate 18 67%

Comprehensive and implementable RP/CDP prepared 2 50%

Natural habitats protected or offsets provided 3 33%

Percent of Projects with Overall Safeguard Consistency 18 83%Addressed Substantially or Higher

Addressed Substantially or Higher(Percent of projects)

2.26 Addressing issue-specific safeguards. In addressing issue-specific safeguards, projects performed better at evaluation than at Board approval. At the Board approval stage not all applicable specific safeguards were addressed in all cases. IEG evaluations noted in particular the Natural Habitats Policy and the Involuntary Resettlement Policy. Nevertheless, those deficiencies that existed at approval were addressed during implementation by guarantee holders, and these projects generally met the requirements of MIGA’s policies at the time of IEG evaluation. This improvement of compliance ratings in the evaluated cases was mainly due to the efforts and initiative of the guarantee holders.

17

Figure 11. Summary Safeguard Performance Criteria at Evaluation (18 projects)


(No. of Projects)

Natural habitats protected or offsets provided 3 100%

Continuing public disclosure and consultation 2 100%

Dam safety measures implemented 1 100%

Full compensation of PAPs 1 100%

Investor’s project implementation EMS effective 18 89%

E&S monitoring fully implemented by investor 18 61%

EAP/EMP fully implemented by investor 18 61%

EHS Guidelines or host country regulations fully met 18 50%

RP/CDP fully implemented 2 50%

Pest Management measures implemented 1 0%

Percent of Projects with Overall Safeguard Consistency 18 61%Addressed Substantially or Higher

Addressed Substantially or Higher(Percent of projects)

2.27 Environmental Guidelines.28 However, for environmental guidelines, the ratings at evaluation were lower than at Board approval. In general, the projects as proposed by the guarantee holders were consistent with MIGA’s and host country environmental, health and safety (EHS) guidelines at Board approval stage. In fact, IEG found that at approval sixteen out of the 18 projects substantially met the set of requirements of the applicable environmental guidelines while at evaluation only nine out of the 18 projects did. In the case of three similar infrastructure projects by the same investor, the Environmental Assessment (EA) at the time of Board approval correctly identified the need for NOx controls to be installed and operated on the gas turbines, and hence, these projects were rated as substantially meeting MIGA’s guidelines. However, at evaluation IEG found that NOx was not being controlled for the three projects and in one case the NOx control equipment had not yet been installed (for one of these plants, MIGA granted a waiver for the NOx requirement under conditions not subsequently met by the investors). Their emissions were significantly above MIGA’s emission limits. Including

28 MIGA’s Environmental and Social Review Procedures note that “MIGA’s environmental guidelines consist of World Bank’s environmental guidelines as established in the World Bank’s Pollution Prevention and Abatement Handbook (http://info.worldbank.org/etools/docs/library/36526/pollutionprevention.pdf). MIGA normally expects that an applicant’s investment will comply with these guidelines. MIGA normally expects that an applicant’s investment will comply with these guidelines.” ”MIGA staff need to ensure that each project insured by the Agency complies or will comply with the more stringent of environmental requirements of the host country or MIGA’s environmental guidelines. Any variances from the guidelines must be reviewed and approved by the Executive Vice President and will be identified in documents submitted to the Board, and accompanied by a summary explanation of its justification. In cases where the project does not fully comply with appropriate guidelines or recognized best management practices, MIGA may require a corrective action plan as a condition of guarantee.” [Public?]

18

these three, four projects failed to comply with MIGA’s air pollutant standards at evaluation.29 In the fourth case, investors claimed that they did not know which MIGA emission limits applied to their facility.

2.28 Environmental Action Plans and Environmental Management Systems. Another aspect where consistency was lower at evaluation than at Board approval was in Environmental Action Plans. IEG found a decline in the number of projects substantially meeting this guideline from 94 percent at Board approval to 61 percent at evaluation (Figure 10 and Figure 11). In many cases, guarantee holders failed to fully implement environmental action or management plans (EAP) agreed at Board approval. By contrast, many more projects had successfully set up effective Environmental Management Systems (EMS) at evaluation (increasing from 67 percent at Board approval to 89 percent at evaluation). Greater attention to ensuring that EMS were effectively set up and operating from the start of the project would have helped improve action plan implementation later on, since guarantee holders would have established up front internal capacity to monitor and ensure compliance. One of the projects with a particularly poor level of compliance was still struggling to set up an effective EMS five years after guarantee issuance.

2.29 Use of Partnerships in Environmental Review and Monitoring. MIGA drew on IFC staff to provide environmental clearance of projects before Board approval for the evaluated projects that were guaranteed between FY96 and FY02 (Box 2 in page 14). Hence, out of 18 evaluated projects, MIGA was the sole environmental reviewer for only one project before Board approval. During implementation, MIGA had primary responsibility for monitoring eight projects (i.e., taking a proactive role in following through on contract conditions, compliance, deadlines for submission of information). In five projects, IFC was also an investor and/or lender, and MIGA relied on IFC to carry out safeguard assessments and/or supervision on its behalf, while in another case, MIGA relied on another multilateral partner involved in the project. In four cases, MIGA considered the risks sufficiently low that no proactive measures were taken. However, out of the evaluated 18 projects, only three had been visited by MIGA’s environmental and/or social specialists prior to an IEG mission, while IFC or EBRD had lead responsibility for the environmental supervision in five additional projects. MIGA files show that in two cases updated information was provided by investors to MIGA on compliance with its EHS Guidelines. IEG found that there was inadequate interaction between MIGA and project staff in most of evaluated projects.

D.3. Safeguard Consistency in Category A and B Projects

2.30 The purpose of this section is to complement the review of safeguards consistency presented in paragraphs 2.24 to 2.29 with information collected for the earlier Extractive

29 The air pollutants that have exceeded MIGA’s emission limits or ambient standards in the projects reviewed include: NOx, SO2, fluoride, HCl, and particulates.

19

Industries (EI) Evaluation,30 when IEG reviewed safeguard consistency of a sample of 11 EI projects. Of a total of 29 projects, 11 were categorized as A and 18 as B projects.

2.31 Safeguard consistency at Board Approval and at Evaluation. For the extractive industries projects reviewed in 2003, IEG had found that there was a significant improvement in safeguard compliance from Board approval to evaluation. Thus, the EI evaluation concluded that “MIGA’s value-added as an insurer of extractive industries projects is in the environmental and social standards it brings with its guarantees.” However, the current sample of 18 ex-post evaluated projects shows an opposite trend, namely a significant decline in safeguard compliance during implementation compared to the level at Board approval (from 83 percent to 61 percent, paragraph 2.24 and Annex 6 and Annex 7).

2.32 Safeguard Consistency for Category A and B projects. Given that most EI projects (nine out of the 11) were category A and the majority of the sample presented in this Annual Report (16 out of 18) are category B, IEG analyzed whether the environmental categorization could help explain the opposite trends. (Figure 12) Among all category A projects, 73 percent were substantially in consistency with MIGA’s safeguards at both approval and at evaluation (the average score is 2.86 at approval and 2.87 at evaluation, Annex 6 and Annex 7). In fact, nine out of 11 category A projects improved compliance during implementation. In contrast, category B projects showed a declining trend: at Board approval, 83 percent of category B projects were substantially in compliance with MIGA’s safeguards, but this ratio declined to 56 percent at evaluation (average score is 2.82 at approval and 2.52 at evaluation), as five projects31 significantly declined in consistency with safeguards to negligible or modest levels.

Figure 12. Comparison of Compliance Ratings at Approval and Evaluation (Category A and B projects)

73%

83%

73%

56%

A Projects B Projects

Approval Evaluation

(18 projects)(11 projects)

(Av.Score2.86)

(Av.Score2.87)

(Av.Score2.82)

(Av.Score2.52)

30 Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience.. 2003. 31 Of these five projects, three were projects of the same investor in the same sector in the same country discussed in Para.2.27.

20

2.33 Key Issues at Board Approval in category B projects. The most notable area in safeguard consistency for the 16 category B projects32 is that almost all of them had an adequate Environmental Action Plan proposed at the time of Board approval (94 percent) but only 56 percent of projects implemented these plans (Figure 13). Other areas of weakness for category B projects at Board approval included: inadequate analysis of feasible project alternatives; inadequate attention to setting up of environmental and social management systems (EMS); and deficiencies in the way the issue-specific safeguards are applied to projects.

2.34 In category B projects with shortcomings in consistency, the environmental and social requirements seemed to be unclear to guarantee holders, as are many of the issue-specific safeguards. Lack of requirements for public consultation and disclosure on category B projects means that project affected people (PAPs) have less opportunity for voicing concerns. Only 61 percent of projects set up Environmental Management Systems (EMS) early in the project cycle (at Board approval) with this increasing to 88 percent at evaluation. As noted earlier, such early establishment would be important to reduce conflicts with communities and adverse environmental impacts at the start of the project. These are areas where investors are seeking substantive advice from MIGA experts, as noted during IEG missions. The need for clear, consistent and timely advice on these often complex matters were the common theme of comments received by IEG from MIGA’s guarantee holders.

Figure 13. Selected Safeguard Performance Criteria at Approval and Evaluation (Category B projects)


(No. of Projects)

EHS Guidelines or host country 16 Approval 88%

regulations comprehensively addressed Evaluation 50%

Adequate Environmental Action Plan 16 Approval 94%

(EAP) proposed and implemented Evaluation 56%

Project investor's Environmental 16 Approval 61%

Management System (EMS) Adequate Evaluation 88%

Addressed Substantially or Higher (Percent of projects)

Sources: Annex 6 and Annex 7

2.35 Key Issues at Evaluation in B projects. Non-compliance in half of category B projects with MIGA’s EHS Pollution Prevention and Health and Safety Guidelines or host country environmental standards at evaluation has been the area of greatest weakness in safeguard performance (Figure 13). Weak implementation of Environmental Management Plans (EMPs) and Environmental Action Plans (EAPs) (only 61 percent of projects) is also an issue at evaluation. In some cases, MIGA is unaware that projects failed to meet its guidelines, since there are no requirements in its policies or procedures (or in most Contracts of Guarantee) to 32 The 16 category B projects come from the sample of 21 reviewed in this report. Two category B projects from the Extractive Industries Evaluation were excluded from this analysis due to lack of direct comparability of subcategories.

21

report annually on implementation of EMPs/EAPs, or on environmental or social performance.33 Projects improved in setting up adequate Environmental Management Systems (EMS) by investors (from 61 percent at Board approval to 88 percent at evaluation), but, as noted earlier, this might be too late in the project cycle to avoid conflict with affected communities.

2.36 Most of the category A projects are complex and highly visible (especially those in the EI sector). Therefore, the guarantee holder, its senior lenders, or the bilateral financing or guarantee agencies are more sensitive to the need to identify and mitigate environmental and social impacts. For the most part, category A projects also have received more attention by MIGA. On the other hand, category B projects are less homogeneous in terms of the nature, scale, and complexity of environmental and social issues involved. Thus, while many of MIGA’s category B projects can have quite significant local impacts, they do not draw the same international attention as the larger projects. Other international agencies do not usually monitor these B projects regularly and host country governments may not have the capacity to enforce and effectively monitor environmental and social safeguard compliance. While environmental plans are generally in accordance with MIGA’s requirements, they at times fall short during implementation. In addition, in the majority of cases, category B projects are not on the list of sensitive/risky projects to be monitored by MIGA environmental staff.

2.37 MIGA management has committed to the Board to update its environmental and social safeguard policies within 12 months from the conclusion of the similar updating process in IFC. The evaluation findings highlight the need for MIGA to consider adapting IFC’s new approach in treating category B projects. Opening up all projects to public scrutiny of social and environmental safeguard provisions may encourage MIGA and guarantee holders to adopt performance standards appropriate for each project. The integration of EHS Guidelines into safeguard performance in an effective way should help to raise the performance of MIGA projects in this important area. Also, investors have expressed their need for expert guidance from MIGA on the implementation of MIGA’s safeguards and guidelines.

33 In certain cases, contracts of guarantee specified investors to submit these reports to MIGA.

22

3. MIGA’S EFFECTIVENESS

MIGA’s effectiveness encompasses its own work in selecting, assessing, underwriting, and monitoring projects, i.e., components that can be influenced by MIGA. More than half (52 percent) of the projects were rated satisfactory for MIGA’s effectiveness. The evaluated projects were generally aligned well with World Bank and host country strategies, and in most projects, MIGA played an important role and brought value added as an insurer. However, MIGA’s assessment, underwriting and monitoring was not adequate in many cases for the evaluated projects, in particular for assessing projects’ financial viability and development impact, and in monitoring project performance and outcome. Due to its line of business, MIGA’s relationship with its projects is more removed than for lending institutions, limiting its scope for influencing development outcomes. A substantial share of the projects in this sample with low ratings for MIGA’s assessment, underwriting, and monitoring also had low ratings for development outcome. MIGA has introduced various measures to strengthen quality assurance during underwriting since FY03, particularly during FY04-05. In this regard, the report found a modest improvement of the quality at entry for guarantees underwritten in FY05 compared with the FY96-02 evaluated projects.

3.1 The evaluation of MIGA’s effectiveness assesses MIGA’s work quality in selecting, assessing, and monitoring its guarantee projects, as well as its additionality and ability to influence project outcomes. IEG evaluates MIGA’s work and involvement through three categories: (i) strategic relevance; (ii) MIGA’s role and contribution; and (iii) MIGA’s assessment, underwriting, and monitoring.

Figure 14. Overall MIGA Effectiveness Ratings


13

5

0

6

6

0

2

4

6

8

10

12

Unsatisfactory PartiallyUnsatisfactory

Satisfactory Excellent

Num

ber o

f Pro

ject

s

9 projects in 2004 RDE 12 projects in 2006 AR

Figure 15. Average MIGA Effectiveness Ratings


0

1

2

3

4Strategic Relevance

MIGA's Role andContribution

MIGA's Assessment,Underwriting and

Monitoring

Source: Annex 5. FY96-02 MIGA Evaluated Projects Summary of Evaluation Results

3.2 Figure 14 presents the average ratings for MIGA’s effectiveness for 21 evaluated projects. Approximately half of the projects (11 out of 21) have a satisfactory rating in MIGA’s overall effectiveness. Nine projects are rated partially unsatisfactory and one project unsatisfactory. On average, Strategic Relevance received the highest ratings, followed closely

23

by MIGA’s Role and Contribution while the lowest rating, on average, is for MIGA’s Assessment, Underwriting, and Monitoring (see Figure 15).

A. CONSISTENCY WITH STRATEGY

3.3 IEG considers whether projects are consistent with MIGA’s strategic priorities, World Bank Country Assistance Strategies, and host country development objectives at the time of guarantee approval. Since all the evaluated projects except one preceded MIGA’s first explicit strategy (MIGA Review 2000)34 that defined regional, sector and other priorities, this aspect did not play a large role in the rating. With this caveat in mind, the majority of projects (19) were rated satisfactory or higher, with only two projects being partially unsatisfactory or lower, reflecting overall consistency of projects with Bank and/or host country priorities (Figure 16). The two low ratings were projects where MIGA guarantees violated an explicit Bank strategy or had very little relevance to the host country.35

Figure 16. Consistency with Strategy Ratings (Twenty-One Projects)

10%

90%

LOW

HIGH


3.4 For three power projects the World Bank strategy and policy advice to countries evolved during the guarantee period, with the result that at evaluation these projects no longer conformed to the World Bank strategy. And some financial sector projects deviated from the stated project objectives, which were in line with World Bank or country priorities, pursuing instead a low risk approach and offering products of lesser strategic relevance to the country.

B. MIGA’S ROLE AND CONTRIBUTION

3.5 As an insurer of political risk, MIGA’s role is to enable projects to proceed in risky environments. To evaluate its role, IEG considers MIGA’s additionality as an insurer, its role in leveraging and complementing partners, as well as its contribution to its clients, for instance by bringing WBG environmental and social safeguards to the design and implementation of their investment projects.

34 In 2000 MIGA presented MIGA Review 2000, which specified for the first time a list of priorities and a strategy for guarantee projects and was in force until 2005. 35 It can be expected that as IEG evaluates more recent projects done following MIGA’s strategies in 2000 and 2005 more differentiation in the ratings should occur.

24

3.6 The results in figure Figure 17 indicate that in the majority of projects (18 out of 21) MIGA has made important contributions. The high ratings reflect investors’ perceptions that MIGA’s involvement in many cases was critical for the investments to proceed, or provided comfort to clients entering new markets or new sectors (especially private provision of public infrastructure). MIGA’s insurance was particularly crucial for investors in post-conflict countries. In one instance, MIGA provided a tailored product that was not available from private insurers for the term required by the investors. In six cases, MIGA coverage was a condition for lenders to provide funds.

Figure 17. MIGA’s Role and Contribution Ratings (Twenty-One Projects)

14%

86%

LOW

HIGH


3.7 MIGA’s role in dispute resolution. Six evaluated projects were involved in disputes, four of which required interventions from MIGA’s Legal Affairs and Claims Group.36 MIGA played a useful role in resolving several disputes related to power purchasing agreements negotiated between investors and local authorities, and contributed to a successful debt restructuring in another case, enabling the project to continue operating.

3.8 On the other hand, MIGA’s impact on project outcomes beyond its role as an insurer has been very limited. Consistent with its guarantee product (that is farther removed from project implementation than a financier or equity investor), it can not be expected to have the capacity to influence project design. However, one aspect where MIGA can have proactive influence to raise standards is through its environmental and social safeguard policies and environmental guidelines. In line with the hands-off approach of the period when the evaluated projects were underwritten, MIGA did not at that time proactively offer services or advice to improve project development outcomes.

C. ADEQUACY OF PROJECT ASSESSMENT, UNDERWRITING, AND MONITORING

3.9 This section assesses the adequacy of project assessment and underwriting with respect to the risks affecting the project, as well as the adequacy and timeliness of monitoring.37 All evaluated projects, except one that was approved in FY02, were underwritten in FY96-99.

36 MIGA has subsequently received and is reviewing a claim for one of these four projects. Six projects were affected by disputes between the investors and the host country. MIGA’s Legal Affairs and Claims Group was involved in dispute resolution in four of the projects. 37 This has been highlighted in previous IEG recommendations.

25

While MIGA’s mandate and policies have largely remained the same since FY96, their application and MIGA’s quality management systems have evolved. Accepted earlier practice reflected the Agency’s greater focus on the business side and its business model as an insurer, rather than on it’s development mandate (see Box 3, on page 30).

Figure 18. Adequacy of Project Assessment, Underwriting, and Monitoring Ratings (Twenty-One Projects)

76%

24%

LOW

HIGH


3.10 The MIGA Convention (Art. 12) requires the Agency to satisfy itself that the projects it guarantees will be economically sound, contribute to the development of the host country, comply with host country laws and regulations, and be consistent with the declared objectives of the host country. In 1999, MIGA formalized the information requirements for underwriting a guarantee (such as project objectives, rationale for MIGA involvement, financial viability, and development impact). IEG applies these internal requirements (rather than FY96-02 accepted practices) for evaluating the adequacy of MIGA’s assessment, underwriting, and monitoring.38

3.11 Evaluation findings show that overall MIGA’s impact assessments and underwriting were not adequate during FY96-02 for ensuring its development effectiveness (see Figure 15). Five projects received a satisfactory rating. Seven projects were rated unsatisfactory, and the remaining partially unsatisfactory in terms of the quality and appropriateness of MIGA’s own work (see paras.3.12-3.16 below).

3.12 Assessment of project financial viability and development impact: Ex-ante assessments of projects’ viability and development impact had shortcomings. There was a tendency to rely almost exclusively on investors’ representations. In many cases MIGA did not verify assumptions or information provided by the guarantee applicants, and documentation was insufficient to carry out an independent assessment of a project’s viability and/or development impact. In some cases, key documents were missing from the files (e.g., power purchase agreements, environmental assessments), and IEG was unable to verify whether this information had been requested from investors or obtained from other sources. IEG noted

38 The current requirements for more rigorous financial analysis including calculation a calculation of financial rate of return and development impact assessment were not adopted by MIGA until FY03. Hence, all evaluated projects predate the current project assessment methodology approved by CODE in 2002, which requires MIGA to apply cost benefit analysis in its underwriting, as well as for ex-post evaluation.

26

many cases in which expected project impacts had been overstated, based on overly optimistic assumptions provided by sponsors. In a case involving a concession the implications of that agreement were not discussed in the underwriting document, and MIGA did not monitor the status and implications of amendments to the concession on the performance bond it had insured. In several cases, assessments by IFC and other partners were not well captured in MIGA’s assessments (ignoring, for instance, the effect of competition).

3.13 Project risk assessment: In general, analyses of the political risks MIGA insures and of risk mitigants appeared to be sound. Country risks were thorough and have been in most cases borne out by subsequent events. In some instances, MIGA raised during the underwriting process issues such as high tariffs, possible conflict of interest, the financial viability of an off-taker, and changes in demand. While the political risks that MIGA insured were addressed, there were shortcomings in assessing risks for financial viability and the development impact of the projects (with implications for MIGA’s business, such as cancellations of guarantees or investment disputes).

3.14 Assessment of environmental and social impacts: In four evaluated projects MIGA’s work related to non-compliance with emission guidelines required more attention. In accordance with its procedures, the Agency granted a waiver for a project which was out of compliance with MIGA’s guidelines; however, two other projects by the same investor, located in the same country, were also non-compliant although not officially covered by a waiver or communicated to MIGA. In another case, the project staff and its guarantee holders claimed that they were unaware of their obligation to meet these standards. MIGA has not received (or required) monitoring reports from this investor, as is the case for many category B projects. More regular communications between MIGA and project staff during implementation would have helped address these issues.

3.15 Partnership Arrangements: When MIGA participated in a project together with other WBG entities or international development agencies, it generally relied on its partners to assess the project and monitor its compliance and performance. While it was prudent of MIGA to leverage its own scarce resources through partnerships, it did not regularly request or receive relevant information on project progress and performance.39 MIGA thus missed opportunities to more effectively conduct due diligence and supervision for its projects.

3.16 Monitoring project performance and development impact remains a challenge for MIGA. The Agency has since 2004 significantly scaled up the environmental and social supervision of its projects (with gaps noted in para. 3.14), but it still does not have an operating system to regularly follow-up on other aspects of project performance with the project enterprise or guarantee holder, and has only in 2006 begun a pilot program of ex-post monitoring of a small group of projects. For most of the evaluated projects, there were no

39 In one case, project information was treated as confidential by the partner supervising the project, and not accessible to IEG staff or MIGA. IEG-MIGA’s 2004 RDE recommended to improve arrangements to share information and documentation with partners.

27

project updates on file, even when the contract of guarantee required investors to submit such updates to MIGA.

D. PROJECT DEVELOPMENT OUTCOME AND MIGA’S EFFECTIVENESS

3.17 MIGA’s relationship with projects is more removed than for a lending institution due to the nature of its business as an insurer. Hence, project success can not be directly attributed to MIGA’s work quality.40 The purpose of this analysis is to explore the two dimensions for evaluated projects together, which were evaluated independently of each other.

Figure 19. Development Outcome and MIGA’s Effectiveness (Twenty-one Projects)

HIGHLOW

LOW

HIG

H 15%High development

OutcomeLow MIGA’s Effectiveness

19%Low development

OutcomeHigh MIGA’s Effectiveness

33%Low development


33%High development


Dev

elop

men

t Out

com

e

MIGA’s Effectiveness

(=48%) (=52%)

(=48%)

(=52%)

HIGHLOW

LOW

HIG



19%Low development


33%Low development


33%High development


Dev

elop

men

t Out

com

e

MIGA’s Effectiveness

(=48%) (=52%)

(=48%)

(=52%)

Figure 20. Development Outcome and MIGA’s Assessment, Underwriting, and

Monitoring (Twenty-one Projects)

HIGHLOW

LOW

HIG

H 29%High development Outcome

Low Assessment,Underwriting & Monitoring

5%Low development Outcome

High Assessment,Underwriting & Monitoring



19%High development Outcome


Dev

elop

men

t Out

com

e

Assessment, Underwriting & Monitoring

(=76%) (=24%)

(=48%)

(=52%)

HIGHLOW

LOW

HIG

H 29%High development Outcome






19%High development Outcome


Dev

elop

men

t Out

com

e

Assessment, Underwriting & Monitoring

(=76%) (=24%)

(=48%)

(=52%)

3.18 Average IEG ratings for MIGA’s Effectiveness are similar to its average Development Outcome ratings. As shown in Figure 19, about one-third of projects have a combination of high development outcomes and high ratings for MIGA’s Effectiveness. A further 15 percent have high development outcomes despite low ratings for MIGA’s Effectiveness. One-third of the evaluated projects have low ratings for both outcome and effectiveness. These results do not indicate a strong relationship between the two dimensions.41 This may be because MIGA’s Effectiveness is an overall rating consisting of three categories, two of which have strong results.

3.19 If one considers a more narrow measure of MIGA’s work quality, MIGA’s Assessment, Underwriting, and Monitoring (a category of MIGA’s Effectiveness), the results

40 Nevertheless, MIGA’s work through the selection of financially and economically sound projects, and thorough assessment and monitoring of safeguards may be related to better project outcomes. 41 Results of the nonparametric correlation test (Spearman’s rho) showed that while there is a positive correlation between Development Outcome and MIGA’s Assessment, Underwriting and Monitoring, there is no evidence of statistically significant association between these two variables among a small sample of 21 evaluated projects.

28

are more prominent (Figure 20). Almost half of the projects (47 percent) have both low development outcomes and low work quality, although only 19 percent are high on both measures. The distribution of ratings in Figure 20 indicates that projects associated with low work quality also had low ratings for Development Outcomes, while a bigger share of projects with high work quality ratings had better development outcomes.

E. IMPROVEMENTS IN MIGA’S QUALITY AT ENTRY

3.20 To see whether there have been improvements in MIGA’s assessment of guarantee projects over time, IEG compared the quality of assessment of the 21 evaluated projects with an assessment of four new projects from FY05.42 In conducting the comparison, IEG applied to both sets of projects the quality at entry methodology presented to CODE in the OEU 2005 Annual Report. The results of the comparison with the 21 projects should be interpreted with caution, given in particular the small sample size for the quality at entry analysis. Next year’s Annual Report will have a larger sample of recently-approved MIGA guarantee projects from FY06-07.

3.21 Modest improvement in quality of analysis of development impact and MIGA effectiveness. As Figure 21 shows, based on the relatively small number of projects assessed, IEG has noted so far a modest improvement in the quality of underwriting compared to projects approved in the 1996-2002 period, particularly in MIGA’s assessment of economic contribution and social impacts. For example, an infrastructure project contained a comprehensive analysis of financial viability and impact on all project stakeholders, including the sustainability of the concession, and supply and purchase contracts with the host government. In a services project, MIGA ensured that the benefits to a social group will be sustained by specifying in MIGA’s contract of guarantee the reporting requirements and the frequency of the monitoring on the conditions of the social group.

Figure 21. Comparison of Aspects of Quality at Entry Development Impact Assessments Between FY96-02 and FY05

42 There are two caveats to this analysis: the small sample sizes, especially for the recently-approved projects, and the applicability of the recent evaluation methodology to the 21 projects which were assessed before the introduction of MIGA’s current appraisal methodology. Regarding the latter, IEG assessed the quality at entry based on whether the analysis contained in MIGA’s decision documents provided enough information and solid assessment to judge the project’s financial viability, economic sustainability, environmental and social effects, and PSD impacts and how the project supported MIGA, the host country and the WBG’s strategy. The assessment of four projects for its quality at entry in the 2005 Annual Report was based on the methodology and guidelines issued by MIGA to its underwriters in FY2002. Hence the quality at entry assessment of the recent projects was assessed according to more stringent standard than the set of 21 older projects.

29

Assessment of PSDImpact

Assessment of BusinessPerformance

Assessment ofContribution to Economic

Sustainability

FY96-02 (21 projects) FY05 (4 projects)

U PU S E

MIGA's Role andContribution

Strategic Relevance

FY96-02 (21 projects) FY05 (4 projects)

U PU S E

3.22 In addition to the improvements in the quality at entry of the analysis of the projects’ economic contribution and PSD impact, FY05 guarantee projects were better in terms of the projects’ strategic relevance and MIGA’s role and contribution to these projects. However, despite the modest improvement, there are still gaps that need to be addressed before MIGA’s assessment of development impacts reaches an average rating of satisfactory. In recent years, MIGA has implemented improvements in its quality management processes and systems (see Box 3 below) to ensure that the assessment of the projects’ development impact is implemented more rigorously.

30

Box 3. Evolution of MIGA’s Quality Management Processes and Systems

MIGA’s quality management processes and systems evolved over the FY96-02 period when the 21 evaluated projects were approved. Inter alia, it began senior management reviews of its projects, implemented a more standardized country risk assessment, and created an in-house environmental unit. More recently, MIGA has strengthened its systems and practices to assess new guarantee projects and has initiated new elements for monitoring and supervising its portfolio. Figure 22 provides an overview of the development of MIGA’s quality management systems. MIGA formed a department responsible for a holistic assessment of expected economic, environmental, and social impacts of its projects, and it has scaled up its economic and social expertise. New guidelines for the assessment of project impacts and new guarantee underwriting templates have also been approved.

Figure 22. MIGA Quality Management Steps

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Board paper on assessing development effectiveness

Board paper on selection criteria for guarantees

Environmental specialist hired

MIGA’s interim safeguard and disclosure policies approved by the Board

Internal Business Process Review: Implements Early Management Screening and Risk Management Committee review for projects guarantee projects

MIGA Review 2000 outlines strategic priorities

MIGA’s safeguard and disclosure policies approved by the Board

Independent Operations Evaluation Unit formed

MIGA to begin using cost benefit analysis in its project assessments

Country risk methodology strengthened and team moved to Finance and Risk Management Department

Reorganization creates Economics and Policy Group and Chief Economist position, a one stop shop for economic, environmental and social assessment of projects as well as country risk; strengthens environmental and social unit in project assessment and supervision

New pricing/costing model for pricing of guarantees

New MIGA Business Model emphasizes role as a development institution

Underwriting process changed to include more in-country assessments, focus on development impact; Project Risk Committee formed

New Guidelines for assessment of development impact of projects

MIGA Review 2005

Approach for guarantees monitoring program and new underwriting paper approved

Projects Approved here…... Projects evaluated by IEG-MIGA

Source: IEG

31

4. CONTRIBUTION TO MIGA’S PROFITABILITY

IEG has developed a preliminary method for assessing the relative contribution of each evaluated project to MIGA’s overall guarantee portfolio profitability using a revenue-cost measure. A ranking of the 21 evaluated projects according to their contribution to profitability shows that the majority have made a relatively higher contribution to MIGA’s profitability. The relative contribution to MIGA profitability was associated with the size of MIGA’s exposure. Comparing the relative profitability ranking to development outcome ratings shows that 29 percent of evaluated projects had a desirable “win-win” combination of high profitability and high development outcomes.

A. BACKGROUND AND OBJECTIVE.

4.1 The approved methodology to evaluate MIGA guarantee projects also included a dimension for the project’s contribution to MIGA’s profitability, in addition to the two other dimensions currently in use. While IEG has implemented this methodology with respect to Development Outcome and MIGA’s Effectiveness since its inception in 2003, it had also committed to develop metrics to assess the contribution of guarantee projects to MIGA’s financial profitability (equivalent to the IFC’s investment profitability dimension). The development of these metrics has been hampered by data availability at the project level. While MIGA regularly monitors its premium income, it has chosen to spread a substantial part of the charge to cover administrative costs across the portfolio in proportion to the size of MIGA’s coverages.

4.2 Why do we need to measure contribution to profitability? As a development institution, MIGA needs to be concerned with the development impact of its projects and also with its own overall financial sustainability. It can only fulfill its mandate as a facilitator of foreign direct investment to developing countries if it is able to remain financially sustainable in the long term. This involves ensuring that the Agency can recover from its premium income, on a portfolio level, the initial and ongoing cost for projects (including provisioning reserves). Ideally, MIGA projects should have high development outcomes and contribute to the Agency’s financial sustainability (i.e., the upper right quadrant of Figure 23 on page 34 ). At a minimum, it implies being aware that certain projects may not contribute to MIGA’s profitability, and support is warranted only if they have high expected development outcomes. MIGA would want to avoid projects with characteristics that are associated with both low profitability and low expected development outcomes.

4.3 This section presents a framework to approximate the relative contribution that each guarantee project makes to MIGA’s profitability, based on certain characteristics and a set of assumptions. Using this framework, IEG has ranked the contributions of the 21 evaluated projects to the agency’s profitability. The methodology behind this framework has been discussed with MIGA staff. IEG plans to use this methodology, with any necessary improvements, to begin assessing the profitability dimension for all of its future guarantee evaluations.

32

4.4 The choice of methodology was guided primarily by the availability of data and the nature of MIGA’s risks and operations. Traditional profitability measures often used in various parts of the insurance industry could not be easily applied to MIGA’s operations due to the relative paucity of accounting cost information at the project level, and the less predictable nature of its political risk insurance business. Consequently, traditional insurance industry metrics for profitability were passed over so that a practical, understandable approach could be developed that was both suitable to the data availability within MIGA and could provide useful information for and be combined with IEG’s guarantee project evaluation, which focuses on MIGA’s development effectiveness.

4.5 MIGA’s premium. MIGA’s method to calculate its premium from its inception until January 2004 has relied on the use of pricing tables, which essentially indicated different risk premiums due to differences in type of coverage, sector, and/or country risk. The pricing of MIGA’s guarantees is central to the Agency’s financial soundness, and it needs to be commensurate to the risk the Agency takes on, as well as covering administrative costs for underwriting and managing guarantees. The introduction of a costing model in January 2004 [confidential?] created an underlying cost basis for premiums that recognizes, in addition to the risk factors used before, additional components for administrative costs, capital costs, and cancellation option costs. This costing model serves as the basis for pricing, but does not facilitate determining the individual project’s contribution to profitability and its main drivers because of the aggregate nature of the cost data. The decision in the cost model was to allocate cost in proportion to both the size of MIGA’s exposure and the number of projects in the portfolio rather than based on the actual cost of the specific project.

33

Box 4. Methodology for Assessing Contribution to MIGA’s Profitability

The IEG methodology incorporates important project characteristics that influence profitability at the project level. It makes use of indirect measures and proxies in the absence of precise and complete data on costs and revenues.43

This methodology recognizes that the two defining elements of profitability (i.e., revenue and costs) may be difficult to measure, particularly ex-ante, within MIGA; while ex-post revenues could be more easily measured from the actual premiums paid. The uncertainties surrounding risk costs (including expected losses, reserve requirements, and capital adequacy), due to little actual data on cost drivers, makes the calculation of precise profitability estimates difficult (estimating profitability within MIGA is inevitably a probability-based exercise).44 Nonetheless, this methodology uses sufficient information within proxies and indirect measures to produce a meaningful measure of profitability across projects to produce a ranking.

Specifically, two estimated values were constructed to facilitate a comparison of projects’ expected profitability: the first measures actual revenue, the other estimated costs. The fact that both values are based upon dollars of cash flow (positive cash flow for revenue and negative cash flow for costs) enables IEG to create a meaningful ratio of the two. The ratio of these two values when both are calculated in present value terms (the revenue/cost measure) approximates a project’s ranking in terms of revenue received per cost expended. The revenue measure uses premium data (net of reinsurance and ceding commissions) over the life of a project. The cost measure provides an approximation of project costs on the basis of initial underwriting and ongoing costs, with modifications made to each project according to its sector, number and types of coverages, environmental category, and existence of an investment dispute situation. A total weight is calculated for each project, based on its ranking in each of these five cost factors, then multiplied by average (portfolio) project initial and ongoing costs. Payment streams are put in present value terms for both cost and revenue values to the time of project inception. The risk related costs for any remaining life of the project are estimated by assigning risk costs to each project based upon its country, and type of coverage risk.

Given the estimates underlying MIGA’s costs, the revenue/cost measure generates scores using several assumptions. One assumption in particular, with regard to the expected duration of projects (for projects not yet closed), means that the individual revenue/cost measures generated for currently active projects are not as precise as those for already closed projects. As information is acquired regarding exactly how long a project lasted, its actual revenues and costs will be known with greater certainty. Nevertheless, all calculations of estimates would require such assumptions as was used in the revenue/cost measure.45 The revenue/cost measure is calculated by dividing the revenue measure by the cost measure for each project.46

43 The refinement and analysis of MIGA’s recently implemented Time Reporting System (TRS) or a similar system might facilitate a more precise measurement of profitability. 44 Estimates of profitability can potentially be generated with enough precision to guide management decisions, however IEG believes that the current methodology behind the revenue and cost measures should be further refined and tested before they could be considered absolute measures of revenue, cost, or profitability. 45 It should be noted that the revenue/cost scores for currently active projects may receive a small upwards bias in comparison to past, completed projects that were canceled early, due to the assumption used that all of the active projects among the 21 evaluated projects will continue until contract expiration. 46 A reference ratio for the revenue-cost measure is established as an index at 100.

34

Figure 23. Development Outcome and Contribution to MIGA’s Profitability (Twenty-one Projects)

Development Outcome

Con

trib

utio

n to

MIG

A's

Pro

fitab

ility

(PV

Rev

enue

/Cos

t Mea

sure

)

15

13

141918

12

9

5104

3

2

16

8

7

611

21

20

1

17

Reference point

Unsatisfactory - Partially Unsatisfactory Satisfactory - Excellent

29%

19% 19%

33%

Low

C

ontri

buto

rH

igh

Con

tribu

tor

Note: Numbers beside the points in Figure 23 correspond to project codes

B. FINDINGS

4.6 A significant number of the evaluated projects are high contributors to MIGA’s profitability. Sixty-two percent (13 out of 21) of the evaluated projects have profitability scores that are above a defined reference ratio, which indicates that these projects have made a high contribution to MIGA’s profitability. On the other hand, the eight projects (38 percent) that are located below the reference point had a lower contribution to MIGA’s profitability. When measured against IEG’s development outcome ratings, 29 percent of the projects are in the win-win quadrant of high development outcome and high contributor to MIGA’s profitability.

4.7 Figure 23 presents IEG’s revenue/cost measure relative to the development outcome ratings for 21 evaluated projects. The graph also shows that projects with low contribution to MIGA’s profitability could have equal chances of having a high or low development outcome. In particular, the 19 percent of projects in the bottom right quadrant indicate that although these projects may have a low contribution to MIGA’s profitability, they are consistent with MIGA’s development mandate. A similar proportion of the evaluated projects (19 percent) were rated

35

low both in development outcome and in their contribution to MIGA’s profitability as shown in the bottom left quadrant of Figure 23. They contribute little to fulfilling either its development mandate or contributing to the Agency’s financial sustainability.

4.8 For this sample, high contributors to MIGA’s profitability (13 out of 21 projects) had a similar likelihood of having either a high or low development outcome. Seven out of the 13 projects had a high contribution to MIGA’s profitability but had low development outcome; while the other six high contributor projects (close to the third of the sample) are in the win-win quadrant (upper right corner) of Figure 23.

4.9 Size of MIGA’s exposure was a factor in the project’s contribution to MIGA’s profitability but did not seem to matter for development outcome. Among the different factors that could affect a project’s contribution to MIGA’s profitability, the size of MIGA’s exposure (or the maximum aggregate liability) had the strongest influence on profitability for this sample. (This does not necessarily reflect the project size but the amount of the investment that MIGA insured.) Regression results shown in Figure 24 indicate a strong correlation between the project’s maximum aggregate liability and the revenue/cost measure. In eleven of the thirteen projects that were ranked as high contributors to MIGA’s profitability, MIGA’s maximum aggregate liability was greater than US$10 million.

Figure 24. MIGA’s Exposure and Contribution to Profitability


R2 = 0.7428

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

0 10 20 30 40 50 60Millions

Dollars of Maximum Aggregate Liability

Prof

itabi

lity

Inde

x

Figure 25. Development Outcome and the Size of MIGA’s Exposure


LargeSmall

LOW

HIG


OutcomeSmall Coverage/Exposure

29%Low development

OutcomeLarge Coverage/Exposure

24%Low development


29%High development


Dev

elop

men

t Out

com

e

MIGA’s Exposure

(=43%) (=57%)

(=48%)

(=52%)

LargeSmall

LOW

HIG



29%Low development


24%Low development


29%High development


Dev

elop

men

t Out

com

e

MIGA’s Exposure

(=43%) (=57%)

(=48%)

(=52%)

4.10 A comparison of exposure size per project to development outcome of the 21 evaluated projects shows that large projects have equal chances of having high or low development outcomes (see Figure 25). Projects with small MIGA exposure however, had a slightly higher percentage of having lower development outcome; however, given the small sample size this variation is not significant.

36

5. EVALUATION OF MIGA’S INFORMATION DISSEMINATION SERVICES (IDS)

MIGA has offered free Internet-based services since 1995 to provide information related to foreign direct investment. While these services, developed in connection with MIGA’s technical assistance program, have been relevant to MIGA and target users in the past, the Agency needs to refine its offerings to better serve clients identified in MIGA’s 2005 strategy and its core guarantee business, and also consider changes in technology and services from other providers. Page requests for most MIGA services have remained constant since 2003, indicating they are not attracting many new users and that there is a need to revamp MIGA’s services to make them more attractive. Current users have indicated that the services are providing reliable and timely information and have, in some instances, led them to change their attitude, improved their knowledge, and even resulted in acting on an investment opportunity.

A. BACKGROUND

5.1 MIGA has provided Internet-based services since 1995 to disseminate FDI-related information. Their objective has been to use web sites to provide information on investment opportunities and facilitate investment flows among member states.

5.2 When MIGA launched IPAnet, the first of a suite of online services, it was considered a pioneer in using the Internet in disseminating information on investment opportunities and the business environment in developing countries. Subsequently MIGA diversified its information services by launching PrivatizationLink (1998), FDIXchange (2002), and the FDI Promotion Center (2004). In addition, a website development tool for Investment Promotion Agencies (IPAs), IPAworks, was launched in 2002.

5.3 MIGA’s information dissemination services (IDS) are intended to serve two types of clients: IPAnet, PrivatizationLink, and FDIXchange are providing information on investment opportunities and the business environment in developing countries, targeting potential investors and investment advisors. These three services also provide a channel for IPAs to publish their content, thus serving and supporting their investment promotion activities. The FDI Promotion Center and the IPAWorks template aim to facilitate capacity building of member country IPAs by providing good practice tools and knowledge for investment promotion, targeting IPA officials and complementing MIGA’s technical assistance activities.

5.4 Until 2004, the online services were organizationally aligned with MIGA’s technical assistance (TA) department, and IDS supported MIGA TA capacity building programs for investment promotion.47 The services were not intended to support MIGA’s core guarantees business. As a result of MIGA’s organizational restructuring in 2004, IDS were moved from

47 For an evaluation of MIGA technical assistance projects, see 2004 Report on Development Effectiveness in MIGA, and Improving Investment Climates. An Evaluation of World Bank Group Assistance. 2005.

37

the TA department to the External Outreach and Partners Group and tasked with serving the whole Operations Group, consisting of guarantees and TA.

5.5 IPAnet, MIGA’s oldest information service, provides information on the business environment, sector analyses, investment opportunities, sources of investment finance, laws and regulations, and investment promotion organizations, sorted by country, topic, and sector. MIGA aggregates content in these areas, pulling information from content partners within the World Bank Group, other development institutions (e.g., UNCTAD), private providers, as well as giving a platform for investment promotion intermediaries to share their business information with a wider audience.

5.6 PrivatizationLink draws on the same database and content providers as IPAnet, but focuses on information on privatization in developing and transition countries in general and profiles of enterprises being privatized.

5.7 FDIXchange is the latest addition of the suite of services providing information to investors and investment advisors. Drawing on the same database as IPAnet, it provides customized email alerts on new information related to investment in developing countries, country information, and sources of financial risk management.

Table 1. Summary View of MIGA’s Information Dissemination Services (IDS) Service Launched Objectives Target Clients

IPAnet 1995 To fulfill MIGA’s mandate of dissemination and exchange of information on investment opportunities and conditions in member developing countries and transition economies.

Investors, Advisors, Researchers

Privatization Link

1998 Reduce transaction costs associated with privatization programs by providing investors with key, timely information on state owned enterprises being divested in developing countries and by providing privatization agencies with a free channel to disseminate these opportunities

Investors, Advisors, Financial institutions

FDIXchange 2002 To link international investors to current investment opportunities and business environment information in developing countries through personalized e-mail updates and a web-based archive

Investors, Advisors, Financial institutions

FDI Promotion Center

2004 Supports the efforts of investment promotion professionals to attract and retain FDI. Serves the growing community of investment promotion professionals, particularly those in developing countries by providing services and knowledge to help practitioners update their knowledge and skills covering the complex and often lengthy process if securing inward investment.

Investment promotion professionals

IPAworks 2002 / 2004

Enable developing country Investment Promotion Intermediaries to establish a professional Internet presence quickly and inexpensively

Investment promotion agencies

38

5.8 The FDI Promotion Center is primarily a learning tool for investment promotion officials to help them update their knowledge and skills in securing inward investment. It is an online version of MIGA’s earlier hard-copy Investment Promotion Toolkit, and covers relevant topics for attracting and servicing foreign investors, case studies, and examples of best practice. IPAWorks is a website template that provides a website structure, instructions and coaching for users. Table 1 provides an overview of characteristics of the services, and Annex 13 a more detailed description of the services and their effectiveness.

B. EVALUATION METHODOLOGY

5.9 IEG-MIGA has developed a method for evaluating these services together with IEG-World Bank. It assesses the relevance of these services to MIGA and to intended target audiences, the effectiveness (including outputs, outcomes, and impact), efficiency, sustainability, and MIGA’s work quality. (For a summary of the evaluation methodology, see Annex 11.) The evaluation consists of a review of relevant documents, interviews with current and former staff from MIGA, comparator institutions, and clients; an assessment of comparator services; analyses of site activity, usability, and prominence; and an online user survey administered to approximately 23,000 registered users.48 (The survey questionnaire is attached as Annex 12.)

C. EVALUATION FINDINGS

C.1. Relevance: Relationship between service objectives and MIGA strategic priorities and positioning vis-à-vis other providers

5.10 Relevance is assessed from an internal perspective, in terms of the consistency with MIGA’s mandate of the strategy and objectives of the services, and their alignment with Bank Group strategies. From an external perspective, this evaluation looked at several comparators to assess how MIGA’s IDS are positioned vis-à-vis other providers (both public and private).

5.11 Mandate and Strategy: MIGA’s role in investment promotion and information dissemination is set out in the MIGA Convention, which positions investment promotion as a contributor to “improving the environment” for foreign direct investment (FDI).

5.12 The IDS are a potentially useful tool to support MIGA in fulfilling its current strategic objectives, and the services should consider how best to serve the complex set of clients identified in the recent 2005 strategy as well as to implement its expanded mandate encompassing both MIGA’s technical assistance (TA) and guarantee products.

48 The survey response rate was approximately 6 percent (over 1,300 responses). It should be noted that the survey results contain some biases. The electronic delivery of the survey favors those already using online services, and since respondents had already registered for one or more MIGA services, they were likely to be more knowledgeable of the services and, potentially, more partial to them.

39

5.13 MIGA’s 2000 strategy49 made a commitment to “intensify capacity building and Internet-based information dissemination activities to effectively promote foreign direct investment by fully leveraging advanced information technologies and know-how, complementary to guarantee activities.” The four pillars of MIGA’s 2005 strategy identified a complex set of clients and partners, and the IDS need to align more closely with the new strategy. Segmenting clients and partners more clearly would help MIGA to target and differentiate better its users and partners, and to reinforce the relevance of IDS to both TA and guarantee operations and the clients they serve.

5.14 In the past, IDS have been relevant, within a more narrow mandate, by supporting and complementing MIGA’s strategy for TA activities to promote foreign direct investment in developing countries. In terms of the ability to serve the targeted users, the client survey indicated that IDS are attracting users from the broad categories they intend to serve. There is diversity within the user base (e.g., between potential investors/investment advisors and professionals seeking to promote inward investments, all geographic regions, high, middle and low income countries), which indicates that IDS provide a channel through which MIGA can reach its target clients.50 Site activity and registrations also show that the share of users from low and middle income countries is increasing, a positive indication given the clients targeted in MIGA’s 2005 strategy.

5.15 One of the challenges is to increase awareness of IDS within the World Bank Group and with potential users and clients. Even within MIGA, outside of the TA staff there is a lack of awareness of IDS and their potential value for the guarantees business. In turn, IDS need to identify what specific services, features and marketing strategies will best help the MIGA operations serve and expand their client groups.

5.16 Complementarity with other providers: Since the inception of the services, the external environment in which they operate and the role of the Internet and technology have both changed, with many new providers entering the field, eroding MIGA’s early pioneering advantage. Potential users have access to other (non-MIGA) FDI-related online service offerings. However, these intend to serve different audiences (for example, policymakers) and tend to be content creators, so there is so far no apparent direct overlap with MIGA’s services. There has been some cross-promotion and linking of services (for example, links to the IDS from the World Bank’s Rapid Results Unit (RRU) site), and efforts to complement each other should continue to avoid duplication of activities.

5.17 Although there is no single provider that can serve as a comparator, several providers of services similar to MIGA’s were reviewed. Figure 26 below indicates that MIGA retains a niche among many other providers of FDI-related information. MIGA’s IDS have a

49 MIGA Review 2000. 50 In the survey conducted for this evaluation a significant percentage of users (37 percent) identified themselves as facilitators of or participants in investment decisions, while few users (1.4 percent) self-identified as potential direct investors. In addition, 19 percent of users indicated that they work for organizations which promote inward investment (n=1,315).

40

comparative advantage where they provide free investor services that aggregate content from their network of IPAs and international organizations, particularly with respect to investment and privatization opportunities. MIGA has also established a niche providing online services that build investment promotion capacity among IPAs, complementing its TA activities.

Figure 26. MIGA’s IDS and Other FDI-related Online Services

Investment opportunities

Privatization opportunities

Business environment

Legal environment

Sectoral research

General FDI

research & statistics

Promotion or policy training

IPA site templates

or platforms

MIGA X X X X X X X X

FIAS (WBG) X X

RRU (WBG) X X X

Development Gateway (WBG)51 X X

UNCTAD X X X X X

UNIDO X X X

OPIC X X X

Business Monitor X X X

Various privatization agency sites

X

Various IPA sites X X

C.2. Effectiveness: Outputs, Outcomes and Impacts on Intended Users

5.18 Outputs: Outputs are assessed for each of MIGA’s online services relative to the quantity and quality of content provided by the services, and actual usage.

5.19 The quality of outputs for the information services has varied. Trends show that site activity (measured by page requests) for the services targeting investors has been constant overall since 2003. IPAnet, the oldest service with the largest share of users among the services trends, does not appear to have consistent growth in traffic.52 PrivatizationLink has a lower and 51 FDI pages on Development Gateway are managed by MIGA. 52 To assess the overall "health" of a service, absolute numbers such as the total number of page hits are less useful than trends that can show growth. Growth in traffic consists of 3 factors: an increase in new users, an increase in return visits to the site, and an increase in activities once users visit the site. A consistent upward trend in page requests indicates a growing service.

41

declining user base, consistent with the slowdown in privatization activity since a decade ago. As expected, FDIXchange, a service that users visit primarily to sign up for email alerts, shows the lowest volume of page requests. On the other hand, the newer FDI Promotion Center has a relatively small, targeted user base, but has shown an upward trend in activity and traffic (See Figure 27).

Figure 27. Page Requests by Service 2003-2005 (By Quarter)53

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

4 1 2 3 4 1 2 3 4

2003 2004 2005

Fiscal Year and Quarter

Page

Req

uest

s

FDI CenterFDIXchangeIPAnetPrivatizationLink

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

4 1 2 3 4 1 2 3 4

2003 2004 2005


Page

Req

uest

s


IPA Net

Privatization Link

FDI Center

FDIXchange

0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

4 1 2 3 4 1 2 3 4

2003 2004 2005


Page

Req

uest

s


0

50000

100000

150000

200000

250000

300000

350000

400000

450000

500000

4 1 2 3 4 1 2 3 4

2003 2004 2005


Page

Req

uest

s


IPA Net

Privatization Link

FDI Center

FDIXchange

5.20 These results indicate that users of the investor services are visiting the sites and leaving, without searching for additional content, which may be in part be due to the inherent content model, which aggregates content developed by others and pushes users to external sites via links. The separate branding of these services (although targeting similar user groups of investors or investment advisors) also leads to fragmentation by topic, which makes it hard to guide users across the services for other items of interest. These results also indicate that the content aggregation model may not attract many new users, and may lose its relevance among the current users.

5.21 The FDI Promotion Center is the one service that has been specifically designed for a defined user group – investment promotion professionals. Its growth in use and activity indicates that MIGA is providing relevant and useful output to the intended user group. More

53 Site activity data were only available for FY03-05. Longer-term trends can not be observed.

42

generally, these results may also point to the value of a targeted service and site design for specific user groups.

5.22 The content provided by the services appeared to be actively managed by MIGA. The investor services (IPAnet, PrivatizationLink and FDIXchange) showed good content diversity for both sectors and countries of origin. The fact that the most common topics of content appear to match MIGA’s strategic priorities further suggests that the content is also being strategically managed.

5.23 On content quality and presentation architecture, MIGA’s IDS have scope for improvement (that is, making it easier for users to find what they are looking for on the sites and channeling them there), particularly where content is distinct and is of strategic value to MIGA and its clients.

5.24 Outcomes and Impact: Outcomes are the benefits users derive from using a service, including time and cost savings, and clients’ perception of content quality (reliability, timeliness, and relevance). They were assessed using a counterfactual scenario.54 Impacts are defined as changes in users’ knowledge, attitudes, and behaviors as a result of using the services, such as acting on an investment opportunity or reconsidering a country or sector.

5.25 Within its niche, users already familiar with the IDS are satisfied with them. Survey respondents55 indicated that the services appear to be achieving their intended outcomes. Respondents indicate that the quality of the services is good (i.e., providing reliable, accurate, timely, and current content, and that using the services saves them time and money). The investor services effectively disseminate information on investment opportunities and conditions in member developing countries. The FDI Promotion Center helps investment promotion professionals to update their skills and knowledge on inward investment.

5.26 Overall, approximately 40 percent of survey respondents agreed or strongly agreed that without the services they could not complete a desired task; approximately the same number had neutral responses. Even in cases where the comparator analysis showed a service to be unique, this view was not strongly reflected in the survey responses.

5.27 The fact that the services are “open”, that is, for most of the services users do not need register to use them, distinguishes them from their comparators. The comparator analysis indicated that the FDI Promotion Center responded more favorably than the others in terms of impact. With respect to the IPAworks service, respondents found it to be an effective service, but did not agree that its value was unique.

5.28 The survey results indicated that the services were successful in achieving impact. Respondents were more likely to indicate that they had reconsidered an opinion as a result of

54 Users were asked whether they would have difficulty finding information related to investment without the IDS. 55 It should be noted that the survey results have an inherent bias since it was only administered to current registered users of the services. It did not include non-users or potential target users.

43

using the service (attitude), than improved their knowledge, or taken action such as considering an investment (see Annex 14).56 Users of the FDI Promotion Center are consistent with this trend, although their responses are more favorable in terms of impact. Sixty percent of these users indicated they had increased the number of inquiries from qualified investors as a result of using the service.

C.3. Efficiency: Resource use for service delivery

5.29 To assess efficiency, IEG-MIGA examines the allocation of resources to achieve the services’ objectives, including the cost of delivery relative to their use, decisions to outsource, and sources of funds. It was not possible to obtain comparable information on the effort and costs of designing, building and operating services for comparators. It is therefore difficult to assess whether MIGA’s IDS are an efficient provider of services, compared to others. Also, since the services are free, there is no true market test to assess the cost of providing the services against what users are willing to pay for them, being more similar to a public good.

5.30 Resource inputs, staff and budgets have shown a declining trend over the past few years even though existing services have been enhanced and new services were launched. Declining inputs and increasing outputs (as measured by increases in the number of services offered, the number of registered users and content and marketing partners) indicate that the IDS have done more with less resources. On average, IDS funding (including donor funds) was equivalent to 5 to 8 percent of MIGA’s administrative budget. The use of partnerships has been a creative approach to reach user audiences cost-effectively and the use of less-expensive off-shore resources is consistent with industry practices.

5.31 Resource availability trends of the past years indicate that should MIGA decide to expand its suite of online services in the future, consideration should be given to reducing, consolidating, or winding down some existing services.

C.4. Sustainability: Risks to the services and their outcomes

5.32 Sustainability is assessed in terms of the risks that may affect the services’ ability to realize their intended outcomes. As the FDI Internet landscape becomes more populated, there is a potential that a comparator could expand into MIGA’s niche, or that MIGA’s services are superseded by a more sophisticated product. In addition to this risk related to its comparator advantage, the most likely risks facing the IDS are related to their internal alignment within MIGA, content quality, and technological advances.

56 Survey responses on the impact of each of the investor services indicated the following: the use of IDS led to changes in attitude (36-40% strongly agreed or agreed; 13-17% strongly disagreed or disagreed; 46-51% neither agreed nor disagreed); identification of an investment opportunity (39-64% strongly agreed or agreed; 2-18% strongly disagreed or disagreed; and 35-42% neither agreed nor disagreed), acting on a specific investment opportunity (25-33% strongly agreed or agreed; 19-22% strongly disagreed or disagreed; and 45-56% neither agreed nor disagreed). (n=523 for Privatizationlink; n=617 for IPAnet; n=637 for FDIXchange). See Annex 14.

44

C.5. MIGA Performance: MIGA’s work quality in developing, managing, and monitoring the performance of the services

5.33 As noted above, users are generally satisfied with the sites’ usability and design and indicate they are able to find what they are looking for. The comparator comments with respect to site maintenance indicate that IDS have refined their management of these activities and are able to capture the benefits that outsourcing was expected to provide. Formalizing user interaction and information architecture in the systems development process could help improve usability and increase user activity on the sites. While the IDS conduct some standard monitoring of site traffic, they have not surveyed users in recent years. The IDS could improve the frequency and methods of tracking service performance and outcomes.

5.34 Overall, the IDS demonstrate sound performance in terms of practices in the context of the current content management approach. A significant portion of content in the investor services database (serving IPAnet, FDIXchange, and PrivatizationLink) is more than two years old; however, some of this content has a long “shelf life” as in the case of laws and regulations, and the team does periodically purge the database of older content. The IDS’ approach may not be the most effective since by its nature will not result in consistently high quality content. However, improving content quality activities is resource intensive and will require MIGA to make different decisions about resource allocations and its technology priorities.

45

6. STATUS OF IMPLEMENTATION OF PREVIOUS IEG-MIGA RECOMMENDATIONS

In FY06, MIGA has continued to make progress in implementing most IEG-MIGA recommendations. MIGA has put in place institutional prerequisites for a more thorough and holistic project assessment and has implemented a supervision program for environmental and social safeguard policies. Continued progress is necessary in consistently applying MIGA’s project assessment methodology. After a phase of rapid change in its business model and procedures, it needs to ensure these are being refined and, where necessary, codified, well understood by staff and applied within the Agency.

6.1 Previous IEG-MIGA recommendations have covered several areas: strengthening of ex-ante assessments of MIGA guarantees; monitoring; safeguard policies; partnerships; and strengthening strategic selectivity and monitoring of technical assistance activities.

6.2 IEG notes MIGA’s progress in implementing its recommendations made in previous reports and anticipates further progress during FY07. Thus, there is a need to systematically review all recommendations in the MATR, retiring those fully implemented or no longer relevant, and consolidating related recommendations in order to provide a set of current and valid recommendations to be presented in the 2007 Annual Report.

6.3 The status of the main areas is noted below:

• Strengthening Project Assessments During Underwriting - MIGA has updated its business model and formed a department to integrate

economic, environmental, and social assessment of its projects. - It has built internal capacity through hiring of staff with economic and social skills,

and through training on development impact assessment. - MIGA has made progress in its assessment of development impact during

underwriting, but still needs to consistently apply the methods and improve certain aspects.

• Social and Environmental Safeguard Policies

- MIGA has upgraded the role of environmental and social specialists, for instance, by requiring site visits for all category A and most category B projects.

- It has issued a guideline to ensure early involvement of environmental and social specialists.

- MIGA is making better use of the Contract of Guarantee for strengthening compliance monitoring and reporting, but needs to ensure that specific requirements are implemented.

- MIGA has scaled up its environmental portfolio review (and its supervision of projects, as noted above), but still lacks formal internal guidelines.

- It has made progress in its engagement with Extractive Industries projects that goes beyond compliance with safeguards.

46

- MIGA lacks a more proactive approach for servicing clients on safeguards towards promoting development effectiveness in its projects.

- MIGA needs to consistently ensure that clients comply with policies prior to Board approval.

- MIGA needs to ensure that the required documents/environmental management systems (EMS) are prepared before the guarantee is issued and Board approval.

- MIGA has committed to update its environmental and social safeguard policies within 12 months from the conclusion of the similar updating process in IFC (i.e., by February 2007).

• Monitoring Development Effectiveness

- MIGA needs to expand its current Guarantee Closing Note to include information on the achievement of project objectives, lessons learned, and development impacts for all expired and cancelled guarantees.

- MIGA has approved a framework and has begun in calendar year 2006 a pilot program to monitor the development effectiveness of some current projects.

• Partnerships

- MIGA has not formalized relationships with partners to ensure accountability and improve information sharing.

• Strategic Selectivity and Monitoring of TA

- MIGA has reoriented its technical assistance program and revised its TA work program in FY06, but needs to implement its criteria for a more strategic client selection.

- MIGA has done self-evaluations of several TA activities, and has approved a monitoring framework in FY06.

47

7. RECOMMENDATIONS

Recommendations for MIGA On Environmental and Social Safeguard Policies 1. MIGA should review and strengthen its supervision of Category B projects to ensure their compliance with environmental and social safeguard policies and guidelines. (Section 2.D.) On Projects Involving Concession Agreements or Similar Agreements 2. MIGA needs to develop rules of engagement for projects involving concessions and similar agreements. Considering that MIGA often gets involved in projects as an insurer after such agreements have been negotiated and signed, it needs to satisfy itself that the underlying business model, terms given to the concession holders, and tariffs are sustainable and reflect sound economic policy in order to ensure a positive development impact. (Box 1) On Guarantee Projects’ Contribution to MIGA’s Profitability 3. MIGA should improve its ability to capture costs associated with underwriting, processing, and monitoring individual guarantees. This information would facilitate IEG's ex-post evaluation of the contribution of individual guarantee projects to profitability. (Section 4) On MIGA’s Information Dissemination Services (IDS) 4. MIGA should undertake a strategic review of its information dissemination services, involving users and target users, strategic partners (including other World Bank Group sites), and comparators, in order to ensure its future relevance, and to better align it with its guarantee business. This involves assessing the relevance of its current and future planned services with respect to offerings of comparators and possible gaps in information services and audiences served. This strategic review should also anticipate the impact of evolving technologies on the information dissemination services and their aggregation service model in the long term, and include a client segmentation analysis to determine how to target user communities, build relationships with strategic partners, and serve MIGA’s strategic priorities. It should also facilitate a consistent identity across the currently separate MIGA brands. (Section 5)

48

5. MIGA should adopt a more robust performance monitoring approach for the information dissemination services that integrates and analyzes user insights, site performance metrics, and content to refine and improve the services. This involves tracking regularly how well content provided by its services matches what user segments demand. (Section 5) Recommendations for IEG-MIGA 6. IEG-MIGA should review and revise all recommendations in previous IEG Annual Reports as well as thematic reports discussed by CODE and the Board, as reflected in the Management Action Track Record, consolidating related recommendations and retiring those that have already been implemented fully. IEG will prepare a new list of remaining recommendations with items that need attention in the future. (Section 6) 7. IEG-MIGA should pilot the application of the methodology for assessing the projects’ contribution to MIGA’s profitability presented in this report in ex-post evaluations of guarantee projects. (Section 4)

50

ANNEXES

51

Annex 1. IEG-MIGA Methodology for Evaluating Guarantees

Project Ratings Guidelines

INDICATOR Guidance Benchmarks Definition

DEVELOPMENT OUTCOME

1. Business Performance a) FRR Excellent: FRR > HR + 2.5%

Satisfactory: FRR = HR

Compare after-tax FRR with 10% hurdle rate

Partly Unsatisfactory:

FRR < HR - 2%

Unsatisfactory: FRR < HR <- 2%

b) if FRR cannot be calculated Excellent: Objectives largely

surpassed

Satisfactory: Objectives broadly

achieved


One or more of the core objectives not met

Judge performance relative to project's business/profit objectives. The guiding principle should remain the project's incremental financial impact on the project enterprise.

Unsatisfactory: Most objectives not met

c) Other financial indicators such as net present value, return on assets, debt service ratios, etc. to complement FRR

2. Project's Contribution to Economic Sustainability a) Non-quantified benefits and cost

Excellent: Objectives largely surpassed

Satisfactory: Objectives broadly achieved


One or more of the core objectives not met

Whether the project had any direct impact - positive or negative - on the poor or on living standards in the local community. If the project had any other contributions to international development goals. Or, whether the non-quantified benefits are sufficient to affect a project's overall assessment.

Unsatisfactory: Most objectives not met

52

INDICATOR Guidance Benchmarks Definition b) Quantifiable benefits and costs

Excellent: ERR >= 20%

Satisfactory: ERR >= 10%


ERR> = 5%

Relates to the project's economic effects on its input and output markets. Consider domestic and international competition and how prices and quantities are determined in relevant markets. Economic costs and benefits include taxes paid to the government, consumer surplus, other divergence of financial and opportunity cost, effects on competitors, and benefits to suppliers, including labor.

Unsatisfactory: ERR < 5%

3. Environmental and Social Effects

REFER TO Annex 2. At Board Approval

At Evaluation

4. Private Sector Development Impact

Consider the relevant indicators below:

Excellent: Impact is significant and meaningful relative to country and sector

a) Extent to which local suppliers of goods and services benefit from the project's operations

Satisfactory: Some positive impact beyond the project enterprise

b) Extent to which businesses are using the project's output as their input


No noticeable impact beyond the project enterprise

c) Extent to which new technology and know-how was introduced

Unsatisfactory: Negative impact on the country and sector

d) Quality and scope of employee training (Development of local management skills, percentage of administrative budget, staff-hours, etc.)

e) Extent to which the project stimulated follow-up investments and/or other feasibility studies

53


f) Contribution of project to domestic capital market development (pioneering listing on stock exchange, enhanced disclosure standards, introduction of international accounting standards, etc.)

g) Company's governance, reputation, and business practices as a corporate role model and quality investment asset

h) Extent to which project-related technical assistance, activities, goods and services bring about changes in the enabling environment

i) Project's impact on enhanced private ownership

j) Project's impact on stronger local entrepreneurship

k) Extent to which the project led to greater competition and competitiveness

l) Identify and discuss the project's impacts on competitors, new entrants, and producers of complementary goods.

MIGA’s EFFECTIVENESS

1. Strategic Relevance

a) Consistency of the project with MIGA's sector/thematic strategy/priorities

Excellent: The project's objectives were fully consistent with (a) and (b)

b) Consistency of the project objectives with CAS and the country's key development priorities

Satisfactory: The project's objectives were mostly consistent with (a) and (b)


The project's objectives were partially consistent with (a) and (b)

54


Unsatisfactory: The project's objectives contradict (a) or (b)

2. MIGA's Role and Contribution

a) MIGA's additionality Excellent: MIGA's role was essential for the project to go ahead and MIGA clearly contributed to its success.

b) Promotion of good practices Satisfactory: MIGA's role and contribution were positive in most areas.

c) Effective partnership and coordination arrangements with other sponsors


MIGA's role and contribution fell short in at least one area.

Unsatisfactory: MIGA's role and contribution fell short in key areas.

3. MIGA's Assessment /Underwriting/Monitoring

a) Quality of MIGA's assessment

Excellent: Comprehensive and high quality assessment, underwriting and monitoring

b) Quality of MIGA's underwriting with respect to assessment of risks affecting the project and the methods, process and procedures put in place to diminish the project's risk profile.

Satisfactory: Good quality assessment, underwriting and monitoring of most areas

c) Adequacy and timeliness of MIGA's monitoring


Some gaps/inconsistencies in the quality of assessment, underwriting and monitoring of key areas.

Unsatisfactory: Major gaps/inconsistencies in the quality of assessment, underwriting and monitoring.

55

Annex 2. IEG-MIGA Methodology for Evaluating Guarantees MIGA Safeguard Policies. Criteria for Consistency

A. MIGA Safeguard Policies - Criteria for Consistency at Approval

Criterion Requirements Comprehensive Environmental Assessment

Applies to both majority and minority owners and lenders (designated herein as “sponsors”) – required for all ‘As’ and for ‘Bs’ if host country legislation requires: (i) Comprehensively includes natural environment, social aspects, human health and safety, major hazards, transboundary/global and cumulative/induced impacts; (ii) prevent, minimize, mitigate or compensate for adverse environmental and social impacts and enhance positive impacts; (iii) potential for independent environmental advisory panel in case of highly risky or contentious project;(iv) properly defined area(s) of project impact; (v) for expansion or modernization projects the entire plant is subject to an EA (usually including an environmental audit) ; (vi) privatization projects require environmental audits; (vii) EAs (including environmental audits) to be carried out or reviewed by independent consultants; and (viii) compliance with more stringent of host country or MIGA environmental and health and safety standards or guidelines

Adequate analysis of feasible alternatives

Proper analysis of project alternatives including: (i) without project alternative; (ii) where appropriate other sector alternatives; (iii) alternative sitings for facilities and routings of infrastructure corridors; (iv) alternative technologies and mitigation arrangements; and (v) analysis of feasible alternatives

EHS Guidelines or host country’s regulations adequately addressed

Identification and application of appropriate EHS Guidelines and host country’s environmental and health and safety regulations that apply to the project and an analysis of the project’s compliance with these guidelines and regulations.

Comprehensive E&S baseline survey

Full description (with adequate support data) of the climatic, geological, topographical, physical, chemical, biological and socio-cultural-economic environment of the area of project impact as a basis for an adequate analysis of project impacts and future monitoring of the efficacy of the mitigation measures incorporated into the project

Adequate EAP or EMP proposed

A detailed plan of the set of mitigation, monitoring and reporting measures proposed to be taken during project implementation to eliminate adverse environmental or social impacts, offset them, or reduce them to acceptable levels – required for all ‘As’ and ‘Bs’.

Project Sponsor’s EMS adequate

Comprehensiveness of environmental, social and safety management system proposed by the sponsor (including contractors) to fully implement the EAP or EMP, as well as appropriateness of proposed measures to strengthen these arrangements

Public disclosure/consultation addressed

(i) consultation with local affected parties and local interest groups during EA process; (ii) disclosure of information in a timely manner and in a language and form understandable and accessible to local groups; (iii) for “A” projects final EA reports disclosed locally and through the World Bank Info-shop at least 60 days before MIGA Board approval.

Comprehensive and implementable RP/CDP prepared

(i) avoid or minimize involuntary physical resettlement or economic displacement; (ii) directly affected and displaced persons should be: (a) informed of their options and rights regarding land acquisition and resettlement as well as alternatives that are available; (b) compensated for their losses at full replacement cost prior to the actual move; (c) assisted with the move and supported during the transition period in the resettlement site; and (d) assisted in their efforts to improve their former living standards, income earning capacity, and production levels, or at least to

56

Criterion Requirements restore them. Particular attention should be paid to the needs of the poorest groups to be resettled; (iii) Land, housing, infrastructure, and other compensation should be provided to the adversely affected population, indigenous groups, ethnic minorities, and pastoralists who may have usufruct or customary rights to the land or other resources taken for the project. The absence of legal title to land by such groups should not be a bar to compensation; (iv) alternative or similar resources provided to compensate for the loss of access to community resources; (v) in new resettlement sites or host communities improve, restore or maintain accessibility and levels of service for the displaced persons and host communities (vi) minimize impacts on host communities including consultation with these communities; (vii) consult and involve affected people in planning, and implementation; (viii) community level impacts require preparation of community development programs to improve the economic and social well-being of the affected communities as well as the affected households; (ix) preparation of a resettlement plan (RP), or other resettlement instrument (e.g., resettlement framework) as agreed with MIGA; and (x) disclosure of RPs involving more than 50 households or 250 people.

Comprehensive and implementable IPP prepared

Appropriate identification of indigenous groups in project area, namely those having: (a) close attachment to ancestral territories and the natural resources in them; (b) self-identification and identification by others as members of a distinct cultural group; (c) presence of customary social and political institutions; (d) economic systems primarily orientated to subsistence production; and (e) and indigenous language. Ensure: (i) avoidance and mitigation of adverse impacts; (ii) informed participation of the indigenous peoples themselves; (iii) culturally appropriate compensatory measures or social and economic benefits; and (iv) in consultation with indigenous peoples preparation of an Indigenous Peoples Plan (IPP).

Natural habitats protected or offsets provided

(i) Project does not significantly convert/degrade a critical habitat; (ii) natural habitats are correctly identified; (iii) alternative analysis examines alternatives to significant conversion; (iv) if conversion can not be avoided, impact are minimized, mitigated and offset requirements are examined.

Comprehensive Dam Safety measures proposed

New Dams: Safety measures from design to operation for dam and associated works, including for: (i) dams >15 meters in final height; (ii) for special case (flood prone, seismic area, difficult foundations, toxic materials, etc) dams between 10 and 15 m; and (iii) for dams initially under 10 m if expected to become large dams during construction, require the following: (a) reviews by independent expertise throughout design and construction of dam and for start of operations; (b) plan for construction, supervision and quality assurance, plan for instrumentation, an O&M plan, and an emergency preparedness plan; (c) construction by fully qualified companies under proper supervision; (d) periodic safety inspections after completion of construction; Existing Dams: (i) independent dam specialist(s) to evaluate safety status, performance history and owner’s operation/maintenance procedures; and (ii) specify remedial works or safety-related measures to upgrade dam to an acceptable standard of safety. Tailings Dams and Ash Lagoons: (i) this policy applies to such dams in excess of 10 m if: (a) the impoundment is cross-valley structure; or (b) after construction of a starter dam, the impoundment structure is made of whole tailings; or (c) standard testing methods indicate net acid generating potential of tailings or ash. However generic safety measures designed by qualified engineers are adequate for such dams less than 10 m in height, if tailings or ash have no net acid generating potential and impoundment is : (a) located in relatively flat terrain, highly arid areas or in permafrost zones; and (b) not

57

Criterion Requirements subject to inflow from streams or rivers: (ii) stream diversions and spillways to be designed for 100 yr flood; and (iii) preparation of closure and abandonment plans.

Cultural Property protection proposed

(i) avoid harm to significant, non-replicable cultural property or with the help of qualified experts mitigate such impacts if loss is judged to be minor or otherwise acceptable; (ii) sponsor addresses protection/management of cultural property in project area including “chance finds”; (iii) sponsor meets host country regulations/laws (or adheres to best practice in the absence of host country laws); and (iv) sponsor consults with relevant stakeholders in documenting presence and significance of physical cultural resources.

The set of requirements for each criterion of safeguard policy compliance were rated according to the following scale:

• High: the set of requirements were fully met, or expected to be fully met, with no shortcomings

• Substantial: the set of requirements generally were met, or expected to be met, with only minor shortcomings

• Modest: the set of requirements were met, or expected to be met, but with significant shortcomings

• Negligible: the set of requirements were not met, or expected not to be met, due to major shortcomings

58

B. MIGA Safeguard Policies - Criteria for Consistency at Evaluation

Criterion Requirements EAP or EMP fully implemented

Assess how effectively the EAP or EMP has been implemented by the sponsor and note any gaps and deficiencies. Note how well EAP or EMP implementation progress has been documented and reported in a timely manner. Note any deviations from the original plan and if these were appropriate considering the circumstances.

E&S monitoring implemented

Assess if the EAP’s or EMP’s E&S monitoring plan has been implemented according to the timing proposed. Assess if the monitoring results are substantiating the effectiveness of the E&S mitigation measures or not. Note if the results are being used to take corrective measures if needed.

Sponsor’s project implementation EMS effective

Determine if the sponsor has implemented the environmental, social and safety management system proposed in the EAP or EMP. Assess the effectiveness of the proposed institutional strengthening measures to improve this system and whether the system has active sponsor management support. Assess its sustainability in the longer term.

Continuing public disclosure and consultation

Determine the extent to which project affected groups and other stakeholders continue to be consulted and involved during the implementation phase of the project. Assess if there have been any complaints by project affected people and how these complaints were dealt with by the Borrower.

Full compensation of PAPs

Assess if displaced persons have been: (a) compensated for their losses at full replacement cost prior to the actual move; (b) assisted with the move and supported during the transition period in the resettlement site; and (c) assisted in their efforts to improve their former living standards, income earning capacity, and production levels, or at least to restore them.

RP/CDP fully implemented

Determine if the RP/CDP has been fully implemented by the sponsor. Assess if the sponsor has adequately monitored and evaluated the activities set forth in the RP/CDP. If upon termination of the contract of guarantee the RP/CDP has not been fully implemented assess what follow up actions the sponsor proposes to meet the objectives of the plan and if these are adequate.

IPP fully implemented Determine if the IPP has been fully implemented by the sponsor. Assess if the sponsor has adequately monitored and evaluated the activities set forth in the IPP. If upon termination of the contract of guarantee the IPP has not been fully implemented, assess what follow-up actions the sponsor proposes to meet the objectives of the plan and if these are adequate.

Natural Habitats protected or offsets provided

Assess if sponsor has taken all necessary measures to limit any significantly conversion/degradation of critical natural habitat and/or provide offset requirements as proposed in the EA. Assess the sustainability of these measures once the project has been implemented.

Dam Safety measures implemented

For new dams covered by the policy, assess if the safety measures recommended by the independent dam expert(s) throughout investigation, design and construction of dam and start-up of operations were implemented. Evaluate effectiveness of plans for construction, supervision and quality assurance, as well as for instrumentation, O&M and emergency preparedness. Assess the results of

59

Criterion Requirements periodic safety inspections after completion of construction. For existing dams, assess if the safety measures proposed by the independent dam specialist(s) have been implemented as proposed and note any deviations.

Cultural Property protected

Assess if appropriate measures were taken by the sponsor to avoid harm to significant, non-replicable cultural property and provide protection/management of cultural property in project area including “chance finds” according to best practice or host country regulations/laws.

Contract of Guarantee for implementation of safeguard policies/ guidelines adequate

Assess if the following requirements have been met in contract of guarantee: (i) Specification of safeguard policies and EHS guidelines that apply to the project; (ii) MIGA ensures that contract of guarantee includes an obligation to carry out the EAP/EMP and includes as additional conditions specific measures under the EAP/EMP, as appropriate for facilitating effective monitoring on EMP implementation; and (iii) the sponsor’s obligations to carry out the RP/CDP and/or IPP (or other instrument agreed with MIGA) and to keep MIGA informed of implementation progress are provided for in the contract of guarantee.

Reporting on safeguard policies by sponsor adequate

Determine if MIGA has specified a comprehensive set of safeguard policy performance indicators that are appropriate for the project under implementation. Assess the timeliness and effectiveness of the reporting of indicators and their evaluation by the sponsor and MIGA, noting any deficiencies.

M&E of safeguard policies by MIGA adequate

MIGA reviews regular monitoring reports on safeguard compliance provided by the sponsor and notes any areas of concern for follow up with sponsor. MIGA bases supervision of the projects environmental/social/safety aspects on the findings and recommendations of the EA, including measures set out in the legal agreements, any EMP and other project documents, and ensures that supervision missions contain adequate environmental and social expertise. During supervision MIGA reviews sponsor’s implementation progress (incl. progress reports) and assesses Borrower’s compliance with agreed environmental actions, particularly the implementation of environmental and social mitigation, monitoring and management measures. If compliance is unsatisfactory, MIGA discusses with sponsor actions necessary to correct non-compliance and follows up on the implementation of such actions.

The set of requirements for each criterion of safeguard policy compliance were rated according to the following scale:

• High: the set of requirements were fully met, or expected to be fully met, with no shortcomings

• Substantial: the set of requirements generally were met, or expected to be met, with only minor shortcomings

• Modest: the set of requirements were met, or expected to be met, but with significant shortcomings

• Negligible: the set of requirements were not met, or expected not to be met, due to major shortcomings

60

Annex 3. IEG-MIGA Ex-Post Guarantee Project Sample Composition As of March 31, 2006

Year of Underwriting 57

Items FY96 FY97 FY98 FY99 FY00 FY01 FY02

FY96-FY02

Total Number of MIGA Guarantee Projects Evaluated Ex-Post by IEG:58

1 3 10 6 0 0 1 21

Still active as of March 31, 2006 1 0 5 2 0 0 1 9 Cancelled after IEG evaluation 0 3 5 4 0 0 0 12 Total Number of MIGA Guarantee Projects: 37 42 21 38 33 35 33 239 Still active as of March 31, 2006 5 9 8 8 7 16 19 72 Cancelled as of March 31, 2006 32 33 13 30 26 19 14 167

57 Extension of the sample to FY01 and FY02 was approved by CODE and the Board as a result of the FY2005 Annual Report discussions. 58 Excludes 11 projects evaluated for the Extractive Industries Study.

61

Annex 4. FY96-FY2002 Evaluated MIGA Projects Composition of Evaluated Projects

By Sector Number of

projects

Agriculture 1 Manufacturing 6 Services 2 Tourism 1 Infrastructure 8 Financial 3

Projects by Sector

Infrastructure38%

Financial14%

Agribusines, M anufacturing and Services

48%

By Region Number of

projects Latin America and the Caribbean 6 Africa 4 Europe and Central Asia 5 Asia and Pacific 6

Projects by Region

LAC28%

AFR19%

ECA24%

ASIA29%

By Income Level Number of projects

IDA-eligible/Blend Countries 7 Middle Income Countries 14

Projects by Country Classification

IDA33%

M iddle Income Countries

67%

By Year of Underwriting Number of projects RDE 2004 AR 2006

1997 2 1 1998 7 4 1999 6 2002 1

2

71

4

6

10

2

4

6

8

10

12

1997 1998 1999 2002

RDE 2004 AR 2006

62

Annex 5. FY96-FY02 Evaluated MIGA Projects

Summary of Evaluation Ratings

Indicators 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Business Performance U U U U PU S E S U E E PU U S E U S S E S PU

Contribution to Economic Sustainability PU U U PU S S E S U PU S S S S S U S S E PU S

Environmental and Social Effects S S E E S PU E N.R. PU U U U E S PU S N.R. PU U S N.R.

PSD Impact S PU U PU PU S E PU S S S S PU E E S PU PU S U S

Development Outcome PU PU U PU PU S E S PU PU S PU PU S S PU S S S PU S

Strategic Relevance E PU U S S S E S E S S S S S E S S S S S S

MIGA's Role and Contribution S PU U E S S S S S S S S S E S E S S PU E S

MIGA's Assessment, PU U U U PU PU S PU U PU PU PU S S S PU U PU U U S

Underwriting, and MonitoringMIGA's Effectiveness S PU U PU S S S S PU PU PU PU S S S S PU S PU PU S

Ratings:E = ExcellentS = SatisfactoryPU = Partially UnsatisfactoryU = UnsatisfactoryN.R.= Not Rated

Projects

63

Annex 6. Safeguard Policy Consistency Ratings of MIGA Projects at Approval

Project No 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21EA Category B B A B B B B C* B B B B B B A B C* B B B C*MIGA Client Category (1) Re/Len

1Maj Min Min/Le

n1Maj Maj Maj Maj

Comprehensive Environmental Assessment (2) PU PU E S PU S E S S S S PU S E PU S PU EAdequate analysis of feasible alternatives S PU S S PU PU E S S S S S PU S S S PU SEHS Guidelines or host country regulations comprehensively addressed

S S E E S PU E S S S S S S S S S PU S

Comprehensive E&S baseline survey S S S SAdequate EAP proposed S S E S S PU S S S S S S S S S S S SProject investor's EMS adequate PU PU S S S PU E PU S S S PU S S S PU S SPublic disclosure / consultation addressed S SComprehensive and implementable RP/CDP prepared U S

Comprehensive and implementable IPP preparedNatural Habitats protected or offsets provided PU U SComprehensive Dam Safety measures proposed ECultural Property protection proposedPest Management provisions proposed SAverage Score 2.8 2.4 3.3 3.2 2.6 2.3 3.3 N.R. 2.6 3.0 3.0 3.0 2.6 2.8 3.1 2.8 N.R. 2.8 2.4 3.2 N.R.Degree of Compliance 60% 47% 77% 73% 53% 43% 77% N.R. 53% 67% 67% 67% 53% 60% 71% 60% N.R. 61% 47% 73% N.R.

(1) Guarantee Holder’s identified as: Maj=Major project investor; Min=Minor project investor; Lenl=Lender to major investor; Len2=Lender to minor investor; Re=Reinsurance. * Category C Projects are financial sector projects, and, therefore, are not subject to MIGA's Safeguard Policies (2) An E.A. (Environmental Assesment) may include: an environmental impact Assessment; environmental audit, and hazard or environmental risk assessment or a combination of these instruments. Individual Safeguard Criteria Rating: E : Excellent: the set of requirements were fully met, or expected to be fully met, with no shortcomings. S : Satisfactory: the set of requirements generally were met, or expected to be met, with only minor shortcomings PU : Partially Unsatisfactory: the set of requirements were met, or expected to be met, but with significant shortcomings. U : Unsatisfactory: the set of requirements were not met, or expected not to be met, due to major shortcomings.

Scoring system used the following values: An average score of 3.3-4.0 means that the set of requirements were fully met, or expected to be fully met, with no shortcomings. An average score of 2.5-3.means that the set of requirements generally were met, or expected to be met, with only minor shortcomings An average score of 1.7-2.4 means that the set of requirements were met, or expected to be met, but with significant shortcomings. An average of 1-1.6 means that the set of requirements were not met, or expected not to be met, due to major shortcomings.

64

Annex 7. Safeguard Policy Consistency Ratings of MIGA Projects at Evaluation Project No 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21EA Category B B A B B B B C* B B B B B B A B C* B B B C*MIGA Client Category[1] Re/Len

1Maj Min Min/Le

n1Maj Maj Maj Min/Le

n1Min/Le

n1Min/Le

n1

EAP/EMP fully implemented by investor S E E E S PU E PU U U U S S S S PU U SEHS Guidelines or host country regulations fully met S S E E E PU E PU U U U S PU U S PU U S

E&S monitoring fully implemented by investor S S S S S S E S U U U PU S U PU S U SInvestor's project implementation EMS effective E E S S E S S PU S S S S S S S PU S SContinuing public disclosure and consultation S SFull compensation of PAPs SRP/CDP fully implemented PU SIPP fully implementedNatural Habitats protected or offsets provided S S SDam Safety measures implemented ECultural Property protectedPest Management measures implemented PUAverage Score at Evaluation 3.4 3.5 3.3 3.5 3.5 2.5 3.6 NR 2.2 1.5 1.5 1.5 2.8 2.8 2.4 2.8 NR 2.4 1.5 3.0 NRDegree of Compliance 80% 83% 77% 83% 83% 50% 87% NR 40% 17% 17% 17% 58% 58% 48% 58% NR 47% 17% 67% NR (1) Guarantee Holder’s identified as: Maj=Major project investor; Min=Minor project investor; Lenl=Lender to major investor; Len2=Lender to minor investor; Re=Reinsurance. * Category C Projects are financial sector projects, and, therefore, are not subject to MIGA's Safeguard Policies (2) An E.A.(Environmental Assesment) may include: an environmental impact Assessment; environmental audit, and hazard or environmental risk assessment or a combination of these instruments. Individual Safeguard Criteria Rating: E : Excellent: the set of requirements were fully met, or expected to be fully met, with no shortcomings. S : Satisfactory: the set of requirements generally were met, or expected to be met, with only minor shortcomings PU : Partially Unsatisfactory: the set of requirements were met, or expected to be met, but with significant shortcomings. U : Unsatisfactory: the set of requirements were not met, or expected not to be met, due to major shortcomings.

Scoring system used the following values: An average score of 3.3-4.0 means that the set of requirements were fully met, or expected to be fully met, with no shortcomings. An average score of 2.5-3.means that the set of requirements generally were met, or expected to be met, with only minor shortcomings An average score of 1.7-2.4 means that the set of requirements were met, or expected to be met, but with significant shortcomings. An average of 1-1.6 means that the set of requirements were not met, or expected not to be met, due to major shortcomings.

65

25%

75%

LOW

HIGH

33%

67%

LOW

HIGH

Annex 8. Development Outcome Results for Projects Evaluated During FY04-05

(Twelve Evaluated Projects) Development Outcome: 58% of the evaluated projects were rated as satisfactory. None of the projects had an excellent overall rating. The overall rating is determined on the basis of the following four underlying dimensions:

Business Performance: Business Performance measures the financial impact of the project on the project financiers as measured by the

financial rate of return (FRR). 67% or eight out of the twelve evaluated projects achieved a satisfactory or excellent financial performance. Four projects were rated excellent for business performance, which implies an FRR of 12.5% or higher.

Contribution to Economic Sustainability: Economic sustainability assesses the net effect of the project on the host country economy, living

standards, quality of life, and the project’s overall social and economic impacts. The indicator reflects the project’s contribution as measured by the economic rate of return (ERR), and also takes into account qualitative economic benefits and costs. Overall, 75% of the evaluated projects were considered to be economically sustainable. Eight of these projects had a “satisfactory” rating, and only one was evaluated as “excellent”.

Environmental and Social Effects: Project’s impact on the physical environment and on social, cultural and health and safety issues,

including compliance with MIGA’s safeguard requirements and relevant policies. Environmental and social effects is the lowest-rated of the four indicators. Since two projects were in the financial sector, only 10 projects could be evaluated for Environmental and Social effects. Of these, 60% had a low performance; four were rated as unsatisfactory.

Private Sector Development (PSD) Impact: Project’s contribution beyond the project enterprise, e.g., impact on

the inflow of foreign capital, on the sector, through demonstration effects, and on local businesses and the extent to which the company is a corporate role model. 67% of the evaluated projects made positive and sustainable contributions to PSD. Of these, four projects were rated excellent

High= Excellent and Satisfactory ratings

Low= Unsatisfactory and Partially Unsatisfactory ratings


33%

67%

LOW

HIGH

60%

40%

LOW

HIGH

42%

58%

LOW

HIGH

66

Annex 9. MIGA’s Effectiveness Results for Projects Evaluated During FY04-05 (Twelve Evaluated Projects)

MIGA’s Effectiveness: Quality of MIGA’s work and project involvement through the following three indicators:

MIGA’s Role and Contribution: Assesses MIGA’s additionality, influence on project design, and synergy

with partners. In only one of the twelve evaluated projects was MIGA’s role considered as low. Within the eleven projects in which MIGA’s relevance was considered high, one project was rated as excellent.

Strategic Relevance: Project’s linkage and consistency with MIGA’s, the WBG’s, and the host

country’s objectives and priorities are considered under this indicator. All twelve projects were found to be strategically relevant; however, only one of them was rated as Excellent. Most of the projects were considered to have a satisfactory strategic relevance, in line with the WBG’s country strategies and particular countries’ situation at the time of implementation and development of the project.

High= Excellent and Satisfactory ratings Low= Unsatisfactory and Partially Unsatisfactory ratings

MIGA’s Assessment, Underwriting, and Monitoring: Quality and appropriateness of

MIGA’s own work, quality of underwriting analysis and monitoring arrangements. In 67% of evaluated projects MIGA fell short in this category. Of these, five projects were rated as partially unsatisfactory and three unsatisfactory.


0%

100%

LOW

HIGH

67%

33%

LOW

HIGH

8%

92%

LOW

HIGH

50%

50%

LOW

HIGH

67

Annex 10. Potential and Actual Risks Affecting Evaluated MIGA Projects at Underwriting and at Evaluation

Flagged at Underwriting Identified at Ex-Post Evaluation Number of

Risks

Changes in Sector/Market

Conditions

Changes in Sector/Market

Conditions

Proj

ect C

ode

Mac

roec

onom

ic a

nd/o

r Pol

itica

l C

risi

s Se

ctor

D

evel

opm

ent

Regu

lato

ry o

r C

once

ssio

n Is

sues

Clie

nt R

isk

Proj

ect S

peci

fic R

isks

Envi

ronm

enta

l & S

ocia

l Sa

fegu

ards

Ris

ks

Mac

roec

onom

ic a

nd/o

r Pol

itica

l C

risi

s

Sect

or

Dev

elop

men

t

Regu

lato

ry o

r C

once

ssio

n Is

sues

Clie

nt R

isk

Proj

ect S

peci

fic R

isks

Envi

ronm

enta

l & S

ocia

l Sa

fegu

ards

Ris

ks

Flag

ged

at U

nder

wri

ting

Act

ual R

isks

Iden

tifie

d at

Ex-

Post

Eva

luat

ion

1 2 2 2 0 3 3 2 3 4 5 5 5 1 2 6 3 3 7 2 0 8 0 1 9 6 5 10 4 4 11 4 4 12 4 4 13 1 2 14 0 2 15 3 1 16 2 2 17 0 1 18 1 2 19 0 1 20 4 1 21 1 2 Note: ( ) Refers to projects involving concession, power purchase or other forms of similar agreements with the government but there were no issues or disputes about the agreement at underwriting and at evaluation. Macroeconomic and/or Political Crisis: Major macro-economic or political crisis within the past 5 years. Refers projects in Russia, Indonesia, Turkey and Brazil. Changes in Sector/Market Conditions: Sector Development: Reform or restructuring in progress; known reform policy issues; vulnerability of the sector to global, political or climatic conditions (agribusiness, monopoly); complex and large projects (oil &

68

gas, power, transport, mining, water supply); fundamental shift in the sector due to technological change, increased competitiveness of other players in the business, and changing consumer preferences. Regulatory or Concession Issues (involving only projects with concession, power purchase and similar agreements with the government): Dispute with host government over provisions; renegotiation of tariffs, other charges and privileges provided in the agreement; unilateral change in the terms and scope of the concession. Client Risk: Inexperienced with business line; weak prior performance; doubtful business reputation, including governance issues; poor financial capacity; minority shareholder and/or investor; history of dispute with government. Project Specific Risks: Greenfield; weak loss of competitiveness advantage in the market; insufficient assessment of potential market; no competitive bidding; unsuitable or outdated technology; and lack of market niche advantage. Environment and Social Safeguards Risks: Category A and B when safeguard compliance is of concern; history of weak compliance by country/sector/client.

69

Annex 11. Methodological Note for Evaluating MIGA’s Information Dissemination Services

Assessment Approach

This annex outlines the approach to the assessment of MIGA’s Information Dissemination Services (IDS). It includes an overview of the major work activities, a discussion on the framework for the assessment including proposed indicators, and a proposal for how information and data is gathered.

Overview of Assessment Activities, Responsibilities and Outcomes The deliveries for this IEG evaluation includes a work plan and a note outlining the assessment approach (this document), and a draft and final report, synthesizing the findings of the evaluation. Activity Interim Checkpoint Outcome 1. Confirm work plan • Reviewed by IEG, Peer

Reviewer • Work plan is agreed to by both teams • Work plan reviewed and accepted by

IEG, Peer Reviewer 2. Confirm project

approach and framework

• Reviewed by IEG, Peer Reviewer

• Approach and framework reviewed and accepted by IEG, Peer Reviewer

3. Conduct interviews and desk research

• Feedback on proposed interview list: o IEG o Suggestions from MIGA

• Overview assessment of services and comparator assessment developed

4. Develop, deliver and analyze results from client surveys

• Input on survey design from: o IEG o Suggestions from MIGA

• Surveys administered and results compiled and analyzed

5. Conduct site usability audit

• Usability audit of five service sites

6. Conduct online prominence study

• Analysis of online prominence of each of the five services

7. Collect and analyze site activity data

• Site activity analysis completed for five service sites

8. Draft and revise report

• Feedback from : o IEG o Circulate to MIGA for

comment

• Draft and final reports reviewed and accepted by IEG

• Assessment report finalized • Presentation made to MIGA team

70

Framework and Approach

The evaluation uses IEG’s objectives-based approach to assess the IDS, examining the overall relevance, effectiveness and efficiency of the IDS. In addition, the sustainability of the services is reviewed as well as MIGA’s performance.

Relevance assesses the context in which the IDS were developed and delivered. This includes the consistency of the strategy and objectives of the IDS with the overall MIGA mandate and strategy. From an internal perspective, relevance looks at the relationship between the evolution of the objectives of the IDS and MIGA’s strategy and priorities and their alignment with the World Bank. From an external perspective, an examination of select comparators identifies how the IDS are positioned relative to other organizations which use the internet to provide FDI information. A client survey gathers information on the how the IDS are perceived as providers of valued services. Effectiveness assesses what the services have achieved by examining the chain of inputs, activities, outputs and outcomes, relative to the objectives59 of the services and the definition of the client groups the services are intended to benefit.

o Inputs include the operating budget, staff and partner relationships for the design, development and ongoing operation of the services. Evidence of inputs is largely comprised of internal strategy, planning and budget documents.

o Activities focus on the work to design, develop and maintain the services themselves including the ongoing management of partner relationships. Evidence of activities involves presentations by and interviews with staff responsible for undertaking them as well as internal documentation such as memoranda of understanding with partners, any process documentation which may exist for activities such as content management, and historic user activity analysis reports.

o Outputs and the evidence used to assess outputs are the content (the number of objects, sources and currency), including learning tools, the usability of the sites and the tracking and analysis of which client groups use them. The sites’

59 Where the objectives of a service are not explicitly stated, assumptions were made based on MIGA’s Convention and other internal MIGA documents and staff interviews. Assumptions are explicitly identified as such.

Figure 28: Assessment Components

Inputs Activities Outputs Outcomes Impact

Relevance Effectiveness Efficiency Sustainability MIGA Performance

71

activity analysis includes an assessment of trends in use such as numbers of registered users (where registration is required) and related site activities. A content audit “spot checks” content for reliability, timeliness and relevance.

o Outcomes are the client benefits derived from using the services. Evidence for outcomes is gathered through surveying clients on their perception of the value and use of the services, and through interviews. Outcomes include the intended clients or users using the sites and their services, the perceived value of the information/training provided to them, perceived savings in time or cost from using the services, and the clients’ perception of the quality of the content provided (reliability, timeliness, and relevance). The evaluation also assesses whether the site content is accurate, current, and relevant through a spot check of selected pages, using a checklist and benchmarks. This analysis also considers the counterfactual scenario, i.e., could the user obtain the information without these services.

o Impact is the final link of the causality chain and is the longer term consequences of using the services. Impact is considered to be the change in knowledge, behaviors or attitudes which clients identify as a result of using the sites and their services; for example, a user acts on information or applies knowledge gained through the services. An investor might proceed with an assessment of a specific opportunity identified through the service, or an IPA might experience an increase in investor inquiries or qualified leads as a result of applying knowledge on investment promotion.

Efficiency examines the allocation of resources to achieve the services’ objectives. The services’ objectives include not only the development and launch of new or improved functions and features, but also their ongoing operation. The cost of delivering the services is considered relative to their use. Decisions related to outsourcing activities or build versus buy are also noted. The resources required to accomplish the work program include the budget and staff complement as well as any additional sources of funds or expertise such as those that may come from programs or donors outside of the MIGA operating budget. Sustainability examines the risk factors which may negatively affect realizing the services’ intended outcomes. The risks identified may include comparator advantage, technology obsolescence, and internal organization changes (such as a reallocation of internal resources for example). The likelihood of these risks occurring is identified through interviews and research rather than statistically modeled. Mitigation strategies are identified where they exist. MIGA’s Performance considers quality assurance processes, practices or controls applied by the IDS team throughout the system development lifecycle. The assessment also considers the feedback provided through the client survey, the activity and prominence analyses, and the site usability analysis. As previously noted, a content audit “spot checks” content for reliability, timeliness and relevance. The comparator assessment may provide insights on good practices.

72

Outputs, Outcomes and Associated Indicators

To assess the IDS, indicators have been identified for the outputs and outcomes in the assessment framework.

Indicators for outputs:

Evidence collected primarily from the usability audit, the online prominence study, and the site activity analysis

Historic documentation, where available, on site usage statistics (e.g. growth in registered users, composition of users)

Indicators for outcomes:

Evidence collected through the client survey

Evidence collected through interviews with MIGA staff and comparators

Analysis, where feasible, of IDS site content and content providers

Analysis of services provided by comparators

Service Objectives60 Target Clients61 Activities Indicators: Outputs

Indicators: Outcomes

IPAnet 1. To fulfill MIGA’s mandate of dissemination and exchange of information on investment opportunities and conditions in member developing countries and transitional

• investors • Advisors • Researchers

• Provides information resources catalogued by country, topic and sector

• Information provided by the WBG, IPAs and government agencies, private sector

1. The site’s usability is acceptable

2. The site has reasonable online prominence

3. The site activity analysis indicates stability or growth in use (based on a

1. Clients perceive the service as saving time and costs (e.g. research)

2. Clients perceive the information provided is reliable, timely and relevant

3. The service is distinct compared to

60 Note that the “best” source for objectives needs to be confirmed where initiating strategy documents are not available; and this may affect the indicators selected. 61 MIGA, Annual Report 2004, p. 20.

Inputs Activities Outputs Outcomes Impact

Relevance Effectiveness Efficiency Sustainability MIGA Performance

73



economies business information providers

several usage indicators)

4. The information providers are credible sources including member developing countries

5. A reasonable proportion of site users are from the target client groups (where this can be tracked by registered users or from the log file)

those provided by other organizations (comparators)

4. MIGA Operations perceive the service as supporting the dissemination and exchange of information

5. Clients learned of and pursued investment opportunities as a result of using IPAnet

Privatizat-ion Link

1. Reduce transaction costs associated with privatization programs by providing investors with key, timely information on state owned enterprises (SOEs) being divested throughout the developing world and by providing privatization agencies with a free channel to disseminate these opportunities

• Investors • Advisors • Financial

institutions

• Provides privatization information on developing countries

• Provides profiles of enterprises being divested by privatization agencies

1. See IPAnet 1.-5.

2. The information provided is on state owned enterprises being divested

1. See IPAnet 1.-5. 2. Clients learned

of and pursued privatization opportunities as a result of using PrivatizationLink

FDIXchange

• To link international investors to current investment opportunities and business environment information in developing

• Investors • Advisors • Financial

institutions

• Provides a customized email alert service on new investment information and related information on country, and sources of

1. See IPAnet1. – 5. (where the service, as opposed to the site is the object of assessment)

2. Clients read the email alerts

1. See IPAnet 1.-5.

74



countries through personalized e-mail updates and a web-based archive62

financial risk management services

• Information provided by the WBG, IPAs and government agencies, private sector business information providers, privatization agencies, chambers and business associations63

pushed to them


• Supports the efforts of investment promotion professionals to attract and retain FDI64

• Serve the growing community of investment promotion professionals, particularly those in developing countries by providing services to help practitioners update their knowledge and skills covering the complex and often lengthy process of securing

• Investment promotion professionals

• Provides a learning toolkit on how to attract and service investors

• Provides promotional tools including case studies and best practice research

• Provides investor research resources

1. See IPAnet1.-5.

See IPAnet1.-3. 1. MIGA Operations

perceive the service as supporting investment promotion professionals to attract and retain FDI.

2. Clients’ use of the FDI Promotion Center has resulted in improved knowledge and skills related to investment promotion

3. Clients have applied the knowledge and skills they have acquired through use of FDI Promotion Center to increase inquiries by

62 MIGA, FDIXchange: A customized information service facilitating Foreign Direct Investment in emerging markets (brochure, no date) 63 MIGA, FDIXchange: A customized information service facilitating Foreign Direct Investment in emerging markets (brochure, no date) 64 MIGA, Foreign Direct Investment Promotion Center: Knowledge and learning resources for investment promotion practitioners (brochure, no date)

75



inward investment65

qualified investors

IPAworks • Enable developing country IPIs to establish a professional Internet presence quickly and inexpensively

• Investment promotion agencies

• Provide a website structure and instructions and coaching to implement it

1. Number of clients using IPAworks to develop their website

2. Clients are able to set up and maintain a website using the toolkit

1. See IPAnet1. & 3

2. Clients have increased qualified investor inquiries since using IPAworks to set up their website

Information and Data Gathering: The methods for collecting information and data include interviews, desk research, a client survey and an examination of the IDS sites including a site review and activity analysis. In addition, a “spot check” is conducted on the content of the IPAnet site (the main content database). The interview list includes current and former IDS team members, and other MIGA staff in the External Outreach and Partners Group and Operations (technical assistance and guarantees). Other interview targets include the World Bank’s Private Sector Development departments (i.e., the Rapid Response Unit) and Foreign Investment Advisory Service. Interviews are conducted with suppliers of FDI (investors) and users of FDI (IPA) clients as well as several comparators. The desk research includes a review of internal documents such as the MIGA strategy and budget documents, memoranda of understanding with partners, and work programs or other documents which provide insight into how the IDS are managed and operated. The client survey includes both a “pop up” and a “long form” survey. The "pop-up" survey is offered to all users who visit one of the MIGA sites during the survey period and is displayed when a user enters the site. All of the responses to this survey are analyzed. The "long form" survey is sent to a list of MIGA's guarantee and technical assistance clients, as well as the roughly 23,000 individuals signed up for MIGA online services. The survey is sent to all of the names on these lists. An incentive is offered to a number of randomly selected respondents in order to increase the response rate. In addition to the survey, a site review is conducted using a suitable usability audit tool and method and an activity analysis examines and extrapolates trends from the log file data. The analysis of log file and registration data identifies usage patterns for the services. The focus of the analysis of the log file data is not on absolute numbers, but rather on trends over time. The assumption is that active, well-managed sites should show consistent traffic growth. The trend analysis includes page views and megabytes transferred which indicate the activity patterns of users. Trends across the number of hosts are also analyzed. Finally the top pages visited for each 65 MIGA, Spotlight: The Foreign Direct Investment Promotion Center, Q305

76

site are analyzed as well as any informative page referral information. For the registration data, the cumulative growth in sign-ups for each service is tracked, with a view to assessing consistent growth. To the extent that sign-ups by profile information can be segmented they are analyzed to determine any patterns among groups of users accessing the services.

77

Annex 12. Survey Administered to IDS Users

Survey Variant

Pop-

up

E-m

ail,

IPIs

E-m

ail,

Inve

stor

s

Respondent profile 1. Please state the country in which you are based or from which you operate. X X X 2. Please select the answer that best describes you:

♦ I am a business executive/officer/entrepreneur. ♦ I am an investor. ♦ I am a consultant/lawyer/adviser. ♦ I am an academic, student, or researcher. ♦ I am an investment promotion professional. ♦ I am a financial analyst/banker. ♦ I am interested in foreign direct investment in general. ♦ Other ___________

X X X

3. Please select the answer that best describes you: ♦ I work for an organization that is a current or potential direct investor in a

foreign country. ♦ I work for a consulting firm providing advice and services to foreign investors. ♦ I work for an institution that promotes inward investment (e.g. investment

promotion agency, etc.). ♦ I work for a multilateral / bilateral development organization. ♦ I work for a non-governmental organization. ♦ I work for a financial institution/intermediary. ♦ I work for an academic or research institution. ♦ Other ___________

X X X

4. What type of investment are you interested in? [multiple selection]: ♦ Greenfield (previously undeveloped) investment ♦ Investment in privatizing companies ♦ Investment in small or medium-sized enterprises ♦ Public-private partnerships (PPP). ♦ Mergers and acquisitions. • Joint ventures and strategic alliances.

X X

5. In what region are you interested? [multiple selection] ♦ East Asia / Pacific ♦ Eastern Europe / Central Asia ♦ Latin America / Caribbean ♦ Middle East / North Africa ♦ South Asia ♦ Sub-Saharan Africa

X X

6. What are the top 3 – 5 countries you are closely following? [Text] X X 7. Have you purchased political risk insurance from MIGA?

♦ Yes ♦ No

X

78

Survey Variant

Pop-

up

E-m

ail,

IPIs

E-m

ail,

Inve

stor

s

Work process and general internet use info 8. Which of the following sources of information do you consider important in assessing

potential investments? [Not important at all, Slightly important, Moderately important, Very important, Extremely important]

♦ Correspondence with country officials (by phone, email, or regular mail) ♦ Promotional presentations by country officials ♦ Meetings with country officials ♦ Visiting potential locations ♦ Consulting/Site selection experts ♦ A country investment promotion web site ♦ Accessing independent foreign investment web sites (e.g. WAIPA.org) ♦ Finding information via general web searches (e.g., using Google or Yahoo!) ♦ Hard-copy country investment promotion materials ♦ General research (e.g., economic profiles, independent research)

X X

9. Do you or someone you work with perform general web searches to research investment information (e.g. using Google, Yahoo!, and others)? [Y/N] X

10. If yes, to what extent do you agree or disagree with the following statements? [Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree]

♦ I can find information needed to assess potential investments quickly through general web searches.

♦ A general web search is a cost effective way to find the information I need.

X

11. What information topics available on MIGA’s online services (IPAnet, PrivatizationLink, FDIXchange) have you actively used? (check all that apply)

♦ Market information (e.g. FDI Research, Market Research) ♦ Investment Opportunities (e.g. Sector profiles, Privatization opportunities) ♦ Business Environment Information (e.g. Country or regional studies,

infrastructure, living standards, operational cost factors, etc.) ♦ Legal Information ♦ Events calendars ♦ World Bank “Doing Business” summaries ♦ World Bank “Investment Climate” summaries ♦ News ♦ Resource links

X

Investment Promotion Professional Profile Information 12. Have you received technical assistance from MIGA?

♦ Yes. ♦ No.

X

13. What type of investment are you targeting (check all that apply)? ♦ Greenfield (previously undeveloped) investment ♦ Investment in privatizing companies ♦ Investment in small or medium-sized enterprises ♦ Public-private partnerships (PPP) ♦ Mergers and acquisitions ♦ Joint ventures and strategic alliances ♦ Investment in low-income countries ♦ Investment in middle-income countries

X

79

Survey Variant

Pop-

up

E-m

ail,

IPIs

E-m

ail,

Inve

stor

s

Investment Promotion-related Usage of Services 14. How effective are the following activities in promoting investment in a country?

(extremely important, very important, moderately important, slightly important, not important at all)

♦ Correspondence with potential investors (by phone, email, or regular mail ♦ Promotional presentations to and outreach to investors ♦ Meetings with potential investors ♦ Hosting country visits ♦ Visits to investor countries ♦ Maintaining an investment promotion office abroad ♦ Building relationships with site selection consultants ♦ Country investment promotion web site ♦ Hard-copy country investment materials ♦ Posting specific investment opportunities on independent web sites

X

15. To what extent do you agree or disagree that the following are effective means to learn about promoting investment opportunities? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree)

♦ One-on-one training ♦ Traditional classroom training ♦ Attending trade conferences ♦ Attending training sessions provided by Government or Development Agencies ♦ Self-guided study using books and other hard-copy training materials and

toolkits ♦ Self-guided study using online training materials and toolkits ♦ Distance learning in a classroom style setting

X

16. To what extent do you agree or disagree with the following statements regarding MIGA’s FDI Promotion Center web site? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree, n/a)

♦ FDI Promotion Center provides me with reliable and accurate content. ♦ FDI Promotion Center provides me with timely and current content. ♦ FDI Promotion Center’s service is unique and not available anywhere else. ♦ I find learning tools on FDI Promotion Center that I cannot easily find anywhere

else. ♦ Without the FDI Promotion Center, I would have difficulty finding the

investment promotion tools I need. ♦ Using the FDI Promotion Center saves my organization time or me. ♦ Using FDI Promotion Center saves my organization money. ♦ FDI Promotion Center increased my knowledge about investment promotion. ♦ The use of FDI Promotion Center helped me improve my investment promotion

skills. ♦ I use the knowledge and skills I have acquired through the FDI Promotion Center

to increase qualified investor inquiries. ♦ I use the knowledge and skills I have acquired through the FDI Promotion Center

to increase my organization’s deal flow. ♦ My use of FDI Promotion Center ultimately resulted in my organization

attracting and securing an investment ♦ FDI Promotion Center is a valuable web site for my work.

X

80

Survey Variant

Pop-

up

E-m

ail,

IPIs

E-m

ail,

Inve

stor

s

17. From what areas of the FDI Promotion Center have you accessed and/or downloaded resources? (Check all that apply)

♦ Investor Research ♦ Best Practices and Case Studies ♦ Toolkits

X

18. To what extent do you agree or disagree with the following statements related to the Investment Promotion Toolkit feature of the FDI Promotion Center? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree, n/a)

♦ To refresh my knowledge of investment promotion techniques ♦ To learn best practices from other Investment Promotion Agencies ♦ To access the library of FDI-related research ♦ To get materials (e.g. checklists) I could use in my work ♦ To prepare for our internal trainings and activities ♦ To prepare my participation in a forthcoming conference and/or seminar

X

19. Did your organization use MIGA’s IPAworks to build an investment promotion web site? ♦ Yes ♦ No

X

IPAWorks Usage 20. If yes, to what extent do you agree or disagree with following statements regarding the

value of IPAworks? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree, N/A)

♦ IPAworks allowed my organization to obtain a reliable web site ♦ IPAworks allowed my organization to obtain a user-friendly web site ♦ My organization can maintain and update our IPAworks web site ♦ The IPAworks service is unique and not available anywhere else ♦ My organization would not have a web site if not for IPAworks ♦ Using IPAworks saved my organization money ♦ Using IPAworks saved our organization time ♦ Our use of IPAworks ultimately brought about new investment opportunities ♦ IPAworks is an important part of MIGA’s technical assistance package ♦ IPAworks proved to be a valuable tool for my work

X66

66 Only those answering “Yes” for Number 19

81

Survey Variant

Pop-

up

E-m

ail,

IPIs

E-m

ail,

Inve

stor

s

General MIGA Services Usage Information 21. To what extent do you agree or disagree with the following statement regarding MIGA’s

PrivatizationLink web site? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree, N/A)

♦ PrivatizationLink provides me with reliable and accurate content ♦ PrivatizationLink provides me with timely and current content ♦ I can always find the information I am looking for on PrivatizationLink ♦ PrivatizationLink is unique and not available anywhere else ♦ I find information through PrivatizationLink that I cannot easily find anywhere

else ♦ Without PrivatizationLink, I would have difficulty finding out about

privatization opportunities ♦ Using PrivatizationLink saves me time in my work ♦ Using PrivatizationLink saves my organization money ♦ I have reconsidered my opinion on a topic related to privatization after reviewing

information I found on PrivatizationLink ♦ I learned about specific privatizations through PrivatizationLink ♦ My use of PrivatizationLink led my organization to act on a specific privatization ♦ My use of PrivatizationLink resulted in my organization making a

bid/investment in a privatization project ♦ PrivatizationLink is a valuable web site for my work

X X

22. To what extent do you agree or disagree with the following statements regarding MIGA’s IPAnet web site? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree)

♦ IPAnet provides me with reliable and accurate content ♦ IPAnet provides me with timely and current content ♦ I can always find the information I am looking for on a specific topic on IPAnet ♦ IPAnet is unique and not available anywhere else ♦ I find information through IPAnet that I cannot easily find anywhere else ♦ Without IPAnet, I would have difficulty finding the FDI information I need ♦ Using IPAnet saves me time in my work ♦ Using IPAnet saves my organization money ♦ I have reconsidered my opinion on a particular investment related topic after

reviewing information I found on IPAnet ♦ I identified and investment opportunity by using IPAnet ♦ My use of IPAnet has resulted in an investment ♦ I regularly read IPAnet Briefing ♦ IPAnet Briefing is a useful tool to keep me informed about developments in

areas I am interested in ♦ IPAnet is a valuable web site for my work

X X

82

Survey Variant

Pop-

up

E-m

ail,

IPIs

E-m

ail,

Inve

stor

s

23. To what extent do you agree or disagree with the following statements regarding MIGA’s FDIXchange e-mail service? (strongly agree, agree, neither agree nor disagree, disagree, strongly disagree)

♦ FDIXchange provides reliable and accurate content ♦ FDIXchange provides timely and current content ♦ I regularly read the FDIXchange email alerts ♦ I can find the information I am looking for through the FDIXchange email

service ♦ The FDIXchange service is unique and not available anywhere else ♦ I receive information through FDIXchange that I cannot easily find anywhere

else ♦ Without FDIXchange, I would have difficulty finding the FDI information I need ♦ Using FDIXchange saves me time in my work ♦ Using FDIXchange saves my organization money ♦ Publishing my organization’s content on FDIXchange has resulted in increased

inquiries from potential investors ♦ I have reconsidered my opinion on an investment related topic after reviewing

information I found on FDIXchange ♦ I identified an investment opportunity on FDIXchange ♦ My use of FDIXchange led my organization to act on a specific investment

opportunity ♦ My use of FDIXchange led to an investment ♦ FDIXchange is a valuable service for my work

X X

83

Annex 13. Profiles and Findings for each Information Dissemination Service

Each of the Information Dissemination Services is profiled below. The summary includes select survey results related to outcomes and impact as well as insights from the comparator assessment, log file analysis, usability review, and prominence study.

IPAnet

Launched: 1995 Objectives: To fulfill MIGA’s mandate of dissemination and exchange of information on investment opportunities and conditions in member developing countries and transitional economies

Description: Provides information resources catalogued by country, topic and sector:

• Business environment • Sector analyses • Investment opportunities • Sources of investment finance • Laws and regulations • Investment promotion organizations

Target Audience67: • Investors • Advisors • Researchers

Key Usage Trends—All services:

0

100000

200000

300000

400000

500000

600000

700000

800000

900000

4 1 2 3 4 1 2 3 4

2003 2004 2005


Page

Req

uest

s

PrivatizationLinkFDIXchangeFDI CenterIPAnet

Outputs

67 MIGA, 2004 Annual Report (Washington, June 30, 2004), p. 20.

84

t

Individual content contributors are diverse (top content provider is the World Bank at 3.4%)

IPAnet Content Providers

IndustryFinancial Institutions &

Business Services

Governments

Research & Education

Business M edia

Business Associations

NGOs

International Organizations

Other OrganizationsIndustry

Financial Institutions & BusinessServicesGovernments

Research & Education

Business M edia

Business Associations

NGOs

International Organizations

Other Organizations

For all the services, 10% of all page requests came from developing countries. The sources of these 10% of requests by region are shown below.

AFR8%

EAP18%

ECA45%

LAC15%

MENA7%

SA7%

Outputs: Summary

IPAnet: Usability Audit Summary (see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design

Navigation & Structure

GoodFairPoor

IPAnet: Usability Audit Summary (see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

IPAnet: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic

Investment Metrics Search

Google Page Rank

GoodFairPoor

IPAnet: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic


Google Page Rank

GoodFairPoor

85

Overall, IPAnet’s outputs appear to be uneven. IPAnet’s usability and prominence are both rated fair which indicates improvements can be made. Governments are the predominant providers of content, which includes government departments and agencies which serve as investment promotion intermediaries. The log file analysis indicates that for all the services, 10% of users are from developing countries (note that a larger number of registrants are from developing countries which is not inconsistent since the log file indicates actual activity of the registrants); however since 2003, IPAnet has shown an increase in users from middle-lower and middle-upper income countries. IPAnet has the most number of users of all the services, but the activities of these registered users do not indicate a growth trend.

Outcomes

Survey Responses: IPAnet's Outcomes

0% 10% 20% 30% 40% 50% 60% 70%

I can alw ays find the information I am looking for on a specif ictopic on IPAnet

Using IPAnet saves my organization money

Using IPAnet saves me time in my w ork

IPAnet provides me w ith timely and current content

IPAnet provides me w ith reliable and accurate content

IPAnet Brief ing is a useful tool to keep me informed aboutdevelopments in areas I am interested in.

IPAnet is a valuable w eb site for my w ork

Strongly Agree Agree Neither agree nor disagree Disagree Strongly Disagree

n~617

n~617

Comparator Summary

Survey Responses: IPAnet's value compared to other information sources

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Without IPAnet, I w ould havedifficulty f inding the FDI

information I need

IPAnet is unique and notavailable anyw here else

I f ind information throughIPAnet that I cannot easily f ind

anyw here else


86

Services Target Users Access: Open = O Registration = R Paid = P

Comments:

IPAnet investors, Advisors, Researchers

O Content aggregator of a wide range of FDI information including investment opportunities

UNIDO Exchange Investors (clients of UNIDO country offices) IPAs Academics

R Registration request vetted by a consultant—not automatic

Individual IPA sites Investors, advisors / consultants O IPAnet aggregates this content

UNCTAD—Investment Compass

Government R Sees itself as complementary to MIGA

UNCTAD—Investment Map

Investors (trans-national corporations) IPAs

R Launched in 2004 in partnership with ITC, WAIPA and MIGA Provides analytical capabilities to IPAs

Development Gateway—FDI Page

Development Professionals “FDI Practitioners”

O Registration required to post materials

World Bank RRU—Investment Climate and Doing Business Surveys

Policymakers O Investment Climate=benchmarking data on host country investment climate Doing Business=benchmarking data on regulatory costs and indicators

FIAS Policymakers in developing country governments

O

Business Monitor Online

Multi-national corporations IPAs Host country governments

P “Few IPAs can afford us” Does not present investment opportunities

Outcomes: Summary Based on survey responses, overall, IPAnet appears to be achieving its desired outcomes. 73% of approximately 617 respondents strongly agreed or agreed that IPAnet provides timely and current content and 80% strongly agreed or agreed that IPAnet provides reliable and accurate content. 68% of respondents strongly agreed or agreed that IPAnet saves them time, and 48% strongly agreed or agreed that IPAnet saves their organization money. With respect to the unique value provided by the service, results were less definitive with 50% of respondents strongly agreeing or agreeing that IPAnet is unique, however, 39% neither agreed nor disagreed with this statement. Approximately 12% of respondents disagreed with the counterfactual statement that without IPAnet a respondent could not find the information needed. 45% either agreed or strongly agreed with this statement. A summary view of IPAnet’s closest comparators shows that it is the only “open” service (no registration is required to access the site) which aggregates a range of content aimed at investors.

87

Impact

Survey Responses: IPAnet's Impact

0% 10% 20% 30% 40% 50% 60%

My use of IPAnet led my organization to act on a specificinvestment opportunity

My use of IPAnet has resulted in an investment

I have reconsidered my opinion on a particular investmentrelated topic after review ing information I found on IPAnet

I identif ied an investment opportunity by using IPAnet


Impact: Summary Generally, survey respondents indicated impacts had been achieved but were more likely to neither agree nor disagree with statements about the impact of IPAnet. Of the approximately 617 respondents, 40% strongly agreed or agreed that they had reconsidered their opinion about an investment as a result of using IPAnet. 46% neither agreed nor disagreed with this statement. Similarly, 40% strongly agreed or agreed that they had identified an investment opportunity through using IPAnet, however 42% neither agreed nor disagreed with this statement. 30% of respondents strongly agreed or agreed that their organization acted on an investment opportunity as a result of using IPAnet. 20% disagreed or strongly disagreed with this statement. Similarly, 23% strongly agreed or agreed that their use of IPAnet has resulted in an investment and 28% disagreed or strongly disagreed with this statement. While it is not surprising that 46% of respondents on average neither agree nor disagree with statements which indicate impact as a result of using IPAnet, it is worth noting that a respectable number do attribute some change in knowledge, attitudes and behaviors. 77% of respondents strongly agreed or agreed that IPAnet is a valuable site for their work.

88

FDIXchange

Launched: 2002 Note: FDIXchange uses the IPAnet database. Objectives: To link international investors to current investment opportunities and business environment information in developing countries through personalized e-mail updates and a web-based archive68

Description: • Provides a customized email alert service on new

investment information and related information on country, and sources of financial risk management services

• Information provided by the WBG, IPAs and government agencies, private sector business information providers, privatization agencies, chambers and business associations69

Target Audience: • Investors • Advisors • Financial institutions

See IPAnet summary for usage trends for all services.

Outputs

See IPAnet summary for content providers and page requests

Outputs: Summary Similar to IPAnet, FDIXchange’s outputs appear to be uneven. FDIXchange does not show a growth trend in traffic for the period analyzed. This is partly due to the nature of the service, where users sign up for a push email and are not expected to do other activities on the site. Overall, FDIXchange’s usability is good, and better than IPAnet’s. The reverse is true for prominence. FDIXchange does not have strong prominence and its prominence is weaker than IPAnet’s. 75% of respondents either strongly agreed or agreed with the statement that they regularly read the FDIXchange email that the service pushes to them.

68 MIGA, FDIXchange: A customized information service facilitating Foreign Direct Investment in emerging markets, Brochure (no date). 69 MIGA, FDIXchange: A customized information service facilitating Foreign Direct Investment in emerging markets, Brochure (no date).

FDIXchange: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

FDIXchange: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

FDIXchange: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic


Google Page Rank

GoodFairPoor

FDIXchange: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic


Google Page Rank

GoodFairPoor

89

Outcomes

n~637

Survey Responses: FDIXchange's Outcomes

0% 10% 20% 30% 40% 50% 60%

Publishing my organization’s content on FDIXchange hasresulted in increased inquiries from potential investors

I can find the information I am looking for through theFDIXchange email service.

Using FDIXchange saves my organization money

Using FDIXchange saves me time in my w ork

FDIXchange is a valuable service for my w ork

FDIXchange provides timely and current content

I regularly read the FDIXchange email alerts

FDIXchange provides reliable and accurate content


Survey Responses: FDIXchange's value compared to other information services

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Without FDIXchange, I w ouldhave dif f iculty f inding the FDI

information I need

The FDIXchange service isunique and not available

anyw here else

I receive information throughFDIXchange that I cannot easily

find anyw here else

Strongly Agree Agree Neither agree nor disagree Disagree Strongly Disagreen~637

Comparator Summary


Comments:

FDIXchange See IPAnet Summary R See IPAnet Summary

Outcomes: Summary Based on survey responses, overall, FDIXchange appears to be achieving its desired outcomes. Similar to the results for IPAnet, of the 637 respondents, 75% strongly agreed or agreed that FDIXchange provides timely and current content and 89% strongly agreed or agreed that FDIXchange provides reliable and accurate

90

content. 64% of respondents strongly agreed or agreed that FDIXchange saves them time and 49% strongly agreed or agreed that FDIXchange saves their organization money Again, like IPAnet, responses to statements about the unique value of the service were less definitive. 54% of respondents strongly agreed or agreed that FDIXchange is unique, however, 36% neither agreed nor disagreed with this statement. 47% of respondents agreed with the counterfactual statement that they could not find the information they needed without FDIXchange and 14% disagreed or strongly disagreed with this counterfactual statement. A summary view of FDIXchange’s closest comparators shows that other sites requiring registration which serve investors are not aggregation services and most provide some sort of analytical capability for users.

Impact

Survey Responses: Impact of FDIXchange

0% 10% 20% 30% 40% 50% 60%

My use of FDIXchange led to an investment

I have reconsidered my opinion on an investment related topicafter review ing information I found on FDIXchange

My use of FDIXchange led my organization to act on a specif icinvestment opportunity

I identif ied an investment opportunity on FDIXchange


Impact: Summary Generally, FDIXchange’s impact results are very similar to those of IPAnet. For the most part, respondents did indicate that there was impact as a result of using the services but were more likely to indicate that they neither agreed nor disagreed with statements about the impact of FDIXchange. Of the approximately 637 respondents, 40% strongly agreed or agreed that they had reconsidered their opinion about an investment as a result of using FDIXchange. 46% neither agreed nor disagreed with this statement. Similarly, 39% strongly agreed or agreed that they had identified an investment opportunity through using FDIXchange, however 42% neither agreed nor disagreed with this statement. 33% of respondents strongly agreed or agreed that their organization acted on an investment opportunity as a result of using FDIXchange. 22% disagreed or strongly disagreed with this statement. Similarly, 26% strongly agreed or agreed that their use of FDIXchange has resulted in an investment and 24% disagreed or strongly disagreed with this statement. 38% of respondents strongly agreed or agreed that investor inquiries had increased as a result of publishing their organization’s content through FDIXchange. 53% neither agreed nor disagreed with this statement. Similar to IPAnet, 46% of respondents on average neither agree nor disagree with statements which indicate impact as a result of using FDIXchange. 72% of respondents strongly agreed or agreed that FDIXchange is a valuable service for their work.

PrivatizationLink

Launched: 1998 Note: PrivatizationLink uses the IPAnet database. Objectives:

91

Reduce transaction costs associated with privatization programs by providing investors with key, timely information on state owned enterprises (SOEs) being divested throughout the developing world and by providing privatization agencies with a free channel to disseminate these opportunities

Description: • Provides privatization information on developing

countries • Provides profiles of enterprises being divested by

privatization agencies 70

Target Audience: • Investors • Advisors • Financial institutions


Outputs

See IPAnet summary for content providers and page requests

Outputs: Summary Like the other services using the IPAnet database, the outputs of PrivatizationLink are uneven. PrivatizationLink shows a declining trend in page requests for the period analyzed Overall, PrivatizationLink’s usability is rated weak and ranks third after FDIXchange and IPAnet respectively. PrivatizationLink has good prominence, showing good results for both Google Page Rank and Traffic. Like the other investor services, PrivatizationLink shows an increase in use among middle-lower and middle-upper income countries.

Outcomes

70 MIGA, FDIXchange: A customized information service facilitating Foreign Direct Investment in emerging markets, Brochure (no date).

PrivatizationLink: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

PrivatizationLink: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

PrivatizationLink: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic


Google Page Rank

GoodFairPoor

PrivatizationLink: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic


Google Page Rank

GoodFairPoor

92

n~523

Survey Responses: Privatization Link's Outcomes

0% 10% 20% 30% 40% 50% 60% 70%

I can alw ays f ind the information I am looking for onPrivatizationLink

PrivatizationLink provides me w ith timely and current content

Using PrivatizationLink saves my organization money

Using PrivatizationLink saves me time in my w ork

PrivatizationLink provides me w ith reliable and accurate content

PrivatizationLink is a valuable w eb site for my w ork


Survey Responses: PrivatizationLink's value compared to other information sources

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Without PrivatizationLink, Iw ould have diff iculty finding

out about privatizationopportunities

PrivatizationLink is unique andnot available anyw here else

I find information throughPrivatizationLink that I cannot

easily f ind anyw here else


Comparator Summary


Comments:

PrivatizationLink Investors (interested in privatization opportunities) Advisors Financial institutions

O Privatization opportunities

Individual privatization agency sites

Investors, advisors / consultants O PrivatizationLink aggregates this content

World Bank—RRU Policymakers O Specifically focus on “practical advice and

93

Privatization Toolkits Advisors tools” for private sector participation in infrastructure services for low and middle income countries

Outcomes: Summary Based on survey responses, overall, PrivatizationLink appears to be achieving its desired outcomes. Similar to the results for IPAnet and FDIXchange , a significant number of respondents agreed that PrivatizationLink provides reliable and accurate content (71%). 62% strongly agree or agree that this content is timely and current. 62% of respondents strongly agreed or agreed that PrivatizationLink saves their organization time and 45% strongly agreed or agreed that the service saves their organization money. Fewer respondents (31%) than for IPAnet or FDIXchange strongly agreed or agreed that they could find what they were looking for on PrivatizationLink. 45% neither agreed nor disagreed with this statement. As with IPAnet and FDIXchange, respondents were not definitive about the service’s unique value. A significant number of respondents (46%) neither agreed nor disagreed with statements about the service’s uniqueness. 17% of respondents disagreed or strongly disagreed with the counterfactual statement that they could not find the information they needed without PrivatizationLink. 40% either agreed or strongly agreed with this counterfactual statement. While respondents did not provide a strong indication that they believe the service to be unique, the comparator analysis did not identify a similar service. For example, the World Bank RRU’s Privatization toolkits target a different audience (Policymakers and Advisors) and are geared more at practical advice and tools rather than specific investment opportunity information.

Impact

Survey Responses: PrivatizationLink's Impact

0% 10% 20% 30% 40% 50% 60%

My use of PrivatizationLink resulted in my organization making abid/investment in a privatization project

I have reconsidered my opinion on a topic related toprivatization after review ing information I found on

PrivatizationLink

My use of PrivatizationLink led my organization to act on aspecif ic privatization

I learned about specif ic privatizations through PrivatizationLink


Impact: Summary Similar to IPAnet and FDIXchange, respondents indicated that some impact was achieved but were not definitive about statements of PrivatizationLink’s impact. Respondents indicate that they learn about specific opportunities through PrivatizationLink (34% strongly agree or agree with this statement), but are much less likely to act on it (25% strongly agreed or agreed with this statement and 56% neither agreed nor disagreed). Similarly 51% neither agree nor disagree that they have reconsidered their opinion as a result of information found on PrivatizationLink. 23% of respondents strongly agreed or disagreed that their use of the service had resulted in a bid or an investment; far more, 56%, neither agreed nor disagreed with this statement. 69% of respondents indicated that PrivatizationLink was valuable for their work. Overall, the survey results indicate that PrivatizationLink has less impact than either IPAnet or FDIXchange.

94


Launched: 2004 Objectives: Supports the efforts of investment promotion professionals to attract and retain FDI71 Serve the growing community of investment promotion professionals, particularly those in developing countries by providing services to help practitioners update their knowledge and skills covering the complex and often lengthy process of securing inward investment72

Description: • Provides a learning toolkit on how to attract and

service investors • Provides promotional tools including case

studies and best practice research • Provides investor research resources

Target Audience: • Investment Promotion Professionals


Outputs

Content Partners include: • WAIPA

• Best practices • Other organizations that manage the service in a language other than English

• Inter-Arab Investment Guarantee Corporation (Arabic) • National Agency for Direct Promotion (Russian) • Fond Za Podrsku Investicija Uvojvodini (Serbian)

See IPAnet summary for page requests.

Outputs: Summary FDI Promotion Center is performing solidly in terms of outputs. The FDI Promotion Center is the only service which shows a growth trend in traffic and user activity for the period

71 MIGA, Foreign Direct Investment Promotion Center: Knowledge and learning resources for investment promotion practitioners, Brochure (no date). 72 MIGA, Spotlight: The Foreign Direct Investment Promotion Center, (Q305).

FDI Promotion Center: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

FDI Promotion Center: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

FDI Promotion Center: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic

N/AN/AN/AInvestment Metrics Search

Google Page Rank

GoodFairPoor

FDI Promotion Center: Prominence Analysis Summary(see Appendix E for full analysis)

Links In

Traffic

N/AN/AN/AInvestment Metrics Search

Google Page Rank

GoodFairPoor

95

analyzed. Overall, the FDI Promotion Center shows good usability and prominence. The service is using partners to support several languages. The FDI Promotion Center has shown a significant increase in 2005 in users from middle-lower income countries.

Outcomes

Survey Responses: FDI Promotion Center's Outcomes

0% 10% 20% 30% 40% 50% 60% 70%

Using FDI Promotion Center saves my organization money

Using the FDI Promotion Center saves me or my organizationtime

The use of the FDI Promotion Center helped me improve myinvestment promotion skills.

FDI Promotion Center provides me w ith timely and currentcontent

FDI Promotion Center is a valuable w eb site for my w ork

FDI Promotion Center provides me w ith reliable and accuratecontent


Survey Responses: FDI Promotion Center's value compared to other sources

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Without the FDI PromotionCenter, I w ould have diff iculty

f inding the investmentpromotion tools that I need

FDI Promotion Center’s serviceis unique and not available

anyw here else

I f ind learning tools on FDIPromotion Center that I cannot

easily f ind anyw here else


Comparator Summary


Comments:

FDI Promotion Investment Promotion R Toolkits, best practices and other learning

96

Center Professionals materials for IP professionals

World Bank—RRU Privatization Toolkits

Policymakers Advisors

O Specifically focus on private sector participation in infrastructure services for low and middle income countries

Outcomes: Summary Overall, respondents indicate that the FDI Promotion Center is achieving its intended outcomes, although respondents were less clear about the uniqueness of the service. Of 553 respondents, 76% strongly agreed or agreed that the FDI Promotion Center provides them with timely, current content and 89% strongly agreed or agreed that the service provides accurate and reliable content. 66% of respondents strongly agreed or agreed that the service saves them time, however, the indication that the service saves their organization money was not as strong. 47% strongly agreed or agreed with this statement and 42% neither agreed nor disagreed. 45% of respondents strongly agreed or agreed that the service is unique and 40% neither agreed nor disagreed with this statement. This is surprising since the comparator analysis indicates that this service is a unique niche for the IDS. Similarly, approximately the same number of respondents (40%) strongly agreed or agreed and 40% neither agreed nor disagreed with the counterfactual statement that without the FDI Promotion Center respondents could not find the tools they need. 72% of respondents strongly agreed or agreed that they had improved their investment promotion skills through using the FDI Promotion Center service and 81% strongly agreed or agreed that they had increased their knowledge of investment promotion as a result of using the service. Respondents indicated that they most value learning about best practices and using materials from the site in their work.

Impact

Survey Responses: FDI Promotion Center's Impact

0% 10% 20% 30% 40% 50% 60%

My use of FDI Promotion Centerultimately resulted in my

organization attracting andsecuring an investment

I use the know ledge and skills Ihave acquired through the FDIPromotion Centre to increasemy organization’s deal f low

I use the know ledge and skills Ihave acquired through the FDIPromotion Center to increasequalif ied investor inquiries


Impact: Summary Given that the FDI Promotion Center is intended to increase capacity and capability among IPIs, and is therefore by nature at one end of the investment promotion continuum--and more removed in this sense than the other services from investment activities--respondents indicate the service does achieve impact. 61% of respondents indicated that they strongly agree or agree that they have increased qualified investor inquiries as a result of using the service. 33% neither agree nor disagree with this statement. 49% strongly agree or agree that they use the skills they acquired through the FDI Promotion Center to increase their organization’s deal flow, and 42% neither agree nor disagree with this statement. Not surprisingly fewer respondents (31%) strongly agree or agree that their use of the service as resulted in attracting an investment. 14% disagree or strongly disagree with this statement. An impressive 83% of respondents strongly agreed or agreed that the FDI Promotion Center is a valuable tool for their work.

97

IPAworks A service now offered through the FDI Promotion Center

Objectives: Enable developing country IPIs to establish a professional Internet presence quickly and inexpensively

Description: • Provide a website structure and instructions and

coaching to implement it

Target Audience: • Investment Promotion Professionals

Outputs

Outputs: Summary Overall, the IPAworks’ usability is rated fair.

Outcomes

Survey Responses: IPAworks' Outcomes

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Using IPAw orks saved my organization money

Using IPAw orks saved our organization time

IPAw orks allow ed my organization to obtain a reliable w eb site

IPAw orks allow ed my organization to obtain a user-friendlyw eb site

My organization can maintain and update our IPAw orks w ebsite

IPAw orks proved to be a valuable tool for my w ork


IPAworks: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

IPAworks: Usability Audit Summary(see Appendix H for full analysis)

Credibility

Links

Writing for Web

Interaction Design


GoodFairPoor

98

Survey Responses: IPAworks' vlaue compared to other tools or resources

0% 10% 20% 30% 40% 50% 60%

My organization w ould nothave a w eb site if not for

IPAw orks

The IPAw orks service isunique and not available

anyw here else


Comparator Summary


Comments:

IPAworks Investment Promotion Professionals

R Users need to contact IDS to access and set up the tool

UNCTAD—Investment Gateway

IPAs Investors Policymakers

R Platform includes “readily available information on investment opportunities and practical information on the legal and regulatory framework IPAs must develop a joint strategy for external funding Cooperates with WAIPA

Outcomes: Summary The number of respondents for IPAworks questions was extremely low at 32, which is likely due to the fact that most respondents had not used the service. Those who responded indicate that IPAworks appears to achieve its intended outcomes. 66% of these respondents indicated that using the service had saved them time and 59% indicated that it had saved them money. On average 78% of respondents strongly agreed or agreed that they had obtained a reliable, user-friendly website by using the service. However, a significant number of respondents, 59% either disagree or strongly disagree that they would not have a website if it were not for IPAworks. So while the service appears to be effective, it does not seem to have a unique value.

Impact

99

Effectiveness of Activities in Promoting Investment

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Building relationships w ith site selection consultants

Posting specif ic investment opportunities on independent w ebsites

Maintaining an investment promotion off ice abroad

Visits to investor countries

Hard-copy country investment materials

Hosting country visits

Promotional presentations to and outreach to investors

Country investment promotion w eb site

Correspondence w ith potential investors (by phone, email, orregular mail)

Meetings w ith potential investors

% Responding

Extremely important Very important Moderately important Slightly important Not important at all

n~205

Survey Reponse: IPAworks' Impact

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

Our use of IPAw orks ultimatelybrought about new investment

opportunities


100

Impact: Summary 42% of the 32 respondents strongly agreed or agreed that IPAworks brought about new investment opportunities while 15% of respondents disagreed with this statement. The interviews with IPAs supported this—an investment promotion website is essential to establish credibility and attract investor inquiries. In a more general question about effective activities to promote investment, over 80% strongly agreed or agreed that a country investment website was important.

101

Annex 14. Survey Results on IDS Impact Impacts: Summary of survey responses SA=Strongly Agree

A= Agree

D=Disagree

SD=Strongly Disagree

IPAnet n=617

FDIXchange n=637

PrivatizationLink n=523

I identified an opportunity by using the service

40% = SA / A 42% = neither A nor D 17% = D / SD



My use of the service led my organization to act on a specific investment opportunity

30% = SA / A 49% = neither A nor D 20%= D / SD



I have reconsidered my opinion on an investment related topic after reviewing information I found through the service




My use of the service led to an investment

23%=SA / A 49% = neither A nor D 28% = D / SD



SA=Strongly Agree

A= Agree

D=Disagree

SD=Strongly Disagree

FDI Promotion Center n=553

I use the knowledge and skills acquired through the FDI Promotion Center to increase qualified investor inquiries


I use the knowledge and skills acquired through the FDI Promotion Center to increase my organization’s deal flow


My use of the FDI Promotion Center ultimately resulted in my organization attracting and securing an investment


independent evaluation group (miga) 2006 annual report€¦ · independence, which are also...

Documents