approach paper technical cooperation experience in new ... · evaluation approach paper: technical...

ab0cd

APPROACH PAPER

Technical cooperation experience in new countries of operations

May 2013

EBRD EVALUATION DEPARTMENT

The Evaluation department (EvD) produces approach papers to guide the conduct of an evaluation and to inform stakeholders of the approach proposed. Although comments are welcome, EvD is the sole decision-maker about incorporation of comments. An approach paper is the first output and milestone of an evaluation. Approach papers will generally approved by the Chief Evaluator 2–4 weeks after a study has started, and be based on a preliminary document review and initial internal consultations. Complex studies may require more time before clarity emerges on the objectives, evaluation questions, conceptual basis for the study and methods to be employed.

Evaluation Approach paper

Technical cooperation experience in new countries of operations

May 2013

EBRD Evaluation department

May 2013

Contents

1 Introduction ............................................................................... 1

1.1 Purpose and scope 1

1.2 Expected results 1

1.3 Other relevant evaluation work 2

2 Monitoring and self-assessment ............................................... 4

2.1 Governance and administration 4

2.2 Project Completion Reports 4

2.3 Transition and other impact monitoring 5

3 Methodology .............................................................................. 6

3.1 Conceptual or results framework 6

3.2 Sample selection 8

3.3 Evaluation questions 9

3.4 Indicators, data sources and analysis 9

3.5 Methods 9

3.6 Potential issues and limitations 10

4 Administrative arrangements .................................................. 11

4.1 Team 11

4.2 Peer reviewers/advisory panel 11

4.3 Timetable 11

Evaluation department Chiara Bocci, Evaluation Manager -TC Tel: +44 20 7338 6306 Email: [email protected]

Evaluation Approach paper

Technical cooperation experience in new countries of operations May 2013

EBRD Evaluation department May 2013

1 Introduction 1.1 Purpose and scope

This special study ‘Technical cooperation experience in new countries of operations’, part of the EvD Work Programme for 2013, sets out to examine the early use of technical cooperation funds in recent countries of operation, Turkey and Mongolia, to identify issues relevant to the EBRD’s first engagements in the Southern and Eastern Mediterranean (SEMed) countries. A desk review in 2012 revealed that out of the two cases identified at an early stage, only the Mongolian experience was a potential source of useful lessons for the future. Turkey became a member and a country of operation at the end of 2008 and was therefore immediately eligible for both investment and technical cooperation operations. For the Mongolian case a multi-donor fund was set up at the early stages thus having more similarities with the strategy for SEMed countries. The multi-donor “Mongolian Cooperation Fund (MCF)” (including donors: Japan, Luxembourg, the Netherlands, and Taipei China) was set up by the EBRD in 2001 to foster Mongolia’s transition to a market economy, promote private and entrepreneurial initiative in the country and implement structural and sectoral economic reforms. Mongolia became a country of operation, and thus eligible for EBRD investments, in 2006, but for the first five years of operation the Bank’s engagement was restricted to the MCF. Fund commitments, mainly approved before end- 2008 but also recently in July 2012, total €9.2 million. A final evaluation of the MCF will provide an overview of the implementation of a multi-donor fund in a country which passed from the status of being a non-recipient country to an EBRD country of operations. The related findings and insights might help the EBRD to manage its own funds, and multi-donor funds entrusted to it, in the new countries of operations (SEMed).

1.2 Expected results

The evaluation team designed an ex-post results framework for the MCF which is due to evolve during the on-going desk research and following the evaluation mission. At present, the evaluation team has identified two main results that the MCF was aimed at, namely: a “corporate”

Evaluation Approach paper: Technical cooperation experience in new countries of operations 2 / 11


May 2013

result to render the EBRD operational in Mongolia and familiar with local context and institutions; and a “transition” result to identify Mongolian transition challenges and start to tackle them. As a consequence, this special study is expected to:

− Assess the relevance, effectiveness, efficiency, impact and sustainability of the EBRD’s technical assistance funded by the MCF.

− Assess the viability of the use of multi-donor technical assistance fund as the EBRD’s entry point in a new country (not yet as operation as Mongolia).

− Assess the strategic relevance of the MCF in light of the EBRD’s operational objectives and structure.

− Assess MCF complementarity with other IFIs and donors active in Mongolia.

− Identify findings of relevance for the EBRD’s first engagements in the southern and eastern Mediterranean (SEMed) countries.

1.3 Other relevant evaluation work

In 2005 EvD produced a ‘Mid-Term Review of the Mongolian Cooperation Fund’ based on findings from the evaluation of the TC committed up to end-2004. Overall, the TC operations funded by the MCF have been considered successful and several recommendations have been crafted. Management did not submit comments on the Mid-Term Review. The Audit Committee discussed the review at its meeting in November 2006 and welcomed the special study. An interesting debate took place on incentives for bankers and the importance of donor and International Financial Insitution (IFI) coordination mechanisms in the EBRD’s countries of operations. The present study will take advantage of the Mid-Term Review recommendations and the topics discussed at the Audit Committee meeting and verify the extent of any positive or negative effects. The Special Study ‘Legal Transition Programme (LTP) Review’ includes Mongolia in the sample of countries selected to assess the LTP’s field performance. Some of the LTP’s activities have been funded by the MCF, namely: judicial capacity assessment; corporate governance standards; infrastructure regulatory reforms (telecommunications and energy); and, secured transactions. This study will take advantage of the assessment

3 / 11 Evaluation Approach paper: Technical cooperation experience in new countries of operations


already done by the LTP review and related findings and recommendations. Other work in the EvD 2012 work programme of relevance to the new study includes: (i) the Operation Evaluation of the ‘Mongolian Financial Sector Framework’, which has benefited from a reasonable amount of TC funded through the MCF aimed at improving the financial sector; and (ii) two Validations of the Operation Performance Assessments that have benefited from technical assistance under the MCF framework. Other complementarities have been identified with the Special Study: ‘Early Transition Country Initiative’ and the on-going Impact Evaluation: ‘Professional Training for Judges in Tajikistan’.



May 2013

2 Monitoring and self-assessment Given the results identified for the MCF by the evaluation team, it will be relevant to look at both the governance and administration of the fund and the technical assistance assignments that have been committed and implemented.

2.1 Governance and administration

The EBRD administered the MCF through its Central Asia Team which delegated the responsibilities to one of its Senior Bankers, who spent about 10 per cent of his time as MCF Administrator. The MCF Administrator exercised general oversight over implementation and worked with the MCF’s Ulaanbaatar office and later on the EBRD’s resident office in Mongolia, to handle liaison with government agencies, other donors, and the parties involved in the TC projects. The MCF benefited from dedicated staff (office manager and analyst) based in Ulaanbaatar. A Steering Committee, consisting of representatives of MCF’s donors, met twice a year (Oct 2001 – Nov 2006) to review the progress of the activities, approve operational priorities and seek approval for individual TC assignments. The process for MCF TC operations paralleled the normal TC process as prescribed in the Bank’s Operations Manual. Once the Steering Committee agreed to a concept, the proposal was submitted to the Technical Cooperation Committee for approval and then finally to the MCF for confirmation that none of the donors objected to the operation. Once Mongolia became an EBRD country of operations in late 2006, the Steering Committee ceased to convene and decided that that the disbursements of the MCF should have been completed by the end of 2008.

2.2 Project Completion Reports

At the Steering Committee meetings, reporting focused on the progress of the assignments in the implementation phase rather than on results of the MCF itself. Internal self-assessment was carried out on individual TC commitments through the Consultant Assignment Reporting (CAR) system. Out of 63 TCs committed, Project Completion Reports (PCRs) have been completed for 36 of them (technical assistance stand-alone and framework) and made available to the evaluation team. Seventeen PCRs have been



completed also for assignments related to administration and human resources related to the MCF. The remaining TCs have been exempted from completing PCRs, which is usual practice for assignments under the framework of the Small Business Support (SBS), namely the Enterprise Growth Programme (EGP, former Turn Around Management TAM programme) and the Business Advisory Services (BAS) Programme. Figure 1 TC projects committed under MCF

2.3 Transition and other impact monitoring

In the MCF Board document there is no indication of the transition impact of the fund itself. As mentioned above, the evaluation team will develop a results framework ex-post, incorporating transition impact elements with input from the Office of the Chief Economist (OCE). Although it is not possible yet to link the results of TC assignments with the EBRD’s Transition Impact Monitoring System (TIMS), the study will ideally find consistencies and a relationship with the transition indicators and challenges were identified once Mongolia became a country of operation (and included in the first Strategy for Mongolia and the Board documents approving the EBRD’s operations in Mongolia).

TA PCRs Ratings Stand alone & Framework

Highly successful 13Successful 15Partly successful 6Unsuccessful 1Not rated yet 1

SUBTOTAL TA PCRs 36N/A (BAS Framework) 1N/A (TAM Framework) 8Administration/HR 17Evaluation 1

TOTAL 63



May 2013

3 Methodology 3.1 Conceptual or results framework

The MCF was conceived for the Bank to facilitate the country’s transition to a market economy and increase its exchanges with the Bank’s countries of operations and other countries. A results framework was not designed up front. At approval, in the Board document, it was mentioned that:

“the Bank’s activities in Mongolia shall be of a technical assistance nature and shall be financed entirely out of grant funds, such grants being used to assist in implementing economic and legal reforms (…), and in supporting the establishment, improvement and expansion of private sector activity (in particular small and medium sized enterprises)”.

An indicative illustration of the utilisation of the MCF was attached, including: legal transition projects; business advice; financial sector support; mining sector; sectoral reforms (energy, telecommunications); and environmental support. Each TC assignment submitted to TC Com and approved for MCF funding was to include specific success indicators. From a preliminary assessment of the available documentation, the evaluation team has designed the following results tree which is subject to changes following the in-depth assessment that will take place during the desk work and the field mission in Mongolia.



Figure 2 Proposed results tree

ACTIV

ITIES

OBJE

CTIVE

OUTC

OMES

OUTP

UTS

To facilitate Mongolia's transition into a market economy

1. Corporate: EBRD operational in Mongolia and familiar with local context and institutions

2. Transition: To identify Mongolian transition challenges and start to tackle them

1.1 Established MCF (EBRD) representation in Mongolia

1.2 Identified medium-term policy and strategic goals for Mongolian capital markets

1.3 Launched and operational TAM and BAS Programmes

1.1.1 Office rent and dedicated staff for the MCF (manager, analyst)

1.2.1 Capital Markets Review and Strategy Recommendations

1.3.1 Feasibility studies for TAM and BAS Programmes

2.1 Improved business environment and corporate governance

2.2 Improved financial sector

2.3 Developed competitive and transparent framework in the natural resources sector

2.4 Promoted commercialisation and private sector participation in public infrastructure

2.1.1 TAM Programme (27 companies) 2.1.2 SME Management Training2.1.3 Assessing (and Improving) Corporate Governance Legal Framework and Secured Transactions2.1.4 Medium sized enterprise projects facility due diligences2.1.5 Sustainable commercial law judicial curriculum

2.2.1 Micro-credit advisors for Khan Bank2.2.2 Trainings for Mongolian bankers2.2.3 Trainings for Bank of Mongolia2.2.4 Trade Facilitation Programme2.2.5 TA to improve secured transaction

2.3.1 Petroleum industry2.3.2 Capacity building to Mining Authority2.3.4 DD MAK Mining

2.4.1 Civil aviation (TA to MCAA) and MIAT2.4.2 Power sector2.4.3 Telecommunications (TA to GoM)2.4.4 Ulaanbaatar urban transport -due diligences



May 2013

3.2 Sample selection

Sixty-three TCs were committed between 2001 and 2012, for a value of approximately €9.2 million. Before Mongolia’s ratification as a country of operation in October 2006, the EBRD’s activities there were mainly restricted to TC. Since the MCF was phased out, the Early Transition Country Multi-Donor Fund and bilateral donors have taken the lead in financing TC activities in Mongolia. Since late 2006, MCF commitments have been focused on specific project types rather than covering all sectors of interest as was previously the case. Figure 3 MCF TC assignments per sector

Out of the total TCs committed, thirty-five have been already subject of the Mid-Term Review carried out in 2005 by the Evaluation Department. The present study will therefore focus for the case studies on the population of TCs committed after 2005 (a total of twenty-eight) and assess the impact of those already subject of the Mid-Term Review. Other evaluation reports, and in particular the Special Study on Legal Transition Programme Review, have already assessed TC assignments funded by the MCF. This study will take advantage of the findings available in those reports and focus its attention on other areas of interest.

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

2001

2002

2003

2004

2005

2006

2007

2008

2012

MCF: TC assignments per sector (2001-2012)

Transport, StorageTelecommunicationsManufacturingFinance, BusinessExtractive IndustriesEnergyCommunity/Social ServicesCommerce, Tourism

Sector Type

Year

EUR



The five DAC evaluation criteria (relevance, effectiveness, efficiency, impact and sustainability) will be used to assess the TC assignments included into the sample.

3.3 Evaluation questions

Evaluation questions will cover among others:

− What was the strategic relevance of the MCF?

− How much does the corporate outcome and transition outcome count?

− How has the MCF contributed to identifying and tackling the transition challenges of the country?

− In which areas is the EBRD focusing on more when approaching work a new country?

− How was the MCF utilised for the then Country Strategy for Mongolia 2006?

− To what extent are the EBRD’s interventions in Mongolia complementary with activities of other IFIs and bilateral donors?

Sub-questions will be derived from the results framework that will be developed by the evaluation team.

3.4 Indicators, data sources and analysis

Indicators and sources of verification will be included into a results framework mainly from the EBRD’s internal documents, such as: Board documents, General Conditions of the MCF, Minutes of the MCF Steering Committees, Project Completion Reports, Consultant’s reports, OCE assessments of transition indicators for Mongolia. Data will be also gathered from other IFIs, such as the IFC, actively working in Mongolia.

3.5 Methods

The methods to be used by the evaluation are expected to be:

− A background document review

− A literature review

− Interviews with Banking and non-Banking teams involved in the MCF

− Interviews with representatives of the donors that contributed



May 2013

to MCF (Japan, Luxembourg, the Netherlands, and Taipei China)

− A field mission in Mongolia to interview clients, beneficiaries, resident office staff, other IFI, other bilateral donors, and all relevant stakeholders

− Case studies will be developed for selected TCs

3.6 Potential issues and limitations

− Limited availability of data and documentation: Internal availability of documentation about the MCF could be very poor. It is already envisaged that the evaluation team will have difficulties gathering the documents related to the early commitments of the MCF from Banking and non-Banking teams. Availability of EBRD staff involved in the MCF could be limited as well. The same can be said in terms of institutional memory for the beneficiaries of the technical assistance provided through the MCF. During the mission in Mongolia the evaluation team will interview beneficiaries identified for the case studies and other stakeholders. It is expected that most of the people involved in the TC assignments will have left.

− The sectors addressed by the MCF are very diverse. The evaluation team could encounter problems in applying the same methodological tools to different sectors at different stages of development.



4 Administrative arrangements 4.1 Team

The evaluation team is composed of Chiara Bocci (Evaluation Manager – Technical Cooperation) and Amelie Eulenburg (Senior Economist). Stephanie Crossley (Administrative Officer) will provide support in terms of research, collection of material and documentation upon request. No consultant is contemplated for this evaluation.

4.2 Peer reviewers/advisory panel

− One EvD staff with cross-cutting experience: TBA

− One EvD staff for policy dialogue and country operations aspect: TBA

− OCE has expressed an interest to collaborate on this study providing inputs particularly on transition impact indicators and the link between those and TC results. Toshiaki Sakatsume (Senior Economist), Alexander Plekhanov (Senior Economist), and particularly Murat Jadraliyev (Economic Analyst) will be involved and act as peer reviewers.

4.3 Timetable

Milestone Date (by week)

Study starts September 2012

Approach paper approved October 2012

Field visit complete November 2012

Draft circulated to internal peer reviewers June 2013

Draft cleared by CE for circulation to external peer reviewers

July 2013

Draft cleared by CE for Management Comments July 2013

Final editing August 2013

Final approved by Chief Evaluator August 2013

Draft Board Summary September 2013

Final Board Summary approved by Chief Evaluator September 2013

Final distribution September 2013

European Bank for Reconstruction and Development One Exchange Square London EC2A 2JN United Kingdom Switchboard/central contact Tel: +44 20 7338 6000 Fax: +44 20 7338 6100 Information requests For information requests and general enquiries, please use the information request form at www.ebrd.com/inforequest Evaluation department Tel: +44 20 7338 6467 Fax: +44 20 7338 6726 Email: [email protected] Web site www.ebrd.com

approach paper technical cooperation experience in new ... · evaluation approach paper: technical...

Documents