baastel macro results framework assessment appendix final · 2018-04-27 · afesd arab fund for...

www.baastel.com

Creating a Macro Results Framework for the Middle East and North Africa

Transition Fund “Feasibility Assessment”

Final Report

Appendix

Submitted by: Le Groupe‐conseil baastel ltée

Requested by: The World Bank

June, 2014

www.baastel.com

Le Groupe‐conseil baastel ltée

85 Victoria Street

Gatineau (Québec) Canada, J8X 2A3

Tel: +(1) 819‐595‐1421

Fax: +(1) 819‐595‐8586

www.baastel.com

Contact: [email protected]

“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix

i

TABLE OF CONTENTS Acronyms ............................................................................................................................... iii

1. Introduction ................................................................................................................ 1

2. Overview of M&E Practices across the ISAs .................................................................... 1

2.1. African Development Bank ............................................................................................ 1

2.2. European Bank for Reconstruction and Development ...................................................... 2

2.3. European Investment Bank ............................................................................................ 2

2.4. International Finance Corporation .................................................................................. 3

2.5. Islamic Development Bank ............................................................................................ 3

2.6. Organisation for Economic Co‐operation and Development .............................................. 4

2.7. World Bank .................................................................................................................. 4

2.8. Others ......................................................................................................................... 5

3. Case Studies .................................................................................................................5

3.1. The Global Environment Facility ..................................................................................... 5

3.2. Global Agriculture and Food Security Program ............................................................... 12

3.3. Neighbourhood Investment Facility............................................................................... 17

3.4. Conclusion................................................................................................................... 20

4. Options ..................................................................................................................... 21

4.1. Option A1 “Enhanced” Status Quo ................................................................................ 21

4.2. Option B1 – Revise and Use the DFID‐based Logic Model ................................................ 27

4.3. Option B2 – Developing Project Cluster/Nesting Model ................................................... 31

4.4. Option B3 – Reporting Against a MENA Transition Fund Theory of Change – “Hybrid” Option

37

4.5. Others ........................................................................................................................ 41

Annex 1. Methodology ........................................................................................................... 42

Annex 2. Glossary ..................................................................................................................46

Annex 3. List of Documents Reviewed .................................................................................... 48

Annex 4. List of Persons Met/Interviewed ................................................................................ 52

Annex 5. Addendum .............................................................................................................. 55


ii


iii

ACRONYMS

Acronym Definition

€ Euros

$ Dollars

ADB Asian Development Bank

AfDB African Development Bank

AFESD Arab Fund for Economic and Social Development

AMF Arab Monetary Fund

AS Advisory Services (IFC)

CAREC Central Asia Regional Economic Cooperation

CEO Chief Executive Officer

CPPR Country Portfolio Performance Reviews

CSI Core Sector Indicator (AfDB)

CSO Civil Society Organization

CU Coordination Unit

DAC Development Assistance Committee (OECD)

DFATD Department of Foreign Affairs, Trade and Development (Canada)

DFID Department for International Development (UK)

DIME Development Impact Evaluation Initiative

DOTS Development Outcome Tracking System

EBRD European Bank for Reconstruction and Development

EC European Commission

EIB European Investment Bank

ENP European Neighbourhood Policy

ENPI European Neighbourhood and Partnership Instrument

EO Evaluation Office (GEF)

EU European Union

EvD Evaluation Department (EBRD)

ESIAF Economic and Social Impact Assessment Framework (EIB)

FAO Food and Agriculture Organization

FIG Finance Institutions Group (NIF)

GAFSP Global Agriculture and Food Security Program

GDPRD Global Donor Platform for Rural Development

GEF Global Environment Facility

GPB Global Program Branch

HS Highly Satisfactory

HU Highly Unsatisfactory

ICR Implementation Completion and Results Report

IDB Inter‐American Development Bank

IE Impact Evaluation

IEG Independent Evaluation Group (WB)

IFAD International Fund for Agricultural Development

IFC International Finance Corporation

IFI International Financial Institutions

IMF International Monetary Fund


iv

IS Investment Services (IFC)

ISA Implementation Support Agency

IsDB Islamic Development Bank

ISR Implementation Status Report

KPI Key Performance Indicator

M&E Monitoring and Evaluation

MDG Millennium Development Goals

MENA Middle East and North Africa

MOPAN Multilateral Organisation Performance Assessment Network

MS Moderately Satisfactory

MU Moderately Unsatisfactory

NGO Nongovernmental Organization

NIF Neighbourhood Investment Facility

OCE Office of the Chief Economist

OECD Organization for Economic Cooperation and Development

OPEC Organization of the Petroleum Exporting Countries

OPS Overall Performance Studies (GEF)

PAR Project Appraisal Reports

PCR Project Completion Reports

PDO Project Development Objective

PIR Project/Program Implementation Report

PISR Project Implementation Support Report

ReM Results Measurement Framework (EIB)

ROtl Review of Outcomes to Impacts (GEF)

RRS Results Reporting System (AfDB)

S Satisfactory

SC Steering Committee

STAP Scientific and Technical Advisory Panel

TC Transition Country

TIMS Transition Impact Monitoring System

U Unsatisfactory

UK United Kingdom

UNDP United Nations Development Programme

UNEP United Nations Environment Programme

USA United States of America

WB World Bank

WFP World Food Programme


1

1. INTRODUCTION his document represents the Appendix to the Final Report of the Feasibility Assessment on Creating a

Macro Results Framework (herein referred to as “Macro Framework” for brevity) for the Middle East and

North Africa (MENA) Transition Fund (also referred to as “the Fund”). This document presents a more

complete discussion on key aspects important for understanding the context and making an informed

decision on the feasibility of developing a Macro Framework for the Fund. In brief, the purpose of this document is

to provide the Steering Committee (SC) members with further details on the following aspects of the completed

Feasibility Assessment:

1. Overview of Monitoring and Evaluation (M&E) Practices across the Implementation Support Agencies

(ISAs);

2. Case Studies of Practices in Funds with Similarities to the MENA Transition Fund;

3. Options for the Way Forward, in particular a detailed discussion of Variations on Creating a Macro

Framework for the Fund.

2. OVERVIEW OF M&E PRACTICES ACROSS

THE ISAS his overview of data collection and M&E practices across the different ISAs could be useful (e.g. as a

baseline) for the next consultancy if the decision is taken to develop a Macro Framework for the Fund. As

important partners involved in project M&E under the Fund, the development of a Macro Framework for

the Fund should take into account, to the extent possible, the particular systems of the ISAs involved and

their internal procedures and capabilities.

2.1. African Development Bank

In recent years, the African Development Bank (AfDB) has been making noticeable efforts to better focus project

supervision on results, for example with the introduction of the new supervision report format and supervision

assessment, and by entrusting sector departments with the responsibility to proactively track Core Sector

Indicators (CSIs). The Bank introduced guidelines and a results‐oriented format in 2008. A new automated Results

Reporting System (RRS) is expected to be launched in 2014. This system will link expected results (as per the

logical frameworks in Project Appraisal Reports (PAR)) to progress toward results (as documented in supervision

reports) and results achieved (as reported in Project Completion Reports (PCR)). Thus, AfDB is moving forward

with more rigorous procedures and M&E systems that will track results internally. However, and as noted in AfDB’s

Management Response to the 2011 Multilateral Organisation Performance Assessment Network (MOPAN)

assessment on organizational performance, “many of these reforms will take some time before they fully yield

T

T


2

their results”.1 Some deficiencies in terms of monitoring and reporting on key results have to do with reporting

that is focused on “processes” rather than “change”.

As confirmed through interviews held with representatives of AfDB, in reporting data for the Fund activities, AfDB

simply transfers information collected by its field staff through the Bank’s internal instrument for project reporting

(i.e. supervision reports and PCR) to the standard Progress Report requested by the Fund Coordination Unit (CU),

which is considered efficient as this way the Progress Report does not constitute an additional reporting burden.

2.2. European Bank for Reconstruction and Development

At the European Bank for Reconstruction and Development (EBRD), the transition impact of every project is

planned for as part of the process of choosing, preparing and appraising projects, in addition to an independent

Evaluation Department (EvD) which closes the cycle. All projects are scrutinized by the credit department to

ensure sound banking principles and by the Office of the Chief Economist (OCE) on their development/transition

impact. The EBRD’s policy requires 100% compliance and the control systems do not allow for diverting from this

policy. Each project document specifies benchmarks for “transition impact” outcomes that are monitored over

time during project implementation. EBRD policy requires annual or bi‐annual reviews of existing projects through

the preparation of monitoring reports, which assess developments in a project’s transition impact, environment

and financial performance. EBRD’s efforts to measure the wider development impact of its transition activities are

noteworthy. To assess the transition impact of its operations, the EBRD has developed a framework that captures

and monitors key transition aspects from the project design stage through to project closure. The OCE monitors

projects under the Transition Impact Monitoring System (TIMS). The TIMS also records assessments of risks to

transition in each project, and how these risks change over the project’s life.2

As stated above, EBRD has very strict procedures that all projects must abide by. Hence, the reporting done

against the project results framework imposed by the Fund reflects EBRD procedures/standards, which are in

general deemed both sufficient and relevant for the kind of activities financed by the Fund.

2.3. European Investment Bank

To further strengthen its assessment, measurement and reporting on the results and impacts of its operations, the

European Investment Bank (EIB) launched in 2012 a new Results Measurement framework (ReM) to replace its

Economic and Social Impact Assessment framework (ESIAF). One intention with this new ReM framework is to

harmonize, to the extent possible, results indicators with other International Financial Institutions (IFIs) and

thereby simplify client reporting requirements for co‐financed operations.3

The EIB’s ReM rests on three pillars:

Pillar 1: An assessment of a project’s consistency with EIB mandate objectives and its

contribution to European Union (EU) priorities and country development objectives, going

beyond a focus on eligibility alone.

1 AfDB. (n.d.). MOPAN Management Response (Draft 23 November). 2 EBRD. (2010). Transition Impact. [online].

3 EIB. (2013).Measuring development results. The EIB Results Measurement (ReM) framework.


3

Pillar 2: A series of sector‐specific standardized indicators to capture a project’s economic,

social, environmental and institutional outcomes. It continues to measure project quality and its

ability to achieve the expected results.

Pillar 3: An assessment of the EIB additionality over market alternatives in terms of financial

product, technical, structuring and sector contribution and standards and assurance.4

As noted above under Pillar 2, EIB uses standardized core indicators for the M&E of its entire portfolio and for

different sectors. It is expected that EIB will make use of these standardized core indicators to report for the Fund

activities and will appreciate the flexibility offered to conduct “business‐as‐usual”.

2.4. International Finance Corporation

The International Finance Corporation (IFC) has a Development Outcome Tracking System (DOTS) to track the

development results of all its active operations continuously throughout their project lives. For investment services

(IS) projects, the overall development outcome rating is a synthesis of four performance categories that are

informed by the indicators agreed with the client: financial performance, economic performance, environmental

and social performance and private sector development impact. For advisory services (AS), the overall

development outcome rating is a synthesis of the overall strategic relevance, effectiveness (as measured by

project outputs, outcomes, and impacts), and efficiency of the services. At project completion, intended results

are compared with achieved results. Some results—medium‐term outcomes and longer‐term impacts—may be

unknown at project completion but can be examined post‐completion.

Each IFC industry department has identified a menu of standard, sector‐specific indicators that capture the

sector’s typical development impacts. In addition, IFC has some wide corporate indicators that are mandatory for

tracking particular impacts related to environmental & social improvements and corporate governance issues. By

standardizing indicators, IFC is able to compare performance and aggregate developmental reach by industry

department and across the corporation as a whole for a particular time frame.5 At the outset of a project, IFC's

staff members identify standardized indicators with baselines and targets. They track progress throughout

implementation and until project closure. This systematic tracking allows IFC to monitor projects, learn from what

does and does not work, and improve performance. The system facilitates data‐aggregation, comparison,

analysis, and reporting on key output, outcome and impact indicators and efficiency of the services. To meet its

monitoring obligations, the IFC requires companies to submit regular reports on financial as well as social and

environmental performance, and information on factors that might materially affect the enterprise. In addition,

IFC monitors progress through it corporate scorecard and reports once a year on the development results of its

interventions in an Annual Report.

2.5. Islamic Development Bank

Evidence collected during interviews suggests that there are various types/levels of monitoring at the Islamic

Development Bank (IsBD). The IsDB has a monitoring tool at the portfolio level which looks like a dashboard of all

projects under implementation. To report on project progress IsDB possesses a Project Implementation Support

Report (PISR). The PISR highlights the challenges/issues the projects are facing in their implementation. The IsDB

4 Idem.

5 5 IFC. (2014). IFC Development Results and Impact. [online].


4

also sporadically conducts Country Portfolio Performance Reviews (CPPRs). The report that is generated goes to

the President directly, requesting guidance, changes and approval, and even in some cases termination of projects

that are not up to par. IsDB can also count on a “results unit” internally to look at all the PISR submitted. Data from

the different performance review tools are entered directly into database software that produces automated

reports as needed. These many layers of monitoring and reporting are particularly evident for IsDB project in

Yemen “Preparation and Implementation Support Project for the Special Industrial Zone Project in Hodeida”. For

this project, the project implementation team has to report at the Transition Country (TC) ministry level, to the

IsDB every quarter, and to the Fund twice a year.

Initiatives taken in recent years to strengthen project monitoring and reporting and direct it towards expected

outcomes include enhancing the IsDB’s field presence, increasing supervision budgets, and designing a supervision

report format that includes a focus on results. Information collected during interviews alluded that at the IsDB,

results‐based logical frameworks are mandatory in all projects and that core indicators are mainstreamed into

these results‐based logical frameworks. Data collected suggests that the IsDB reports on its own key performance

indicators (KPIs) in the Progress Report submitted to the Fund. The KPIs are collected through supervision

undertaken by the implementation unit at the project level.

2.6. Organisation for Economic Co‐operation and Development

The Organisation for Economic Co‐operation and Development (OECD) is generally known for being a forum in

which governments can work together to share experiences and seek solutions to common problems. OECD

usually provides analyses and recommendations on a variety of subjects and matters and is less thought of as a

“direct implementer” of projects in developing countries. Maybe it is for this reason, that OECD has comparatively

fewer demanding rules in terms of M&E vis‐a‐vis the other ISAs involved with the Fund, which are for the most

part quite used to dealing with such projects and tracking performance rigorously. Based on discussions held with

OECD representatives, the organization shows great flexibility with regards to the method and template for

project progress reporting. Thus, for the interventions funded by the Fund, the OECD uses only the Progress

Report template provided in Fund Operations Manual (p.37) and this reporting imposes no major burden on them.

2.7. World Bank

Generally speaking, the designs of the World Bank (WB) operational activities incorporate a framework for M&E.

During the project life, the Bank monitors and evaluates its own contribution to results using this framework.

Project teams compile information and data regularly in Implementation Status Reports (ISRs) and when projects

are complete, project teams are also required to complete an Implementation Completion and Results Report

(ICR). The latter document includes for example results achieved, problems encountered, and lessons learned.

These are later reviewed by the Independent Evaluation Group (IEG). In brief, the WB requires performance

reporting on a regular basis for all projects, and institutional procedures and systems are well in place to ensure the

tracking of progress and achievements of projects supported.


5

2.8. Others

M&E systems and procedures of the following ISAs were not looked at for this assessment in‐depth as the

representatives did not respond to the request to participate:

Arab Fund for Economic and Social Development (AFESD)

Arab Monetary Fund (AMF)

International Monetary Fund (IMF)6

Organization of the Petroleum Exporting Countries (OPEC) Fund for International Development

3. CASE STUDIES he case study will describe, for each fund, the structure of the logic model: the results chain from

outputs/activities to outcomes and impacts, related theory of change and results aggregation strategies.

This section will also describe the organization of the monitoring and reporting function: the

implementation arrangements by which results are collected as well as roles and responsibilities of the

various organizations involved in the funds/case studies.

3.1. The Global Environment Facility

The Global Environment Facility (GEF) is a financial mechanism for international cooperation, established in 1991,

based on partnerships, that provides new and additional grant and concessional funding to meet the agreed

incremental costs of measures to achieve agreed global environmental benefits ‐‐ in six focal areas7 ‐‐ in

developing countries and countries with economies in transition8. The GEF is a partnership involving 183 member

countries, 10 agencies (the WB, United Nations Development Programme (UNDP), United Nations Environment

Programme (UNEP), the regional development banks, Food and Agriculture Organization (FAO), the International

Fund for Agricultural Development (IFAD), and the United Nations Industrial Development Organization),

recipient countries, nongovernmental organizations (NGO), and the private sector.

GEF’s governance architecture is set out in the Instrument for the Establishment of the Restructured Global

Environmental Facility and consists of an Assembly, a Council, the Secretariat, the Implementing and Executing

agencies and the Scientific and Technical Advisory Panel (STAP). The Assembly and Council are the governing

bodies empowered to take decision. The others, while an integral part of governance, have only an advisory role.

As financial mechanism for four international environmental Conventions (Biodiversity, Climate Change,

Persistent Organic Pollutants, and Combating Desertification) the GEF helps fund initiatives that assist developing

6 It should be noted that representatives from the IMF said they were keen to participate in the assessment, but had nothing of direct relevance to share at the time of the data collection. 7These areas are: biological diversity, climate change mitigation and adaptation, international waters, land degradation, ozone layer depletion, and persistent organic pollutants, with sustainable forest management cross‐cutting relevant focal areas. 8 GEF. (2013). GEF Website. Investing in our Planet. [online].

T


6

countries in meeting the obligations of the Conventions. GEF also collaborates closely with other treaties,

protocols and agreements9.

3.1.1. Overview of the GEF Results Framework

The following presents an overview of the GEF results framework, referring to its logic model and theory of change

as well as the results aggregation strategy, the role of evaluations, and finally, the roles and responsibilities for

M&E between GEF partners.

Logic Model and Theory of Change

GEF developed in 2012, 21 years after the inception of GEF, a fund level theory of change (please see Figure 1

below). The general framework theory of change seems to be meant to capture lessons learned.

Figure 1. General Framework for GEF Theory of Change10

Aggregation and Reporting

GEF, monitoring take place on three levels:

1. Project and program level: progress toward results, including outcomes, implementation issues, and

portfolio‐wide trends is based on information submitted by the GEF Agencies through the project or

program implementation reports (PIR) and focal area tracking tools;

2. Portfolio level: The aggregation of findings across the portfolio of projects and programs can be done by

focal area, type, theme, or issue. Portfolio monitoring tools are intended to roll up indicators from the

9 Pérez del Castillo, Carlos (GEF). (2009) OPS4 Technical Document # 5: Governance of the GEF.

10 GEF EO. (2012). Annual Thematic Evaluations Report (2012).


7

individual project level to the portfolio level and track overall portfolio performance in focal areas. Each

focal area has developed its own tracking tool to meet its unique needs, which brings uniformity. It is a

requirement of the GEF that the implementing agencies fill these tracking tools to report to the GEF;

3. National and global level: mainly of global environmental status, stress, trends, and benefits, based on

independent data gathering and analysis by national bureaus of statistics and/or international bodies and

organizations; these monitoring exercises occur mostly between replenishment periods when reassessing

and redesigning indicators for country allocations under any GEF system for allocation of resources, and

overall when formulating the focal area strategies.11

In brief, there is no single, overarching macro framework per se for GEF. GEF results are organized using what can

be called a “Project Portfolio” approach composed of results frameworks for each “focal” area of work against

which each project in that focal area has to report.12 This is a key element of GEF’s results aggregation process. As

stated, aggregation at the project and portfolio level of the GEF requires the completion of a PIR. However,

reporting of results at this level is based on a narrative and a self‐evaluation using a standard set of categories,

much like the ratings in the standard Progress Report for the MENA Transition Fund, except that the rating for the

GEF range from Highly Satisfactory (HS) to Highly Unsatisfactory (HU). Table 1 below is an excerpt from a PIR.

Table 1. Excerpt from GEF PIR13

OVERALL RATING OF IMPLEMENTATION PROGRESS PREVIOUS RATING

NEW RATING

U MS

Outputs and Outcomes

Component 1: Most of the products established to achieve the outcomes of this component have been executed according to plan. However, there are significant delays in the demarcation of the Marine Protected Area, currently just 20% of the area has been demarcated while the project life is already almost 50% complete. Also, the Government has issued a law to consolidate the protected areas management category of the national system. If this regulation is approved, the outputs related with the Integrated Management Plan updating and demarcation could be affected. In conclusion, four (4) of the twelve (12) outputs planned for the component show delays in their implementation.

The consequences to such a broad‐based, generalized, subjective approach to results aggregation is that the

discussion on results achievement at the fund level is limited. Given the self‐evaluative nature of the ratings, there

is also a risk that the results quoted above could be due to bias. Secondly, the narrative approach to performance

reporting may contribute to the perception by key informants that GEF imposes a reporting burden on projects.

In addition, GEF has developed performance tracking tools14 per work or focal area as follow:

Biodiversity Tracking Tool;

GEF International Waters Tracking Tool;

GEF Climate Change Mitigation Tracking Tool;

GEF Climate Change Adaptation Tracking Tool;

GEF Persistent Organic Pollutants Tracking Tool;

GEF Land Degradation Tracking Tool;

GEF Sustainable Forest Management/REDD Tracking Tool.

11 GEF. (2010). M&E Policy.

12 For more details, please see GEF. (2010). Focal Area Framework and LDCF/SCCF Framework. Excerpts from GEF‐5 Programming

Document at the Sixth Meeting for the Fifth Replenishment of the GEF Trust Fund, April 7, 2010. 13 Inter‐American Development Bank (IDB). (2012). Project Implementation Report (PIR) FY2012.

14 For more details on these tracking tools, please refer to GEF Website: http://www.thegef.org/gef/tracking_tools.


8

These tracking tools can be conceptualized as a results framework per area of work or focal area. The results that

are tracked are both “hard” indicators such as, “Landscape/seascape area directly covered by the project (ha)”15

and the standardization and quantification of “soft” indicators related to institutional reform, see excerpt below.16

The main benefit of basing fund‐level aggregation on focal area results is that reporting at higher levels is

facilitated.

Evaluation

The GEF fund is typical of OECD Development Assistance Committee (DAC)‐based17 evaluation standards where

performance and evaluation reporting take place at every level (project, country, and agency)18. Established 23

years ago, the scope and reach of the GEF means that it is possible to report frequently and reliably on results up

to impact level. Reflecting its longevity and resourcing, the GEF Evaluation Office (EO) publishes annual results at

the fund and country levels, as well as by theme19. The EO has three key functions which permits the

dissemination of M&E standards fund‐wide: the setting of minimum standards; quality‐control and oversight on

the design of M&E systems at the program and project levels; as well as the sharing of evaluation evidence20.

Roles and Responsibilities for M&E

M&E are a shared responsibility in the GEF. The GEF Council provides the overall framework, starting with

agreement on objectives and corporate and focal area results frameworks. The GEF Secretariat proposes to the

Council how these objectives and results should be monitored, and the GEF EO proposes to the Council how these

should be evaluated. Emerging environmental and development trends, and GEF results and performance within

the context of these trends, are reported on in the overall performance study prepared by the GEF EO as one of

the key documents of the fund replenishment process. Based on this information, the GEF Council makes strategic

and policy‐level decisions. The GEF Agencies and their partners execute project, program, and portfolio M&E. The

GEF STAP provides advice on indicators, targets, and evaluation approaches. The GEF EO collaborates with the

independent evaluation units of the GEF Agencies to enhance collective capacity to fulfill evaluation needs

effectively and efficiently.21

Consistent with the decentralized organization of the GEF, implementing agencies are expected to use their

own M&E systems. Each GEF Agency has its own system of governance and rules and regulations governing the

implementation of activities, as well as the M&E of these activities. However, the GEF Council can decide on which

partners it collaborates with and can require minimum standards and minimum procedures to be applied in

activities that it funds.22

The decentralized roles and responsibilities for M&E are counter‐balanced by an independent GEF EO that reports

directly to the GEF Council. In addition, the M&E Office has a separate budget and its own program of work, both

to be approved by the Council.23 Roles and responsibilities for M&E at the GEF are summarized in Error! Reference

source not found. and graphical form in Figure 2 below.

15 GEF. (2011). GEF‐5 Tracking Tool for Biodiversity Focal Area.

16 Idem.

17 GEF. (2010). GEF M&E Policy.

18 Binnendijk, Annette. (2000). Results Based Management in the Development Co‐operation Agencies: a Review of Experience.

19 GEF. EO. (2013). Independent Evaluation Office Website. [online].



22 Idem.

23 De Crombrugghe, Dominique, et al. (2009) Peer Review: the Evaluation Function of the Global Environment Facility (GEF), Final

Report.


9

Table 2. Key Roles and Responsibilities of GEF Partners in M&E25

Partner Key Roles and Responsibilities in M&E

GEF Council Policy making on M&E Oversight of M&E functions Enabling environment for M&E

GEF EO Independent GEF evaluation Oversight of project and program evaluations Oversight of the relevance, performance, and overall quality of monitoring systems Set minimum requirements for GEF M&E Evaluative knowledge sharing and dissemination

GEF Secretariat Set results frameworks at focal area and corporate levels GEF portfolio monitoring across Agencies and focal areas Report on and incorporate lessons from portfolio monitoring Review of GEF M&E requirements in project and program proposals Coordinate partnership knowledge management activities

Agency GEF operational units

Monitor the Agency GEF portfolio Report Agency’s project, program, and portfolio progress, results, learning, and lessons Ensure monitoring at the project and program levels Adaptive management of project and program implementation Systematically involve national partners and share project M&E information at the national level

Agency evaluation units

Project, program, and/or corporate Agency independent evaluations Mainstream GEF into relevant Agency evaluations

STAP Advice on scientific/technical matters in M&E Support to scientific and technical indicators Support knowledge management and information sharing

GEF operational focal points

Collaborate on M&E at project, program, and portfolio levels

Others (NGOs and civil society organizations (CSOs), private sector, community members)

Participate in monitoring activities and mechanisms Provide views and perceptions to evaluations

24 Volonte, Claudio. (GEF EO). (2011). The GEF M&E Policy. PPT Presentation.


Figure 2. Flow Chart of M&E Reporting in the GEF

24


10

Another counter‐balance to the shared responsibility for M&E at the GEF is strong knowledge management of the

M&E function at fund‐level. As per its mandate, the EO publishes guidelines on M&E in the form of policy

documents26. In addition to policy statements on M&E, the EO also produces technical guidelines and specific

tools27. A good example of the relatively high‐quality advice given on M&E is the Review of Outcomes to Impacts

(ROtI) methodology document28, which also reflects the predictable shift away from monitoring to evaluation29

(i.e. a focus on outcomes and impacts), given the longevity of GEF. Another key aspect of EO activity is the role of

facilitator in the exchange of M&E perspectives within the GEF “family” (i.e. consultants, implementing agencies,

national recipients, and so on). The GEF EO has sponsored such events as an Evaluators’ Summit and an

International Conference on Evaluation, Environment and Sustainable development.30 The key consequence of

these knowledge management activities, widely disseminated digitally, is the possibility to aggregate results at

the fund level.

3.1.2. Strengths, Weaknesses and Opportunities for

the MENA Transition Fund

Each of the funds discussed in this case study section have strengths and weaknesses in terms of learning lessons

for the potential development of a Macro Framework for the MENA Transition Fund. However, each fund also

have “checks and balances” to offset the weaknesses of their respective approaches.

The weakness of GEF is its performance reporting at project level which seems to yield a limited amount in the

form of aggregated results. The “check and balance” against this is the development of what is effectively a “layer”

of results frameworks per area of work or focal area and by developing, ex ante, results categorizations for a fund‐

level theory of change. Yet, perhaps because of the narrative format of the PIR, this particular tool and the

tracking tools are at the same time perceived as imposing a reporting burden on projects.

In terms of strengths, fund‐level governance of M&E at GEF is strong with reporting lines for M&E going directly to

the GEF Council. The appetite for the production of (self‐described) “cutting‐edge” M&E knowledge at fund‐level

is also strong. The “check and balance” against a central approach to the M&E function is a strong knowledge

management function that widely distributes M&E publications produced at the fund‐level. In terms of value for

money, a “top‐down”, central approach where evaluation knowledge is produced at the fund‐level can mean

increased administrative costs at the fund level. Value for money is questionable with respect to an ex‐ante

approach to results aggregation. This process requires primary research.

There are many aspects of the M&E regime at GEF which can be adopted by the MENA Transition Fund, albeit

within a solid understanding of the key differences. The following are a few points to consider:

The MENA Transition Fund could continue with its “devolved” M&E function at the project level,

but put more emphasis on building sustainability and capacity for M&E on the part of ISAs and

TCs. Strengthening the M&E capacity of line ministries in TCs will indirectly strengthen fund‐

level reporting while improving capacity;


27 Please see, for example, GEF. (2011) GEF‐5 Tracking Tool for Biodiversity Focal Area.

28 GEF EO. (2009). The RoTI Handbook: Towards enhancing the impacts of environmental projects, Methodological Paper #2.

29 De Crombrugghe, Dominique, et al. (2009). Peer Review: the Evaluation Function of the Global Environment Facility (GEF),

Final Report. 30 Idem.


11

Careful planning of the project‐level performance reporting regime to avoid costly “re‐

engineering” of Fund‐level aggregated results;

Adopting a “layered” results framework approach to facilitate aggregation of the diverse type of

projects funded;

GEF produces M&E research on impacts (and other topics) in addition to report on information

coming from the projects only. These additional pieces are not currently necessary for the MENA

Transition Fund. Keeping things simpler and more focused at this early stage could allow the

MENA Transition Fund to keep costs at a more manageable level compared to GEF.

Notwithstanding the greater sectoral breadth that the MENA Transition Fund covers, there are lessons to be

consider: for example, there is a decentralized approach to M&E at the GEF that is offset by: an independent M&E

function at the fund level; the distribution of high‐quality M&E guidance regarding policy and technical issues; as

well as a focus on knowledge management.

The approach to aggregation of results with respect to developing a theory of change could also be instructive for

the MENA Transition Fund. “Aggregation” is the classification of project results identified as important to bringing

about the desired impact(s). The development of a theory of change for a focal area is a combination of inductive

(“bottom up”) and deductive (“top down”) reasoning. It is inductive in that the results frameworks for projects in a

focal area are analyzed and a properly‐designed theory of change is developed for each. It is deductive in that,

using some taxonomy, an internationally‐accepted standard for that focal area is preferred, intended project

results are then categorized, based on projects’ theory of change. GEF provides many valuable technical insights

on the development of an aggregation method using both inductive and deductive reasoning. However, the

“downside” of an inductive approach as developed by the GEF is that projects’ results frameworks and theory of

change have to be analyzed and, depending on analysis findings, redesigned retrospectively. The approach is

therefore time/cost intensive, most likely requiring primary research. Also, compared to MENA Transition Fund,

GEF has a critical mass of completed projects from which to draw conclusions about project commonalities

through an impact analysis.

There are other factors which militate against wholesale adoption of the GEF M&E approach by the MENA

Transition Fund. Since 2003, GEF has focused on the measurement of impacts and lessons learned. This is to be

expected, given the longevity of and resources channeled through the GEF. However, the MENA Transition Fund is

on the other end of the spectrum. It is a facility barely out of the inception stage, with a fraction of funds in

operations compared to GEF. In other words, the focus of the MENA Transition Fund would be on results “lower”

down the results chain: activities; outputs and outcomes. In this, the GEF has relatively little guidance to offer,

except that implementing agencies should use their own systems and that the GEF only has a quality assurance

and standards‐setting role with respect to these. Moreover, the functions created by the GEF took years to

develop, are costly to maintain and were set up for a much larger fund with a long term horizon. Thus, the

strategies and methodologies used by the GEF would need to be carefully adapted for potential consideration in

the context of the MENA Transition Fund in particular as the latter could not bear a comparability cost footprint.


12

3.2. Global Agriculture and Food Security Program

The Global Agriculture and Food Security Program (GAFSP) is a multilateral mechanism to assist in the

implementation of pledges made by the G20 in Pittsburgh in September 2009. The GAFSP is a financial

intermediary fund (FIF) under external governance, established by the WB Group to support inclusive, strategic

and country‐led agriculture and food security investment plans in poor countries. GAFSP makes funding available

for countries that exhibit both need and significant readiness to use funds well for increased long‐term food

security. To date, Australia, the Bill & Melinda Gates Foundation, Canada, Ireland, Japan, the Netherlands, South

Korea, Spain, United Kingdom (UK), and the United States of America (USA) have contributed resources to the

GAFSP. These ten donors have pledged a total of US$ equivalent 1.4 billion to the Public and Private Sector

Windows. The resources received from the donors as of December 31, 2013 amounted to approximately US

equivalent $1.2 billion. Eligible countries must demonstrate a high level of need and submit a comprehensive,

technically sound project proposal along with their national agricultural investment plans. GAFSP (public and

private sector windows) has allocated funds for the following countries: Bangladesh, Burkina Faso, Burundi,

Cambodia, Cote d’Ivoire, Ethiopia, Haiti, Honduras, the Gambia, Kenya, Kyrgyz Republic, Liberia, Malawi, Mali,

Mongolia, Nepal, Nicaragua, Niger, Pakistan, Rwanda, Senegal, Sierra Leone, Tajikistan, Tanzania, Togo, Uganda,

Vanuatu, Yemen, and Zambia. Each GAFSP grant focuses on filling a specific need for the nation’s food security

and agricultural growth.

In terms of governance, GAFSP is similar if not identical to that of the MENA Transition Fund. In GAFSP, the WB

plays three roles: (1) Trustee (to carry out fiduciary responsibilities as set out in the Trust Fund Administration

Agreement), (2) CU (a small team to support the GAFSP SC and Technical Advisory Committee and facilitate

communications between the SC and the other partners of GAFSP), and (3) potential Supervising Entity of GAFSP

funded projects, if selected by the SC. IFC, the WB’s private sector arm, manages the Private Sector Window of

GAFSP. Allocations are determined by a SC composed of a balance of representatives from donor and recipient

countries, CSOs, and multilateral organizations, following advice from a technical committee of independent

experts. 31

3.2.1. Overview of GAFSP Results Framework

The following is a discussion of the GAFSP results framework, referring to its logic model and theory of change as

well as the results aggregation strategy. The section also looks at the role of evaluation and the roles and

responsibilities of the different partners involved.


Figure 3 below is a depiction of the GAFSP’s objectives and areas of intervention which shows the intervention

logic and how the fund expects to deliver results (i.e. theory of change).

31 GASFP. (2011). GAFSP Website. [online].


13

Figure 3. GAFSP Logic Model and Theory of Change32


The aggregation strategy for the GAFSP includes the development of common indicators to enable monitoring of

results across projects: “Each GAFSP financed project will select from the menu of core indicators (…) at least one

output or intermediary outcome indicator for the results areas of focus of the GASFP project. These indicators will

be included in the respective project results frameworks. This will allow for aggregation of results across

projects.”33 An excerpt of the “menu” of indicators appears in Table 3 below.

32 GAFSP. (2013). GAFSP Annual Report.

33 GAFSP. (2011) Monitoring and Evaluation Plan.


14

Table 3. Sample of “Menu” Approach to the Fund‐level Results, GAFSP34

Results Area Menu of Common Project Level Progress Indicators (Output and Intermediate Outcomes)

Higher‐yielding technologies adopted

Technology generated

1. Number of collaborative research or extension sub‐projects implemented 2. Number of client days of training to raise agricultural productivity provided to scientists, extension agents, agro‐dealers, farmers, community members etc (disaggregated by gender) 3. Number of client days of extension services provided to farmers, community members etc (disaggregated by gender) 4. Number of farmers who have adopted the technology being promoted 5. Number of additional hectares which have adopted the technology being promoted

In developing these core indicators for the GAFSP, the CU leveraged on the work of others to diminish the burden

of this task. They reviewed the IDA core sector indicators and core indicators listed in the publication by the Global

Donor Platform for Rural Development, FAO, and the World Bank (GDPRD/FAO/World Bank 2008). The selection

of core indicators was a participatory process. An earlier draft was circulated to the Supervising Entities and their

comments have been incorporated before moving forward. GAFSP document mention that these indicators were

selected using the following principles:

1. “The indicators are deliberately set at the output and intermediate outcome levels and have been

selected in order to have parsimonious but wide applicability to GAFSP‐funded interventions.

2. The indicators do not provide an added burden for individual project M&E frameworks, rather provide

indicators that would be included anyway, but in a form that is readily comparable across different GAFSP

projects.”35

There are two noteworthy aspects of the example above. First, it is the specificity of the results indicator to the

GAFSP component or area of work, which is, possible, to a certain extent, because of the sectoral focus on

agriculture of the GAFSP. Secondly, given that it may take many years of project operation before it is possible to

report on progress made on higher‐level results and the associated indicators, GAFSP focuses on aggregated

results reporting at the output and intermediate outcome level.

Evaluation

GAFSP SC selected the WB's Development Impact Evaluation Initiative (DIME) to implement in‐depth impact

evaluations for certain GAFSP Public Sector Window projects. DIME is a global program hosted in the WB's

Development Research Group. Its purpose is to increase the use of impact evaluation (IE) in the design and

implementation of public policy and increase institutional capacity and motivation for evidence‐based policy. The

GAFSP M&E Plan notes that all GAFSP projects will undergo an independent evaluation as well as an IE. The

following table outlines the IE policy for the GAFSP.

34 Idem.

35 Idem.


15

Table 4. Impact Evaluation Policy for GAFSP36

Type of IEs Target Indicative Cost Implementation Funding

In‐depth IEs (experimental or quasi‐experimental IE)

Approximately 10‐30% of all GAFSP projects selected according to lesson‐learning priorities

$800,000 for each IE CU to centrally contract a specialized organization to carry out all in‐depth IEs

To be funded separately by the SC

Rapid IEs (non‐experimental IE)

All GAFSP projects (except for those that have been chosen to carry out the in‐depth IEs)

$50,000 for each IE Each project to contract a specialized organization to carry out its rapid IE

To be funded from each GAFSP‐awarded Grant amount


Similar to the GEF and to the MENA Transition Fund, GAFSP delegates project implementation to the

“supervising entity”. As a matter of fact, GAFSP has to a great extent “pushed” M&E capacity down to the

national government level, and provided support for this explicitly in the design of the program. Supervising

agencies are to use their own systems to conduct project‐level supervision. This combined with strengthened

national government capacity of M&E systems means increased donor coordination at the country level for fund‐

level results management. As a counterbalance, each supervising entity that receives funding under GAFSP have

to report on progress and results for all activities to the GAFSP CU at the WB, which then consolidate reporting

and submit as an annual implementation results report to the GAFSP SC, with copy to the Technical Advisory

Committee.37

The potential weakness of this approach has to do with “The accuracy of all reporting is the responsibility of the

originating supervising entity, and not the GAFSP CU, whose role is consolidation and not quality control of

individual contributions.”38, which is similar to the role of the MENA Transition Fund CU. However, an off‐setting

factor is an explicit mention of M&E as a specific (therefore, presumably fundable) component of GAFSP, under

Technical Assistance, Institution‐Building, and Capacity Development.39 In addition, as mentioned above, the

evaluation function is delegated to the DIME.

Below is a summary table describing the roles and responsibilities for M&E between the GAFSP supervising

entities; CU, SC, and the DIME.

Table 5. Roles and Responsibilities for M&E, GAFSP

Partner Key Roles and Responsibilities in M&E (not exhaustive)

Supervising Entities for the Public Sector Window (Asian Development Bank (ADB), AfDB, FAO, IDB, IFAD, WB, World Food Programme (WFP)

Follow their own guidelines on project M&E during all phases of the project cycle Projects under implementation: Submits supervising entity’s own progress report (or a disclosable version) every 6 months Projects under preparation: Submits a brief progress update with updated milestone dates every 6 months Completes geographic information form Submits supervising entity’s own completion report (or a disclosable version) Submits supervising entity’s own independent evaluation report (or a disclosable version) Submits rapid IE report (if the project is not selected for an in‐depth impact

36 Idem.

37 GAFSP. (2009) Framework Document for a Global Agriculture and Food Security Program (GAFSP).

38 Idem.

39 Idem.


16

Partner Key Roles and Responsibilities in M&E (not exhaustive)

evaluation)

Supervising Entity for the Private Sector Window (IFC)

Follows its own guidelines on project M&E during all phases of the project cycle To develop M&E Plan for the Private Sector Window including any policy on IEs Submits program level annual development impact indicators based on IFC’s DOTS to CU for onward transmission to the SC

CU Prepares template TOR for rapid IEs Contracts specialized organization to geo‐code submitted geographic information and carry out mapping of GAFSP projects Contracts specialized organization to carry out in‐depth IEs Submits GAFSP annual progress report to the SC Discloses relevant M&E documents on the GAFSP website (including updating the progress of GAFSP core indicators every six months) Carry out other dissemination activities such as organize global and regional workshops and liaise with relevant organizations

SC Reviews annual progress report and provides overall guidance on project implementation

DIME Responsible for IEs

3.2.2. Strengths, Weaknesses and Opportunities for


Overall, key informants rated the GAFSP macro results framework as having many strengths:

“supports planning and approval of project proposals;

tells the story of the fund at the highest level;

provides evidence to make the business case for financing;

macro results are tied to Millennium Development Goals (MDGs) and international commitments

on food security;

incentivizes the partners to go into certain programming areas that are underserved;

provides a basis to “adjust the dial” on incoming proposals by tweaking the Call for Proposals to

ensure the kinds of proposals you require to achieve the full range of results will be produced”.40

A strength of GAFSP is the devolved nature of the M&E function from the “centre” (i.e. Fund Level). The self‐

described role of the GAFSP CU at the fund level is that of coordinator for reporting. There is therefore reason to

believe that the administration cost of the M&E function at fund level is less compared to a more centralized

approach such as that of GEF. On a US$1.2B fund, there are five people in the secretariat, with a low overhead

ratio).41

The weakness of a more devolved approach is that of the accuracy of information reported. With respect to

GAFSP, a “check and balance” against a “hands‐off” approach is to support partners’ M&E capacity by making

M&E systems development and knowledge management a fundable program component. This “push” of the M&E

capacity to the country level means that M&E systems development is sustainable as both the national

government and the ISAs are implicated in supporting it past the life of the fund. Therefore, the GAFSP approach

to M&E is also good value for money as M&E capacity development is not an administration cost (i.e. fund

“overhead”) but part of program spending. Another “check and balance” is the provision of results reporting

40 DFATD. (2014). MENA Transition Fund Results Feasibility Study: Questions and Answers for an Interview with Baastel Ltd.

41 DFATD. (2014). Op. cit.


17

standards at the project level as per the “menu” approach to fund‐level results reporting. This allows both ISAs to

continue to use their own systems but also to aggregate results at the Fund‐level.

A significant aspect for consideration by the MENA Transition Fund is the priorities of the DIME program which

aligns well with the MENA Transition Fund objectives. The four pillars of DIME are: Governance (local

development, public sector governance, and legal justice); Shared Prosperity (Finance, Private Sector, and

Agriculture and Food Security); Infrastructure (Transport, Urban Development, Energy and Mining; and Water);

Human Development; and Climate Change. In addition, the DIME website also cites the cross‐cutting theme of

Fragile and Conflict Situations as another area of work. However, and as outlined in Table 4 above, IE are costly to

implement.

In sum, with respect to what can be applied to the MENA Transition Fund, the approach where: a) the M&E

function is devolved; b) but with offsetting capacity development for M&E “pushed” to country level via project

spending; c) while developing standards (such as a “menu” of results), should be given serious consideration.

3.3. Neighbourhood Investment Facility

The Neighbourhood Investment Facility (NIF) represents the “counter‐factual” case. That is to say, it is an example

of a funding facility that does not have a macro‐level results framework. Thus, in some aspects, NIF represents the

status quo for the MENA Transition Fund.

Launched in May 2008, the NIF has been designed to finance capital‐intensive infrastructure projects in partner

countries covered by the European Neighbourhood Policy (ENP) as well as to support their private sector. For the

2007‐2013 period, the European Commission (EC) has earmarked a total amount of €767 million for the NIF, which

are complemented by direct contributions from Member States kept in a trust fund managed by the EIB. To date,

the NIF contributes €678.9 million to infrastructure and private sector projects, leveraging a total project volume

of more than €19.3 billion. Financing and implementing large infrastructure projects requires considerable

financial resources. The NIF aims to create a partnership, pooling together grant resources from the EC and the EU

Member States and using them to leverage loans from European finance institutions as well as own contribution

from the ENP partner countries.42

The organization of the NIF follows the principles of the three‐tier governance architecture agreed between the

EC and the Member States. The partner countries submit projects to the European finance institutions to seek

finance. Between them, the European finance institutions designate a Lead Finance Institution, which presents the

project to the Finance Institutions Group (FIG). The FIG is a group made up of all the eligible European finance

institutions and chaired by the EC. The Lead Finance Institutions then present the NIF grant requests to the

Operational Board for final or provisional approval. The lead administrative body is the NIF Secretariat.43

3.3.1. Overview of NIF Results Framework

The following is a discussion of the NIF results framework, referring to its logic model and theory of change as well

as the results aggregation strategy, and roles and responsibilities for M&E.


42 Idem.

43 NIF. (2013). Activity Report: 2008‐2012 Overview of Activities.


18

The NIF logic model depicted in Figure 4 below was developed retrospectively for the purposed of the mid‐term

evaluation, c0nducted in late 2013.

Figure 4. NIF Logic Model44


In the mid‐term evaluation the aggregation strategy used was that project results and objectives were analyzed

retrospectively and evaluated for “fit” against the retrospectively designed intervention logic model. The

evaluators noted in their report under the efficiency section that, “The Governance would benefit of a feed‐back

system (monitoring, evaluation and lesson learning) from projects to inform decision making and strategies.” They

also noted the following, “The establishment of a comprehensive monitoring for all projects is an appreciable

effort which would require adequate resources and management capacities. The monitoring should not be

considered a duplication of Finance Institutions’ monitoring as its focus would be on the grant component and

achievement of blending and EC objectives. Costs would be appreciable, but contained, likely less than 1% of the

value of the grant. Benefits in terms of aid effectiveness and improvement of the instruments would be significant

(…) Improved reporting should not significantly accrue the cost and the time for Financing Institutions and

partners, but may help to structure the information for an improved understanding and assessment of NIF

achievements.”45

To strengthen the process, the evaluators made the following recommendations which are noteworthy:

44 NIF. (2013). Mid‐Term Evaluation of the Neighbourhood Investment Facility under the European Neighbourhood and Partnership

Instrument (ENPI) 2007‐2013. 45 DRN‐ECDPM‐ECORYS‐PARTICIP. (2013). Mid‐Term Evaluation of the Neighbourhood Investment Facility under the European

Neighbourhood and Partnership Instrument (ENPI) 2007‐2013.


19

“Develop a result based framework for all NIF projects with harmonized impact and performances

indicators (e.g. a system of impact and performance indicators should be developed to allow data

management and analysis);

Monitor and measure NIF induced changes at project, regional and portfolio levels;

Strengthen evaluation arrangements at project level.”46


“NIF has not developed specific and harmonized procedures for monitoring the facility and has not established a

mandatory result‐based framework for following up the interventions.”47 Project monitoring is based on periodic

(annual) assessment of progress on delivery of specified project results towards achievement of project objectives.

Under the Joint management implementation modality, responsibilities for management and implementation

(and hence monitoring) are delegated by the EC to the financing institution, although the EC maintains the

“prerogative of control and verification”.48

Project monitoring and follow‐up are delegated to the Lead Finance Institution. NIF mentions its intention is to

avoid duplicating tasks already performed by a Finance Institution for which procedures have been positively

assessed (i.e. are supported by result‐based systems). Then the EC relies on external independent audits and

evaluation foreseen by the Lead Finance Institution whenever it is appropriate.

Table 6 below briefly summarizes the main roles and responsibilities of the different partners.

Table 6. Key Roles and Responsibilities for M&E ‐ NIF49

Partner Key Roles and Responsibilities in M&E

Strategic Board Oversees NIF’s activities Determines general strategic orientations

Operational Board Approve (or reject) projects submitted

FIG Discuss technical aspects of projects presented with the EC

Finance Institutions Present projects to the FIG

Beneficiaries Country’s systems are used whenever possible

3.3.2. Strengths, Weaknesses, and Opportunities for


As mentioned above, NIF represents in many ways the status quo for MENA Transition Fund. The following

strengths and weaknesses for the NIF model are thus also those which potentially apply to the MENA Transition

Fund.

In terms of value for money, there are some potential cost‐efficiencies achieved in the short term; however, in the

longer term, the lack of a fund‐level macro framework can be less efficient, as noted by the evaluators during the

mid‐term evaluation. Among the main reasons for this is that the lack of harmonized procedures or ways to

aggregate results at a macro level can hinder the ability to assess whether the fund is achieving its intended results

or not over time, and may impede the ability to take corrective actions in time or to learn from experience over

46 DRN‐ECDPM‐ECORYS‐PARTICIP. (2013). Op. cit.

47 EC. (2013). Development and Cooperation – EuropeAid. Promoting investment through the Neighbourhood Investment Facility

(NIF). [online]. 48 Idem.

49 EuropeAid. (2011). Neighbourhood Investment Facility (NIF). PPT Presentation.


20

time. Overall, the mid‐term evaluation report seems to point towards the argument that, moving forward, a

macro‐results framework is necessary to measure the impact of NIF and must be supported institutionally by

reporting coordination (“harmonization”) and an M&E system at fund‐level. This likely increases costs, but also

provide important benefits as it helps to structure the information for an improved understanding and assessment

of the achievements.

3.4. Conclusion

This section provides a brief synthesis of lessons learned relevant to the development of a Macro Framework for

the MENA Transition Fund.

The development of a Macro Framework is more often than not a sign of maturity of the fund. As was mentioned

above it is reasonable to expect that GEF would focus on outcomes and impacts given its longevity. On the other

hand GAFSP, a relatively recent fund, focuses on outputs and intermediate outcomes. NIF made a tactical decision

not to develop a fund‐level results framework at inception, as in the case of the MENA Transition Fund. However,

the authors of the mid‐term evaluation suggested that a Macro Framework be developed for NIF as a tool to track

and assess performance overall.50

With respect to the aggregation strategy for a Macro Framework, the GEF methodology of a “layered” approach

to fund‐level results framework is instructive. Essentially, there is no overarching results framework; rather there is

a series of results frameworks per work area. This provides a certain reporting flexibility, but most importantly,

allows to aggregate results from vast sectors by categorizing projects under certain “pillars”. Instructive as well is

the GEF “top down/bottom up” approach of developing fund‐level results based on an analysis of GEF projects.

There are also lessons to be learned from GAFSP strategy for results aggregation which involves requiring projects

to choose from a “menu” of outcome and intermediate results specific to the area of work, notwithstanding the

narrower sectoral focus of the GAFSP as compared to the MENA Transition Fund.

For the case studies that do have a Macro Framework (i.e. GAFSP and GEF), with respect to the governance of

M&E at fund level, the approaches are fairly similar but with different emphases. Both are similar in a sense that

results coordination function is filled by the provision of standard reporting tools and templates. The emphasis

placed on specific M&E functions is determined by the perceived role placed by fund management, regarding

M&E. GEF, compared to GAFSP, emphasizes knowledge management or, more precisely, knowledge production;

publishing several documents on M&E policy, technical issues and the results of primary research. GEF also

communicates M&E best practices through global conferences such as International Conference on Evaluation,

Environment and Sustainable Development in Alexandria, Egypt. On the other hand, GAFSP self‐described role is

less as a producer of cutting‐edge evaluation practice and more as a coordinator of fund‐level reporting and a

developer of capacity for M&E, delegating important functions to the implementing agencies and funding capacity

for M&E and knowledge management at the country level. Also, GAFSP seemingly chooses to concentrate on

intermediate results, delegating impact assessments to DIME. Importantly, GAFSP has “pushed” M&E capacity to

the country level and provided explicit support for this in program design.

50 DRN‐ECDPM‐ECORYS‐PARTICIP. (2013). Op. cit.


21

4. OPTIONS his section presents in details the following options: a) the “enhanced” status quo (i.e. with some potential

minor changes to consider that might improve on the current process); b) moving forward with developing

a Macro Framework for the Fund. This latter option entails various options: B1 ‐ Department for

International Development (DFID)’s model for Macro Framework; B2‐ Project cluster/nesting model for

aggregation of results at a macro level, and; B3 ‐ an hybrid option. These options reflect a major effort to capture

the numerous stakeholders’ expectations and to translate them into concrete options to help guide the way

forward.

4.1. Option A1 “Enhanced” Status Quo

This section presents some slight alterations to the status quo suggesting how the information collected by the

CU, through the Progress Report, could eventually provide interesting information over time at the Fund level, and

might contribute to addressing the need expressed by some partners who want to be able to report on the Fund’s

impact, without changing the way the monitoring and reporting is currently structured. The current Progress

Report that ISAs are required to complete twice a year could better contribute to aggregated reporting at the

Fund‐level over time, with some very minor adjustments.

4.1.1. Portfolio Reporting ‐ Overview

Project portfolio reporting could be a powerful tool to ensure that the Fund resources work together effectively

and deliver results in the TCs. Portfolio reporting would let SC members look holistically across the projects

supported to understand the portfolio’s strategic alignment and its potential to deliver expected results. Using the

current tools for reporting set out by the Fund, the aggregation of findings across the portfolio of projects could be

done by country, focal area/pillars, and overall project performance. Such a simple portfolio monitoring system

would allow the aggregation of some information sets, instead of looking at each project as an isolated event. This

would permit development of an overview of how projects with different objectives are contributing as a set to the

overall contribution of the MENA Transition Fund.

Focal Areas/Pillars Portfolio Reporting

Aggregation of information in this way would allow, in addition to country portfolio reporting (as shown in Figure

5), an assessment of whether the overall portfolio of projects approved consistently covers all areas of results (i.e.

pillars) set out by the Fund (exemplified in Figure 6). In addition, given the fact that projects submitted for

financing are obliged to respond to TCs’ priorities, it could also provide information about which “pillars” are

contributing most towards meeting these. This could eventually help balance the overall portfolio to most

effectively address TC priorities over time.

T


22

Figure 6

Figure 6. MENA Transition Fund Approved Portfolio by Pillars, YR X (by amount ~ Total amount) – Fictional

Data

This approach to aggregate data would not provide a clear portrait of the macro results achieved by projects

funded at an overall regional level. Nevertheless, it could prove useful for managing investments and ensuring that

thematic areas are covered in proportion to the contribution they make, thereby enabling donors and ISAs to

assess what types of projects TC value more on the basis of results achieved; thereby ensuring better targeting of

resources for enhancing impact over time in the region.

To facilitate this process moving forward, this information about pillars could be transposed under section A of the

Progress Report template as demonstrated by yellow highlights below52:

Table 7. Snapshot of Revisions to Improve the Progress Reporting Template

A. Basic Project Information

ActivityName:

Country(ies)Name: Name of Implementation SupportAgency(ies):

Total Amount Approved by theTransitionFund(US$):

Additional Funds Leveraged, if any(US$):

Steering Committee ApprovalDate:

ProjectImplementationStartDate: ProjectClosingDate:

NameofISAProjectLeader:EmailofISAProjectLeader:

Pillar(s)towhichActivityResponds(aspertheproposalapprovedbytheSteeringCommittee)

Investing in SustainableGrowth.

Enhancing EconomicGovernance

Inclusive Development andJobCreation

Competitiveness andIntegration

Primary:

Secondary:

Primary:

Secondary:

Primary:

Secondary:

Primary:

Secondary:

51 MENA Transition Fund. (2013). MENA Transition Fund Annual Report December 31, 2013.

52 Since the report was submitted, there has already been some movement along the lines suggested.

Figure 5. Percent of Funding Allocated by Country

Figure 5. Percent of Funding Allocated by Country51


23

Potential Additional Revisions to Reinforce the Process of Focal Area/Pillars Portfolio Reporting

Developing clearer results statements under each pillar could further improve the effectiveness of the results

reporting process and assist the work in term of aggregating data. This could simply be done by creating sub‐

results category. Based on the broad definitions given for each pillar, these sub‐categories of results could look like

the following:

Table 8. Examples of Sub‐Categories for MENA Transition Fund Objectives53

Pillars

Sub‐category

Pillar 1: Investing in Sustainable Growth

Pillar 2: Enhancing Economic Governance

Pillar 3: Inclusive Development and Job

Creation

Pillar 4: Competitiveness and

Integration

Innovation and Technology

Good Governance (e.g. transparency, corruption, accountability, integrity, etc.)

Youth employability Trade and Negotiation

Business, private sector development, Investment, and Competition

Finance (e.g. asset recovery, financial management, audit, banking system, etc.)

Female labor participation

Infrastructure

Energy Justice (e.g. regulatory reforms, consumer protection, etc.)

Integration of people with disability

Transport

Urban & rural planning Partnership with Civil Society (e.g. open government

Aging people (e.g. pension reform)

Private Sector Development

Then based on the sub‐category defined, it would be possible to report in the Fund Annual Report, for example, a

graph as depicted in the figure below:

53 The below is provided as an example and would have to be refined based on the current portfolio of projects approved and

the priorities defined by the partners involved in the Fund.


24

Figure 7. Snapshot of Transition Objectives Pursued by Fund Projects, 2013‐2015 – Fictional Data

This would enhance the reporting of relevant data to partners. For example, by presenting to ISAs and TCs a set of

“macro results” statements to which their project could align, these types of graphs could render an even more

precise view about how the Fund is supporting TCs in their reform processes. Another main benefit from this

refinement would be to potentially better “adjust the dial” for incoming proposals by slightly modifying and

improving the Call for Proposals to ensure the kinds of proposals required to achieve the full range of results will be

targeted. It could also help for improving understanding of future resource needs and improve alignment and

leveraging of resources. Moreover, it could enhance coordination between partners (e.g. donors, TC, ISAs, and

other actors in the region) and catalyze synergy by more clearly showing what type of projects are currently

underway/supported by the Fund. Finally, it could potentially incentivize more partners to join if they feel they

have a vested interest in the kind of support the Fund provides. Even though all of the above represent interesting

benefits for reporting in an aggregated way at the Fund level, this type of reporting would not catalyze enhanced

reporting on the contribution of the Fund in terms of development outcomes as requested by key donors.

Overall Portfolio Performance

Another aspect which could be “aggregated” easily at the Fund‐level is the overall project performance. During

project implementation, each project is required to monitor the expected achievements of objectives as follows:

Satisfactory (S): The project is likely to achieve almost all or exceed its major objectives

efficiently without any significant shortcomings.

Moderately Satisfactory (MS): The project is likely to achieve the majority of its major

objectives efficiently with moderate shortcomings.

Innova

tion and Tec

hnology

Business, P

SD, Inve

stmen

t, and Competition

Energy

Urban & rural p

lanning

Good gove

rnan

ce

Finan

ce

Justice

Partnersh

ip with Civil Society

Youth employa

bility

Fem

ale labour participation

Peo

ple with disab

ility

Aging peo

ple

Trade an

d Neg

otiation

Infrastructure

Transp

ort

PSD

60%

30%

5% 5%

25%

40%

20%15%

70%

30%

0% 0%

40%

10%

25% 25%25%

40%

20%15%

40%

10%

25% 25%

60%

30%

5% 5%

60%

30%

5% 5%

40%

10%

25% 25%

60%

30%

5% 5%

25%

40%

20%15%

25%

40%

20%15%

2013 2014 2015

Pillar 1 Pillar 2 Pillar 3 Pillar 4


25

Moderately Unsatisfactory (MU): The project is not likely to achieve at least half of its major

objectives efficiently with moderate shortcomings.

Unsatisfactory (U): The project is not likely to achieve most or any of its major objectives

efficiently.

Systematic analysis of data provided under this category of the Progress Report could enable the Fund to

continuously monitor the quality/performance of the projects under implementation (see the tables below for

examples). Based on the assumption that each project defines a Project Development Objective (PDO), which is

expected to contribute to the transition/reform process in its TC and measures progress towards achieving

relevant results against robust indicators, then it would be possible to report on the extent to which the Fund is

supporting “transformative” projects that will benefit the transition phase in TCs.

Table 9. Example of a Summary of Performance Ratings Table ‐ Fictional Data

Rating FY1 (40 projects) FY2 (50 projects) FY=n (*n project)

Progress towards achievement of objective (PDO)

S 27% 18% X%

MS 65% 74% X%

MU 5% 0% X%

U 3% 8% X%

Implementation progress

S 20% 18% X%

MS 67% 66% X%

MU 1% 16% X%

U 12% 0% X%

Table 10. Example of Performance Ratings by Country for YR X54

Country Number of Projects

Progress Towards Achievement of Objectives %

Implementation Progress %

S MS MU U S MS MU U

Egypt XX % % % % % % % %

Jordan XX % % % % % % % %

Libya XX % % % % % % % %

Morocco XX % % % % % % % %

Tunisia XX % % % % % % % %

Yemen XX % % % % % % % %

54 These tables are presented merely as suggestions and for example purposes only.


26

Table 11. Example of Performance Ratings by Pillars for YR X

Pillars Number of Projects

Progress Towards Achievement of Objectives %

Implementation Progress %

S MS MU U S MS MU U

Pillar 1 XX % % % % % % % %

Pillar 2 XX % % % % % % % %

Pillar 3 XX % % % % % % % %

Pillar 4 XX % % % % % % % %

Tables above could help to assess which thematic area (i.e. pillars) are generally more/less successful at delivering

key results or potentially facing more/less difficulty during implementation; thus, helping identify opportunities in

the future. Over time, the systematic use of this type of aggregate reporting through the Progress Reports

submitted by ISAs, would allow SC members to assess what approaches or interventions contribute most

effectively to achieving specific development objectives, and would thereby help to identify good practices for

replication.

Revisions to Reinforce the Process of Portfolio Performance Reporting

As discussed above, ratings and rating definitions are given for progress towards the achievement of objectives in

the current Progress Report. In addition to this existing information, it would be possible to add definitions for the

ratings for overall implementation progress (as shown below), which would enable the introduction of a

harmonized system for ISA reporting on this aspect. For example:

Possible Definitions for Implementation Progress Ratings

Satisfactory (S): Implementation of most components is in substantial compliance with the

original/formally revised plan except for only a few that are subject to remedial action.

Moderately Satisfactory (MS): Implementation of some components is in substantial

compliance with the original/formally revised plan with some components requiring remedial

action.

Moderately Unsatisfactory (MU): Implementation of some components is not in substantial

compliance with the original/formally revised plan with most components requiring remedial

action.

Unsatisfactory (U): Implementation of most components is not in substantial compliance with

the original/formally revised plan.

Not applicable (NA): Implementation has not yet started.

The inclusion of this information could also help pave the way for more evaluative analysis at a later stage. For

example, a comparison of the results on progress on implementation with those on progress towards

achievements could reveal areas in which the original project concepts (notably their explicit or implicit theories of

change) have or have not proved correct.


27

4.1.1. Steps Required in Moving Forward with this

“Enhanced” Status Quo Option

As mentioned, there are some small steps required in order to develop this option and these mostly require only

slight modifications to the current progress reporting system, specifically:

1. Information about pillars to which projects contribute could be included under section A of the Progress

Report template to facilitate focal areas/pillars portfolio reporting55;

2. Ratings and definitions for overall implementation progress could be added to the Progress Report

template. This would harmonize how ISAs report on this aspect and allow aggregation at the portfolio

level.

In addition, developing clearer results statements (i.e. sub‐results categories) under each pillar could further

improve the effectiveness of the results reporting process and assist in term of aggregating data. However, this

could take some time to become effective, as all partners would need to agree on a set of results sub‐categories.

As exemplified in GEF case study, given the fragility of any system based on self‐reported monitoring systems, it

could be useful to also consider how targeted mid‐term evaluations, or even formative evaluations (e.g. country,

project, sector or thematic evaluations56) might be used to refine and calibrate the monitoring‐based system. The

issue of the need to verify self‐reported data through some system of cross‐checking/quality assurance should also

be given some further thoughts.

Finally, the SC members may wish to consider building on the above by setting targets for performance. This

aspect is not developed/suggested further in this section purposefully, as given the context and stage of the Fund,

this may not be the most efficient and effective way forward.

4.2. Option B1 – Revise and Use the DFID‐based Logic Model

This variation for developing a Macro Framework involves using the logic model and theory of change developed

by DFID. The following is the logic model for the MENA Transition Fund interventions as it appears in the business

case document for UK involvement in the Fund.

Figure 8. Intervention Logic ‐ DFID Model57

55 Since the report was submitted, there has already been some movement along the lines suggested.

56 Please refer to the Glossary of terms in Annex 2 for more details on each type of evaluation mentioned.

57 DFID. (n.d.) Business Case and Intervention Summary. Middle East and North Africa (MENA) G8 Deauville Transition Fund.


28

As outlined in this DFID model, the Fund is primarily aimed at catalyzing institutional reforms that will enable

better policies and in turn drive improved economic performance. In addition, the agreed aim of the Fund is to

improve the development and implementation of government policy in the TCs, focusing on four priority thematic

areas: 1. investing in sustainable growth; 2. inclusive development and job creation; 3. enhancing economic

governance; and 4. competitiveness and integration. These thematic areas will all contribute to the outcome of

building effective policies and institutions that promote inclusive and sustainable growth as demonstrated below.

Figure 9. Theory of Change Diagram ‐ DFID Model58

To measure these different articulations of results, DFID proposes a “top‐down”” approach and has already

defined indicators and sources to track these at the Fund‐level as displayed in Figure 10 below. This model is

categorized as following a “top‐down” approach since the indicators used to track achievement of results do

not take into account the project‐level results frameworks (and indicators) already in use. Rather, the

58 Idem.


29

indicators for each results level have been defined without considering the work on the ground (i.e. the

project‐level).

Figure 10. Preliminary Logical Framework ‐ DFID Model59

Of note is that many stakeholders are skeptical that at this stage, the long‐term effect of the MENA Transition

Fund can be measured. In practice, the Fund is likely to continue to support many projects in a variety of sectors

within each thematic area; thus, the impact and outcome level of this proposed results framework might be too

general and broad to give an accurate picture of the contribution of the Fund in the region. The gap between

outputs and impacts might be too vast given the contribution of the Fund in the region and the large diversity of

projects it funds. Some have argued that the “macro” impact of the Fund should be limited to helping strengthen

59 Idem.

Indicators Sources

Annual real GDP growth in priority countries

IMF Regional Economic Outlook (projections for 2012

and 2013)

Unemployment (and female unemployment) rate

IMF Regional Update 2011; Government statistics or World Bank Development Indicators

(latest available data)

Percentage of population living on less than $2 per day (2005 prices)

World Development Indicators (latest year available)

Government Effectiveness ScoreWorld Bank Governance Indicators: Government

Effectiveness Score (annual)

Foreign direct investment (BOP, current $) total of all positive flows

World Bank, World Development Indicators

Open Budget Index Open Budget Index

Volume of funds leveraged directly as a result of technical assistance ($m)

WB Coordination Unit Six monthly report

Volume of technical assistance provided ($m) disaggregated by Implementing Support Agency (ISA)


Average percentage progress against agreed work plan (disaggregated by ISA)


Client's imprression of technical assistance effectiveness (based on Client survey, average rating on a scale of 0-10, 10 being very favourable)


80%

World Bank coordination unit

working effectively

Average time from approval to first disbursement (Number of months from approval to first disbursement) disaggregated by ISA

WB coordination unit six monthly report

Number of countries receiving at least 10% of cumulated allocated technical assistance (over achievement will be regarded as anything over 10%)


20%Percentage of allocated technical assistance being managed by Regional ISAs


IMPACT

Inclusive and sustainable growth

in transition countries

OUTCOMEEffective policies and institutions

OUTPUTS

Effective technical assistance which

meets country needs


30

capacity to deliver reforms. In order to minimize the reporting burden, current results reporting may need to focus

on intermediate outcomes such as institutional reform and capacity building. However, others consider that it

remains important to specify the long‐term results of the MENA Transition Fund using a logic model and theory of

change.

4.2.1. Steps Required In Moving Forward with this

Option

To build a Macro Framework following the model above, indicators would have to be developed and vetted by

partners. DFID stated in the business case document that it would like to work with the SC and other donors, to

see if the Macro Framework for the Fund can be strengthened by the addition of sector specific indicators that are

included in Figure 11 below.60

Figure 11. Possible Additional Indicators to be Considered for the MENA Transition Fund Pillars

Moreover, this option would entail, even if in a limited way, a bureaucratic burden on the Fund structure and

reporting that perhaps only a few stakeholders are interested in. This means that the roles and responsibilities for

M&E reporting in the Fund Operations Manual will likely need to be revised to fit the new “system”. Data collection

on the new indicators should represent only a slight additional burden since indicators are collected through

secondary sources, and provided that the approach is implemented with a recognition that higher‐level results

cannot and should not be measured frequently.

Of note also is that the lower level (i.e. output) of this framework uses the current Progress Report as a basis.

Again, given the fragility of any system based, to some extent, on self‐reported monitoring systems, it could be

useful to also consider how targeted mid‐term evaluations or even formative evaluations (e.g. country, project,

sector or thematic evaluations) might be used to refine and calibrate the monitoring‐based system. To ensure

integrity of individual project reporting, a “Quality Assessment Group” could be helpful.

60 Idem.

•No of micro loans (of which to women)

•No of SME loans

•No of procedures to legally start and operate a company

•Time required to complete each procedure (calendar days)

Investing in Sustainable Growth

•Estimated no of jobs created indirectly

•No of people trained (of which women or youth)

•Estimated no of people who have indirectly benefitted from the Transition Fund (calculated for each programme then quality assured by Coordination Unit)

Inclusive Development and Job Creation

•Public Expenditure Financial Assessment ScoresEnhancing Economic

Governance•Revenue Watch Index

•No of documents to export

•Time to export (days)

Competitiveness and Integration


31

4.3. Option B2 – Developing Project Cluster/Nesting Model

In this model derived from DFATD61, the concept is to have two sets of results frameworks for monitoring and

reporting; one at the project level and the other one at the Fund‐level. The relationship between the two follows a

“nesting” model. Briefly, nesting refers to the process of inserting/creating detailed project level results

frameworks that sit within a larger macro Fund‐level results framework. Each nested project results framework

depicts in greater detail a specific component that is part of a broader Fund‐level objective. For example, in Figure

12 below, projects are categorized based on the pillar to which they contribute, which also represent, to some

extent, the immediate outcomes in the macro Fund‐level results framework.

Figure 12. DFATD Nesting Model ‐ Linkages

These two sets of results frameworks are linked together because ever though the outcomes in the project results

frameworks vary depending on the nature of the project, they always link directly to the pillar to which they

contribute. However, the PDO are more specific and granular in terms of who, what and where than the outcomes

in the macro Fund‐level results framework as exemplified in Figure 13 below.

61 This option is adapted from documents provided by DFATD and collected during the interview phase. To look at the

originally proposed DFATD model, please refer to Annex 5.


32

Figure 13. Project Cluster/Nesting Model Linkages – Example62

The rationale for this model is that the Fund finances projects of short duration, low budget and limited scope.

This type of project may realistically only achieve their defined output/intermediate results by the end of the Fund‐

financed support (or even later). The collective impact of the Fund projects on a country or area of work, (i.e. pillar)

will take longer to manifest itself than is covered by the project results system. In order to take into account this

lag in measurement, ISAs and TCs still need to identify the PDO and the “pillar” to which their project contributes,

but cannot in the short‐term realistically measure impact results achievement. Thus, individually, each project

results framework would convey its respective results and output information against the macro Fund‐level result

framework. This way is then recognized that it is not the ISAs’ role to monitor and report on the progress towards

or on the achievement of the “pillars” or the intermediate and ultimate outcomes of the macro Fund‐level results

framework, but their project level reporting can be aggregated and categorized accordingly for this type of

reporting at the higher levels. The expected outcomes in the macro Fund‐level result framework would describe

broad changes for the entire Fund which would allow reporting on the “impact” or at least the “intermediate

outcomes” of the Fund. Reporting on the immediate outcomes would reflect the clusters/pillars of the overall Fund

as visible below in Figure 14, which represents the linkages (for pillar 2 and 3 only) between the project results

framework (through their PDO) and the Fund‐level results framework.

62 Project‐level results frameworks are based on information as found in the Project Progress Report (2013).


33

Figure 14. Linkages between Macro Fund‐level Outcomes (2&3) and PDO63

Fund‐level

Ultimate Outcome (Fund's impact):

To improve the lives of citizens in TCs; and to support the transformation currently underway in TCs

Intermediate Outcomes (Fund's Objectives)

To foster sustainable inclusive economic growth To strengthen governance and public institutions

Immediate Outcomes (Fund's Pillars)

Investing in sustainable growth/Strengthening enabling environment

Inclusive development and job creation Enhancing economic governance Competitiveness and integration

Project‐level

PDO:

Improving employability of graduates and unemployed youth through increasing access to career guidance, job search, and on the job training. (Jordan: Support to Building Active Labor Market Program)

(Morocco: Strengthening Micro‐Entrepreneurship for Disadvantaged Youth in the Informal Sector)

To improve individual employability and SME capabilities for graduates and firms participating in a job‐intensive SME support pilot scheme and to inform related policies and programs. (Yemen: Enterprise Revitalization and Employment Pilot Project)

To enhance the capacity of targeted accountability institutions to provide access to information and to improve enforcement of the anti‐corruption law. (Yemen: Accountability Enhancement Project)

Contribute to the strengthening of government transparency, accountability and public participation (Morocco: New Governance Framework Implementation Support Project)

To mobilize domestic resources to foster sustainable economic growth and income redistribution (Tunisia: Enhancing Domestic Resource Mobilization Through Effective Tax Systems)

This model is regarded as following a “bottom‐up” approach because to measure the results in the macro Fund‐level results framework, the indicators

would come from the individual project results framework.

63 Based on DFATD. (2013). CIDA‐PAD – Annex B: Logic Model for MENA Transition Fund. This figure is adapted from DFATD’s preliminary logic model posited for the Fund. To

look at the originally proposed DFATD model, please refer to the Appendix, Annex 5.


34

The main advantage of this option is to build from the “bottom‐up”; thus, allowing ISAs to continue to report their own indicators in the project results

framework and not change the way they conduct the monitoring and reporting for their project. This type of monitoring and reporting is possible through the

development of consolidated indicators.64 Consolidated indicators are project level indicators that are clustered together with other related indicators to form

one unit that will allow aggregation of related data at the macro Fund‐level. Table 12 exemplifies how this could be done for pillar 1 and 2 of the Fund.

Table 12. Linkages from Project Results Framework Outcomes to Macro Framework – Pillars 1 &2

Pillars Macro Fund‐level Consolidated Indicator65 Project level indicators66

Investing in Sustainable Growth Number of policies, laws, agreements or regulations promoting sustainable growth that are implemented as result of the MENA Transition Fund assistance

1) A comprehensive energy pricing and fuel switching strategy for Egypt is developed 2) Acceptance and adoption of the SME development strategy in Libya 3) Number of policy reforms identified in view of strengthening the legal framework for PPPs in Tunisia

Inclusive Development and Job Creation Number of people who have benefitted from the MENA Transition Fund

1) Number of stakeholders participating in training 2) Number of direct green jobs created 3) Number of women entrepreneur assisted 4) Number of youth micro‐entrepreneur who receive post‐creation follow‐up support (of which female %)

The process is also shown in Figure 15 below.

64 Of note is that DFATD developed guidance on how to select indicators at the program (i.e. fund level) and how to strategically select other indicators from the project level.

DFATD’s Global Program Branch (GPB) also has a Tip Sheet on consolidated indicators which provides guidance on the key steps required to conduct this process. DFATD’s initial model is discussed further in Annex 5. 65 The below is posited merely as an example and would have to be refined based on the current portfolio of projects approved.

66 MENA Transition Fund. (2013). MENA Transition Fund Project Progress Report.


35

Figure 15. Diagram Illustrating Different Requirement for Indicators and their Data at Different Levels67

67 Adapted from DFATD. (n.d.) Master Slide EDRMS No. 6065211. To look at the originally proposed DFATD model, please refer to the Appendix, Annex 5.

Indicators at the Ultimate and

Intermediate Outcomes to report

on the Impact of the Fund

Consolidated Indicators at the

Immediate Outcome level to

report on the results achievement

under each pillars

Project level indicators as set out in

the individual project results

framework


36


Option

This option requires the creation of project clusters/categorizations. An even simpler way would be to decide that

the “cluster” simply reflects the “pillars” defined by the Fund as each project is already required to identify a

“pillar” for which its project align in the Fund proposal form.

This option also requires the creation of appropriate consolidated indicators to measure the results at the macro

Fund‐level. This step would require extensive analysis and comparison of the projects approved to build from

these the Fund‐level consolidated indicators. DFATD’s Global Program Branch (GPB) has a Tip Sheet on

consolidated indicators which provides guidance on the key steps required to conduct this process and they are as

follow:

1. “Review portfolio of projects, by sector (multidisciplinary team);

2. Look for project level indicators that meet the following criteria:

a. Dealing with the same sector;

b. Measuring similar change, at the same level;

c. Same of comparable units of measure;

d. Same/similar units of analysis that could be grouped into a broader unit;

e. Same of similar context which speak to the expected result

3. Using the project level indicators that fulfilled the criteria in Step 2, try and group the units of analysis

from the project indicators into a single broader (generic) category;

4. Add a unit of measure and context (that is related to the expected result from your program) to your unit

of analysis and your indicator is complete.”68

The main burden of this option has to do with the “mapping” of the current project level indicators to the macro

level and do the same process for all newly approved projects to the defined consolidated indicators once these

have been defined. A temporary ad hoc technical group at Fund level could lead this task or this could be

delegated to the next consultancy (to be engaged if the Macro Framework option is selected). If it is decided to

move forward with developing a Macro Framework for the Fund, then given there are consultants coming on

stream to guide this process, it could be their task to kick‐start this process and define/posit categories and

possible consolidated indicators based on the current universe of project level indicators. While doing so, they

could map the current project level indicators to these newly created consolidated indicators. They could also

develop simple guidance material and tools that would guide this process for future project design and ongoing

reporting, which would ease the reporting burden on the consolidated indicators when the time comes to produce

the Annual Report.

Potential Additional Steps to Reinforce the Process

Eventually, it could be useful to provide in the proposal form a “menu” of results statements reflecting the

categories developed and even a “menu” of indicators for the implementers (e.g. ISAs and TCs) that they could

select from. Should this avenue be pursued, it could be made clear that these “menu items” would only be posited

as suggestions and not represent a straight‐jacket; thus keeping with the principles of flexibility and appropriation

set out by the Fund at its inception. When the same outcome is shared by more than one project, harmonizing or

standardizing the indicator within and across pillars would help to compare and aggregate results for different

68 DFATD. Coherence Division‐RBM Technical Unit. (2014). GPB Tip Sheet. DFATD Consolidated Indicators.


37

sectors, countries, and the Fund as a whole. As noted above, some guidance material to assist ISAs and TCs in this

process into the future could be developed. This could be developed by the consultants that might be engaged to

assist in the process of developing the Macro Framework itself. This would have the added benefit of addressing

the challenge of standardization of terminology use with regards to ISA reporting.

4.4. Option B3 – Reporting Against a MENA Transition Fund Theory of Change – “Hybrid” Option

This variation combines the DFID and project cluster/nesting approaches together. Both approaches focus on

results, as well as themes in the realm of governance, capacity building or institutional reform. Thus, it is possible

to construct a combined “top down/bottom up” approach, focusing on the intermediate outcomes related to

capacity building, institutional reform and/or governance.

Figure 16. Top‐down and Bottom‐up Approach

However, instead of categorizing projects as per the project cluster/nesting model, the aim is to categorize results,

as per the GEF methodology. In addition, instead of developing a categorization of project results according to an

analysis of their own activities and declared expected results, the categorization could be based on a Fund‐level

logic model and theory of change, building on that developed by DFID. Based on the previous, individual work‐

area results framework could be derived from a single logic model and theory of change for the Fund. This could

be done by combining the “top down/bottom up” methodology applied by DFID and the project cluster/nesting

model respectively.

In going this route, it needs to be well understood that the macro objectives results respond to different

information needs and imperatives than the results at the ISA‐project level. In order to reflect fully the

decentralized and diverse reality, the possibility of a single, fully integrated system in which country and project

goals cascade neatly from pre‐defined broader objectives should likely be discarded. To reconcile potentially

competing needs, an “empirical” approach could be adopted. The macro level elements of the Fund needs to be

developed based on extensive consultation and maybe even pilot testing, while the results (outcomes and

outputs) are defined by the TC and ISAs and reflect the diversity of country needs.

Thus, it is possible to construct a combined “top down/bottom up” approach, focusing on the intermediate

outcomes related to capacity building, institutional reform and/or governance.


38

Figure 17. Revised Logic Model for the MENA Transition Fund

Given the revised (albeit provisional) model, the proposed logic model and theory of change portrayed in Figure 17

can be summarized as follows.

Technical assistance from the Fund increases beneficiary countries’ institutions (specifically line

ministries) capacity for administrative reform such as: M&E, Knowledge Management,

Information Management, Public Financial Management, Policy and Planning and Grants and

Contributions Management.

More efficient line ministries affect the ability of national government to implement reforms in

the key areas of a) increased service delivery in social and economic sectors and b) governance

(anti‐corruption justice, etc...)

Such key institutional reforms lead to full participation of national country citizens in the

economic and social life of the country (i.e. economic and social inclusion), which in turn yields

sustainable (i.e. broad‐based) economic growth.

The intermediate results of two Fund projects were categorized using concepts taken from the revised, provisional

logic model and theory of change described above. The results are depicted in Table 13 below.

Table 13. Proposed Categorization of MENA Transition Fund Project Results

MENA Trust Fund Project

MENA Transition Fund Pillar Intermediate Result

Indicator

Intermediate Outcome (DFID Logic Model, revised)

Enterprise Revitalization and Employment Pilot Project (Yemen)

Primary:

Investing in Sustainable Growth

Successfully facilitate internships and business services to private enterprises

Number of firms requesting business services

Institutional Reform (Service Delivery)

Secondary:

Inclusive Development and Job Creation

Competitiveness and Integration

Number of beneficiaries completing internships

Regional Affordable Housing Project (Morocco & Tunisia)

Primary: Investing in

Sustainable Growth

Evaluation of Government Housing Policy and Programs

# of reports completed

Institutional Reform (Policy and Planning)

Secondary: Inclusive Development

and Job Creation Enhancing Economic

Governance Competitiveness and

Integration


39

As can be seen, the results of two relatively different projects, one having to do with service to firms and

employment training and the other targeting affordable housing, can both be classified using the concepts

contained in the revised DFID logic model and theory of change. This points to the possibility of developing a

Macro Framework for the Fund.

In order to avoid the risk of banality, a “layered” approach to developing a Macro Framework for Fund could be

adopted. This means that, instead of developing a single overarching results framework, a series of results

framework per pillar/work area is developed. This should provide a certain level of specificity that would address

the risk of an overly‐generalized Macro Framework. A “layered” results framework approach means flexibility of

reporting horizontally and vertically (i.e. by country and/or pillars/area of work). Focusing on results related to

governance and/or institutional reform offers another dimension of reporting, one that cross‐cuts both area of

work and country as displayed in below.

Figure 18. Layered Approach

Aggregation Strategies

In order to discuss possible aggregation strategies for a Fund Macro Framework, it is useful to make a distinction

between two types of results aggregation.

1. Count: This is defined as the categorization of results according to concepts such as administrative or

institutional reform. This aggregation method is described above in the GEF case study. Using this

strategy, it is possible to make statements like the following, “25% of MENA Transition Fund projects

used policy redesign as a tactic for institutional reform”; “30% of MENA Transition Fund Projects

enhanced M&E Systems as part of administrative reform”.

2. Sum: This involves the mathematical addition of results metrics. See an excerpt below from DFATD’s Tip

Sheet regarding the consolidation of indicators which exemplifies this methodology.69

69 DFATD. Coherence Division‐RBM Technical Unit. (2014). GPB Tip Sheet. DFATD Consolidated Indicators.


40

Referring again to Table 13 above, it is possible to employ the “Count” and the “Sum” aggregation strategy to the

hypothetical Fund Macro Framework defined in this section.

Count. Referring to the column “Intermediate Outcome” one would count the number of results

categories in this column.

Sum. Referring to the column “Indicator” and using some change in wording, it is possible to add similar

results (beneficiary counts, for example) from different projects together.


Option

As this variation is a combination of both the DFID and project cluster/nesting approach, steps in moving forward

with this option are similar to those described above. To recap, this option entails:

1. Refine/elaborate logic model and theory of change;

2. Categorization of current Fund project results, using the new logic model and theory of change to provide

“bottom‐up” validation;

3. Define/build consensus on indicators for each level of results in the logic model and theory of change;

4. Develop pillar/area of work results frameworks;

5. Revise the M&E reporting structure/arrangements to fit the new “system”;

6. Consider how audits and evaluations might be used to refine and calibrate the monitoring‐based system.

As alluded to above, the DFID logic model and theory of change, while a good start, requires further elaboration.

Some of the relationships between results need to be further specified. For example, examining DFID logic model,

there is an implied but under‐specified relationship between social/economic inclusion and sustainable economic

growth. The demonstration model attempts to give more specificity to the relationship between social/economic

inclusion and sustainable growth. Also, the demonstration logic model makes a distinction between

administrative and institutional reform, positing that administrative reform is a precondition of key institutional

EXAMPLE– How an Existing Consolidated Indicator Might Be “Unpacked” or Reflected at Project Level

Consolidated Indicator: #/total of birth attendants trained in emergency obstetrical care (F/M).

A program may have three different projects, each with a different indicator related to the training of midwives:

Project A: #/total of midwives trained in the management of common complications during

delivery in country X (f/m) Project B: #/total of midwives that received refresher training on how to recognize signs of more

severe complications in country Y (f/m) Project C: #/total of nurse practitioners trained in emergency obstetrics (f/m)

In this case, for the purposes of rolling up data against a common agency standard, DFATD created

the consolidated program‐level indicator: #/total of birth attendants trained in emergency obstetrical care (F/M).


41

reform such as governance improvement or service to the public. These hypothesized relationships between

expected results must be first rooted in the current literature and the practice of donors and ISAs70.

Once this is done, the categorization of current Fund project results, using the new logic model and theory of

change, is carried out, providing “bottom‐up” validation. In addition, as mentioned earlier, consensus results

indicators will need to be developed for the logic model. These will form the basis of results framework to be

developed per area of work, i.e. pillar.

In order to reduce reporting burden, several implementation strategies can be employed. To posit two examples:

1. Choice: Projects are given a choice of common, fund‐level indicators from which to choose and against

which they must report (i.e. a menu);

2. Prioritization: Careful selection of which indicators projects have to report on frequently and which,

because of their slow‐changing nature (i.e. intermediate outcomes or impacts), will only need to be

reported on at the end of the project. In the case of slow‐changing indicators, only require subjective

evaluations (Satisfactory/Unsatisfactory) of progress; Reserve measurement of slow‐changing, higher

level results (such as catalytic effects) at fund‐level for evaluations;

In addition to the above, the “hybrid approach”, to some degree, relies on subjective evaluations of results

progress at project level and therefore project‐level audits should likely be conducted to verify the subjective

evaluations. Furthermore, as was noted, slower‐changing fund‐level results at outcome and impact level should

likely be measured less frequently, as there is a lag in effect. A related issue is the use of evaluation at project level.

A RoTI approach71 as per the GEF, focusing on key issues, can provide some indications of impacts at fund‐level.

4.5. Others

Possible interesting resources to consult, if decided to move forward in developing a Macro Framework for the

Fund, are posited below:

The WB’s Approach to Public Sector Management (2011‐2020), which is interested in how to

support delivery of reforms;

High Level Forum on Aid Effectiveness in Busan (2011) and the Cairo Consensus on Capacity

Building (2011), which set internationally agreed frameworks for strengthening capacity

development through technical assistance which emphasizes the importance of country

ownership, transparency and knowledge transfer.

Other case studies: Central Asia Regional Economic Cooperation (CAREC) Program72, EU

Platform on Blending.

70 It is recognized that this represents a significant step and one which may not find extensive support.

71 Please refer to GEF EO. (2009). The RoTI Handbook: Towards enhancing the impacts of environmental projects, Methodological

Paper #2. 72 Please refer to http://www.carecprogram.org/index.php?page=carec‐results‐framework to see CAREC’s three levels Results

Framework.


42

ANNEX 1. METHODOLOGY

This independent Feasibility Assessment was carried out from Mid‐February to April 2014. The following section

describes the assessment questions, approach and methods, and examines some constraints and limitations.

The approach and methodology presented in this Annex are also of utmost importance to clarify and determine

the scope of the mandate. These have been presented and discussed during early stages with the CU’s Program

Manager.

Approach & Questions

In relation to the objectives mentioned in the main report, the following aspects were covered: the current project‐

specific results framework; the motivation and challenges for establishing a Macro Framework for the Fund; a

review of best practices in funds of similar nature, and; the main options and key next steps in establishing a Macro

Framework for the Fund.

A general introduction to each of these and some sample core questions are presented below:

The current project‐specific results framework: An assessment and mapping of the status quo,

in order to identify the strengths and weaknesses of the processes associated with the

monitoring and reporting function of the Fund;

o What are the results that are currently targeted and measured through the Fund?

o How and when is data collected by the CU? By individual ISAs?

o What is the capacity of implementing partners to conduct M&E of their individual

projects, data gathering information technology (IT) systems in place (or not), etc.?

Motivation and challenges: An assessment of the extent to which a Macro Framework could

meet the objectives set forth by the stakeholders working with the Fund and would be consistent

with requirements, needs, priorities, and partners’ and donors’ policies. This refers more

specifically to the opportunities, challenges, and risks.

o What do stakeholders involved in the Fund see as possible benefits and challenges for

establishing a Macro Framework? What do stakeholders involved in the Fund suggest

about the possible motivations for establishing a Macro Framework? What would be the

value (or lack thereof) for it?

o What are (or could be) the strengths of having a Macro Framework when managing a

fund?

o What are (or could be) the weaknesses of having a Macro Framework when managing a

fund?

Review of best practices in funds of similar nature: A brief study of the options, challenges and

risks based on the experience of funds of similar nature.

o How do these funds operate in terms of monitoring and reporting? What are the M&E

systems in use?

o Where evidence exists, how effective is their framework? What are the challenges?

o How could each option be used in the context of the MENA Transition Fund (i.e. what

can we learn)?


43

Based on answers to the previous questions and the assessment made, the recommendations – which will be more

fully exposed in the final report ‐ will cover the following aspects:

What recommendations can be made on the way forward? Where are the opportunities and

where is the resistance? What are the pitfalls and what are the key success factors in the way

forward?

What options is there for the MENA Transition Fund?

What are the main steps required for creating a Macro Framework for the MENA Transition

Fund?

Matrix

The main element guiding the Feasibility Assessment approach and methodology is the matrix, which is presented

below at the end of this Annex. The matrix links overall questions to more detailed sub‐questions distilled from the

TORs, which would be measured through the different data gathering and analysis methods and from varying

data sources, as presented in the following sub‐sections.

Methods

The two main data gathering and analysis method, or lines of inquiry for the Feasibility Assessment were

document review and interviews with key stakeholders. In very rare cases, surveys were sent to stakeholders to

replace the interview and to accommodate informants. Everyone involve with the Fund was contacted: TCs,

donors, and ISAs, the CU. Key focal points in each country and at each ISA were provided by the CU and the

consultant reached to all of them. However, not all of them responded.

Document Review

The document review was the first step in the mandate and allowed the consultant to prepare for interviews.

Further document were also gathered from the interviews. A complete list of document reviewed by consultant

can be found in Annex 3.

Interviews

Significant time has been spent in conducting interviews with members of the SC. These included representatives

of TCs, donors, and ISAs. Stakeholders were contacted mostly through telephone/Skype interviews. A complete

list of persons interviewed during the Feasibility Assessment can be found in Annex 4.

Key Deliverables and Calendar of Activities

Deliverables/Key Activities Date of Submission

Kick off Conference Call February 12, 2014

Interview with Key Informants February 24 to March 28, 2014

First Draft Report The week of April 7, 2014

WB/CU Comments on 1st Draft Report By April 14, 2014

Second Draft Report By April 25, 2014

WB/CU Comments on 2nd Draft Report By April 28, 2014

Final Report By May 1, 2014


44

EVALUATION QUESTIONS AND SUB-QUESTIONS INDICATORS DATA COLLECTION METHODS

SOURCES OF INFORMATION

1. Management/Operations Overview What are the results that are currently targeted and measured through the MENA Transition Fund?

Document Review

MENA Transition Fund Operation Manual Progress Reports submitted ISAs policies and procedures (Financial Procedures Agreements (FPAs) signed with recipient entities)

Interviews Recipient entities ISAs Coordination Unit Trustee Steering Committee

2. Efficiency: How effective is the current project-oriented results framework of the Fund?

Costs/Resource Allocation Demonstrating Results/Impact Simplicity/Users Satisfaction

2.1. Efficiency of Resource Allocation and Utilization How and when (frequency) is data collected? Who is involved (i.e. how many persons/for how many days)?

Interviews Recipient entities ISAs Coordination Unit Trustee

Do you consider that the project have access to appropriate financial resources to report and conduct M&E appropriately/according to the agreements signed/set forth in the MENA Fund Operation Manual? What is the ratio of resources invested in M&E to date versus those planned?


2.2. Efficiency in Demonstrating Results How are indicators defined, and does this vary between ISA? Do definitions vary by country?

Interviews ISAs

To what extent do the current annual/progress reports provide a picture of results/progress at the macro level? What do these annual/progress reports say about the MENA Transition Fund impact? What could the annual/progress reports have said about the MENA Transition Fund impact? How can these documents be revised /enhanced to better allow for capturing effects at a more macro level?

Document Review MENA Transition Fund Annual Report Interviews Coordination Unit

Steering Committee

2.3. Simplicity/User-friendliness Do the modalities and M&E procedures constitute challenges? Interviews Recipient entities

ISAs Are the definitions of roles in terms of M&E and management clear? How do individual ISAs and transition countries use the results data in their respective work programs and strategies? How do the results influence policy decisions/actions?

Interviews Recipient entities ISAs

Would you have any recommendation to improve the efficiency of the M&E process (both in terms of costs and utilisation of findings generated)?

3. Relevance What do stakeholders involved in the MENA transition Fund suggest about the possible motivations for establishing a macro/results framework? What would be

Interviews Recipient entities ISAs


45

the value (or lack thereof) for it? Coordination Unit Trustee Steering Committee

What do stakeholders involved in the MENA transition Fund see as possible challenges for establishing a macro results framework?


Would you make any recommendation concerning the future or opportunity to pursue or not the establishment of such macro-aggregate results framework for the MENA Transition Fund?


What are the strengths (general, based on the experience of other similar Funds) of having a macro results framework when managing a Fund?

Document Review Studies and Assessments of M&E frameworks and the performance and governance of international funds

Interviews/Surveys Funds targeted for Case Study What are the weaknesses (general, based on the experience of other similar Funds) of having a macro results framework when managing a Fund? Challenges, risks, costs


Interviews/Surveys Funds targeted for Case Study Is there value for money of such a macro-aggregated results framework based on the experience of other similar Funds?

Value added v.s. Costs Extent to which Funds of similar nature uses information generated for decision-making


Interviews/Surveys Funds targeted for Case Study 4. Recommendations Opportunities

Threats Resources implications Possible options Next Steps

4.1. What are the prospects for success in establishing a macro results framework for the MENA Transition Fund?

Existing capacity & resistance Costs

To what extent is there sufficient ownership of the process to sustain the implementation of the macro results framework?

Information collected through interviews about “relevance”

Generally speaking, has the Fund planned enough financial resources to cover recurring costs of this macro results framework?

Interview Coordination Unit Steering Committee Trustee

4. What recommendations can be made on the way forward regarding structuring and implementing a macro-aggregate results framework?

IT system requirements Main actions required/Next Steps

What capability exists to set up cost-efficient IT-systems? What do other Funds of similar nature use as IT-system? With what costs?

Interview Funds targeted for Case Study WB IT staff

Document Review Similar Funds website What are the main steps required for creating a macro/aggregate results framework for the MENA Transition Fund?

Document Review Funds targeted for Case Study Interviews RBM & M&E experts

What are the pitfalls in the development of a macro/aggregate results framework for the MENA Transition Fund? What are the key success factors?


46

ANNEX 2. GLOSSARY73

Term Definition

Activity Actions taken or work performed through which inputs, such as funds, technical assistance and other types of resources are mobilized to produce specific products and services

Activity indicator Measure what happens during implementation. Often they are expressed as a set of completion or milestone events taken from an activity plan, and may measure the time and/or cost required to complete them (examples: date by which building site is completed, cost of developing textbooks)

Beneficiary The individuals, groups, or organizations, whether targeted or not, that benefit, directly or indirectly, from the development intervention.

Country evaluation Evaluation of one or more donor’s or agency’s portfolio of development interventions, and the assistance strategy behind them, in a partner country.

Evaluation The systematic and objective assessment of an on‐going or completed project, program or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision–making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or program. An assessment, as systematic and objective as possible, of a planned, on‐going, or completed development intervention.

Impact Positive and negative, primary and secondary long‐term effects produced by a development intervention, directly or indirectly, intended or unintended.

Impact indicator Measure the longer‐term and more widespread development changes in the society, economy or environment to which the project contributes. Often these are captured via national sector or subsector statistics (examples: reductions in percent of the population living below the poverty line, declines in infant mortality rates, reductions in urban pollution emission rates)

Indicator A variable that allows the verification of changes in the development intervention or shows results relative to what was planned

Input The financial, human, and material resources used for the development intervention

Input indicator Measure quantities of physical, human or financial resources provided to the project, often expressed in dollar amounts or amounts of employee time (examples: number of machines procured, number of staff‐months of technical assistance provided, levels of financial contributions from the government or co financiers)

Lessons learned Generalizations based on evaluation experiences with projects, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact.

Logical Framework Management tool used to improve the design of interventions, most often at the project level. It involves identifying strategic elements (inputs, products and services, expected results, impact) and their causal relationships, indicators, and the assumptions or risks that may influence success and failure. It thus facilitates planning, execution and evaluation of a development intervention.

Milestone A significant point or event in the project

73 OECD. (2002). Glossary of Key Terms in Evaluation and Results Based Management.


47

Monitoring A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.

Outcome The likely or achieved short‐term and medium‐term effects of an intervention’s products and services. The post‐intervention state of the target group or the social conditions that an intervention is expected to have changed

Outcome indicator Measure relatively direct and short‐to‐medium term effects of project products and services on intermediary organizations or on the project beneficiaries (clients, customers) ‐‐ such as the initial changes in their skills, attitudes, practices or behaviors. Often measures of the clients’ preferences and satisfaction with product/service quality are also considered as expected results (example: percent of clients satisfied with quality of health clinic services)

Output The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of expected results.

Output indicator Track the most immediate results of the project ‐‐ that is, the physical quantities of goods produced or services delivered (examples: kilometers of highway completed, number of classrooms built)

Program A development program is a time bound intervention involving multiple activities that may cut across sectors, themes and/or geographic areas

Project evaluation Evaluation of an individual development intervention designed to achieve specific objectives within specified resources and implementation schedules, often within the framework of a broader program.

Results The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention

Results chain The causal sequence for a development intervention that stipulates the necessary sequence to achieve desired objectives beginning with inputs, moving through activities and products and services, and culminating in expected results, impacts, and feedback.

Sector evaluation Evaluation of a cluster of development interventions in a sector within one country or across countries, all of which contribute to the achievement of a specific development goal.

Stakeholder Agencies, organizations, groups or individuals who have a direct or indirect interest in the development intervention or its evaluation.

Thematic evaluation Evaluation of a selection of development interventions, all of which address a specific development priority that cuts across countries, regions, and sectors.


48

ANNEX 3. LIST OF DOCUMENTS REVIEWED

General Documentation about the MENA Transition Fund and the DP

(2011). Deauville Partnership. IFI Statement.

(2013). MENA Transition Fund Fourth Steering Committee Meeting December 4 ‐ 5, 2013 (Brussels, Belgium). Meeting

Minutes.

DFID. (n.d.) Business Case and Intervention Summary. Middle East and North Africa (MENA) G8 Deauville Transition

Fund.

G8 Research Group. (2013). 2012 G8 Camp David Final Compliance Report.

G8 UK. (2013). The Deauville Partnership with Arab Countries in Transition open economies, inclusive

growth.[https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/183533/DP_pamphlet_FS

__SOS___CH_signoff.pdf].

G8 UK. (2014). The Deauville Partnership with Arab Countries in Transition. Progress Report 2013.

IMF. (2013). Arab Countries in Transition: Economic Outlook and Key Challenges. Deauville Partnership Ministerial

Meeting April 19, 2013 Washington DC.

MENA Transition Fund. (2013). Terms of Reference. Feasibility Assessment on Creating a Macro Results Framework

for the Deauville Partnership’s Transition Fund.

MENA Transition Fund. (2013). MENA Transition Fund Operations Manual.

MENA Transition Fund. CU. (2013). MENA Transition Fund Project Progress Report.

MENA Transition Fund. CU. (2013). MENA Transition Fund Annual Report December 31. 2013.

[https://www.menatransitionfund.org/sites/mena_trans_fund/files/documents/MENA%20Transition%20Fund%20

Annual%20Report%20December%202013%20FINAL.pdf].

MENA Transition Fund. Trustee. (2013). Financial Report.

MENA‐OECD Investment Programme. (2013). Challenges for Investment, Growth and Job Creation in the MENA

Region: Consequences for the MENA‐OECD Investment Programme. PPT.

Documents Collected through Interviews

DFATD. (n.d.) Master Slide EDRMS No. 6065211.

DFATD. Coherence Division‐RBM Technical Unit. (2014). GPB Tip Sheet. DFATD Consolidated Indicators.

DFATD. (2014). MENA Transition Fund: a proposal for the clustering of projects. Unpublished.

DFATD. (2014). CIDA PAD – Annex B: Logic Model for Mena Transition Fund. Unpublished.

Ministry of Foreign Affairs. DANIDA. TAS Department. (2013). Danish Arab Partnership Programme. MENA

Transition Fund. Desk Appraisal Report.


49

Documents Related to the ISAs

AfDB.(n.d.). MOPAN Management Response (Draft 23 November).

AfDB. (2008). Multilateral Development Banks’ Common Performance Assessment System 2007 Report.

AfDB. (2011). Monitoring and evaluation frameworks and the performance and governance of international funds.

AfDB. Quality Assurance and Results Department (ORQR). (n.d.). AfDB’s Action Plan for Quality and Results:

Recent Progress Communicating for and about Results. PPT.

AfDB. Quality Assurance and Results Department (ORQR). (2010). Bank Group Results Measurement Framework.

AfDB. Quality Assurance and Results Department (ORQR). (2012). Staff Guidance on Project Completion Reporting

and Rating.

AfDB. North Africa Regional Department (ORNA) Department. (2012). The Bank Group’s Participation in the Middle

East And North Africa Transition Fund.

BESLEY, T., DEWATRIPONT, M. & GURIEV, S. (n.d.). Transition and transition impact. A review of the concept and

implications for the EBRD.

BINNENDIJK, A.(2000). Results Based Management in the Development Co‐Operation Agencies: A Review of

Experience. Background Report.

DFID. (2011). Multilateral Aid Review: Assessment of International Finance Corporation (IFC).

DFID. (2013). Multilateral Aid Review ‐ Update 2013 progress rating: European Bank for Reconstruction and

Development (EBRD).

EBRD. Office of the Chief Economist. (2009). Transition Impact Retrospective: Results of EBRD Operations 2005 to

2009.

EIB. (2013). Measuring development results. The EIB Results Measurement (ReM) framework.

EIB. (2012). Report on results of EIB operations outside the EU.

IFC. (n.d.). DOTS Indicator Framework.

IFC. (2012). IFC ROAD MAP FY13‐15. Creating Innovative Solutions in Challenging Times.

Multilateral Organisation Performance Assessment Network (MOPAN) Secretariat. (2012). Performance

Assessment of Organisational Effectiveness and Reporting on Development Results. African Development Bank

(AfDB). Volume I.

Multilateral Organisation Performance Assessment Network (MOPAN) Secretariat. (2012). Performance

Assessment of Organisational Effectiveness and Reporting on Development Results. World Bank. Volume I.

World Bank. (n.d.). The World Bank Operations Manual.

Documents Reviewed for Case Studies

De Crombrugghe, D., et al. (2009). Peer Review: the Evaluation Function of the Global Environment Facility (GEF),

Final Report.


50

DFATD. (2014). MENA Transition Fund Results Feasibility Study: Questions and Answers for an Interview with Baastel

Ltd. Unpublished.

EuropeAid. (2011). Neighbourhood Investment Facility (NIF). PPT Presentation.

GAFSP. (2009) Framework Document for a Global Agriculture and Food Security Program (GAFSP).

GASFP. (2011). Fact Sheet: GAFSP Public Sector Window.

GAFSP. (2011). Monitoring and Evaluation Plan.

GAFSP. (2013). GAFSP Annual Report.

GEF. (2005). Project Implementation Review 2004 – Overview Report – World Bank.

[http://www.thegef.org/gef/sites/thegef.org/files/documents/GEF.C.25.ME_.Inf4_.pdf].

GEF. (2010). Focal Area Framework and LDCF/SCCF Framework. Excerpts from GEF‐5 Programming Document at the

Sixth Meeting for the Fifth Replenishment of the GEF Trust Fund, April 7, 2010.

GEF. (2010). M&E Policy.

GEF. (2011). GEF‐5 Tracking Tool for Biodiversity Focal Area.

GEF. (2012). Annual Performance Report: 2012.

GEF EO. (2007). Final Report on a Proposed Approach to GEF Impact Evaluation. Impact Evaluation Information

Document No. 2.

GEF EO. (2009). The RoTI Handbook: Towards enhancing the impacts of environmental projects, Methodological

Paper #2.

GEF EO. (2012). Annual Thematic Evaluations Report (2012).

IDB. (2012). Project Implementation Report (PIR) FY2012.

Le Groupe‐conseil baastel ltée. (2005). Overview of Monitoring Approaches in the GEF Family.

Le Groupe‐conseil baastel ltée. (2008). Catalogue of Results‐Based Management Practices of Key Organizations.

NIF. (2013). Activity Report: 2008‐2012 Overview of Activities.

NIF. (2013). Mid‐Term Evaluation of the Neighbourhood Investment Facility under the European Neighbourhood and

Partnership Instrument (ENPI) 2007‐2013.

Pérez del Castillo, Carlos (GEF). (2009) OPS4 Technical Document # 5: Governance of the GEF.

Vivideconomics. (2011). Monitoring and evaluation frameworks and the performance and governance of international

funds.

VOLONTE, C. (GEF EO). (2011). The GEF M&E Policy. PPT Presentation.

Sitography

EBRD. (2010). Measuring transition impact. [http://www.ebrd.com/pages/research/analysis/impact.shtml].

EC. (2013). Development and Cooperation – EuropeAid. Promoting investment through the Neighbourhood

Investment Facility (NIF). [http://ec.europa.eu/europeaid/where/neighbourhood/regional‐

cooperation/irc/investment_en.htm].


51

GASFP. (2011). GAFSP Website. [http://www.gafspfund.org/].

GEF. (2013). GEF Website. Investing in our Planet. [http://www.thegef.org/gef/].

GEF EO. (2013). Independent Evaluation Office Website. [http://www.thegef.org/gef/eo_office].

IFC. (2014). IFC Development Results and Impact.

[http://www.ifc.org/wps/wcm/connect/Topics_Ext_Content/IFC_External_Corporate_Site/IDG_Home/Monitoring_

Tracking_Results/DOTS_Investment/].

MENA Transition Fund. (2013). MENA Transition Fund Website. [https://www.menatransitionfund.org/]

The White House. Office of the Press Secretary. (2012). Fact Sheet: G‐8 Action on the Deauville Partnership with

Arab Countries in Transition. [http://www.whitehouse.gov/the‐press‐office/2012/05/19/fact‐sheet‐g‐8‐action‐

deauville‐partnership‐arab‐countries‐transition].

US Department of State. (n.d.). Deauville Partnership with Arab Countries in Transition.

[http://www.state.gov/e/eb/ecosum/2012g8/deauville/].

WB. (2011). Financial Intermediary Funds. [http://fiftrustee.worldbank.org/index.php?type=page&ft=fifoverview].

Newspaper

EBRD. (2014). EBRD to lead the IFI Coordination Platform of the Deauville Partnership in 2014.

[http://www.ebrd.com/pages/news/press/2014/140106.shtml].

European Commission. (2012). EU launches new Platform for blending funds in external cooperation.

[http://ec.europa.eu/europeaid/news/2012‐12‐12‐platform‐blending‐funds_en.htm].

G8 Information Centre. (2012). Fact Sheet: G‐8 Action on the Deauville Partnership with Arab Countries in Transition.

[http://www.g8.utoronto.ca/summit/2012campdavid/g8‐transition‐factsheet.html]

Others

(2013). Memorandum regarding IFIs Harmonized Development Results Indicators for Private Sector Investment

Operations.

DFID. (2011). Briefing Note Indicators and VFM in Governance Programming.

EuropeAid. DG DevCo. (2013).Results Study.

OECD. (2002). Glossary of Key Terms in Evaluation and Results‐Based Management.

[http://www.oecd.org/dac/evaluation/2754804.pdf].

ZALL KUSEK, J. & R. C. RIST. (2004). Ten Steps to a Results‐Based Monitoring and Evaluation System. A Handbook

for Development Practitioners.


52

ANNEX 4. LIST OF PERSONS

MET/INTERVIEWED

Donors

Country Organization Name Position

Canada DFATD

Gina Watson Deauville Partnership Coordinator

Yvetta Pass Manager, Results‐Based Management Technical Unit

François‐Philippe Dubé Analyst, Policies and Planning, Europe and Middle East Bureau Development

Andrew Spezowka Director, a.j. Food Security Division, Global Issues and Development Branch

Denmark Ministry of Foreign Affairs of Denmark

Pernille Mortensen Head of Section, Department For Middle East And North Africa

Jørgen Georg Jensen Advisor

France

Ministry of Finance and Industry

Cécile Pot Deputy

Ministry of Economy and Finance

Stephane Tabarie Civil Administrator, Treasury Directorate General

Kuwait Ministry of Finance Ishaq Abdulkarim Director, International Economic Co‐operation Department

Saudi Arabia Saudi Fund for Development

Fahad A. Alnowaiser Director General of Export Credit Dept

Turkey Undersecretariat of Treasury

Zafer Düzenli Senior Treasury Expert

DG for Foreign Economic Relations, Multilateral Development Banks and Economic Co‐operation Department

UAE Ministry of Finance H.E.Younis Alkhoori Undersecretary

UK DFID

Helen Winterton Head, Middle East and North Africa Department

Tim Williams Senior Governance Adviser, Middle East


53


and North Africa Department

US US Treasury and International Affairs

Andrew Baukol Deputy Assistant Secretary for Africa and Middle East

Mary Svenstrup Deputy Director, Middle East Office

Elizabeth Shortino Director for Middle East North Africa

ISAs

Organization Name Position

AfDB Jabob Kolster Regional Director for North Africa

EBRD

Adrian Keler Principal Manager, Donor Co‐Financing

Frederic Lead, Transition Team

Akhil Patel RBM Expert

Alan Rousso Managing Director for External Action and Political Affairs

EIB Julien Serre Policy and Strategy Advisor

IFC Luke Haggarty Regional Head, IFC Advisory Services, Middle East and North Africa

IMF Mark Horton Assistant Director, IMF Middle East and Central Asia Department

IsDB Amadou Thierno Diallo

Director, Office of the Vice President Operations

OECD Andreas Schaal Head of Sherpa Office

Nejla Saula Legal Adviser at the OECD Sherpa Office

World Bank (WB)

Preeti S. Ahuja Manager, Development Effectiveness, MENA region

Fareeba Mahmood Senior Operations Officer (MNADE)

Franck Bousquet Coordinator, MENA region

Victoria Gyllerup Operations Officer, Development Effectiveness, MENA region


54

TCs


Jordan

Ministry of Planning and International Cooperation

Saleh Al‐Kharasbeh Secretary General

Zeina Toukan Director of International Cooperation

Morocco Ministry of Economy and Finance

Abderrazak Yassir Director, Treasury and External Finance

Tunisia Ministry of Development and International Cooperation

Kalthoum Hamzaoui General Director of Multilateral Cooperation

Yemen Ministry of Planning & International Cooperation

Mohammed AlHawri Deputy Minister

CU

Organization Name Position

World Bank

Hayat Al‐Harazi Program Officer, MENA

Yogita Mumssen Program Manager, MENA

Jonathan Walters Director, Regional Programs & Partnership, MENA

Pauline Zwaans Operations Officer, AES


55

ANNEX 5. ADDENDUM

The following represents DFATD’s clarification on its initial conception of a Project Cluster/Nesting Model.

Parallel Nesting Model

The DFATD model is a parallel nesting model. The draft results feasibility assessment for an overall results

framework document portrays this model as if it were a linear model where projects feed upward to the macro

results framework (refer to figure 7 in the main report), whereas in the DFATD nesting model, project results

frameworks relate to the macro results framework in parallel as follows:

The ultimate outcome (impact) of the project would be a more granular expression of the ultimate outcome

(impact) of the fund. In other words, the meaning of the ultimate outcome does not change. At the project level

and fund level, we are talking about a change in state for beneficiaries, ie., improving the life of people. Except

that at the project level, the ultimate outcome describes an improvement of the life of the specific project

beneficiaries (improved life of the people in the geographic location of the project), where as a the fund level, the

ultimate outcome describes improvement of beneficiaries of a portfolio of projects in the same sectoral or

thematic area (improved lives of people in five geographic locations).

The intermediate outcome (outcome) of the project would be a more granular expression of the intermediate

outcome (outcome) of the fund. In other words, the meaning of the intermediate outcome (outcome) does not

change. At the project level and fund level, we are talking about changes in performance, behavior, practice or

even access. Except that at the project level, the intermediate outcome describes an improvement in the

performance, practice, and behavior of the intermediaries (e.g., ministry, etc.) directly associated with a specific

project. Whereas at the fund level, the intermediate outcome describes broader changes in performance,

behavior, and practice for intermediaries of a portfolio of project in the same sectoral or thematic area.

The output of the project would be a more granular expression of the output in the fund. In other words, the

meaning of the outputs does not change. At the project level and fund level, we are talking about products and

services, for example technical assistance provided. Except that at the project level you would specify what type

of technical assistance.

Figure 19. Nesting Model Linkages ‐ DFATD's Model


56

Indicators

The report only focuses on the consolidated indicator idea from our tip sheet on how to select indicators at the

program level. However, our criteria for selecting indicators at the program (fund level) includes guidance on how

to strategically selected other indicators from the project level. This includes identifying common indicators, that

are being used to develop standards, such as a menu of commonly used indicators. Usually if you have several

projects trying to achieve similar changes in different countries, you will find one or two indicators that are more

or less the same. In the case of consolidated indicators, please note that there are no consolidated indicators at

the project level. Consolidated indicators are created at the fund level. They are based on project level indicators.

In addition the pyramid diagram on consolidated indicators (page 36 of the Appendix) that came from our input

misrepresents the concept. In referring to a consolidated indicator at the meso level, a link must be created

between a consolidated indicator at the meso level to several (for example, 2‐3 indicators) at the project level.

However, there are other issues with the pyramid as presented, such as the mixing of levels, treating the projects

as if they were leading to an immediate outcome at the meso level, which is not the case. In other words every

indicator in the pyramid is at the same outcome level, for example, intermediate outcomes at project (micro),

program (meso) and corporate level (macro). Since in the context of the fund, they only have two layers ‐ micro

and macro, it would be a two layered pyramid. The diagram in the report is adapted from the following diagram:

Figure 20. Diagram Illustrating Different Requirement for Indicators & their Data at Different Levels (Simplified)74

The DFATD parallel nesting model requires a full indicator selection menu to be effective. This is explained in a

document that we provided to the consultant as way of explaining the DFATD nesting model in its entirety.

74 DFATD. Master Slide EDRMS No. 6065211. Adapted from diagram found in: http://www.qp.org.nz/monitoring/monitor‐

tools.php.

Founded in 1989, Baastel’s mission is to provide decision‐makers, managers and implementation partners with the knowledge, tools and skills necessary to promote effective and efficient sustainable development. Baastel has earned its reputation as a consultancy

firm fully committed to providing sound advice to development partners on how to strengthen the impacts of their policy and development efforts around the world

North American Office 85 Victoria Street Gatineau QC J8X 2A3 CANADA Tel: + 1 819 595‐1421 Fax: + 1 819 595‐8586 European Office Le Groupe‐conseil baastel sprl Rue de la Croix de fer 23, 1° B‐1000 Brussels BELGIUM Tel: + 32 (0)2 893 0032 Fax: + 32 (0)2 503 3183

www.baastel.com