dhaka revpres160814

26
Preliminary Findings and Conclusions of the Mid-term Project Evaluation by Christian Bugnion, Subur Consulting S.L., Team Leader Dr. Nizam Al-Husainy, National consultant www.suburconsulting.es Dhaka, 16 August 2014 AID EFFECTIVENESS PROJECT Mid-term Evaluation Preliminary Findings Implemented by the ERD, MoF With technical and financial support by UNDP and co-funded by DFID, DANIDA, Australia

Upload: drnizam-al-hussainy

Post on 19-Aug-2015

41 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Dhaka revpres160814

Preliminary Findings and Conclusions of the Mid-term Project Evaluation byChristian Bugnion, Subur Consulting S.L., Team LeaderDr. Nizam Al-Husainy, National consultantwww.suburconsulting.es Dhaka, 16 August 2014

AID EFFECTIVENESS PROJECTMid-term Evaluation Preliminary Findings

Implemented by the ERD, MoF

With technical and financial support by UNDP and co-funded by DFID, DANIDA, Australia

Page 2: Dhaka revpres160814

• Feedback and accountability to stakeholders: Findings, conclusions and recommendations;

• Inform about programme performance: relevance, efficiency, performance, effectiveness, ownership, and sustainability;

• Identify good practice and lessons, looking at the achievements to date;

• Make forward-looking recommendations taking into consideration the remaining two years time-frame of the Project, but also the wider Development Effectiveness Agenda

Presentation objectives

Page 3: Dhaka revpres160814

Mix-methods but primarily qualitative analysis based on the following:

• Documentary review of project documents and updates received during the in-country mission

• Semi-structured individual interviews with (20) key informants from the following stakeholder groups:

A. GoB (4 ERD, 1 Cabinet, 1 MP, 1 Planning Commission)

B. UNDP (2 ACD, 1 CD, 1 AE specialist)

C. AE Unit (1 NPM, 1 DPM, M&E specialist, ICT Specialist, Aid Specialist) and a group discussion jointly with UNDP

D. Development Partners: DFID (1), DANIDA (1)

E. Civil society (2), Ernst and Young (1)

• One field visit to interview participants from AE awareness workshop given at district level (Moulvbazar) by a member of the evaluation team

METHODOLOGY :

Page 4: Dhaka revpres160814

• Relevance in responding to identified needs

• Efficiency of the project (use of resources)

• Performance of the project including design issues and implementation

• Effectiveness in achieving project results

• Ownership (degree to which ERD is committed to the process)

• Sustainability after the end of the project

EVALUATION CRITERIA:

Page 5: Dhaka revpres160814

• Sunday 3rd August 2013 : Start review of documents for the MTE and consultations between the AE team and the national consultant and field visit

• Sunday 10 August 2014: arrival of Evaluation Team leader and start of formal key informant interviews, including joint AE team and UNDP joint meetings.

• Key informant interviews from 10th to 14th August 2014

• 15th August 2014: review and analysis of data and notes and preparation of the presentation

• 16th August 2014: presentation to primary stakeholders of the preliminary findings, conclusions and recommendations

• End August 2014: submission of draft evaluation report

Evaluation Agenda

Page 6: Dhaka revpres160814

Evaluation Preliminary Findings, Conclusions and Recommendations

Page 7: Dhaka revpres160814

• The project addresses the issue of Aid Effectiveness, a wider unit of analysis that cannot be achieved by a limited intervention such as the AE project in its limited time-frame. However as the AE project stems from the highest level of ERD authority, it is clearly responsive to the needs of the government and in line with the current debate on what is now called Development Effectiveness.

• GoB planning is based on the current 6th 5-year plan (p. 239) and AE contributes to shaping future planning and partnership processes (such as JCS 2010-2014)

Relevance

Page 8: Dhaka revpres160814

• The project has known a number of delays both linked to procurement and human resource constraints and limitations from both UNDP and GoB’s side. In addition the year prior to the election more than half a year of work was lost given political unrest and violent strikes.

• The project is not particularly efficient as delivery rates varied per year and some activities had to be postponed or cancelled or changed in light of the context. Late start also means low initial delivery : 47,5% of expenditures (US$ 1.46 mio or 32.6% of initial budget and 47.5% of current funds available). BUT delivery ≠ effectiveness!!

Efficiency

Page 9: Dhaka revpres160814

BUDGET AND DELIVERY

Page 10: Dhaka revpres160814

Need to understand the context and environment of the Project – a number of factors led to delay – enabling factor JCS 2010-2014

• 2011 – late recruitment and start of the project delayed to 2012• Feb. 2012: change of NPD (15 Feb. 2012) – staffing and international staffing

gradually falling in place – TOR for studies/assessment - Holding of awareness raising activities at Dhaka and Division levels – support to LCG/WG - Cambodia Aid database identified as model – Development of AIMS

• 213 – part of the year lost strikes and security situation - Visit to VN for aid policy, capacity assessment of ERD

• 2014 – Beta version of AIMS, final E&Y report ERD capacity assessment, aid policy outline established, IMC set-up

Performance

Page 11: Dhaka revpres160814

•Reasonable performance given the constraints – could have been better, but could also have been worse

•Essential characteristics: low efficiency in part cause no clear road map to guide the vision of the AE (insufficient coord. and information sharing between GoB and UNDP on modalities and incremental approach and theory of change missing in the design) by also by lack of an enabling environment (e.g. initially low awareness of the AE, purpose of AE, no clear understanding of AE, no previous preparation within ERD or GoB except for 5 YP and JCS, so AE new and not streamlined)

•Better reporting on results necessary, low RBM

Performance

Page 12: Dhaka revpres160814

• Despite average performance, project rating from the key informant interviews is: 4 out of 5 (1 being lowest, 5 being highest) regarding the effectiveness = Good.

• This is based on the following:A. Awareness within GoB (high-level) of the importance of the AE agendaB. Development of the AIMS Database (win/win for GoB and DPs) essential to provide quality data

on aid flowsC. On-going process to develop an aid policy (outline ready and objective to finalize the policy by

end of 2014)D. Revitalisation of the LCG and WGsE. Participation in international events (feeds national ownership and 5YP and JCS so multiplier

effect)F. Ownership trough creation of ERD AE Wing and announced co-funding from GoB (leading to

sustainability)G. Inclusiveness of the consultation process (NGOs) but more can be done (Parliament, media, etc.)

Effectiveness

Page 13: Dhaka revpres160814

•Rating Grid (1 lowest to 5 highest)Effectiveness

ERD2UNDP3 UNDP4ERD5 CS6 DP7 DP8 AE9 GoB12 AE14 CS15 AVG

4 4 4 4 3,5 4 4 4 4 4,5 4 4

Page 14: Dhaka revpres160814

•The development of an AIMS database is a key achievement. Actually not yet live on line, but potential for win/win provided GoB and DPs are committed to its use

•Will contribute to aid transparency and provide credible evidence and management information regarding aid flows to all authorized stakeholders

•Slow in its shaping as no clear technical specs set out from the onset (tweaking of Cambodia’s model based on cost-effectiveness considerations)

RESULTS PER PROJECT COMPONENT – 1 AIMS

Page 15: Dhaka revpres160814

•Many awareness raising activities undertaken but wider inclusiveness and participation required (civil society, NGO, Parliament, South-South Coop)

• Established a capacity assessment ERD wide but will not be able to generate evidence by the end of the project: at best some recommendations will be implemented but remains too soon to affect overall ERD aid management capacity

•Questionable if this is correct entry-point (ERD-wide assessment) rather than more limited but incremental approach to capacity development (e.g. selected wings and future AE Wing)

•Other LM capacity development envisaged but not materialised (although CA are being foreseen over the rest of the project – TOR developed)

COMPONENT 2 : ERD Aid management capacity

Page 16: Dhaka revpres160814

•Very positive feedback on the dynamics and revitalisation of the structures and positive appraisal of its performance – of course varies from WG to WG.

•Unclear follow-up of key findings and recommendations stemming from WGs

•More monitoring efforts and RBM into facilitating the process, and also should be under ownership of ERD when project ends (LCG Secretariat within ERD Wing? Currently all done by AE Unit – substitution?)

COMPONENT 3 : LCG/WG mechanisms

Page 17: Dhaka revpres160814

• Slow process and back-and-forth movement to the process. Initially unclear level of commitment to developing an AP – current process now based on VN experience requires technical support (int. and nat. consultants hired) and wide consultations with civil society, parliament, private sector, etc.

• Draft outline ready• Inter-ministerial committee tasked with follow-up• Again initial lack of reference or preparation for taking up the aid policy

development process (hence visits and examples from other countries)• Seen as critical to demonstrate ownership and leadership from GoB and

create a coherent framework for DP assistance

COMPONENT 4: AID POLICY

Page 18: Dhaka revpres160814

•Participation in international events and support of DPs fuels national ownership

•ERD S and CS and other high-level officials committed to AE agenda and the development of AE/DE Wing in ERD

•Ownership ≠ Leadership – GoB committed to both but technical support from DPs necessary (such as UNDP)

• Long-term process that requires a WoG approach beyond the life of the project

Ownership

Page 19: Dhaka revpres160814

•Seems guaranteed through ERD AE/DE Wing creation• Considering also GoB co-funding to the Project (last SC)•Full operational AIMS (likely by end of project)•Approval and implementation of the Aid Policy (likely by end of the

project)•Gradual merging of AEU into ERD AE Wing with hand-over and training

Sustainability

Page 20: Dhaka revpres160814

1. The project has still two years to complete its objectives (June 2016)

2. Important milestones have been achieved so far (AIMS, Draft outline of AP, support to LCG/WG, develop capacity assessment). Project outputs are nonetheless an essential contribution to the outcomes, but the outcomes are yet to be appraised.

3.AE agenda (or DE agenda) cannot be accomplished over the short time-frame with a single small project. The AE project is the initial building block from which AE (or DE) efforts must be mainstreamed in GoB with the support of DP. Long-term planning and commitment from both GoB and DPs to this effect is essential (incremental approach of 15 years or more).

4. Opportunity for synergies and linkages exist within the AE agenda that are unexploited. Capacity Development should be a harmonized programme for GoB under the leadership of a DP (UNDP) and inclusive of all DP to work across sectors

CONCLUSIONS

Page 21: Dhaka revpres160814

5. Capacity of other LM not yet started, could be envisaged outside of the project (under a Capacity Development Programme, see point 4.

6. ERD personnel capacity development needs to be given priority with adequate hand-over from AEU to new ERD wing over the next two years

7. Longer-term support requires a long-term roadmap for how and where support will be ensured from DPs over the next fifteen years as it is needed to ensure mainstreaming of leadership and ownership of AE into GoB

8. More attention needed to show results and RBM value – trainings to ERD

CONCLUSIONS

Page 22: Dhaka revpres160814

• Importance of giving sufficient time to create enabling conditions for any change process. Given that AE is a long-time process, the creation of a critical mass of knowledgeable and committed partners is a pre-condition for success (e.g. importance of awareness raising and dissemination)

• AE requires a specific communication strategy to enable to engage the various partners and visualize stakeholders’ buy-in and gains.

• Difficulty to report on achievements given untangible nature of some processes. Hence the importance of a robust RBM in the Project design to inform proper Monitoring and Evaluation and a proper Results Framework mindful of the hierarchy of results.

• Importance of leveraging national leadership and ownership through international exposure and participation in regional initiatives (Post Busan Conference, APD, etc.) – so capacity to grasp emerging opportunities beyond the prodoc.

Lessons Learnt

Page 23: Dhaka revpres160814

1. Continue and consolidate the project achievements, particularly:▫ Consolidate and roll-out functional AIMS to be used by GoB and all DP including

UNDP in the first phase to show commitment to the process▫ Continue the process and provision of support to the development of the Aid

Policy and ensure its finalisation▫ Continue providing support to ERD capacity development and ensure hand-

over, training and coaching of the ERD new DE Wing ▫ Continue to support participation in LCG/WG structure and participation in

international events 2. Capacity development for development effectiveness is not limited to ERD or

specific LM. A harmonized approach amongst DPs is required for an effective WoG capacity development programme under the aegis of the UNDP. This global and harmonized CD programme should be inclusive of all CD projects and initiatives but provide an overall framework for actual results of the CD process.

RECOMMENDATIONS

Page 24: Dhaka revpres160814

3. The value of the inclusion of the Capacity Assessment foreseen amongst now 5 LM under the AE project is not evident for this project and should be dropped and included in the wider CD programme mentioned in the previous point to ensure no duplication of other DP efforts.

4. Develop a fifteen year road-map and strategy for achieving mainstreaming of AE/DE into WoG with commitment and support from DPs and incremental approach from ERD to WoG.

5. Plan incremental developments on the basis of achievements and synchronized with the National FYPs

6. Improve RBM knowledge and capacity through commissioning JOINT EVALUATIONS of future AE/DE interventions

RECOMMENDATIONS (cont.)

Page 25: Dhaka revpres160814

7. Develop a monitoring mechanism for LCG/WG to ensure that key issues are being addressed (high number of issues do not appear to be followed up or no evidence of referral). Important issues raised in LCG and WG should lead to action. The ERD owned LCG Secretariat needs to be properly staffed and equipped for this function.

8. UNDP and ERD need to develop jointly a comprehensive communication strategy for AE/DE that can be used for addressing various stakeholders (media, CS, Parliament) based on a shared understanding of the AE/DE agenda, including at field level (sub-district).

9. High visibility needs to be given to achievements (when AIMS will be used and the first analytical report prepared, when the NAP will be officially adopted, etc.) to contribute to leadership and ownership.

RECOMMENDATIONS (cont.)

Page 26: Dhaka revpres160814

Questions or comments welcome

Thank you very much!

Christian Bugnion de Moreta, Team LeaderNizam Al-Husainy, National Consultant

www.suburconsulting.es