monitoring & evaluation - wyg · monitoring & evaluation ... by women and girls in...

MONITORING & EVALUATION Capability Statement 2016

www.wyg.com creative minds safe hands

Learning lessons; improving lives

WYG - M&E track record across a broad range of sectors www.wyg.ngwww.wyg.ng WYG - M&E track record across a broad range of sectors 32

CONFLICT AND SECURITY

NIGERIAEvaluation of the Nigeria Stability and Reconciliation Programme (NSRP), DFID (2013-2018).

The NSRP is a five year, £39m programme (2012-2017), contracted via a consortium of NGO’s led by the British Council, International Alert and Social Development Direct. The objectives of the programme are to reduce violence and improve stability in selected regions of Nigeria (the northern states of Kano, Borno, and Yobo; the middle belt states of Kaduna and Jos; and the southern states of Delta, Bayelsa, and Rivers). The programme is premised on the achievement of 4 key outcomes, namely: • Improved conflict management mechanisms: Broader

societal participation in and oversight of conflict management mechanisms at federal, state and local level.

• Mitigating the drivers of conflict: Reduced grievances in target areas around economic opportunities and distribution of resources.

• Increasing participation of, and reducing violence against, women and girls: Increased and more influential participation by women and girls in institutions and initiatives relevant to peace building, with reduced prevalence and impact of violence against women and girls.

• Improved conflict prevention policy and practice: Research, advocacy and media have an increasingly positive influence on policy and practice relevant to stability and reconciliation in Nigeria.

Upper Quartile is carrying out a formative and summative programme level evaluation of NSRP that is tasked with exploring the combined effect, overall course and quality of the programme over its duration. The evaluation involves significant data collection to support both of these evaluation components, often in dangerous conditions and with datasets that are often inconsistent, poor quality, or missing altogether. To overcome these challenges, we have deployed a methodology that triangulates available data, drawing on national datasets, programme M&E data wherever possible, and instigating primary data collection to fill gaps or supplement existing data where this is necessary.

AFGHANISTAN Conflict Pool Evaluation, FCO (2014).

Upper Quartile have been contracted to conduct an Evaluation of Conflict Pool/Strategic Programme Fund mentoring activity in Afghanistan from UK Financial Year 11/12 to 13/14. This evaluation seeks (1) to establish what impact has been delivered by SPF and CP mentoring in Afghanistan and (2) to articulate why and how different types and styles of mentoring have been effective or ineffective. As such, this assignment features both process and impact evaluation components.

DEMOCRATIC REPUBLIC OF CONGO (DRC) M&E Support Unit to the International Security and Stabilisation Support Strategy (I4S), DFID (2015-2017).

The main vehicle for development interventions in DRC is I4S, which is managed by the Government of DRC and the UN Stabilization Mission (MONUSCO). The I4S is an umbrella of projects, funded and implemented by different donors, UN agencies, local and international NGOs. WYGI is establishing an M&E Unit to design and implement an M&E Strategy for the ISSSS, which is able to keep track of progress of all projects and of the Strategy itself. The main focus during inception is to scope out the “political” and “technical” feasibility of this initiative. Stakeholders in DRC (MONUSCO, Government, Implementing Partners - IPs, Donors) have been interviewed to assess their level of buy-in and their M&E capacities. The team will then develop an M&E Strategy that links vertically the projects to the Strategy and horizontally all projects and programmes, that is agreed among all stakeholders and that is implementable in the difficult operational context of Eastern DRC.

If the initiative is deemed feasible, the implementation phase will focus on finalizing its design and implementation. The Team will develop a logframe (common indicators and milestones to all IPs) for the five pillars of the ISSSS, a reporting and data collation mechanism across all implementing partners. The team will also carry out primary data collection and area based annual studies (quantitative and qualitative data) to keep track of ISSSS progress, and will commission specific thematic studies to fill knowledge gaps. The Team will also provide capacity building support to partners to ensure proper implementation of the M&E strategy.


GOVERNANCE AND CIVIL SOCIETYZAMBIAIndependent Evaluation of the Zambia Accountability Programme (ZAP), DFID (2015-18).

ZAP has two significant programme components dealing with both national and local level accountability; Institutions of Democratic Accountability Programme (IDEA) and the Citizen Service Engagement Programme (CSEP). ZAP will directly contribute to creating open societies and stronger political institutions in Zambia, both key enablers of development. The overarching purpose of the evaluation being delivered by WYG is to identify gaps in the evidence base in relation to democracy and accountability interventions and to directly contribute to these gaps. This will involve the development and implementation of an Evaluation Framework which will allow the programme to enable clear articulation of the results against the expected outputs and outcomes, evidence to support or challenge the Theory of Change and lessons for current and future programme design. The evaluation framework will include a detailed methodology and approach for conducting a Knowledge, Perception, Attitude and Opinion Survey (KPOAS) (including baseline and follow-up surveys) which will assess changes in knowledge, perceptions, opinions attitudes resulting from ZAP on the rights of citizens, quality of public services, attitudes to women in leadership, the role and performance of elected officials, voting behaviour, use and effectiveness of radio (national and community) and the ability of citizens to influence officials.

SOMALIA End-Term Evaluation of the Democracy and Human Rights Programme – Phase IV, SIDA (2013-14).

This 3 year programme had the overall objective of promoting democracy, human rights and gender equality in North-East and Central-South Somalia; with the specific objective of empowering the community through civil society to participate in the democratisation process in Sool, Sanaag, Puntland and Galmudug. Delta conducted the end of term evaluation which covered Phase IV of the programme (April 2011 – March 2014), with reference to the previous three phases. The overall objective of the evaluation was to provide SIDA, the programme implementers and other key stakeholders with necessary data and information for making informed decisions regarding programme performance in terms of its efficiency, effectiveness, impact and sustainability, and also to take appropriate action on any required changes in the programme design, development and implementation strategies for the future. The evaluation outputs are being used in the design of the next programme phase which will expand to areas in the south central region of Somalia.

UGANDA Developing an implementation plan for the operationalization of the National Monitoring and Evaluation policy of Uganda, UNDP (2013-14).

Delta undertook a comprehensive mapping of existing M&E capacities within the Ugandan government institutions, identifying gaps and draft recommendations that will serve as basis for the full operationalization of the National M&E Policy. As part of the review we assessed existing capacities for M&E (human resources, tools/equipment and technology for generating M&E information) across MDAs and Local Governments, examined the existing demand (internal and external) for institutional M&E outputs and the capacity of public sector institutions to satisfy this demand, and provided recommendations for implementing the M&E policy.

MULTI-COUNTRY Governance and Transparency Fund (GTF), DFID (2007, 2012-14).

DFID designed the Governance and Transparency Fund (£1.2 million) to enhance the capacity of a wide range of civil society and other groups to call for transparent, accountable and equitable access to decision making processes and government services. In 2007 Delta worked to complete a technical appraisal and performance assessment of the fund. In follow-up, from 2012-14 Delta provided advice on data analysis in order to produce a series of papers, across 38 grantees, looking at grantees Theories of Change, strength of evidence and impact. This involved appraising all annual reports, evaluation reports and programme completion reports and providing advice and guidance to grantees.


PRIVATE SECTOR DEVELOPMENT AND TRADELIBYAMonitoring and Evaluation Component of the Libya Private Sector Development Institutions Programme (LPSDI) (2013-2016).

WYG have recently conducted baseline data collection from Libyan job seekers and enterprises. The data gathered provides a baseline against which measures of effect and indicators for impact assessment can be measured. There were several challenges involved in the collection of data, including the fact that official data sources provided limited information due to limited capacity of data collectors within responsible ministries, and limited incentives for individuals and SMEs to register with the relevant official databases. Recent efforts by the OECD to collect information on the Libyan economy were stymied due to the security situation. Using WYG’s networks in Libya and rigorous duty of care procedures, the research team were able to access job seekers and SMEs. Libyan fixers organised all face to face meetings, using networks to open doors that would otherwise have been unavailable, for example a meeting with a job centre manager to discuss the Ministry of Labour database, which is a politically sensitive subject as there are many ghost registrants and accusations of rent seeking. The research team had limited access to jobseekers or SMEs outside of Tripoli and Misrata due to security concerns, and so used Computer Assisted Telephone Interviews (CATI) from a call centre in Tripoli to reach SMEs in Benghazi and Sebha. Local staff were trained to implement the CATI surveys. The team reached jobseekers in these areas through increased use of Arabic-speaking regional staff, complemented by 24 hour visits by expats with close protection security arrangements. Flexibility in data collection methods was vital, for example in smaller cities such as Sebha where it was difficult to interview jobseekers next to the Job Centre as people did not wish to appear to speak to foreigners in close proximity to state buildings, respondents were instead interviewed in a nearby café. In total, 1,108 randomly selected Libyan jobseekers were interviewed using a 63-question questionnaire, complemented by 60 in-depth Key Informant Interviews (KIIs) with jobseekers, and with four Job Centre managers at Libya’s main Job Centres in Tripoli, Benghazi, Misurata and Sebha. Data from enterprises was collected using a quantitative survey conducted with 300 Micro, Small and Medium Enterprises (MSMEs) across a variety of sectors from the Libyan cities of Tripoli, Misurata, Benghazi, and Sebha.

SOUTHERN AFRICA Evaluation of TradeMark Southern Africa, DFID (2013).

WYG conducted a mid-term evaluation of the Trade Mark Southern Africa (TMSA) initiative - a 5 year, £100 million DFID South Africa-funded programme to promote regional integration in Southern and East Africa by working with the COMESA-EAC-SADC Tripartite, its 26 member states, business and civil society organisations. The evaluation assessed TMSA’s activities and outcomes against its logframe and Theory of Change to assess the contribution of the programme to regional integration; and also assessed institutional arrangements and the Value for Money (VfM) achieved by the programme. WYG employed a mixed method approach to the mid-term evaluation comprising contextual analysis through documentary review, site visits, and VfM analysis.

EAST AFRICA Evaluation and Value for Money Framework to assess the impact of TradeMark East Africa’s project and programme portfolio, TMEA (2012).

Upper Quartile was commissioned by TMEA to research and recommend methods by which it could evaluate the impact and value for money of its activities. The outputs from this were captured in a practical evaluation framework which will be used by TMEA to structure its evaluation activity over the next four years. Upper Quartile worked with TMEA to help it clarify its theory of change at the impact level. The team then identified the projects which were likely to offer the greatest economic, social and environmental impacts. Options for evaluating the impact of these interventions were made in formal recommendations. Options included experimental, quasi-experimental and theory based approaches.


AGRICULTURAL AND RURAL DEVELOPMENTGHANA Market Development in the North of Ghana Independent Evaluation, DFID (2014-2018).

Ghana MADE is a Making Markets Work for the Poor (M4P) programme, intended to stimulate economic growth and reduce poverty in the Northern Savannah, through improved incomes and resilience of poor farmers and small-scale rural entrepreneurs in northern Ghana. Upper Quartile is currently conducting an evaluation of the Ghana MADE programme using a theory based approach, and will use qualitative and quantitative data collection and analysis to conduct an impact and process evaluation. The objective of the MADE evaluation is to: • Assess whether M4P delivered within the context of “thin

markets”, such as the northern savannah can deliver benefits for poor producers and entrepreneurs.

• Contribute to the evidence base on what works in Northern Ghana.

• Inform the international debate on the value of the M4P approach, and thereby feed into future policy and funding decisions relating to tackling poverty reduction and achieving the MDGs.

• Inform DFID and Government of Ghana of sustainable approaches to developing market systems and which interventions have the greatest impact.

The evaluation is using a theory based, participatory approach, drawing on data collected by the Ghana MADE programme as well as conducting primary data collection to fill gaps or supplement existing data. The evaluation is using mixed quantitative and qualitative methods to build up a picture of the programme in action, and to set out, revise, and substantiate or invalidate the programme Theory of Change. Contribution Analysis is being used as an analytical framework to come to robust conclusions about the programme’s contribution to outcomes.

NIGERIA Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (PrOpCom) Mai-karfi Programme in Nigeria, DFID (2013-2018).

Using the M4P approach, PrOpCom Mai-karfi will seek to address inequitable growth in Nigeria by increasing the incomes of 650,000 poor men and women in northern Nigeria by a) stimulating sustainable, pro-poor growth in selected rural markets, and b) improving the position of poor men and women within these market systems, to make them more

inclusive for poor people. WYG/Upper Quartile is conducting a formative and summative process and impact evaluation which is running alongside the programme implementation. The aims and objectives of the evaluation of PrOpCom Mai-karfi are two-fold. Firstly, to evaluate the achievement of PrOpCom Mai-karfi in relation to key OECD DAC criteria for evaluating development assistance – relevance, efficiency, effectiveness, impact, sustainability. Given the focus and ambitious targets in place for PrOpCom in relation to gender issues, coverage and inclusivity will also be considered. Secondly, the evaluation will contribute to the, as yet limited evaluative evidence base around what works, for whom and in what circumstances in relation to M4P programmes. The evaluation is using a theory based, participatory approach, combining quantitative and qualitative evaluation methods to assess the impact of the PropCom Mai-karfi programme. The data collection methods will include the undertaking of quantitative beneficiary surveys in two of the selected target markets to explore the impact of the interventions in more depth at beneficiary level. These surveys are to be conducted twice in each chosen market; once at the start of the evaluation period and once again at the end. In total 3,800 farmers/small rural enterprises will be surveyed (1600 in the baseline and 2200 in the in final survey), using a successive independent random samples design.

UK/ KENYA/ SOUTH AFRICA/ ETHIOPIA/ MALAWI Evaluation of Future Agricultures Consortium (FAC), DFID (2014).

Upper Quartile undertook an independent evaluation of the Future Agricultures Consortium (FAC), an international alliance of research organisations coordinated by the Institute of Development Studies (IDS). The focus of the evaluation was on understanding the pathways from high quality outputs to policy and practice change outcomes and impacts and what can be learnt from the FAC experience in relation to these. The objectives of the evaluation were to:• Assess the relevance of FAC’s policy research and

communication work to agricultural policy in Africa;• Document lessons from FAC, and accordingly, make

recommendations and/or outline options for commissioning agricultural policy research; and

• Assess FAC’s performance with respect to the achievement of logframe indicators.

The evaluation was commissioned by DFID at the end of a nine year period of core funding that began in 2005. Upper Quartile’s evaluation design combined elements of both theory-based and case-based approaches.

AFGHANISTAN Evaluation of the Comprehensive Agricultural Rural Development Facility (CARD-F), DFID (2015-2019).

In 2015 Upper Quartile was commissioned to undertake a four-year, theory-based mixed-method evaluation of DFID’s Comprehensive Agriculture and Rural Development Facility (CARD-F). CARD-F seeks to strengthen agricultural value chains with the aim of increasing rural incomes, job creation, and improving the trade balance. The centrepiece of the programme are Economic Development Packages (EDPs); each EDP targets and addresses particular value chains in one or more agricultural sub-sectors in a specific geographical area.

The purpose of the evaluation is to consider if, how, and in what circumstances CARD-F is providing jobs and increasing the incomes of beneficiaries. The theory-based, mixed-methods approach of the evaluation will draw on contribution analysis to explore cause and effect in order to reach conclusions about the contribution CARD-F has made to planned outcomes. Identifying the critical elements for success (‘keystone nodes’) as a basis for refining the programme and learning lessons pertinent to similar interventions (in Afghanistan and elsewhere) is central to this approach. Quantitative impact evaluation analysis is another important element of the evaluation involving, where possible, quasi-experimental impact evaluation techniques. Some aspects of CARD-F support do not fit attributive quasi-experimental approaches and, accordingly, quantitative and qualitative approaches will be combined in gathering evidence. Conflict and gender sensitivity are also key and will cut across all elements of the evaluation.

KENYA Mid Term Evaluation of Kenya Arid and semi-arid Lands Research Programme (KASAL), EU (2010).

The Kenya Agricultural Research Institute (KARI) was implementing KASAL, a research programme for the arid and semi-arid districts of Kenya, intended to improve access to innovative agricultural and livestock knowledge and technologies for rural populations in the arid and semi arid zones in Kenya. Upper Quartile conducted a midterm review of the programme at the end of the second implementation year.


CLIMATE CHANGE RESILIENCERWANDAIndependent Interim Reviews and Final Evaluation of Rwanda Climate Change and Environment Fund (FONERWA) support projects, DFID (2014-2017).

The National Fund for Climate Change and Environment in Rwanda (FONERWA) has been established with a view to it becoming Rwanda’s primary vehicle for improving climate change resilience, particularly among the most vulnerable households across the country. The purpose of the independent evaluation, being delivered by WYG/Upper Quartile, is twofold. The process evaluation will assess the projects’ progress towards effectiveness (outputs to outcomes) as well as the pathways to impact (through testing the assumptions in the Theory of Change). The impact evaluation will assess FONERWA’s contribution to impact (have the Fund and FMT catalysed low carbon/climate resilient growth and what is their respective contribution to wealth creation and poverty reduction?). The overarching purpose of the evaluation is to learn if and how the intervention impacts on the lives of beneficiaries for an array of outcomes including climate change resilience and poverty reduction, and to research the effectiveness of the national fund as a model for climate finance.

UGANDA Monitoring & Evaluation Design Consultancy Enhancing Resilience in Karamoja Programme (ERKP), DFID (2014).

The Karamoja Programme is working to increase resilience to extreme climate and weather events in semi-arid Karamoja through strengthening nutrition programmes, livelihoods and food security for vulnerable communities. The ERKP aims to contribute to, and be informed by, current resilience practice, both in terms of the measurement of resilience outcomes, and in the development of evidence that illustrates the factors that contribute to resilience. Resilience is inherently difficult to measure, however developing information which will assist in its measurement is critical for assessing the relative potential of different approaches. Delta developed a detailed approach and methods for performance monitoring, review, evaluation and research of the ERKP. This entailed conducting a review of existing DFID and other organisation approaches and methodologies on resilience measurement and the development of recommendations on indicators / data collection methods that could be readily incorporated into both DFID and individual partner log frames. We also conducted a review and verification of the theory of change (TOC) and log frame developed for the ERKP, and developed TOCs for the individual partner projects. We then developed a monitoring plan for the programme.

UGANDA GET FiT M&E support in the energy sector, KFW (2013-14).

The main objective of the GETFiT programme is to assist East African nations in pursuing a climate resilient low carbon development path resulting in growth, poverty reduction and climate change mitigation. The programme fast tracked a portfolio of 10-15 small scale renewable energy (RE) generation projects promoted by private developers with a targeted total of 125W, with the aim of transforming Uganda’s energy mix within a period of 3-5 years. Delta worked as part of a consortium to provide M&E services for the GETFiT programme, through a results-based monitoring and evaluation (M&E) framework designed to help the Government of Uganda, other national governments, donors and other stakeholders to better plan for renewable energy investment facilitating energy security, growth, poverty reduction and low carbon development. In addition to designing the M&E framework the consortium developed an M&E toolbox with user-friendly and practical tools and templates to use in monitoring of the GET FiT programme in Uganda, and conducted a baseline study for the programme.

SOMALIA Real Time Learning (RTL) Services to the Energy Security and Resource Efficiency Programme (ESRES), DFID (2015-2018).

The RTL assesses whether hybrid (primarily solar) mini-grid pilots enabled through the ESRES programme can provide access to affordable green energy for communities, institutions and businesses. Using a locally-developed evidence base, the RTL will update the ESRES Theory of Change and improve programme implementation during its 30-month first phase. The ESRES implementation programme itself is adaptive and the RTL methodology mirrors this by adopting a flexible research and engagement programme running alongside the activities of the implementing agent and the wider stakeholder group (community institutions, community members, businesses, IPPs and service providers such as hospitals and schools). RTL engagement activity will include household surveys, qualitative focus groups discussion with community representatives, key informant interviews among all stakeholder groups and study visits involving IPPs. As well as producing recommendations for the design and implementation of ESRES Phase 2, RTL will be used to gather and disseminate evidence on the feasibility and challenges of hybrid mini-grids in Somaliland, highlighting lessons of wider applicability in other fragile and conflict-affected settings. RTL ‘products’ will include a mix of published materials (a portfolio of case studies, thematic reports/summaries, newsletters and regular reports), and ‘softer’ RTL processes including feedback meetings, stakeholder and community events and ongoing, in-depth engagement between the RTL team and ESRES service provider.


EDUCATION

RWANDAEvaluation of Results Based Aid in Rwanda, DFID (2013-2015).

Upper Quartile is currently delivering this evaluation on behalf of DFID Rwanda. The objective of the RBA programme is to pilot the provision of additional results-based aid on improvements in the number of students completing primary (P6), lower secondary (S3) and upper secondary (S6) education; and the competency of teachers in Rwanda to use English as the means of instruction. RBA is composed of three key elements: • Payments based on results; • Recipient discretion on how results are achieved; and • Verification of results acts as the trigger for disbursement. Adopting a realist evaluation approach, Upper Quartile will undertake process and impact evaluations of the programme between 2013 and 2015, focusing on: • Exploration of the theory of change that informs the

programme and the existing evidence base to gather the necessary information to explore context, mechanisms and outcomes

• Systematic testing of if and in what situations the intervention has contributed to the intended outcomes

• Interpretation of the findings to explore how and why the intervention has contributed to these outcomes.

Currently in the inception phase, one of the key challenges being addressed by our team is the assessment of the evaluability of available primary datasets, which is shaping our final process and impact evaluation design and approach.


HEALTH AND SANITATIONNIGER, SOUTH SUDAN, MADAGASCAR, YEMEN, PAKISTAN, NEPAL, BANGLADESH, CAMBODIA, MYANMAR Independent Process Monitoring and Evaluation of the DFID contribution to the UNICEF Programme – Accelerating Sanitation and Water for All (ASWA) in Neglected, Off-Track Countries (2014-2016).

DFID is supporting UNICEF’s implementation of the Accelerating Sanitation and Water for All in Neglected, Off-Track Countries programme, which involves delivery of water, sanitation and hygiene both through direct implementation and strengthening of the enabling environment to improve and sustain levels of access to WASH services. Both approaches require independent quality assurance and supportive supervision to keep the process on track for a timely delivery of sustainable WASH outputs for 5 million people by December 2015 with a concomitant range of benefits.

The aims and objectives of the independent evaluation of IPME, being delivered by WYG, are two-fold.

Firstly, it will assess programme results and performance, and formulate recommendations to allow programme improvements to be made in real time. Given that women and girls are disproportionately impacted by lack of access to WASH facilities, the evaluation will pay particular attention to gender issues to ensure that differential gender impacts of WASH interventions are being robustly measured and addressed. The IPME focuses on 9 countries (Niger, South Sudan, Madagascar, Yemen, Pakistan, Nepal, Bangladesh, Cambodia, Myanmar) and aims at supporting the UNICEF’s WASH country programmes in:

• Strengthening their M&E systems: ensuring that baseline are in place and are relevant for measuring changes over time; harmonising DFID/UNICEF WASH indicators and milestones to ensure reporting is consistent and meaningful; strengthening quality of data collection and analysis processes.

• Building UNICEF country offices’ M&E capacity, with a particular focus on the Value for Money (VfM) component.

UGANDA Establishment of a knowledge management and communication centre for HIV/AIDS, DFID (2011-2014).

DFID is implementing an HIV/AIDS Prevention Programme which aims to reduce the number of annual new HIV infections in Uganda by improving effectiveness of the national prevention response. A key output is improved, evidence-based policy and practice; through DFID support over a 3 to 4 year period to establish a Knowledge Management and Communication Centre (KMCC). The KMCC is a multi-disciplinary knowledge hub that synthesizes, analyses and communicates research to inform evidence-based policy and practice. Delta has improved access to research findings and analytical work by systematically storing, synthesising, analysing and disseminating content in appropriate forms for use by stakeholders. We have also sought to inform emerging policy and practice with stronger communication of evidence and lessons learned from research on the HIV epidemic and response. Delta has delivered capacity building of other HIV/AIDS entities to improve the generation and sharing of information on the multi-sectoral HIV response; including capacity building of the Uganda AIDS Commission for development of its documentation centre.

CONTACT US

To find out more about our M&E work please email [email protected]

Follow us

monitoring & evaluation - wyg · monitoring & evaluation ... by women and girls in...

Documents