building resilience through asset creation and enhancement ......rima resilience index measurement...

DFID Contract Reference: PO 7532

Mid-Term Evaluation Report Department for International Development For the Period: February 2016 to October 2018 Date of Final Report: June 2019 Prepared by: IMC Worldwide In association with:

Building Resilience Through Asset Creation and Enhancement (BRACE) II – Monitoring and Evaluation Services

BRACE II Mid-Term Evaluation Report

i

IMC Worldwide Ltd 64-68 London Road Redhill RH1 1LG Tel: +44 (0)1737 231 400 Fax: +44 (0)1737 771 107 www.imcworldwide.com

Forcier Consulting Monitoring, Evaluation & Learning Unit Bilpam Road Juba www.forcierconsulting.com

http://www.imcworldwide.com/

http://www.forcierconsulting.com/


i

Contents

1 Background to the Mid-term Evaluation ........................................................................................... 1

1.1 Context ................................................................................................................................. 1

1.2 BRACE II Overview ................................................................................................................ 2

1.3 Purpose of the BRACE II Mid-Term Evaluation ......................................................................... 3

1.4 The Structure of the Report .................................................................................................... 5

2 BRACE II: Resilience and the Evaluation Methodology ...................................................................... 5

2.1 BRACE II Activity Packages, Theory of Change and Resilience .................................................. 5

2.2 Evaluation and Analysis Methodology ...................................................................................... 8

2.3 Quality assurance and data analysis ...................................................................................... 13

2.4 Methodological Issues and Challenges ................................................................................... 14

3 Why or why not activities have worked? ........................................................................................ 18

EQ1 To what degree is the programme progressing against its logframe outcomes? ............................ 20

3.1 What Has Worked ................................................................................................................ 20

EQ2: Are there differences in performance between Components I & II and if so, why? ...................... 31

4 Programme Questions .................................................................................................................. 32

EQ3 What are emerging pathways to resilience and how do we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts? ........... 33

4.1 Pathways and Milestones to Resilience .................................................................................. 34

4.2 BRACE II Contributions ......................................................................................................... 35

4.3 Emerging Final Outcomes and Impacts .................................................................................. 38

EQ4: Number of people whose resilience has been improved as a result of project support ICF-KPI 4 .. 39

4.3 What is the climate related problem? .................................................................................... 39

4.4 Pathways to Climate Resilience ............................................................................................. 41

4.5 BRACE Contribution to Climate Resilience .............................................................................. 41

4.6 Underlying project hypotheses .............................................................................................. 43

4.7 Additional Learning Questions ............................................................................................... 49

5 Summary & Recommendations ..................................................................................................... 51

5.1 Summary ............................................................................................................................. 51

5.2 Resilient pathways and climate resilient change ..................................................................... 52

5.3 Social protection .................................................................................................................. 53

5.4 Governance and social mechanisms ...................................................................................... 53

5.5 Resilience versus Humanitarian programming ........................................................................ 54

5.6 Internal BRACE II ................................................................................................................. 54

6 Works Cited ................................................................................................................................. 56

Annex I: BRACE II Logframe Summary ................................................................................................ 57

Annex 2: BRACE II Theory of Change .................................................................................................. 58

Annex 3: Detail of Data Gathering ....................................................................................................... 59

Annex 4: Performance Summary of Selected Outcomes ........................................................................ 61

Annex 5: Historical Rainfall for South Sudan ......................................................................................... 66


ii

Annex 6: Key demographics of beneficiaries ......................................................................................... 67

Figure 1: BRACE II Locations ...................................................................................................................... 2 Figure 2: Increases in vegetable farming ................................................................................................... 21 Figure 3: Change in income proxies .......................................................................................................... 22 Figure 4: Change in sources of income by Component ................................................................................. 23 Figure 5: Total production value by farmer type ......................................................................................... 27 Figure 6: Sources and changes in cash ...................................................................................................... 28 Figure 7: Resilience dimensions ............................................................................................................... 36 Figure 8: Beneficiary perceptions of change in community conflict since the start of BRACE ............................... 38 Figure 9: Food Consumption Scores in detail & by component ...................................................................... 44 Figure 10: Application of agricultural practices ........................................................................................... 45 Figure 11: Rainfall 1991-2015 .................................................................................................................. 66 Figure 12: Rainfall 1961-1990 .................................................................................................................. 66 Figure 13: Rainfall 1931-1960 .................................................................................................................. 66 Figure 14: Rainfall 1901-1930 .................................................................................................................. 66

Table 1: Quantitative data summary ......................................................................................................... 10 Table 2: Different Assessments of Resilience .............................................................................................. 12 Table 3: Framework to assess resilient change ........................................................................................... 13 Table 4: MTE Methodology and the Standard OECD/DAC Evaluation Criteria ................................................... 17 Table 5: Summary of MTE fieldwork results by intermediate and final outcome ............................................... 18 Table 6: BRACE II Contributions to Resilience ............................................................................................. 36 Table 7: Reporting Against KPI4 ............................................................................................................... 42

Box 1: Resilience, terminology and confusion ............................................................................................. 31 Box 2: Targeting and chiefs ..................................................................................................................... 37 Box 3: Third Party M&E, and learning ....................................................................................................... 49


iii

Abbreviations

BPMC Boma Project Management Committee

BRACE Building Resilience through Asset Creation and Enhancement

BRACED Building Resilience and Adaptation to Climate Extremes and Disasters

CBPP Community Based Participatory Planning Process

CCT Conditional Cash Transfers

CFA Cash for Assets

CFSAM Crop & Food Supply Assessment

CPA Comprehensive Peace Agreement

CSI Coping Strategy Index

DFID Department for International Development

DTL Deputy Team Leader

EQ Evaluation Question

FAO Food and Agriculture Organisation

FCS Food Consumption Score

FE Final Evaluation

FFA Food Assistance for Assets

FO Final Outcome

Forcier Forcier Consulting

HDDS Household Dietary Diversity Score

ICF International Climate Fund

IMC IMC Worldwide

IO Intermediate Outcome

IPC Integrated Phase Classification

KPI Key Performance Indicator

KPI4 International Climate Fund’s Key Performance Indicator No. 4

M&E Monitoring and Evaluation

MHHH Male Headed Households

MoU Memorandum of Understanding

MTE Mid-term Evaluation

NAPA National Adaptation Programme of Action

NGO Non-Governmental Organisation

OECD The Organisation for Economic Co-operation and Development

PET Pictorial Evaluation Tool


iv

RIMA Resilience Index Measurement & Analysis

SE Sorghum Equivalent

SSP South Sudan Pound

TL Team Leader

TPM Third Party Monitoring

VfM Value for Money

VSLA Village Savings and Loans Associations

WFP World Food Programme

WV World Vision


v

Executive Summary Building Resilience through Asset Creation and Enhancement (BRACE) II is a five-year programme (2015-2020). This is the final report of the mid-term evaluation (MTE) of the BRACE II programme. The executive summary includes a programme overview, the MTE approach and methodology, limitations, the four evaluation questions and key findings, and recommendations. These are set out in further detail in the full report. The BRACE II programme offers a resilience-building approach, intended to reduce dependence on food assistance and relief, and to improve community relations and reduce climate vulnerability. Food insecure families work to create productive assets at the household level (block farms for crops or vegetable gardens) or community assets (such as access roads, water ponds, or anti-flooding dykes) to bring about a sustained change in the amount of food produced or income generated, and to protect livelihoods against climatic shocks. Other activities intended to improve local governance, social cohesion or access to markets are intended to deepen and sustain those changes. Beneficiaries are provided labour opportunities during the lean season (i.e. before harvest when previous food stocks run low) to help them access cash and meet their short-term food needs.

BRACE II is being implemented in two phases of 2.5 years each, and has three live components. Component I is managed by the World Food Programme (WFP) and the Food and Agriculture Organisation (FAO). Component II is managed by World Vision (WV). The third is a monitoring, evaluation and learning component delivered by IMC Worldwide (IMC) & Forcier Consulting (Forcier). Component I began with the start of the programme and Component II was launched in 2018. BRACE II builds on lessons learned in a successful three-year pilot project phase - BRACE I.1

Components I and II have similar project designs in terms of project activities, to which Component II also adds establishing village savings and loans associations (VSLA), micro-businesses, and encouraging specific nutrition-positive behaviours. Component II provides cash transfers to beneficiaries in exchange for work, as does Component I, although for a shorter time period.

The MTE is a theory-based assessment of progress towards programme outcomes which will gauge the appropriateness of the theory of change. The MTE should help to answer four questions:

1. To what degree is the programme progressing against its logframe outcomes?

2. Are there differences in performance between Components I and II of BRACE II, and if so, why?

3. What are emerging pathways to resilience and how do we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts?

4. Number of people whose resilience has been improved as a result of project support (International Climate Fund (ICF) Key Performance Indicator (KPI) 4)?

The MTE adopts a bottom-up process that examines firstly what was done by implementers; secondly, what change those activities made; and thirdly, how changes impacted the lives of beneficiaries. The MTE seeks to provide unambiguous evidence of change attributable to the programme, and to serve as the evidence base for the four MTE questions. The MTE uses a mixed methods approach to verify change which occurred and could be attributable to the programme, and to provide critical analysis to interpret those findings. A complementary compendium to this report: “Detailed Analysis of Individual Outcomes at the Mid Term” documents the results of the quantitative data collection exercise and preliminary analysis against each outcome. The compendium on its own provides a wealth of insight into the programme mechanisms of change. It in turn is the basis of the responses to the four evaluation questions in this report.

1 2012 to 31 July 2015.


vi

Field work to collect data for the MTE was carried out in October and November 2018. A household survey was undertaken to measure progress towards BRACE II outcomes, for Component 1 from 2016 to date and for Component II from 2018 to date. Field researchers also conducted a post-harvest survey to triangulate household survey results. They kept observation diaries to capture information that would inform responses to the MTE evaluation questions or reflections on broader BRACE successes and failures, enablers and constraints, or emerging pathways to resilience.

The MTE provides the following responses to the four evaluation questions:

Evaluation Question 1: To what degree is the programme progressing against its logframe outcomes?

In differing degrees, the programme is making a difference in the lives of beneficiaries. Overall, the programme reported an 89% increase in the total value of farm production. That increase is due to the 33% of beneficiaries who adopted groundnuts (a higher value crop) and increased their farm size. A decrease in the value of farm production was reported by 33% of beneficiaries. Another 33% reported no change at all. Farm productivity increased 68% to 100% for groundnut adopters, while others reported yield decreases of varying degrees. Farm sizes overall increased by 1 feddan (acre). Farm size had been expected to increase by 3 feddans for year three beneficiaries, but that change was not evident. The number of vegetable farmers increased from 8% to 20% (Component I & II). Incomes may have increased by 20% to 40%. Increased income is mainly attributed to cash transfers in exchange for labour.

Slightly more than half of household expenditures are made on food. Another 40% are made on occasional expenditures such as health, education and essential household goods. The proportion of occasional expenditures was seen to increase to 45% in year three. Groundnut adopters, and particularly those with horticulture fields, reported increases in their incomes from their farms. They were also more likely to make expenditures on durable goods. A positive indication of change in the community is that approximately 90% of beneficiaries feel that community conflict is less likely now. Improvements in the economic situation and the level of social interactions of programme beneficiaries may have contributed.

It is estimated that approximately 43% of programme beneficiaries, principally groundnut adopters and more commercially oriented horticulture farmers, are likely to make the sustained improvements envisioned in the programme design. Also, it is valuable to bear in mind that BRACE II labour opportunities or cash transfers, though not sustainable, have helped stabilise conditions for a large majority of beneficiaries.

Evaluation Question 2: Are there differences in performance between Components I and II of BRACE II, and if so, why?

Component I and II demonstrate broadly similar trends, but the degree of change is higher in Component I. At the mid-term, it is premature to draw any conclusion from these differences. Component I has been implemented for almost three years. Component II had been implemented for six months at the time of the MTE. As the MTE documents considerable improvement in performance against indicators from year 1 to 2, the final evaluation will provide a more meaningful assessment of differences between components.

Evaluation Question 3: What are emerging pathways to resilience and how do we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts?

An emerging pathway to resilience can be framed around two branches; one aims to create enabling stability in the home, and the second aims to educate children so they may eventually find work elsewhere. The programme’s outcomes contributed towards that pathway, if not as envisioned at the outset of the programme. For groundnut adopters and more commercial horticulture farmers, their financial or food gains may have contributed to one or both of the branches along this pathway. Also, if cash transfers had been initially justified on a need to meet food expenditures, over the course of the year, beneficiaries report 40% to 45% of their expenditures are on health, education and essential household goods. For some, the cash transfers are considered particularly important to meet education costs and depending when the


vii

transfer is made, it may be entirely used for education. Social support relationships are intertwined with these two aims and BRACE II may be making a positive contribution. Sharing in the target communities is integral to identity and wellbeing. Without a baseline, the MTE is not able to determine if there has been a change in sharing. However, cash transfers provided by BRACE II were expected to be shared but also allowed people to share. It may be that a beneficiary’s involvement in the programme deepened their engagement in their social support relationships.

The overarching purpose of the MTE is to assess programme relevance. Beneficiary articulated aims or pathways provide an important benchmark. The success of groundnut adopters and horticulture farmers reflects well on the relevance of BRACE II support for them. For others, the cash transfer or labour component proved to be more relevant. Based on this, a reflection on BRACE II’s assumptions and targeting may be valuable (see recommendation 3). BRACE II assumes that its beneficiaries are farmers, but this is more often not the case. Its vulnerability selection criteria has led to the inclusion of people who are unable to farm or for whom increased farming is not a rational option (and who would be exposed to greater climate risk as a result). Most BRACE II beneficiaries are poor labourers, petty traders or fisher people who require reliable income opportunities. Others, the elderly or ill, require unconditional assistance. A final reflection is that beneficiary children may need assistance to attend school. It is a transformative opportunity to which BRACE II is contributing, even if indirectly.

Evaluation Question 4: Number of people whose resilience has been improved as a result of project support (International Climate Fund (ICF) Key Performance Indicator (KPI) 4)?

Evaluation question (EQ) 4 takes the pathways and contribution of BRACE II discussed in EQ3 and considers them with regard to climate related hazards. Perhaps 43% of BRACE II beneficiaries have improved climate resilience, and another 67% could be considered shock-proofed, (i.e. better able to manage a climate shock if not demonstrating a sustained change that reduce impacts of that shock). The vulnerability of BRACE II beneficiaries was seen to be a product of their poverty, exacerbated but not caused by climatic factors. Their two-branch resilience pathway, to a certain degree, recognises this situation. It has identified an option that can transform current exposure to hazards and risk. Educating children offers potential for more than climate adaptation. It may be considered an intergenerational strategy of climate resilience. Whether through the success of groundnut adopters, commercial vegetable producers or cash transfers, BRACE II helped meet immediate needs and as an unintended benefit, contributed towards longer-term household resilience strategies.

Long-term resilience strategies are however predicated on the sustained improvements of BRACE II beneficiaries. A sustained and improved income stream through groundnut or horticulture farming helps absorb climate shocks, but the degree to which it can be expected to absorb a shock is limited as they too will be impacted by the same shock. Greater emphasis on anticipatory mechanisms or linkages to wider value and marketing chains would make these income streams more risk informed. Cash transfer in exchange for work is an effective adaptive mechanism, reducing exposure to climatic hazards and increasing capacity to cope with their effects. These labour opportunities, however, will not be sustained.

There is an opportunity for BRACE II to help its beneficiaries risk-inform their work or family plans to manage transitory hardship. In addition to risk informing groundnut or horticulture farms, it could also help individuals to save, find short-term labour opportunities, or access remittances from family elsewhere. Contingencies can look beyond traditional coping strategies, towards emerging opportunities better aligned to achieving family goals.

Cross-cutting issues

The MTE reflects on several issues such as duration of benefits, gender and the programme’s hypotheses. Observations include:

▪ The greatest changes and benefits resulting from economic advancement or “graduation” activities occur in years one and year two of implementation. Cash transfers after the second year may be increasingly expended on health, education or basic household needs.


viii

▪ Women are equally represented in programme activities and on Boma Project Management Committees. They report realising benefits from the programme with very few reports of negative consequences.

▪ Several of the assumptions underpinning the programme and its hypothesis were shown to be partially relevant. For example, rural people are not necessarily farmers. Farming is in many cases a complementary food and income activity for beneficiaries, and it has a higher risk than other income options. For many, income opportunities beyond the farm are preferred or preferable.

Limitations

Answering the four evaluation questions faced a number of obstacles. BRACE II’s logframe, indicators, monitoring processes and tools captured activity implementation and reported against high level food security indicators, i.e. Food Consumption Score (FCS), Household Dietary Diversity Score (HDDS) or Coping Strategy Index (CSI). Results from activity implementation are not measured nor is attribution of changes in high level food security due to programme activities possible. The current logframe now includes intermediate indicators, which are directly related to programme activities i.e. changes in agricultural production, income, expenditures, sharing, and functioning of committees. As there is no baseline for or reporting against intermediate outcomes, the degree to which the MTE could determine progress or change against them was limited. However, the MTE made observations of how activities are performing as well as documenting pathways of resilience and the contribution of the programme.

Also, while at the mid-point of BRACE II, Component II was still in the early phase of project implementation, making it too early to detect some outcomes or make robust comparisons to Component I. The MTE was not intended to assess operational issues already are captured in regular Third Party Monitoring (TPM) exercises.

Key MTE Findings

▪ With the help of the programme, groundnut adopters and commercial vegetable farmers (43% of beneficiaries) have made important production gains which may be sustained.

▪ The same 43% of beneficiaries, the groundnut adopters and commercial vegetable farmers, may be considered more climate resilient. Risk informing those activities can make them more durable.

▪ Economic improvements are seen to peak in the second year and plateau in the third.

▪ Groundnut adopters, commercial vegetable farmers and cash transfer activities are helping households meet health and education expenses. This may be an emergent and potentially transformative – even if unintended - impact.

▪ Targeting led to the inclusion of beneficiaries inappropriate for the programme’s farming or labour components, following a broad assumption that all vulnerable, rural people are, can or wish to be farmers. That inclusion error, rather than activities themselves, meant that many people did not conform with the programmes’ expected impact.

▪ The programme can be considered to be on track for 43% of its beneficiaries. These are ones to whom programme activities are well aligned. Many may be better served by the programmes’ conditional cash transfer component, and others in turn may be better served by unconditional cash transfers. It may be helpful to reconsider the metrics of BRACE II success.

Recommendations

As BRACE II is at its mid-point of implementation, the evaluation findings and recommendations identify opportunities to deepen the progress it has made and to adjust areas where performance is not as intended. They address progress that was evidenced against logframe indicators, but also progress against pathways to resilience, and climate resilience in particular. Pathways to resilience or climate resilience are pinned to future, transformative opportunities for children through education and the ability of their


ix

parents to provide enabling stability in the home to keep the children in school. Within this framework of climate resilience, below are some recommendations for BRACE II to consolidate its success as the programme moves towards finalisation:

1. Reinforce the successes with groundnut adopters and horticulture farmers by risk-informing their work. This might be done by helping farmers anticipate poor seasons, building linkages to wider networks (such as value chains), and helping individuals make plans for their families should their harvest fail.

2. Work being carried out in Component II to generate progressive incomes streams (employment or business development) outside of agriculture resonates strongly with MTE findings. It should be encouraged. Opportunities for financial inclusion (i.e. to existing financial service providers) in Component I may be possible in the last year of implementation.

3. BRACE II can use social protection approaches to reframe its work to better suit the different categories of vulnerability found within its beneficiaries. Existing programme activities could be adjusted to better align with broad social protection categories: i.e. unconditional payments to the poorest, most vulnerable and less able to work; seasonal employment for the very poor and able to work, and economic growth (graduation) activities such as agriculture or micro-business support for high potential farmers or business people. Equally, under a social protection umbrella, a subsidy to defray education or health costs could be provided instead of or in addition to the provision of unconditional or seasonal employment payments to parents.

4. Extend current governance work within BRACE II activities to explore areas of shared interest. These areas might include helping educate the poorest children or providing stability in homes of the poorest families. Where there are opportunities to build on traditional sharing mechanisms or local taxation, it could allow for local leadership, greater local control and sustainability. Building on existing structures may be an opportunity to help communities build their own initiatives for equitable development or protection, even in the last year of the programme.

5. Use BRACE II learning to engage the humanitarian community on growth/graduation, agriculture and protection. BRACE II lessons show how one might bridge humanitarian and development objectives in the context of chronic crisis. The Department for International Development (DFID) and its BRACE II partners could extend an invitation to future learning and action workshops with relevant organisations beyond those directly involved in BRACE II. They can use these platforms to share any learning products more widely (for example, beginning with the learning workshops in March 2019).

6. BRACE II to revise its theory of change based on clear, measurable interpretations of terms such as resilience, climate risk, climate resilience, vulnerability, food insecurity, and climate related causes of food insecurity. This would address challenges encountered during the MTE and enable the final evaluation to provide richer results.


1

1 Background to the Mid-term Evaluation 1.1 Context

The 2013 conflict in South Sudan caused widespread displacement, human rights violations, deaths, disease and injuries, severe food and nutrition insecurity, and disrupted livelihoods. Despite a peace agreement in August 2015 and the formation of a Transitional Government of National Unity in 2016, conflict resurged and spread from three to seven of the ten former states of South Sudan. Markets and trade were severely disrupted, and the currency was in free-fall. Inflation was dramatic (ranging between 400-800%), placing extreme pressure on ordinary people to afford or find basic food items and commodities. Conflict exacerbated vulnerabilities arising from climate variability and extremes as well as environmental degradation. Relative stability appears to have returned to South Sudan with the signing of a peace agreement in September 2018.

During the period of conflict above, 1.8 million people were internally displaced and over 1.2 million fled to surrounding countries. The combination of insecurity, economic crisis and climate shocks led up to 40% of the population to be classified as severely food insecure. In response, levels of food aid increased across the country. The conflict exacerbated extreme and chronically high levels of food insecurity. While food security gains had been made after the 2005 Comprehensive Peace Agreement (CPA), many of those gains were reversed by the more recent conflict. Vast rural areas, including most of the communities assisted by BRACE II, were however less directly impacted by the 2013 conflict and have seen continuing improvements in their subsistence. They still face significant challenges related to climate related shocks, an eroding natural resource base, and lack of appropriate knowledge, skills and agricultural supplies according to WFP2. Some additionally face challenges of localised low-intensity social and natural resource conflicts.

The BRACE II business case referenced WFP South Sudan’s four main drivers of chronic rural food insecurity (Dfid, 2018), noted as the following:

1. Conflict and increased pressure on resources – primarily driven by post-2013 hostilities;

2. Climate variability and extremes – particularly drought, floods, consequences of climate change and environmental degradation;

3. Poor agricultural and environmental practices - lack of farming knowledge by returnees, subsistence farming, pastoralism, charcoal production and the destruction of forests;

4. Extremely limited transport and market infrastructure - seasonal and poor roads, high marketing costs, expensive imports, and conflict leading to an increase in prices.

The BRACE II Business Case further emphasised the role of climate variability and extremes as well as environmental degradation as a driver of rural vulnerability.3

Donors have been providing support through massive humanitarian and resilience programmes, but often based on annual plans and there have been calls for donors to improve multi-year support.

DFID responded with a resilience programme, BRACE II, as a deliberate programmatic shift towards more durable solutions for populations affected by chronic crisis. BRACE II as a resilience programme supports a combination of humanitarian and developmental objectives and activities through implementing partners to deliver support flexibly in the target counties. Understanding what these programmes are achieving is

2 Climate risk and food security in South Sudan: Analysis of climate impacts on food security and livelihoods’, report by WFP/VAM Regional Bureau and VAM South Sudan Office, 2014. 3 : …the dramatic effects of climate change and environmental degradation in South Sudan are: the drying up of permanent rivers resulting in seasonal waterways; reduction of water tables; erratic rains; soil degradation, wind and fire erosion; accelerated deforestation due to wood collection, charcoal production; competition for drinking water between people and livestock; habitat degradation for livestock and desertification; and loss of fish species and reduction of fish size as a result of rivers becoming increasingly seasonal. (DFID, 2018, p. 5)


2

vital for both accountability and learning purposes. Differentiating the added value of complex multi-year programmes from other traditional humanitarian or development approaches is important to decide on future ways of providing support.

1.2 BRACE II Overview

BRACE II is a five-year programme (2015-2020). The programme offers a resilience-building approach, intended to reduce dependence on food aid and relief, and to improve community relations and reduce vulnerability climate variation. Food insecure families work to create productive assets at the household level (block farms for crops or vegetable gardens) or community assets (such as access roads, water ponds, anti-flooding dykes) to bring about a sustained change in the amount of food produced or income generated. Other activities intended to improve local governance, social cohesion or access to markets are intended to deepen and sustain those changes. In exchange for their work, beneficiaries are provided labour opportunities during the lean season (i.e. before harvest when previous food stocks run low) to help them to meet their short-term food needs. A map of BRACE II locations is provided below in Figure 1. Figure 1: BRACE II Locations

BRACE II is being implemented in two phases of 2.5 years each, and has three live components. Component I is managed by the WFP and FAO. Component II is managed by WV. The third is a monitoring, evaluation and learning component delivered by IMC & Forcier. Component I operates in two former states (Northern Bahr el Ghazal and Warrap) with five international Non-Governmental Organisations (NGO) as cooperating partners4. Component II is delivered in three former states (Northern Bahr el Ghazal, Warrap, and Eastern Equatoria) with two national NGOs as cooperating partners5. Component I began with the start of the programme and Component II was launched with the second phase of programme, BRACE II in 2018. BRACE II builds on lessons learned in a successful three-year pilot project phase - BRACE I6.

Components I and II have similar project designs in terms of project activities, to which Component II also adds establishing VSLA, micro-businesses, and encouraging specific nutrition-positive behaviours. WV included these activities as they were part of the organisation’s understanding of “resilience building”. Also, Component II provides short-term labour opportunities for 8 months versus 18 months in Component I. This change was introduced as it was anticipated that there would be no additional need to provide labour incentives beyond the second year of the programme (an assumption which is included in the MTE).

4 World Vision, Joint Aid Management (JAM), Concern Worldwide, Action Contre le Faim, Norwegian Refugee Council 5 Smile Again Africa Development Organization (SAADO), and Support for Peace & Education Development Programme (SPEDP) 6 2012 to 31 July 2015.


3

Published BRACE documentation can be found at: https://devtracker.DFID.gov.uk/projects/GB-1-204888/documents.

Key project aims of supporting climate-sensitive agricultural techniques and agroforestry are in line with the Government of South Sudan’s National Adaptation Programme of Actions to climate change (NAPA).

As with 1, BRACE II is framed around three main approaches: Resilience and climate adaptation; conflict sensitivity and gender equality:

▪ Climate adaptation and climate resilience look to address climate-related causes of food insecurity, by increasing capacity and knowledge to deal with climatic variability; and diversifying water and irrigation options beyond rain-fed agriculture;

▪ A conflict sensitivity approach sees the operating environment as central to the programme and examines how the programme can address causes of conflict and avoid negative impacts;

▪ Gender equality refers to the programme’s understanding of gender dimensions and the programme’s impact on women, girls, boys, youth and men. BRACE II aimed to ensure that it did not create conflict between men and women e.g. over cash transfers.

The monitoring, evaluation and learning component of BRACE II, delivered by IMC & Forcier (referred to collectively as the Monitoring and Evaluation (M&E) team), works closely with DFID South Sudan and BRACE II partners to design and undertake the four principle M&E activities:

▪ Third Party Monitoring (TPM) monitors implementation, observes evolving gender issues and identifies emerging outcomes or impact. Three TPM’s were conducted by the time of the MTE. A fourth was conducted during the same field data collection exercise as the MTE. The results of these TPMs have made important contributions to the MTE.

▪ Thematic Learning uses findings from TPMs, and the MTE once complete, to support and inform learning for DFID and the IPs.

▪ Evaluation measures progress against outcomes and impact and generates evidence to answer programmatic questions about the BRACE II approach. The first evaluation exercise is this evaluation at the mid-term. The second, the final evaluation FE is tentatively planned for October/November 2019 (field work) and a final report in April 2020.

▪ Value for Money (VfM): The M&E Component will generate two sets of programme level VfM studies costing changes made and performance to cost implications of different DFID contracting approaches in BRACE II.

1.3 Purpose of the BRACE II Mid-Term Evaluation

The overarching purpose of evaluation in BRACE II is to provide evidence and learning about the approach for DFID and BRACE II partners. The MTE will contribute to that overall evaluation purpose by presenting field-based evidence of progress and through a summary and brief discussion of progress towards outcomes, differentiated by Component I and II. The mid-term evaluation is a theory-based exercise which will test the relevance of the theory of change and effectiveness of the programme through the four evaluation questions, as follows:

1. To what degree is the programme progressing against its logframe outcomes?

2. Are there differences in performance between Components 1 and 2 of BRACE II, and if so, why?

3. What are emerging pathways to resilience and how do we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts?

4. Number of people whose resilience has been improved as a result of project support: ICF’s KPI 4?

https://devtracker.dfid.gov.uk/projects/GB-1-204888/documents

https://devtracker.dfid.gov.uk/projects/GB-1-204888/documents


4

Issues of relevance and sustainability, while not explicitly mentioned, are integral to the four evaluation questions. The overarching purpose of the MTE is to examine relevance by considering performance of outcomes and their alignment with aims of beneficiaries. Secondly, sustainability is fundamental to the concept of resilience and is included as a resilience dimension (examined in Evaluation Question 3). Value for Money is examined in a separate study, outside of the MTE.

This report presents a summary and brief discussion of progress towards outcomes, differentiated by Component I and II. That evidence also informs reflections on what worked, for whom and why. It aims to understand how beneficiaries conceive of and work towards a more resilient future and how the programme has contributed. It also provides the basis from which to ascertain if BRACE II contributions have promoted climate adaptation, peace, and social cohesion. It uses realist and purposive evaluation approaches. These approaches will build from the bottom up with implemented packages of activities as a starting point to understand the specific outcomes they have generated. Those outcomes will then be tested to understand how they might be sustained, reduce risk or may lead to wider change. The approach will also build a foundation from which the number of people whose climate resilience has improved, and on what basis, can be estimated.

Answering the four evaluation questions at the mid-term will face a number of obstacles. BRACE II’s logframe, indicators, monitoring processes and tools captured activity implementation and reported against high level food security indicators, i.e. FCS, HDDS or CSI. Results from activity implementation are not measured nor is attribution possible of changes in high level food security due to programme activities. In May 2018 (the start of phase 2), the M&E team worked with DFID to revise the logframe and address this gap. The current logframe now includes intermediate indicators, which are directly related to programme activities i.e. changes in agricultural production, income, expenditures, sharing, and functioning of committees. As there is no baseline for or reporting against intermediate outcomes, the degree to which the MTE can determine progress or change against them is limited. Some analysis is possible, and the MTE will make observations of how activities are performing as well as documenting pathways of resilience and the contribution of the programme. The findings of the MTE will however enable the Final Evaluation (FE) to better answer the evaluation questions. Also, while at the mid-point of BRACE, Component II will still in the early phase of project implementation, making it too early to detect some outcomes or make robust comparisons to Component I.

The intended users of this report are:

▪ DFID: DFID will be the primary user of the MTE report. By providing a measure of progress towards BRACE II outcomes, and an assessment of contribution to resilience, the MTE can enable reflection on BRACE II assumptions, hypotheses and the theory of change. The approach provides a rich review of how packages of activities (agriculture, short-term labour, local governance and public works construction) have performed. It is hoped the report can assist with decisions about potential changes at the mid-term of the programme. The findings should also be useful for future planning considerations: administrative (contracting mechanisms), operations, design and deeper understandings of how programme activities can influence peace or social cohesion.

▪ BRACE II Partners: The MTE will report provide a snapshot of progress towards outcomes and suggest some explanations for that progress (or lack thereof). It will highlight where the programme has worked particularly well, where it did or did not work as hypothesised or where it provides unanticipated benefits. It is hoped these insights can be helpful to partners to inform potential changes that will in turn increase the impact and climate adaptation of their current programmes.

▪ Others designing, implementing, funding and evaluating resilience-building or climate resilience programmes: BRACE II is an innovative approach which has demanded similarly innovative performance metrics and evaluative approaches. The MTEs application of realist or purposive evaluation approaches, its bottom up approach to building evidence and its application of resilience


5

concepts to test for resilient change, may offer helpful insights and learning for others facing similar evaluative challenges.

1.4 The Structure of the Report

The report has five sections. Section 1 provides the reader with an understanding of the context and the programme itself. Section 2 describes the BRACE II programme in further detail, its interpretation of resilience and the methodology identified which could best answer the evaluation questions. Section 3 looks at “what worked?”, or whether the individual outcomes worked in synergy as envisioned in the programme design, and answers Evaluation Questions 1 and 2. Section 4 recasts the same individual outcomes, but against beneficiary defined resilience pathways to assess the contribution of the programme, and answers Evaluation Questions 3 and 4. Section 5 concludes with a summary of MTE findings and recommendations.

A complementary compendium to this report: “Detailed Analysis of Individual Outcomes at the Mid Term” documents the results of the quantitative data collection exercise and preliminary analysis against each outcome. The compendium on its own provides a wealth of insight into the programme mechanisms of change. It in turn is the basis of the responses to the four evaluation questions in this report.

2 BRACE II: Resilience and the Evaluation Methodology

This section of the report will discuss BRACE II’s activity packages, theory of change and understandings of resilience within the programme. It will also outline the evaluation approach and methodology, and describe how the evaluation questions will be answered.

2.1 BRACE II Activity Packages, Theory of Change and Resilience

DFID South Sudan designed BRACE II as an innovative approach to build on WFP’s general Food Assistance for Assets (FFA) programming, introducing strategic modifications such as cooperation with FAO to ensure cleared land has sufficient inputs, provision of extension services and training, and using cash instead of food when appropriate based on market conditions. It was designed to be an alternative to traditional humanitarian programming. Based on its theory of change and understanding of resilience, it identified packages of activities which would form the programme. Here we discuss those elements, how they have evolved over time and their implications on the MTE.

2.1.1 BRACE II Activities & Outcomes

Both components of BRACE are framed around a number of activities packages. While each package of activities is expected to deliver on its own intermediate outcome(s), they are also intended to work together in synergy. The design of the programme posits that each activity and outcome is necessary for the resilient final result, or expected impact. A brief overview of the logframe is provided in Annex I: BRACE II Logframe Summary.

Activities can be grouped into five packages: agriculture, public works, local governance, savings, and health behaviour. Agriculture activities include input provision, training, help to establish new fields in block farms as well as paid labour if they prepared and weeded the fields in the block farms. Public works refer to the construction of roads or ponds which include activities such as short-term employment and tools provision. Local governance activities related to the committees which would supervise the programme at the community level and its activities include different types of training. Savings and loans activities include a range of numeracy, financial and business trainings. Health behaviour activities include a range of nutrition related trainings. Labour opportunities are an important element in the BRACE design in both the


6

agriculture and public works activity packages. As it is a means towards those ends, it was not included as a separate package of activities.

The concept of resilience in BRACE II reflects an aspiration for these packages of activities to generate synergies resulting in beneficiaries who will have sustained, improved income sources from their farms. This is in part how the programme distinguishes itself from humanitarian work: a result which gives people the means to independently manage their affairs without the need, or with less need, for humanitarian relief. Public works, local governance and health behaviour, while important in their own right, in this sense are subordinate to the intended intermediate objective of a sustained source of increased income from farms. These packages of activities in and of themselves or in the manner in which they were implemented were designed to be conflict sensitive, gender informed, and climate adaptive.

2.1.2 Revisions to Logframe Outcome Indicators and Impact measurement

An initial logframe, based on a draft theory of change, programme aims and activities was developed in 2017 and published in February 20187. However, at the start of the second component of BRACE II, in early 2018, a need was identified to revise the logframe and the M&E components. The initial logframe did not include the metrics that could measure progress robustly. M&E team discussions identified a gap in the logframe. There were no intermediate level outcome indicators to directly reflect performance of activity packages nor was there means of attributing activities to high level final outcomes or impact. As a result, in June 2018 the BRACE II Logframe was revised and included “intermediate outcomes”. The intermediate indicators directly reflect changes in agricultural production, income, expenditures, sharing, and function of committees attributable to programme activity packages. The current logframe outcomes are summarised in Annex I: BRACE II Logframe Summary.

The inclusion of intermediate outcomes in the logframe clarified potential impacts that might be expected from BRACE II activities. The Impact level changes in the logframe were changes in the Integrated Phase Classification (IPC) for a specified county; a change in the Resilience Index Measurement and Analysis (RIMA) for a county; or a change in crop (mostly sorghum) production for a county. These indicators may reflect the types of changes made by BRACE II activity packages, but as population level measures, they did not reflect the scale of BRACE activity (which involves perhaps 5 to 10% of a county’s population) nor the likely magnitude of change resulting from BRACE activities. In May 2018, M&E Team discussions also reflected that resilience, as an impact, may be articulated differently by BRACE beneficiaries than those described in the logframe. The discussions concluded that the logframe indicators should remain, but that TPM, the MTE and FE would identify pathways to resilience as articulated by beneficiaries. That way, intermediate and final outcomes could be compared against the two interpretations of resilience – providing a more robust understanding of impact and contributing to BRACE II learning.

No baseline or monitoring was in place for the new intermediate outcomes, and there was no provision for BRACE II partners to collect or report on those indicators. The M&E team built collecting that information into the MTE data collection plans, recognising that it would limit the degree to which the MTE could assess progress or change. However, the MTE would establish baselines which would be important to the FE.

2.1.3 Revisions to the Theory of Change

With the start of the second phase of BRACE (BRACE II) a need was recognised to revise the theory of change which had been developed for the business case. DFID was working with partners in early 2018 to evolve the initial theory of change into one that was more specific about the changes that should be effected by BRACE II activities and how they would be brought about (see

7 http://iati.dfid.gov.uk/iati_documents/27251755.xlsx


7

Annex 2: BRACE II Theory of Change). Work with partners was not finalised but it fed into the Logframe revisions and ultimately shaped the design of the MTE and FE.

Using the revised logframe with its intermediate outcomes helped translate the generalised terms and the conceptual logical flow of the initial theory of change into one describing discrete and measurable changes and relationships with BRACE II activity packages. There is no finalised revision of the theory of change. Rather, the original logframe with the revisions of early 2018, discussions with DFID, and the design of the MTE and FE have evolved the theory of change which could be described as follows (summarised by the M&E Team Lead):

…by incentivizing and helping extremely poor or “vulnerable” people, who are interested in and committed to farming, by providing them with inputs (seeds, tools, training), irrigation ponds, labour opportunities (payment for land preparation and weeding or in public works) and helping them to prepare and farm a feddan on block farms, they will increase their farm sizes by one feddan per year…larger farms will enable these farmers to grow more crops than before the project. The increase in production will translate into more food and an improved, sustainable source of income, one which could better manage climate extremes. By making these gains, and working collectively, people will be more able and willing to share, increasing social cohesion in the community. Also by helping beneficiaries form and manage committees, conflict can be reduced. As well, by building roads farmers can sell more surplus and food traders can bring more goods. These sustained and integrated improvements will translate into expenditures on more and better food and less need to engage in negative coping strategies. Making these sustainable gains will lead to the programmes’ envisioned impact, measured by improved food security and resilience rankings.

2.1.4 Resilience in BRACE II

Resilience, as described in the BRACE II business case (Dfid, 2018), encompassed three different conditions:

▪ a state of development (resilience as an end point or final state);

▪ an ability to reduce the consequences of hazards, and climate change in particular8;

▪ a programmatic approach which is an alternative to humanitarian actions (i.e. resilience as a unique development programme design);

However, the programme wished to distinguish resilient from non-resilient change and grappled with the practicalities of evidencing these three different conditions. As a result, during the logframe revision process, BRACE II understandings of resilience were made more specific, meaningful and measurable. It was agreed that for BRACE to claim a change in resilience, it must be a function of the programme’s outcomes. Outcomes must be shown to have changed and also to have done so in ways that that demonstrate the three different conditions of resilience:

• Resilience as a state of development: programme outcomes demonstrate alignment and contribution to the developmental aims of populations served.

▪ Resilience as risk reduced9: programme outcomes demonstrate how they have reduced exposure to a hazard or how they have increased an ability to manage the consequences of those shocks.

▪ Resilience as an alternative programmatic approach: programme outcomes demonstrate changes which would not be expected from traditional, humanitarian programmes. (Note: resilience-building approaches became an appropriate response relatively recently in many parts of South Sudan).

8 Specific references in the BRACE II business case include …ability of countries, communities and households to manage change, in the face of shocks or stresses - such as earthquakes, drought or violent conflict - without compromising their long-term prospects.

9 The understanding of risk reduced used here, is premised on a combination factors: hazard exposure, sensitivity to impact, and adaptive capacity https://climatescreeningtools.worldbank.org/content/key-terms-0 or (UNISDR, 2015, p. 14)

https://climatescreeningtools.worldbank.org/content/key-terms-0


8

The more that BRACE II outcomes could demonstrate progress towards these three conditions, the stronger the evidence base to support claims of, improved resilience to climate shocks and stresses, as envisioned in UK ICF’s KPI4.

2.2 Evaluation and Analysis Methodology

This section describes the evaluative approach, data sources, data gathering and analytical process used to answer the four evaluation questions.

The mid-term evaluation is theory-based, documenting progress to date and testing the theory of change. At the mid-term, it is an early indication of success or of programme impacts framed around outputs. However, as a theory-based exercise, it will provide an indication of how implementing activities may or may not be leading to the envisioned programmatic impacts.

Forcier implemented a household survey to measure progress towards BRACE II outcomes from 2016 to date for Component I and from 2018 to date for Component II. Field researchers also conducted a pre and post-harvest survey to triangulate household survey results. They kept diaries to capture information that would inform responses to the MTE evaluation questions or reflections on broader BRACE successes and failures, enablers and constraints, or emerging pathways to resilience.

At completion of the field work component of the MTE, the M&E Team Lead (TL) and Deputy Team Leader (DTL) were debriefed by field researchers in Juba. Preliminary analysis of data and researcher findings began following the debriefing and continued through early December 2018. A presentation of preliminary MTE results was made to DFID South Sudan in early December.

2.2.1 Evaluation Questions & Data Sources

The table below outlines described briefly the four evaluation questions and indicates their sources of data:

Evaluation Question (EQ) Explanation Sources of Data

EQ1: To what degree is the programme progressing against its logframe outcomes?

To-date progress measured against the intermediate outcomes.

▪ Household Survey

▪ Pre & Post-harvest assessment

▪ South Sudan historical agricultural data & literature

▪ Regression analysis

EQ2: Are there differences in performance between Components 1 and 2 of BRACE II, and if so, why?

Results from EQ1 are disaggregated by Component.

▪ Results from EQ1

▪ TPM Key Informant Interviews

EQ3: What are emerging pathways to resilience and how do we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts?

EQ1 results are compared for their contribution against final outcomes or impacts.

Secondly, emerging pathways refers to beneficiary expenditures as a proxy for impact level goals sought by beneficiaries themselves. Those goals and how beneficiaries intend to reach those goals, map out a “pathway to resilience”. EQ1 results are also

▪ Results from EQ 1&2

▪ Researcher Diaries

▪ TPM Key Informant Interviews

▪ BRACE II reports

▪ South Sudan census data

▪ South Sudan historical agricultural data

▪ South Sudan Livelihoods Baselines


9

tested for their contribution to these pathways.

EQ4: Number of people whose resilience has been improved as a result of project support (International Climate Fund (ICF) Key Performance Indicator (KPI) 4)?

Using EQ3 results, outcomes can be ranked by the level of resilience they demonstrate. BRACE II beneficiaries benefitting from each outcome can be estimated and weighted according to the degree of resilience of that outcome.

▪ Results from EQ3

▪ BRACE beneficiary data

NOTE: In the final revised logframe EQ4 is included as an intermediate outcome. In order to include it most efficiently in the MTE methodology, it is included solely as one of the evaluation questions rather than also being included in the review of logframe indicators.

2.2.2 The MTE Approach

The four evaluation questions are answered using a mixed methods approach which builds from the bottom-up. It required evidenced change to be produced, attributable to the programme’s activity packages. Secondly, from that evidence base it applied an analytical framework to distinguish resilient from less resilient change. As the evaluation questions effectively build on each other (i.e. results of EQ1 are the basis of EQ2; EQ1 and 2 are the basis of EQ3; EQ4 is an extension of EQ3), the foundation of the MTE depends on clear evidence of what happened, and to what degree was the programme responsible for what happened.

The approach used in the MTE built on the approach used for the mid-term and final evaluation of DFID’s global resilience programme, Building Resilience and Adaptation to Climate Extremes and Disasters (BRACED). That approach emphasised evidence of change attributable to packages of activities as the basis of demonstrating changes, which were later analysed for their resilience attributes (Leavy, Boydell, McDowell, & Sladkova, 2018). We discuss this approach more fully in Section 4.1. The approach can provide an evidence base of programme generated change, as well as facilitate understandings of who or who not has benefitted from these changes and in what context. The BRACED approach was based on Realist Evaluation Approaches (Westhorp, 2016). Other similar approaches equally describe this bottom-up approach to evidence, attribute and understand: Purposive Evaluations (Funnel & Rogers, 2012), Outcome Harvesting (Wilson-Grau & Britt, 2012) or ODI learning on evaluation methodologies for public works programmes for protection and climate resilience (Ludi, Levine, & McCord, 2016) or (McCord, 2016).

These approaches share a common view that an objective change must firstly be identified. Secondly, understanding and explaining that requires that an evaluator must look broadly to a range of different factors. It is against this foundation that the role of a programme’s activities may be assessed as well as to test for change and change processes hypothesised in logframes or theories of change. While these are not specifically resilience measurement approaches, they are used in resilience measurement to evidence that a change was made. The modification of the BRACED approach was to then test evidenced changes for characteristics associated with resilience to learn if it was a resilient change that was made or not.

The BRACE II evaluative approach adds further to the BRACED approach to seek objective evidence of change and an understanding the programme’s role in that change. It also solicits insight from beneficiaries on the how and why change has happened.

2.2.3 Data Gathering

Data gathering utilised a mixed methods approach, including quantitative household-level data, in-depth interviews with staff members, local stakeholders, and beneficiaries, observation data on yields (both pre-harvest with standing crops and post-harvest with containers), and researcher logs on key questions and


10

themes. Complementary information was provided through project reports, such as beneficiary numbers and implementation issues.

At national level in South Sudan, there is no body which reviews or gives ethical clearances to research outside of the medical sector. However, data collection requests and approvals were reviewed and validated at the local level by the Governor’s office, the Commissioner, and the Relief and Rehabilitation Commission. At organisational level, Forcier carries out standard ethics research training with all new researchers. Forcier also has processes in place to ensure ethics are adequately followed with regard to data collection and that no harm is caused to beneficiaries.

2.2.3.1 Household Survey

A quantitative household survey was conducted in all project locations. The tool was designed to speak to the main evaluation questions posed by the MTE, as well as the joint logframe indicators. Its main purpose was to provide robust information on the magnitude of change(s), understand better for whom change has occurred, and to a lesser degree the mechanisms of change. The application of the household survey between Components I and II was consistent, but also included some differences.

The main aspects of the household survey are listed below by component:

Table 1: Quantitative data summary

Component I (WFP/FAO) Component II (WV)

Role of External ME team

Overseen by Forcier national researchers, and conducted by locally hired and trained enumerators.

Supported by Forcier researchers during training and initial days of data collection. Conducted by enumerators hired locally by WV.

Sampling Sample of bomas selected from the full boma list through the application of 36 clusters of 24 observations to bomas using PPS (done by Forcier).

Random sample of 24 individuals from the full beneficiary lists per boma (done by WFP).

All bomas included in data collection, the number of observations each determined through PPS (done by WV).

Random sample of ultimate respondents from beneficiary lists (done by WV).

Tool Developed by External ME team to measure joint logframe indicators.

Component I tool with some additional questions to answer internal WV logframe.

Sample size

864 884

Beneficiary years

3rd year, 505 (58.45%) 2nd year, 242 (28.01%) 1st year, 117 (13.54%)

All 1st year (all Component II beneficiaries were in their first year during data collection)

Locations and partners

Aweil North (WV): 144 (16.67%) Aweil South (JAM): 144 (16.67%) Aweil West (CWW): 216 (25%) Aweil East (ACF): 144 (16.67%) Aweil Centre (JAM): 24 (2.78%)10 Twic (NRC): 192 (22.22%)

Aweil North (SAADO): 249 (28.2%)

Gogrial (WV): 277 (31.3%)

Magwi (SPEDP): 358 (40.5%)

10 Smaller figures in Aweil Centre due to the lower number of bomas and the priority of getting significant numbers per partner.


11

2.2.3.2 TPM Key Informant Interviews

The MTE used qualitative data collected through the TPMs. Despite the TPM being primarily an accountability tool, the dataset provided through this mechanism has been strongly grounded in an overall learning agenda.

TPM interviews have included questions regarding the change made by the project, both in its magnitude, its directions and its mechanisms. Their main contribution to the MTE relates to mechanisms of change, direction of final outcomes and articulating beneficiary pathways to resilience.

2.2.3.3 Post-Harvest Assessments

In order to triangulate the yield data provided by the household data, the MTE also collected observational data on the crops produced by the project beneficiaries. This was done with two separate tools applied as appropriate: 1) the Pictorial Evaluation Tool (PET), developed by FAO for South Sudan, to estimate yields for standing crops; and 2) a post-harvest assessment estimating volume and weight of containers found at the household level in cases where the farmland was already harvested (also based on the PET).

Standing crops were found in Aweil West. In other locations the crops had just been harvested and the post-harvest assessment tool was applied. In each location the assessment included two randomly selected project beneficiaries and two randomly selected non-beneficiaries in their vicinity. Some logistical challenges lead to slightly fewer non-beneficiaries being interviewed than beneficiaries. The approach controlled for large differences in land quality and individual production issues and also allowed the MTE to situate current production against other community members in the programming areas. Significant logistics-related delays for Component II limited the ability of the researchers to conduct yield assessments in their locations.

2.2.3.4 Researcher diaries

Each Forcier researcher kept a daily diary to capture their observations or comments from beneficiaries regarding key elements or assumptions of the theory of change. They provided both a control for quantitative findings used in EQ1, as well as explanatory detail which was useful to EQ3 and in the discussion on why activities worked and for whom (section 3).

Upon return from the field, the researchers presented, debated, and discussed these hypotheses with the TL and DTL in a workshop. This allowed for the discussion to go deep into the mechanisms through which change had been created or why it had not been, as well as help validate the patterns observed in the quantitative data.

2.2.3.5 Regression Analysis

Over the course of analysis, certain elements of the data set, e.g. increased value in farm production and expenditures, were referred to Forcier experts for regression analysis to test for possible statistical relationships. Those analyses helped to determine the degree to which synergy or relations between intermediate and final outcomes might be claimed. As the BRACE II theory of change implies an inter-relationship or synergy from its different outcomes leading to a “resilient state” regression helped to test for synergy.

2.2.4 Analysis

This section describes the three-step analytical process that was used to distinguish resilient from non-resilient change. Firstly, the technical application of the term resilience is discussed. Next, the process to identify pathways to resilience is presented. Lastly, the process by which the contributions of BRACE II towards resilience is discussed. The process was used to answer EQ3 and 4 and by building on data gathered for EQ1 and EQ2. The analysis also brought in qualitative data gathered from researcher diaries, TPM key informant interviews and notes from researcher debriefing sessions.


12

2.2.4.1 Technical application of the term resilience

The uses of resilience in BRACE II, as a developmental objective and an approach to risk reduction (primarily from climatic hazards), required a clearly understood and measurable or rather a technical understanding of the term, resilience. The everyday usage of the term resilience helps make it a powerful concept but in the absence of a specific, technical definition that same familiarity leads to multiple understandings of what it means. Section 0 described the three concepts of resilience in BRACE II, and Table 2 describes how they were assessed:

Table 2: Different Assessments of Resilience

Notion of Resilience Assessment Approach

Developmental State

Beneficiaries articulate the pathway to resilience and milestones. Contributions by BRACE II outcomes towards these pathways will be assessed using the resilient change criteria

Risk Reduction The resilient change criteria will analyse individual outcomes for evidence that they reduced exposure to hazards and increased coping to consequences of hazards

Climate Resilience The resilient change criteria will analyse individual outcomes for evidence that they reduced exposure to hazards and increased coping to consequences of hazards for climate hazards

Unique Programme Approach

Evidence that outcomes have made a meaningful and risk informed contribution towards pathways to resilience and their milestones

Because there are multiple understandings and interpretations of what resilience is, there is no correct way to define or measure resilience. The RIMA (used in the BRACE II logframe) is one example. They are each designed to serve different purposes and notions of resilience. However, practical, meaningful and comparable measurements of resilient change are recognised to have many challenges (Schipper & Langston, 2015).

2.2.4.2 Pathways to resilience

Pathways to resilience were identified by both the TPM and MTE from key informant interviews, researcher debriefings and expenditure components of the household survey. Responses presented a fairly consistent picture of future aspirations and immediate term priorities and how they related to each other. Each outcome or change was examined for its relevance to beneficiary identified pathways to resilience.

2.2.4.3 Resilience Dimensions Analysis

This analysis tests each outcome for the contribution it makes to beneficiary defined goals. The analysis uses five criteria or resilience dimensions, identified by BRACED: transformative, risk informed, interconnected, future proofed, and inclusive. These dimensions help assess the degree to which an outcome is likely to make sustained impact.

Testing against the five resilience dimensions required that BRACE II intermediate outcomes were re-configured as “changes”. Some BRACE II intermediate outcomes were in fact different aspects of the same change. The analysis also considers the varying strength of data for each reconfigured change. An appreciation of the quality of data would qualify or limit conclusions that might be drawn. Table 3 below outlines the analytical framework used and shows how the outcomes were reconfigured into changes:


13

Table 3: Framework to assess resilient change

Changes

Alignment to Beneficiary Goals

Is it a significant contribution (resilience dimensions)?

Strength of Data

Increased food and income as a result of Increased production

Increased expenditures as a result of employment

Increased sharing due to increased food or income

Increased Conflict reduction due to new governance mechanisms

Increased trade due to new roads

2.2.4.4 KPI4: Number of people with improved climate resilience

Climate resilience is a sub-set of the broader resilience pathways discussed in EQ3. Resilience can be distinguished from climate-resilience by the role of climate hazards in the risk profile of beneficiaries. Demonstrating a reduction in exposure to a climate hazard or the ability to cope and manage the consequences of a climate hazard are defining elements of climate resilience.

The approach used in the MTE to identify the number of people with improved climate resilience draws on an approach developed for DFID Tanzania (McDowell, Goodman, Fitzgibbon, & Wallis, 2017). That approach is firstly premised on beneficiary-articulated developmental priorities. Secondly, it examines climate variation, climate change and associated risks that threaten their developmental gains. To assess programme contribution to beneficiary priorities or to reduce climatic risks, outcomes are examined against five resilience dimensions. These resilience dimensions have been grouped under the 3A’s (Adaptive, Anticipatory, Absorptive) to align with recent KPI4 guidance (Climate Change Compass, 2018).

This approach can highlight where BRACE II outcomes are more aligned and making a greater contribution to beneficiary aims and risks. BRACE II outcomes are ranked and placed in one of three categories of resilient change (climate resilience, shock-proofed, improved well-being). As the MTE identified the proportion of beneficiaries showing progress against each outcome, it provides an estimate of people found in each category of resilient change. Each category is weighted (between .3 and 1) allowing a single number to be reported against KPI4. While the methodology is consistent with recent KPI4 guidance, the categorisation of those results (based on the analysis using resilience dimensions), provides a higher threshold than is demanded by current KPI4 advice. Beneficiaries categorised as climate resilient or shock-proofed may meet thresholds of “improved resilience” for purposes of reporting against KPI4.

2.3 Quality assurance and data analysis

2.3.1 Assuring robust data

Answers to the four evaluation questions built on a common data set. Robust data was imperative. The research skills and contextual knowledge of Forcier’s researchers was a strong quality assurance factor. They are young South Sudanese, trained and working for Forcier, and some conducted the BRACE II MTE. The have strong contextual knowledge and understanding; they speak the local languages, are drawn from the same ethnic groups and some grew up in the programme’s locations.

The researchers received lengthy training by the M&E TL and DTL, to understand the programme and the M&E approach, and the household survey allowing the tools to be tested and researchers to become familiar with them. They were also trained by AA International, which developed the Pictorial Evaluation


14

Tool for South Sudan11, on pre and post-harvest techniques. Researchers in turn provided multiple-day training of the household survey enumerators. Researcher also oversaw and supervised the work of the enumerators.

Tools were tested to ensure they could capture nuanced perceptions as well as objective and observable reality on the ground. This was done particularly for measures of agricultural production. Beneficiary self-reporting (household survey) was triangulated with visits to households. At the homestead, harvested bags and sacks were verified (or fields assessed). For other indicators, the quantitative questions themselves were formulated in precise, non-value loaded, and verifiable ways. The language and structure of the questions was intended to avoid bias of beneficiary perceptions or value-judgments. This included guarding against optimism bias by concentrating on what has happened rather than what the beneficiaries think will happen – the most important variable of what had happened, agricultural production, was then triangulated through observation. Lastly, lengthy, team-wide discussions were held to minimise the level of individual biases that may otherwise emerge from researcher’s in-depth interviews and observations. These discussions included training on the most common types of biases that can emerge in research situations from both the sides of the researcher (particularly confirmation and selection biases) and the respondent (social desirability bias), and how to remain conscious of and able to mitigate them.

WV’s teams were included in household survey and pre/post-harvest assessment methodologies. Forcier researchers and helped to train World Vision’s enumerators to ensure comparability of data for Components I and II.

Results of the household survey and pre-harvest assessment were recorded on tablets and uploaded in the evenings or when there was connectivity. Uploaded data was reviewed by the TL as it became available on the server, monitoring for potential issues. Photos were taken of standing fields of harvested crop, with notes of the farmer, date and location. TPM key informant surveys were recorded by hand and transcribed. Researcher diaries were updated each night, in notebooks and laptops. Researchers were connected to each other and the TL and DTL through a Whatsapp group allowing them to share experiences, problems and solutions during field work. All researchers attended a two-day debriefing session, immediately at the end of the field work.

Post-harvest assessment results were compared with the household survey as part of quality assurance. Results were highly consistent with each other. The total value of harvest was 1,341kg sorghum equivalent (SE) according to the post-harvest assessment and 1,402kg SE according to the HH survey (4.6% difference).

The design of the assessment and analysis borrowed from the knowledge and experience of IMC and Forcier staff of South Sudan, its culture and livelihoods. Design of the sample frame was done by experts within Forcier who also undertook regression analysis of specific variables.

Analysis of household survey and pre-harvest data was led by the DTL. Data required little cleaning; outliers were identified and managed. Data sets were tested for internal continuity. Analysis of the data used an iterative approach. Each outcome was analysed and reanalysed. Findings were compared to livelihoods and agriculture baselines.

The resilience analysis provided a “reality check” of the findings which incorporated strength of data (more on this in 2.4.1).

2.4 Methodological Issues and Challenges

The MTE navigated a number of issues and challenges, which required creative solutions and equally must be appreciated as limitations on the findings at the mid-term.

The programme is only at the mid-term, but many of the evaluation’s questions can only or at least be better answered in the final evaluation, when implementation is complete. This is particularly relevant for

11 http://www.aainternational.co.uk/

http://www.aainternational.co.uk/


15

evaluation question 2, difference between Component I and Component II, where component I has been implemented for almost three years, and Component II started implementation in mid-2018. This report also addresses “impact” to the extent possible, bearing in mind that findings can only comment on progress towards impact targets at this mid-term stage.

One of the main challenges for the MTE was the lack of baseline with a control group, which limits the level to which changes seen among the population can be attributed to the project activities. Given that no baseline existed for the majority of the indicators, considerable effort was spent on ensuring that change could be measured to begin with, before issues of attribution could be considered. Including control groups in the MTE could address some of these concerns and was discussed. The additional financial costs and human resource demands to include a control group in the MTE would have been significant and considerably beyond the budget allocated to the MTE. A slimmer survey with a control group might produce more convincing results for the same resources, particularly for the agricultural component, and was discussed as an option. However, an ability to segment MTE results generally by Component, geography and implementing partner was prioritised, which could not be achieved with a slimmer survey to adequate levels of confidence. The survey questions were selected to help clarify issues of attribution. Nonetheless, a small control group was present for the harvest assessment, which mainly informed how the current agricultural production levels among beneficiaries compare with non-beneficiaries, but could not help to measure change and thus attribution to the project.

FAO’s general Crop & Food Supply Assessment (CFSAM) also provides an opportunity to compare beneficiary performance against county averages, as well as an approximation of how much environmental variation year to year may account for the changes measured by beneficiaries. The 2018 CFSAM was not yet available at the time of the main analysis and writing for the MTE, and as such these additional analyses could not be carried out for the MTE.

Given the lack of control groups being part of the project’s M&E design from the baseline, its introduction in later stages will be of limited value in any subsequent activities, as change cannot be measured against them for most indicators. For the main indicators where a baseline was estimated (yield levels), it would be difficult to determine against which year the control group should be comparing against (for project beneficiaries, the yield was from the year before their status as beneficiaries) and as such of limited value in accurately measuring a treatment effect. Its introduction should be discussed for the FE, though mainly for the purpose of establishing their level across all indicators against the wider population to understand their end-point.

While the BRACE II activities are aligned with NAPA, the scope of the evaluation does not include examining the actual relationships between BRACE II partners and government policy or programmes.

There was difficulty to translate programmatic terminology used to describe main areas of activity (social cohesion, resilience, vulnerability, conflict, risk, climate vulnerability, etc.) and the theory of change into clear and measurable indicators. As well, the absence of intermediate indicators prior to the logframe revisions in 2018 left an explanation gap in terms of what was expected from activity packages and how activity packages would translate into final outcome.

As the intermediate outcome indicators were introduced in July 2018, there is no baseline against which change can be measured. That baseline is being effectively created by the MTE to better answer those questions.

The MTE addressed additional questions to the extent possible using MTE data. These were not evaluation questions, and included queries such as: testing the implicit project hypotheses; participation of and benefit to women; does the provision of work for longer or shorter periods make a difference.

Other challenges, of a lesser order, include:


16

▪ The use of different evaluations teams for Component I and Component II (Forcier evaluated Component I and World Vision evaluated Component II); differences in researchers, enumerators and supervision could limit the ability to compare across datasets.

▪ The general security situation in the country was improving during the time of the fieldwork. The programming areas were generally stable (a condition for the success of the program). Some security risks were present in relation to cattle raiding and criminality, mostly near the border areas in Twic. These did not meaningfully affect the data collection. Only in one location, Twic a small road to the boma could not be used due to security concerns. No exclusions had to be made in the boma list sample frame and no replacements were required for these reasons.

▪ Road conditions make logistics a challenge in programme areas. Road condition led to delays during data collection, though the disruption was not significant. At times, researchers would use bodas (motorcycles) or walk, particularly to block farms outside of road coverage.

▪ Determining the value of agricultural production was complicated. Measurements of agricultural production are rarely in kilograms but rather in units of volume such as sacks, bags, safias, maluas, basins, etc. but market prices are reported South Sudan Pounds (SSP) by kilograms. Secondly, the extremely high inflation makes any measurement of change over time in SSP difficult, or even impossible. Thirdly, as BRACE II altered the composition of crops in the fields (mainly groundnuts), production varied considerably from pre-programme levels, meaning weight was no longer an appropriate metric. For these three reasons production was measured in sorghum equivalent values.

▪ The term “vulnerable” was operationalised differently by agencies on the ground with the result that different categories of people were targeted by the programme. Some organisations understood the vulnerable to indicate those in need of safety need support (widows, orphans, sick, and aged); others understood it to mean poor people, who were high potential farmers and needed assistance; yet others believed the vulnerable were those who had not yet benefitted from an NGO project. Ability, interest in or relevance of farming varies across these categories of beneficiaries with implications for performance against the intermediate outcomes.

2.4.1 Strength of Data

Strength of data was considered with regards to the degree to which the four evaluation questions could be answered. Firstly, the absence of baseline values for the measurement of changes against intermediate outcomes was recognised. Secondly, outcomes relating to sharing (social cohesion) and functional committees (reduced conflict) were much more subjective and qualitative than the other outcomes. Lastly, the evaluation was conducted at the mid-term and outcomes will evolve over the course of the programme.

Quality assurance measures helped to control for strength of data. Analysis took into account findings based on more robust data, and conclusions took into consideration the strength of data.

2.4.2 OECD Framework

The MTE methodology is based on four EQs, (see section 1.4). The methodology is consistent with the standard evaluation methodology developed by the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD/DAC), and focuses on qualitative consultation about the nature of resilience. EQ1 [To what degree is the programme progressing against its logframe outcomes?], EQ2 [Are there differences in performance between Components 1 and 2 of BRACE II, and if so, why?] and EQ4 [Number of people whose resilience has been improved as a result of project support, ICF’s KPI4 can be viewed as high level evaluation questions relating to effectiveness and impact, respectively. The framework in Table 3 above offers some detailed reflections on the EQs 1 & 2, arranged around logframe outcomes.


17

EQ3 [What are emerging pathways to resilience and how to we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts?] can be viewed as an open evaluation question that encourages the emergence of new detailed EQs about the nature of resilience, as sub-sets of EQ4. In the OECD/DAC methodology, this is equivalent to accepting that the detailed EQs in the evaluation ToRs will not be comprehensive and that new EQs will emerge during consultation. In practice, it is rare that new EQs are allowed to emerge and so EQ3 aims to ensure that this does happen.

The table below summarises the relationship between the MTE approach and the OECD/DAC criteria.

Table 4: MTE Methodology and the Standard OECD/DAC Evaluation Criteria

OECD/DAC Evaluation Criteria

BRACE II MTE Methodology

Relevance The overarching purpose of the MTE is to assess relevance, and the conclusions include lessons on the appropriateness of the approach in the specific situation in South Sudan.

Efficiency The MTE considers narrow efficiency (i.e. unit costs) and the data collected allowed some assessment of broader efficiency, defined as outcomes divided by inputs, with a particular emphasis on whether household benefits are sufficient to justify labour inputs.

Effectiveness EQ1 and EQ2 focus on this and relate to the Immediate Outcomes in the logframe.

Impact and Sustainability

For resilience projects, the assessment of impact and sustainability are merged, since resilience is defined as the ability to sustain impact. EQ4 assesses the increase in resilience, as defined in the BRACE II logframe and ICF KPI4. The MTE uses EQ3 to consult with beneficiaries on other dimensions of resilience beyond those in the logframe.

In the OECD/DAC standard methodology, risk is part of the assessment of sustainability, since the risks are reasons why impact might not be sustained. In a resilience project, the merging of impact and sustainability means that risk is an integral part of the assessment of impact, as well as sustainability.


18

3 Why or why not activities have worked? In this section, evaluation questions 1 and 2 will be addressed. They look at BRACE II progress against logframe outcomes, and differences in these results across Components I and II. The key distinction between this section and section 4 is the benchmark of success. In this section, progress is not only evidence of the change against individual indicators, but also if they are working together in synergistic ways, drawn from the programme design and theory of change. Impact, in the logframe, is premised on both achieving intermediate and final outcomes and their synergistic relationship. That impact is integral to the justification for the programme.

Notwithstanding the non-finalised nature of the theory of change, the intention of the BRACE II programme is reasonably established and clear. The programme is generally understood to have a single primary focus: to see farmers make a sustained increase their production through an increase in their farm size. That increase is intended to be enabled and sustained by a number of outcomes working together in concert. Progress must incorporate the synergy across outcomes.

The supplementary report, the Mid-Term Evaluation Compendium, provides the results of the MTE fieldwork against each intermediate and final outcome indicator. Those results are the foundation of this report and are summarised in Table 5 below:

Table 5: Summary of MTE fieldwork results by intermediate and final outcome

Indicator Results

C1 C2

IO1: Number of people with improved resilience

IO2: Households with good nutrition No data yet

IO3: Households using CSA

Proxy: number of skills used

Y1 2.2 2.4

Y2 3.6

Y3 3.4

IO4: Households receiving support from others

Whether households are givers or receivers of support, or both

Both 72% 74%

Giver 9% 10%

Receiver 2% 4%

None 17% 12%

IO5: Communities with conflict resolution structures 88% 91%

IO6: Increased household income

Proxy: ability to afford health, education, sugar, tea, lean season dinner

Better 30-45% 20-30%

Same 50-60% 45-55%

Worse 5-10% 5-14%

Proxy: livestock ownership, cattle

Increase 23% 6%

Same 71% 91%

Decrease 6% 3%

Total annual income (USD) Total ~710 ~444

Consumables as % total expenditure 52% 61%

Reason for improvement

Project cash 59% 83%

Farm output 33% 13%

Market/wage 9% 4%

IO7: Increase in household grain production


19

Increase in cereal production 12% -10%

Increase in value of crop production Total 89% 19%

Increase in area of production Hectares 0.55 0.41

Increase in productivity kg SE/ha 17% -7%

Reasons for increased production Land 89%

Productivity 37%

Reasons for reduced production Land 34%

Productivity 80%

IO8: Increase in food/income sources

Change in feddans cultivated 1.4=>2.3 1.4=>2.2

Change in number of crop types cultivated 2=>2.2 1.9=>2

Change in vegetable gardeners 26=>46% 32=>40%

Change in number of income sources 2.2=>2.2 1.7=>1.8

FO1: Poor/borderline food consumption (FCS), by gender

Male headed households 26% 64%

Female headed households 32% 63%

FO2: Diet diversity score (HDDS), by gender

MHHH 6.2 5.1

FHHH 6.3 5.2

FO3: Household crisis strategies (CSI), by gender

MHHH 34% 43%

FHHH 31% 51%

FO4: Reduced vulnerability to climate risks

Household making sustainable gains in resilience

43%

FO5: Reduced vulnerability to communal conflict - qualitative evidence is positive


20

EQ1 To what degree is the programme progressing against its logframe outcomes?

Broadly, the BRACE II theory of change and design seeks to promote an increase in farm production (by increasing the land cultivated and by using different techniques) for very poor families (i.e. vulnerable and food insecure). The increase in farm production is intended to better meet food and income needs in ways that can be sustained. The labour activities are to help beneficiaries to prepare additional land (block farms). The public works, such as ponds and roads, would increase production and trade. Committees are intended to reduce conflict that may impact on farming or trade. Increasing food and income through the programme’s collective work approach was expected to lead to more sharing (social cohesion), helping farmers cope with future disruptions. The net result of these activities working together would be that programme beneficiaries would not require humanitarian assistance.

This understanding has been used to determine not simply if outcomes are showing progress, but if they are working together as planned.

3.1 What Has Worked

A number of programme features worked and worked together for some people as envisioned in the programme design and theory of change. The changes below describe where there is progress against individual outcomes if not the theory of change.

▪ 40% of beneficiaries increased the land they cultivated, and increased the value of their produce (in sorghum equivalent), mostly through the adoption of groundnuts. 75% increased their land under cultivation, and 66% marginally increased their production. It is probable that this increased income at least in part was expended on more and better food and more likely on durable goods.

In differing degrees, the programme is making a difference in the lives of beneficiaries. Overall, the programme reported an 89% increase in the total value of farm production. That increase is due to the 33% of beneficiaries who adopted groundnuts (a higher value crop) and increased their farm size. A decrease in the value of farm production was reported by 33% of beneficiaries. Another 33% reported no change at all. Farm productivity increased 68% to 100% for groundnut adopters, while others reported yield decreases of varying degrees. Farm sizes overall increased by 1 feddan (acre). Farm size had been expected to increase by 3 feddans for year three beneficiaries, but that change was not evident. The number of vegetable farmers increased from 8% to 20% (Component II & I). Incomes may have increased by 20% to 40%. Increased income is mainly attributed to cash transfers in exchange for labour.

Slightly more than half of household expenditures are made on food. Another 40% are made on occasional expenditures such as health, education and essential household goods. The proportion of occasional expenditures was seen to increase to 45% in year three. Groundnut adopters, and particularly those with horticulture fields, reported increases in their incomes from their farms. They were also more likely to make expenditures on durable goods. A positive indication of change in the community is that approximately 90% of beneficiaries feel that community conflict is less likely now. Improvements in the economic situation and level of social interactions of programme beneficiaries may have contributed.

It is estimated that approximately 43% of programme beneficiaries, principally groundnut adopters and more commercially oriented horticulture farmers, are likely to make the sustained improvements envisioned in the programme design. Also, it may be valuable to bear in mind that BRACE II labour opportunities or cash transfers, though not sustainable, have helped stabilise conditions for a large majority of beneficiaries.


21

Their expenditures also included health and education expenditures. There is a certain degree of probability that this increase in production and hence income will be sustained after the project.

o It must also be recognised that yields per feddan were stable or even decreased, and farm sizes did not increase as expected. Farmers increased their farm sizes by one feddan - not the one feddan per year as envisioned in the programme (i.e. we would have expected that farm size increased by 3 feddans for Component I at the time of the MTE).

▪ 14% of programme respondents reported starting vegetable farming during the project (43% of beneficiaries currently report growing vegetables), see Figure 2: Increases in vegetable farming. The MTE did not have the means to determine at what scale vegetables are being produced, but up to 26% reporting gaining money from horticulture (up from 14% pre-project). Almost all vegetable gardens were reported to be irrigated, with many of these new vegetable farmers using project-built ponds for irrigation. It is not certain, but it is probable that sold vegetables were used to buy more and different types of food. There is a certain degree of probability that this increase in production and income will be sustained after the project.

o As we do not know the quantity of vegetables produced nor how much revenue it generated, we cannot know the degree to which vegetables have contributed to a change

in income or food consumption. Households that reported only growing vegetables, (but not the degree to which it was an income source), saw no improvements in their income levels or food consumption/diversity.

▪ Beneficiary farmers who adopted groundnuts or started vegetable farming reported the greatest progress towards BRACE II intermediate outcomes: increased production, increased income, new income and new food sources. Furthermore, differences in groundnut adoption accounted for the higher production values measured between beneficiaries and non-beneficiaries.

▪ Proxy indicators suggest an increase in income for beneficiaries (Figure 3: Change in income proxies). The main source of the increased income is the BRACE II conditional cash transfer (i.e. labour guarantee). Some also report an increased investment in wealth accumulation (goats or cattle). Beyond changes in total household income, the contribution of individual sources also changed. Some income sources have increased (agricultural/horticultural sales and agricultural labour) and others decreased (collecting wild foods, charcoal and business i.e. petty trade). Levels

26.2%

46.3%

31.9%40.2%

77.9%

89.8%

49.6%

64.8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Before project Currently Before project Currently

Component I Component II

Vegetable gardening and irrigation

Vegetable gardening Irrigation

Figure 2: Increases in vegetable farming


22

in these changes vary from Component I to Component II (see Figure 4: Change in sources of income by Component).

NOTE 1: Changes in income are suggested for half or more of Component 1 beneficiaries and perhaps a quarter of Component 2 beneficiaries. The main driver of that change for Component 1 is a change in cash, largely made available through programme supported labour opportunities. Also, Component I reports substantially larger improvements across all changes in income proxies, (as may be expected due to their length of enrolment in the programme). After cash, most beneficiaries in both Components report there has been no change in expenditures except for less than 10% reporting a decrease in their expenditures. More Component II respondents reported having less cash than in Component I. Closer inspection showed they are more likely to report improvements on other indicators such as school, medicine, or livestock (as is also seen with those reporting less cash in Component I). It would seem that the higher levels of decreased amounts of cash are actually indicative of investments in services, assets, or productive capacity, rather than of HH wealth or status – and suggest stabilisation or development.

77%

44%

9%

6%

5%

20%

62%

3%

7%

14%

6%

8%

9%

10%

14%

27%

44%

52%

54%

47%

28%

6%

22%

19%

32%

23%

20%

15%

5%

11%

4%

0% 20% 40% 60% 80% 100% 120%

Amount of cash

Number of cattle

Number of goats

Afford school

Afford medicine

Afford dinner (lean season)

Afford sugar

Change in income proxies (Component 2)

Remain bottom Decrease No change Increase Remain top

Figure 3: Change in income proxies

49%

20%

5%

1%

1%

2%

23%

6%

5%

7%

7%

7%

7%

5%

22%

29%

51%

42%

53%

51%

72%

23%

46%

30%

44%

34%

34%

8%

7%

5%

6%

0% 20% 40% 60% 80% 100% 120%

Amount of cash

Number of cattle

Number of goats

Afford school

Afford medicine

Afford dinner (lean season)

Afford sugar

Change in income proxies (Component I)

Remain bottom Decrease No change Increase Remain top


23

NOTE 2: There are significant increases in the degree to which beneficiaries report the core activities of agriculture and horticulture being sources of money rather than simply food. For Component I, more than half of the farmers are selling part of their harvest. Although constant groundnut farmers are much more likely to sell portions (around 60% versus 46% for adopters and 34% for non-groundnut farmers), whether the farmer reports planning sales seems to be predicted best by total production value and presence of sesame. This suggests that particularly adopters are not yet entirely clear of the commercial potential for groundnut. Furthermore, almost all Component I vegetable farmers report gaining an income from it, where this is not, at least not yet, the case for Component II.

Attribution: Initially, there was concern that it may be difficult for the MTE to identify changes in agricultural production which could be attributed to BRACE II’s activities (provision of seeds, tools, training, or the additional fields created on block-farms). Without a control group, it would be difficult to differentiate programme activities from other factors (climatic, soil, pest or weed issues; farmer motivation, etc.) as the mechanism or driver of change.

Actual results mooted much of that concern. Most beneficiaries in the programme – who principally grow sorghum (with some also growing considerable amounts of groundnut) – reported harvests largely unchanged from pre-programme levels. Their 2018 harvest was consistent with historic averages and trends for their county. There was no indication of a change in the factors of production – rainfall, pest, disease, agricultural techniques, seeds – from previous years. Also their production was comparable to the small sample of control farmers. There was no evidence to suggest that results may have been poorer if there had not been an intervention. The MTE suggested that had been no change in agricultural production for this group of farmers. However, if these farmers did not indicate a change in total production, MTE results indicated a change - a decrease - in productivity per feddan. The MTE indicates the total amount of land farmed increased by a level consistent with the programme’s mechanisms (i.e. farmers received a cash transfer if they worked on one-feddan in a block farm). While the programme may have induced farmers to put more land under cultivation, with total production likely unchanged, the programme may have simply induced farmers to farm more land to produce the same amount of sorghum. This unintended change may not continue after cash transfers stop at the end of the programme.

However, the MTE established that farmers who adopted groundnuts and vegetables at scale did indicate changes in the value of their total farm production. Comparing the MTE results in 2018 to pre-programme levels, baseline trends for the county or performance of neighbouring farms (control farmers), suggests that groundnut and vegetable farmers had changes in the total value of their production. They were

0%

10%

20%

30%

40%

50%

60%

Before project Current Before project Current

Component I Component II

Change in sources of income

Agriculture sales Horticulture sales Livestock rearing Agricultural work

Other work Collect wild materials Fell trees / charcoal Business

Hunting Remittance

Figure 4: Change in sources of income by Component


24

exposed to the same climatic, disease and pest factors as other farmers in their community and or block farms. Distinguishing them was their decision to adopt higher-value groundnuts and vegetables and at scale. The comparison between beneficiary and non-beneficiary households in the post-harvest assessment further highlights the impact of groundnuts. The difference in total production values was driven entirely by groundnut production (1,341kg SE for beneficiaries versus 1,086kg SE for non-beneficiaries). The post-harvest assessment also showed that production for groundnut producers reached higher levels than their community averages.

While the MTE found evidence indicating a change in the value of production for groundnut and vegetable adopters, attribution to programme activities was not direct and may have been unintended. BRACE II – generally - did not promote groundnut farming or marketing. Its focus was on staple grain production (sorghum). Most seeds distributed were sorghum seeds. However, BRACE II promoted block-farms. Some block farms were situated on sandier soils. As these soils were not appropriate for sorghum, some farmers planted groundnuts, which often required they provide their own seed.

Horticulture adopters (at scale) reported a change in the value of their production. Their adoption attributes more directly to programme mechanisms (seeds, tools, irrigation ponds). The MTE noted that not all BRACE II horticulture farmers were new or had adopted horticulture as a result of the programme. Also, it noted that not all new horticulture farmers elected to farm at scale. While the change in horticulture production may attribute more strongly to the programme, one must be aware that its mechanisms worked for some farmers but not universally.

The results of the 2018 FAO Crop and Food Supply Assessment became available in May 2019, providing an important information source to triangulate the MTE results. This preliminary agricultural assessment can provide a useful basis for a more in-depth examination during the final evaluation of BRACE II12.

▪ Other intermediate outcomes such as households giving or receiving support, and presence of functional conflict resolution mechanisms, associated weakly with these positive changes and also indicated progress on their own.

o Sharing may have been enhanced by the programme’s activities, where support is predominantly given to close family members/relations. It may not be that the programme changed the amount of sharing; more likely participation in the programme may have changed the scale or regularity of giving and offered more accrued social capital to the giver. The cash transfers were a key mechanism helping beneficiaries to increase their level of giving or sharing. The cash transfers also had important social implications for some of the most vulnerable; those who were barely able to take care of themselves suffered a “pariah status”13 in the community due to the local custom of “hunger courts”. In these courts, a well-to-do member of the community, usually a relative, is ordered by the chief to take care of the person in need. Due to fear of being assigned with this responsibility, community members would avoid being in the presence of a person who may seek help through the hunger court. Through having access to basic subsistence through the cash transfers, these individuals have no longer been shunned by the community.

o The programme changed the ability of some beneficiaries to pay local taxes, payment of which has important social implications. By paying local taxes, it gives one dignity and a voice in community affairs. Also, the programme increased some beneficiaries’ ability to purchase on credit, as their participation changed how they were perceived by local

12 In the late stages of the finalisation of the MTE report (May 2019), FAO released the results of the 2018 CFSAM. The CFSAM reported that farm field sizes were not reported to have changed in NBEG and Magwi. Farm sizes of BRACE II farmers was 50 to 100% larger than the county averages reported in CFSAM. The CFSAM reported that sorghum yields were unchanged in Magwi, Aweil East & Twic from 2016 to 2018. Yields increased 15% in Aweil North and decreased 30% in Aweil West from 2016 levels. It is difficult to compare CFSAM county production averages for 2018 against MTE results (differences in samples). Broadly, they share similar trends. BRACE II did not observe changes in total household production for sorghum farmers who did not adopt groundnuts as a new crop. 13 A term directly translated from local language.


25

businesses or service providers. These examples do not imply a change in social cohesion, but they reflect that due to the programme, beneficiaries have been more able to engage in local social structures.

o Engagement in reciprocal relationships correlates positively with Food Consumption Scores (FCS) and occasional expenditures (most likely education or medical services). Beyond cash transfers, we do not know what has enhanced respondents’ ability to engage more deeply in reciprocal relationships. To the degree it is dependent on labour opportunities, the benefit may not be sustained. Conversely, for the successful groundnut farmers or vegetable farmers, this may be a more durable change.

o TPMs verified that Boma Project Management Committees (BPMC) were established, were functioning to support programme implementation, and addressed programme related issues (including conflict amongst beneficiaries). One of the main purposes of the BPMC is to conduct a community based participatory planning process (CBPP). The CBPP will select public works activities (referred to as “asset creation”) or identify the location for a group farm. BPMCs and community planning are processes and as such are examined in TPM missions, rather than in the MTE. However, it is worth considering how BPMCs have performed and in turn have or have not influenced outcomes examined more closely in the MTE.

TPMs verified that BPMCs facilitated community-based participatory planning processes. They helped balance a range of local interests and priorities in order to select activities to be implemented from a range of programme provided options. However, the options provided to the communities are limited and may not capture local priorities. Decisions by BPMCs were also subject to approval by the relevant NGO and WFP. As CBPPs were directly linked with resources, they were subject to external pressures or could be co-opted. In one case a commissioner assigned a block farm to a location where there was no water. It was done to pressure the programme to provide water. It was noted in many locations that the inclusion of women in the BPMC did not equate to voice or influence on decision-making. More influential BPMC members tended to shape decisions.

BPMCs intended to bridge programme and local governance. They were intended to include local leaders. It was expected that participation in the BPMC should be a public service. No remuneration or resources were to be provided. Communities feel that a wealthy programme has down-loaded an administrative responsibility and cost to them. They consider BPMC work as that of a public servant rather than a volunteer, and are aware that public servants are paid. Normally, local leaders are remunerated, even for local issues. For example, if they are engaged to resolve a conflict they are paid by the person who brings the claim. The absence of remuneration and the limited decision-making powers of BPMCs was reported to have led to a disconnect with some local leaders. Some BPMCs were reformed. Some were re-configured to include only programme beneficiaries and excluded local leadership. Others included family members of the chief in the programme as a de facto payment for services.

Local ownership over programme processes or assets offers insights on how BPMCs and CBPPs may be perceived. TPMs noted that programme constructed ponds were dry when they should have water; some ponds were partially built; tree seedlings were not cared for or were inappropriate for the soil; roads were built but led nowhere; other roads reported to be built were modifications to existing roads. The establishment and management of block farms was the responsibility of BPMCs. Each year additional block farms were to be established to increase the size of beneficiary farm holdings. However, in several locations, the same land was “cleared” each year (and reported as new land cleared). Many block farms are not expected to continue after the programme. Without programme provided


26

“conditional cash transfers”, or due to soil nutrient depletion, the farms are expected to be abandoned. BPMCs and CBPPs performed to a level as might be expected given programme processes and local context.

As BPMCs included local leadership, it did bridge the programme and wider governance structures which in turn gave the programme credibility that helped the programme work. Respondents felt that they were less “vulnerable” after the programme than before. Perhaps the BPMCs were effective at helping the programme “do no harm”.

The benefit of BPMC to the programme’s intentions to reduce conflict is limited, as they only dealt with programme created conflict, but may have helped to reinforce the effectiveness of local institutions. The BPMC is not expected to be sustained after the project. While the BPMCs were limited in their scale within communities to participants in the programme, given programme coverage, an expectation to influence wider conflict may not be met. Also, due to inclusion of local leaders, targeting inclusion errors were noted, or rather people were included in the programme who would otherwise not qualify.

▪ Lastly are activities which TPMs verified had been implemented, but which did not translate into changes of intermediate outcomes (agricultural training, and public works, such as dykes and roads).

o Agricultural trainings on improved or climate-sensitive agricultural practices showed no change in adoption by farmers (see Figure 10: Application of agricultural practices), nor did those trainings correlate with increases in levels of farm production. Increased levels of production correlated more strongly with the adoption of a new crop and not topics of BRACE II agricultural trainings. It was observed that the techniques included in BRACE II improved and climate-sensitive agricultural trainings have been the subject of agriculture trainings in South Sudan for at least 20 years.14

The spread of agricultural skills organically between beneficiaries and non-beneficiaries was reported anecdotally.15 The MTE examination of these skills among beneficiaries shows very low levels of adoption of most techniques, which would not support the hypothesis that the organic spreading of skills was widespread.

o Dykes were built, but there was not strong evidence of the intended benefit. Flooding continued to be an extremely common issue and in multiple locations the type of flooding could not be addressed through dykes. One case was reported where BRACE II built dykes diverting water which flooded non-beneficiary farmland.

o Roads were included as public works. Those roads were used and for a range of purposes. However, they were often not new roads but rehabilitations or widening of existing roads. As such it’s unclear to what change was made from pre-programme conditions.16

3.1.1 Why did some things work and only for some people?

All activities worked, although for certain groups of people or to a limited degree. The MTE was designed to provide an indication why change occurred or not. Different types of investigation would be necessary to further explain these changes. This section includes some the potential explanations, based on

14 The team leader worked on food security programmes in Northern Bahr el Ghazal in the 2000’s and those agriculture techniques were being taught at that time. 15 WFP cites that many beneficiaries are returnees from Sudan and have limited or no agricultural skills. The MTE data does not support this, as 3.9% of Component I respondents (through random sampling of beneficiary lists) identified as returnees. Such levels do not suggest that the project is systematically responding to the needs of returnees in the area, either because their presence is limited or because they are not selected through the beneficiary targeting (the MTE did not cover this aspect). 16 In at least one case, the road allowed a school to be connected to WFP’s school feeding programme. It is not clear if the road has changed enrolment or attendance. In another, the main benefit seems to have been the ability of the partner to monitor a block farm by car. In another, the constructed road led nowhere.


27

researcher observations and comments in their observation diaries, TPM key informant interviews, and comparisons with other baseline data and literature.

The agriculture activity packages, including input provision, seeds, labour payments, and group farm arrangements, worked very well for groundnut adopters. They also reported to motivated by the friendly competition on block farms, which got people showing off skills and determination. They were felt to have responded to programme incentives as the programme “opened up their eyes” to a new opportunity. They felt that groundnuts were a good pathway for them to increase their income. The vegetable farms had a similar effect on beneficiaries involved with those activities. It was also noted that there was a high level of correlation between groundnut adopters and vegetable farmers. In other words, this group who responded most strongly to the agriculture activity package were more likely high-potential farmers.

It was also observed that farmers rarely increased their land holding beyond a certain point (i.e. groundnut farmers on average tend 4 feddans). The programme expected farmers to increase their farms by a feddan per year, but even the groundnut adopters in the programme for three years increased land by a feddan or slightly more over the entire period of the project. It may be that within those production systems, there is a limit to how big they can or will grow.

NOTE 3: With regard to changes in value of production, four different relationships emerged: 1) constant groundnut farmers; 2) groundnut adopters; 3) groundnut dropouts; and 4) constant non-groundnut farmers. Disaggregating data using these four categories of relationships to groundnuts reflects a range of

1435

503

1060

494

1683 1740

499602

0

500

1000

1500

2000

Constant groundnut farmers(n=239/212)

Groundnut adopters(n=355/322)

Groundnut dropouts(n=25/22)

Non-groundnut farmers(n=220/185)

Value of production by farmer type (kg SE) (Component I)

Total harvest value before project Total harvest value at 2018

Figure 5: Total production value by farmer type

656

260

540

281

697569

249 325

0

500

1000

1500

2000

Constant groundnut farmers(n=275/196)

Groundnut adopters(n=195/165)

Groundnut dropouts(n=44/38)

Non-groundnut farmers(n=298/273)

Value of production by farmer type (kg SE) (Component II)

Total harvest value before project Total harvest value at 2018


28

relationships to mechanisms of agricultural change and suggest a role or influence of programme activities in these production outcomes. It appears that certain beneficiaries – groundnut adopters - had a strong positive relationship with the programme’s activities intended to increase production, for example. Equally, it appears that other beneficiaries did not. The two graphs in Figure 2: Total production value by farmer type summarise changes in total household production (measured in sorghum equivalent) by the four categories.

The amount of land cultivated increased for three of the four categories, and in levels that reflect programme activities and conditionality. Labour payments were conditioned on beneficiaries preparing, seeding and weeding one feddan in the group block farms. Fields in block farms were intended to be additional to the home and/or far farms of beneficiaries.

However, the majority of beneficiaries did not respond to these incentives to increase their production (Figure 5: Total production value by farmer type). Those who had already been growing groundnuts or sorghum with larger or smaller farms increased their farms enough to access programme inputs and income from labour opportunities. It led them to become involved in activities which consumed time, effort and lead to a decrease in yield per feddan.

The main incentive to increase land size was short term employment, referred to in the programme as conditional cash transfers. People were paid if they worked on their own farms, or on public assets such as ponds or roads. Cash disbursements did induce people to increase their farm size and the cash income was valued. It did not lead to an increase in production, however. There were isolated reports that some beneficiaries worked, were paid and gave the farm to others to tend and harvest. Also, the increase in land led to a decrease in yields per feddan. As a cash transfer it may have been effective (see Figure 6: Sources and changes in cash but the conditionality was counter-productive for many.

Perhaps part of the explanation of why some embraced the programme design and others did not may be that not every beneficiary is a farmer. Section 4.4 describes rural livelihoods more fully and from the perspective of resilience, and it makes the point that for most BRACE II beneficiaries farming is a contributing but not a main source of income. It is also higher risk. According to livelihoods baselines, (Muchomba & Sharp, 2006), many target beneficiaries depend more on labour, petty trade and fishing than they do farming. Other income streams may be more attractive. Figure 6: Sources and changes in cash, as a proxy for income, suggests that labour or cash transfer has made the biggest change. Income (cash) from farm surplus (agricultural sales) may be important, but for fewer people. Choosing not to comply with the BRACE II design may have been a rational decision. Lastly, due to the different understandings of “vulnerability” used to target beneficiaries, some may simply not be able to farm at all due to age, illness, physical condition, or due to the labour situation in their home. Rather they are

Figure 6: Sources and changes in cash

0.0%

20.0%

40.0%

60.0%

80.0%

Reasons for increased cash by beneficiary years (Component I)

Year I Year II Year III

NOTE: 2 Out of those who report more cash available, 65.5% attribute it to the programme supported labour opportunities (cash transfers) and only 26.9% to increased agricultural production (a total under 10% combined attribution to better market access, and other livelihood or wage labour opportunities). However, there is large variance between the Components, as the Component I respondents report 58.6% for cash transfers and 32.5% agricultural production versus Component II’s 82.7% cash transfers and 12.9% agricultural production.


29

vulnerable, but in a safety net meaning of the term.

A possible explanation for the success around issues of sharing and social cohesion may be that the programme simply helped people to share more than previously. Sharing is an act of fundamental importance in these communities. The programme may have facilitated greater sharing by people previously less able to do so.

Within these broad categories of what worked and what did not are a number of observations on individual components of activity packages:

▪ Some block farms were very productive while many others were not. The block farm itself did not consistently explain success or failure. Rather the motivation of the farmer was more explanatory. Motivated farmers may have valued the collective aspect of the block farm, but a reasonable question is whether the block farm was necessary for their success. It is not clear if access to land, a farmer’s motivation or both are required to increase levels of groundnut production. There were some block farms, for example in Twic (planted with sorghum), whose poor performance was due to the siting of the farm and not a reflection of farmer motivation. Those farms were flooded.

▪ Seeds may have been a helpful incentive to groundnut adopters, but they generally did not lead to a change in total household production.

Please refer to Annex 4: Performance Summary of Selected Outcomes, to see how mechanisms within individual outcomes performed.

3.1.2 Progress towards Logframe Impact?

This section has looked at the performance of individual outcomes and also how they have worked together to create the synergies envisioned in the programme design. These outcomes are the building blocks of any future impact.

BRACE II had set out performance benchmarks for the logframe’s final outcomes. Generally, and with the exception of increases in cereal production at the household level, the MTE reports good progress against these benchmarks in Component 1. There are differences in the results for Component 2 (discussed in section EQ2). Component 1’s performance against these benchmarks is largely explained by cash transfers and by increases in production, and for the groundnut adopters/commercial vegetable production possibly due to their increased income. Where the cash transfers explain the changes, those gains may not be sustained. As well, these benchmarks do not capture experience of groundnut and horticulture adopters nor the breadth of expenditures made with the cash transfers.

Component I’s progress against these benchmarks is important, but it may be disingenuous to imply it reflects the intended contribution or impact of BRACE II. While improvements in basic food security were desired and intended to be sustained, those outcomes are subordinate to the overarching purpose of the programme. The programme intended to make meaningful, sustained changes in how people generated their food, income and improvements to social cohesion.

BRACE II Outcome Target June 2018 MTE October 2018 Result17

Proportion of targeted households implementing crisis and emergency strategies, disaggregated by sex of household head (Livelihood coping strategy index)

50% reduction from Baseline: 71% all 85% male-headed 63% female-headed

53% reduction from Baseline – target met: 33% all 34% male-headed (60% reduction) 31% female-headed (50% reduction)

17 Baselines and targets only for Component I. As such, results also provided for Component I only.


30

Consumption-based Coping Strategies Index (reduced CSI): rCSI of targeted households, disaggregated by sex of household head

50% reduction from Baseline: 18.96 all 19.61 male-headed 18.65 female-headed

Indicator no longer present in the logframe at time of MTE. No data.

Prevalence of poor and borderline food consumption, disaggregated by sex of household head

30% reduction from Baseline: 92% (no gender-disaggregation available)

70% reduction from baseline – target met: 28% all 26% male-headed 32% female-headed

% of increase in the diet diversity score of targeted households, disaggregated by sex of households head

15% Increase from Baseline: 2.51 all 2.56 male-head 2.48 female-head

151% increase from baseline – target met:

6.24 all (149% increase)

6.22 male-headed (143% increase)

6.27 female-headed (153% increase)

% of increase in cereal production at household level, disaggregated by increase in production and cultivated area

No baseline. Target: 50% production 20% area

Cereal crops – target not met: 5% production Area unknown18 Total Farm production (cereal and non-cereal crops) – target met: 64% production 43% area

18 Issues with intercropping, unreliability of size estimations, and issues of recall prohibit conclusive statements on land size by specific crop.


31

Impact, in the programme design and theory of change, is intended to be a function of all intermediate and final outcomes being realised and working in concert. EQ1 indicates that in the best case, only some of the outcomes are being realised and for just less than half of the beneficiaries, primarily the groundnut adopters and vegetable farmers. Other objectives are being realised but not in ways intended by the programme, namely labour, which is useful for a number of reasons but not to increase production. Moreover, the decision by labourers not to expand farm production may well be justified, rational and unlikely to change. Some groundnut adopters and vegetable farmers may sustain their gains.

EQ2: Are there differences in performance between Components I & II and if so, why?

Broadly, similar trends in Intermediate Outcomes (IO) and Final Outcomes (FO) were seen in both Component I and Component II. Despite those similarities, the reported magnitude of change is significantly higher in Component I than Component II. There are a number of considerations for this difference:

1. The most obvious explanation is that Component II is in its first year of implementation. It was seen in Component I that there is a significant increase in performance against IOs from the first to second year of implementation. It seems there are a number of possible explanations for this increase. One relates to internal improvements (establishing the operational and administrative

Component I and II demonstrate broadly similar trends, but the degree of change is higher in Component I. At the mid-term, it is premature to draw any conclusion from these differences. Component I has been implemented for almost three years. Component II had been implemented for six months at the time of the MTE. As the MTE documents considerable improvement in performance against indicators from year 1 to 2, the final evaluation will provide a more meaningful assessment of differences between components.

A staff member of a BRACE II partner, who is implementing the programme, proudly said at the end of the MTE training, “Now I understand what we were supposed to be doing in BRACE II!”. An M&E researcher reacted and asked, “So then what have you been doing in the field?” The staff member replied sheepishly: “…well, the norm”.

Similarly, in several occasions during field work, researchers responded to requests by beneficiaries being interviewed to explain the programme.

One thing has become clear, frontline staff initially did not understand the project, or at least the language of it – though for some partners this has improved over the years of programming. The rhetoric of resilience programming can be baffling. For this reason, the revision of the logframe included intermediate outcomes, or rather outcomes which logically should be anticipated by the different packages of activities implemented by the programme. They were clear, measurable and attributable to programme activities. To some degree, incomprehensible outcomes may lead staff, charged with implementation, to simply focus on the “norm”, rather than the more ambitious intent of this resilience programme. The vague articulation of resilience may also explain the programmes emphasis on delivering outputs and not achievable outcomes.

Box 1: Resilience, terminology and confusion


32

systems), or a deepened understanding of programme activities by implementing staff. Another consideration is that it takes time for beneficiaries to understand the programme, its mechanics and to trust staff. Qualitative data suggests that Component I struggled heavily with these issues in their first year.

2. The farmers in Component II may be more vulnerable or poorer than those of Component I. This could explain lower farm production starting points and change observed in Component II. The initial Component II TPM visit (a Smile Again Africa Development Organization location) observed that many beneficiaries were elderly and unable to effectively cultivate their plots.

3. It may be that Component II has issues with its design or implementation not seen in Component I. This explanation could explain the lower levels of change of its farmers but not their lower starting points. However, the activities marking the main design differences between Component I and Component II have not yet been implemented (employment and savings and loans associations begin in year 2). The only significant difference in design and implementation noted so far, is that while Component I conducts an input trade fair for seed delivery, Component II distributes directly seeds directly. Component II failed to deliver groundnut seeds this year (most seeds were rotten by the time of delivery and could not be used) and this may have affected the adoption of groundnuts. Farmers said that they could have bought groundnuts locally, but as they were anticipating free NGO seeds, they chose to avoid the expense and wait for the free seeds.

4. Another possible explanation may relate to data collection, particularly for reported yield data. Beneficiaries in Component II may have been hesitant to accurately report yields out of concern they would be removed from the programme (a fear reported to have been present during the first year of Component I as well). As noted, it takes time for the programme to gain the trust of beneficiaries. This hypothesis would help explain both the lower starting points and lower levels of change for beneficiary farmers in Component II.

As the programmes are in very different phases of implementation, and with more time required to understand potential differences in data collection, one cannot meaningfully compare outcomes between Component I and II. As such, we would suggest not to draw conclusions at this point on the ultimate success or soundness of aspects of one component over the other. However, given that similar trends are emerging and highlighting the relevance of the theory of change, and programme elements for groundnut adopters versus non-groundnut adopters, there may be scope for Component II to usefully learn from Component I.

4 Programme Questions Section 4 looks at evaluation questions 3 and 4, emerging pathways to resilience, and progress toward outcomes and impact. This includes the programme contribution to those pathways and how that contribution is changing levels of climate resilience. While the benchmark of success for Section 3 was how outcomes progressed and work in concert according to the BRACE II logframe design, this section uses a different benchmark, that of beneficiary articulated pathways to resilience. This view of progress considers the programme from the perspective of beneficiaries and their developmental priorities. It also allows for reflection on the programme’s theory of change and hypotheses.

In addressing EQ3 and 4, this section effectively builds on progress documented in section 1, and considers the contribution of outcomes, even ones that have not worked according to the logframe design. The considerable data and analysis in the Compendium document “Detailed Analysis of Individual Outcomes at the Mid Term” provided insights into the performance of individual outcomes. Another important difference provided by this section is its inclusion of risk reduction, either reduced exposure to hazards or increased ability to cope with their impacts, necessary to assess changes in climate resilience.


33

EQ3 What are emerging pathways to resilience and how do we measure actual change and progress along those pathways (and for whom)? What are emerging, actual final outcomes or impacts?

EQs 1 and 2 examined the evidence of progress against logframe outcomes. In doing so, they offer a high level of reflection on effectiveness and impact. EQ3 proposes consideration of factors outside the logical framework. EQ3 could be viewed as an open evaluation question; it encourages the emergence of new detailed EQs about the nature of resilience during field consultation, and accepts that this is possible and perhaps helpful. As a result, it must also accept that the detailed EQs in the evaluation Terms of Reference (ToR) will not be comprehensive. Lastly, EQ3 recognises that it is not standard practice that new EQs are allowed to emerge, but it aims to ensure that this does happen.

The MTE uses EQ3 to consult with beneficiaries on other dimensions of resilience beyond those in the logframe. A defining feature of a resilience programme is its ability to sustain impact. By giving voice to those being served, it allows an additional perspective on assessments of impact and sustainability. It also appreciates that impact and sustainability owe much to process. Sustained and substantive change may require engaging with and addressing social norms, or underlying political or economic factors. Such perspectives view resilience as an intermediate outcome, not an end in itself but a step towards improving wellbeing. (Bene, Frankenburger, & Nelson, 2015)

EQ3 builds on that perspective of resilience to gauge the contribution made by BRACE II outcomes (discussed in EQs 1 and 2). It can gauge the role of BRACE II in resilience building in a way that has been used in DFIDs BRACED programme:

An emerging pathway to resilience can be framed around two branches; one aims to create enabling stability in the home, and the second aims to educate children so they may eventually find work elsewhere. The programme’s outcomes contributed towards that pathway, if not as envisioned at the outset of the programme. For groundnut adopters and more commercial horticulture farmers, their financial or food gains may have contributed to one or both of the branches along this pathway. Also, if cash transfers had been initially justified on a need to meet food expenditures, over the course of the year, beneficiaries report 40% to 45% of their expenditures are on health, education and essential household goods. For some, the cash transfers are considered particularly important to meet education costs and depending when the transfer is made, it may be entirely used for education. Social support relationships are intertwined with these two aims and BRACE II may be making a positive contribution. Sharing in the target communities is integral to identity and wellbeing. Without a baseline, the MTE is not able to determine if there has been a change in sharing. However, cash transfers provided by BRACE II were expected to be shared but also allowed people to share. It may be that a beneficiary’s involvement in the programme deepened their engagement in their social support relationships.

The overarching purpose of the MTE is to assess programme relevance. Beneficiary articulated aims or pathways provide an important benchmark. The success of groundnut adopters and horticulture farmers reflects well on the relevance of BRACE II support for them. For others, the cash transfer or labour component proved to be more relevant. Based on this, a reflection on BRACE II’s assumptions and targeting may be valuable (see recommendation 3). BRACE II assumes that its beneficiaries are farmers, but this is more often not the case. Its vulnerability selection criteria has led to the inclusion of people who are unable to farm or for whom increased farming is not a rational option (and who would be exposed to greater climate risk as a result). Most BRACE II beneficiaries are poor labourers, petty traders or fisher people and who require reliable income opportunities. Others, the elderly or ill, require unconditional assistance. A final reflection is that beneficiary children may need assistance to attend school. It is a transformative opportunity to which BRACE II is contributing, even if indirectly.


34

We also recognise that the BRACED programme plays a contributory role in what, for many people, will be an intergenerational process of building resilience. In other words, the changes (outcomes or results) of project activities should be intermediary steps in a longer-term, (strategic) process of transformation (Leavy, Boydell, McDowell, & Sladkova, 2018, p. 23)

EQ3 creates a space for emergent evaluation questions, driven by the perspectives of beneficiaries. Those perspectives provide us with a framework from which to consider the alignment and contribution of BRACE II outcomes. To answer EQ3 we first consider beneficiary articulated impact-pathways or “pathways to resilience”. Secondly, and relative to this beneficiary defined pathway, we also ask:

1. Is BRACE II making a contribution?

2. Is it a significant contribution?

Lastly, we reflect on how these contributions shape emerging final outcomes and impact, relative to these beneficiary defined perspectives of improved well-being.

4.1 Pathways and Milestones to Resilience

The term “pathway to resilience” is consistent with the logframe and OECD/DAC approach, but seeks to reframe them in a way which aligns with the aspirations of resilience programming. It fundamentally supports logframe outcomes and indicators, rather than replaces them.

Identification of these pathways began in TPMs conducted in 2018. In the TPMs, researchers verified that programme payments (cash transfers) were received. Interviewees also told researchers that money was used for food, health and education costs as well as other important household expenditures.

These anecdotal reports had implications: BRACE II had defined its resilience impact in terms of food security. Given expenditure patterns, it may only partially align to aims of its beneficiaries. Expenditures on health, education and other important household expenditures suggested a range of other objectives were being sought.

MTE results for both Components 1 and 2 observed that expenditures on immediate needs (i.e. food) in the first year of the programme were significantly higher (53% and 61% respectively) than expenditures on other categories. By year three that expenditure pattern was changing. For Component 1 beneficiaries, 49% of their expenditures were on food and 45% on health, education and clothing. For BRACE II beneficiaries who must purchase more food than they produce on their farms, this finding was unexpected.

Forcier researchers, based on their interviews, personal experience and observation diaries were asked to consider why such extremely poor people were making such a broad range of expenditures. In particular, as cash had been provided on the assumption that without it, they may not be able to feed themselves, it was important to know why all the money was not spent on food. The responses of researchers were consistent, and illustrated by these anecdotes:

…they said that if money comes at the times when schools are open, then they directly channel them (the money) towards school requirements. They said that they consider schooling of children to be investment on them (children) for future benefit. And this makes them appreciate much the support being given to them by NGO. The problem mostly reported by beneficiaries was that the money can't meet all school needs if one has more than one kid and make them end up not sending other children to school. Mawien Madut Awer, Forcier Researcher.

Reasons for prioritising school are because now many people have realised that it's a future investment, they have evidence through seeing the child of so and so went to school and lifted their family from the poverty and even they see the NGO staffs they wish that one day one's child will be able to do the same so they want to change the lives of their children not to be as hard as their own. Angelina G, Forcier Researcher.

These perspectives on education were particularly surprising. Education historically had relatively lower value in traditional, subsistent Dinka and Nuer homes. The shift described by Forcier researchers is however


35

consistent with trends seen in other rural communities in east Africa. It presents a different perspective of resilience beyond subsistence farming and food security.

These insights suggest aims and objectives of BRACE II households may be more complex than the programme assumptions allowed for. Based on these insights, impact pathways had both immediate and long-term aims that were inter-related. It led to the hypothesis of a two-pronged strategy or pathway to resilience. The first prong related to household aims for stability (likely to be achieved through existing income opportunities - a mix of labour, petty trade, farming, fishing, livestock for a few and maintaining social/reciprocal relations). The second prong aimed to see children seize opportunities outside of the village - employment or self-employment (to be achieved through education). Both prongs are mutually reinforcing. The first prong may be well conceptualised by BRACE II, but it must be under-stood in relation to the second.

This two-pronged hypothesis would bring together BRACE II’s diverse outcome performance in a single coherent framework. Using the mechanism of a pathway to resilience, outcomes are validated but casts within in a different narrative. This hypothesis could explain why labour opportunities were more attractive than more farming. It could help explain why more commercial forms of farming, i.e. groundnuts, performed better than subsistence forms of farming, i.e. sorghum. Lastly, it helps explain expenditure patterns. All of these outcomes are essential but subordinate to a more strategic aim, vested in the future success of their children.

In terms of impact, the aims of the two-pronged strategy particularly speak to transformation as a dimension of climate resilience, and the need to pursue policies that relate to power imbalances in society (Devereux & Sabates-Wheeler, 2004, p. 9). As observed in the BRACED evaluation, this strategy conceives of climate resilience intergenerationally. By virtue of transformative opportunities for children, and possibly their migration from rural villages, we may see i) a dramatic and perhaps permanent reduction in exposure to current climate hazards and consequences and ii) an improved capacity to cope and manage consequences of climate hazards. An important consideration for BRACE II, is that this pathway to intergenerational climate resilience is predicated on current, very fragile, livelihood options of the parents. This hypothesis is an interim assumption, developed to enable the MTE to answer EQ3 & 4. The hypothesis will be further examined in TPMs in 2019 as well as the final evaluation. For the MTE, this hypothesis will be used as a benchmark to gauge the relevance or alignment of BRACE II outcomes. It will also be the basis from which to gauge their contribution or rather their effectiveness or sustained impact.

4.2 BRACE II Contributions

Assessing the contribution of BRACE II towards this beneficiary articulated pathway to resilience was determined by asking two questions for each outcome:

1. Is it making a contribution to this notion of a pathway to resilience? 2. Is it a significant contribution?


36

The first question asks if BRACE II outcomes and changes are aligned to beneficiary goals. The second question examines the degree to which BRACE II may be contributing to those goals. A greater contribution and a more sustained impact is inferred from outcomes demonstrating five resilience dimensions (Figure 7). It is an approach developed by BRACED:

In ‘assessing’ whether or not the reported results are indeed ‘resilient’, we need to exercise a certain degree of professional judgment. These are not formal, quantifiable indicators, and an outcome or process may not ‘score’ well in one dimension but could still be considered to be resilient because it scores highly elsewhere. The dimensions of resilience are essentially a guide for examining the nature of the relationship between outcomes and resilience in a systematic and comprehensive way so that we can avoid inadvertently favouring some kinds of context or interventions over others, for example, projects focusing on easy-to-reach people starting from a higher base, compared to those implementing activities in fragile and/or crisis contexts.

Intermediate outcomes were re-configured as “changes” as some outcomes were different aspects of the same result. Below, Table 6: BRACE II Contributions to Resilience, summarises the contribution of BRACE II changes to beneficiary articulated goals. Changes are ranked and the darker colours indicate more positive results:

Table 6: BRACE II Contributions to Resilience

▪ Income for groundnut adopters is significant and can be sustained. Also, these farmers tended to also be the commercial vegetable farmers, making this change perhaps the most important

19 Risk informed refers to the degree to which either the change itself or the people involved in that change have reduced their exposure to probable hazards or increased their ability to cope. The analysis builds on the UNISDR notions of the relationship between exposure and vulnerability to a hazard versus one’s ability to cope with the consequences of the event. See https://www.unisdr.org/who-we-are/what-is-drr

Changes

Alignment to Beneficiary Goals

Is it a significant contribution (resilience dimensions)?19

Strength of Data

increased food and income as a result of increased farm production

Increased expenditures as a result of employment

Increased food and income as a result of increased vegetable production

Increased sharing due to increased food or income

Increased Conflict reduction due to new governance mechanisms

Increased trade due to new roads, or productivity due water control

Figure 7: Resilience dimensions Source: BRACED Final Evaluation

https://www.unisdr.org/who-we-are/what-is-drr

https://www.unisdr.org/who-we-are/what-is-drr


37

contributor towards the pathways to resilience. Increased sharing – which is a function of all income creating activities – may be an important form of crisis mitigation or management. Increased groundnut farming does however expose people more to climatic hazards and does not demonstrate actions that would either risk inform the production system or farmers themselves. To a certain degree that may be offset by their irrigated vegetable farms.

▪ Vegetable gardens when done at a scale provided a new source of income which may be significant and sustained. Also, where irrigation ponds are maintained and well managed, they can enable production through large parts of the dry season in a normal rainfall year. By virtue of irrigation, to a limited degree it reduces exposure of people to climatic hazards, but the activities do not demonstrate actions that would risk inform the production system or farmers themselves.

▪ Income from labour proved to be an important source of income and allowed priority expenditures to be made. It also moves people away from climate hazards or traditional hazards in the village and builds on an important pillar of their livelihoods. However, that income source will not be sustained.

The other changes, which were hypothesised to be integral to the success of the programme, in this analysis proved weakly aligned to the priorities and strategies of beneficiaries themselves. That is to say, that while the BRACE II activities were implemented, and progress toward the intended intermediary outcomes was achieved, the outcomes themselves were poorly aligned to beneficiary resilience pathways.

▪ BPMCs have played a positive role in establishing activities related to local issues and engaging with local governance structures (see Box 2: Targeting and chiefs). After the project however, the BPMCs are unlikely to be sustained.

▪ While multi-use ponds are integral to vegetable production, it is not clear what their value-added is to non-vegetable farmers or the wider community20. Similarly, the value added of roads or road/dams to beneficiaries or their income activities is not evident21. It is not clear if these investments will be maintained and ultimately sustained.

▪ Sharing may not have increased per se as a result of the project, but it appears that it possibly influenced how and the scale at which beneficiaries engaged in reciprocal sharing relationships. Those relationships are of enormous value to beneficiaries as a source of social cohesion, and

20 It is reported that cattle drank from these ponds. However, cattle would have been taken to water elsewhere in the absence of the programme provided pond. By allowing cattle to drink there is no indication of it has made a difference to animal production or productivity nor substantive value to the livestock management system. 21 It is claimed that the roads improved access to basic services and social interactions. No evidence was provided of how the existence of roads provided some demonstrable change from pre-programme conditions i.e. a change in clinic attendance, increased sharing, etc.

Involving chiefs in beneficiary selection as well as programmes played a role in BRACE II success, even if not as planned.

Many researchers came across situations where chiefs had been engaged in BPMCs at the outset of BRACE II activities. Implementing partners engaged chiefs as they believed their support to BPMCs was critical to the project. They provided final approval on the allocation of land for block farms, ponds or gardens, roads or dykes. It was implicitly understood that a chief would require remuneration in return for his support. For this reason, his wives or family members were often included as beneficiaries. There was no implication that this practise was considered nefarious nor was it abused. In fact, it was felt to either be just an operational reality or a creative solution that improved programme implementation.

A more problematic issue emerged in NBEG, where chiefs are mandated to collect state taxes from their constituents. Their constituents are determined by socio-political rather than geographic location. It means a few households in a boma are not represented by their local chief. As those households do not pay taxes to the local chief who does beneficiary selection, they will not be selected. Some chiefs may also give prefer to select those who reliably pay their taxes (or exclude those who have refused in the past).

Box 2: Targeting and chiefs


38

mutual help to manage crises and to make investments in the future. It is an area which could be investigated more deeply.

4.3 Emerging Final Outcomes and Impacts

Emerging final outcomes and impacts could be gauged by understanding how beneficiaries choose to spend their money. Final outcomes may include the completion of primary or secondary school, establishment of new sources of income, meeting health expenditures, as well as existing logframe final outcomes of improved HDDS or FCS. We are also seeing emergent, if not well understood, changes such as increased engagement in reciprocal sharing arrangements, and an improved sense of community security (see Figure 8: Beneficiary perceptions of change in community conflict since the start of BRACE). Better understandings of those changes might be valuable to capture BRACE II’s final outcomes as well.

At an impact level, they may simply be able to continue to achieve those outcomes over time, requiring them to be better able to adapt to changing conditions and protecting themselves from risks, whether climate related or not. A related impact indicator may be savings or progressive income growth. One of the differences between Components I and II is the inclusion of savings and loans associations, which speaks to these emerging final outcomes.

The analysis also highlights that risk can be reduced by the objectives sought by a programme, but equally the activities or outcomes must be risk informed (i.e. groundnut farms need to be able to access inputs to control for pest and disease) and individuals themselves must have contingencies to manage crisis (i.e. when the harvest fails completely, there needs to be a contingency for the family until next season).

A final note on pathways to resilience and resilient change is a consideration of sustainability and scale. The programme reached approximately 5 to 10% of the population in the counties where it was implemented (see Annex 6: Key demographics of beneficiaries). Perhaps 43% of those beneficiaries (groundnut adopters and commercial vegetable gardeners) or 1.5 to 3% of a county may sustain changes. That level of coverage and in the absence of system linkages is unlikely to make a population level difference (either in well-being outcomes for the county or adoption of new income streams). However, if we are to speak of “community resilience”, supporting 5 to 10% of the poorest households in a county, may be a reasonable threshold to provide stability in the county. For example, Tanzania’s Productive Social Safety Net programme has targeted 10% of the country (National Bureau of Statistics, 2016). Coverage would be that much higher in the sub-counties and bomas where the beneficiaries actually reside. However, that benefit to the county, sub-county or boma exists only for the life-span of the project.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Less likely No change More likely

Male-headed HH Female-headed HH

Figure 8: Beneficiary perceptions of change in community conflict since the start of BRACE


39

EQ4: Number of people whose resilience has been improved as a result of project support ICF-KPI 4

Assessing improved climate resilience builds on the analysis of EQ3 to understand where BRACE II has made a significant contribution to beneficiary developmental priorities (pathways to resilience). It further examines the risk informed elements of those contributions, looking for the degree to which they reduce exposure or increase coping specific to climatic hazards and related risks.

Depending on the degree of resilience demonstrated, they will be ranked (by stability, livelihood improvement and climate resilience). From this analysis, we can then compile the different levels of “resilience” which have been realised through the assistance of the project.

4.3 What is the climate related problem?

BRACE II provided general guidelines of how it understands climate resilience:

Disaster Resilience is the ability of countries, communities and households to manage change, in the face of shocks or stresses - such as earthquakes, drought or violent conflict - without compromising their long-term prospects. Climate adaptation will be central in the BRACE II approach through the combination of:

▪ addressing climate-related causes of food insecurity through interventions including flood protection, reversing environmental degradation and supporting development of more diverse livelihood options;

▪ increasing capacity and knowledge to deal with climatic variability; and

▪ diversifying water and irrigation options beyond rain-fed agriculture. (Dfid, 2018, p. 7)

EQ4 takes the pathways and contribution of BRACE II discussed in EQ3 and considers them specifically to climate related hazards. Perhaps 43% of BRACE II beneficiaries have improved climate resilience, and another 67% could be considered shock-proofed, (i.e. better able to manage a climate shock if not demonstrating a sustained change that reduce impacts of that shock). The vulnerability of BRACE II beneficiaries was seen to be a product of their poverty, exacerbated but not caused by climatic factors. Their two-branch resilience pathway, to a certain degree, recognises this situation. It has identified an option that can transform current exposure to hazards and risk. Educating children offers potential for more than climate adaptation. It may be considered an intergenerational strategy of climate resilience. Whether through the success of groundnut adopters, commercial vegetable producers or cash transfers, BRACE II helped meet immediate needs and as an unintended benefit, contributed towards longer-term household resilience strategies.

Long-term resilience strategies are however predicated on the sustained improvements of BRACE II beneficiaries. A sustained and improved income stream through groundnut or horticulture farming helps absorb climate shocks, but the degree to which it can be expected to absorb a shock is limited as they too will be impacted by the same shock. Greater emphasis on anticipatory mechanisms or linkages to wider value and marketing chains would make these income streams more risk informed. Cash transfer in exchange for work is an effective adaptive mechanism, reducing exposure to climatic hazards and increasing capacity to cope with their effects. These labour opportunities, however, will not be sustained.

There is an opportunity for BRACE II to help its beneficiaries risk-inform their work or family plans to manage transitory hardship. In addition to risk informing groundnut or horticulture farms, it could also help individuals to save, find short-term labour opportunities, or access remittances from family elsewhere. Contingencies can look beyond traditional coping strategies, towards emerging opportunities better aligned to achieving family goals.


40

Before we can assess the reduction in climate-related causes of food insecurity, clarity on the problem itself is required. The claims that climate is responsible for food insecurity and undermining long-term prospects needs a technical specificity. That specificity can allow us to measure the changes made by BRACE II outcomes. BRACE II documentation suggests a relationship between climatic factors, farming, and food production. The implications, in food security terms, may have been assumed to be a decrease in quantity and quality of food available. It is not clear what long-term prospects which are compromised.22

BRACE II beneficiaries have a complex relationship with climatic factors. Firstly, the majority of livelihood options of BRACE II beneficiaries are not directly exposed to climatic factors. According to livelihood and wealth group baselines (Muchomba & Sharp, 2006), BRACE II beneficiaries are drawn from poor and middle wealth groups. The poor meet 20% of their annual food needs from their farms, and homes from the middle wealth group up to 40%. The poor realise no income from their farms. Middle wealth group households realise 15% of their income from their farms. Labour is an important source of income for both wealth groups. Labour, together with fishing, wild foods collection and petty trade (and livestock the middle wealth group) are as or more important than farming for food and income. They are not directly impacted by climatic factors.

Secondly, losses in farms due to weather variability for these wealth groups are generally managed by an expansion of other food and income sources. Muchomba and Sharp highlight how the diversity of their traditional livelihoods is the foundation of their wellbeing and has proven robust enabling them to manage not only normal hardship but almost continuous war since the 1950’s23. These systems recognise the vulnerability of farming to rainfall or other hazards such as pest, disease or conflict. Of their livelihood components, perhaps farming is the most vulnerable. Crop losses can be accommodated by an expansion of fishing or wild food collection or increasingly, migratory labour.

Thirdly, and more broadly, it is not entirely accurate to assume farm production issues are climate related. Distinctions between agricultural drought and meteorological drought emphasis that distinction. Human factors, more than climatic factors may be the cause of the production problem. Crop selection, production systems, location of fields, soil types and many other factors determine the interaction of plant growth and rainfall both in quantity, intensity and temporal distribution. Pests and disease are equally important and if not managed will reduce production. Rainfall variation will exacerbate pre-conditions that are manmade or due to other factors.

Fourthly, food insecurity is not necessarily the result of production losses. The MTE observed several block farms in Twic County where the entire crop was lost to pluvial flooding. No crisis has been reported in Twic as a result of the loss of the block farms. Shock analysis using livelihood baseline would show such losses could be compensated by an expansion of fishing (which is better in high rainfall years), wild food collection, labour migration to Sudan or Aweil, or simply by increased production on farms situated on higher ground that benefitted from higher rainfall levels. IPC rankings of Twic may not change as a result of these flooded and failed farms.

Beyond farms and food, long-standing acute food insecurity in Northern Bahr el Ghazal is embedded in livelihoods systems, cultural norms and gender relations, not simply a function of variation in food or income factors. Climate is a factor, but not necessarily the direct cause. For example, one of the primary indicators used to indicate acute food insecurity in the IPC system is acute malnutrition. Gender division of labour and childcare norms have been shown to be more proximate causes of acute malnutrition (only measured in children under 5), than food security. Global acute malnutrition (GAM) levels double from wet season levels, and peak well before the hunger gap, and wealthy and poor children are equally affected (McDowell S. , Causes of Acute Malnutrition in South Sudan, 2007). Rates of GAM and severe acute malnutrition (SAM) remain unchanged from the 1990’s. Those rates continue to almost double from the

22 A review of WFP Annual Needs Assessments (and similar reports) going back to the 1990’s describes a situation the same or worse in NBEG or Magwi. The conditions may be poor, but they are chronic (i.e. stable). There is no historical evidence that current conditions, without external assistance, deteriorate and compromise long-term prospects. 23 The high population growth rate – excluding return – is a reflection of the robustness and success of their livelihood systems.


41

end of the rains to their peak late in the dry season. While the prevalence of GAM and SAM is unchanged, key food security factors have improved significantly, such as farm sizes, yields and total farm production, movement between towns and rural areas, roads, trade, labour migration with Sudan or elsewhere).

These high levels of acute food insecurity have continued have an ambiguous relationship with climate factors generally. Rainfall trends since 1900 have varied both in quantity and timing. Data indicates a total seasonal rainfall variation of 20% from mean (see Annex 5: Historical Rainfall for South Sudan).

Given the limited data, this variation could simply be a norm in South Sudan (i.e. it is normal to experience variability across inter-annual or inter-seasonal rainfall). Historic rainfall variation would have affected different components of BRACE II livelihood systems, including their farms – both positively and negatively – but without significant change in performance of key food security indicators.

In the future, the International Panel on Climate Change anticipates temperature increases of around 2 degrees for South Sudan by mid-century and doubling by the end of the century. Rainfall projections for the east Africa region broadly anticipate that by the end of the 21st century there will be a wetter climate with more intense wet seasons and less severe droughts during October-November-December and March-April-May Climates (Niang, 2014, p. 1210). While this suggests a reversal of an observed drying trend over the last 60 years, there is not yet consensus across different models of what might be expected from future rainfall trends or rather, South Sudan may experience an increase or decrease in rainfall as a result of climate change. Even if clear trends of a future climate are not yet established, there is consensus that more variability should be anticipated.

The role of climate, in acute food insecurity outcomes is complex, and subordinate to other more proximate causes. It may be more accurate to say that traditional livelihoods, cultural and gender norms, in which climate is embedded, are the sources of acute food insecurity. The implication is that climate resilience will be a function of substantive changes to those livelihoods and norms.

4.4 Pathways to Climate Resilience

The pathways to resilience, as identified by BRACE II beneficiaries, reflect a movement away from the cultural, traditional and climate related factors driving food security and poverty. The emphasis on education marks an important change. It implies that parents see a better future for their children and their grandchildren, in opportunities outside of the village. Educating a child is done with the expectation that the child will seek opportunity elsewhere and be of greater value to the family. In pastoral terms, it is referred to as the “expanding rangeland”, where household resources seek new opportunities in new lands (McDowell S. G., 2013).

If parents are investing in a generational transformation away from their traditional livelihoods, in the short-term, the MTE has no indication of what pathway parents are pursuing to ensure those inter-generational aspirations are met. Expenditure data did not indicate investments in progressive economic activities, which is referred to by some as “graduation”. The decision of many BRACE II parents not to increase their exposure to farming but to only pursue the paid labour is rational, and “climate informed.” Even if one branch of the pathway to resilience is transformative and can be climate resilient, the other branch upon which the first is predicated, remains fragile. Parents are not making the substantive changes to their livelihoods that may be valuable to protect their children’s future opportunities and ultimately the family’s climate resilience.

4.5 BRACE Contribution to Climate Resilience

If climate resilience for BRACE II beneficiaries is ultimately vested in their children’s ability to find new opportunities outside of the village, activities which make a meaningful, sustained and risk informed contribution to that would be very important. Secondly, activities which increase income, at a meaningful level and which are also risk informed, to provide stability and ensure children go to school, are be essential gains but of second order.


42

The risk reduction components of the changes in Table 6: BRACE II Contributions to Resilience were re-examined both for the degree to which they can claim to have made a contribution to educating children, helping families meet quotidian needs, and for how they demonstrate risk reduction to climate hazards. Those changes were weighed and ranked according to three Levels of Climate Resilience: Climate Resilience, Shock-proofed and Improved well-being. The most climate resilient changes are those making meaningful and sustained contribution towards both the transformative aspirations of beneficiaries as well as their immediate needs. Stability captures helpful but not sustained changes that serve immediate needs.

In order to report against KPI4, DFID must report the number of BRACE II beneficiaries who have had their resilience improved, which is done by weighting the number of beneficiaries in each level of resilient change: 1 for Climate resilience; 2/3 for Shock-proofed and 1/3 for Improved well-being can be used to count the number of people with “Increased Climate Resilience”.

A revised guidance on KPI4 was issued in December 2018/January 2019. It requires an analysis of climatic factors and how they come to bear on livelihoods or well-being, and the programme to specify the outcomes it will deliver to modify that relationship. It also requires that those outcomes are in labelled as adaptative, absorptive or anticipatory.

The revised KPI4 guidance and the approach used in the BRACE II MTE are fundamentally the same. Differences are complementary rather than substantive. Both are based on the BRACED programme, for which the 3As were central. The point of divergence is how outcomes are considered evidence of resilient change. The revised KPI4 guidance requires outcomes to be labelled. The principles of the 3As were retained by the BRACED evaluation team, but to meet their evaluation requirements the notions of 3A’s were expanded into resilience ‘considerations’ or dimensions. Notions of the 3As are specifically found in one of the five dimensions, the risk informed dimension. BRACED also required outcomes to demonstrate transformation to be considered resilient (it is one of the resilience dimensions). Transformation is not required in the revised KPI4 guidance. BRACED used a much higher threshold than is demanded than of KPI4. By adopting the BRACED evaluation approach, BRACE II effectively meets and exceeds the updated requirements of KPI4 reporting. Moreover, BRACE II goes further than even the requirements of BRACED by ranking outcomes according to their performance against the resilience dimensions. It recognises that there is a spectrum of resilient change likely to be made, even within a single programme.

Table 7: Reporting Against KPI4 summarises the results of the ranking. It ranks the change as one of the three levels of climate resilience, provides a brief explanation of why it has been ranked that way and lastly, proposes the number of beneficiaries who would be considered to have benefitted by that level of change:

Table 7: Reporting Against KPI4

Level of Climate Resilience

Changes in… Main Explanation for Classification Number of BRACE II Beneficiaries – direct/indirect

Climate Resilience

(generational change)

1. Additional income or food from increased farm production

2. Increased income from vegetable sales

Increased investments in stability, education and wealth that are likely to be sustained.

Vegetable production provides small levels of income/food, allowing limited expenditures or modest reduction in household expenses – but with the likelihood of sustained change.

43%


43

4.6 Underlying project hypotheses

As mentioned in the section on Evaluation and Analysis methodology, the MTE examines the current level of BRACE II hypotheses. To the degree possible, hypotheses have been drawn from those implicit in the programme design choices and logframe outcomes, including that they should be synergistic.

4.6.1 Partially true but only for some

Some elements of the BRACE II hypothesis - that people can be incentivised by providing seeds, block-farms, labour to increase the farm size leading to increased production, or value of production, that will be a sustainable source of income, enabling families make more expenditures on food - was shown to be true for 43% of beneficiaries. Encouraging people to invest more in farms in fact increases their exposure to climatic factors and the strategy of training to become “climate resilient”, while shown to be well conceived, proved ineffective in practise. Much can be done to address these issues in the final phase of the programme and it forms one of the recommendations – which must also consider the farmer him/herself can risk inform themselves and their families. There were a number of other elements, hypothesised to create important synergies such as a decrease conflict, and in increase in social cohesion, roads, dykes and ponds. Social cohesion and conflict management mechanisms existed prior to the programme and were not substantively changed by it. It is likely that the programme helped reinforce those mechanisms, which nonetheless is a valuable contribution if not integral to the adoption of new farming approaches by individuals. Benefits of roads and dykes were not evidenced as having an impact on the main mechanisms of increased food production or income. They were reported to be used for accessing basic services or to maintain social connections. No evidence was provided to indicate how these roads or dykes changed rates of basic service utilisation from pre-programme levels. Ponds were useful to vegetable farmers but will not provide “climate resilience” as they may help manage dry seasons, but ponds are equally affected by meteorological drought.

The programme activities were also expected to improve FCS (see Figure 9: Food Consumption Scores in detail & by component, below). They did this, with the change due to the purchase of meat and dairy. It

HHs Shock-proofed

(protect gains)

3. Increased income from new labour opportunities

4. Increased Sharing

Labour income was significant, allowing a range of expenditures, but it is not sustainable.

Sharing is potentially very important – but as much of it appears to be dependent on labour income which will not be sustained, it may only be a transitory change.

57%

HHs with improved wellbeing

(stabilise)

5. Increased ability to prevent and manage conflict

6. Public Works

7. Adoption of improved farming techniques

Helped implementation of BRACE II.

Benefits was limited to a small number of people.

No evidenced benefit from roads or dykes and unlikely to be sustained.

No evidence that the programme training changed techniques used by farmers. Also techniques in training were not associated with improvements by groundnut adopters or vegetable farmers.

0% - beneficiaries of these activities were involved in all other activity packages.


44

was not due to an increase in consumption of sorghum (from block farms) or vegetables (from kitchen gardens).

4.6.2 Some sub-hypotheses were true

BRACE II, while ostensibly an agriculture and economic development programme, equally had a social protection agenda. Part of that agenda was to provide subsidy for the economic inclusion of a marginalised population (the farmer component). It also wanted that same cohort to have access to more cash. The cash was meant to both incentivise and enable larger farms as well as to help people cope. Secondly, it hypothesised that they would spend the money on food. All of these sub-hypotheses proved to be true. They may not have used the opportunity to change their farms, but that may have simply been a rational response. Cash, principally from employment, was used on food and possibly sharing. Significant amounts of cash were spent on health and education.

4.6.3 Some Sub-hypotheses were not true

Equally, sub-hypotheses or rather assumptions of what was necessary were not evidenced by the MTE:

▪ Increased production was not achieved through increasing farm size. Farm sizes were increased, but productivity decreased. There was an opportunity cost associated with expanding farms. That cost has not been fully explored.

▪ Of those who reported gains in the value of the farm production, success factors perhaps had more to do with the support, working collectively (block farms) and the cash from labour.

▪ The framing of risk did not include actual risks such as the need to have provision to manage idiosyncratic risks, or contingency plans when farms or vegetable gardens fail.

NOTE 5: The large differences in scores are driven almost exclusively by the consumption of meat and dairy, which Component I beneficiaries consume more than twice as often. Because of the high score value of the two food groups (each day consumed is multiplied by four), the differences become extremely large.

Basically no differences exist in staples, legumes, vegetables, fruits or oil. The difference in sweets is large in quantity, but almost irrelevant due to its low factor in the FCS (factor of 0.5). All in all, the consumption pattern differences are consistent with Component I beneficiaries being more well-to-do, whether this is caused by the project or otherwise.

0.0

2.0

4.0

6.0

8.0

10.0

12.0

0.0

2.0

4.0

6.0

8.0

10.0

12.0

FCS in detail

Component I Food group eaten

Component II Food group eaten

Component II Value

Component I Value

Figure 9: Food Consumption Scores in detail & by component


45

Regression analysis can shed some further light into the potential mechanisms through which the project may have contributed to the FCS. The main statistically significant (p<.5) determinants of FCS scores emerge as being a constant groundnut farmer, value of the harvest, income/expenses, being part of reciprocal social networks, cultivating vegetables commercially or having a business, receiving remittances, and being a year 2-3 beneficiary (though no significant difference was recorded between years 2 and 3, only against first year beneficiaries). Importantly, gender differences disappear when controlling for these factors, suggesting that it is better explained by male-headed households having higher yields and a higher likelihood of being in reciprocal networks rather than by gender itself. A negative gender effect for Male Headed Households (MHHHs) actually exists for dietary diversity, though on aggregate this is negated by the positive effects of higher yields

The BRACE II theory of change posited a number of problem statements and assumptions (Dfid, 2018, p. 9) which were not evidenced in the MTE:

▪ Agricultural and natural resource management practices not adapted to climate challenges (see Figure 10: Application of agricultural practices)

▪ Large scale return of populations since 2005 who have lost or do not have agricultural knowledge and skills

▪ Divided communities lacking social cohesion to work together on common challenges ▪ Inter-community based conflict resulting from competition for depleted resources ▪ Lack of transport infrastructure and value chain market development

Concerns that the programme may exacerbate relations between men and women were also not borne out. Rather, the programme may have played a positive role within those relations whether in the family or wider community.

Figure 10: Application of agricultural practices


46

NOTE 3: Error! Not a valid bookmark self-reference. summarise the results from the household survey. The results provide the percentage of the sample that apply the practice and in which fields (only counting those who currently cultivate those land types). No skill has more than 2/3 coverage with weeding and topping at 66% and 64% spread respectively. Weeding is clearly considered by the majority as an important technique, as evidenced by its almost universal application across all farm types (in only block farms is it monetarily incentivised). However, given that farmers are paid to weed on their block farms fields, it would have been expected to be more prevalent. Skills that require either large monetary or labour investment, such as manuring, ridge planting, tree planting, and animal ploughing, are least reported.

CLOSER LOOK: Assumptions that a lack of access to farmland was a source cause of food insecurity

The BRACE II project design of increasing agricultural production through the increase of land cultivated has strong implications for land rights that are worth discussing in detail. For every boma where programming takes place, each beneficiary should theoretically clear and cultivate three additional feddans of land. On average this would require the project to identify and negotiate access to around 230 hectares of land within the boma. The block farm approach further complicates the process in this regard, as land plots need to be large enough to accommodate a group of beneficiaries. Horticulture similarly requires access to land, though at a much smaller scale as a beneficiary requires only 400m2 rather than a feddan (4,200m2).

The responsibility for land identification and access negotiations tend to fall on the shoulders of chiefs who are expected to provide the service on a voluntary basis. The vast majority of block farms in the former states of NBeG and Warrap are established on communal land that was previously not used or under-utilised. BRACE partners have generally negotiated a five-year memorandum of understanding (MoU) with the communities for the block farm use. Anecdotal cases of people being pushed out of their far farms to make space for the block farms were found.24 In Magwi, this dynamic is different due to higher population density, relying more on individually owned land where BRACE II block farm access will be limited to the programme’s duration, according to the TPM visit. Vegetable plots additionally require a water source for irrigation.

There are a few important implications to this approach to land that are relevant to the project’s aims and outcomes:

▪ The land is communally owned and right of use does not translate to ownership. This likely contributes to the finding that particularly skills that contribute to the longevity of the land are very rare in the block farms. The prevailing perception within the beneficiaries is that the plots will not be fertile beyond the MoU regardless of rights to use, and although many like farming in a group, the block farms themselves are unlikely to be sustained much beyond the project period;

▪ Block farms on the larger end (around 100 feddans) were noted to have issues with freeloaders, particularly around fencing. In cases where these issues cannot be solved, crops remain at risk of being destroyed by animals. It also means that farmers are unlikely to use these fields beyond the programme’s incentivised periods;

▪ In several locations, the sizes of block farms were not as large as they had been reported. It is not clear if the under-reporting is due to land availability or difficulties measuring. Also, within block farms, TPM 11 was able to measure individual, demarcated feddans. They were substantially smaller than 60m*70m. In one location, it was reported new land for a block farm had been cleared but the field was shown to be the same as the previous year. The MTE data for Component I shows that only a third of year 3 beneficiaries increased their farms by two feddans. Most, after 3 years, only increased their farms by one feddan. Overall, there appears to be a systematic issue mitigating an increase in farm sizes. Explanations might be a lack of labour or a lack of suitable land;

24 In one case, a person had spent significant amounts of time and resources building up a piece of land through manuring, increasing its fertility. It seems he was given beneficiary status and access to a piece of this land within the block farm for accepting the block farm in the area. In another, a person who was not a long-standing member of the community was driven out of the plot to make space.


47

▪ Areas of unused or underutilised land are likely to have been left for a reason. They may be low-lying and prone to flooding; they may not have ready water sources or they may have poor soils. However, ironically one soil type of unused or under used land (sandy soil) inadvertently led to the largest project success – the adoption of groundnuts at scale (groundnuts are grown in sandy soils, often found in gok or higher lands, that are often drier);

▪ For vegetable gardening, available land is even more limited, as it requires water inputs. Most vegetable plots were found around existing ponds that were created during the construction of main roads. These were not ponds created by the programme. New ponds created by the programme, inasmuch as they hold water in quantities equal to the demands of horticulture farmers and for periods required for plant maturation, do increase the availability of suitable land for horticulture.

4.6.4 Rethinking the hypothesis

The findings in this section also raise questions about fundamental assumptions underpinning the programme hypothesis:

▪ Rural South Sudanese are farmers ▪ Even modified traditional farming is a pathway to climate resilience ▪ Food Security is resilience

Rather, given the livelihoods systems of this wealth group of beneficiaries it is a very rational response to choose to not be a farmer. Labour may be a more appropriate options for many in the target communities.

These fundamental distinctions also bring into question how people were identified for the programme. Vulnerability was a main criterion for selection and TPMs have noted that vulnerability is understood differently by different implementing organisations. Based on the success of groundnut farmers and some vegetable farmers, some of the right or intended people were identified. However, vulnerability, the conceptual descriptions of the project and basic assumptions made it difficult be able to categorise people effectively for the programme. Targeting was framed more in social protection terms or more specifically safety net/welfare or seasonal employment terms, yet activities heavily emphasised farming skills and motivation to farm. It may explain why progress against logframe outcomes was observed for some but not all beneficiaries. It is an interesting alternate explanation where much of the programme design was in fact useful, only that the confusion over targeting – economic growth versus safety net – led to the targeting of a large number of people inappropriate for the programmes intended outcomes. Perhaps an important factor determining why the programme did not perform as anticipated is because of targeting.

Also, this perspective provides a reflection on the potential value of segmenting beneficiaries according to different policy objectives and related instruments more appropriate to their situation see (Rawlings, 2015) (World Bank, 2018). It would also help the programme to better align its expectations i.e. sustainability, economic graduation versus stability, relative to the different policy objectives.

CLOSER LOOK: Social Protection in BRACE II

Examining BRACE II “conditional cash transfers” (CCT) or “cash for assets” (CFA) with a social protection lens provides a different perspective on BRACE II outcomes. BRACE II CCT has three functions: labour guarantee, public works construction, and basic safety net provision. These three components are generally intended to work in concert by providing transfer income in return for work, and secondly through the construction of infrastructure that supports economic activity (DFID, 2011, p. 51). BRACE II provides additional in-kind support to farmers (e.g. seeds, tools and advice) to improve agricultural outcomes. Considering expenditures made on health and education, in part as a result of the cash transfer, CCT may also be thought of as a de facto subsidy for access to education or health care.

Using a Transformative Social Protection categorisation can help clarify how BRACE II actions would align to different beneficiary populations and social protection policy objectives (Devereux & Sabates-Wheeler,


48

2004)25. Transformative, promotive, preventive and protective are the four broad categories used in Transformative Social Protection. Importantly for BRACE II, transformative social protection shares a similar conceptualisation of transformation as resilience in DFID’s literature.26 This conceptualisation of transformation also resonates well with the developmental context of South Sudan27.

Within this transformative social protection framework, BRACE II’s objectives and activities can be seen to serve a number of different social protection purposes and populations. Some BRACE II cash payments would be categorised as protective and would include: old age pensions (persons over 65), access to health services (extremely poor households) or basic social safety net (chronically ill, orphans, widows, indigent). BRACE II cash provided in return for labour, would also be categorised as a protective action and would be referred to as a rural employment guarantee (landless labourers or extremely poor farmers). BRACE II support to small farmers, a package of conditional cash transfer payments including in kind commodities and training, might be categorised as a promotive or market intervention for rural economic inclusion. Promotive assistance of this nature is generally provided to increase the chance of graduation from poverty (DFID, 2011, p. 56). A final category might be the de facto subsidy for access to education, which may be considered transformative if it sought to substantively change the social and economic situations of impoverished rural children.

BRACE II’s targeting criteria, mechanism and its food security objective can help explain why BRACE II activities were considered to work for some and not for others. The same activities, using a social protection framework, may be considered to have worked more widely. BRACE II uses a “vulnerability” or food insecurity criteria and a community-based targeting mechanism. BRACE II’s CCT assumes that the vulnerable people it identifies can or wish to work. Secondly, it assumes identified persons can or wish to use the asset to be built28. BRACE II’s vulnerability criteria however has a broad range of understandings29. Not all vulnerable people identified were aligned to the demands or expectations of the CCT. It encompassed aged, ill, able-bodied but poor individuals as well as a group of high potential farmers.

BRACE II targeting did work, if not efficiently. 43% of the beneficiaries it targeted advanced towards a modest level of graduation.30 The other 57% of beneficiaries who did not report this economic improvement could be considered inclusion errors (i.e. individuals who should not have been included). However, EQ3 reports that most of the 57% did benefit from programme activities. Support to them might be considered a success against wider social protection metrics, if not the BRACE II CCT. Targeting for social protection on the other hand employs a range of different criteria and targeting mechanisms, to align beneficiary populations to activities and policy objectives.

The CCT also faces a number of challenges, beyond identification of persons appropriate to its activities and purpose. Firstly, conditional programmes are considered complex and challenging to implement31. Secondly, results from well run programmes are difficult to distinguish from the results of non-conditional programmes32. Moreover, beyond a discussion of mechanism, the issue which BRACE II was trying to confront is structural, extreme rural poverty. As such, there is space for a promotive (i.e. graduation) oriented programme such as BRACE II. Also, with regard to transformation, there is space to consider the de facto indirect subsidy for education. Most of the BRACE II caseload may be better served by actual social protection mechanisms, ones grounded in institutions and providing reliable and appropriate forms of support. Given conditions in South Sudan, it may be some time before these institutions and services emerge. In the meantime, it may be useful to disentangle humanitarian forms of targeting and objectives

25 The World Bank is revising the categorisation of social protection actions in its ASPIRE initiative. While ASPIRE is a technical advancement on the classification of social protection policy and actions, it omits notions of social investments to transform impoverished segments of society, captured in Transformative Social Protection. 26 A socio-economic transformation fundamentally changing power relationships see (Bahadur, et al., 2015) or (Bahadur & Tanner, 2014) 27 Social protection as applied in Rwanda illustrates a “transformative social protection” agenda applied in policy and practise. 28 It also assumes the asset will function as designed. See (Ludi, Levine, & McCord, 2016) 29 See (Smith, 2014, pp. 69-70) 30 See (DFID, 2011, p. 19) 31 See (DFID, 2011) or (Adato & Hoddinott, 2010) 32 See (Giles, 2014) or (Huntington, 2010)


49

from those of social protection, and better align different forms of assistance to different groups of people, opening the door for greater relevance, efficiency and potentially sustained impact.

4.7 Additional Learning Questions

The MTE uses its findings to provide reflections on the following additional questions or areas of interest. Defining learning points is a challenge to BRACE II, as it touches on so many important themes and issues. The process of preparing for the MTE led to a focus on basing learning around changes made by the programme and understanding why they have taken place (see Box 3: Third Party M&E, and learning).

4.7.1 Participation of and Benefit to Women

Some of the reasons for specific questions on the participation and benefit of the programme for women arose from concerns about how providing cash to women might alter household relations. Few problems of this nature were reported in either TPMs or the MTE.

There were also concerns about the ability of women to control the income they earned from working. Again, there were no reports of problems relating to this, though women in MHHH would bring their income as part of the household income of which they do not have sole decision-making capacity. Moreover, the MTE reported that cash income was an important contributor to increased expenditures on food, basic household requirements, health and children’s education.

There were reports of programme activities which were felt to be onerous: labour was difficult for elderly, ill or those with a limiting physical condition or disability, and distances to some block farms was considerable. These were issues for both men and women, but disproportionately for women due to cultural norms on types of work, household and childcare duties. Furthermore, block farms that were far from homes have important gendered implications. For those with small children, it lessens the amount of time they have for childcare, a factor closely associated with chronic malnutrition. It was also noted in multiple occasions that mothers would bring young children with them to the block farms where they end up working.

Female-headed households were systematically worse-off than male-headed households on most IO/FO measures. However, it is important to note two factors. Firstly, the change in IOs that had a constructed baseline showed that the level to which female-headed households and male-headed households had benefited by the project were similar – i.e. there was no gender effect on the level of change. Secondly, female-headed households correlated strongly with single-headed households in general in their status and performance. It is thus unclear to what degree the issue is a gender issue rather than a single parent issue, though being a single parent itself is more common for females due to male mortality (17% of females widowed vs 3% male). Most likely, this reflects reduced labour capacity at the household level.

When third party monitoring is effective, a contracting client can not only contract for specific tasks but can tap into experience translating programmatic aims into metrics that are measurable and attributable to the programme activities. It can provide access to resources committed to designing and managing high quality field work. The results can deliver evidence which in turn can answer basic questions on the design and implementation of the programming. As those questions are answered and used to change design or modify work, information generated transforms into learning.

The MTE process highlighted a range of strengths and weaknesses in internal partner M&E resources and practice. There are high performing M&E resources, but designed to serve internal purposes and questions, and so encounter challenges in addressing wider programmatic questions. Other systems are designed to address programmatic questions, but lack the capacity to design and field the necessary field research in spite of a high level of willingness.

The MTE process has led to high level s of collaboration and brought with it a range of skills and depth of ability. The evidence provided by the MTE addresses the M&E questions, and provides recommendations relating to adaptive management and learning.

Box 3: Third Party M&E, and learning


50

The programme also made important if unanticipated contributions to women. Researchers reported how cash from the programme helped women to improve their situation in the home, and to change how they were perceived by the community and led to new opportunities. They could access credit and deepened their ability to pay taxes and help others. It meant they were not classified as “nyop” or someone who cannot care for themselves, and a liability and collective reasonability of the community. Men also provided similar testimonials.

4.7.2 Does the provision of work for longer or shorter periods make a difference?

Component I supports short term labour for 3 years (6 months x 3 years) and Component II for just over a year (6 months + 2 months). While there is an important change in income and expenditures from year one to two, it then stabilises and little change is seen in year three.

▪ Expenditures by expenditure type do not differ significantly over time. ▪ There was no investment in “graduation” or investments to provide long term economic

betterment, i.e. livestock, new businesses or migration.

Employment was also provided for multiple years with the expectation that it would be invested to continue increasing farm size, i.e. households would add a feddan each year while continuing to farm the previously cultivated land. Farms sizes were increased to one feddan on block farms in the first year but not systematically beyond this point, despite labour payments in year two or three (it is worth noting that year 3 beneficiaries of Component I began the project by being paid to cultivate their household land and as such no block farms were created in 2016). Providing labour opportunities to these farmers past the second year did not change the amount produced nor how it is produced.

4.7.3 Reflections on the OECD-DAC Criteria

Discussions of the OECD-DAC criteria lead to a nuanced picture of BRACE II. Its different performance metrics reflect on a range of strengths and limitation of the programme design and its results.

Both relevance and effectiveness are discussed fully in sections EQ1 and EQ2. The potentially sustained and significant changes being made by groundnut adopters reflects well on the relevance of the programme and their priorities. In contrast, the same activities were less relevant and effective to the majority whose agricultural output showed no change despite an increase in their farm size. The programme’s ability to increase income, whether through agricultural growth or more simply the conditional cash transfers (guaranteed labour schemes), are well aligned to household priorities to not only feed themselves but to meet health costs and ensure their children access education. Given that the changes for groundnut adopters were made as an unintended benefit, reconsidering the effectiveness of those agricultural activities would be useful. Lastly, the cash transfers proved to be an effective mechanism to modify food consumption or dietary diversity scores. That those changes will not be sustained diminishes their effectiveness. Ultimately, their effectiveness must be measured against the objectives of a resilience programme rather than a humanitarian, food security programme.

Narrow efficiency analysis (unit costs) are captured outside of the MTE. For example, WFP’s BRACE II value for money analysis indicates that it is more cost efficient to provide cash than food. These are important reflections with potential operational implications. The MTE has identified another aspect of efficiency central to BRACE II. There is an opportunity-cost to beneficiaries of working on the programme’s block farms, in order to access the cash transfer. The MTE has shown that beneficiaries valued the cash. However, by working on the block farms, many were not able to adequately look after their other fields (which may have led to the decrease in farm productivity). Future conditional cash transfers or guaranteed labour schemes may wish to factor in opportunity-cost to their design, to better appraise the value-add of conditionality.


51

Assessment of impact is found in the resilience analysis of the MTE. The MTE noted that there is positive progress towards the logframe’s impact indicators. The means of making those changes provides valuable reflections on their sustainability. Changes that were predicated on cash transfer is transitory and will not be sustained. Impact which is the product of income generated through the adoption of groundnuts, may well be sustained.

5 Summary & Recommendations 5.1 Summary

The changes sought by BRACE II are evidenced by the mid-term evaluation. Approximately 43% of its beneficiaries are increasing farm size, production, yield and with the potential for sustainable increased income. These are beneficiaries who have adopted groundnut production and to a lesser degree, horticulture. Other beneficiaries increased their farm size but without a change in their total farm production. Mid-term results suggest their yields have decreased.

These bifurcated results led to reflection on why the programme helps some but not others. Design issues rather than implementation factors can explain the diverse results. Specifically, key assumptions and targeting mechanisms help explain results. It was assumed that individuals desired improved subsistence through increased production of staples, i.e. sorghum but rather it was more commercially oriented ground nut or horticulture production that saw increases. It was assumed that households had the labour necessary to increase their farms but many did not. It also assumed that the risks associated with agriculture or increased exposure to climatic hazards was tenable or even desirable by beneficiaries. However, beneficiaries in the programme for three years increased their farm size by only one feddan (on average), rather than three as per the programme design. The choice to limit increases in farm size contrasts with the positive response to the programme’s employment component. It was assumed that conditional payments of cash would lead to graduation or increased, sustained farm production. Cash transfers likely helped the groundnut and horticulture adopters, but issues such as motivation or labour availability may more likely explain their production changes. For the others in the programme, they worked on block farms or public works and earned additional income, but conditionality did not lead them to increase their farm production. A final consideration relates to targeting or rather the targeting criteria. By using a vulnerability criterion, the programme implicitly assumed homogeneity or rather that all rural people were farmers and wished to farm more. That assumption and criteria led to inclusion errors, or rather included people who could not or would not farm more.

Notwithstanding these assumptions, the broad net used to identify beneficiaries did include a significant proportion of high-potential farmers who responded to the programme’s activities and incentives. The broad net also identified people unable to farm, unable to farm more or for whom farming more may not have been rational nor desirable. Even if this group did not expand their production, they did respond positively to the programme’s labour component or cash transfers. At the mid-term, both of these results reinforce the programme’s emphasis on cash and income. Increases in income or at least cash was associated with expenditures to meet short and long-term objectives; greater sharing and for some, improved status. Enabling households to invest more in health, education and children at the same time as meeting immediate needs may reflect how BRACE II is contributing to climate resilience. The fundamental challenge for BRACE II is that unlike changes in production, betterment through employment in the programme’s works activities will not be sustained.

This led to the following reflections on central programme assumptions or hypotheses:

▪ Rural people are not necessarily farmers. Farming is a complementary food and income activity for beneficiaries. It has a higher risk than other income options. Other income opportunities may be preferable.


52

▪ While all BRACE II activities may fall under a social protection umbrella, they seek very different policy objectives – basic safety net versus graduation - and speak to different categories of person. The bifurcated MTE results may simply reflect a problem associated with targeting based on vulnerability, which makes no distinction on the appropriateness for labour intensive, agriculture-based activities versus those requiring unconditional assistance.

▪ Resilience, for BRACE II beneficiaries, looks past subsistence and food security to education and opportunities for children beyond the village.

▪ Climate may not be a direct cause of food insecurity, but it remains an important factor in the lives of BRACE II beneficiaries. Much can be done to risk inform BRACE II results as well as to help beneficiaries plan and have contingencies for themselves.

The trends we see emerging at the mid-term have important implications for the remainder of Component I or II and future programme designs:

▪ Component I: There is little scope for modification to the work under Component 1 as the programme has or soon will be finishing in those locations. However, where time allows, implementers could look for opportunities to enhance relations between farmers and input or financial service providers (where they exist). These relations could contribute towards farm sustainability or improved ability to weather crisis. Component 1 may also wish to relook at its theory of change and metrics of success given the mixed results with regards to agriculture performance.

▪ Component II: Component II has scope to incorporate mid-term learning. For its farmers, it may wish to consider strengthening market/value chain and financial services linkages (where they exist). The discussion on targeting and mixed agriculture results might warrant strengthening the Component’s savings and micro-business elements to better serve those for whom increased agriculture activities are not appropriate.

▪ Future Programme Design: MTE lessons around improving agricultural production (i.e. land as a limiting factor), targeting, social protection options and practical application of resilience principles should be of use to DFID and BRACE II partners to inform the design of future programmes.

The M&E Team discussed the findings from the MTE and their implications for agriculture and resilience with DFID in March 2019. DFID also held sessions at that time for the M&E team to share learning with BRACE II partners and the South Sudan donor community. These plans and any further agreed final learning activities constitute the M&E Team’s Use and Influence Plan within the remaining contract period. DFID South Sudan will determine the extent to which the MTE findings are to be used and how, both within BRACE II and to inform future programme design.

Without modification to the programme, the MTE does paint a picture of what we might expect in the final evaluation: that the programme has worked partially and well for perhaps 43% of its beneficiaries. The others will still have benefitted, but not in the ways foreseen by the programme nor necessarily in sustainable ways. The MTE and its approach can provide the baselines from which the changes can be better ascertained and understood in the FE. Below are recommendations for BRACE II to consider, based on the MTE results.

5.2 Resilient pathways and climate resilient change

Supporting BRACE II beneficiaries to achieve their own objectives, which would both see them manage current challenges as well as support a generational transformation through their children, is not limited to agricultural support. Moreover, where that support seeks to make risk informed change and embed itself in the private sector or social systems offers a deeper support with great opportunity for sustained change.

BRACE II can consider:


53

▪ Deepening its support to farmers entering more commercially based forms of crop or vegetable production. It can make them more resilient by building linkages with wider systems. It may consider linkages to value chain actors to ensure a commercial supply of inputs and goods, and to better position these farmers to sell into local markets, for example. Similar considerations can be made for business people to be supported by Component II.

▪ Incorporating complex land tenure, labour and water access issues into future appraisals of agriculture support programmes or labour /public works schemes (i.e. food for assets).

▪ Risk informing farms or businesses (i.e. contingencies for how a farm or business will manage drought, conflict, etc.) as well the individual farmers, businesspeople or labourers (i.e. preparing contingency in the form of savings, labour migration plans, plans to access remittances, etc.) in times of crop failure, loss of employment, etc.

▪ Economic graduation or alternate options for economic growth of parents can be explored e.g. as with micro-entrepreneurs in Component II.

▪ Working with business or other donors to create seasonal employment opportunities.

5.3 Social protection

Social protection can be part of a resilience building approach. Continuing to frame BRACE II as a resilience programme and incorporating social protection approaches may help broaden programme relevance to more BRACE II beneficiaries. It may better align beneficiaries with activity packages and objectives.

BRACE II can consider reformulating its existing activities to include:

▪ Basic safety net (unconditional, protective assistance) for defined categories of beneficiaries (i.e. older people, chronically ill, orphans, widows or indigent).

▪ Short-term employment (preventive assistance) at key points in the year to assist the able but very poor, and not necessarily linked to a “public asset”. This could be planned around other seasonal activities and gender informed, so as to make a minimal disruption to a mother’s childcare responsibilities (e.g. creches near the worksite).

▪ Promotive forms of assistance, such as reimbursement of health costs (or insurance), assistance to develop new income opportunities, or to link to financial services (an extension of activities in Component II).

▪ Support poor families to meet education costs, or to have their youth gain employment in urban areas (transformative support).

▪ Exploring different vehicles to make investments in public works, such as roads or water reserves, into which it can provide subsidised labour, so taking responsibility for the labour component but not the design, prioritisation and maintenance of the public good.

5.4 Governance and social mechanisms

Although the MTE was not able to quantify changes in security or social cohesion, it contributed in valuable ways.


▪ Formalising current BPMC linkages to local authorities, and reporting programme progress and learning through those governance mechanisms while increasing their stake in the success of BRACE II individuals and households.

▪ Extend current governance work to build local partnerships to support towards the climate resilience objectives – education of children, stability in their homes and in the homes of the poorest. Exploring opportunities to adapt traditional sharing mechanisms or local taxes could be a


54

means for local communities to control and sustain their own initiatives for equitable development or protection.

▪ Understand better how increased liquidity has helped poor people engage in reciprocal relationships and why that is important.

5.5 Resilience versus Humanitarian programming

The BRACE II approach is delivering sustainable livelihood changes and contributing to the ability of its beneficiaries to achieve their own long- and short-term objectives, something not seen in traditional humanitarian approaches.

BRACE II can consider ways in which it can further distinguish itself from traditional humanitarian approaches by:

▪ Using social protection approaches to show how it is possible for a single programme to respond to diverse conditions and opportunities within a single population. It can use absolute categories (i.e. age, physical condition, chronic illness, indigent, sex) or clearly demarcated socio-economic categories (i.e. chronic, extreme poverty and able, poor and stable but unable to meet unexpected expenditures, high economic potential but lacking skills) for targeting rather than “vulnerability” assessments. It can also re-categorise its programmes under protective, preventive and promotive policy objectives, to align categories of populations with programme instruments and their policy objectives (which can also help clarify challenges with the theory of change).

▪ Leading discussion with humanitarian donors and agencies to inform their understandings of rural resilience and share BRACE II learning on how one can make a contribution towards it.

▪ Conducting a simple VfM study on discrete, evidenced benefits (intermediate outcomes) against expenditures to generate those benefits.

▪ In the BRACE II learning agenda, examine the opportunity-cost of working on block-farms, with attendant land tenure, labour, and water access issues, against other income or labour opportunities.

5.6 Internal BRACE II

5.6.1 Contracting Approaches

Given that Component II has not completed its first year of implementation, it is premature to assess differences between the two components. Also, it is important to ensure that data collection is consistent across both Components.


▪ Deferring the evaluation of the different contracting approaches between Component I and II until the final evaluation.

▪ Continue new longer-term contracting models that reduce the likelihood of prolonged contract negotiations causing delays to programming in the beginning of the critical cultivation period.

5.6.2 Theory of Change & Logframe

The programme faced challenges as much of the theory of change language was not translated into clearly understood, demonstrable programmatic features and goals, and also included assumptions which should be revised:


▪ Revising the theory of change based on clear, measurable interpretations of terms such as resilience, climate risk, climate resilience, vulnerability, food insecurity, and climate related causes


55

of food insecurity. The revision could also include a more specific focus regarding skills development and which skills should be emphasised and promoted.

▪ Rethinking key assumptions such as that increased farm size results in increased production; and conducting an opportunity-cost analysis, particularly for women, comparing the approach of increasing land under production via block farms versus an expansion of other, non-farm related sources of income.

▪ Revising the logframe intermediate indicator IO7 from “Increase in cereal production at household level, disaggregated by increase in production and cultivated area (MT)” to “Increase in the value of total crop production at household level, disaggregated by increase in production and cultivated area (MT)” to properly capture the contribution of groundnuts to programme results, and differences in crop values not reflected by weights.

▪ Change the impact indicators to better reflect actual pathways to resilience and ones more likely to reflect the value addition of BRACE II outcomes.


56

6 Works Cited Dfid. (2018). Building Resilience through Asset Creation and Enhancement Phase Two. South Sudan.

London: Dfid.

Fielding, W., Sharp, B., Gullick, C., & Coutts, P. (2001). An Introduction to the Food Economy Research in Southern Sudan 1994-2000. Nairobi: WFP & Save the Children (UK).

Funnel, S., & Rogers, P. (2012). Purposeful Program Theory: Effective Use of Logic Models and Theories of Change. San Franciso: Jossey-Bass/Wiley.

Harigan, S., & Chol, C. (1998). The Southern Sudan Vulnerabilty Study. Nairobi: Save the Children Fund (UK).

IMC Worldwide. (2017). BRACE II Inception Report. Redhill: IMC Worldwide.

Leavy, J., Boydell, E., McDowell, S., & Sladkova, B. (2018). Resilience Results: BRACED final evaluation. Brighton: ITAD.

Ludi, S., Levine, S., & McCord, A. (2016). Assessing the Livelihoods Impact of NRM PWP Assets. In R. Beazley, A. McCord, & A. Solórzano (Ed.), Public works programmes for protection and climate resilience: theory of change and evidence in low-income countries. Brasilia: IPCIG.

McCord, A. (2016). Methodologies for Assessing the Medium Term Livelihoods Impact of Public Works Assets. Human Science Research Council. Pretoria: ODI.

McDowell, S. (2007). Causes of Acute Malnutrition in South Sudan. Nairobi: CARE South Sudan.

McDowell, S. G. (2013). Change in the Arid Lands: The expanding Rangeland. Nairobi: Save the Children, Oxfam & British Red Cross.

McDowell, S., Goodman, R., Fitzgibbon, C., & Wallis, C. (2017). Final Report. London: DAI Europe.

Muchomba, E., & Sharp, B. (2006). Southern Sudan Livelihood Zone Profiles. Juba: Southern Sudan Centre for Statistics & Evaluation.

National Bureau of Statistics. (2016). Tanzania's Productive Social Safety Net: Findings from the Evaluatino Baseline Survey. Dar es Salaam: GoUT.

Niang, I. O. (2014). Africa. In V. C. Barros (Ed.), Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part B: Regional Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge, United Kingdom: Cambridge University Press.

Rawlings, L. (2015). Overview of social protection: pension core course. . Washington, D.C.: World Bank.

Schipper, L., & Langston, L. (2015). A Comparative Overview of Resilience Measurement Frameworks. London: ODI.

Westhorp, G. (2016). Realist Impact Evaluation: An Introduction. London: ODI.

Wilson-Grau, R., & Britt, H. (2012). Outcome Harvesting . Cairo: Ford Foundation.

World Bank. (2018). The State of Social Safety Nets. Washington: World Bank.


57

Annex I: BRACE II Logframe Summary Impact, Final & Intermediate Outcomes

Impact: Improved food security and better community relationships within the targeted

communities & Partners' community feedback mechanisms established

I1: Reduced prevalence of moderately and severely food insecure households (in counties where

BRACE II is implemented)

I2: Resilience Index Measurement and analysis (RIMA II) (in counties where BRACE II is implemented)

I3 CESAM

Final Outcome: Improved food security and better community relationships

FO1: Prevalence of poor and borderline food consumption, disaggregated by sex of household head

(Food Consumption Group)

FO2: Diet diversity score of targeted households, disaggregated by sex of households head (HDDS)

FO3: Proportion of targeted households implementing crisis and emergency strategies,

disaggregated by sex of household head (Livelihoods CSI)

FO4: Reduced vulnerability to climate risks and shocks

FO5: Reduced vulnerability to communal conflict

Intermediate Outcome: Increased capacity to absorb, anticipate and adapt to shocks and stresses

(including climate variability and extremes)

IO1: Number of people with improved resilience (ICF indicator)

IO2: Proportion of targeted households using good nutrition practices promoted in BRACE trainings

IO3: Proportion of targeted households using improved and climate-sensitive agricultural practices

IO4: Proportion of households giving or receiving support from other households (Social cohesion)

IO5: Proportion of targeted communities where there is evidence of functional structures for

conflict resolution

IO6: Increase in incomes at household level

IO7: Increase in cereal production at household level, disaggregated by increase in production and

cultivated area (Metric Tonne)

IO8: Increase in number of food / incomes sources at households level


58

Annex 2: BRACE II Theory of Change SOURCE: DRAFT Feb 2018: BRACE II Partners Meeting

SOURCE: DRAFT October 2017 BRACE Partners Meeting


59

Annex 3: Detail of Data Gathering TPM Qualitative Interviews

The number of interviewees by type are as follows:

All data (apart from one Project Manager (PM) interview conducted by the DTL during a field visit) were conducted by Forcier national researchers between July 2017 and November 2018 (rounds 2-4 between April and November 2018)

Table of Summarising Farms Assessed

Details of Assumptions included in Researcher Diaries

11 key hypotheses derived from the theory of change and EQs:

1. The project has increased the production of food for beneficiaries;

2. The project has increased income for beneficiaries; 3. The project has made beneficiaries work together more than they used to (social cohesion); 4. The project has contributed to peace in the community; 5. The project has helped beneficiaries create sustainable livelihoods; 6. The project has reduced the mentality of dependency; 7. The change created by the project has contributed to a long-term improvement in the quality of

life; 8. The project is targeting the right people; 9. The project is addressing the main obstacles to food production/agriculture; 10. The project has led to negative impacts on the beneficiaries or communities; and 11. The 3rd year beneficiaries are now resilient.

Determining the value of agricultural production

To establish a kilogram yield per crop, it was decided to allow respondents to report their yield using the unit they were most comfortable with and apply the weight measurements in the analysis phase. This was done in order to maximise the reliability of the data, avoiding calculation errors that respondents or enumerators could make converting known measures to kilograms. For each of the volume measures, the kilogram amount was known for sorghum (50kg for a bag, 3.5kg for a malua, 17.5kg for a safia, etc.). Other crops were thus calculated based on their litre to kilogram ratio in relation to sorghum.

33 Data is complete for all crops for the MTE apart from cowpea, which had data missing for 13 months over the 46-month period. The effects of missing data are small as cowpea was not a common crop amongst the beneficiaries. 34 Grain bulk density factors provided by AgriTechTalk Africa.

In-depth Interview type Interview number

Beneficiaries 42

Committee members 10

Local authorities (chiefs, boma administrators) 8

Project staff 16

Total 76

Pre & Post-Harvest Assessment

Component I Beneficiaries 81

Non-beneficiaries 68

Component II Beneficiaries

Non-beneficiaries

Total 149

Crop Value ratio33

Weight ratio34


60

Knowing the yield per crop in kilograms enables creating a value index centred again around the most common crop – sorghum. The data used is monthly market data from 2015-18 at the main Aweil market. The value of each crop is compared at each of these points to sorghum, and the overall average ratio is recorded and applied to each crop yield for analysis.

The value ratios and weights for each crop are summarised in Table 2. To correctly read the changes in the value of agricultural production, it is thus important to understand that the unit of change is one kilogram of sorghum. I.e. a reported change in value of 600 is equivalent to 600kg of sorghum. To change it to a kg yield of any other crop, the value needs to be divided by the value ratio. The SSP and dollar value can be estimated at a value at a given point by multiplying the kg with the SSP/kg for that month.

35 Unshelled weight and price are used for groundnut as this is the form in which respondents store the crop. This was confirmed in the household tool.

Sorghum 1 1

Maize 1.02 .99

Groundnut35 3.90 .41

Sesame 4.89 .81

Cowpea 1.44 1.03

Beans 5.01 1.04

Millet 1.57 0.86


61

Annex 4: Performance Summary of Selected Outcomes This summary is drawn from the additional MTE compendium, Detailed Analysis of Individual Outcomes at the Mid Term

Activity What worked What didn’t work

Cash ▪ In relation to the main project objective or food security, cash transfers were consistently used to purchase food that positively affected both the quantity and availability of food – even if the precise impact of the cash cannot be measured from the overall project effect;

▪ Beyond food, the cash was used strategically by beneficiaries to improve their quality of life through access to basic services – particularly health and education. The former contributes significantly to long-term resilience as it reduces the likelihood of deaths within the household as well as the impact of illnesses on the household productive capacity. The latter is a strategic investment into the family future, as more educated children have a higher variety of livelihood opportunities available – thus improving generational resilience;

▪ Some cases were found of cash transfers being used to invest in further productive capacity. These included seed money for businesses such as tea shops, roasting groundnuts, buying goods in bulk and selling at retail (for example, sugar). Some beneficiaries also used the money to invest in agriculture, particularly through hiring labour to clear and cultivate more land;

▪ Cash transfers were also used for dignity products, such as beds, clothing, serving trays, etc. Although not directly relevant to food security or productive capacity, these types of items had a significant impact on the self-perception and social standing of the recipients;

▪ Due to the strong cultural norms for sharing, it is extremely common for beneficiaries to give out some of the cash transfer particularly to family connections. Only around 17% overall had not engaged in any type of

▪ The cash transfers had some implementation issues, particularly delays in the early stages of the project. These created large amounts of distrust and caused in some anecdotal cases beneficiaries to temporarily stop work on the incentivised assets;

▪ The determination of the SSP value (pegged to a central dollar value) was also not always well understood by beneficiaries who saw slight changes in their SSP entitlements, at times attributing this to corruption with the cooperating partner. The issue is particularly acute with Component II as WFP was able to negotiate a preferential rate leading to divergence between what the beneficiaries of each Component received. Although they do not live in the same bomas, communication between the Components’ beneficiaries led to suspicions among WV beneficiaries that the partner was stealing the difference;

▪ Some recipients of cash transfers perceived that they were working for the cooperating partner. This is a potential challenge mainly for the household-level productive assets (farmland and vegetable plots), as these require strong buy-in to be sustained;

▪ Some beneficiaries expressed hopes and expectations that the cash transfers would continue, not fully understanding their strongly time-bound nature. This may cause issues with spending behaviour as those who expect support to continue may be less likely to invest in productive capacity;

▪ In cases where investments in productive capacity are not made, the better quality of life is unlikely to be sustained, limiting the utility of the cash transfer to temporary welfare;


62

giving in the past 3 months (though this is not exclusive to cash transfers);

▪ As a secondary effect of the cash transfers, trust was created towards the recipients, which allowed them to access services and purchase goods on credit. This was highly beneficial for them as particularly illnesses are not predictable, and their treatment is highly time-sensitive;

▪ The predictability given by the cash transfers was also noted to have a strong effect on the mindsets of beneficiaries. Those that would suffer from the constant stress of trying to find the next meal could instead take stock of their situation and think more strategically of their next steps. This furthermore reduced the tensions particularly between spouses on how meagre resources were spent, which was said to positively contribute to reductions in domestic violence;

▪ For those who lived in abject poverty, the cash transfers were particularly meaningful. Due to the strong norms of helping those in need, these people who were constantly in need were considered as liabilities and shunned for fear of being ordered to support them (hunger courts). With the cash transfers, these individuals no longer required support but could be contributing members of the community;

▪ By allowing those previously in constant need to become contributing members of the community, those people also report having more voice in the community matters as a whole, thus improving political inclusion.

▪ Due to its universal application, cash transfers are highly desirable. As such, they can cause tensions in relation to targeting. Sharing with other community members was mentioned by some beneficiaries as a way to mitigate these tensions.

The objective of the cash transfer was to make them work to create both household and community assets, as well as provide resources that could be used to stabilise the household. In this respect, the cash transfers worked effectively, even going beyond meeting the immediate need of stabilised food security.

The benefits were particularly large for those households living in abject poverty, where the cash allowed for vastly improved critical service delivery on credit, as well as affects social relationships both within the household and in the society writ large. In this way, it acts as social protection or welfare for those that lack the ability to produce sufficient quantities of food or marketable labour to ensure their basic needs are met. It is also reported to have a transformative effect on a poverty mindset, opening up space for thinking strategically and reducing particularly intra-household tensions and conflict.


63

The main limitation of the cash transfer is that for those households that have not been able to increase their productive capacity, the effects are likely to fade away after the cash transfers cease. This eventuality has the highest likelihood amongst those households that needed the cash transfer and benefited most from them (i.e. the poorest).

Therefore, cash transfers did contribute to resilience and a dignified life during the project period with some potential to contribute to generational resilience. Its impacts were wider reaching than the project’s logic would necessitate, and as a social protection tool it was highly successful. However, the sustainability of its impact is inherently tied to the success of the other project activities, namely increasing household productive capacity through agriculture or horticulture. It is also questionable whether the cash transfer is the most appropriate modality for creating strong buy-in for productive assets, as they are often perceived particularly in the early stages of the project to belong to the organisation rather than the beneficiaries. It can thus help enable resilient change, but in itself has mainly a temporary effect.

Land cultivation

▪ Highly significant improvements in food production were reported by around 30-40% of beneficiaries, who were disproportionately comprised of beneficiaries that adopted farming groundnut at scale during the project period (for full details on production, see IO2);

▪ Increases in cultivated land were reached with the average change of 1.13 feddans across the beneficiaries;

▪ More beneficiaries are selling their crop than before the project, suggesting larger surpluses being produced. The figure has risen from 35% to 43% across the project;

▪ The increased value in harvest correlates positively with quality of life indicators and wealth of the household, suggesting that farming has significantly contributed to the overall conditions of those who got higher yields, including food consumption, access to services, and investments in wealth (mainly livestock);

▪ Assembling the additional farmland on block farms had some benefits for ease of monitoring as well as beneficiary interaction, where working in the same location was said to foster both positive relationships and a healthy competitive spirit where individuals wanted to show they were hard workers;

▪ Researchers noted that both beneficiaries and communities felt positively about their ability to produce their own food and reduce reliance on aid.

▪ Large numbers of people only farm the amounts incentivised, as over 50% of 2nd are 3rd year beneficiaries only cultivated 1 feddan of block farm this year;

▪ Apart from those who changed their farming system to include the higher value groundnut crop saw decreases in their productivity (yield per feddan), which negated a large portion of the production of the additional feddan. Productivity per feddan remains largely below county averages;

▪ Some beneficiaries related that they let someone else farm their block farm. In this case, the beneficiary would retain the cash transfer and the farmer the harvest. According to the researchers, these were mainly 1) old people who could not do the labour, and 2) those who feel like they have enough and aren’t convinced of the added benefit of farming more;

▪ The sustainability of the block farms beyond the project period is questionable given 1) that substantial numbers of people have only farmed the feddan that was tied to the year’s (first point above) cash transfers; 2) the low levels of techniques that maintain or develop the land applied at block farms (see IO2b); and 3) lack of clear ownership by beneficiaries, coupled with unclear formal structures to support block farm management;

▪ Techniques are altogether not widely adopted with even the highest (and incentivised) technique of weeding practiced by only


64

65% of the beneficiaries. Lack of belief that the added labour demands of applying the technique justifies the added value (or in some cases absolute lack of labour) seems to be the most relevant obstacle to wider adoption;

▪ For Component II, seed distributions failed due to delays and rot. It is generally questionable whether continued seed distributions are vital for production particularly after the first year, as seed storage and market access in the areas in question are possible.

▪ Beneficiaries continue to suffer from predictable shocks, such as short dry spells, flooding, pests, and weeds. The project has as of yet not succeeded in building sufficient resilience against these. It seeks to actively address flooding, but at least in some locations with dykes, they do not affect the type of flooding experienced.

Land cultivation was designed to contribute to food security and resilience through increased production of food. The implied assumption of the activity is that the issue with cultivation is access to land. This assumption was by and large not born out by the data, which suggests that the median land size for beneficiaries before the project was already 2 feddans, which should be sufficient for any household farming for sustenance. Only 25% of beneficiaries reported farming 1 feddan or less prior to the project (only 2% under 1 feddan).

Regardless of the assumption regarding land access and food production, incentivising cultivation did lead to a net aggregate increase in crop production across the project. However, the gains were driven mainly by those people who adopted groundnuts and moving from food production mainly for conception towards more commercial farming and were negligible or even negative for a substantial number of beneficiaries.

The pathway to resilience through increasing production thus works only for a part of beneficiaries – those both willing and able to do farming beyond subsistence levels. Those beneficiaries have seen substantial development that feeds into their resilience through multiple mechanisms. Those beneficiaries who do not have this capacity or will may temporarily increase the size of the land that they farm in order to get the universally useful cash transfers, but this does not significantly increase their total production due to productivity losses, and is unlikely to be sustained beyond the project. There are some indications that positive examples of success (social proof) may motivate others to produce more, as year 2-3 beneficiaries have higher production than first year beneficiaries.

MTE results indicate that a substantial percentage, around 42% are in the latter category of people who have not seen significant benefits in the agricultural production (less than 50% and less than 180kg SE increase), whereas around 37% have seen highly significant benefits (over double and 500kg SE increase). The remaining fifth have seen only modest increases that are either small in relation to their existing production or small in absolute figures. For these households, it is unclear whether the opportunity cost of the time spent cultivating the additional land is covered by the increased production. Those who have benefited in large absolute quantities seem to be transitioning to better wealth status, having higher likelihoods of being able to make longer term investments in items, including cattle.


65

Vegetable cultivation

▪ Vegetable cultivation amongst beneficiaries has grown significantly, from 29% of households to 43% of households;

▪ Simultaneously, the level to which vegetable patches are irrigated beyond the dry season has grown from 62% to 78%. Multi-purpose ponds are likely to have contributed to this development;

▪ Diversity of vegetable crops also increased;

▪ Vegetables as a source of income also increased from 14% to 26%, contributing thus not only to household food consumption. 64% of those who currently farm as an income source have reached this level during the project, the rest 36% were doing so before the project began.

▪ Vegetable production remains small within the population. Unless it reaches the scale of becoming a source of income, it has no impact on the household quality of life indicators;

▪ Availability of irrigatable land remains limited despite the project efforts to create more irrigation sources. This caps the potential of scaling up vegetable production;

▪ Pests remain a significant issue for horticulture, as does the lack of seed multiplication for other than the core traditionally cultivated vegetables.

Horticulture was designed as an alternative household productive activity to block farms. More commonly, it has been done as an additional activity, as most beneficiaries report having been primarily supported for crop production. Through the production of vegetables, contributions should be made to food consumption as well as incomes, insofar as some product could be sold.

The MTE data shows clear increases in the levels of horticulture among the beneficiaries, but shows impact on quality of life indicators only insofar as it is done at a scale of being an income source rather than pure subsistence. However, it is possible that it would contribute more to FCS and HDDS if the data collection was done during the dry season thanks to the large level of irrigated vegetable patches. Furthermore, vegetables only receive a factor of 1 in the FCS, and as such are a priori unlikely to make substantial contributions to it. Nonetheless, in the cases where commercial scale is reached, clear correlations were found on indicators relating to wealth accumulation and increased income, such as investment in durable goods and livestock, and consumption of meat and dairy.


66

Annex 5: Historical Rainfall for South Sudan The graphs are provided by the World Bank, Climate Change Knowledge Portal http://sdwebx.worldbank.org/climateportal/?page=country_historical_climate&ThisRegion=Africa&ThisCCode=SSD. Data for the graphs was produced by the Climatic Research Unit (CRU) of University of East Anglia (UEA).

Figure 11: Rainfall 1991-2015 Figure 12: Rainfall 1961-1990

Figure 13: Rainfall 1931-1960 Figure 14: Rainfall 1901-1930

http://sdwebx.worldbank.org/climateportal/?page=country_historical_climate&ThisRegion=Africa&ThisCCode=SSD

http://sdwebx.worldbank.org/climateportal/?page=country_historical_climate&ThisRegion=Africa&ThisCCode=SSD


67

Annex 6: Key demographics of beneficiaries

State (former)

County Pop Sept '18 (IPC)

Beneficiary Reach (HH)

Beneficiary reach (HHx6)

Beneficiary reach (%)

NBeG Aweil

Centre 109,954 1,776 10,656 10%

NBeG Aweil East 550,956 5,067 30,402 6%

NBeG Aweil North

280,284 3,062 18,372 7%

NBeG Aweil South

149,848 2,919 17,514 12%

NBeG Aweil West 310,919 3,616 21,696 7%

Warrap Gogrial West

340,952

Warrap Twic 425,538 3,168 19,008 4%

EE Magwi 185,028


68

IMC Worldwide Ltd 64-68 London Road Redhill RH1 1LG Tel: +44 (0)1737 231 400 Fax: +44 (0)1737 771 107

building resilience through asset creation and enhancement ......rima resilience index measurement...

Documents