Monitoring and Evaluating the Food Security and Nutrition Effects of

Agricultural Projects

F. James Levinson and Anna Herforth

Purpose

Monitoring and evaluation of ag projects seeking to address Ag2Nut considerations in addition to their generally primary production objectives is important:

• because there is, to date, so little empirical data documenting successes and failures; and

• because the possibility of adverse effects is a real one – and such effects need to be identified and addressed rapidly.

Constraints

To date, however, Ag2Nut issues, while of great interest to many in the nutrition community and to a growing number of agriculturalists, has not yet captured the attention of agriculture project managers.

Ag Manager Ag2Nut Expert

Problems in Ag M&E

Meanwhile, international agriculture project have had their own problems with M&E.

An FAO/World Bank study (2010) found the M&E of agriculture projects in Asia, Africa and Latin America to be seriously problematic.

Among the most common problems:

Common shortcomings of Ag M&E • M&E is often perceived as an externally imposed

obligation with findings seldom integrated into management and action systems

• Ag managers complain of unmanageable data collection and reporting demands

• What M&E is carried out gives primary attention to physical achievements to the neglect of project outcomes

• Inadequate institutional capacity consistently limit M&E

(All of these, of course, are problems common to M&E in development projects more generally.)

Yet we want to see Ag2Nut considerations included in Ag projects, we want to see them monitored and evaluated, and we want to be able to document successes

Recognizing:

• The seriousness of existing Ag M&E problems; and

• The traditionally lower prioritization of nutrition-related issues in agricultural planning

Might there be value in examining alternative avenues rather than seeking to incorporate Ag2Nut in mainline ag project management information systems?

• Ag project managers would likely feel less encumbered

• Ag2Nut M&E staff could focus on the quality of these M&E efforts and their utilization;

• and, in the process, provide Ag project managers with invaluable up-to-date information on their projects.

With a more circumscribed Ag2Nut M&E posture

Ag2Nut Expert

Sentinel Sites

A useful approach to M&E in ag projects with Ag2Nut interests may be the establishment of geographically representative sentinel sites within the overall project area with large enough samples to have statistical power – all subject to understandings presented later in this presentation.

Baseline data would be followed by the collection of quantitative and qualitative data at 6 month intervals from these sites, and from comparable sites in non-project areas (serving as controls.)

Sentinel Sites

This data would include: • Basic socio-demographic information • Information indicating the extent to which households have been

reached/affected by the project • Data on household food insecurity levels and on the diversity of food

consumption • Data on malnutrition among young children and women (collected

annually) • Women’s empowerment data (quantitative and qualitative) • Information taken from the above which might indicate harmful food

security or nutrition effects • Data on a subset of essential data of primary interest to project

managers All would be complemented by qualitative data to better understand the dynamics of project effects

Assuring Quality and Establishing Successful Precedents, and Moving

Toward Sustainability But: • All of the above requires M&E skills not always

readily available to such programs Our suggestion: • With external support, skilled Ag2Nut M&E

teams, working with local ag staff in initial projects would: – Create much needed initial successes, and – Develop prototypes, training modules and TA

mechanisms for subsequent utilization

Data to be Collected

Have households had been

reached/affected by the project? • If the project involves the distribution of

inputs or the provision of subsidies, have families received them?

• If the project seeks to generate employment, have previously un- or underemployed individuals been employed?

Data on HH food insecurity and on the diversity of HH food consumption

• In the case of most agriculture projects, positive

effects on household food consumption may be more readily accomplished than effects on child malnutrition levels.

• Agriculturalists are also more likely to be able to relate to these effects.

• One useful, although sometimes challenging way to assess household food insecurity is using standardized measurement instruments that have been validated for a region.

Household Food Insecurity Measuring Instruments

• A model for such measurement is the 15 item standardized household food insecurity measurement scale recently validated for all of Latin America by FAO together with academic affiliates.

• Comparable scales are required for sub-Saharan Africa and for South Asia.

Data on household food insecurity levels

A second useful way is to utilize a household dietary diversity score assessing the diversity of foods (often the forgotten element of food security) consumed by household members on the previous day.

Assessing food diversity also can be coupled with qualitative data to provide invaluable insights to guide future food security-oriented action.

Dietary Diversity Scores

• Household level – indicates food access (an indicator of food security)

• Individual level (woman and/or child) – indicates dietary quality

• Can be measured using qualitative 24-hour recall (FAO 2011) or a food group questionnaire (FANTA 2006), depending on the interviewer skills and level of information sought.

Data on child malnutrition levels

• While recognizing that effects on child stunting may take time to show from agricultural projects (unless coupled with explicit nutrition counseling by health staff), it may be useful to measure the height for age of children under age 2 on an annual basis.

• These data also could show synergies between multisectoral project components (e.g. does diet quality + WASH have a greater effect on nutritional status than either alone?)

• But malnutrition levels must be analyzed via impact pathways – the specific means by which the project services are affecting child height for age.

Data on reproductive-age women

• In many countries the nutritional status of reproductive age women is critical both for themselves and for their offspring.

• At the same time, the nutrition transition problems of overweight and obesity are becoming increasingly serious in many low income countries.

• To track both issues, BMI data will be collected on reproductive-age women, assessing percentages below 18.5 and over 25.0.

Identifying harmful food security or nutrition effects

The above information, collected at six monthly intervals is likely to help identify harmful effects resulting from the project. Special attention should be given to results indicating that, compared to the control area: • Employment levels have remained static or deteriorated; • Small producers have been excluded; • Household food insecurity has deteriorated (overall or seasonally); • Intra-household equity of income has declined; • The labor burden of women has increased; • Debt burden has increased; • In irrigation/water use projects, changes in water-borne diseases; • In livestock projects, changes in zoonotic disease; • In cash crop production/major mechanization, possible harmful effects.

Data of particular interest to project managers

In addition to data on Ag2Nut effects, it may be useful to collect data which will be of particular interest to project managers, data which might not be collected as often (or as well) in primary ag M&E systems.

Data of particular interest to project managers

Among possible indicators, amenable to collection at surveillance sites without difficulty are the following: • Percentage of households considering themselves

better off now than 12 months ago • Percentage of the labor force underemployed or

unemployed • Access, use and satisfaction with services provided

under the project • Changes in farmer income • Ratio of average income of the richest quintile to the

poorest quintile

Ag2Nut Expert Ag Manager

The Larger Context

Such sentinel site data collection is likely to be useful if:

1. Ag2Nut staff is itself sufficiently involved in project design, and is successful in including, e.g. employment generation; production diversity (with special attention to affordable nutrient-dense foods); active involvement of women

The Larger Context (conditions)

2. Good quality data can be sensibly aggregated and presented to project management in timely fashion

3. There is an explicit understanding that harmful effects identified by Ag2Nut sentinel site teams – or data indicating shortcomings in project implementation – will be directly and seriously addressed by project management.

4. Ag2Nut surveillance teams are prepared with mitigation plans in cases of harmful effects

Comparing Nutrition-Sensitive Ag Projects

Unlike nutrition-specific projects, cost-effectiveness analysis in nutrition-sensitive ag projects may be too context-specific, and involve too many assumptions, to be useful.

• Costs of ag projects vary enormously

• Trying to define specific “nutrition” marginal costs minimizes the importance of the projects’ primary production objectives, and the potential for integrated win-win projects.

Comparing Nutrition-Sensitive Ag Projects

Instead, projects

• should be reviewed jointly by project managers and Ag2Nut M&E teams, and

• should be considered “successful” and worthy of replication if they are effective in meeting both production and food security/nutrition objectives and are found to be economically viable for producers and funders.

Ag2Nut Expert Ag Manager

In Sum

Through the creative use of separately managed sentinel site-based M&E, it should be possible to:

• Generate cooperative efforts

• Generate much needed data

• Generate much needed successes in nutrition-sensitive agriculture.