Monitoring and Evaluating the Food Security and Nutrition Effects of

Agricultural ProjectsAnna Herforth - June 13, 2013 - LCIRAH

Nutrition Indicators in Agriculture Survey:

Preliminary results

Anna Herforth and Terri BallardFAO consultants

Funded by the EU-FAO Improved Global Governance for Hunger Reduction Programme (2012-2015)

Nutrition indicators in Agriculture projects - Survey

• Aim: to understand how agriculture projects will measure impact on nutrition: which indicators are being used, how, and why

• Why?– Researchers may be able to connect if desired;

informal technical support possible– Discussions about the evidence base can be informed

about the current generation of new evidence• How:– Follow up on DFID-funded LCIRAH mapping study

• 50% of studies identified were applicable• Excluded secondary data analysis, unspecified research, and

research with no agricultural intervention

Survey questions based on theory• Key principles:– Incorporate explicit nutrition objectives and indicators

into their design– Maintain or improve the natural resource base – Empower women – Facilitate production diversification, and increase

production of nutritious foods – Expand markets and market access for vulnerable

groups– Incorporate nutrition promotion and education to

enhance the impact of production and income– Collaborate and coordinate with other sectors

See: http://unscn.org/files/Agriculture-Nutrition-CoP/Agriculture-Nutrition_Key_recommendations.pdf

Preliminary Results: Response

• 77 project PIs contacted

• 68 responded (88%)– 3 of these excluded– 2 did not complete82% with complete

data

Preliminary Results: IndicatorsType of measure % measuring NotesFood consumption or diet almost all Many measuring HDDS, WDDS, and

IDDS for kids; intake of specific foods

of these, specific varieties? half (of above) Biofortified varieties; natural varieties

Food security most HFIAS, HHS, seasonality, coping strategies

Knowledge or behaviors most Specific to project

Women’s empowerment or labor

half Indicators somewhat unclear; a couple using/testing WEAI

Economic outcomes many of these, 2/3 disaggregating by gender

Nutritional status many stunting, underweight, BMI, anemia, indicators of VAS

Link with water, health, or sanitation

many

Natural resource management some Few indictors described; e.g. use of soil and water conservation practices

Preliminary Results: Design

• Majority are measuring in a comparison population

• Majority are collecting qualitative data• Sample sizes range from 120 to 4000 (one

9000)

Early conclusions• Focus is on nutrition impact among producers• Total shift from previous generation of research

regarding measurement of diet quality• Newly developed indicators get used• High number of studies measure nutritional status,

but available sample sizes suggest low power• Seems to be attention to program impact pathways– Indicators chosen mostly because important to project

goals, or evaluates a specific aspect of project– Interest in support for: “Adapting indicators to fit your

particular study activities and aims”

What these results represent

• This sample describes projects that set out to affect nutrition

• It does not describe larger agriculture programs or investments where nutrition is not necessarily the primary goal

Monitoring and Evaluating the Food Security and Nutrition Effects of

Agricultural ProjectsF. James Levinson and Anna Herforth

Ag2Nut Community of Practicehttp://knowledge-gateway.org/ag2nut

Purpose

Monitoring and evaluation of ag projects seeking to address food security and nutrition (in addition to their generally primary production objectives) would be highly desirable:

• there is, to date, so little empirical data documenting successes and failures; and

• possible adverse effects need to be identified and addressed rapidly within programs

Constraints

• Impact on food security on nutrition not necessarily first priority of agricultural managers

• Not familiar with how to measure• M&E in general is often not very strong– FAO/World Bank study (2010) of M&E in agriculture

projects in Asia, Africa and Latin America

Common shortcomings of Ag M&E• M&E is often perceived as an externally imposed

obligation with findings seldom integrated into management and action systems

• Ag managers complain of unmanageable data collection and reporting demands

• What M&E is carried out gives primary attention to physical achievements to the neglect of project outcomes

• Inadequate institutional capacity consistently limit M&E

(All of these, of course, are problems common to M&E in development projects more generally.)

A role for external Ag-Nut M&E teams?• Roles– identify sensible indicators to measure nutrition-

relevant impact based on the type of activities in the program

– carry out the key M&E necessary for tracking progress

– feed back to the program management (monitoring)

– Understand reasons for impact or lack thereof (eval)

– could support nutrition-sensitive program design or adjustment

Sentinel SitesGeographically representative sentinel sites within

the overall project area are one approach to M&E in ag projects with Ag2Nut interests

Baseline data would be followed by the collection of quantitative and qualitative data at 6 month intervals from these sites, and from comparable sites in non-project areas.

Examples of data• Extent to which households have been

reached/affected by the project• Relevant outputs specific to project (e.g. certain

crops/foods consumed)• Household food insecurity levels• Dietary quality• Women’s empowerment• If it makes sense, nutritional status among young

children and women

All complemented by qualitative data to better understand the dynamics of project effects

Identifying harmful food security or nutrition effects

• Employment levels have remained static or deteriorated;

• Small producers have been excluded;• Household food insecurity has deteriorated (overall or

seasonally);• Intra-household equity of income has declined;• The labor burden of women has increased;• Debt burden has increased;• In irrigation/water use projects, changes in water-

borne diseases;• In livestock projects, changes in zoonotic disease• Harm to natural resources (particularly water, soil,

biodiversity)

Data of particular interest to project managers

Among possible indicators:• Access, use and satisfaction with services

provided under the project• Changes in farmer assets• Percentage of households considering

themselves better off now than 12 months ago

• Percentage of the labor force underemployed or unemployed

Conditions

Such sentinel site data collection is likely to be useful if:•Good quality data can be sensibly aggregated and presented to project management in timely fashion– Challenge to collect data that are meaningful enough

to be useful, and brief enough to be usable•Information indicating harmful effects or shortcomings in project implementation – will be addressed by project management

How would it be supported?

Adequate staff and funding:•There is a need to identify Ag-Nut M&E teams capable of participating actively in an initial stream of nutrition-sensitive agriculture projects. •External funds probably necessary

In Sum

Through the creative use of separately managed sentinel site-based M&E, it should be possible to:

• Generate cooperative efforts• Generate much needed data• Document successes in nutrition-sensitive

agriculture• Develop prototypes, training modules and TA

mechanisms for subsequent use

Building the evidence base

• Need more examples of successful programs– Not just nutrition outcomes, but win-wins with

other goals• Ultimately, what is it that we want to scale

up?– Probably not individual programs– Rather, principles that explain how individual

programs have positive impact– Evaluations need to offer generalizable lessons

learned

Discussion

Ag2Nut Community of Practice http://knowledge-gateway.org/ag2nut