an agenda for assessing and improving conservation impacts ... · jeffrey c....

Essay

An agenda for assessing and improving conservationimpacts of sustainability standards in tropicalagricultureJeffrey C. Milder,∗ ††††† Margaret Arbuthnot,† Allen Blackman,‡ Sharon E. Brooks,§Daniele Giovannucci,∗∗ Lee Gross,†† Elizabeth T. Kennedy,∗ Kristin Komives,‡‡Eric F. Lambin,§§∗∗∗ Audrey Lee,††† Daniel Meyer,‡‡‡ Peter Newton,§§§ Ben Phalan,∗∗∗∗

Gotz Schroth,†††† Bambi Semroc,‡‡‡‡ Henk Van Rikxoort,§§§§ and Michal Zrust∗∗∗∗∗∗Rainforest Alliance, Evaluation & Research Program, 233 Broadway, 28th Floor, New York, NY, 10279, U.S.A.†World Wildlife Fund, 1250 24th Street NW, Washington, D.C., 20037, U.S.A.‡Resources for the Future, 1616 P Street NW, Washington, D.C., 20036, U.S.A.§UNEP-WCMC, Cambridge CB3 0DL, United Kingdom∗∗Committee on Sustainability Assessment, Philadelphia, PA, 19147, U.S.A.††EcoAgriculture Partners, 1100 17th Street NW, Suite 600, Washington, D.C., 20036, U.S.A.‡‡ISEAL Alliance, 50-52 Wharf Road, London, N1 7EU, United Kingdom§§School of Earth Sciences and Woods Institute for the Environment, Stanford University, 473 Via Ortega, Stanford, CA, 94305, U.S.A.∗∗∗Georges Lemaıtre Centre for Earth and Climate Research, Earth and Life Institute, Universite Catholique de Louvain, Place LouisPasteur 3, 1348, Louvain-la-Neuve, Belgium†††Roundtable on Sustainable Palm Oil, Kuala Lumpur, Malaysia‡‡‡Round Table on Responsible Soy Association, Florianopolis, Brazil§§§International Forestry Resources and Institutions research network, School of Natural Resources and Environment, University ofMichigan, 440 Church Street, Ann Arbor, MI, 48109, U.S.A.∗∗∗∗Department of Zoology, University of Cambridge, Cambridge, CB2 3EJ, United Kingdom††††Rainforest Alliance, Lombardi 34, 6708LT, Wageningen, The Netherlands‡‡‡‡Conservation International, 2011 Crystal Drive, Suite 500, Arlington, VA, 22202, U.S.A.§§§§UTZ Certified, De Ruyterkade 6, 1013AA, Amsterdam, The Netherlands∗∗∗∗∗Zoological Society of London, Regent’s Park, Outer Circle, London, NW1 4RY, United Kingdom

Abstract: Sustainability standards and certification serve to differentiate and provide market recognition togoods produced in accordance with social and environmental good practices, typically including practices toprotect biodiversity. Such standards have seen rapid growth, including in tropical agricultural commoditiessuch as cocoa, coffee, palm oil, soybeans, and tea. Given the role of sustainability standards in influencingland use in hotspots of biodiversity, deforestation, and agricultural intensification, much could be gainedfrom efforts to evaluate and increase the conservation payoff of these schemes. To this end, we devised asystematic approach for monitoring and evaluating the conservation impacts of agricultural sustainabilitystandards and for using the resulting evidence to improve the effectiveness of such standards over time. Theapproach is oriented around a set of hypotheses and corresponding research questions about how sustain-ability standards are predicted to deliver conservation benefits. These questions are addressed through datafrom multiple sources, including basic common information from certification audits; field monitoring ofenvironmental outcomes at a sample of certified sites; and rigorous impact assessment research based onexperimental or quasi-experimental methods. Integration of these sources can generate time-series data thatare comparable across sites and regions and provide detailed portraits of the effects of sustainability standards.

†††††email [email protected] submitted April 12, 2014; revised manuscript accepted July 25, 2014.

1Conservation Biology, Volume 00, No. 0, 1–12C© 2014 Society for Conservation BiologyDOI: 10.1111/cobi.12411

2 Conservation and Sustainability Standards

To implement this approach, we propose new collaborations between the conservation research communityand the sustainability standards community to develop common indicators and monitoring protocols, fosterdata sharing and synthesis, and link research and practice more effectively. As the role of sustainabilitystandards in tropical land-use governance continues to evolve, robust evidence on the factors contributing toeffectiveness can help to ensure that such standards are designed and implemented to maximize benefits forbiodiversity conservation.

Keywords: biodiversity, certification, evaluation, evidence-based conservation, monitoring, voluntary sustain-ability standards

Una Agenda para Evaluar y Mejorar los Impactos de Conservacion de los Estandares de Sustentabilidad sobre laAgricultura Tropical

Resumen: Los estandares de sustentabilidad y de certificacion sirven para diferenciar y proporcionarreconocimiento de mercado a los bienes producidos de acuerdo con las buenas practicas sociales y ambientalese incluyen tıpicamente a las practicas para proteger a la biodiversidad. Dichos estandares han tenido unrapido crecimiento, incluso en comodidades de la agricultura tropical como el cacao, el cafe, la palma deaceite, la soya y el te. Dado el papel de los estandares de sustentabilidad en influenciar el uso de suelo enzonas crıticas de biodiversidad, deforestacion e intensificacion agrıcola, mucho podrıa ganarse de los esfuerzospara evaluar e incrementar la indemnizacion de conservacion de estas estrategias. Para este fin, disenamosuna estrategia sistematica para monitorear y evaluar los impactos de conservacion de los estandares desustentabilidad agrıcola y para usar la evidencia resultante para mejorar la efectividad de dichos estandarescon el tiempo. La estrategia esta orientada a partir de un juego de hipotesis y preguntas de investigacioncorrespondientes sobre como los estandares de sustentabilidad estan pronosticados para entregar beneficiosde conservacion. Estas preguntas se abordan a partir de datos de multiples fuentes, incluyendo informacioncomun basica de auditorıas de certificacion; monitoreo en campo de resultados ambientales en una muestrade sitios certificados; e investigacion de evaluaciones rigurosas de impacto con base en metodos experimentaleso casi experimentales. La integracion de estas fuentes puede generar datos de series de tiempo que soncomparables a lo largo de sitios y regiones y proporcionan retratos detallados de los efectos de los estandaresde sustentabilidad. Para implementar esta estrategia, proponemos colaboraciones nuevas entre la comunidadde investigadores de la conservacion y la comunidad de estandares de sustentabilidad para desarrollarindicadores comunes y protocolos de monitoreo, fomentar la sıntesis y el compartir los datos y enlazar conmayor efectividad la investigacion y la practica. Conforme el papel de los estandares de sustentabilidad en lagobernacion del uso de suelo continua con su evolucion, la evidencia fuerte de los factores que contribuyen a laefectividad puede ayudar a asegurar que dichos estandares son disenados e implementados para maximizarlos beneficios para la conservacion de la biodiversidad.

Palabras clave: biodiversidad, certificacion, conservacion con base en evidencias, evaluacion, monitoreo, nor-mas voluntarias de sostenibilidad

Introduction

The expansion and intensification of agriculture to meetgrowing demand for food, feed, and fuel is a major causeof tropical biodiversity loss and ecosystem degradation(Laurance et al. 2014). These effects occur either directly,through the conversion of natural ecosystems to croplandor pastures and the degradation of on-farm habitats, orindirectly, through habitat fragmentation, water pollu-tion or diversion, spread of invasive species, greenhousegas emissions, and other off-farm environmental impacts.Agriculture may also disrupt a range of ecosystem ser-vices, from water cycle regulation to soil protection, thatunderpin food production and other aspects of humanwell-being.

Voluntary sustainability standards have emerged as apromising response to the serious challenges associatedwith tropical commodity agriculture, which include not

only ecological impacts, but also social issues such aschild labor, land and water conflicts, and the perpetua-tion of severe poverty and inequality (Potts et al. 2014).Typically established by nonprofit organizations, definedthrough multistakeholder processes, and adopted vol-untarily by producers and buyers in agricultural supplychains, such standards provide a nonstate governanceapproach to addressing sustainability concerns (Cashoreet al. 2004). Sustainability standards serve to codify thepractice of sustainable agriculture in ways that couldsupport the widely held objective of increasing agricul-tural output while reducing agriculture’s ecological im-pacts and supporting rural livelihoods (Foley et al. 2011;Garnett et al. 2013). Such standards also provide a marketmechanism to convert demand for more sustainable agri-cultural products into farm-level incentives (COSA 2014).

Voluntary sustainability standards systems typically in-clude 4 components: the standard itself, which identifies

Conservation BiologyVolume 00, No. 0, 2014

Milder et al. 3

putatively sustainable practices or outcomes; assurancesystems, including certification audits, intended to assessconformance with the standard by specific productionor business units; sustainability labels (ecolabels) that en-able producers, brands, and retailers to differentiate theirproducts in the eyes of consumers, intermediary buyers,government regulators, and civil society; and trainingand technical assistance activities to assist producers inachieving certification. Hereafter, we use the term sus-tainability standards to signify this entire set of linkedcomponents. We refer to farms, groups of farms, or otherproduction units (e.g., a mill plus its supply shed) thathave complied with sustainability standards as certifiedentities. We focus on third-party sustainability standardsin which standard setting and assurance are carried outby independent parties unrelated to the entity being eval-uated; however, the issues we consider are relevant to abroader set of sustainability standards initiatives, includ-ing standards set by industry associations and individualcompanies.

Most agricultural sustainability standards address biodi-versity conservation and environmental quality. A reviewof 12 such standards showed that all 12 included require-ments for habitat protection; 10 prohibited clearance ofcertain land-cover types; 9 specified criteria for priorityhabitat areas; 8 addressed impacts to threatened species;and 7 included measures to address invasive species(UNEP-WCMC 2011). Most standards also address soil ero-sion, pesticide use, water pollution, waste management,and, increasingly, greenhouse gas emissions (SteeringCommittee 2012). The degree to which each standarddelivers environmental benefits depends not only onits requirements, but also on how it is implementedin practice, including the quality and consistency ofauditing.

Agricultural sustainability standards have seen rapiduptake in recent years and are now applied on tens ofmillions of hectares worldwide. From their position asa small niche only a decade ago, such standards nowgovern a major proportion of the global production ofcoffee (38% of all production), cocoa (22%), palm oil(15%), and tea (12%), and small but growing productionsegments for cotton (3%), sugar (3%), soybeans (2%), andbananas (3% of global production but 18% of exportedproduction) (Potts et al. 2014). This growth has beenaccompanied and accelerated by the increased use ofsustainability standards as proxies for acceptable practicewithin government and nonstate regulatory frameworks,including public-sector procurement policies and lendingcriteria by private and multilateral banks (Giovannucciet al. 2014). In addition to these regulatory and quasi-regulatory drivers, the business case for adopting sustain-ability standards may include increased net revenue ormarket access for producers, and increased market share,improved branding position, and decreased operationalor reputational risks for companies (Levin 2012).

Despite the growing uptake of sustainability standardsand their explicit conservation objectives, there is littleevidence about their conservation impacts. A review ofevaluation literature on sustainability certification in se-lected sectors found 26 peer-reviewed studies on bananasand coffee but only 8 of these addressed environmentalresults and none of these 8 were judged to have beendesigned rigorously, based on quantitative measures anda credible counterfactual scenario (Blackman & Rivera2011).

This dearth of evidence poses risks for many stake-holders and a missed opportunity for conservation. First,companies, investors, and governments rely upon sus-tainability standards as a credible means to implementsustainability policies and commitments. Without ro-bust evidence of impact, it is difficult to know whetherthese commitments are being fulfilled and translated intoprogress for biodiversity conservation and other objec-tives. Second, sustainability standards are rapidly evolv-ing, but the lack of data on impacts impedes continuousimprovement and leads to fragmented policy messagesthat are not rooted in solid evidence (Newton et al. 2013).Third, better evidence on conservation impacts—andtheir relationship to social, economic, and productivityimpacts—is needed to improve the cost-benefit calculusand business case for producers and businesses to con-sider sustainability standards.

In short, better evidence on impacts could helpposition sustainability standards as a powerful adaptivemanagement framework in support of globalsustainability—that is, a structured set of initiativesunder which key tenets of efficient, equitable, andconservation-friendly agriculture are widely applied,monitored, and analyzed in consistent ways, with theresulting evidence used to revise standards systemsand improve their benefits over time. Conversely, acontinued lack of evidence could undermine support forsustainability standards (especially the more demandingand costly standards), causing businesses to select lessdemanding standards or less transparent approaches tosustainable sourcing. A manifest risk is that sourcingprograms billed as sustainable might satisfy publicdemand for green products without actually deliveringconservation benefits.

Efforts to generate robust evidence on sustainabilitystandards could draw on recent advances in evidence-based conservation, including best practices for system-atic monitoring, evaluation, and adaptive management ofconservation interventions (Pullin & Knight 2003; Ferraro& Pattanayak 2006). Several features of sustainabilitystandards make them both compelling and tractableas subjects for collaborative, systematic evaluation fol-lowing an evidence-based conservation paradigm. First,sustainability standards present features of both con-sistency and variation that make them well suited tolongitudinal, cross-sectional, and comparative evaluation



designs. Sustainability standards can be considered over-all as a similar set of nonstate governance interventions,and individual standards are usually applied consistentlydue to the codified nature of standards. However, thereis considerable variation among agricultural sustainabil-ity standards, which presents opportunities to comparethe effectiveness of different approaches. Second, sus-tainability standards have been adopted widely acrossmany contexts, and ongoing expansion presents op-portunities to compare adoption and nonadoption tra-jectories. Third, most standards systems can generateconsistent, time-series data through audit processes.Finally, coordination presents strong opportunities toimprove efficiency, reduce duplication, and enhancedata interpretability and cross-context learning associ-ated with the growing, but now largely separate, set ofevaluation efforts emerging from the research and NGOcommunities.

Although both the need and the opportunity for sys-tematic evaluation and improvement of agricultural sus-tainability standards are clear, it now remains to turnthese possibilities into reality. We propose an approachand operational strategy for doing so.

Key Considerations

Conservation Hypothesis

Conservation interventions may be described accordingto a theory of change or results chain that logically con-nects activities to results (i.e., outputs, outcomes, and im-pacts) (Mascia et al. 2014). Such results-based approachesare now widely used to characterize the causal relation-ships and assumptions that underpin conservation effec-tiveness (Margoluis et al. 2013).

We recognize 3 main ways by which agricultural sus-tainability standards may deliver conservation benefits:conserving existing natural ecosystems and their asso-ciated biodiversity; improving the conservation value offarms and production landscapes through restoration andconservation-friendly management; and reducing off-siteenvironmental impacts of agriculture, such as water pol-lution (Fig. 1). Each of these results may be delivered atthe scale of individual certified entities or, more broadly,across landscapes and regions through aggregate or indi-rect effects.

Many of these results are hypothesized to be achievedin a sequential and cumulative manner: direct resultssuch as improved farm management practices supportpositive changes in the local environment (intermediateresults), which in turn contribute to benefits over largerspatial and temporal scales (broader results) (Fig. 1).Sustainability standards also interact reciprocally withother state and nonstate governance approaches to influ-ence standard-setting processes, demand for sustainably

produced products, and the adoption of sustainabilitystandards and their component land use and farm man-agement practices (Steering Committee 2012; Eberleinet al. 2014). This results chain provides an overarch-ing metahypothesis around which specific research andevaluation questions can be designed to interrogate eachhypothesized linkage (small right-pointing horizontal ar-rows in Fig. 1). The stepwise logic supports robust eval-uation design by clarifying the intermediate factors thatinfluence the achievement of ultimate impacts (Mitevaet al. 2012). The relative importance of each result, andthe associated causal linkages, may differ among com-modities and contexts.

Specific Demands for Evidence

Efforts to evaluate and improve conservation impacts ofsustainability standards will hold greater value if they ad-dress key stakeholder priorities. A widely held priority—shared by companies, civil society, governments, agri-cultural financiers, and consumers—is to documentavoidance of the worst environmental practices, suchas destruction of mature forests and other ecosystemsof high conservation value. Such evidence is neededto demonstrate adherence to various regulations andvoluntary commitments, such as the Equator Principlesand the Consumer Goods Forum zero net deforestationpledge (Brown & Zarin 2013). Similarly, many companieshave made sustainability commitments around issuessuch as water conservation, pollution mitigation, and in-creasing on-farm habitat (e.g. Unilever 2010) and requirecredible monitoring to report on these. Although compa-nies are currently using sustainability standards as a proxyfor demonstrating progress, stakeholders are increasinglydemanding more direct evidence. Information on con-servation outcomes may also inform consumer behavior,although such information may have limited reach unlessamplified by marketing campaigns (e.g., to highlight fa-vorable evidence) or activist campaigns (e.g., companyscorecards or name-and-shame tactics that shine a spot-light on negative evidence).

Different actors prioritize different evidence in differ-ent contexts. For commodities where climate changeor land degradation threaten future supply (e.g.,cocoa, coffee, and tea), farmers, companies, and indus-try groups seek information on the condition of natu-ral resources that underpin productive farming systems.Certification for developing country markets is incipient,and stakeholders may emphasize legal compliance, socialand environmental management systems, and food safetyover demonstrated conservation outcomes (Newtonet al. 2014). Especially for crops that can be grown inagroforestry systems, the composition, abundance, andpersistence of native species assemblages are of particularinterest to conservationists (Tscharntke et al. 2014).


Milder et al. 5

Figure 1. Generalized results chain illustrating the ways in which agricultural sustainability standards arehypothesized to deliver conservation benefits and reduce conservation threats. Black rectangles elaborate the 3sets of potential conservation benefits identified in the text. Feedback arrows at the bottom indicate thatmonitoring and evaluation of results at all levels should inform the adaptation and improvement of standardssystems over time.

Sustainability standards bodies seek information on im-pacts to understand and improve the effectiveness oftheir standards over time; to increase the credibility ofsustainability claims; and to assert competitive advantagebased on positive results. Finally, donors and interna-tional institutions that support sustainability standardsseek information on the effects of their investments onkey conservation and human development objectives.

Methodological Challenges and Remedies

To develop a robust assessment framework, it is impor-tant to recognize key limitations that have character-ized most evaluations of sustainability standards to date(Table 1). Many such challenges can be largely addressedthrough improved design and execution of future moni-toring and research (Table 1).

Other limitations are more difficult to address and maycontinue to impede the interpretation of evaluation find-ings. One challenge is to distinguish the role of sustain-ability standards in improving practices and outcomesversus simply recognizing or rewarding existing goodmanagement. If the most sustainable producers adoptsustainability standards whereas poorer performers avoidthem, then the incremental benefit (additionality) maybe limited. However, this issue requires a nuanced lensbecause in some cases simply maintaining good practicesin the face of pressures to expand or intensify agriculture

may constitute success. In other cases, producers makemajor improvements in preparation for certification, buttheir results are not captured if precertification baselineshave not been established. An additional challenge is thateach sustainability standard is actually a bundle of inter-ventions (e.g., specific requirements of the standard) thatmay differ by context and change over time as standardsare revised. Accordingly, when evaluating sustainabilitystandards, it is important to select the appropriate inde-pendent variable or variables and understand the extentto which evaluation findings may be legitimately gener-alized across standards, contexts, and time.

Framework for Assessing Conservation Impacts ofAgricultural Sustainability Standards

Based on the preceding considerations, we devised aframework for a coordinated effort to monitor, evalu-ate, and adaptively manage the conservation impacts ofagricultural sustainability standards. The framework is ori-ented around 8 research questions (Table 2) designed totest the causal logic and associated hypotheses in Fig. 1.These questions are aligned with many of the top 100important questions for the future of agriculture (Prettyet al. 2010) and biodiversity conservation (Sutherlandet al. 2009), thereby linking evaluation of agriculturalsustainability standards to these broader agendas. The



Table 1. Key limitations of the existing evidence base on impacts of agricultural sustainability standards and corresponding recommendations forfuture activities or approaches that could help improve the evidence base.

Limitations of existing Needs and recommendations forapproaches and evidence base generating a more robust evidence base

Evaluation literature is focused on isolated case studyinvestigations limited to a few crops and geographies.

Apply a systematic and adequately dense sampling frameworkrepresenting key crops and geographies.

Different studies use different methods, preventingcomparative analysis:

Apply standard methods and indicators to facilitategeneralization and comparison across contexts.

different treatment variables (interventions)different outcome variables (indicators)different, or unaccounted for, control variables (covariates)different data collection protocols.

Studies evaluate only one dimension of sustainability (e.g.,social, environmental, or economic).

Evaluate multiple dimensions in a given system to understandsustainability outcomes more holistically, including therealization of synergies or trade-offs.

Studies lack counterfactual comparisons. Construct credible counterfactuals using experimental orquasi-experimental methods.

Short time frame of studies fails to reveal the trajectory ofecological and other outcomes, especially for slowlychanging variables.

Conduct repeated monitoring of key variables to generaterobust time series and panel data (including forcounterfactual scenarios).

Scale of evaluations (e.g., certified farms) is not matched to thescale of key conservation outcomes (e.g., landscape,regional, or global).

Complement site-level evaluations with analysis at largerspatial and temporal scales to evaluate cumulative effectsand interactions of farms with surrounding areas.

Studies lack transparency, are not conducted to high researchstandards, or are not appropriately interpreted relative tothe aims and mechanisms of sustainability standards.

Use theory-based evaluation approaches; adhere to evaluationbest practices; improve collaboration between researchersand standards bodies; and publish results in peer-reviewedjournals.

Studies on environmental results focus on changes inpractices, rather than outcomes, or use indicators of lowcredibility (e.g., farmer perception).

Select and quantitatively monitor changes in keyenvironmental parameters that sustainability standards arehypothesized to influence.

questions are also defined to address stakeholder de-mands for evidence (identified above).

Research questions are posed in Table 2 in a gen-eralized way; researchers would further specify thesequestions for specific evaluation studies. Individual evalu-ations will often focus on a specific sustainability standardor on a subset of the practices or outcomes requiredunder that standard. For instance, question 3 in Table 2,related to the protection of on-farm conservation values,would be approached quite differently for a coffee stan-dard that promotes diversified agroforestry systems thanfor a sugar standard that promotes efficient productionmonocultures with conservation set-aside areas. Collec-tively, the body of evidence from many such individualevaluations can clarify which practices (i.e., requirementsof different sustainability standards) are most effectiveat delivering conservation benefits in different contexts.This information can elucidate the relative effectivenessof different standards at meeting different objectives and,more importantly, support all standard systems in becom-ing more effective over time. In addition to addressing theeffects of individual sustainability standards or their com-ponent practices, the research questions also considerthe cumulative and interactive effects of sustainabilitystandards applied broadly across a sector or region, rela-

tive to the biophysical, spatial, and governance contexts.To address these questions will require a monitoring

and evaluation approach that combines breadth, depth,and integrative analysis. We therefore propose a hierar-chical, nested framework that integrates data and learningat 3 levels (Fig. 2). At the broadest level (Fig. 2, level1), system-wide monitoring across most or all of thecertified estate is necessary to characterize sustainabilitystandards at a basic level, understand variation acrosscontexts, quantify aggregate results, and contextualizeplace-specific research. To move beyond basic character-izations and understand intermediate conservation out-comes (Fig. 1) requires more in-depth field monitoringthat is too costly to conduct for every certified entity.Such sampled monitoring (Fig. 2, level 2) should be con-ducted across a stratified sample of sites to understand theeffects of sustainability standards in different settings andthereby support more generalized conclusions and re-porting of aggregate outcomes. Finally, focused research(Fig. 2, level 3) is needed to improve the credibilityof evaluation findings, address existing methodologicallimitations (Table 1), and evaluate broader and indirecteffects associated with sustainability standards. Below wedescribe each level of the framework, provide examples,and identify the most critical next steps.


Milder et al. 7

Table 2. Key questions to be addressed by a coordinated approach to monitor and evaluate conservation impacts of agricultural sustainabilitystandards.

Research Key Scope ofquestiona themesb investigationc Referencesd

1. To what extent do sustainability standards leadto changes in on-farm environmentalmanagement practices?

soil and water, noncropvegetation,agrochemicals, farmland use

certified lands A79, A100, C95

2. Do sustainability standards help maintain orimprove the on-farm natural resource base andassociated ecosystem services to agriculture?

soil structure andfertility, water, pestcontrol, climateresilience, pollination

certified lands A8, A21

3. Do sustainability standards help maintain orimprove on-farm biodiversity conservationvalues?

noncrop vegetation,shade canopy, riparianprotection,conservation setasides

certified lands A32, C41

4. When and to what extent are conservationoutcomes synergistic or conflicting withproductivity and profitability outcomes?

above themes combinedwith key economicand productivityresults

certified lands A15, A66, A67, A92,C40, C41

5. Do sustainability standards contribute tobiodiversity-friendly landscapes and land-usemosaics?

habitat connectivity,natural ecosystemconservation andcondition, populationviability

certified lands in context A19, C37, C40

6. To what extent do sustainability standardsmitigate negative off-site environmentalimpacts or support ecosystem service flows tooff-site beneficiaries?

water quality, water flowregulation,sedimentation,eutrophication,greenhouse gases

certified lands in context A21

7. Do sustainability standards help reducedestruction and degradation of naturalecosystems and areas of biodiversityimportance at landscape to regional scales?

protection of key areas certified lands in context C95

8. How do sustainability standards interact withother land use, governance, and marketincentive systems and dynamics to influencethe answers to the preceding questions?

governance, land-useplanning andregulation, marketincentives, agriculturalsupply chains

sustainability standardsin context

aResearch questions about the effects of sustainability standards can be answered at multiple levels: effects of implementing specific requirementsof one or more sustainability standards, such as the adoption of specific conservation-friendly management practices; effects of adopting a specificsustainability standard; or uptake of sustainability standards and certification more generally across a sector or locale. The research questionsshould be construed as including all these aspects, although the salience of each aspect differs from question to question.bFurther suggested foci for each research question.cThis column identifies whether each question needs to be answered by examining certified lands alone; certified lands within their biophysical,economic, or sociopolitical contexts; or entire sustainability standards systems within such contexts.dThis column links the research questions to previously identified top 100 important questions for the future of global agriculture (questionswith an A [Pretty et al. 2010]) and biodiversity conservation (questions with a C [Sutherland et al. 2009]).

System-Wide Monitoring

System-wide monitoring (level 1) comprises data col-lected for every certified entity under a given sustainabil-ity standard. Such data collection is typically integratedinto certification processes to achieve full data coverage.However, this means that level 1 data are limited to thosethat can be feasibly collected through certification ap-plications (i.e., information self-reported by producers)and audits (which verify self-reported data and collectadditional information to assess conformance with a stan-

dard). Key level 1 data include the location and land areaof certified entities, area under production and conserva-tion land uses, crop types and production systems, whenand for how long certificate holders achieve and maintaincertification, and whether a certified entity has multiplecertifications. Such data enable basic characterization ofcertified lands and are critical for informing secondaryanalysis or research on broader impacts. For example,contributions to landscape conservation cannot be under-stood without knowing the location of certified entities.



Figure 2. Proposed framework for a coordinated system to evaluate the conservation impacts of agriculturalsustainability standards. The system aligns evaluation themes, indicators, and methods to integrate data collectedsystem-wide (i.e., for all certified entities; level 1), data collected using comparable methods at a representativesample of certified operations (level 2), and focused research conducted using experimental andquasi-experimental designs as well as methods to understand broader contributions to, and indirect effects on,conservation outcomes (level 3). Each level is explained further in the text.

System-wide monitoring can also be used to collectdata on farm performance relative to specific practices oroutcomes identified in a standard, to the extent that audi-tors evaluate these parameters to assess conformance. Forexample, Starbucks’ Coffee and Farmer Equity (C.A.F.E.)Practices sustainability standard and verification programhas used audit data to track farm practices over sev-eral years, thereby informing adaptive management byhighlighting potential improvements to the standard andpriorities for farmer training (Thomas et al. 2012). In sus-tainability standards that are strictly practice based, auditdata would be restricted to information on practice adop-tion. However, some standards also include outcome-based requirements. For example, compliance to theBonsucro sugar standard considers outcome-level dataon greenhouse gas emissions, water use efficiency, andchemical oxygen demand of agricultural runoff, amongothers (Bonsucro 2011).

At level 1, the priority is to achieve complete datacoverage, consistency, and comparability for a coreminimum data set that is feasible to collect through certi-fication processes and that spans different sustainabilitystandards and contexts. This could be achieved by defin-ing common indicators and guidelines for data collectionto facilitate data comparability. If data are to be aggre-gated across multiple sustainability standards and made

available to the research community (as we recommend),policies on data confidentiality, spatial precision (e.g.,blurring farm location data to protect individual farmeridentity while still enabling spatial analysis), and terms ofuse will also be needed.

Many processes necessary for the establishment of aminimum data set are already underway. For example,the ISEAL Alliance (a membership organization for sus-tainability standards) requires its members to developcredible monitoring and evaluation systems and encour-ages the collection of basic common information, suchas the location of certified entities (ISEAL Alliance 2010).However, additional work is needed to better incorporateconservation-related indicators, support all major stan-dards systems to participate in a common level 1 datacollection effort, and define mutually agreeable policiesfor data quality, confidentiality, and sharing of data withthe research community. These efforts will help rede-fine the certification audit as not solely a conformanceassessment tool but also one that facilitates monitoringand evaluation.

Sampled Monitoring

Effective monitoring of intermediate conservation out-comes (level 2) usually requires going beyond the scope


Milder et al. 9

of certification audits. Given resource constraints, thisis best achieved through a stratified sampling approachdesigned to represent key axes of variability most likelyto affect conservation outcomes, such as crop type, geo-graphic setting, local regulatory context, identity of thesustainability standards applied, and type of productionsystem (e.g., plantation vs. smallholder group). This ap-proach recognizes that outcomes are likely to vary bycontext; therefore, data on multiple certified entities percombination of context variables can help elucidate ef-fectiveness and the factors that influence it. Level 1 dataprovide a basis for establishing stratified samples.

Sampled monitoring may assess indicators such asland-cover composition, tree diversity and vegetationstructure, presence of natural ecosystem set asides andrestoration zones, and water quality of streams withincertified units. Depending on the conservation values ofinterest, it may be more appropriate to monitor changethrough the use of proxy variables (e.g., habitat qual-ity metrics) than direct measures (e.g., populations ofspecies of concern). Pairing sampled monitoring of in-termediate results with system-wide monitoring of directresults (e.g., practice adoption) is especially valuable fortesting the intermediate hypothesized linkages within theresults chain.

Sampled monitoring conducted on certified propertieswithout control sites is useful for tracking change trajec-tories and generating suggestive evidence of the effects ofcertification. Adding control groups can provide greaterconfidence about the causality of any observed effects.With or without a control group, sampled monitoringshould be conducted on a time-series basis, for instance,at baseline and every 2–3 years thereafter. This intervalis generally suitable to detect changes associated withinitial adoption of standards, subsequent improvementsor regressions, and the effects of periodic standard re-visions. Monitoring can be conducted by auditors whohave received additional training on the relevant fieldmethods (as a supplement to the audit process itself) orby local researchers, students, or paraprofessionals.

Several initiatives have defined sets of environmentalindicators at the sampled monitoring level, includingwork by the Committee on Sustainability Assessment(COSA 2014); the SAI Platform, representing the foodindustry (Elferink et al. 2012); and the Zoological Societyof London, focusing on high conservation value areas inthe context of oil palm production (Zrust et al. 2013). Inaddition, several calculator tools are available that modelenvironmental outcomes by integrating farmer-provideddata on land management practices with environmentalcontext data. For instance, the Field to Market tool esti-mates outcomes related to water pollution, greenhousegases, and biodiversity, which can be analyzed at the farmlevel or aggregated across supply chains or watersheds(Field to Market 2013). Each of these initiatives offersuseful tools and insights, but none has yet been applied

widely to evaluate intermediate conservation outcomesof sustainability standards.

Two sets of activities are needed to advance sampledmonitoring. First, data collectors and data users need tomove toward sets of common indicators and methods fortracking key conservation results. Doing so will increasedata comparability and facilitate comparative analysis andmeta-analysis. The indicator sets and tools mentionedabove provide helpful starting points but may need tobe expanded, customized, or streamlined to fit the spe-cific contexts where sustainability standards are applied.Given differing interests of different stakeholders and theneed for context-sensitive indicators, some metrics mightbe widely standardized whereas others will need to beadapted to specific crops or contexts.

Second, sampled monitoring needs to be operational-ized, which may best be achieved through initiatives fo-cused on regions where commodity production is a keyconservation threat. These efforts would address specificquestions nested within the overall framework (Fig. 1,Table 2), for instance, do sustainability standards ap-plied to cocoa-producing areas in West Africa’s GuineanForests biodiversity hotspot support sustainable crop in-tensification and reduced deforestation and forest degra-dation? In this example, multiple sustainability standardsworking in cocoa in West Africa would conduct sampledmonitoring at a network of sites to evaluate effects oftheir efforts, both individually and collectively. Monitor-ing could be conducted by local or regional institutions(e.g., universities) and coordinated under the auspicesof research teams or grant-funded projects. In this way,impacts can be understood beyond the level of individualcertified farms and linked to conservation priorities atlarger scales.

Focused Research

Level 3 includes 2 types of focused research that requiremore specialized or in-depth methodologies than are typi-cally possible at levels 1 and 2. First, to overcome method-ological limitations (Table 1), there is a need for researchthat uses rigorous program evaluation techniques. Suchtechniques measure causal effects by comparing the per-formance of a group subject to an intervention (e.g., sus-tainability standards) to a counterfactual scenario, thatis, a credible approximation of performance absent theintervention (Blackman & Rivera 2011). Counterfactu-als may be established through experimental methods(randomized controlled trials) in which producers arerandomly assigned to either a treatment group, whichadopts sustainability standards, or a control group, whichdoes not. However, this approach is uncommon due tologistical and ethical challenges of intentionally exclud-ing some producers. It is more common to mimic anexperimental design by allowing producers to interactwith sustainability standards in the usual way and then



Table 3. Proposed initiatives to implement the coordinated approach for assessing and improving conservation outcomes of agricultural sustain-ability standards.

Tangible productsInitiative or consequences of Key groups toor activitya this activityb be involved

Develop common indicators, datacollection guidelines, and datapolicies for a core minimum data setfor system-wide monitoring (level 1).

foundation for a collaborative systemestablished based on best-practicemonitoring approaches;conservation-related indicatorseffectively incorporated

standards bodies, convening body suchas ISEAL Alliance, monitoring experts,funders

Support standards systems to implementsystem-wide monitoring (level 1).

time-series data on basic characteristicsof all certified entities collectedaccording to a common monitoringframework

standards bodies, convening body suchas ISEAL Alliance, funders

Establish a data sharing platform (levels1, 2, 3).

data management facility and functions(including clear data sharing policiesand terms of use) that support use ofdata for analysis and research

standards bodies, neutral organization tohost platform, data systems expertsand other advisors, funders

Develop common indicators and metricsfor sampled monitoring ofconservation outcomes (level 2).

framework and field protocols forcarrying out coordinated monitoringof key conservation results

data collection organizations (e.g.,universities or research centers withinfocal regions), standards bodies,companies, researchers, governments,funders

Establish regional sampled monitoringinitiatives in key regions wherecommodity production drivesconservation threats (level 2).

comparable panel data from stratifiedsamples of certified properties inregions where commodity productiondrives key conservation threats

data collection organizations, standardsbodies, companies, researchers,governments, funders

Establish a research network to define aresearch agenda, coordinate researchactivities, and synthesize andcommunicate results (level 3).

research agenda guides project selectionand design; data and knowledgesharing facilitates collaborativeresearch and meta-analysis; newevidence supports adaptivemanagement and continuousimprovement of conservation benefits

researchers, standards bodies,convening organizations for thenetwork, other groups interested in astronger evidence base (e.g., civilsociety), funders

aInitially, these activities could be developed and launched in a modular fashion through the groups indicated in the right column. Over time,the activities would coalesce into the integrated 3-tiered assessment approach described in the text.bSpecific products or results of value to “key groups to be involved” and to society at large that each activity will deliver.

use statistical techniques to identify a control group thatis similar to the treatment group but did not adopt thestandards. Under either approach, the control group’s av-erage performance constitutes a credible counterfactualagainst which the treatment group is compared.

Second, level 3 research is needed to understand con-servation impacts over larger spatial and temporal scales(i.e., results 1b, 2b, and 3b in Fig. 1) to which sustainabil-ity standards may contribute only partially or indirectly.Doing so requires analyzing certified lands in their spa-tial, biogeographical, political, institutional, and marketcontexts. For example, understanding impacts on land-cover change or species’ populations may require dis-entangling the effects of sustainability standards fromthose of other landscape dynamics or broader trends,such as climate change. Spatial analysis can clarify contri-butions of certified properties to landscape connectivityor conservation networks and, when paired with statisti-cal methods, reveal whether sustainability standards arehelping to arrest threats such as forest degradation orpeatland burning. Level 1 data, such as the spatial loca-tion and basic characteristics of certified units, provide

important input to such research. Focused research is typ-ically best conducted by independent researchers whopossess specialized methodological expertise and whosework is less likely to be perceived as biased than moni-toring associated with standards bodies or supply chainactors.

The priority at level 3 is not only to increase the amountof relevant research on sustainability standards, but alsoto target, align, and use this research more effectivelyto improve conservation benefits. Doing so requires 3sets of activities. First, a priority research agenda shouldbe codefined by researchers, sustainability standards or-ganizations, and other stakeholders. As an example, theRoundtable for Sustainable Palm Oil (RSPO) and RoyalSociety recently collaborated to define a coordinated5-year multidisciplinary research program on the impactsof RSPO certification, the results of which will supportefforts to evaluate and improve RSPO’s sustainability stan-dard (SEARRP 2012). Second, data management functionsshould be established within a neutral, trusted organiza-tion to archive level 1 and 2 monitoring data and makeit available to researchers for legitimate purposes. Third,


Milder et al. 11

an active network for research on conservation impactsof sustainability standards should be established to sup-port researchers in aligning their work with the overallevaluation framework articulated in this article, facilitatemeta-analysis and other data synthesis, and communicateresearch results more effectively to standards bodies andother stakeholders.

From Concept to Action

Experience to date suggests that coordinated monitoring,evaluation, and data management systems are certainlypossible but are challenging to develop because they re-quire agreement of many people and institutions regard-ing shared goals and processes, while addressing issuessuch as data management and access, confidentiality, andfunding (Wollenberg et al. 2007; COSA 2014). Coordi-nated efforts are most likely to succeed if they advancein a modular fashion, leading with discrete initiatives thatoffer clear benefits and synergies for specific groups, suchas precompetitive collaboration on common indicators orincreased monitoring efficiency by consolidating duplica-tive effort in specific regions. Table 3 identifies a set ofinitiatives that should be feasible to launch over the nextfew years and that could coalesce into the fully formed3-tiered approach advocated here.

New resources will be needed to deliver stronger evi-dence on the conservation impacts of sustainability stan-dards. There is now a growing business case for suchinvestment, as companies require new monitoring todemonstrate fulfillment of sustainability commitmentsand as sustainability standards and supply chain actorsseek to leverage the value of existing data (e.g., certifica-tion audits and farmer record keeping) to better under-stand and improve supply chain performance. However,not all outcomes of importance for global biodiversityconservation are of high interest to the private sector—particularly those that manifest across wider landscapesor have less direct impact on future commodity supply.Rather, these are public goods that require governmentand donor support to safeguard—in this case, by con-tributing to the generation of a robust evidence base toassess and continuously improve such outcomes. Suchsupport is strategic and critical: comparatively modestinvestment in monitoring systems for sustainability stan-dards could help solidify and expand the role of suchstandards in leveraging market forces to deliver conserva-tion benefits, whereas failure to invest could underminethe continued viability of such mechanisms.

Conclusions

Agricultural sustainability standards have gone wellbeyond niche status and now govern substantial propor-

tions of key tropical commodities implicated in biodiver-sity loss. But despite the increasing reach of sustainabilitystandards, relatively little is known about their conserva-tion impacts. Sustainability standards may therefore nowbe at a fork in the road. If robust evidence on results andmechanisms can be generated—thereby substantiatingclaims, identifying limitations, and supporting continu-ous improvement—then sustainability standards couldbecome increasingly potent vehicles to translate publicdemand and corporate commitments for sustainabilityinto real conservation benefits. However, if there remainslittle evidence to substantiate (or refute) standards’ im-plied sustainability claims, then widespread skepticismand loss of credibility could follow. In this scenario,sustainability marketing campaigns (i.e., greenwashing)could crowd out the more transparent and independentstandard-setting and auditing processes that are the hall-marks of third-party sustainability standards.

Given the potentially pivotal role of sustainability stan-dards in addressing stubborn social and environmentalproblems, the development of a robust evidence baserelated to these standards must be considered as amongthe highest investment priorities for increasing the sus-tainability of tropical agriculture. We have demonstratedhere that it is both possible and necessary to deliversuch evidence by better coordinating a range of existingmonitoring and research activities and supplementingthese with targeted new initiatives. The private sectorand sustainability standards bodies can support this workto a considerable degree, building on current efforts.However, in view of the public good value of benefits thatsustainability standards may deliver, donors and govern-ments must also invest in the supportive infrastructure—particularly robust monitoring systems—that is needed toleverage societal and corporate demand for sustainabilityto deliver biodiversity conservation and other benefits atfull scale.

Literature Cited

Blackman, A., and J. Rivera. 2011. Producer-level benefits of sustainabil-ity certification. Conservation Biology 25:1176–1185.

Bonsucro. 2011. Bonsucro production standard, version 3.0. Availablefrom http://bonsucro.com/site/production-standard (accessed De-cember 2013).

Brown, S., and D. Zarin. 2013. What does zero deforestation mean?Science 342:805–807.

Cashore, B., G. Auld, and D. Newsom. 2004. Governing through mar-kets: forest certification and the emergence of non-state authority.Yale University Press, New Haven, Connecticut.

COSA (Committee on Sustainability Assessment). 2014. The COSA mea-suring sustainability report: coffee and cocoa in 12 countries. COSA,Philadelphia.

Eberlein, B., K. W. Abbott, J. Black, E. Meidinger, and S. Wood. 2014.Transnational business governance interactions: conceptualizationand framework for analysis. Regulation & Governance 8:1–21.

Elferink, E., G. Kuneman, A. Visser, and E. van der Wal. 2012. Sus-tainability performance assessment of farming practices: guidelines



for developers of quantitative monitoring tools. Available fromhttp://www.saiplatform.org/activities/alias/SPA (accessed Decem-ber 2013).

Ferraro, P. J., and S. K. Pattanayak. 2006. Money for nothing? A call forempirical evaluation of biodiversity conservation investments. PLoSBiology 4 DOI: 10.1371/journal.pbio.0040105.

Field to Market. 2013. Fieldprint calculator. Available fromhttp://www.fieldtomarket.org/fieldprint-calculator (accessed De-cember 2013).

Foley, J. A., et al. 2011. Solutions for a cultivated planet. Nature478:337–342.

Garnett, T., et al. 2013. Sustainable intensification in agriculture:premises and policies. Science 341:33–34.

Giovannucci, D., O. von Hagen, and J. Wozniak. 2014. Corporate so-cial responsibility and the role of voluntary sustainability standards.Pages 359–384 in C. Schmitz-Hoffmann, M. Schmidt, B. Hansmann,and D. Palekhov, editors. Voluntary standards systems—a contribu-tion to sustainable development. Springer, Berlin.

ISEAL Alliance. 2010. Assessing the impacts of social and environmentalstandards systems v1.0: ISEAL code of good practice. ISEAL Alliance,London.

Laurance W. F., J. Sayer, and K. G. Cassman. 2014. Agricultural ex-pansion and its impacts on tropical nature. Trends in Ecology &Evolution 29:107–116.

Levin, J. 2012. Profitability and sustainability in palm oil production:analysis of incremental financial costs and benefits of RSPO compli-ance. WWF, Washington, D.C.

Margoluis, R., C. Stem, V. Swaminathan, M. Brown, A. Johnson,G. Placci, N. Salafsky, and I. Tilders. 2013. Results chains: atool for conservation action design, management, and evalua-tion. Ecology and Society 18:22. Available from: http://www.ecologyandsociety.org/vol18/iss3/art22/.

Mascia, M., S. Pailler, M. L. Thieme, A. Rowe, M. C. Bottrill, F. Danielsen,J. Geldmann, R. Naidoo, A. S. Pullin, and N. D. Burgess. 2014. Com-monalities and complementarities among approaches to conserva-tion monitoring and evaluation. Biological Conservation 169:258–267.

Miteva, D. A., S. K. Pattanayak, and P. J. Ferraro. 2012. Evaluation of bio-diversity policy instruments: What works and what doesn’t? OxfordReview of Economic Policy 28:69–92.

Newton P., A. Agrawal, and L. Wollenberg. 2013. Enhancing the sustain-ability of commodity supply chains in tropical forest and agriculturallandscapes. Global Environmental Change 23:1761–1772.

Newton, P., H. N. Alves-Pinto, and L. F. Guedes Pinto. 2014. Certi-fication, forest conservation, and cattle: theories and evidence ofchange in Brazil. Conservation Letters DOI:10.1111/conl.12116.

Potts, J., M. Lynch, A. Wilkings, G. Huppe, M. Cunningham, andV. Voora. 2014. The state of sustainability initiatives review2014. International Institute for Sustainable Development, Win-nipeg and International Institute for Environment and Development,London.

Pretty, J., et al. 2010. The top 100 questions of importance to thefuture of global agriculture. International Journal of AgriculturalSustainability 8:219–236.

Pullin, A. S., and T. M. Knight. 2003. Support for decision makingin conservation practice: an evidence-based approach. Journal forNature Conservation 90:83–90.

SEARRP (South East Asia Rainforest Research Programme). 2012. SEn-SOR: an integrated multi-disciplinary research programme for sus-tainability. Available from http://www.searrp.org/sensor (accessedFebruary 2014).

Steering Committee of the State-of-Knowledge Assessment of Standardsand Certification. 2012. Toward sustainability: the roles and limita-tions of certification. RESOLVE, Washington, D.C.

Sutherland, W. J., et al. 2009. One hundred questions of importance tothe conservation of global biological diversity. Conservation Biology23:557–567.

Thomas, M., E. Baer, B. Semroc, and J. Sonenshine. 2012. C.A.F.E.practices results assessment, fiscal years 2011–2012. ConservationInternational, Arlington, Virginia.

Tscharntke, T., J. C. Milder, G. Schroth, Y. Clough, F. DeClerck, A. Wal-dron, R. Rice, and J. Ghazoul. 2014. Conserving biodiversity throughcertification of tropical agroforestry crops at local and landscapescales. Conservation Letters DOI:10.1111/conl.12110.

UNEP-WCMC. 2011. Review of the biodiversity requirements ofstandards and certification schemes. CBD Technical Series No.63. Secretariat of the Convention on Biological Diversity,Montreal.

Unilever. 2010. Unilever Sustainable Agriculture code. Available fromhttp://www.unilever.com/aboutus/supplier/sustainablesourcing(accessed December 2013).

Wollenberg, E., L. Merino, A. Agrawal, and E. Ostrom. 2007. Four-teen years of monitoring community-managed forests: learning fromIFRI’s experience. International Forestry Review 9:670–684.

Zrust, M., et al. 2013. HCV threat monitoring protocol. Zoological Soci-ety of London, London.


an agenda for assessing and improving conservation impacts ... · jeffrey c....

Documents