theme 3 - iisd · module 12: adaptive management and evaluation stakeholder feedback • each taw...

Theme 3 Adaptive Thinking and

Committing to Action to

Prepare for the Future

Module 12: Adaptive Management and Evaluation

Module 12Adaptive Management

and Evaluation


Learning objectives

After completing this module, you will be able to:

1. Describe the principles of adaptive management and

summarize how watershed management can respond

to anticipated and unanticipated uncertainties.

2. Explain the role of monitoring and evaluation (M&E) and

key performance indicators and describe how to

implement appropriate M&E systems for adaptive

watersheds.



Key questions

• What is adaptive management? Why is it necessary? What are the key

elements of adaptive management?

• How is adaptive management implemented?

• How can we develop key performance indicators (KPIs), including long term

and near term?

• How can we develop effective M&E systems?


Rationale for adaptive systemsBased on Holling, 1978

Brought together principles of ecology, sociology and

economics in the context of environmental management

and policies.

• Our knowledge of a system is often surpassed by our

ignorance of aspects of it; plan for uncertainty.

• Humans have dealt with uncertainty through trial and

error.

• Ecological systems demonstrate a resilience that has a

counterpart in institutional and other systems.

• Complex systems (including watersheds) comprise

multiple variables that shift and move. Management

policies often try and minimize this variability, but must

instead understand and adapt with them.


Adaptive management

• Monitor and review impacts to understand what’s working

• Revise actionsfor maximum impact

• Implementing key actions based on goals

• Establishing goals and responsibilities Plan Do

CheckCorrect


Cycle of strategic adaptive management


Adaptive ecosystem-based management in watersheds

Source: Adapted from Roy, Barr and Venema, 2011


Visioning your watershed of the futureAdaptive watershed management brings together ecological, social, hydrologic,

climatic, economic and other priorities to develop management systems.

Fisheries

IrrigationAgriculture

Sanitation

and Waste

Drought

Local

bioenergy

Recreation

Source: IISD, 2010


Piloting portfolioof ecosystem initiatives• Implementation stage of strategic adaptive ecosystem management

• Manager pilots ecosystem initiatives aimed at achieving a shared long-term

outcome


Need for formal review (after Holling 1978)

1. Variability of ecological systems provides a self-

monitoring system that maintains resilience.

2. Regular review, even when the program is performing

well, and the use of well-designed pilots throughout the

life of the program to test assumptions related to

performance, can help address emerging issues and

trigger important adjustments for better impact.

3. Monitoring the wrong variable can falsely indicate no

change even when drastic change is imminent.

4. Impacts may not be gradual; they may appear abruptly

due to climate, human, or other drivers or pressures.

5. Existing impact assessment methods (e.g., cost-benefit

analysis, input output, cross-impact matrices, linear

models, discounting) do not recognize importance of

these interactions.


Formalize review systems

• Specified time period identified in planning stages

• Annual to 5-year reviews usually recommended

• Monitoring key performance indicators (KPIs) may

indicate shorter time interval needed to respond to

changing conditions

• When performance of an initiative is sensitive to a

specific input parameter or if impacts are potentially

serious but uncertain, KPIs can indicate key times for

additional review.

• Stakeholder feedback, including new scientific

information


Key performance indicators

(KPIs)• KPIs are the primary tool during this stage

• Different levels and time scales: Process; Outcomes, near-term/long-term;

Impact on ecosystem, community and resilience KPIs


Anatomy of a KPI Anatomy of a KPI

Title and axis labelunits

source

data legend

Target

Source: UNEP/DEWA/GRID-EUROPE, GEO Data portal: Compiles from Forest Stewardship Council (FSC) data)


Key characteristics of a KPI

SMART indicators

• Specific: Target a specific area of improvement

• Measurable: Quantify/qualify progress

• Achievable: Results must be possible in time and resources available

• Responsible: Specify who will actually do it

• Time-bound: Specify when the results can be achieved.


Near-term KPIs

• Activity-based outputs to measure completion of tasks and their results

• Early gauge of whether hypothesis upon which the initiative is based was

correct

• Can the actions deliver a positive ecosystem benefit?

Outputs

(knowledge

generated or

services

delivered)

Ecosystem initiative results show the impact of water retention

on fish population and water availability

KPI: % increase in downstream fish population

Target: 20%

Activities

(ecosystem

management

and adaptation

projects)

Development of water retention structures to improve fish

habitat and mitigate flood damage in the region.

KPI: Progress in developing small dams

Target: Completed on schedule and budget


Long-term KPIs

• Measure performance of actions based on impacts, priorities and

effectiveness as agreed-upon by stakeholders.

Results Chain

Ecosystem Initiative #1

Water retention pilot initiative

Long-term outcomes (change in state of society, economy, environment)

Improved water access for irrigation during dry seasons, flood damage mitigation and improved fisheriesKPI: Flood damage mitigated downstream; total fish populationTarget: 20% increase in fish population

Intermediate outcomes (increased capacity, awareness, access,etc.)

Awareness among regional land use managers, watershed managers and communities about the relationship between water storage, floods/drought and community resilience.KPI: # of planners attending planning sessionsTarget: Specific # of influential roles/people identified in the impact strategy)


Selecting your KPIs

• Aim to measure your strategic objectives

• Aim for a manageable number of KPIs

• Choose SMART KPIs, including a mix of near-term,

long-term, process, ecological, social, etc.

• Look for recurrent and consistent patterns

• Select KPIs based on available data, and also inputs

from key stakeholders to ensure the most legitimate,

evidence-oriented and culturally appropriate

measurement.

• Ensure inclusion of gender-responsive KPIs, as well as

those representing the needs of vulnerable groups


Long-term KPIs

Results Chain


Spillway Pilot Initiative


Fish Ladder Pilot Initiative

Ultimate Outcomes

(change in state of

environment, society,

economy)

Restoration of salmon population and hydropower that can meet demand

KPI: Total salmon population

Target: 20% above baseline counts within 5 years

Intermediate

Outcomes

(new/improved policy

or practice)

More frequent spillway operation

KPI: Total spillway operation time

Target: X hours more per month

Permanent increase in fish ladder capacity

KPI: salmon count immediately downstream of ladder

Target: X% of upstream count

Intermediate

Outcomes

(increased

awareness, capacity

or access)

Awareness among hydropower policy-makers that

increased spillway operation is a feasible means to

increase salmon population

KPI: # of hydropower planners and policy-makers

attending presentation on results of spillway experiments

Target: (this target should include the specific names of

influential persons identified in the impact strategy)

Awareness among hydropower policy-makers that

improved fish ladder technology can increase salmon

population

KPI: # of hydropower planners and policy-makers attending

presentation on results of fish ladder experiments

Target: (this target should include the specific names of

influential persons identified in the impact strategy)

Outputs

(knowledge

generated or

services delivered)

Ecosystem initiative results showing the impact of

spillway operation on salmon population

KPI: % increase in downstream salmon population

Target: 20%

Ecosystem initiative results showing the impact of fish

ladder operation on salmon population

KPI: % increase in downstream salmon population

Target: 20%

Activities

(ecosystem

management

projects)

Ecosystem initiative to test the impact of increased

spillway operation on salmon population (including

salmon population and stream flow monitoring)

KPI: Progress toward completion of ecosystem experiment

Target: Completed on schedule

Ecosystem initiative to test the impact of improved fish

ladder design on salmon population (including salmon

population and stream flow monitoring)

KPI: Progress toward completion of ecosystem experiment

Target: Completed on schedule


Activity: KPIs - Part I (20 min)

Based on your draft watershed plan and priorities, develop 10 illustrative KPIs,

including actions, and social, economic and environmental impact indicators.

Once you have 10, test these against the SMART criteria

KPIs

# of small check dams built (#) (Act.) Outputs/actions

Fish population increased (#) (Env.) Short/long-term outcomes

Fisheries-based economic growth ($/year) (Econ.) Short/long-term outcomes

Volume of flood-water retained (m3) (Env.) Short-term outcomes

# of irrigable acres increased (Econ.) Short/long-term outcomes

# of people with better access to water

(disaggregate by gender and community) (Soc.)

Long-term outcome

Improved resilience to floods and drought (Soc.) Ultimate outcome


Activity: KPIs - Part II

KPIs Type SMART?

# of small check dams built (#) (Act.) Outputs/action

s

Fish population increased (#) (Env.) Short/long-

term outcomes

Fisheries-based economic growth ($/year) (Econ.) Short/long-

term outcomes

Volume of flood-water retained (m3) (Env.) Short-term

outcomes

# of irrigable acres increased (Econ.) Short/long-

term outcomes

# of people with better access to water

(disaggregate by gender and community) (Soc.)

Long-term

outcome

Improved resilience to floods and drought (Soc.) Ultimate

outcome


Monitoring, evaluation and reporting• Recognize the role of monitoring and reporting in the development and

implementation of KPIs, and the adaptive watershed.


Monitoring, evaluation and reporting• Monitoring: Systematic process of observation. Follows

a course of activities, comparing what happens with

what was expected. Observes delivery of ecosystem

services and that progress is made towards meeting

societal goals

• Evaluation: Assesses achievement against preset

criteria. Evaluation of an EM plan determines extent to

which ecosystem service levels meet intended

purposes; identifies lessons learned for subsequent

phases of EM

• Reporting: The process of regular sharing of

information with project stakeholders and the public to

ensure participation, transparency, accountability and

innovation.


Developing effective monitoring and evaluation systems• Begin by reviewing the high-level KPIs appropriate for

your adaptive watershed plan.

• Identify primary users of evaluation, including those

most affected by actions, people whose perception will

control whether or not evaluation is used to guide

evolution of our EM plan

• Identify existing monitoring systems available in your

region.

• Compile ecological, social and economic measures that

could help with the completion of your KPIs

• Prioritize component parameters to understand how

systems are functioning, and how they respond to

development, climate and other pressures.


Climate-hydrologic monitoring systems


A role for citizen science

• Citizen science includes the collection and

analysis of data by members of the general

public, typically as part of a collaborative project

with professional scientists.

• Citizen Science is gaining popularity in different

parts of the world, particularly in the field of

weather and water-related monitoring.


Integrated evaluation

• Along with information and data on climate, land

and water systems, incorporate key information

and data related to KPIs selected for the health of

the watershed.

• E.g., for community water access, you need to

think about access data (e.g. time travelled to get

water, or hours of running water/day)


Integrated evaluation

Integrated Evaluation Systems

Evaluation Summary and Reporting

Summarizing available

data/info

First-hand monitoring

systems

Citizen science efforts

with formal protocols

Monitoring protocols for all desired elements (the what, when, how, and who

of monitoring systems)

KPIs selected for process and outcomes

Stakeholder priorities and requirements


Stakeholder feedback

• Each TAW initiative should have an expert team that reviews feedback and

develops ways to respond.

• Evaluation should incorporate sound data, but also views of key

stakeholders to ensure that both qualitative and quantitative information are

captured.

• Aggregated stakeholder feedback can tell an important story about an

emerging issue or an unintended consequence of an initiative.


Activity: monitoring and evaluation (20 min)

1. Looking back at the illustrative KPIs you developed for your watershed, can

you link them clearly to broader adaptive watershed priorities?

2. Are all the primary users represented in your evaluation? Where are the

gaps? (e.g., women’s use of local fisheries not included)

3. Identify key/potential sources of data for these KPIs; specify quantitative

(incl. analytical) and qualitative (deliberative) approaches

4. Set priorities, areas and frequency of monitoring.


Final discussion points

1. Your monitoring plan is a key tool to evaluate your

watershed adaptive plan and change course as

necessary.

2. Ensure that your KPIs and monitoring take into account

climate-related vulnerabilities as well as women and

Indigenous community needs.

3. Your KPIs and monitoring systems should be adaptive

and can change with changing priorities.

4. Consider how evaluations will be reported to different

stakeholders. There is a range of reporting mechanisms

including data portal, websites, published “report cards”

and meetings.

theme 3 - iisd · module 12: adaptive management and evaluation stakeholder feedback • each taw...

Documents