skunk works evaluation tools: how do we know if we are having an impact?
TRANSCRIPT
Skunk Works Evaluation Tools:
How do we know if we are having an impact?
What are we trying to achieve?
Strategies
Mind Shaping
Process: Activities
to demonstra
te application
Capacity
Building
Accessible ES Tools
Improved State of HWB and Biodiversity (Ecosystem Services)
Outcomes
Use of ES information in decisions
Operationalizing Outcomes
• Material living standards
• Physical health• Mental health• Social
connections• Spiritual
fulfillment• Cultural
identity• Security• Life
satisfaction• Equity
Use of ES and Policy
Decisions
Improved ES Improved HWB
• Increased policy documents with reference to ES as desired outcomes or factors in decision-making
• Increased discussion of ES in policy meetings
• Clean drinking water
• Reliable water flow in the dry season
• Reduced, in-stream sediment concentration
What other factors influence our intended outcomes?
Individual experience and personalities
Social and institutional context
Political climate
Other initiatives in place
Resources
• Land-use change (outside of decision)
• Climate change • Extreme events
(e.g., fire or flood)
• Other sources not related to natural resources
• Background factors (physical, mental, social health)
• Biophysical context
Use of ES and Policy
Decisions
Improved ES Improved HWB
So what to we need to consider?1) Stage of Evaluation
– At what stage are we in completing our strategies?
2) Design – How confident can we be that our intervention
caused the outcomes?
3) Data Collection Methods– Which methods are most valid to measure
expected outcomes?
Evaluation Stages
Developmental• What is promoting innovation in
the project?
Formative• How well are our strategies
getting us to our goals?
Summative (Impact)• Did it work?
Design
• Control and Experiment group/site receive same measurement before and after intervention
Quasi-experiment/
BACI• Control and experiment group/site
randomized and measured post-intervention
Posttest Only
• Pre and post measure the experimental group/site
One Group Pre-Posttest
• Control and Experimental group/site non-randomly assigned, measured post intervention
Natural Experiment
• Post intervention measurement onlyCase Study ✔
Some Example Methods…
To Test Mind Shaping:
e.g.• Stakeholder focus on ES• Mediation of different ideas• Understanding and Awareness of ES
Method: Mental Models
Method 1: Mental Models
Tells us: To what extent ES are being incorporated into Structure and Content of Knowledge and preferences
• Accomplished during an interview (1hr)• All or sample of participants, individual or group• Content analysis and stakeholder comparison• Improved with pre-post design, control group,
and/or randomization• Necessary: control for the hypothesized
intervention
To Test ES use in Decisions: e.g.• Policy documents that refer to ES as
desired outcomes or factors in decision-making
Method: Content Analysis
Method 2: Content Analysis
Tells us:The frequency and parameters of ES used to make a decision, and the relative influence of ES to other factors • Desktop document review• Can combine with clarifying interviews
with document preparers• Either inductive or deductive analysis (a
rubric helps)
To Test Increased HWB: e.g. • Physical Health• Life Satisfaction and Happiness• Political Voice and Governance
Method: Community-level Survey
Method 3: Community-level Survey
Tells us:Baseline and change values for actual and perceived HWB attributes
• Usually best developed based on ideas from interviews or focus groups
• In-person, paper or Web-based• Distributed among the target population• Some indicators may already exist from
local health departments or other agencies
A note on HWB measurements Qualitative and Quantitative
approaches necessary to hone in on appropriate indicators
Other Methods
• Network analysis: measure increase in information exchange, collaboration, and social capital related to ES and/or HWB
• Focus groups/interviews: Just about anything
• Participant Observation: Participant interaction with the tools and which are most effective at what
• Observation Protocols: Adaptation of ES specific terminology in meetings or trainings
Water quality
high
low
time
Conservation impact site
Control
Difference in mean values
improved water quality!
To Test Improved ES
e.g. improved water quality and dry-season flow
Methods: BACI and modeled controls
Method 4: BACI and modeled controlsTells us: Water quality relative to before the conservation impact and to control areas • Experimental backbone: Before-After-Control-
Impact (BACI) (conservation impact)• Modeled controls and exploration of “minimum
monitoring”• Post-treatment/impact only• Experimental controls to validate NatCap
models: model “no action” as a comparison