Water, Land & Ecosystems Intervention Decisions

Where are the high information values?

Keith Shepherd & Doug HubbardNov 2012

Vagen

Water, Land & Ecosystems (WLE)Organizing research around a conceptual framework of basins

and landscapes

WLE Decisions

Our strategic objectives = our system level outcomes: (i) decrease food insecurity(ii) manage environmental resources(iii) reduce poverty among farmers(iv) increase nutrition, health and wellbeing We aim to improve stakeholder decisions on policies, intervention programmes and intervention designs through research

What information has high value for improving decisions to achieve these outcomes?

•How to prioritize research under uncertaintyWhich interventions will reduce risk, increase security, and

improve lives the most? What are the trade-offs between competing objectives, like agricultural productivity and the environment? What are the risks of intervention failure?How to measure and monitor development outcomes

•Potentially huge investments in monitoring but not all metrics will be of equal value to support intervention decisions. How should we determine what data gathering costs are justified?How to show the value of research

How can we show how the expense of research is justified by better intervention decisions and improved outcomes?

Challenges Facing Researchers

•Development of systems to measure the impact of CGIAR investments (of relevance to DFID as a significant funder) at the level of the 4 system outcomes. Mechanisms to analyse the impacts and trade-offs associated with sustainable intensification at different scales (sub-national, national, regional).Value for money metrics for measuring agriculture, ecosystem and poverty and nutritional outcomes.

Interests of donors

Why must quantify uncertainty

•Averages are wrong on average

•Uncertain events (floods, droughts, erosion, market fluctuations)

•Security is a development outcome (food/nutritional security; risk is the complement of security)

•Value of information

Walsh

How much information do we need?

What defines whether information is unreasonably expensive?

What is the value of doing one more survey or experiment, or creating another database?

Organizations often spend 10 times the value of information on surveys and trials, etc

[Ron Howard]

We need a method to quantify information value

How to make preferences explicit

Objective trade-offs •The trade-offs between productivity, ecosystem and welfare outcomes

Valuation of outcomes (Preferences, Policy)•Valuing one outcome relative to another (production vs environment)•Time (benefit now versus later)•Uncertainty (risk aversion)•Equity (increasing income of poor worth more than non-poor)

Making preferences explicit improves transparency and multi-stakeholder decision processes

Applied Information Economics

HubbardHubbard

© Hubbard Decision Research, 2012

Uses of Applied Information Economics

AIE was applied initially to IT business cases. But over the last 17 years it has also been applied to other decision analysis problems in all areas of Business

Cases, Performance Metrics, Risk Analysis, and Portfolio Prioritization.

• Prioritizing IT portfoliosPrioritizing IT portfolios• Risk of software Risk of software

developmentdevelopment• Value of better informationValue of better information• Value of better securityValue of better security• Risk of obsolescence and Risk of obsolescence and

optimal technology optimal technology upgradesupgrades

• Value of infrastructureValue of infrastructure• Performance metrics for the Performance metrics for the

business value of business value of applicationsapplications

ITIT

• Risks of major engineering Risks of major engineering projectsprojects

• Risk of mine floodingRisk of mine flooding

EngineeringEngineering

• Movie / film project selectionMovie / film project selection• New product developmentNew product development• PharmaceuticalsPharmaceuticals• Medical devicesMedical devices• PublishingPublishing

BusinessBusiness Civilian GovernmentCivilian Government

• Environmental policyEnvironmental policy• Procurement / auction Procurement / auction

methodsmethods• Grants managementGrants management

MilitaryMilitary

• Forecasting battlefield fuel Forecasting battlefield fuel consumptionconsumption

• Effectiveness of combat Effectiveness of combat training to reduce roadside training to reduce roadside bomb / IED casualtiesbomb / IED casualties

• R&D portfoliosR&D portfolios

Payback is 20:1 to 300:1

The AIE Process

Identify important metrics for monitoring implementation

Improve the intervention design to reduce chance of negative outcomes

Hubbard

Forecasting intervention impacts

Value of information

Game theory provided a formula for the economic value of information over 60 years ago:

Expected Opportunity Loss = the chance of being wrong x the cost of being wrong

Expected Value of Information is the reduction in the EOL as a result of the additional information.

AIE Empirical Evidence•We are not as clear as we think on the decisions we are trying to

influence

•Expressing uncertainty dissolves assumptions & allows all benefits, costs and risks to be included, however intangible (especially environment!)

•We need calibrating to reliably estimate probability distributions

•There are usually only a few variables with high information value

•We are often measuring the variables that have least economic value

•And completely missing the ones that do have value (e.g. tend to measure costs but ignore benefits, which are typically uncertain).

•Measurement is uncertainty reduction, not a gold standard

•Often need different data than we think

•Often need less data than we think

•Even small reductions in uncertainty can have considerable value

© Hubbard Decision Research, 2012

Safe Drinking Water Information System

• The EPA needed to compute the ROI of the Safe Drinking Water Information System (SDWIS)

• As with any AIE project, we built a spreadsheet model that connected the expected effects of the system to relevant impacts – in this case public health and its economic value

Input sheet

Need for calibration training

Cash flow page

Risk report page

Cost-effective Measurement• Fermi decompositionEstimate no. of piano tuners in Chicago= No. households (population/people per

household)x % of households with tuned pianosx tuning frequency per year / (tunings per day x work days per year

• Secondary research - measured before? Historic data

• Observation - sampling, tracers, experiment

[From Hubbard 2010]

Value of Information

A Probability Management SystemDecision modelling defines the metrics

Smart data - Smart decisions

Quantifying WLE Intervention Outcomes

Time value preference

Years

Environmentally rational

Poor farmer

Next stepsPhase 1•Analysis of 4 - 6 WLE intervention categories/cases in 2013•The outcomes define the agro-ecosystem metrics databases•Decision Analyst

Phase 2•Standardized databases and stochastic libraries•Generic intervention screening model (triage method)

Phase 3•Develop intervention decision modelling platform (linked stochastic libraries, visualization tools) •Analysis of WLE or CGIAR project portfolio

Smart data - Smart decisions