User studies: contributing to developing regional decision support for climate

services

Andrea J. Ray, Roger S. Pulwarty and

Robert S. Webb

NOAA-CIRES Climate Diagnostics CenterClimate Prediction Assessments Workshop

Tallahassee, FL

9-11 March 2004

“The timely production and delivery of useful climate data, information, and knowledge to decision makers.”

National Research Council, 2001

• NRC recommends that a regional focus is needed to address societal needs, developed through regional decision support systems.

• Pillars of climate services include:

–Regional decision support networks,

–Research,

–Observing systems, and

–Mechanisms to enable user-centric design and improvement.

To be effective, regional decision support systems must not only provide data and forecasts, but also determine:

–what problems and questions are important and critical, and why, to be determined by consensus between users and providers of products;

–what information is needed to solve problems, on short and long-term time scales;

–what transaction costs are involved in developing and maintaining long term partnerships and interactions.

• Many “regional decision support,” networks exist,

–NOAA weather forecast offices,data centers, river forecast centers,

–Regional climate centers,

–State climatologist offices,

–Agricultural extension services

–Some private companies

• These networks provide data and forecasts, but in order to meet the NRC user-centric goals of climate services, these organizations would also:

–adapt products to meet specific user needs,

–support major user groups,

–encourage dialogue with users, and

–develop a regional focus to address societal needs.

• These networks generally are not set up to do effective user-studies

Effective user-centric design and implementation

• To achieve user-centric design and implementation these interactions must:

–identify entry points for information within existing decisionmaking and the space for communicating this information;

–identify research needed, including strategic, applied, and adaptive;

–create nodes for users to engage with researchers and providers, to help determine what products are to be developed and tested;

–Characterize needed products: e.g., scale, units, formats, how accessible, when issued.

–Inform the development of alternative decisionmaking strategies as needed

• Mechanisms for studying users need to be built in to regional networks.

Effective user-centric design and implementation

• Sustained and systematic communication is required on three fronts:

–between users and providers (NOAA and regional);

–between researchers and providers, and

–between researchers and users.

• Why user studies?

–More than consultancy

–More than a simple conversation or communication (not anecdotal)

–Understand the complexity of user context and differences in perspectives between users and scientists

Designing effective user studies

• User-study approach

• Based on case studies

• Based on

–need to get past differences in perspective;

–the innovation-decision process;

–definitions of “use”

• Dimensions to characterize about a user-case

Differences in perspectiveFactor Scientist’s perspective Water Manager’s Perspective

Identifying a critical issue

Based on a broad understanding of the nature of water management

Based on experience of a particular system

Time frame Variable Immediate (operations)

Long-term (infrastructure)

Spatial resolution Defined by data availability or funding Defined by institutional boundaries or authorities

Goals Prediction

Explanation

Understanding of natural system

Optimization of multiple conditions and minimization of risk

Basis for Decisions Generalizing multiple facts and observations

Use of scientific procedures and methods

Availability of research funding

Disciplinary perspective

Tradition; Procedure

Professional judgment; Training

Economics; Politics

Job risks

Expectation Understanding

Prediction

Ongoing improvement (project is never actually complete)

Statistical significance of results

Innovations in methods/theory

Accuracy of information

Appropriate methodology

Save money and time; Protect the public;

Protect their jobs, agendas or institutions

Product Characteristics

Complex

Scientifically defensible

As simple as possible without losing accuracy

Importance of context

Frame Physical (atmospheric, hydrologic, etc.) conditions as drivers

Dependent on scientific discipline

Safety and well being

Profit

Consistency with institutional culture, policy, etc.

Nature of Use Conceptual Applied

Innovation-Decision process: conceptual steps in deciding whether to adopt an innovation

• Knowledge phase: learns about the existence of an innovation, perhaps thru an outreach function

• Persuasion phase: forms a favorable/unfavorable opinion• Decision phase: engages in activities leading to adopting or

rejecting• Implementation: puts the innovation into use • Confirmation phase: decision to adopt is reinforced, or could be

reversed if experience is adverse (Rogers,1995)

Path towards adoption and use: studies and mechanisms for developing user-centric products should take advantage of this

Definitions of “use”

• Consult: the product is consulted, e.g., looked up on a web page or received as a briefing or from other source (type1);

• Consider: after consulting the product, it is considered in management deliberations as a factor potentially influencing decisions, but not necessarily in operational models (type 2);

- forecasts may be used in this way when they are not in appropriate forms for use in operational models

• Incorporate: some form of the forecast is incorporated into an operational model that is utilized in operational decisions (type 3).

• Dialogue about risks: communication of risk, i.e. the forecast is used to communicate with other managers and stakeholders about the risk of certain conditions and about the need to take actions, or to justify actions (type 4).

User Study Approach

• One mechanism for assessing user needs• Dimensions to characterize about the user-context:

– Identify critical problems which are sensitive to climate– Identify decisionmakers and their key stakeholders – Assess how climate variability interacts with their critical

problems• Decision calendar may help organize recurring decisions and

show when products are needed

– Identify user groups who may be willing partners in testing and prototyping this new technology

• Potential users in a rapidly evolving phase of their critical problem may be more open to interacting as partners

• Social changes, legal or policy change, or a climate event such as drought or flood

User group is the reservoir managers in the Colorado headwaters and Gunnison basins who operate reservoirs originally built to provide reliable irrigation supply, hydropower, and recreation

Redlands Diversion dam and fish ladder, Gunnison river near Grand Junction, CO

Case study: Multi-purpose Reservoir Management in the Upper Colorado River Basin

Dimensions for Aspinall Unit Case

• Critical problem sensitive to climate– Evolving reservoir management in which demands now are closely

balanced with supply but new uses are being legally proscribed– More intensely managed water system is more sensitive to

shortage because there is less buffer in the system

• Decisionmakers and their key stakeholders– USBR has authority to manage, but USFWS, NPS, have interacting

legal authorities, other stakeho

• How climate variability interacts w/ the critical problem– Decision calendar for annual operating plan helped organize

recurring decisions and ID potential climate

• User groups who may be willing partners/adopters– Deadlines to write ESA recovery plans and formal filing for a federal

reserve water right for a national park are requiring USBR to accommodate these in their operations

– 2002 drought as a climate event (not forecasted) also resulted in finding flexibility and new ways of operating

Potential Uses in the Aspinall case

• Identified a number of potential uses of seasonal climate forecasts in their operational models, as well as shorter term forecasts

• However, seasonal forecasts are currently not available in a form that can be incorporated into their operational models. Current use is to consult these products and consider them in management deliberations

• Aspinall decisionmakers use climate information (e.g. during drought) in a dialogue with their stakeholders about the risk of certain conditions and about the need to take actions, or to justify actions.

• Projects with Martyn Clark (WWA) and Dave Brandon (CBRFC) are working to translate both MRF forecasts and CPC seasonal forecasts

Conclusions I

• Regional decision support networks exist, but are not designed to carry on the systematic study of and interaction with users to ensure the development of user-centric products and services

• Mechanisms need to be built into the regional decision support networks

• These mechanisms could ensure user-centric design and implementation by:

–identifying entry points for information within existing decisionmaking and the space for communicating this information, e.g., through basin water management groups, water availability task forces, emergency management meetings

–identifying research needed, including strategic, applied, and adaptive;

–enabling RDS networks to function as nodes for users, researchers and providers to engage in dialogue and to help determine what products are to be developed and tested;

–effectively characterizing needed products and synthesize user needs from among many interactions (beyond consultancy)

Conclusions II

• User studies are a means to create and sustain the communication and learning required for user-centric design and development – and also a cost.

–create a dialogue about climate risks and not just provide information;

–create a dialogue about the types of climate products desired by a broad spectrum of users by integrating knowledge about what climate products are

needed.

• NOAA’s mechanisms for developing user-centric products and services need to:

–Recognize that differences in perspective exist and limit scientists ability to create user-centric products – unless we deliberately work to see other perspectives

–“Use” of products is broader than just in operational models: recognize that there is a range of uses and create products for all types. Evaluation of “use” should consider all types.

–Adoption is a process: develop mechanisms to draw potential users along this path and evaluate whether activities support evolution of use

• Training needed across the RDSS networks in how to conduct user-assessments

EXTRAS

Summer season forecast for Peak Augmentation planning

Jan Feb Mar Apr May Jun July Aug Sep OctOct Nov DecAug Sep

Provide for late Summer/early Fall irrigation while maintaining target flows

Next water year runoff unknown, reserve water until February snowpack observations

Winter season precipitation forecast for Fall release decisions

Winter releases based on January/February snowpack observations

Operational issuesOperational issues

Potential use of forecastsPotential use of forecasts

Legend:

Planning ProcessPlanning Process

Winter/Spring forecast for Winter release decisions

Peak Flow Augmentation - fill curve

Week 2 forecasts for Peak Augmentation

Peak Flow Augmentation releases

Plan releases for Summer irrigation & hydropower

Week 2 forecasts for Summer irrigation & hydropower release decisions

Reservoir Management Decision Calendar

Provide for Summer irrigation & hydropower needs while maintaining target flows

Water Year Planning Next Water Year Planning

Ray

et

al.

(200

0)

Figure 4-28

Differences in perspective

• Identifying a critical issue

• Time frame

• Goals

• Basis for decisions

• Expectiation

• Product characteristics

• Frame

Characteristics of effective user studies

• Dimensions to characterize about a user-case

Desired information not available

Understanding Barriers New Services

How to they understand climate? What do they use now?

Can we produce forecasts in a manner related to how they think about climate?

Are the problems critical to the user influenced by climate? If so, are there

products available to address these?

Are there circumstances that minimize constraints?

User studies seek answers to the right hand questions

Other factors more

important

Constraints prevent flexible response

Fail tounderstand