Numerical Model: MM5 Satellite

The MURI* Uncertainty Monitor (MUM) David W. Jones, Applied Physics Laboratory, University of Washington, Seattle, Washington and Susan Joslyn, Department of Psychology, University of Washington, Seattle, Washington

Introduction

UW researchers asked the question: Would Navy forecasters use additional uncertainty information? To answer the question UW researchers conducted two studies:

Study 1 Results: Forecasters used few sources of information, mostly models, and already evaluated model uncertainty on every forecast, but relied on rules of thumb to avoid the computationally intensive procedure of model comparisons.

Study 2 - Post-TAF Questionnaire Forecasters filled out questionnaires after their TAF indicating their sources of information and rating the numerical models’ performance. Researchers wanted to find out if they increased their evaluation techniques or used additional information sources when the models were judged to be less reliable.

Study 2 results: Forecasters’ routine varied little from one forecast to the next. There was no correlation between model ratings and forecasters’ evaluation strategies or their use of other information sources such as satellites or buoys.

Major Findings:

Naval forecasters are concerned about model uncertainty, but tend to avoid computationally intensive procedures such as

Examination and comparison of multiple models Comparisons to multiple sources Head-to-head comparisons between models Adjustments to the forecasting process to the

perceived model uncertainty Evaluation of model performance over previous

few days

Forecasting tasks that are working memory intensive are either offloaded to long term memory or avoided

Why? Human information processing tells us that people have large long-term memory capacity, but limited working memory capacity, which is aggravated under time pressure and the task switching demands common in the Navy.

Prototype Solution: MUM (MURI* Uncertainty Monitor)A software prototype that assembles, processes, and visualizes uncertainty information for weather forecasters.

Frees forecasters from the computational demands on their working memory.

Lets forecasters use models and techniques appropriate to specific location and situation.

Alerts forecasters to problems and encourages thorough evaluation.

Allows quick assessment of model uncertainty.

MUM Architecture – Based on Java Server Page technology that

only requires a browser to read and interact with the system.

Model data is produced at the lowest tier, post-processing at the second, and user interaction at the top tier.

MUM Interface – Information presented in past-present-future on left

control panel, bottom center and right control panels. Users select information needed on the control panels

and a visualized representation appears in the center. Task analysis revealed forecasters spent most of their

time reviewing model initialization. Thus, MUM’s default is the current model initialization field overlaid on top of the most current satellite picture.

MUM with ensemble spread meteogram

The ensemble spread meteogram displays information about the MM5 ensemble performance for a single geographic location and parameter over a four day period. The most recent 00 hour prediction lies at the center, marked by a bright vertical line.

* This research is supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745. The support of the sponsor is gratefully appreciated. The MUM system is based on Java Server Page (JSP) and servlet technology. It is hosted at the Applied Physics Laboratory (APL-UW) on a Linux system running a Tomcat server. The model data used in the system comes from the UW SREF. This includes the global fields used for the SREF boundary conditions and the individual ensemble members of the SREF. This data is stored and archived on the APL-UW server.

Approach: University of Washington researchers studied the forecasting task and the users who perform it, then built a tool to increase the forecasters’ ability to evaluate uncertainty.

Study 1 - Think aloud Verbal Protocol Analysis of Terminal Aerodrome Forecast (TAF) Forecasters thought aloud through the generation of their forecast to explain what they were doing and how they did it. They functioned under severe time pressure and often had to do other tasks simultaneously while producing their TAF.

Rules of thumbSet routines

Knowledge about model biases & strengths

Mental model of the atmosphere• Goals and steps to reach e.g.Make comparisons Models to satellite Models to models Avoid• Calculate differences• Maintain results of calculations & comparisons• Make decisions Sources of information Evaluation techniques Forecast

Long Term Memory Working Memory

GlobalVerificationStoplights

MM5 Uncertainty Stoplights

RMSEVerification Plots

SREFStoplight

ProbabilisticEnsemble forecasts

Meteograms

StoplightServlet

GlobalArchive

StoplightTable Tag

PlotImageGenServlet

StoplightServlet

MM5 Data Extractor

StoplightTable Tag

MeteogramPlotServlet

Zero-Hour Analysis /Satellite Overlays

External Link to AtmosphericSciences

External Link to At.Sci.

MM5 Data Extractor

Verification Portal

Uncertainty Portal

Link to Verification Portal

Link to Uncertainty Portal

MUM Architecture and

Data Flow

Mesoscale Error Plots

MVIRServletMesoscale ImageArchive

For more information, contact: David Jones, APL-UW (206-543-3236 dwjones@apl.washington.edu)

Problem: Navy weather forecasters rely on numerical models to produce forecasts, but also have to consider the level of uncertainty in the models, which can be crucial in the tactical decisions the military has to make.

ConclusionMUM will continue to be used to test methods of presentation and user interactivity toward the goal of improving forecast quality, timeliness, and usefulness. Forecasters can now use probabilistic information in new and innovative ways.