Hydrologic Verification

Verification of Deterministic River Stage Forecasts

Edwin Welles

OST SeminarJune 6, 2007

Outline of This Talk● Introductory comments about river forecasts.

– Is there value in verifying stage forecasts?

● Results of evaluating limited sample of NWS stage forecasts.– Proposal: Hydrologists should verify their forecasts.

● Results of a hindcasting study on headwater basins.– Proposal 2: We can use verification information to guide our

forecast process development.

● Standardized Hydrologic Verification– Just advertising

2

The beneficial .....and not so beneficial uses of water.

3

Impacts of Floods and Droughts

– Floods kill about 100 people a year

– Approximate annual economic losses: Floods - $5 billion

Droughts - $6-8 billion

Flood

Hurricane

Winter Storm

Lightning

Tornado

Cold

Heat

Average Annual Deaths (1990-99): 5021414

HeatHeat

FloodFlood

193193

9999

5555

5757

5858

2626

4

Mitigating the Impacts

● Forecasts and data can help us manage our water resources and mitigate the impacts of floods.

● NWS started issuing hydrologic forecasts in 1890.

5

Data and Forecasts to Meet Diverse Needs

Radar Data

River Gage Data Weather Observations

Snow Cover/Melt DataPrecipitation

Forecasts

Climate PredictionsReservoirReleases

PrecipitationEstimates

Satellite Data

The Approach to River Forecasting

Models

6

Forecast Modeling Structure

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARYFREE

SUPPLE-M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARY

FREE

SUPPLE-

M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARY

FREE

SUPPLE-M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARY

FREE

SUPPLE-

M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARY

FREE

SUPPLE-M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARYFREE

SUPPLE-

M ENTALFREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

Snow

Runoff

Overland FlowRouting

ChannelRouting

Reservoir

...for many rivers,

with many reservoirs.

Elevation zoneboundary

ForecastPoint

RatingCurves

7

The Forecast Process

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARYFREE

SUPPLE-M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

BULLETINFLOOD WARNINGNATIONAL WEATHER SERVICE SHREVEPORT LA1033 AM CDT WED APR 13 2005

Data Assimilation

Forecast:p, T, PE, Q

res, Q

Observed:p, T, PE, Q

res, Stg

Verification

8

Where’s the …..?9

● … Verification

● Little verification of hydrologic forecasts has been conducted to date.

● My Main Point – Need to fill this void in both

Research and Operations.● So let’s look at some

verification metrics

10

Where’s the …..?

Description of the Data

● Two sets of River Stage forecasts from NWS RFCs

● OK Dataset– 4 Locations in Oklahoma– 1993 to 2002– 1 to 4 basins above the forecast point– Response time measured in hours

● MM Dataset– 11 locations along the mainstem of the Missouri river– 1983 to 2002 – 500 to 1000 basins above the forecast point– Response time measured in days

● Generated a Persistence forecast as a reference– Observation at time of forecast persisted into the future

11

Acrobat Document

12

13

Acrobat Document

14

Initial Observationsfrom this limited sample

• Day 1 and Day 2 forecasts are accurate and skillful (as compared to persistence).

• Little skill in day 3• Little change over the periods of record.

• Need to conduct a more complete study to establish a comprehensive baseline so we can answer this basic question.– What is the skill of hydrologic forecasts?

15

V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N

Leverage Data and Systems from NOAA and Collaborators

Valuable Data and Forecasts to Satisfy Diverse Customer Needs

Radar Data

River Gage Data Weather Observations

Snow Cover/Melt DataPrecipitation

Forecasts

Climate PredictionsReservoirReleases

PrecipitationEstimates

Satellite Data

Improving The Approach to River Forecasting

Models

16

Some More Basic Questionsfor which we do not have answers

● How is new science improving the forecasts?

● What are the largest sources of error in the forecasts?

● What should be done to improve the forecasts?

How can verification can help us answer these questions?

17

A Hindcasting Experiment to Analyze Sources of Error

● Generated hindcasts – Three headwater basisns in Oklahoma and Missouri.– Out to three day lead times – Four years (1997 to 2000)

● Twelve total scenarios using…– 2 calibrations

● A “good” and an “a priori” calibration– 2 state updating methods

● Updated initial conditions and not-updated initial conditions– 3 QPF scenarios

● Perfect QPF– The observations

● Actual QPF – Computed QPF for 24 hours and then zero for days 2 and 3

● Zero QPF– Zero for all 3 days

18

Analysis Method● For each scenario

– Computed standard verifiction metrics on 4 subsets of the data

● Sorted into high and low stages● Sorted by observations and by forecasts

– Sorting by observations – Discrimination – Sorting by forecasts – Reliability

– Presenting just RMSE for High stage discrimination

● Compared scenarios with simple differences between the metrics.

● Evaluated error in the QPF also.

19

Comparing the Calibrations

Discrimination ReliabilityRMSE ME R RMSE ME R

Low: Calibrated 1.3 0.2 0.85 1.3 0.2 0.85Low: Uncalibrated 3.5 -0.1 0.51 2.6 -0.6 0.50

High: Calibrated 2.8 0.0 0.65 3.0 0.0 0.75High: Uncalibrated 5.8 3.2 0.55 11.5 10.5 0.35Table 8. Summary statistics to compare the model calibrations.

20

ME (byobs)

RMSE(by obs)

ME (byfcst)

RMSE(by fcst)

Samples(by obs)

Samples(by fcst)

Actual QPF <=25 mm 0.2 2.5 0.1 3.0 31800 31920Zero QPF <=25 mm -0.7 2.6 -0.7 3.5 31800 31946

Actual QPF >25 mm -22.8 25.7 15.5 20.3 146 26Zero QPF >25 mm -33.2 34.7 NA NA 146 NA

Table 9. The actual QPF compared to the zero QPF for the three hindcast basins.

ME(by obs)

RMSE(by obs)

ME (byfcst)

RMSE(by fcst)

FAR POD

National <=25 mm 0.1 2.4 0.1 2.4 na na

Hindcast basins <=25 mm 0.2 2.5 0.1 3.0 na naNational >25 mm -24.7 29.1 16 23 0.76 0.10

Hindcast basins >25 mm -22.8 25.7 15.5 20.3 0.77 0.04

Table 10. National Precipitation Verification Unit QPF statistics and the QPF statisticsfor the 3 hindcast basins.

The Skill of the Input Forecasts 21

Comparing the State Updating Scenarios

● State updating appears to bring similar skill to the hindcasts as the calibration in the early periods.

● QPF and state updating skill appear independent.

22

23

Comparing theCalibration Scenarios

● State updating appears to bring similar skill to the hindcasts as the calibration in the early periods.

● Improving the calibration may degrade forecast skill depending upon the QPF characteristics.

● Need good QPF to realize benefits of improved calibration.

24

25

Comparing the QPF Scenarios

● Using some QPF almost always improves the forecasts (1 exception) for the High Stages– Not so for the low stages.

● Can capitalize on the improved QPF even with poor calibration.– But get more improvement with better calibration

26

27

28

Improving The Forecast Process

INTERFLOWSURFACERUNOFF

INFILTRATIONTENSION

TENSION TENSION

PERCOLATION

LOWERZONE

UPPERZONE

PRIM ARYFREE

SUPPLE-M ENTAL

FREE

RESERVED RESERVED

FREE

EVAPOTRANSPIRATION

BASEFLOW

SUBSURFACEOUTFLOW

DIRECTRUNOFF

BULLETINFLOOD WARNINGNATIONAL WEATHER SERVICE SHREVEPORT LA1033 AM CDT WED APR 13 2005

Data Assimilation

Forecast:p, T, PE, Q

res, Q

Observed:p, T, PE, Q

res, Stg

Verification

Verification

VerificationVerification

Verification

Verification

Verification

Verification

Verification

Verification

Help needed for > day 1 forecasts

The hindcasts tell us ....

29

Critical for < day 1 forecasts

Standardized Verification for Hydrologic Forecasts

U N W SNWSU

Without Verification With Verification

It is hard to communicate without a common language. StandardizedVerification is the common language forecasters and researchers need.

This Professor

has a really

good idea.

This one used

verification to

explain her idea.

30

Standardized Verification for Hydrologic Forecasts

● Supports research by identifying needs AND by clarifying the value of results.

● Supports operational agencies by defining acceptable methods.

● They are expected to evolve, but we must start somewhere.

● Use peer review to establish validity.

31

HydrologicForecastProcess

Supported by a complete verificationsystem.

ModelsRuntime

Observations

Parameters

Model States

Final Forecast

Automatic Forecast

DataAssimilation

ForecasterAnalysisReivew

Forecasts Models

DataQualityControl

ModelCalibration

HistoricalObservations

Models

Model Simulationwith No DA

Model Stateswith No DA

Automatic Forecastwith No DA

AutomaticForecast

InputForecasts

Model Setup

Data Assimilation

ForecastGeneration

ForecastReview

FinalForecast

Veri f icat ion

32

Summary

● Verification methods can identify sources of error.– Demonstrated with hindcast experiment

● Propose Standardized Verification will enhance verification efforts and moving research to operations.

● Identified a need● Fill the hydrologic verification void.

● Documented an initial description of NWS river stage forecast skill.

● Need to verify more forecasts to update baseline

33