Are Numerical Weather

Prediction Models Getting Better?

Cliff Mass, David Ovens, and Jeff Baars

University of Washington

Getting Better?

• The overall performance of numerical weather prediction has gotten better over the past several decades.

• But such improvements could have come from a number of sources:

• Increased resolution• Better data assimilation• Improved global data assets (e.g., satellite data)• Better models and associated parameterizations.

The Question of This Talk

To what degree are our modeling systems IMPROVING in their ability to deliver better simulations and forecasts?

Additional Question

To What Degree Do We Actually KNOW how much our models are improving?

Most modeling systems have shown a flattening of the

improvement curve the last 5 years or so—even with improving

data for initialization and increasing resolution

ECMWF Flat

GFS and NAM Heavy Precipitation Threat Scores: Flat-lined

500 hPaDay 2

Most Global Models Show Minimal Improvement the Last 5 Years

What about mesoscale models like WRF?

Developmental Testbed Center (DTC) Reference Configuration Tests and

Comparisons (3.6 versus 3.4)

Purple: 3.6Green: 3.4

Upper AirTemperature

No real overall improvement between WRF 3.4 and 3.6

Let’s Check Out the Long-Term Verification of the Northwest

Regional Prediction System• Frozen MM5 driven by operational National

Weather Service NAM • Continuous upgraded WRF driven by

operational NWS GFS and recently by RAP• So we are driving the local models with

supposedly improving national guidance.

July 1, 2006 June 1, 2015

Bottom Line: No evidence of significant improvement in UW

WRF over 9 years, even with synoptic models getting slightly

better

But let’s do this more rigorously

• Run a history of WRF models from substantially different “eras” using the SAME large scale forcing.

• Jan 1-Feb 28, 2010, Pacific Northwest Domain• Attempted to keep physics parameterizations

consistent (but they are being updated as the modeling system changes)

WRF Versions Tested

• 2.2.1 - October 31, 2007 • 3.1.1 - July 31, 2009 • 3.5 - April 18, 2013• 3.7 - April 20, 2015

A stroll through WRF history

• WRF version 2.2.1 with Kain-Fritsch cumulus parameterization, simple ice physics, YSU PBL, Noah LSM, and RRTM longwave and Dudhia shortwave radiation

• WRF version 3.1.1 with Kain-Fritsch cumulus parameterization, Thompson microphysics, YSU PBL, Noah LSM, and RRTM longwave and Dudhia shortwave radiation

• WRF version 3.5 with Kain-Fritsch cumulus parameterization, Thompson microphysics, YSU PBL, Noah LSM, and RRTMG longwave and shortwave radiation

• WRF version 3.7 with Kain-Fritsch cumulus parameterization, Thompson microphysics, YSU PBL, Noah LSM, and RRTMG longwave and shortwave radiation

Some Results

Sea Level Pressure (24h Forecast)

2.2.1 WINSForecasts GET WORSE OverTime

Wind Speed

3.5 and 3.7 Better Than Early Versions3.5 lower MAE than 3.7

2-m Temperature

3.1.1 has lowest MAE, 2.2.1 Worst MAE and ME

Wind Direction

3.5 best, followed by 3.72.21 better than 3.11

Conclusions (based on many more stats)

• The WRF modeling system does not get steadily better, but staggers forward in a halting manner

• A new version of the modeling system is not necessarily better.

• Some statistics have generally improved, others have not.

• Precipitation (not shown above) has been one of the winners (improvement over the last decade). Positive Definite advection helped.

Other modeling systems are no better than WRF

Why Slow Progress? Some Possibilities

• NWP is now a mature technology and we should not expect significant advances

• There is no organized, coherent system in place to improve physics parameterizations and model numerics/structure.

• There is a lack of organized verification/evaluation of model simulations to find problems.

Without detail evaluations of the strengths of weaknesses of our modeling system are we driving

blind?

Our Challenge• We need a completely new paradigm on how

we develop and fund NWP model development, and how we apply the enormous resources of our community more effectively.

• Need to bring together the huge, uncoordinated NWP research effort in the U.S

• Reduce the number of models, determine the key weaknesses in NWP systems, and direct resources to rectify them.

The END