verification & weather impacts: past and future

April 22, 2023 Southern Region SOO-NASA/SPoRT Joint Workshop 1

Verification &Weather Impacts:Past and Future

Verification &Weather Impacts:Past and Future

Mark Fox

Regional Training Officer

Mark Fox

Regional Training Officer


National Weather ServiceNational Weather Service

Currently Verify:– TOR / SVR / FFW ( POD – FAR – CSI )– Max / Min T ( in various forms ) – PoP ( brier, reliability, and other forms )– Marine Wind Speed / Wave Heights– TAF (various forms)

• 50% Offices say their “Best Service” is Direct and Personal Communications

Currently Verify:– TOR / SVR / FFW ( POD – FAR – CSI )– Max / Min T ( in various forms ) – PoP ( brier, reliability, and other forms )– Marine Wind Speed / Wave Heights– TAF (various forms)

• 50% Offices say their “Best Service” is Direct and Personal Communications


A Goal: A Goal:• Begin the culture change of “beating the model” to ‘Get

it right when it matters.’• Focus should be on:

– Impact Weather– Accurate Grids

• How ?– Change Habits– Change perceptions of the models

• “Garbage that the model spits out”

– Change definition of success ?– Change what we measure ?– Reward what we do (even if it fails!) ?

• Good AFD with a “bad” MAE• Or: Praise a “bad” forecast for the right reasons

• Begin the culture change of “beating the model” to ‘Get it right when it matters.’

• Focus should be on: – Impact Weather– Accurate Grids

• How ?– Change Habits– Change perceptions of the models

• “Garbage that the model spits out”

– Change definition of success ?– Change what we measure ?– Reward what we do (even if it fails!) ?

• Good AFD with a “bad” MAE• Or: Praise a “bad” forecast for the right reasons


Weather ImpactsWeather Impacts

• The Customer Defines the Impact– Not always what we think the impact is

– We know the meteorology (and the limitations of the meteorology)

• SSDs Working Definition of Impact Weather:– A weather event that causes a substantial departure

from the normal routine. The event must be forecastable with sufficient lead time to allow decision makers to take appropriate action that results in a saving of lives and/or a reduction in property damage or other adverse economic impacts.

• The Customer Defines the Impact– Not always what we think the impact is

– We know the meteorology (and the limitations of the meteorology)

• SSDs Working Definition of Impact Weather:– A weather event that causes a substantial departure

from the normal routine. The event must be forecastable with sufficient lead time to allow decision makers to take appropriate action that results in a saving of lives and/or a reduction in property damage or other adverse economic impacts.


Forecasting for Weather ImpactsForecasting for Weather Impacts• SR Grid policy is designed to allow

forecasters to concentrate on the weather that will create impacts

• WISE Concept came out in 2005– SSD / CWWD survey in early 2006:

• 90% said our office are using the grid policy, but our neighbors are not.

• Idea behind WISE, or looking for “windows of opportunity” is to let the model do what it does best and allow the forecaster to do what he/she does best.

• SR Grid policy is designed to allow forecasters to concentrate on the weather that will create impacts

• WISE Concept came out in 2005– SSD / CWWD survey in early 2006:

• 90% said our office are using the grid policy, but our neighbors are not.

• Idea behind WISE, or looking for “windows of opportunity” is to let the model do what it does best and allow the forecaster to do what he/she does best.


WISE? What about ConOps ?WISE? What about ConOps ?

• ConOps is discussing infrastructure (grid size, cluster, equipment, etc.), but not operations

• We have the opportunity to influence operations for the ConOps

• Already being done in many offices

• ConOps is discussing infrastructure (grid size, cluster, equipment, etc.), but not operations

• We have the opportunity to influence operations for the ConOps

• Already being done in many offices


Southern Region Grid PolicySouthern Region Grid Policy• Does the Model Make Meteorological

Sense ?– Yes : Use it– No : Don’t

• Everyone uses the same thing as the starting point to collaborate the forecast

• Concentrate on the Impact Events or where the model is off base– Routine errors minimized

• Does the Model Make Meteorological Sense ?– Yes : Use it– No : Don’t

• Everyone uses the same thing as the starting point to collaborate the forecast

• Concentrate on the Impact Events or where the model is off base– Routine errors minimized


Southern Region Grids - BeforeSouthern Region Grids - Before

Max / Min T Forecasts by RegionSeptember 2004 - May 2005

-10-8-6-4-202468

10

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR CR ER WR


-10-8-6-4-202468

10

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR CR ER WR


Southern Region Grids - AfterSouthern Region Grids - After


-10-8-6-4-202468

10

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR CR ER WR


-10-8-6-4-202468

10

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR CR ER WR


Why the Jump ?Why the Jump ?• Regime ?

– Western drought – Above normal winter– Model MAE

• 4.2 in 2005 / 2006• 4.1 in 2004 / 2005

• Man / Machine Mix ?– Forecast more than 4 degrees from MOS– 13 % of the time in 2005 – 2006– 17 % of the time in 2004 - 2005

• Regime ?– Western drought – Above normal winter– Model MAE

• 4.2 in 2005 / 2006• 4.1 in 2004 / 2005

• Man / Machine Mix ?– Forecast more than 4 degrees from MOS– 13 % of the time in 2005 – 2006– 17 % of the time in 2004 - 2005


A Possible ReasonA Possible Reason

Temperature ForecastsWhen Differing from Guidance > 4 Degrees

-20

-10

0

10

20

30

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

2004 - 2005 2005 - 2006

Temperature ForecastsWhen Differing from Guidance > 4 Degrees

-20

-10

0

10

20

30

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

2004 - 2005 2005 - 2006


Southern Region Grids - BeforeSouthern Region Grids - Before

PoP Forecasts by RegionSeptember 2004 - May 2005

-10

-8

-6

-4

-2

0

2

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR ER CR WR


-10

-8

-6

-4

-2

0

2

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR ER CR WR


Southern Region Grids - AfterSouthern Region Grids - After


-10

-8

-6

-4

-2

0

2

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR ER CR WR


-10

-8

-6

-4

-2

0

2

4

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

SR ER CR WR


Why the Jump ?Why the Jump ?• Regime ?

– Western drought – Above normal winter– Model

• 0.11 Brier Score in 2005 / 2006• 0.08 Brier Score in 2004 / 2005

• Man / Machine Mix ?– Forecast more than 20 % from MOS– 6 % of the time in 2005 – 2006– 10 % of the time in 2004 - 2005

• Regime ?– Western drought – Above normal winter– Model

• 0.11 Brier Score in 2005 / 2006• 0.08 Brier Score in 2004 / 2005

• Man / Machine Mix ?– Forecast more than 20 % from MOS– 6 % of the time in 2005 – 2006– 10 % of the time in 2004 - 2005


A Possible Reason ?A Possible Reason ?

PoP ForecastsWhen Differing from Guidance > 20 %

-20

-15

-10

-5

0

5

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

2004 - 2005 2005 - 2006

PoP ForecastsWhen Differing from Guidance > 20 %

-20

-15

-10

-5

0

5

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14

Forecast Period

% I

mpr

ovem

ent

over

MO

S

2004 - 2005 2005 - 2006


The Future ?The Future ?• But…wouldn’t grid PoP verification

make more sense ??

• [email protected]

• But…wouldn’t grid PoP verification make more sense ??



Severe Weather WarningsSevere Weather Warnings• Public Forecast Verification Nice for

Forecasters– Not Tied to GPRA Goals

• Hazardous Weather Warnings are in the Bread and Butter of NWS– Tied to GPRA Goals

• Public Forecast Verification Nice for Forecasters– Not Tied to GPRA Goals

• Hazardous Weather Warnings are in the Bread and Butter of NWS– Tied to GPRA Goals


Tornado WarningsTornado Warnings

All Tornado Events

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

POD LT > 0

All Tornado Events

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

POD LT > 0


Tornado WarningsTornado Warnings

Tornado EventsWith One or More Fatalities

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

POD LT > 0

Tornado EventsWith One or More Fatalities

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Year

POD LT > 0


Making the ShiftMaking the Shift• Get away from the routine

– Need a database instead of ‘Stats on Demand’• [email protected]

• Verify the stuff that matters– Grids instead of points (lots of issues here)

• BOI Verify

– Large temperature swings– Critical Temperatures– PoP timing– Hazards and threats

• Get away from the routine – Need a database instead of ‘Stats on Demand’


• Verify the stuff that matters– Grids instead of points (lots of issues here)

• BOI Verify

– Large temperature swings– Critical Temperatures– PoP timing– Hazards and threats


Making the ShiftMaking the Shift• Verify the stuff that matters

– Mississippi Valley Tornado Outbreak– DFW Snow / Sleet– West Texas December Cold Snap– Florida Fires– Fire Weather Season in General– Anything else I can’t think of…

• How many of us were told:– Great job beating MOS !!!

• Verify the stuff that matters– Mississippi Valley Tornado Outbreak– DFW Snow / Sleet– West Texas December Cold Snap– Florida Fires– Fire Weather Season in General– Anything else I can’t think of…

• How many of us were told:– Great job beating MOS !!!

verification & weather impacts: past and future

Documents

weather event

definition of impact

verification weather

weather impactssr grid

bad forecast

brier score

normal routine

impact iswe