joshua m. boustead and daniel nietfeld noaa/nws wfo omaha/valley, ne ray wolf
DESCRIPTION
An Experiment to Evaluate the Use of Quantitative Precipitation Forecasts from Numerical Guidance by Operational Forecasters. Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf NOAA/NWS WFO Davenport, IA. Presentation Overview. Study purpose and methodology - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/1.jpg)
An Experiment to Evaluate the Use of An Experiment to Evaluate the Use of Quantitative Precipitation Forecasts from Quantitative Precipitation Forecasts from
Numerical Guidance by Operational Numerical Guidance by Operational ForecastersForecasters
Joshua M. Boustead and Daniel NietfeldJoshua M. Boustead and Daniel Nietfeld
NOAA/NWS WFO Omaha/Valley, NENOAA/NWS WFO Omaha/Valley, NE
Ray Wolf Ray Wolf
NOAA/NWS WFO Davenport, IANOAA/NWS WFO Davenport, IA
![Page 2: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/2.jpg)
Presentation Overview
• Study purpose and methodology
• Data results– Survey results– Snowfall forecast– Watch/warning statistics – Gridded forecast results
• Forecasting implications, conclusions, and future work
![Page 3: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/3.jpg)
Study Motivation
• Strong interest in the role of the future forecaster– Can we still add value to the everyday
forecast?– How can we better concentrate on high-
impact weather?– How can we better utilize increasingly high-
tech tools into the everyday forecast?• How does this increasingly high-tech information
affect the forecaster?
![Page 4: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/4.jpg)
ExampleNSSL 4km WRF 00Z 8/14/07
![Page 5: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/5.jpg)
Results
![Page 6: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/6.jpg)
Study Purpose
• To evaluate if and how operational forecasters are biased by numerically generated quantitative precipitation forecasts (QPF)
• Use these results to develop an updated methodology for operational forecasters on how to approach a daily forecast and utilize the latest technology, including high resolution model output
![Page 7: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/7.jpg)
Study Methodology
• Utilizing the National Weather Service’s (NWS) Warning Event Simulator (WES) operational forecasters from two NWS offices made two forecasts for two winter weather case– The forecasters first completed the forecast,
including making a warning decision, without the use of model QPF
– The forecasters then went through the same case again with model QPF, again making a snowfall forecast as well as a warning decision
• Once each scenario was completed, the forecasters completed a survey about the specific case
![Page 8: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/8.jpg)
Study Methodology
• Two winter weather cases were chosen from the Central and Northern Plains– December 7-8, 2005
from the Pleasant Hill, MO (EAX) forecast area
– February 28 – March 1, 2004 from the Bismarck, ND (BIS) forecast area
![Page 9: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/9.jpg)
Survey ResultsDistribution of Forecaster Experience
0%
10%
20%
30%
40%
50%
60%
0 to 3 yrs 3 to 10 yrs 10 to 20 yrs 20 + yrs
Years of Operational Forecasting
Perc
en
t o
f F
ore
caste
rs
• Forecaster Demographics:– Forecasters were from the NWS offices in
Omaha/Valley, NE and Davenport, IA– Operational forecasters involved were of a high
experience level
![Page 10: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/10.jpg)
Survey Results
• Forecaster confidence without using model QPF:– Majority of operational forecasters felt confident in making a
forecast without model QPF– Potentially due to the high experience level of the forecasters
Forecaster Confidence Level without Using QPF
0%
10%
20%
30%
40%
50%
60%
Very Confident Confident Neutral Unsure Very Unsure
![Page 11: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/11.jpg)
Survey Results
• Forecaster confidence after seeing QPF:– Most forecasters indicated that seeing QPF either
increased their forecast confidence or it was unchanged
Change in Forecaster Confidence with QPF
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Sig increased Increase Unchanged Decreased Sig decreased
![Page 12: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/12.jpg)
Snowfall Forecast Results
• MAE was computed for each location and then averaged for before and after the use of QPF
• MAE decreased 0.5 inches for both the EAX and BIS case post QPF
Pre and Post QPF Mean Absolute Error for EAX and BIS
00.5
11.5
22.5
3
3.54
4.55
Pre and Post QPF EAX Pre and Post QPF BIS
Inch
es Pre-QPF
Post-QPF
![Page 13: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/13.jpg)
Snowfall Forecast Results
• Majority of the forecasts were unchanged post QPF
• Majority of the forecasts that did change their forecast, increased accuracy
Distribution of Change in Forecast Accuracy with QPF among Forecasters for EAX
34%
29%
37% Improved Degraded
Unchanged
Distribution of Change in Forecast Accuracy with QPF among Forecasters for BIS
38%
26%
36% Improved Degraded
Unchanged
![Page 14: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/14.jpg)
Warning Results
• The probability of detection (POD) and false alarm ratio (FAR) were computed by county for each of the forecast areas
• Forecasters showed improvement in both the POD and in FAR ratio once QPF was used
Warning Statistics
0%
10%
20%
30%
40%
50%
60%
70%
80%
Pre-QPF POD and FAR Post-QPF POD and FAR
POD
FAR
![Page 15: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/15.jpg)
Warning Results
Distribution of Change in Probability of Detection Accuracy with QPF for BIS
30%
13%
57%
Improved Degraded
Unchanged
Distribution of Change in False Alarm Ratio Accuracy with QPF for BIS
29%
25%
46%
Improved Degraded
Unchanged
Distrubution of Change in Probability of Detection Accuracy with QPF for EAX
29%
21%
50%
Improved Degraded
Unchanged
Distrubution of Change in False Alarm Ratio Accuracy with QPF for EAX
33%
22%
45%
Improved Degraded
Unchanged
![Page 16: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/16.jpg)
Gridded Forecast ResultsEAX Case
• EAX Pre and Post QPF MAE– Forecasters had
the most confidence in the northern CWA
– Much better agreement over the southern CWA post QPF
– Also a 2 to 3 inch decrease in MAE over the south
![Page 17: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/17.jpg)
Gridded Forecast ResultsEAX Case
• EAX Pre and Post QPF Standard Deviation– Forecast
differences decreased over the north and south
– Slight increase in differences over the center
![Page 18: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/18.jpg)
Gridded Model ForecastsEAX Case
• Greatest agreement of snow band across central CWA
• Viewing QPF increased the forecast confidence in the southern CWA
![Page 19: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/19.jpg)
Actual SnowfallEAX Case
![Page 20: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/20.jpg)
Gridded Forecast ResultsBismarck Case
• BIS Pre and Post QPF MAE– Good agreement
and low error over the northwest forecast area
– Mean errors of 5 to 6 inches over the southern and eastern forecast area
![Page 21: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/21.jpg)
Gridded Forecast ResultsBismarck Case
• Pre and Post QPF Standard Deviation– Significant
increase in forecaster clustering across the central forecast area
– Greater than 4 inch differences continue over the southern forecast area
![Page 22: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/22.jpg)
Gridded Model ForecastsBIS Case
• Models agree northwest CWA to get least QPF
• Larger uncertainty in the south
• Forecasters tended to pick a model– Led to continued
large MAE in the southern CWA
![Page 23: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/23.jpg)
Actual SnowfallBIS Case
![Page 24: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/24.jpg)
Discussion• Only a slight improvement in snowfall forecasts was
noted once forecasters viewed QPF– When snowfall forecasts were modified, a higher percentage
were improved than degraded• Model QPF seemed best utilized to resolve snow-no
snow areas– This led to improvements in both FAR and POD
• High MAE did not always mean high standard deviation, which can indicate a systematic forecasting error
• Doesn’t clearly answer the question does model QPF bias forecasters– Some evidence in the BIS case where model agreement was
poor• Possible forecast methodology
– Make entire forecast without QPF– Utilize QPF for placement for defining snow-no snow areas
![Page 25: Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fea550346895daae0ed/html5/thumbnails/25.jpg)
Future Work
• Conduct the study using two warm season convective cases
• Investigate forecaster philosophy from the surveys where standard deviation is low and mean absolute error is higher
• Investigate what, if any, synoptic patterns increase forecaster uncertainty and MAE
• Continue to increase the number of forecasters in the study, and from different areas of the CONUS