1 what’s new in verification? a verification briefing for the saws iv workshop october 26, 2011...
TRANSCRIPT
1
What’s New in Verification?A Verification Briefing
for the SAWS IV Workshop
October 26, 2011Chuck Kluepfel
National Weather Service HeadquartersSilver Spring, Maryland
301-713-0090 x132
http://www.metacafe.com/watch/1686664/fedex_planes_dodging_storm/
Starter Videos
http://www.youtube.com/watch?v=6E_Z_Ve-ayA
3
History
• NOAA (GSD) was contracted to measure the lead time with which NWS TAFs predict the onset and cessation of IFR conditions (2-category, IFR vs. VFR).
• This software package has been available thru the NWS Performance Management Web site for 2 years.
• Next 6 months: Joint NWS/FAA TRWG (Traffic Flow Management Weather Requirements Working Group) is directing the NWS Performance Branch to measure TAF performance, using more specific Decision Support requirements.
4
GSD Software
• Allows for tight and loose accuracy standards.
• Verifies IFR onsets similar to the way NWS verifies warnings.
• LT = IFR onset time - TAF issuance time.
• Uses the TAF with the longest LT.
9
TRWG Decision Support Software
• Requirements have just been defined.
• Code writing should begin soon.
• Very specific requirements set by TRWG.
• Measure NWS skill in forecasting the Onset and Cessation of events critical to operations.
10
TRWG Decision Support Software
• Measure the Onset and Cessation of
• IFR (Phase 1)• VFR (Phase 1)• MVFR (Phase 1)
11
TRWG Decision Support Software
• Measure the Onset and Cessation of
• IFR (Phase 1)• VFR (Phase 1)• MVFR (Phase 1)• Wind shifts (Phase 2)• Wind speed changes (Phase 2)
12
TRWG Decision Support Software
• Measure the Onset and Cessation of
• IFR (Phase 1)• VFR (Phase 1)• MVFR (Phase 1)• Wind shifts (Phase 2)• Wind speed changes (Phase 2)• Thunderstorms (Phase 3)• Freezing precipitation (Phase 3)
13
TRWG Decision Support Software
The Onset of these conditions:
• IFR (Onset - always deterioration)• VFR (Onset - always improvement or clearing)
14
TRWG Decision Support Software
• Measure the Onset and Cessation of the Following Events
• IFR • VFR• MVFR (from IFR – Onset represents clearing)• MVFR (from VFR – Onset represents deterioration)
20
TRWG Decision Support SoftwareMethodology
• For each hit, determine whether the forecast timing error met the TRWG requirements for scoring a POD hit:
• For 1-hour LT: ± 15 minutes• For 2-hour LT: ± 15 minutes• For 4-hour LT: ± 30 minutes • For 6-hour LT: ± 45 minutes • For 8-hour LT: ± 60 minutes
30
Traditional Stats forModified Southwest United States:
ColoradoNew Mexico
UtahArizonaNevada
California
Minus these WFOs: San Diego, Los Angeles, San Francisco
32
Modified SW US – Fiscal Year 2011Scheduled 3-6 hr IFR and Below
GFS LAMP
(569K - 264K) K ÷ (511K – 482K) ~ 10.5The 3-6 hr GFS LAMP false alarmed over 10 times for every additional hit it got over the forecasters!
POD 0.56
FAR 0.53
CSI 0.35
ForecastYes No
O
b
s
Yes 511 K 402 K
No 569 K 13.1 Million
Prevailing
POD 0.53
FAR 0.35
CSI 0.41
ForecastYes No
O
b
s
Yes 482 K 430 K
No 264 K 13.4 Million
33
WFOs MTR, LOX, SGX FY 2011Scheduled 3-6 hr IFR and Below
GFS LAMP
POD 0.54
FAR 0.43
CSI 0.39
ForecastYes No
O
b
s
Yes 213 K 182 K
No 158 K 2.7 Million
Prevailing
POD 0.64
FAR 0.31
CSI 0.50
ForecastYes No
O
b
s
Yes 251 K 144 K
No 113 K 2.7 Million