© crown copyright met office standard verification system for long-range forecasts (svslrf) richard...
TRANSCRIPT
© Crown copyright Met Office
Standard Verification System for Long-range Forecasts (SVSLRF)Richard Graham, Met Office Hadley Centre. With acknowledgements to all GPCs, LC-SVSLRF and members of the WMO Expert Team on Extended and Long-range Forecasting (ET-ELRF)
WMO/THORPEX/WCRP sub-seasonal to seasonal implementation planning meeting, WMO HQ, Geneva,
2-3 December 2011
Lead Centre for Standard Verification of Long-range forecasts (LC-SVSLRF)
• repository of global SVSLRF scores for GPC forecasts
• Core of SVS: ROC maps and diagrams, reliability diagrams, MSSS
• documentation on verification scores (Attachment II-8 of the WMO manual on the GDPFS)
• observation datasets for verification, and code for calculating the SVSLRF scores
http://www.bom.gov.au/wmo/lrfvs/
Jointly operated byBureau of Met. BoM) & Met. Service of Canada (MSC)David Jones, Bertrand Denis
SVSLRF: skill measures
Probability forecasts (for tercile categories):
• Relative Operating Characteristics (ROC): diagrams and scores (area under ROC curve)
• Reliability diagrams and sharpness diagrams
Deterministic forecasts (ensemble mean):
• Mean Square Skill Score (MSSS) and decomposition terms (correlation, variance, bias) – climatology as reference
Variables:
• T2m, precipitation, SST, Nino3.4 index
Level 1: ‘bulk’ scores and diagrams for regions/indices (for admin.)
• ROC curves and scores; reliability diagrams and freq histograms; MSSS
• Variables/indices: T2m, precip, Nino3.4
• Regions: Tropics (20N to 20S), N. Extratropics (20-90N); S. Extratropics (20-90S)
• Seasons: MAM, JJA, SON, DJF (Nino3.4: calendar months)
• Lead-times: all available (but not > 4 months for T2m and precip; not > 6 months for Nino3.4)
Level 2: scores at grid points for generating skill maps (for users)
• ROC scores; MSSS (and decomposition)
• Variables/indices: T2m, precip, SST
• Regions: global, each defined grid point
• Seasons: 12 rolling 3-month seasons
• Lead-times: all available (but not > 4 months)
• Stratification: Where sufficient hindcast years, stratify into seasons with El Niño active and seasons with La Niña active
Lead Centre SVSLRF
ROC curvesROC score maps
Reliability/sharpness diagrams
Courtesy: David Jones (BoM)
Sample verification: ROC scores for SON precipitation, 1-month lead
ToulouseWashington TokyoMelbourne
above
below
0.6
Some issues
• Each GPC calculates own scores and submits to LC-SVSLRF. Centralised verification would help standardisation (e.g. in category definition, verification datasets used) and speed processing;
• Good practice = all forecast products have corresponding verification products – big task, we are still catching up to a degree;
• Only GPC minimum variable set are required to be verified (T2m, precip, SST, Nino3.4) – but forecasts for other variables (e.g. Z500) are available
• No verification yet of individual month forecasts (available on LC-LRFMME website)
• No verification yet of the LC-LRFMME multi-model combinations
Thank you! Any questions?