Deutscher Wetterdienst Preliminary evaluation and verification of the pre-operational COSMO-DE Ensemble Prediction System Susanne Theis Christoph Gebhardt, Zied Ben Bouallgue, Carlos Peralta, Michael Buchhold EMS September 2011 Presentation Overview Setup of COSMO-DE-EPS First results of pre-operational phase Objective verification Forecasters feedback; evaluation of case studies Deutscher Wetterdienst Setup of COSMO-DE-EPS EMS September 2011 COSMO-DE-EPS status pre-operational phase has started: Dec 9 th, 2010 pre-operational setup: 20 members (upgrade to 40 member in 2012) grid size: 2.8 km convection-permitting lead time: 0-21 hours, 8 starts per day (00, 03, 06,... UTC) variations in physics, initial conditions, lateral boundaries model domain EMS September 2011 Generation of Ensemble Members multi-configuration different configurations of COSMO-DE model Variations in Forecast System for the Representation of Forecast Uncertainty Initial Conditions BoundariesModel Physics multi-model driven by different global models multi-model COSMO-DE initial conditions modified by different global models EMS September 2011 Generation of Ensemble Members COSMO-DE-EPS 2.8km COSMO 7km plus variations of initial conditions model physics BC-EPS GME, IFS, GFS, GSM Ensemble Chain Deutscher Wetterdienst First Results of Pre-operational Phase - Objective verification - Forecasters feedback; Evaluation of case studies EMS September 2011 Focus on precipitation Ensemble Members Probabilities of Precipitation SYNOP RADAR Verification EMS September 2011 PREC 1h accumulation Do the ensemble members have different long-term statistics? (multi-model / multi- configuration) Are there many cases with the same best member or wettest member? Only small differences in long-term statistics Members may be treated as equally probable Only small differences in long-term statistics Members may be treated as equally probable DETERMINISTIC SCORES for Individual Members } 20 members Forecast Time [h] IFS GME GFS GSM Equitable Threat Score JUNE 2011 FBI threshold 1 mm threshold 0.1 mm EMS September 2011 PREC 1h accumulation How well does the ensemble spread of the forecast represent the true variability of the observation? RANK HISTOGRAM a) Underdispersiveness relatively small b) Four groups Many cases with large influence by global models a) Underdispersiveness relatively small b) Four groups Many cases with large influence by global models Frequency JANUARY 2011 Rank JUNE 2011 Rank Frequency EMS September 2011 PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) JANUARY Forecast Time [h] > 0.1 mm > 1 mm > 2 mm EMS September 2011 PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) JUNE Forecast Time [h] > 0.1 mm > 1 mm > 2 mm EMS September 2011 PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) - for all foreast lead times BRIER SKILL SCORE For larger precipitation amounts (summer): even more additional value For larger precipitation amounts (summer): even more additional value MAY - JULY Threshold [mm/h] EMS September 2011 PREC 1h accumulation RELIABILITY DIAGRAM Reliability diagram shows some bias and underdispersiveness Lines are sloped additional calibration has good potential Reliability diagram shows some bias and underdispersiveness Lines are sloped additional calibration has good potential log (# fcst) > 0.1 mm > 1 mm > 2 mm JUNE 2011 Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say % did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) EMS September 2011 Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,) Ensemble underdispersiveness is relatively small Ensemble members may be treated as equally probable Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation Summary of Verification (Precipitation) EMS September 2011 Other Variables T_2M and VMAX have been verified ensemble spread is far too small COSMO-DE ensemble prediction system has been developed with focus on precipitation COSMO-DE ensemble prediction system has been developed with focus on precipitation Upgrade to 40 members will also look at other variables V_MAX 10m Jan./Feb./ Mar. nevertheless, ensemble provides additional value to COSMO-DE First Results of Pre-operational Phase - verification - forecasters feedback; evaluation of case studies Observation: 12h-precip, 18 UTC, Synop and Radar Severe weather Rostock EMS September Rostock (2) COSMO-DE: 12h-precip Synop: 18 UTC, 12h precip COSMO-DE 00 UTC + 18 h Synop: 18 UTC, 12h precip COSMO-DE 06 UTC + 12 h EMS September UTC-run (previous day) +21h 00 UTC-run UTC-run +15 COSMO-DE-EPS: Prob > 25 mm, 12h-RR UTC Good consistency of successive model runs Rostock (3) EMS September UTC-Lauf (Vortag) +21h 00 UTC-Lauf UTC-Lauf +15 COSMO-DE-EPS: Prob > 40 mm, 12h-RR UTC Probabilities provide good information on location and intensity of the event Rostock (4) EMS September h-prec, SRNWP PEPS 00+18h 12h-prec, COSMO-DE-EPS 00+18h Rostock (2) Prob > 40 mm, 12h-prec UTC COSMO-DE-EPS is able to capture heavy convective precip events provides optimal guidance for forecasters (in this case) COSMO-DE-EPS is able to capture heavy convective precip events provides optimal guidance for forecasters (in this case) EMS September , 06 UTC 1h precip [mm/h] Synop Radar : Severe storm Bremen , 06 UTC 1h precip [mm/h] Synop COSMO-DE deterministic 21 UTC + 09 h Forecaster: Where to expect the main shower activity and what will be maximum intensity? EMS September 2011 Synop 06 UTC, 1h precip C-EPS upscaled prob >15mm 18 UTC + 12 h Synop 06 UTC, 1h precip C-EPS Max of all members 18 UTC + 12 h Probability defines area of occurrence Maxmem (or 90% quantile) provides estimation of possible intensities : Bremen (2) EMS September : cold front NRW Radar and Synop: 19 UTC, 1h precip COSMO-DE deterministic 12 UTC + 07 h Large location error in C-DE, horizontal extent of severe precip. too small EMS September : cold front NRW (2) Radar, Synop: 19 UTC, 1h precip C-DE-EPS upscaled Prob >15mm/h 12 UTC + 7 h Radar, Synop: 19 UTC, 1h precip C-DE-EPS Perc90% 12 UTC + 7 h C-DE-EPS: better localization and intensity estimation But: BC-EPS does not adequately capture uncertainty on synoptic scale C-DE-EPS: better localization and intensity estimation But: BC-EPS does not adequately capture uncertainty on synoptic scale EMS September : severe storm Frankfurt Radar 19 UTC, 1h precip Synop 19 UTC, max wind (1h) COSMO-DE max wind (1h), 09 UTC + 08 h Synop 19 UTC, max wind (1h) C-DE determ. run: no indication of gale-force winds (fx>29 m/s) EMS September : Frankfurt (2) C-EPS 90%quant max wind (1h), 06 UTC + 11 h Synop 19 UTC, max wind (1h) C-EPS maxwind prob > 29m/s 06 UTC + 11 h Synop 19 UTC, max wind (1h) C-DE -EPS: clear signals for gale-force winds above 29 m/s. Useful guidance for forecasters despite of timing error EMS September 2011 Forecasters Feedback what they prefer to use: 90%-quantile or maximum of all members of precipitation precipitation probabilities for an area (10x10 grid points) what they appreciate: early signals for heavy precipitation indication that deterministic run may be wrong what they criticize: jumpiness between subsequent runs lack of spread in T_2M and VMAX what they are learning: dealing with low probabilities (10% probability for extreme weather issue a warning?) EMS September 2011 Generation of Ensemble Members Perturbation Methods Peralta, C., Ben Bouallgue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research. Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115123. Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallgue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, (contains status of 2009) EMS September : Thunderstorm Mnchen C-EPS upscal prob > 10mm, 00 UTC + 21 h Synop 21 UTC, 1h precip C-DE -EPS: 21 and 12h probability forecasts provide useful information on the storm C-EPS upscal prob > 10mm, 00 UTC + 12 h EMS September 2011 C-EPS upscal prob > 10mm, 12 UTC + 09 h Synop 21 UTC, 1h precip C-EPS upscal prob > 10mm, 15 UTC + 06 h C-DE -EPS: Event lost in forecast 12 UTC+9h. Jumpiness of successive runs is a problem! C-DE -EPS: Event lost in forecast 12 UTC+9h. Jumpiness of successive runs is a problem! EMS September 2011 COSMO-DE-EPS plans ( ) upgrade to 40 members, redesign statistical postprocessing initial conditions by LETKF lateral boundary conditions by ICON EPS reach operational status 2013