Evaluation of the ability of Numerical Weather Prediction models run in

support of IHOP to predict the evolution of Mesoscale Convective Systems

Steve Koch, Linda Wharton, Andy LougheForecast Systems Laboratory

Bill Gallus, Jeremy Grams

Iowa State University

Beth EbertAustralian Bureau of Meteorology

Background Background ((Weisman, Skamarock, and Klemp, 1997: The resolution dependence of explicitly

modeled convective systems, Mon. Wea. Rev., 125, 527-548))

1. Results from 3D midlatitude squall-line simulations suggest that 4 km grid size is needed to reproduce much of the mesoscale structure and evolution of MCSs seen in 1-km simulations.

2. Evolution at resolutions coarser than 4 km is characteristically slower, due largely to a delayed strengthening of the cold pool.

3. Cold pool is crucial to the evolution of an MCS into an upshear-tilted mature system.

4. An overly strong mesoscale circulation and overprediction of precipitation results at resolutions coarser than 4 km.

MCS: Mesoscale Convective System

Objectives of this StudyObjectives of this Study1. Determine forecast precipitation properties for each type of observed MCS for

the 12-km NWP models run during IHOP (Eta, MM5, and WRF). Operational models in the U.S. use resolutions of 12-20 km presently, not 4 km.

2. Traditional “measures-oriented” verification statistics (RMS, bias, etc.) severely penalize an incorrectly located precipitation system that may be forecast with only small positional or shape error, yet have practical forecast utility. We use an “object-oriented” verification technique in this study.

3. For each MCS type, we obtain systematic model performances by using the Ebert-McBride (2000) technique to determine the fractional contribution of forecast precipitation displacement, intensity, and shape errors.

4. The EM technique was implemented in the Real-Time Verification System (RTVS) at FSL and changes were made to the EM code as needed to make it applicable to mesoscale systems in the central U.S.

Ebert-McBride (EM) Verification TechniqueEbert-McBride (EM) Verification TechniqueEM Technique takes maximum of observed and forecasted rain at all pointsEM Technique takes maximum of observed and forecasted rain at all pointsand determines Contiguous Rain Areas (CRAs) exceeding specified isohyetand determines Contiguous Rain Areas (CRAs) exceeding specified isohyet

Forecast is permitted to shift within expanded CRA box by user-defined amount, until either RMSE is minimized or correlation coefficient maximized

OBS

FCST

FCST

CRA BOXCRA BOX

Expanded CRA boxExpanded CRA box

Application of the EM technique to 6h accumulated precipitation ending at 0600 UTC 13 June 2002

Strategy using Ebert-McBride technique

• Eta, WRF, and MM5 12-km runs from IHOP period were analyzed using the EM Contiguous Rain Area (CRA) technique

• Observed CRAs were assigned a morphology based on 2-km composite reflectivity radar images (30 minute time resolution)

• We classified the morphology of MM5 and WRF model convective systems using hourly reflectivity output from the models. Only six-hourly output was available for the Eta model.

• Required that an MCS meet the following criteria for at least 3 hours:

30 dBZ (~3 mm/h) over at least a 100 x 100 km area

40 dBZ (~13 mm/h) over at least a 50 x 50 km area

Improvements to EM CRA code

1. Percent of grid points allowed to shift off domain reduced from 50% to 0.1% for RMSE minimization and to 25% for correlation coefficient maximization (removes problem of displacements usually being off edge)

2. Plots changed to show entire expanded CRA region, with shifted forecast overlaid on observed rainfall chart

3. Correlation Coefficient maximization seems to produce more reasonable results than minimization of RMSE, but the error decomposition required development of a new decomposition (four terms) based on Murphy (1995)

4. Critical mass threshold for 24 hr periods was reduced by a factor of 4, allowing a greater number of smaller CRAs to be identified

Improvements to EM CRA code

5. Increasing threshold amount from .25 to .50 inch (13 mm) per 6 hours helps to identify distinct systems, but statistics are then computed over too small an area (so we kept the threshold at .25 inch)

6. Because models at best only resolve 6x features, a Lanczos filter was introduced to filter observations – creating patterns more similar to that forecast in the models

7. Sensitivity tests of tuneable parameters such as expansion of CRA domain, and minimum size (grid points) of CRA/observed rainfall area were performed, but results did not indicate the need for changes

8. CRA statistics for error decomposition continue to be computed over union of observed, forecasted and shifted forecast of rain. Volume and average rain rate are determined just over appropriate portion of CRA

•Linearo Linear (CL)

o Linear Bowing (CLB)

o Linear (CL & CLB) sub-classifications (Parker and Johnson 2000): Trailing Stratiform (TS), Leading Stratiform (LS), Parallel Stratiform (PS)

o Squall Line Developmental Types (Bluestein and Jane 1985): Broken Areal (BA), Broken Line (BL), Backbuilding (BB), Embedded Areal (EA)

•Non-linearo Continuous Non-Linear (CNL)

o Discontinuous Areal (DA)

o Isolated Cells (IC)

•Orographically Fixed (OF)

In the 6-hour CRA window, if multiple types were observed, then the type dominating most of the time was used.

If multiple systems were observed in one CRA, then the system with greater temporal, spatial, and rain volume was used.

MCS Morphology ClassificationMCS Morphology Classification

Number of Linear MCSs in Radar Data

111

19

89

9 8

20

4

61

38

28

30

20

40

60

80

100

120

General Squall Development

CL CLB TS LS PS TS_PS LS_PS BA BL BB EA

121

96

37

0

20

40

60

80

100

120

140

CNLDAIC

Number of Nonlinear MCSs in Radar Data

Rain Volume

•The version of the MM5 model (“Pre-MM5”) run in real-time during IHOP by FSL exhibited very large wet biases.

•These results compelled FSL to make major changes to the “Hot Start” diabatic initialization, both during the field phase and for several months thereafter.

• Those changes resulted in a removal of the bias for linear MCSs, but a slight wet bias was maintained with non-linear MCSs

1.21

4.67

2.46

7.49

3.08

0.51

012345678

All Linear Non-Linear

Pre-MM5

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

CL CLB CNL DA IC

MM5-fcst

MM5-obs

Eta-fcst

Eta-obs

Maximum Rainfall

• The post-MM5 overpredicts rainfall maxima for all MCS categories except CNL

• Conversely, the Eta underpredicts rainfall maxima for all categories except IC

Frequency Classification of MCSs

MCS Type Radar (%) Eta (%) MM5 (%) WRF (%)

CNL 31 33 36 34

DA 25 25 10 19

IC 10 4 3 1

Total Nonlinear 66 62 49 54

CL 29 33 42 38

CLB 5 5 9 8

Total Linear 34 38 51 46

Distribution of Forecast Rainfall Distribution of Forecast Rainfall Errors by Error Type for the Five Errors by Error Type for the Five

Basic MCS categoriesBasic MCS categories

Displacement Errors: ETA

Displacement Errors: MM5

Displacement Errors: WRF

Conclusions• Displacement vectors (polar plots):

o None of the models displays a strongly preferred (or systematic) direction and magnitude of displacement vectors, either for any particular MCS or between MCS classes, except for the linear CL and CLB types, which were forecast too slowly (north or northwest) by the WRF and MM5 models

o This may suggest cold pools for squall line systems were forecast to be too weak, which if true is consistent with the idealized study of squall lines by Weisman et al. (1997)

• Decomposition of rainfall errors (histograms):o Overall, for all three models and all MCS types, the largest contributors to

total MSE are conditional bias and pattern errors, with volume error consistently being smallest

o This was a major problem for CLB type, suggesting poorly forecast bowing structures and with the wrong intensity for the convective lines

o Nonlinear systems (CNL and DA) were forecast with the least error, though all three models did display a significant bias for these systems