evaluating climate models - university corporation for ...ericg/talks/gillelandems2016.pdf · 16th...

Evaluating Climate Models Eric Gilleland

Research Applications Laboratory, 16th EMS Annual Meeting & 11th European Conference on

Applied Climatology, 16 September 2016 . Co-authors: Caspar Ammann, Barb Brown, Melissa Bukovsky, Seth McGinnis,

Linda Mearns, and Christopher L. Williams Support for this work provided by the NSF via the Weather and Climate Impacts Assessment Science Program (http://www.assessment.ucar.edu) and Earth System Modeling (EaSM) Grant number AGS-1243030.

UCAR Confidential and Proprietary. © 2016, University Corporation for Atmospheric Research. All rights reserved.

●

●

●

●

●

●

Estes Park

Boulder

Fort Collins

DenverDIAGolden

NCAR/NCEP reanalysisCCSM3 Global Climate Model

Severe Storm Environments


0.00

000

0.00

010

0.00

020

Non−severeSevereSignificant Non−tornadicSignificant Tornadic

0.00

00.

001

0.00

20.

003

0e+00 2e+05 4e+05

0.0

0.2

0.4

0.6

0.8

1.0

0 2000 6000 10000

0.0

0.2

0.4

0.6

0.8

1.0

CAPE × Shear (J kg-1 × m s-1)

Wmax × Shear (WmSh, m2 s-2)

NCEP reanalysis

Community Climate System Model

3rd Generation Coupled Global Climate Model

Hadley Centre Coupled Model, v. 3

abbreviation NCEP CCSM3 CGCM3 HadCM3

Canadian Regional Climate Model (CRCM)

X

X

X

Hadley Regional Model 3 (HRM3)

X

Pennsylvania State University/NCAR mesoscale model (MM5I)

X

X

Weather Research and Forecasting model (WRFG)

X

X

X


http://www.narccap.ucar.edu/ http://www.emc.ncep.noaa.gov/mmb/rreanl/

Lingo


q75: Univariate time series giving the upper quartile of a random variable (CAPE or WmSh) over space at each time point.

WmSh: As before, but set to zero if CAPE < 100 J kg-1 or 5 ≤ Shear ≤ 50 ms-1

High “field energy”: when q75 > its 90th percentile over time.

κ: Frequency of CAPE ≥ 1000 J kg-1 conditioned on the presence of high field energy.

ω: Frequency of WmSh ≥ 225 m2s-2 conditioned on the presence of high field energy.


κ

−120 −100 −80 −70

3035

4045

NARR

−120 −100 −80 −70

3035

4045

CRCM−CCSM

−120 −100 −80 −70

3035

4045

CRCM−CGCM3

−120 −100 −80 −70

3035

4045

HRM3−HadCM3

−120 −100 −80 −70

3035

4045

MM5I−CCSM

−120 −100 −80 −70

3035

4045

MM5I−HadCM3

−120 −100 −80 −70

3035

4045

WRFG−CCSM

−120 −100 −80 −70

3035

4045

WRFG−CGCM3

−120 −100 −80 −70

3035

4045

CRCM−NCEP

−120 −100 −80 −70

3035

4045

WRFG−NCEP

020406080100


ω

−120 −100 −80 −70

3035

4045

NARR

−120 −100 −80 −70

3035

4045

CRCM−CCSM

−120 −100 −80 −70

3035

4045

CRCM−CGCM3

−120 −100 −80 −70

3035

4045

HRM3−HadCM3

−120 −100 −80 −70

3035

4045

MM5I−CCSM

−120 −100 −80 −70

3035

4045

MM5I−HadCM3

−120 −100 −80 −70

3035

4045

WRFG−CCSM

−120 −100 −80 −70

3035

4045

WRFG−CGCM3

−120 −100 −80 −70

3035

4045

CRCM−NCEP

−120 −100 −80 −70

3035

4045

WRFG−NCEP

020406080100


Image Warping

Objective Function = SSE(Warped Forecast and Observation) + Penalty term for too much bending/movement


% error reduction ≈ 40% minimum bending energy = 2.0042

RMSE0 = 0.2665 RMSE1 = 0.1605

Image Warping

−120 −100 −90 −80 −70

3035

4045

0−energy field

0.0 0.2 0.4 0.6 0.8 1.0

−120 −100 −90 −80 −70

3035

4045

1−energy field

0.0 0.2 0.4 0.6 0.8 1.0

−120 −100 −90 −80 −70

3035

4045

Error Field

−0.8 −0.4 0.0 0.2 0.4

−120 −100 −90 −80 −70

3035

4045

Distance Travelled

5 10 15 20

−120 −100 −90 −80 −70

3035

4045

Deformed 1−energy field

0.0 0.2 0.4 0.6 0.8 1.0

−120 −100 −90 −80 −70

3035

4045

Error Field(after warping)

−0.8 −0.4 0.0 0.2 0.4

MM5I-CCSM3 κ


Image Warping

RMSE0 RMSE1 RMSE Reduction

Minimum Bending Energy

CRCM-CCSM3 0.214 0.139 35% 0.96 CRCM-CGCM3 0.147 0.103 30% 1.07 HRM3-HadCM3 0.157 0.110 30% 0.25 MM5I-CCSM3 0.267 0.161 40% 2.00 MM5I-HadCM3 0.148 0.084 43% 0.69 WRFG-CCSM3 0.249 0.096 61% 3.27 WRFG-CGCM3 0.241 0.092 62% 3.32 CRCM-NCEP 0.214 0.173 19% 0.25 WRFG-NCEP 0.171 0.092 46% 0.43

ω results


SPCT1 + Image Warping2 Model 1 Model 2 Mean Loss

Differential p-value

CRCM-CCSM3 MM5I-CCSM3 -1.31 0.19 HRM3—HadCM3 MM5I-CCSM3 -1.64 0.10 MM5I-CCSM3 CRCM-NCEP 1.63 0.10 MM5I-CCSM3 WRFG-NCEP 1.38 0.17 WRFG-CCSM3 CRCM-NCEP 1.76 0.08 WRFG-CCSM3 WRFG-NCEP 1.36 0.17 WRFG-CGCM3 CRCM-NCEP 1.33 0.18 WRFG-CGCM3 WRFG-NCEP 1.36 0.17

Significant results for ω at the 20% level 1See Hering and Genton (2011, Technometrics, 53 (4), 414 - 425, DOI 10.1198/TECH.2011.10136) 2See G. (2013, Mon. Wea. Rev., 141, (1), 340 – 355, DOI: 10.1175/MWR-D-12-00155.1)

10% level

ω results

Conclusions •  Climate models should reproduce observed distributional properties for

the current-period climate, making spatial forecast verification methods particularly useful, and easy to implement in this context.

•  Approach, here, focuses on spatial patterns of frequencies of severe-storm environments for the top ten-percent of high-energy days. §  Provides a means of comparing climate models to observations without

requiring day-to-day match-ups. •  Models more-or-less agree with NARR about spatial location and

overall pattern of high severe storm frequencies (κ and ω). •  Models tend to under-project the spatial extent of high-frequency κ and ω areas compared to NARR.

•  WRFG configurations not coupled with NCEP (i.e., “observations”) have the least agreement with NARR.

•  Full analysis including many other spatial methods in G. et al. (submitted to ASCMO, available at http://www.ral.ucar.edu/staff/ericg/GillelandEtAl2016.pdf)


evaluating climate models - university corporation for ...ericg/talks/gillelandems2016.pdf · 16th...

Documents