April 24, 2023

1

A Real-Time Automated Method to A Real-Time Automated Method to Determine Forecast Confidence Determine Forecast Confidence

Associated with Tornado WarningsAssociated with Tornado WarningsUsing Spring 2008 NWS Tornado WarningsUsing Spring 2008 NWS Tornado Warnings

John CintineoJohn CintineoCornell UniversityCornell University

Travis SmithTravis SmithValliappa LakshmananValliappa Lakshmanan

Kiel OrtegaKiel OrtegaNOAA - NSSLNOAA - NSSL

April 24, 2023

2

BackgroundBackground Warning Decision Support System – Integrated

Information (WDSS-II) Uses merged, multi-sensor CONUS radar network combines model, lightning, and GOES satellite data Short-term severe weather forecasting products

Objective: To examine how WDSS-II products can be used as

predictors for issuing NWS tornado warnings. Assign objective probabilities to warnings based on

varying the attribute threshold.

April 24, 2023

3

Radar-derived productsRadar-derived products Storm Environment DataStorm Environment Data Maximum Expected Size of Hail

(MESH) Probability of Severe Hail (POSH) Severe Hail Index (SHI) Vertically Integrated Liquid (VIL) Area of VIL +30 Echo Tops of 50, 30, & 18dBZ 3-6 km & 0-2 km Azimuthal Shear Lowest level max dBZ Reflectivity at 0C, -10C, & -20C Overall max reflectivity Height of 50dBZ above 253K

isotherm

Environmental Shear Storm Relative Flow 9-11km AGL Storm Relative Helicity 0-3km CAPE, CIN LCL min heightSATELLITE:SATELLITE: IR band-4 min temp. (cloud tops)

Total of 23 products

April 24, 2023

4

Methodology: Investigated archived NWS spring 2008 CONUS tornado

warnings with WDSS-II radar-derived products Each storm attribute maximum (or minimum) values computed

every 1 minute of the warning Compared attribute values from the issuance of the warning

(initial values) and the expiration of the warning (lifetime max/min).

Composite time series of each attribute Warnings broken down by verified vs. unverified Verification data obtained from the Storm Prediction Center’s

storm data (preliminary). storm environment data provided by 20-km RUC model.

April 24, 2023

5

1,617 Tornado WarningsFrequency of Hits = 0.256 (414 verified warnings)

False Alarm Ratio = 0.744 (1,203 unverified warnings)Average Warning Duration: 38.6 mins

Dataset:Dataset: 2 May – 1 July for 0-2 km Azimuthal shear, VIL, Area of VIL

+30, and reflectivity products 15 May to 1 July for 3-6 km Azimuthal shear 20 random days for Storm Environment attributes NB: for 1 May – 10 May 0-2 km Azimuthal shear is replaced by

0-3km Azimuthal shear

April 24, 2023

6

Initial 0-2 km Azimuthal ShearInitial 0-2 km Azimuthal Shear

Mean: 0.0053 s^-1 Mean: 0.0078 s^-1SD: 0.0044 s^-1 SD: 0.0053 s^-1

UNVERIFIEDUNVERIFIED VERIFIED VERIFIED

April 24, 2023

7

Lifetime Max 0-2km Azimuthal ShearLifetime Max 0-2km Azimuthal Shear

Mean: 0.0078 s^-1 Mean: 0.0109 s^-1SD: 0.0051 s^-1 SD: 0.0055 s^-1

UNVERIFIEDUNVERIFIED VERIFIEDVERIFIED

April 24, 2023

8

5 10 15 20 25 30 35 40 45 50 55 60 650.0000

0.0010

0.0020

0.0030

0.0040

0.0050

0.0060

0.0070

0.0080

0.00900-2 km Azimuthal Shear

VERIFIEDUNVERIFIED

Time (mins)

Az.

She

ar (s

^-1)

April 24, 2023

9

Probability that a warning verified, given an Probability that a warning verified, given an initial initial 0-2 km Az. Shear0-2 km Az. ShearP( Ver. | shear in bin ) for 0-2km Az. Shear

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

< 2 2 – 4 4 – 6 6 – 8 8 – 10 10 – 12 12 – 15 > 15

Shear bin ( x 10 -̂3 s -̂1)

Prob

abilit

yity

P(V | in bin)

April 24, 2023

10

Initial Vertically Integrated Liquid (VIL)Initial Vertically Integrated Liquid (VIL) UNVERIFIEDUNVERIFIED VERIFIED VERIFIED

Mean: 27.76 kg/m^2 Mean: 34.44 kg/m^2SD: 20.46 kg/m^2 SD: 18.66 kg/m^2

April 24, 2023

11

Lifetime Maximum VILLifetime Maximum VIL

Mean: 37.00 kg/m^2 Mean: 46.35 kg/m^2SD: 20.27 kg/m^2 SD: 18.05 kg/m^2

UNVERIFIEDUNVERIFIED VERIFIED VERIFIED

April 24, 2023

12

5 10 15 20 25 30 35 40 45 50 55 60 6520

22

24

26

28

30

32

34

36

38Vertically Integrated Liquid (VIL)

VERIFIEDUNVERIFIED

Time (mins)

kg/m

^2

April 24, 2023

13

Probability that a warning verified, given an Probability that a warning verified, given an initial initial Vertically Integrated LiquidVertically Integrated Liquid

< 10 10 – 20 20 – 30 30 – 40 40 – 50 50 – 60 > 600

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4( Ver. | VIL in bin )

P(V | in bin)

VIL bin (kg/m^2)

Pro

babi

lity

April 24, 2023

14

Initial 0-2 km Az. Shear (s^-1)Initial 0-2 km Az. Shear (s^-1)

Ver: 41Unv: 238

PROB: 0.147

Ver: 10Unv: 61

PROB: 0.141

Ver: 5Unv: 34

PROB: 0.128

Ver: 5Unv: 19

PROB: 0.208Ver: 29Unv: 70

PROB: 0.293

Ver: 32Unv: 49

PROB: 0.395

Ver: 18Unv: 42

PROB: 0.300

Ver: 24Unv: 26

PROB: 0.480Ver: 15Unv: 41

PROB: 0.268

Ver: 23Unv: 69

PROB: 0.250

Ver: 26Unv: 35

PROB: 0.426

Ver: 25Unv: 27

PROB: 0.481Ver: 0Unv: 14

PROB: 0.000

Ver: 9Unv: 36

PROB: 0.200

Ver: 10Unv: 15

PROB: 0.400

Ver: 8Unv: 9

PROB: 0.471

x < 0.004 0.004 <= x < 0.008 0.008 <= x < 0.012 x >= 0.012

In

itial

Ver

tical

ly In

tegr

ated

Liq

uid

(kg/

m^2

)In

itial

Ver

tical

ly In

tegr

ated

Liq

uid

(kg/

m^2

)

y

>= 6

0

60 >

y >

= 40

40

> y

>= 2

0 y

< 2

0

CONDITIONAL PROBABILITY CONTINGENCY TABLECONDITIONAL PROBABILITY CONTINGENCY TABLE

April 24, 2023

15

SummarySummary Provide warning guidance for the NWS Once a NWS tornado warning is issued,

WDSS-II can automatically assign a probability that it will verify, in real-time

More years of warning data will lead to a better climatology of warning probabilities

With more warning data, create a contingency table based on 3 or 4 of the best predictors

Forecasters can use such probability data to reduce their FAR

April 24, 2023

16

Future avenues of researchFuture avenues of research Extend the data set to include past springs Examine environment just outside the

warning polygons (to capture the entire storm)

Compare spring and fall tornado warnings Compare attributes in tornado and severe

T-storm warnings Compare warning data based on region Investigate warnings issued in watches,

and those outside of watches

April 24, 2023

17

SummarySummary Provide warning guidance for the NWS Once a NWS tornado warning is issued,

WDSS-II can automatically assign a probability that it will verify, in real-time

More years of warning data will lead to a better climatology of warning probabilities

With more warning data, create a contingency table based on 3 or 4 of the best predictors

Forecasters can use such probability data to reduce their FAR

April 24, 2023

18

AcknowledgementsAcknowledgements Travis Smith Lak Kiel Ortega Owen Shieh This research was supported by an

appointment to the National Oceanic and National Oceanic and Atmospheric AdministrationAtmospheric Administration Research Participation Program through a grant award to Oak Ridge Institute for Science and Education.

April 24, 2023

19

ReferencesReferences Erickson, S. A., Brooks, H., 2006: Lead time and time under tornado warnings: 1986-2004. 23rd

Conference on Severe Local Storms Guillot, E., T. M. Smith, Lakshmanan, V., Elmore, K. L., Burgess, D. W., Stumpf, G. J., 2007:

Tornado and Severe Thunderstorm Warning Forecast Skill and its Relationship to Storm Type. Lakshmanan, V., T. M. Smith, K. Cooper, J. J. Levit, G. J. Stumpf, and D. R. Bright, 2006: High-

resolution radar data and products over the Continental United States. 22nd Conference on Interactive Information Processing Systems, Atlanta, Amer. Meteor. Soc.

Lakshmanan, V., T. Smith, K. Hondl, G. J. Stumpf, and A. Witt, 2006: A real-time, three dimensional, rapidly updating, heterogeneous radar merger technique for reflectivity, velocity and derived products. Weather and Forecasting 21, 802-823.

Lakshmanan, V., T. Smith, G. J. Stumpf, and K. Hondl, 2007: The warning decision support system - integrated information (WDSS-II). Weather and Forecasting 22, 592-608.

Ortega, K. L, and T. M. Smith, 2006: Verification of multi-sensor, multi-radar hail diagnosis techniques. 1st Severe Local Storms Special Symposium, Atlanta, GA, Amer. Meteo. Soc.

Ortega, K. L., T. M. Smith, G. J. Stumpf, J. Hocker, and L. López, 2005: A comparison of multi-sensor hail diagnosis techniques. 21st Conference on Interactive Information Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Amer. Meteo. Soc., P1.11 - CD preprints.

Witt, A., Eilts, M., Stumpf, G. J., Johnson, J. T., Mitchell, D. E., Thomas, K. W., 1998: An Enhanced Hail Detection Algorithm for the WSR-88D.

April 24, 2023

20

BIN: 5 10 15 20 25 30 35 40 45 50 55 60 65Verified 1174 860 1051 711 1027 769 792 473 469 264 277 255 117Unverified 3174 2244 2675 1819 2649 1971 2011 1161 1247 802 890 725 326

BIN: 5 10 15 20 25 30 35 40 45 50 55 60 65Verified 1176 1120 1047 936 1024 965 797 617 472 367 293 343 142Unverified 3089 2896 2733 2393 2613 2492 1978 1517 1169 977 821 852 332

April 24, 2023

21

Initial 3-6 km Azimuthal Shear

Mean: 0.0054 s^-1 Mean: 0.0076 s^-1SD: 0.0041 s^-1 SD: 0.0046 s^-1

UNVERIFIEDUNVERIFIED VERIFIED VERIFIED

April 24, 2023

22

Lifetime Max 3-6 km Azimuthal Shear

Mean: 0.0084 s^-1 Mean: 0.0112 s^-1SD: 0.0049 s^-1 SD: 0.0052 s^-1

UNVERIFIEDUNVERIFIED VERIFIED VERIFIED

April 24, 2023

23

5 10 15 20 25 30 35 40 45 50 55 60 650.0000

0.0010

0.0020

0.0030

0.0040

0.0050

0.0060

0.0070

0.0080

0.00903-6 km Azimuthal Shear

VERIFIEDUNVERIFIED

Time (mins)

Az.

She

ar (s

^-1)

BIN: 5 10 15 20 25 30 35 40 45 50 55 60 65Verified 951 706 860 595 851 636 642 398 402 240 252 228 108Unverified 2415 1739 2061 1429 2053 1535 1569 938 987 641 722 575 254

April 24, 2023

24

Probability that a warning verified, given an Probability that a warning verified, given an initial initial 3-6 km Az. Shear3-6 km Az. Shear

< 22 – 3

3 – 44 – 5

5 – 66 – 7

7 – 88 – 9

9 – 1010 – 12

12 – 15> 15

0

0.1

0.2

0.3

0.4

0.5

0.6P( Ver. | shear in bin ) for 3-6 km Az. Shear

P(V | in bin)

Shear bin ( x 10^-3 s^-1)

Pro

babi

lity

April 24, 2023

25

Initial Max LL Reflectivity

Mean: 49.88 dBZ Mean: 55.09 dBZSD: 14.63 dBZ SD: 11.02 dBZ

April 24, 2023

26

Lifetime Max LL Reflectivity

Mean: 56.86 dBZ Mean: 60.51 dBZSD: 10.39 dBZ SD: 7.11 dBZ

April 24, 2023

27

5 10 15 20 25 30 35 40 45 50 55 60 6544

46

48

50

52

54

56

58

60

Maximum Low Level Reflectivity

max_LL_dBZmax_LL_dBZ-UNV

Time (mins)

dBZ

BIN: 5 10 15 20 25 30 35 40 45 50 55 60 65Verified 1187 1125 1051 942 1038 983 810 636 489 374 296 351 144Unverified 3270 3073 2862 2516 2765 2652 2098 1627 1264 1052 882 940 364

April 24, 2023

28

Initial dBZ @ -20C

Mean: 46.58 dBZ Mean: 53.08 dBZSD: 14.21 dBZ SD: 10.78 dBZ

April 24, 2023

29

Lifetime Max dBZ @ 20C

Mean: 52.75 dBZ Mean: 57.86 dBZSD: 11.97 dBZ SD: 8.43 dBZ

April 24, 2023

30

BIN: 5 10 15 20 25 30 35 40 45 50 55 60 65Verified 1192 1127 1055 939 1034 982 807 630 487 374 294 348 142Unverified 3342 3127 2909 2549 2796 2688 2118 1629 1273 1051 878 944 368

5 10 15 20 25 30 35 40 45 50 55 60 6540

42

44

46

48

50

52

54

56

58

60Maximum Reflectivity @ -20C

max_dBZ @ -20Cmax_dBZ @ -20C-UNV

Time (mins)

dBZ

April 24, 2023

31

Probability a warning verified, given a certain Az. shearProbability a warning verified, given a certain Az. shear

0-2km Az. Shear0-2km Az. Shear 3-6km Az. Shear3-6km Az. Shear

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

P( V | shear in bin) for 3-6km Az. Shear

shear bin (s^-1)Pr

obab

ility

0-0.0

01

0.001

-0.00

2

0.002

-0.00

3

0.003

-0.00

4

0.004

-0.00

5

0.005

-0.00

6

0.006

-0.00

7

0.007

-0.00

8

0.008

-0.00

9

0.009

-0.01

0.01-

0.011

0.011

-0.01

2

0.012

-0.01

3

0.013

-0.01

4

0.014

-0.01

5

0.015

-0.01

6

0.016

-0.01

7

0.017

-0.01

8

0.018

-0.01

90

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

P( V | shear in bin ) for 0-2km Az. Shear

0-2km Az. Shear bins

Pro

babi

lity