© Crown copyright Met Office

WAFC turbulence and Cb hazard verificationRecent results and future plansDr Philip G Gill

WAFSOPSG 7/14, 30th April 2013

© Crown copyright Met Office

Contents

This presentation covers the following areas

• Introduction

• Turbulence verification

• Cb verification

• Future plans

• Questions and answers

© Crown copyright Met Office

Introduction

• Turbulence and Cb are a major cause of aviation incidents

• Turbulence and Cb hazard forecasting are part of the World Area Forecast Centre (WAFC) service provided by WAFC London and WAFC Washington

• Forecasts currently produced both by forecasters – Significant Weather Chart (SIGWX) and from gridded model output

• Objective verification was set up in 2009

• Forecasts are global but verification is currently limited

Photos © P Gill

© Crown copyright Met Office

Objective turbulence verification

© Crown copyright Met Office

Verification

• Objective verification of gridded clear air turbulence (CAT) forecasts

• Verification by severity of turbulence (eg. moderate or greater)

• Verify against automated aircraft observations from the Global Aircraft Data Set (GADS)

• Restricted to cruise level (above 28, 000 ft) to limit unwanted effects on observations of manoeuvring at lower levels

© Crown copyright Met Office

Global Aircraft Data Set

• Fleet of Boeing 747-400 aircraft

• Global coverage, but flights mainly over northern hemisphere

• Automated aircraft observations available every 4 seconds

10-19 January 2009

Good coverage of N Atlantic, US and Europe

Poor coverage of E Asia/Pacific region and southern hemisphere – need more observations

© Crown copyright Met Office

Turbulence observations

• Pilot Reports (PIREPS) are issued by pilots when turbulence is encountered – useful additional information but are subjective, aircraft specific, temporal and spatial errors, no routine null reports

• AMDAR are automated reports from aircraft but are not regular and can differ between aircraft

• Derived Equivalent Vertical Gust (DEVG) - Indicator of turbulence derived from automated in-situ flight data: vertical acceleration, aircraft mass, altitude and airspeed

• Eddy Dissipation Rate (EDR) also a measurement of turbulence derived from automated in-situ data – ICAO standard

At present access to EDR data limited to US – DEVG only option for wider coverage

© Crown copyright Met Office

Verification methodology

Aircraft track within time window

Turbulent event

Turbulence forecast field

Ellrod TI1

© Crown copyright Met Office

Forecast assessment

• Turbulent/non turbulent event defined on 10min aircraft track ~120km - approx grid size

• Forecast turbulent event –

CAT potential >= Threshold

• Observed (moderate or greater) turbulent event - DEVG>=4.5m/s

• Construct 2x2 contingency tables for each threshold

• Sum entries in contingency tables over the verification period

Turbulence observed

No turbulence observed

Turbulence forecast

Hit False alarm

No turbulence forecast

Miss Correct rejection

2x2 contingency table

Forecast skill

• Contingency tables can be used for various probability thresholds to produce a reliability table

• From the reliability table various scores can be calculated

• The skill of the forecast at discriminating between events and non-events can be measured using the Receiver Operating Characteristic (ROC) by plotting hit rate against false alarm rate

• The curve from a probabilistic forecast and single point from a deterministic forecast can be compared.

ROC Curve

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hit

rat

e

Low threshold

High threshold

Line of no skill

Perfect forecast

Area under curve measures skill

© Crown copyright Met Office

WAFC CAT verificationRobert Coulson

Latest results – mean CAT

ROC for global WAFC max CAT forecasts against GADS data for moderate or greater turbulence Nov 2011-Oct 2012

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hit

ra

te

WAFC blended max CAT

Acceptable hit rates?

Acceptable false alarm rates?

Area of interest could be measured by partial area under ROC curve

Latest results – max CAT

ROC for global WAFC mean CAT forecastsagainst GADS data for moderate or greater turbulence Nov 2011-Oct 2012

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hi r

ate

WAFC blended mean CAT

Latitudinal variation

50N to 90N

20S to 20N

20N to 50N

50S to 20S

Good performance from above 20N

Summary of CAT verification

• Blending process continues to produce both consistent and skilful turbulence forecasts.

• Some variation in skill by latitude with best performance in the mid latitudes.

• Rolling 12-month results produced monthly since June 2012.

© Crown copyright Met Office

WAFC Cb verificationTeresa Hughes

Cb verification

• Verification against lightning strikes (SFERICS) from Met Office ATDnet

• Gridded SFERICS field produced using observations at +/- 30 minutes from forecast validity time

• T+24 forecast verified from all four model runs

ATDnet Domain

Latest results

ROC for global WAFC Cb forecasts against SFERICS lighning data Nov 2011-Oct 2012

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hi r

ate

WAFC blended Cb

Latitudinal variationROC for global WAFC Cb forecasts against SFERICS lightning data

Nov 2011-Oct 2012 (50N - 80N)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hi r

ate

WAFC Blended Cb

ROC for global WAFC Cb forecasts against SFERICS lightning data Nov 2011-Oct 2012 (20N - 50N)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hi r

ate

WAFC Blended Cb

ROC for global WAFC Cb forecasts against SFERICS lightning data Nov 2011-Oct 2012 (20S - 20N)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hi r

ate

WAFC Blended Cb

ROC for global WAFC Cb forecasts against SFERICS lightning data Nov 2011-Oct 2012 (40S - 20S)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

False alarm rate

Hi r

ate

WAFC Blended Cb

50N to 80N 20N to 50N

20S to 20N 40S to 20S

Summary of Cb verification

• Blending process continues to produce both consistent and skilful Cb forecasts.

• Some variation in skill by latitude

• Rolling 12-month results produced quarterly since June 2012.

© Crown copyright Met Office

Future verification plans

Improving turbulence verification

• Switch from DEVG to EDR

• ICAO standard

• Forecasting and observing EDR

• Source more EDR observations from other airlines to increase the area of coverage

• Increase coverage to fill gaps in current GADS database

Improving Cb verification

• Extend verification against lightning reports by outside of the ATDnet domain

• Source additional lightning reports outside [80N, 40S, 100W, 80E]

• Use geostationary satellite data in addition to lightning data

• MSG [60N, 60S]

• Overshooting convection product (work underway)

• New satellites over America (GOES-R) and East Asia (Himawari-8)

• Verify Cb cloud top height and Cb extent

MSG - the EuropeanGeostationary Imager

•Scans Africa, Europe, Middle East and Atlantic every 15 minutes

•3/1 km spatial resolution at 0° N, 0° E

•12 channels (different wavelengths)

•Meteosat-10 operational, Meteosat-8,-9 backup

Severe convection product

•Data from Meteosat Second Generation

•Calculated from the visible and infrared channels.

•Generated every 15 mins.

•Current product over Europe identifies area where severe convection is occurring (day-time only)

•New overshooting convection product will identify intense convective areas with overshooting cloud tops.

•Could be merged with lightning data from ATDnet

•Could be merged later with Meteosat third generation lightning data

Cloud top height product

• Data from Meteosat Second Generation

• Calculated from the infrared channels at wavelengths of 10.8, 12.0 and 13.4 microns.

• Generated every 15 mins

• Covers Europe, Atlantic, Africa

• Extend to global coverage following GOES-R and Himawari-8 satellite launches (~ 2015)

• Mask with Cb product

Verification – all forecasts

• Verify probabilistic forecasts using suitable metrics

• Forecast skill – ROC (partial area under curve?)

• Reliability diagram

• Value – relative economic value measure

• Publish verification results – plots and contingency tables

• Breakdown into WMO regions

• Forecast ranges T+12, T+24, T+36

• Rolling 12-month scores

• Geostationary satellite data limited to 60N, 60S. Polar orbiter data could supplement this if necessary.

Timescales

• 2013/14

• Preliminary work on satellite data specifications

• Routine CAT and Cb verification published by WMO region.

• 2014/15

• Include satellite data in Cb verification

• Develop Cb cloud top height verification

• Move to using EDR in turbulence verification

• 2015/16

• Expand coverage of Cb verification using new satellite data over East Asia (Himawari-8)

• 2016/17

• Expand coverage of Cb verification using new satellite data over America (GOES-R)

© Crown copyright Met Office

Questions & answers