The purpose of forecast verification is to check for consistency, quality and value of a forecast. A forecast has high quality, if it predicts the observed conditions well according to some objective or subjective criteria. It has value if it helps the user to make a better decision. In aviation, value of a forecast is of prime importance as it has direct links with the safety, economy and regularity of operation.

Aviation Forecast Verification by IMD: The old verification procedures were based on the “Instructions” issued in 1956. Except for the landing (TREND) forecasts and aerodrome warnings, all other forecast were verified on a sampling basis. In the sampling method only the occurrence or non-occurrence of weather was analysed. Effectiveness of the forecast and warnings not studied so far.

New criteria came in use with effect from 01st April 2011. It is based on the ICAO specified “Operationally Desirable Accuracy of Forecasts”. Annex-3, Appendix B. Presently all elements in TAF, TREND forecast, Forecast for take-off & Area/Local forecast are verified on a near real time basis. All aerodrome warnings and SIGMETs are also verified. Verification of Route/ flight forecast is continued on sampling basis. According to the new criteria, forecasts are verified as “correct” and “incorrect” when the forecast value falls within prescribed limits. Monthly statement of percentage correct occurrence is prepared for individual forecasting offices (18 in number) for individual forecast type and for all elements.

Forecasts Validity period

Areal validity

Update frequency

TAF 9 hours and 30 hours

Over the aerodrome

3 and 6 hours

TREND 2 hours Over the aerodrome

Half hour or when SPECIal conditions are observed

Area/ Local forecast

8 hours Aerodrome and 100 nautical miles around

3 times a day

Route forecast Valid from 1 hour before the ETD and upto 2 hours after ETA

Along the route Need based

Take off forecast xxxxxxx Over the aerodrome

xxxxxxxx

Warnings Validity period Areal validity Update frequency

Aerodrome warning 4 hours or less Over the aerodrome

Need based

SIGMET Upto 4 hours (upto 6 hours for TC SIGMET)

FIR or part of FIR Need based

World Area Forecast System : - wind and temperature charts - Significant weather charts

(provided by designated centres, not verified by IMD)

General considerations of verification: • Verification is done manually. • The forecasts and warnings are human generated (except for

WAFC products). • Where ever possible the forecaster verifies the forecast of his

reliever, using the actual observations. • The aim is to analyse the forecast as and when it is being

experienced. • Aims to help the forecaster in understanding the wrong

diagnosis. Tools for verification • METAR, SPECI,Synop, Radio Sonde Ascent data, RADAR

data and Satellite observation are used for comparison. • No pilot reports are received for verification. • Continuously measured sensor data are not used as all

airports do not have automatic instruments.

Verification of TAFOR • TAF is considered a forecast for time periods rather than for

points of time. • Changes within the TAF period are indicated mostly using

BECMG, and TEMPO. • ‘PROB’ is not used • TAFS are produced for all domestic and international

airports, and for other destinations as per the requirement. • Used for flight planning by operators and is considered one

of the standardized forecast worldwide

• In case of TAF, only the current weather observations from the aerodrome during the validity period of the TAF are used for verification purpose.

• For verification of present weather , Recent weather

and weather in the vicinity are also considered. • In TAF, the change group is verified as a separated

forecast. • Amended forecast is also verified seperately. • Syntax errors are also checked, but not as a part of

the verification scheme.

Element to be forecast

Operationally desirable accuracy of forecasts

Minimum percentage of cases within range

Wind Direction ± 20° 80%of cases Wind speed ± 10 km/h (5kt) 80% of cases Visibility ± 200 m up to 800m

± 30% between 800 m and 10 km 80% of cases

Precipitation Occurrence or non-occurrence 80%of cases Cloud amount One category below 450m (1500 ft)

Occurrence or non-occurrence of BKN or OVC between 450 m (1500 ft) and 3000 m (10000 ft)

70%of cases

Cloud height ± 30 m (100 ft) up to 300m (1000 ft) ± 30% between 300m (1000 ft) and 3000m (10000 ft)

70% of cases

TAF

• The verification of TAF is done manually, the basic forecast is considered to be one instance.

• Change groups: Assumed that the forecast value takes effect at the time specified after the change indicator ‘BECMG’ or ‘TEMPO’

• Hence, the values which are applicable outside the change group are not considered to be applicable within the change group.

• If forecast value falls within the specified range, it is considered as a correct occurrence.

• The ratio of total number of correct occurrences against the total number of forecast (basic+ change group or amendments) is considered for calculating percentage occurrence.

• The major draw back of these considerations is that the score of the forecaster can be very high, but the value of the forecast can be very low as far as a planner is considered.

• At a particular window of time, in which the operator is

interested, the forecast values may be very much different from the actual, but the forecast may be correct from the point of view of the forecaster.

• This may be the reasons why the verification statements show the accuracy for all elements of TAF from all the forecasting offices are very high (close to 90%) in most cases.

SIGMET

Element to be forecast

Operationally desirable accuracy of forecasts

Minimum percentage of cases within range

Significant en-route weather phenomena

Occurrence or non-occurrence 80% of cases Location: ± 100 km (60 NM) 70% of cases Vertical extent: ±300 m (1000 ft) 70% of cases Expected changes in intensity: Occurrence or non-occurrence of the same category (INTSF or WKN or NC)

70% of cases

Considerations for SIGMET verification: • In the case of SIGMET all the available observations during

the validity period of the SIGMET are consulted for verification, so that, in effect a composite of all the observations over the area under consideration are obtained for comparison.

• Verification is done only for SIGMET for forecast

phenomenon and not for observed.

• Most frequently issued SIGMETs are for TC and TS.

• Lack of observations or reporting limits the issue of SIGMET for other phenomena.

• TC verification is done based on weather bulletins, and satellite bulletins, satellite observations and RADAR observations.

• Widespread sandstorm/ duststorm Verification done based on actual observations (METAR and SYNOP) and satellite products.

• Thunderstorms and Cumulonimbus clouds are verified based on Cloud top temperature and actual observations (conventional and radar) along with satellite imageries and corresponding satellite bulletins.

• Turbulence (severe turbulence): Presently no observations or reports for verifying turbulence outside clouds.

• Same is the case with Mountain wave (severe mountain wave).

Verification of Aerodrome warning: • As the warning is meant for the aerodrome under

consideration, only the observations from the airfield during the warning period are used for verification.

• Verification of warning is carried out only if the warning is based on a forecast condition.

• The warning elements usually observed (in our conditions) are:

• Wind: Gusts and Squall. • Tropical cyclone • Dust storm & Sandstorm • Thunderstorm & Hail

General: If a weather phenomena is expected to affect the aerodrome and if it occurs, then warning is to be considered as correct. For each warning element, a minimum of 80% of cases percentage within range is considered desirable. a)Weather phenomena: Occurrence or non-occurrence are the criterion. In case of expected occurrence of thunderstorms, if lightning is observed or thunder is heard, the warning is taken as correct. If, CB is observed at the station, then also the warning may be taken as correct.

b) Surface wind: For “Strong surface wind and gusts”, and for “Squall” Speed: ± 5 knots; ie., when the deviation of the forecast speed from the observed speed is 5 knots, the warning may be taken as correct. Otherwise it may be taken as wrong. Wind Direction: ± 30°; the warning may be taken as correct when the expected direction differs from the actual direction by 30° or less. c) Change in wind direction: In cases of aerodrome warning for rapid wind direction change by 45 degrees or more of wind with a speed of 20 KT or more, the occurrence of the event may be taken as “correct” and non-occurrence as “wrong”.

TREND forecast Element to

be forecast Operationally desirable accuracy of forecasts

Minimum percentage of cases within range

Wind Direction

± 20° 90%of cases

Wind speed ±10 km/h (5kt) 90% of cases Visibility ± 200 m up to 800m

± 30% between 800 m and 10 km 90% of cases

Precipitation Occurrence or non-occurrence 90%of cases Cloud amount

±One category below 450m (1500 ft) Occurrence or non-occurrence of BKN or OVC between 450 m(1500 ft) and 3000 m (10000 ft)

90%of cases

Cloud height ± 30 m (100 ft) up to 300m (1000 ft) ± 30% between 300m (1000 ft) and 3000m (10000 ft)

90% of cases

Verification method Trend forecast is similar to TAF verification. The weather phenomena that are included in the TREND forecast are: • Moderate or heavy precipitation (including showers thereof) • thunderstorm (with or without precipitation) • Dust storm • Sandstorm • low drifting dust, sand • blowing dust, sand • Squall • other weather phenomena which are expected to cause a

significant change in visibility.

Area/ Local Forecast forecast

Element to be forecast

Operationally desirable accuracy of forecasts

Minimum percentage of cases within range

Upper-air temperature

± 2°C (Mean for 100 NM) 90%of cases

Upper wind ± 20 km/h (10 kt) 90%of cases Significant weather phenomena and cloud

Occurrence or non-occurrence within 100NM area Vertical extent: ± 300 m (1000 ft)

80% of cases 70%of cases

• Area/ Local Forecast is issued for an aerodrome and 100 nautical miles around.

• Hence there can be areas which are not covered by an area forecast.

• Issued for the use of low level flights, mainly training sorties.

• Does not follow the format of ICAO • The observations available for evaluation is limited owing

to the low resolution of observatories and limited number of vertical profiles.

• The SYNOP/ METAR/ RADAR/ Satellite observations that fall within the 100 NM around the aerodrome is considered for verification.

Route forecast • The route forecast are issued in the MET- T3 / MET- T4

form.

• It is for the use of low level flights/ helicopters, which are not covered by the use of WAFC products.

• In case of ROFOR, synoptic observations reported by the stations along the route also is considered apart from the METARs/ RADAR observations/ satellite observations.

• The observation that fall within 50 kilometers on either side of the route is also considered for verifying a route forecast & observations during the entire validity period are taken into account.

Element to be forecast Operationally desirable accuracy of forecasts

Minimum percentage of cases within range

Upper-air temperature ± 2°C (Mean for 900 km (500 NM))

90% of cases

Upper wind Direction: ± 30° 90% of cases Speed: ± 10 kts 90% of cases

Significant en-route weather phenomena and cloud

Occurrence or non-occurrence

80% of cases

Location: ± 100 km (60 NM) 70% of cases Vertical extent: ±300 m (1000 ft)

70% of cases

Flight level of tropopause: ±300 m (1000 ft)

80% of cases

Max wind level: ±300 m (1000 ft)

70% of cases

ROUTE/ FLIGHT FORECASTS

Conclusion • Verification results should help in developments by evaluating new

methods and products. • Tracking the trend of a results not done so far.

• And hence verification results are not considered for the

developmental activities at present.

• Verification is not done specific for different aviation user requirements.

• The same forecast may not be equally effective for all users.

• Hence the effectiveness or impact study of the forecast is to be carried out, for evaluating the economy, safety and regularity of the operations.

• The forecast which can be correct for the forecaster may not be correct for a particular user, who plans his flight for a particular window.

• In the present method of verification of TAF, only the comparison of individual elements is being done.

• The weather at a particular time period is not considered in totality and hence, the weather events within the forecast period could be properly captured in the forecast cannot analysed.

• This concept is not helpful for the forecaster in understanding the evolution of the weather.

• The entire forecast period is considered a single stretch outside the change groups. This might be the reason for the persistently high accuracy values for all the forecasting offices. Hence, splitting the forecast into smaller windows of one hour duration or so is to be tried.

• This will require automation of the process.

• The amended forecast is verified as a separate forecast. It should replace the original forecast, as the conditions could not be anticipated in the original forecast.

• The transition to change groups is not smooth. • Probably the use of continuous observations and use of smaller

time windows can make the verification more realistic. • Different treatment for TEMPO is needed as the fluctuation is not

steady within the TEMPO period.

• Automatic verification will make the verification process more objective.

• Even though the details a human can observe may not be possible in the automatic method, the present way of doing it manually is laborious.

• Hence automation is recommended.

92

52

94 97

83 84 85 82

67

75

91

72

64 67

34

89 86

20

30

40

50

60

70

80

90

100

VABB VAAH VOMM VOBL VOTV VOHS VEGT VEMN VEAT VIDP VIJP VILK VECC VEBS VEPT VANP VABP

Wind direction Wind speed Visibility Precipitation Cloud amount Cloud height

VERIFICATION OF TAF FOR THE YEAR 2012

THANK YOU