A comparison of automatic model selection procedures for seasonal adjustment

Cathy Jones

Motivation

• Automatic forecasting procedures are required for seasonal adjustment of official statistics time series

• Current seasonal adjustment software offers two automatic procedures

• Aim of this work is to compare the performance of these procedures in the context of seasonal adjustment

Overview

• Time series background• What is seasonal adjustment?• X-13ARIMA-SEATS• ARIMA models• Forecasting• Methods for automatic model selection• Results• Future work

Time Series Background

Data source: ONS

Seasonal Adjustment

• What is seasonal adjustment?“process of removing from a time series variations

associated with the time of year and/or the arrangement of the calendar”

• Why seasonally adjust?• primary interest of users is movements in time

series• removes predictable variation from time series in

order to aid interpretation

Seasonal Adjustment

Data source: ONS

X-13ARIMA-SEATS

• Chosen as the seasonal adjustment software for use in official statistics (agreed by the Statistical Policy and Standards Committee in 2012)

• Produced by US Census Bureau• RegARIMA modelling used to ‘clean’ series• Can seasonally adjust with X-11 algorithm or

SEATS

RegARIMA models

• Time series regression models with ARIMA errors used to deal with autocorrelation

• Used to forecast/backcast • reduces revisions to seasonally adjusted series

caused by asymmetric moving averages used in the X-11 method

• also used to forecast some components of National Accounts

• ‘Cleans’ the time series• estimation and removal of outliers, level shifts etc

End point problem

Data source: ONS

Forecasting

Data source: ONS

Why selecting a good model is important

• Forecasts needed to deal with end point problem (asymmetric averages give an implied forecast)

• Good forecasts minimise revisions to the seasonally adjusted estimates

• Selection and estimation of regressors

Why is automatic model selection needed?

• ONS seasonally adjust thousands of time series -last year TSAB reviewed over 13,500 series

• Manual selection of ARIMA models is very time consuming- we’d never get through them all

Automatic model selection

• X-13ARIMA-SEATS provides two automatic model selection routinesPickmdl

Automdl

• Pickmdl was used in older versions of the software (from X-11-ARIMA onwards)

• Automdl is based on routine from TRAMO (available from X-12-ARIMA onwards)

Pickmdl

• Chooses the first model in the following list that satisfies a number of tests

(0,1,1)(0,1,1)s

(0,1,2)(0,1,1)s

(2,1,0)(0,1,1)s

(0,2,2)(0,1,1)s

(2,1,2)(0,1,1)s

• The tests are:• testing that the average absolute percent error of

forecasted values are within certain limits• test that the residuals are not correlated• no sign of over differencing

Automdl

• fits a (0,1,1)(0,1,1)s model

• identification of differencing orders by empirical unit root tests

• iterative procedure to determine ARMA model orders (maximum orders are set by default and can be changed)

• identified model is compared to default• final model checks

Data

• Monthly GDP• 176 monthly series• 40 quarterly series

• Data spans from January 1995 to

October 2014• Automatic detection of calendar effects and

outliers• Removed series that included seasonal

breaks

Results

• Stability• Forecast errors• Forecast differences• Differences in seasonal adjustment estimates

Stability of model selected

5 Years

7 years

9 years

11 years

Full series

.

.

.

.

.

.

.

.

.

.

.

.

Stability of model selected

5 Years

7 years

9 years

11 years

Full series

.

.

.

.

.

.

.

.

.

.

.

.

Stability of model selected

5 Years

7 years

9 years

11 years

Full series

.

.

.

.

.

.

.

.

.

.

.

.

Stability of the model selected

Forecasts

Data source: ONS

Average absolute percentage error in within-sample forecast values

Average absolute percentage error in within-sample forecast values

Automdl lower 30% of the timePickmdl lower 28% of the timeSame 42% of the time

On average, automdl's error is roughly 8% lower than pickmdl's

Average absolute percentage error in within-sample forecast values

Last three years

Average absolute percentage error in within-sample forecast values

Last three years

Automdl lower 35% of the timePickmdl lower 23% of the timeSame 42% of the time

On average, automdl's error is roughly 7% lower than pickmdl's

Forecast results

                 

F = Failures (%)AFD = Average Absolute Forecast Difference (%)BFP = Best Forecast Performance (%)LMV = Lowest Model Variance (%)

Monthly

Automdl Pickmdl

F 0 27.6

AFD 11.54 11.54

BFP 47 53

LMV 61 36

Quarterly

Automdl Pickmdl

F 0 5

AFD 10.21 12.07

BFP 58 42

LMV 68 32

Seasonal adjustment estimates

Data source: ONS

Seasonal adjustment estimates

• Pickmdl produced better adjustment in 6% of cases

• Automdl was better in 33% of cases

• 41% were exactly the same

• 20% showed very little difference

Data source: ONS

Revision analysis

Span 1

Span 2

Span 3

Span 4

Span 5

Revision analysis

Span 1

Span 2

Span 3

Span 4

Span 5

Revision analysis

Span 1

Span 2

Span 3

Span 4

Span 5

Revision analysis

Span 1

Span 2

Span 3

Span 4

Span 5

Seasonal adjustment revisions

Conclusions

• Pickmdl performs better when considering model stability

• Automdl has a lower average absolute percentage error in within-sample forecast values

• Out-of-sample forecasts performance:• Pickmdl appears slightly better for monthly series • Automdl appears much better for quarterly series

• Little difference in seasonal adjustment revisions between methods

Future work

• SEATS• Series with differing volatility• Simulated series• How different model selection impacts on

regressor identification and estimation• Errors on shorter spans• Sliding spans stability• Using current regressors

Thanks for listening!