improving trend forecasting in sap apo dp

Improving Trend Forecasting in SAP APO DP © 2014 Crone | All rights reserved.

Dr. Sven F. Crone Assistant Professor - Director

Improving Trend Forecasting in SAP APO DP An empirical evaluation of different initialization procedures for Holt’s ETS

1 July 2014, ISF’14, Rotterdam, Netherlands

Agenda

• Background – Relevance for ETS initialisation – Best practices in Research – Established practices in Software

• Experiments – Research question & design – Results of an empirical evaluation – Developing an add-on to SAP APO DP

• Feedback & Suggestions

Improving Trend Forecasting


Initialisation procedure is influential for ETS forecasting accuracy

User choices to implement Exponential Smoothing (ETS) • initial values, {S0, T0, I0,…,p}

• parameters (fixed or adaptive) {α, β, γ, ϕ}

• loss functions, {MSE, MAD, …}

• decide whether to (re-)normalize the seasonals [Gardner (2006)]

• (individual selection of adequate model form)

ETS Initialisation Relevance of Initial Values

“Parameter selection is

not independent of initial

values and loss functions” 2-stage approach in standard ETS

simultaneous in state-space ETS

DA-M ( aka DM aka ETS(A,D,M)

Damped additive trend &

multiplicative seasonality

Optimisation dominates: full opt./2-stage/heuristic only [Hyndman et al. (2002)], MLE [Broze and Mélard (1990)], nonlinear programming [Segura and Vercher (2001)], etc.

How to determine initial smoothed values? • Question posed since ETS origin [Wade (1967), Cogger (1973), McClain (1981), Taylor (1981)]

• Various „innovative“ approaches in research papers [see, e.g., Bates & Granger (1969)]

• Few guidelines and no empirical evidence [Gardner (1985), Chatfield and Yar (1988)]

Approaches

(1) Least squares estimates [Brown (1959)]

(2) Backcasting [Ledolter and Abraham (1984)].

(3) Global values of Training set [Makridakis et al. (1983)]

(4) Convenient (heuristic) initial values [Makridakis and Wheelwright (1978)]

(5) Zero values (e.g. for all or some) [Makridakis & Hibon, 1991]

• ‘Starting values and loss functions don’t make any difference as optimal smoothing

parameter(s) found compensate for various starting values’ [Gardner (1990b)]

• Makridakis & Hibon (1991): first evidence & propose guidelines on 1001 M-series

(3 nonseasonal ETS methods, 7 types of initial values, multiple loss functions)

”contrary to expectation accuracy is not affected by the type of initial values used”

ETS Initialisation Best practices in Research

“most widely used in practice” [Makridakis & Hibon, 1991]

• Using (longer) averages

• Global Averages of values with

(bounded) parameter Optimisation

• Backcasting of values with


• Optimisation of values with


From a practical point of view the prevalent use of OLS estimates … seems adequate, … it makes no sense to consider more elaborate alternatives … since such alternatives are more difficult to program and require more computer time.

Initialization named as area of “most of the

work since 1980”[De Gooijer & Hyndman (2006)]


Initialisation problem for ETS AAN and 2nd order ETS (in APO) is significant best practice recommendation: avoid all trend models in forecasting!

poor “Naïve” Initialisation will impair complete forecast

5 10 15 20 25 30 35 40 4570

80

90

100

110

120

130

140Out-of-sample

Month

GD

P

Data

APO

External

5 10 15 20 25 30 35 40 4580

90

100

110

120

130

140

Out-of-sample

Month

GD

P

Data

APO

External

Heuristics dominate commercial software(not (Hyndman & Khandakar, 2007)

• proc forecast: run time-trend regression on NSTART=8 values (or customize)

• S0 , I0, T0 , J0 are set „equal to reasonable guesses based on the data“.

• S0 =X0 and T0 = X2 -X0

ETS Initialisation Best practices in Practice

3 years in-sample data not enough to forget a bad initialisation requires higher

smoothing parameters

Using higher α and β smoothing parameters Does not filter noise

adequately

Agenda

• Background – Relevance for ETS initialisation – Best practices in Research – Established practices in Software

• Experiments – Research question & design – Results of an empirical evaluation – Developing an add-on to SAP APO DP

• Feedback & Suggestions

Improving Trend Forecasting

http://www.sap.com/index.epx


• OLS (optimisation)

• Backcasting

• Global training values

Empirical evaluation on aggregate vs. individual selection of ETS inits

Experiments Empirical Design

• Heuristics

• SAP APO DP

• Naïve

• 3-period Average

ETS Initialisations

• ETS(A,A,N)

• ETS(A,D,N)

ETS Models

• Individual selection of

„best“ ETS initalisation

• α & β (& init)-optimisation

• LBFGSB

• Constrained parameter

ranges α,β=[0.01,0.4], ϕ

• Min! SAE t+1,2,...,12

• Model selection on in-

sample MAE t+1,2,…,12

“However, we must emphasize that our results apply to the average of forecasts that have been found mechanically (i.e., using an automatic approach)

without studying each series separately to determine the best initial values or optimal loss function.

In our view additional research will be required to determine if our findings also apply when

single series are studied and optimized individually.” Makridakis & Hobon (1991)

Aggre

gate

Sele

ctio

n

Simple empirical evaluation to improve on SAP APO DP

Dataset • Empirical dataset of FMCG of skincare manufacturer Beiersdorf

• 35 time series of corrected history (no OOS, promotions etc.)

• Time series exhibit trend (manually identified) across 6 affiliates

Experimental Design • Model evaluation on out-of-sample data (18 observations)

• Using multiple robust error metrics: sMAPE, MASE

• Estimated on multiple step ahead horizons

• Operational: t+2,…,t+4 14 rolling origins

• Tactical: t+1, t+2, … , t+12 6 rolling origins

Hypothesis • All inits (heuristics, backcasting, optimisation etc.) perform equally well

• Individual selection of ETS initialisation does not improve

Experiments Empirical Design


Reduces sMAPE from 73.08% 58.41% on test t+1…12 (Reduces sMAPE from 106.42% 43.13% on training)

Experiments Examples of Initialisations

SAP® APO DP Standard Initialisation

Novel Initialisation with IF

Increases sMAPE from 55.54% 59.82% on test t+2…4 Reduces sMAPE from 82.58% 39.81% on train t+2…4





Reduces sMAPE from 104.8% 64.77% on test t+1…12 Reduces sMAPE 94.79% 41.33% on train t+1…12




Reduces sMAPE from 72.31% 66.18% on test t+2…4 Reduces sMAPE from 63.24% 34.98% on train t+2…4





Increase sMAPE from 54.33% 31.75% on test t+1…12 Reduces sMAPE 60.93% 33.63% on train t+1…12




All but 1 initialisations outperform SAP APO DP trend initialisation Largest reduction of sMAPE through individual selection of inits

t+1,2,…,12 mean sMAPE *100 Median sMAPE*100

Train Test Train Test Improve Rank 1a Optmise AAN 41.44% 50.23% 38.16% 42.73% -12,75% 5 1b Optmise ADN 42.34% 52.47% 39.14% 43.71% -11,77% 6 2a Global AAN 36.26% 46.71% 34.33% 40.21% -15,27% 3 2b Global ADN 37.75% 49.14% 36.55% 40.98% -14,50% 4 3a Backcast AAN 42.30% 59.91% 37.67% 48.25% -7,23% 9 3b Backcast ADN 36.30% 49.27% 33.10% 39.57% -15,91% 2 4a Heuristic MovAV3 AAN 34.37% 56.53% *32.03% 45.73% -9,75% 8 4b Heuristic MovAV3 AAN 56.88% 58.74% 58.37% 51.40% -4,08% 10 5a Heuristic Naive AAN 36.90% 54.96% 37.27% 45.46% -10,02% 7 5b Heuristic Naive ADN 81.16% 64.26% 82.61% 60.58% 5,10% 11 6a Heuristic SAP APO DP 68.84% 59.72% 72.34% 55.48% - Inividual Selection Init *32.96% *46.70% 32.11% *38.17% -17,31% 1

Results consistent across error metrics (MASE) and horizons (t+2,…,4)

Experiments Results of Initialisations


Reduction of sMAPE through individual selection of inits Significance? Conditions? tdb!

Experiments Results of Initialisations

Extend SAP APO DP New Methods outside of APO

Solutions exist: – Use bolt-on systems for model selection – Use bolt-on systems to run new methods

Forecast Profile Model Type Mosel-specific Settings Forecast Profile Model Type

Seamlessly implement new trend models & initialisations

Implement other methods: ETS (ANA), (AAA), Fourier Reg.

APO-DP

http://www.sap.com/index.epx


Take aways

• Commercial software ignores academic best practices Forecasting practices can be enhanced

• Research use of Aggregate OLS optimised initial values is not the most accurate for all datasets need to consider initialisation in ETS specification

• Forecast Models outside APO can be included in APO use novel algorithms from within APO, incl.

• ETS with damped trend, with additive seasonality etc. • Neural Networks, Support Vector Regression, Decision Trees

• Future Work – Extend to more representative datasets – Assess conditions under which different trend models

work well – Assess heuristics provided (e.g. Hyndman et al) …

Dr. Sven F. Crone Assistant Professor, Director

Lancaster University Management School Research Centre for Forecasting

Lancaster, LA1 4YX

Tel. +44 1524 5-92991 [email protected]

Questions?


Dr. Sven F. Crone

Assistant Professor, Director Lancaster University Management School

Research Centre for Forecasting

Lancaster, LA1 4YX Tel. +44 1524 5-92991

[email protected]

Copyright

These slides are from a workshop on forecasting.

You may use these slides for internal purpose, so long as they are

clearly identified as being created and copyrighted © by Sven F. Crone.

You may not use the text and images in a paper, tutorial or external

training without explicit prior permission from the centre.

Selected References

• Hyndman, R. J., Koehler, A. B., Snyder, R. D., & Grose, S. (2002). A state space framework for automatic forecasting using exponential smoothing methods. International Journal of Forecasting, 18, 439– 454.

• Makridakis & Hibon (1991) Exponential smoothing: The effect of initial values and loss functions on post-sample forecasting accuracy, IJF, 7, pp. 317-330

• Ledolter, J., & Abraham, B. (1984). Some comments on the initialization of exponential smoothing. Journal of Forecasting, 3, 79– 84.

• Broze, L., & Me´lard, G. (1990). Exponential smoothing: Estimation by maximum likelihood. Journal of Forecasting, 9, 445–455.

• Segura, J. V., & Vercher, E. (2001). A spreadsheet modeling approach to the Holt–Winters optimal forecasting. European Journal of Operational Research, 131, 375– 388.

• SAS Institute Inc., SAS/ETS User’s Guide, Version 8, Cary, NC: SAS Institute Inc., 1999. 1546 pp. (Chapter 12, The FORECAST Procedure)

improving trend forecasting in sap apo dp

Documents