1.

DemetraDoc.docx

0. Introduction

Demetra+ is aimed at providing facilities to make seasonal adjustment. Users are able to analysein details a specific time series (single analysis mode) or to process many series at once (batchprocessing). The software is designed to handle series coming from various sources and to processTramo-Seats and X13 in a similar way. The results and the diagnostics can be analysed through avariety of tools, which are to a large extent common to both algorithms.

The documentation is organized as follows: in a first point we take a look on the ways ofintroducing external data in the application. In a second paragraph, we show how to use thesingle analysis mode. The third paragraph focuses on batch processing. In the forth point, weshortly give some explanations on peripheral topics. Finally, we present in the fifth paragraphdetailed information -functional and statistical - on the various graphical components.

1. Time series providers

1.1. Introduction

The different time series providers produce the time series that will be used in the application.They are displayed on the top-left panel of the application. The list of the available data sourcesdepends on the local installation. By default, the following time series providers appear on thescreen:

Workspace: Xml files (Demetra+ specific)

Txt: tab separated text files

Excel: Excel 2003/2007 files

ODBC: generic ODBC module

TSW: time series in the format supported by TSW

USCB: time series in some of the formats supported by X12/X13

SDMX: SDMX file (compact model)

That list can be extended by users' defined implementations (such as the "GTAB" provider of theNBB)

2.

DemetraDoc.docx

The default providers are shortly described below.

1.2. Xml files

1.2.1. Description

Xml files are written in a specific xml format, which is used by Demetra+ to store time seriesinformation1. The "workspace" panel shows the time series contained in such a file; a workspaceis a set of time series collections. It is presented in a two-level tree

The first nodes (Exports, tmp2) represent time series collections (lists) and the final nodesrepresent time series (Union européenne...). The name of a collection can be modified (click on itand wait for an instant to edit the name of the collection).

1.2.2. Opening an existing Xml file

To open an existing, use the command File Workspace Open...

1 The specific xml format will probably be replaced by an sdmx compact format in future versions.

3.

DemetraDoc.docx

The default location for xml files is "My Documents\Data\Xml".

It is possible to load and to analyse files coming from other places. However, series integrated in"reproducible" processing should belong to the default location. The xml provider follows anapproach based on a "relative path": files are identified by the default location (or base folder),which is common to all files and which can be changed easily, and by their position in that folder2.

1.2.3. Creating a new xml file

The user can create a new xml file and import external data in it.

To create a new workspace, use the command File Workspace New

To import external data (for instance from Excel), the user must copy the suitable information andpaste it in the workspace by means of the command File Paste

A new time series collection (called tmpXX) is added to the current workspace. It can be savedusing the command File Workspace Save

1.2.4. Importing data from Excel

We present below an example of Excel worksheet that can be integrated in Demetra+.

It must contain a first column/row of true dates and a first row/column with the names of theseries; the top left cell should be empty. Missing values are identified by empty cells.

2 However, the location of the default folder itself can be changed (for the time being, no graphical interface for thatfeature)

4.

DemetraDoc.docx

It should be noted that external data can also be directly dropped or paste in other componentsof Demetra+, like containers...

1.3. Text files

Demetra+ is able to read tab-separated text files. The location of those files is also based on therelative path approach (see xml files).

The data may be presented vertically or horizontally. They must contain an array of "true" dates(first column/row) and an array of titles (first row/column); the top left cell must be empty. Datesand figures should be formatted using "culture invariant" formats (for example "yyyy-MM-dd" fordates)

See below for an example

5.

DemetraDoc.docx

The Txt provider appears in a tree, just like the Xml provider. However, the user must double clickany item corresponding to a file (for example "Insee1.txt") to open it and to see the series itcontains.

1.4. Excel files

Demetra+ supports Excel 2003 (.xls) or Excel 2007 (.xlsx) files3. The data contained in Excel may bepresented vertically or horizontally. They must contain:

an array of "true" dates (first column/row)

an array of titles (first row/column)

Missing values are identified by empty cells.

See below for an example

3 Some problems appear with old versions of Excel, especially when there are missing values at the beginning of someseries. Those problems, which come from the ADO engine of Excel, cannot be solved easily.

6.

DemetraDoc.docx

For loading Excel workbooks, go to the Excel panel and click on the "Open" button.

Some warnings could appear if some parts of the workbook cannot be successfully read. Furtherimplementations of the Excel provider should be more robust (in the sense that they wouldaccept "dirty workbooks" more easily).

The current Excel provider doesn't use any "default location"; it is based on absolute locations;files may be everywhere, but they should not be moved if they are integrated in batch processing.

That option could be changed in the future by the relative location approach used for xml files.

Demetra+ is delivered with some examples of workbooks, located in [My documents]\data\excel.

The Excel provider appears in a tree, where the nodes are defined by the open workbooks, theirworksheets and their columns (rows).

1.5. ODBC provider

7.

DemetraDoc.docx

The default implementation contains a generic solution that allows the connection to databasesthat supports ODBC. The current solution can be easily modified to fit the design of specificdatabases. However, in many cases, the use of views (or of queries for MS-Access databases)should be sufficient to provide a simpler solution.

Let's suppose that our data can be found amongst a database with an ODBC "data source" called"Demetra". This database would have a table such as this one:

TSTABLE

TsDomain TsPeriod TsSerie TsValue

Domain_1 Serie_1 Period_1 Value

Domain_1 Serie_1 Period_2 Value

Domain_1 Serie_2 Period_1 Value

Domain_2 Serie_3 Period_1 Value

... ... ... ...

A new ODBC provider could be added from the ODBC panel using the following properties:

In this example, we suppose that the "Demetra" data source has been successfully configured,using the ODBC Data Source Administrator (see below).

8.

DemetraDoc.docx

Starting from the previous example, it is easy to create, for example, the corresponding quarterlyseries by means of the following SQL Query (MS-ACCESS).

SELECT *

FROM

[SELECT SUM(TsValue) AS QValue, MAX(TsPeriod) AS QPeriod, TsDomain, TsSerie

FROM TsTable

GROUP BY TsDomain, TsSerie, YEAR(TsPeriod), ROUND((MONTH(TsPeriod)+1)/3)]. AS [TEMPORARY]

WHERE MONTH (QPeriod) IN (3,6,9,12);

If we suppose that that query is called QTable, we can now define the corresponding ODBCprovider as follows:

9.

DemetraDoc.docx

It will be able to provide dynamically the quarterly time series.

The ODBC provider appears in a tree, where the nodes are defined by the data sources(DataSrcName), their loaded tables (TableName), the defined domains (DomainColumn)and theirseries (TsColumn).

Demetra+ is delivered with some examples of Access databases, located in [Mydocuments]\data\ODBC.

1.6. TSW files

Demetra+ is able to read time series files written for TSW. The location of those files is based onthe relative path approach (see xml files). The TSW folder can contain several levels of sub-folderswith TSW files. They will appear in the tree navigator of the TSW provider. The series in a sub-folder will be grouped in a collection called "All series"

1.7. X12/X13 files

10.

DemetraDoc.docx

The X12/X13 files (input as well as output format) are handled by Demetra+ exactly like the TSWfiles. This version is not able to read X12/X13 metafiles.

1.8. SDMX files

SDMX files (compact data format) may be read by Demetra+ and displayed in a navigation treecontaining the open files and all the series (unstructured).

Demetra+ is delivered with some examples of SDMX files, located in [My documents]\data\SDMX.This provider is in a very experimental state.

1.9. Interaction

Time series collections and time series displayed by the different providers can bedragged/dropped to other components (like tables) or to other software (like Excel)

When a time series is double-clicked, the default action (usually the execution of a seasonaladjustment) is started, using that time series.

1.10. Settings of the providers

11.

DemetraDoc.docx

The list of the available providers and of their properties can be displayed by means of thecommand Options Data providers...

The user can (de)select providers and/or modify their properties4

Unavailable providers or graphical interfaces are mentioned by disabled check boxes.

Those settings are saved between different sessions of Demetra+.

2. Single analysis

2.1 Creation of a SA Analysis

You create a new seasonal adjustment analysis by the menu item "File->Analysis->TramoSeats/X13"

Those windows are initialized by default specifications, which can be defined for each method(see 4.3).

4 The current modules provide very few options. Future versions will be much more flexible.

12.

DemetraDoc.docx

It is possible to create as much SA windows as you will. On way to start the actual execution of theseasonal adjustment(s) is to double click on any series coming from any time series provider (Xml,Txt, ODBC, Excel...). If several windows are opened, each of them is updated with the chosenseries (except if it has been locked; see below).

It should be noted that, if no unlocked SA window is available, a double click on a series canautomatically open a new one, in function of the default settings of the application (see 4.3).

Another way to start a seasonal adjustment is to import a time series in an already open (andunlocked) SA window. It can be done either by dropping a time series object in the left top chartor by pasting it through the command "X13/Tramo-Seats Edit Paste"

2.1. Serialization of a single processing

A single processing of X13 and of Tramo-Seats can be saved by means of the command"X13/Tramo-Seats Edit Save/Save as". The results is an Xml file (stored by default in the folder[My Documents]\Eurostat\Demetra+\X13 or TramoSeats). A saved document is locked andcannot be unlocked. That means that the series cannot be changed, but that the specifications

13.

DemetraDoc.docx

used to compute the seasonal adjustment can still be modified. A saved processing can be re-open through the command "File Analysis Open". That command might fail if the document isalready open.

When it is save in "the single analysis mode", a time series becomes "static"; its link with possibleexternal providers is removed, so that the document is not automatically modified by newobservations when it is re-open. However, it is still possible to refresh manually the data by meansof the command "TramoSeats/X13 Refresh time series".

It should be noted that information to refresh a time series (its source and identifier) and the lastrefresh are stored in its metadata; they can be visualized in the "TSProperties" component (seebelow)

2.2. Other commands on a single processing

It is possible to display the specifications of the different methods, to copy/paste5 the originalseries, to lock the current view, which means that it is no longer modified by a double click of aseries...6

The different panels of the SA window are automatically updated when the specificationsdisplayed in the properties dialog are modified and when the user click on the "apply" button toconfirm his modifications. That window is modeless, which means that it is not necessary to closeit to do something else (for example for clicking on a different panel of the SA Window).

5 The "Paste" command allows a direct SA processing of external series (without using the "workspace" panel).

6 The "Save" command has still to be implemented.

14.

DemetraDoc.docx

A complete description of the SA windows is provided in point 5 (forthcoming).

3. Batch processing

Document in progress. Will be modified

3.1 Creation of a batch processing

3.1.1. Command

"File->Batch processing->New..."

The user must first chose the SA method (TramoSeats or X13).

15.

DemetraDoc.docx

The default specifications are then used (see 4.3). The user can change the default specification byasking for the details of the method (click on the button [...] in the "details" field).

3.2. Importing series in the batch processing document

16.

DemetraDoc.docx

The user can import series in the batch processing document by drag and drop or by copy/paste(the paste is done through the command "SA Processing->Edit->Paste". The series can beimported one by one or by groups (drag the "Turnover" node in the next example)

3.3. Running the SA processing

In the case of a new SA Processing, use the command "SA Processing->Run->Start", which shouldbe enabled.

The series processed are indicated on the SA Processing view (in blue) and the quality of theestimation appears (green, orange or red). In this prototype, the symbols (+?-) indicate tests onthe residuals (successively normality [1], independence (Ljung-Box) [2], periodogram peaks ontrading days frequencies [3] and on seasonal frequencies [4]).

Detailed information on a processing is obtained by a double click on an item (series part). Itshould be noted that, once that window is visible, you just have to change the selected item in thelist (by a single click or through the keys ) to get the corresponding results.

17.

DemetraDoc.docx

The user can modify the specifications of the selected item, apply them and, if necessary, savethem. The results are then automatically updated.

3.4. Saving the batch processing

The batch processing is saved by the command "SA Processing->Save". The saved processingappears in the bottom left panel.

3.5. Re-executing a batch processing

3.51 Double click the SA Processing in the bottom left panel (or use the local menu "Open").

3.52 Generate the specifications for that processing (for example "SA Processing->Generatespecifications->Intermediary").

The new specifications are generated as follow:

The specifications that correspond to the estimated model in the previous estimation areloaded (as they were saved).

18.

DemetraDoc.docx

Following the estimation policy corresponding to the menu item, those specifications aremodified: for example, in the case of "free parameters", the form of the model(regression variables + ARIMA orders) is kept but all the parameters are re-estimated. Acomplete description of the possible options is provided in appendix?

3.53 Run the processing ("SA Processing->Run->Start").

3.6. Displaying the last estimation

It is possible to see the last saved estimation (local menu "Results->Last estimation") or thecurrent (unsaved) one (local menu "Results->New estimation")

3.7. Others

After a complete processing, it is possible to show a short report ("SA Processing->Report"), todisplay the results by components (for example, all the seasonally adjusted series), through theresults panel and to generate outputs ("SA Processing -> Generate output"). The output dependson the available modules and on the configuration of the application; the current prototypeprovides the creation of detailed xml files for each series and the creation of light text files withthe main time series generated by the processing (original, seasonally adjusted, seasonal, trend,irregular).

Through the menu "SA Processing->Save definition", It is also possible to save the "definition" ofthe processing (specifications and identifiers of the series, without data). That file should be thebasic input for batch processing using sacruncher.exe (console application. See thedocumentation of sacruncher for further information).

3.8. Remarks

The diagnostics, the storage of the processing and the outputs may be modified/extended byexternal IT teams. See the documentation on Demetra+'s extensibility for further information.

4. Various

4.1. Auxiliary statistical tools

19.

DemetraDoc.docx

The current prototype contains two main statistical tools, which reflect some parts of thealgorithms involved in Tramo-Seats and in X13. The "differencing tool" searches the unit roots ofthe series (based on Tramo-Seats) and the "trading days analysis" compares the approaches ofTramo-Seats (T-Stat), of X13 (AIC Test, Spectral diagnostics) and alternative solutions (F-Test,Periodogram) in the detection of calendar effects.

The series that the user wishes to analyse must be dropped in the top left chart (differencing) orin the left listview (calendar). Those tools should only be considered as examples of what ispossible and certainly not as definitive solutions.

4.2. Containers and tool windows

The application provides several containers or tool windows, where the user can drop (paste)time series coming from other containers (charts, table...) of the application (or from externalsoftware). See points 5 for more information.

Most of the time series displayed in the application (in the form of list items, lines in a chart,column in a table...) may be drag to the different containers or tool windows.

For example, in the next snapshot, the "trend" was dragged from the results tree of"TramoSeats[1]" to the "chart-1" container; it was then dropped in the top left chart of the"Differencing tool"; the irregular component was dragged from the top-left cell of the table to theperiodogram tool window.

20.

DemetraDoc.docx

4.3. Options

The application contains several general options, which have an impact on many features. Themost important ones are available through the "Settings..." sub-item

21.

DemetraDoc.docx

The default action defines what happens when a series is double clicked (and if there is no openedSA window).

The user's level modifies the number of options available in the specifications dialog boxes.

Finally, the default specifications, used in batch processing as well as in SA analysis may bemodified through the X13 details and the Tramo-Seats details.

All those choices are saved between different sessions of Demetra+

The other options are only partially implemented.

5. Description of the graphical components

5.1. Introduction

Complex graphical components, like a complete view on a seasonal adjustment, are composed ofsome basic graphical components, such as a simple time series graphs. A same basic componentreused in many other more complex ones will present everywhere the same behaviour. Wedescribe in this point the different graphical components used in Demetra+. Detailed statisticalinformation will also be provided if need be.

5.2. Template for the description of the graphical components

A complete description of all the graphical components is a long but necessary work. We intend touse for that goal a common template for the different components (some items could be missingin some components). It is described below and an example is provided for the "Periodogram" see5.4.1.

5.2.1.1. Description

Snapshot and general description of the component

5.2.1.2. Functionalities

5.2.1.2.1. CommandsActions that can be applied on the components through menus

5.2.1.2.2. Interactivity

22.

DemetraDoc.docx

Actions that can be generated through the mouse

5.2.1.3. Statistics

5.2.1.3.1. UsesExamples on the statistical use/signification of the graphical component

5.2.1.3.2. DetailsExact description of the statistics displayed by the component. Some algorithmic details can alsobe provided

5.2.1.3.3. Bibliography

5.2.1.4. Comments

5.2.1.4.1. BugsBugs encountered by the user could be added here

5.2.1.4.2. ProposalsIdeas for the improvement of the component (graphical or methodological)

5.2.1.4.3. Planned actionsModification of the component already planned

5.3. Basic time series components

5.3.1. Chart

5.3.2. Growth chart

5.3.3. Table (single series)

5.3.4. Table (multiple series)

5.3.5. List

5.3.6. Time series tree

5.3.7. Seasonal component chart

23.

DemetraDoc.docx

5.3.8. Seasonal SI chart

5.4. Basic time statistical components

5.4.1. Periodogram

5.4.1.1. Description

The periodogram component is a powerful tool that explores a set of data in the frequencydomain. It is defined for any set of ordered values (represented by doubles), with possible missingobservations (NaN).

The graphical component shows some special regions that correspond to the seasonalfrequencies (blue) and to trading days frequencies (violet), provided that the periodicity of theobservations is available. It also contains a horizontal yellow rule, which delimits highly suspectpeaks.

The X-axis shows the different frequencies; the periodicity (expressed as a number of time

periods for an observation) of a phenomenon at frequency f is ; for example, in the case of a

monthly series, the frequency correspond to a periodicty of 6 months.

5.4.1.2. Functionalities

5.4.1.2.1. CommandsNo specific commands

5.4.1.2.2. InteractivityTime series can be dropped in the periodogram components; their observations are thenanalysed, using the periodicity of the series.

5.4.1.3. Statistics

5.4.1.3.1. Uses

24.

DemetraDoc.docx

The yellow rule corresponds to a significance level of 0.5%. It is not abnormal to get a few valuesabove that level. However, if a peak corresponds to a seasonal or to trading days frequencies, theseries should be considered in details.

In the example above, the periodogram of the residuals computed on a monthly series shows asignificant half-yearly peak and a significant trading days peak. The other peaks should not beconsidered.

We present below the periodogram of some specific effects

Monthly Easter effect generated by Tramo (duration=6) Pure working days effect

Over-differenced series (abnormal peaks on the high frequencies)

Under-differenced series (abnormal peaks on the low frequencies)

5.4.1.3.2. Details

For a given series( ) , which may contain missing values, the periodogram is computed asfollows:

In a first step, the series is standardized:

25.

DemetraDoc.docx

( , )( )

In a second step, we compute at the so-called Fourier frequencies ( = , 0 < ) the

values of the periodogram:

2

where is the number of non missing values.

To speed up the processing, is computed recursively (using ( ) ).

Under the white noise hypothesis, the values of the periodogram should be asymptoticallydistributed as a Chi-square with 2 degrees of freedom.

The default frequency (td) for trading days is computed as follows (for series of frequency q):

= 365.25

= 7 7

Other frequencies correspond to trading days frequencies:

For monthly series, 2.714 ( default = 2.188)

For quarterly series, 1.292, 1.850, 2.128 (default = 0.280)

5.4.1.3.3. Bibliography

Quinn B.G. and Hannan E.J., "The estimation and tracking of frequency", Cambridge UniversityPress, 2001)

5.4.1.4. Comments

5.4.1.4.1. Bugs

5.4.1.4.2. Proposals

5.4.1.4.3. Planned actionsUse of different X-axis: periodicity, removal of the factor (axis ranging from 0 to 0.5)

Possibility to copy / drag the values

Possibility to paste /drop different kind of information (not only time series)

26.

DemetraDoc.docx

Modification of the "alarm" level (yellow rule)

5.4.2. Auto-regressive spectrum

5.4.3. Pseudo-spectrum

5.4.4. ACGF function

5.4.5. Autocorrelations

5.4.6. Squared gain function

5.4.7. Phase effects

5.4.8. Linear Filters

5.4.9. Distribution

5.4.10. Residuals statistics

5.5. Composite components

5.5.1. Time series properties

5.5.2. Time series comparison

5.5.3. Revisions history

5.5.4. Sliding spans

5.5.5. Differenced series

5.5.6. Differencing tool

5.5.7. Residuals analysis

27.

DemetraDoc.docx

5.5.8. Wiener-Kolmogorov filter analysis

5.5.9. Seasonality tests