Implementation of SDMXfor data and metadata exchange

SDMX Basics Course12 April 2013

Daniel SuranyiEurostat B5Management of statistical data and metadata

• What is SDMX?

• Why use it?

• How do we get there?

• Where are we today?

• What can data providers do?

Questions

The SDMXComponents

3

...not just a transmissionformat !

SDMX – opportunities

4

Streamline data flows Central management

(SDMX Registry)

Software tools Data sharing

Data structures, concepts, code lists

Across domains

Across organisations

Questions

• What is SDMX?

• Why use it?

• How do we get there?

• Where are we today?

• What can data providers do?

6

Eurostat SDMX Implementation Process

6

Questions

7

• What is SDMX?

• Why use it?

• How do we get there?

• Where are we today?

• What can data providers do?

• JVS/LCI(Job Vacancy Statistics/ Labour Cost Index)

• Maritime• STS

(Short Term Statistics)

• TEC(Trade by Enterprise Characteristics)

• International Trade*

• National Accounts*

• R&D*(Research & Development)

• Balance of Payments*

• EGR IS(EuroGroups Register Identification Service)

• Aviation• Fisheries*• Orchards• Pesticides• Waste

Eurostat project status (data exchange)

* Other international organisations involved

Implementation projects – Some milestones

Domain International actors Production phase

Fisheries UN (FAO) 2011

Balance of Payments SDMX Sponsors(IMF coordinating)

09/2014

National Accounts SDMX Sponsors(Eurostat coordinating)

09/2014

Research & Development

OECD, UNESCO ~2014

International Trade UNSD, OECD To be defined

• Census• HICP/

Compliance monitoring

• Pesticides• R&D

(Research & Development)

• Waste(additional workflows)

• ESAW• Tourism• Waste

• COMEXT(3rd countries)

• COSAEA• Education

(Pilot phase)

• JVS/LCI(Job Vacancy Statistics/ Labour Cost Index)

• PRODCOM

• HICP• LFS-Q• EU-SILC• FATS• STS

(Short Term Statistics)

• LFS-A

Eurostat implementation projects (metadata exchange)

* Other international organisations involved

Questions

• What is SDMX?

• Why use it?

• How do we get there?

• Where are we today?

• What can data providers do?

SDMX Implementation in Member States

12

Tools offered Action

Web Forms No costs for organisation

Manual work for senders (type, copy/paste)

SDMX Converter

(sender’s PC)

Installation on sender’s PC

Manual work for senders (convert)

SDMX Converter

(batch mode)

Installation on server

SDMX Reference Infrastructure

Installation on server

Mapping of database to DSDs

Own SDMX Implementation

Local development in organisations

SDMX Implementation in Member States

13

Example SDMX Converter

1. Generate CSV or GES2. Take DSD and data file3. Produce SDMX-ML

All steps each time

Example SDMX-RI(installed in ~20 NSIs for Census Project)

1. Take DSD2. Link to database3. Produce SDMX-ML

Step 1 & 2 only once Only step 3 each time

Data Validation Process

Before/During Transmission(“First Level”) - Performed in Member States

- Format Check (SDMX-ML) - Code Check (SDMX DSD)

After Transmission( “Second Level”) - Performed in Eurostat

- Detailed value check - Mirror check - …

Support for data providers

• SDMX Reference Infrastructure Installed in ~20 countries (NSIs) for Census Hub Can be extended to other domains

• 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX

• 2nd contact point - Eurostat support teams

Questions?

16

• SDMX Support Team• ESTAT-Support-SDMX@ec.europa.eu

• SDMX Website• http://www.sdmx.org

• Eurostat SDMX Info Space• https://webgate.ec.europa.eu/fpfis/mwikis/sdmx