webinar on sdmx implementation in research & development

Eurostat

Webinar on SDMX implementation in Research & Development

Eurostat

SDMX-RD Webinar – Agenda

• SDMX Overview• SDMX-RD Project overview• SDMX – how do I get there?• Demonstrations

• SDMX Converter• SDMX Reference Infrastructure (SDMX-RI)• SDMX-RI Mapping Assistant• SDMX-RI clients (PUSH and PULL Mode)

• Questions & Answers

Eurostat

The SDMXComponents

3

IT Infrastructure for exchange and sharing

Content Oriented Guideline

s

SDMX

Informati

on Model

Eurostat

The SDMXComponents

4

Describe statistics in a standard way Objects and their relationships

Data Structure Definition (DSD), Concepts, Code List Central management and standard access

SDMX Registry, SDMX Web Services

Cross Domain Concepts Cross Domain Code Lists Statistical Domains Metadata Common Vocabulary

Push Provider generates and sends file to receiver

Pull Provider opens web service to data Receiver downloads regularly

Hub Special case of pull: receiver downloads on end user request

Eurostat

SDMX and Validation Before/During Transmission

(“First Level”) - Covered by SDMX standard

- Format Check (SDMX-ML) - Code Check (SDMX DSD)

After Transmission( “Second Level”) - Not yet covered(planned end-2014)

- Detailed value check - …

Eurostat

Data flows in R&D

Eurostat

SDMX-RD project overview• Participants:

• Eurostat, OECD, & UNESCO

• Scope and objectives• Create joint DSDs, harmonised codelists, and

simple validation rules for the R&D data• Operationalise the use of these SDMX artefacts

into our processes

Eurostat

SDMX-RD "Pilot Package"• DSD Matrix

• R&D Coding sheet

• SDMX Data Structure Definitions

• Related tools and support

Eurostat

Technical Integration

National Organisations International

National databases (Mapping)

Source

Source

Source

SDMX Data Structure Defintions

& Data Flows

National data sources

Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

Eurostat

Data provider view:One goal – different possibilities

WebformsEDAMIS

Single Entry Point

SDMX-ML file

Database export SDMX Converter SDMX-ML file

Database SDMX Reference Infrastructure

SDMX-ML fileSDMX Web Service

Same DSDSame Format

Excel sheets SDMX Converter SDMX-ML file

Eurostat

SDMX-RD Webinar – Agenda

• SDMX Overview• SDMX-RD Project overview• SDMX – how do I get there?• Demonstrations

• SDMX Converter• SDMX Reference Infrastructure (SDMX-RI)• SDMX-RI Mapping Assistant• SDMX-RI clients (PUSH and PULL Mode)

• Questions & Answers

Eurostat

SDMX-RD Webinar – Agenda

• SDMX Overview• SDMX-RD Project overview• SDMX – how do I get there?• Demonstrations

• SDMX Converter• SDMX Reference Infrastructure (SDMX-RI)• SDMX-RI Mapping Assistant• SDMX-RI clients (PUSH and PULL Mode)

• Questions & Answers

Eurostat

Data Provider Data Collector

Non-SDMXlocal database

SDMX data set

SDMX Registry

DSD

SDMX-RI

Mapping Assistant

Web Svc

Test Client

SDMX Reference Infrastructurehttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI

Web Client

Eurostat

Data Provider Data Collector

Non-SDMXlocal database

SDMX data set

SDMX Registry

DSD

SDMX-RI

Web Svc

Test Client

SDMX Reference Infrastructurehttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI

Web Client

Mapping Assistant

Eurostat

SDMX-RD Webinar – Agenda

• SDMX Overview• SDMX-RD Project overview• SDMX – how do I get there?• Demonstrations

• SDMX Converter• SDMX Reference Infrastructure (SDMX-RI)• SDMX-RI Mapping Assistant• SDMX-RI clients (PUSH and PULL Mode)

• Questions & Answers

Eurostat

Data Provider Data Collector

Non-SDMXlocal database

SDMX data set

SDMX Registry

DSD

SDMX-RI

SDMX Reference Infrastructurehttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI

Mapping Assistant

Web Svc

Web Client

Test Client

Eurostat

SDMX-RD Webinar – Agenda

• SDMX Overview• SDMX-RD Project overview• SDMX – how do I get there?• Demonstrations

• SDMX Converter• SDMX Reference Infrastructure (SDMX-RI)• SDMX-RI Mapping Assistant• SDMX-RI clients (PUSH and PULL Mode)

• Questions & Answers

Eurostat

SDMX Tools – Summary• Eurostat tools Eurostat SDMX Info Space

https://webgate.ec.europa.eu/fpfis/mwikis/sdmx• SDMX Converter• SDMX Reference Infrastructure & Mapping Assistant• SDMX Registry & Data Structure Wizard

• Other tools availble in the communityhttp://www.sdmxtools.org• Fusion Weaver (validation, transformation, creation)• Fusion Transformer (command line conversion)• Fusion Registry• ECB Checker/Loader (conversion and validation)• FAO OpenSDMX project

Eurostat

Aim of the pilot / open questions• Feedback on data structures and coding

• For pilot: DSDs in Eurostat test registry• For production: Global Registry• Note: maintenance agencies not yet final

• Setup / configuration of national systems

• Support is given by international organisations

Eurostat

Questions?

20

• SDMX Support Team• ESTAT-Support-SDMX@ec.europa.eu

• SDMX Website• http://www.sdmx.org

• Eurostat SDMX Info Space• https://webgate.ec.europa.eu/fpfis/mwikis/sdmx