core essnet report @msis 2011

MSIS Meeting - Luxembourg May 23-25 2011 1

CORE ESSnet(COmmon Reference Environment)

progress report

Carlo VaccariIstat - Italy


Outline

Introduction and history

CORE objectives

CORE where we are

Architecture implementation

Information model

CORE and SDMX

CORE and GSIM


CORA ESSnet

Financed by Eurostat under 2009

Statistical Workprogramme

Countries involved: it (coordinator),

ch, dk, lv, nl, no, se

Duration: October 2009 - October

2010


CORA Technical

Architecture

CORA Model: two dimensions

Functional dimension

Construction dimension

Functional dimension

Adoption of GSBPM 4.0

9 subprocesses of level 2

1Specify Needs

7Disseminate

2Design

3Build

4Collect

5Process

6Analyse

8Archive

9Evaluate


Construction Dimension:

Layers

Figures

Time Series

Statistic

Population

Unit

Variable

Value

A domain of interest documented by statistical products

Statistical series over time

Integrated or simple statistical product for a given time

A statistical unit at a given time

A population at a given time

A statistical variable at a given time

A logical representation of the value of a variable


CORA Model Grid

Statistical processes compliant to CORA model are

intended to be designed by statisticians


After CORA…CORE!

COmmon Reference Environment (CORE),

financed by Eurostat under 2010

Statistical Workprogramme

Countries involved: it (coordinator), fr, nl,

no, pt, se

Duration: December 2010 - January 2012


CORE Principal

Outcomes

Environment for the definition and

execution of statistical processes

Definition of a process in terms of

services selected from an available

repository

Execution of the composed workflow


CORE Outcomes:

Design

CORA model → CORA information

model

Design of CORE services and

processes


CORE Outcomes:

Implementation

Selection of available middleware solutions for

process execution

Realization of an environment able to permit the

execution of processes:

Interfaces (GUIs) for defining CORE processes for

statistical users

Integration APIs

Repository of integration layers


CORE Outcomes:

Testing

Realization of processes starting from

services implementing some GSBPM phase

Evaluation of costs related to integration

Prototype implementation (to be

engineered)


CORE Architecture (1)

GUIs to support modelling of CORE processes

according to the CORA grid

Modeling & control flow constructs

Drag & drop facilities for process design

Global schema

Implementation: we are evaluating the usage of

an open process editor tool (Oryx -

http://bpt.hpi.uni-potsdam.de/Oryx/WebHome )


Process Runtime

Controlled execution of services

Implementation: integration of existing workflow solutions, currently in evaluation phase

Service Runtime

Integration APIs (in-out data transformation)

Service execution

Implementation: CSV and SQL data transformations are currently being implemented

Service Repository

Deployment of services

CORE Architecture (2)


CORE Information Model (1)

First draft of CORE information model

Del. 2.1 released: requirements for the

model of the interface through which

statistical services will communicate

Information Model to be released,

currently in discussion phase


Design Principles (in discussion):

Rectangular data sets (rows & columns)

Strong typing (data, rules, parameters)

Dataset kinds (eg micro/aggreg)

Free-style arguments (eg scripts tool

dependent)

Other (service arguments and infos)

CORE Information Model (2)


CORE and SDMX

Both initiatives foster standardization

CORE

Focus on standardization of processes and data

exchanges (mainly) intra-NSI

SDMX

Focus on standardization of processes and data

exchanges (mainly) inter-NSIs (or between NSIs

and international organizations)


CORE

Focus on all phases of statistical processes

Both micro and macro data considered

SDMX

Focus (mainly) on dissemination phase

Mainly macro data considered

CORE and SDMX - 2


Information Model

Both propose an information model

CORE information model

Takes explicitly process dimension into account

through GSBPM

Data dimension

SDMX information model

Mainly focused on data dimension


CORA Information

Model

Object

Figure Time series Statistic Population Variable

Constructorprescript

Service

n

1

n +implements

1

Layerlevel+contains

+belongs_to

Construct

Unit

Element

+has

+belongs_to

n

n

+inputn

+output n

1

n

+represented_by

1

noutput.belongs_to.level = input.belongs_to.level + 1


SDMX Data & Metadata Information

Model

Category Scheme

Data or Metadata Structure Definition

Category

can have child categories

comprised of subject or reporting categories

Data or Metadata

Flow

Data Provider

Provision Agreement

uses specific data or metadata structure

can be linked with categories from multiple category schemes

conforms to business rules of the data or metadata flow

can get data from multiple data providers

Data or Metadata

Set


On Information Models

Different abstraction levels

CORE

“Higher” modelling level

E.g.: statistics as tabular data

SDMX

“Lower” modeling level

E.g.: aggregated data set with dimensions, attributes

and measures


Open Issues - 1

Can we use SDMX for micro and macro

data exchanges in a CORE process?

Need for mapping of information

models


On-Going Work - 1

CORE implementation scenario within

Istat

Main phases: Sample selection and

allocation

CORE wrapping of available SAS and

R procedures


Design and implementation of CORE

Integration APIs

Possible in/out SDMX translations

IAPIIAPI TOOLSDMX

CORE TOOL

SDMX

On-Going Work - 2


Open Issues - 2

What about metadata?

CORE: Data and metadata managed in the

same way

SDMX:

Distinction between structural metadata and

reference metadata

Dedicated effort for metadata management


Collaboration between

CORE/SDMX ESSnets

CORE planned deliverable on “Feedbacks

on SDMX Usage in CORE”

Periodical meetings inside Istat between

coordinators of the two ESSnets

Exchanges of resources between the two

ESSnets


CORE and GSIM

GSIM: Generic Statistical Information Model

deliverable from OCMIMF Operationalising a

Common Metadata/Information Management

Framework activity inside Statistical Network

Ambiguity on the acronym: reference to

“generic statistical information model” in CORE

ESSnet proposal

In March started activity to clarify relationships

(thanks to J.P.Kent and A.Hamilton)


First analysis and discussions: the deliverables

from the two initiatives are complementary in

intent and do not overlap in concept

Necessary to avoid gaps and/or duplications

and ensure the complementary relationship

CORE and GSIM


CORE Information

ModelCORE will define a very generic information model

(CORE-IM) for the interface through which statistical

services will communicate with each other within

the framework of the CORA model

As a communication protocol, CORE-IM focuses on

the “postal envelope” used when passing

information between services, rather than focusing

in detail on the information being communicated (ie

what is inside)


CORE-IM current hypothesis: to support a flag to

indicate if the information being communicated is

described within GSIM

→ without claiming to align the semantics of the

content (eg, “classification”), but only to alert a

consuming service which “understands” GSIM that

it can relate the content to GSIM

CORE and GSIM


Complementary nature

CORE-IM supporting semantic interoperability at a very high, abstract level (“here is an information object, along with the ‘envelope’ information about it”) where GSIM can provide greater semantic precision to a subset of information objects communicated using CORE

CORE supporting communication between services

→ substantial interoperability benefits

Information aligned with GSIM semantics

→ further level of interoperability


GSBPM reference model for statistical business processes

GSIM reference model for information input to, used by and produced by those processes

Models are independent → it's possible to use one without the other

CORE-IM recognizes and uses GSBPM and (hopefully) will do the same with regard to GSIM, giving them a potential contact point

Complementary nature


Coordination in

practiceNeed to maximize the extent to which these synergies are achieved in practice: Members in common (no, se) ABS leader for the OCMIMF and observer in CORE CORE members external reviewers for GSIM material CORE WP2 “co-ordination input” from the OCMIMF collaboration team in regard to deliverables Half day session at METIS workshop (October) presenting CORE and OCMIMF works to external metadata specialists Common documents in preparation ...

core essnet report @msis 2011

Technology

sdmx core

core architecture

core outcomes

infosmsis meeting luxembourg

gsimmsis meeting luxembourg

history core objectives

modelling of core processes

standardization core