9 feb 2004mikko mäkinen & saija ylönen joint unece/eurostat/oecd work session on statistical...

9 Feb 2004Mikko Mäkinen & Saija Ylönen

Joint UNECE/Eurostat/OECD work session on statistical metadata (METIS) Geneva, 9-11 February 2004, Topic (ii): Metadata interchange / WP.6

MetaAPI – improving the usability of

existing metadata systems by application programming interface

9 Feb 2004 2Mikko Mäkinen & Saija Ylönen

Statistics Finland’s metadata systemsCurrent metadata systems of Statistics Finland

Unified File Description System

File 1 metadata

File 2 metadata ...

DataFile x

File xmetadata...

Field/variable x.1 metadata

Field/variable x.2 metadata ...

Field/variable x.n metadata ...

Classification Database

Concepts Database

Classification 1

Classification 2 ...

Classifi-cation ymetadata...

Item y.1 metadata

Item y.2 metadata ...

Item y.n metadata

Statistical concept 1

Statistical concept 2 ...

Statistical concept z...

Other metadata systems(System Register, Archive

System, etc.)

Database APIs

Applications


Classification database (main menu screen)

Classifications

Correspondence tables

Definitions

Key words


Classification screen

Code

Label

Metadata

Metadata window


Classification versions in the database

Classification version Text version


Problems with the present metadata systems

User interfaces are somewhat outdated Technical and statistical metadata are intermingled There is no direct data connection to several file formats

Text descriptions are not sufficiently structured and the description power of plain text is not adequate for mathematical formulas, figures, highlighting, etc.

Training in application development does not yet have any practical connection to metadata systems


Production model project Improves the production processes in our office Covers collecting, storing, processing and disseminating of data Utilises modern technologies, tools and IT methods

Data warehousing, software architectures, XML, web services, application development tools etc.

Different kinds of metadata is needed in all phases of production

We had no time (or knowledge) to find the perfect and final metadata solutions

We had no time to rewrite everything we already had A flexible, software independent solution with added change tolerance was needed to start with


MetaAPI structure

Classification db

Concepts db

ClassificationAPI

UFSAPIUnified File System

Statistical prod. app 1ConceptsAPI

MetaAPISFMeta

Statistical prod. app 2

Statistical prod. app ...N

Archive SystemArchiveAPI

SAS

SuperStar

Application X

MetaData

MetaData

MetaData

MetaData

MetaAPI = Metadata Application Programming Interface


MetaAPI services

At the moment we have some basic classification servicesFor getting codes, labels, definitions, key wordsRead only, internal only

Basic services to produce large result sets concerning a single metadata object (e.g. one classification, statistical concept or survey data with all metadata concerning it)

Schemas for the structure of objects are neededWith these few services we can access all stored metadata

Special services for more specialised common tasks (e.g. getting a label for a single code)

Using “sub schemas” of the above


MetaAPI services in the future

All classification services (including correspondence tables, etc.)

Concept services Services for the metadata of data files Services to create and maintain metadata Services for mixed tasks (e.g. producing metadata rich publication tables)

External services?


How does it work, what the user sees Example from our testing site: Getting classification codes and labels

someserveractivity


How does it work, what really happens Getting classification codes and labels

Data Access

MetaServices

EasyMeta

TKMeta

ClassificationAPI

ConceptAPI UFileAPI

Class.db

Conc.db

UFDdb

etc.

etc.

SOAP reques

t

SOAP respons

e

...


MetaAPI implementation

Server side.NET web service components (public interface).NET binary components (internal interfaces)Sybase and MS SQL Server databases to store metadata

Client side.NET, PowerBuilder, SAS applications XML based systems in dissemination BrowsersAnything (with support of using web services)


Change management We want to

… have the possibility to adopt new standards and common tools in the future without changing everything

… isolate the effects of system changes to a known area That is why

… we should not allow direct connections to metadata systems and databases

… all metadata business should take place through MetaAPI

However, the uncertainty of the remaining lifetime of the present systems has slightly slowed down the speed of creating new services into MetaAPI


Change management Example: all background systems are replaced

Some Common Standard Metadata Solution


SomeCommonStandardMetadataAPI

MetaAPISFMeta


Statistical prod. app ...N

SAS

SuperStar

Application XMetaData


Experiences so far

MetaAPI has now been used in one real production system and in some test cases

Experiences have been positive Web services are easy to use with modern application development tools

XML skills are needed after receiving the result sets Selected techniques seems to be efficient (but no benchmarking has been done)

Some client side optimisation may be necessary because XML documents can be large and parsing them can be time-consuming


What’s next?

More services Using MetaAPI in new projects Guidance of users, client side tools, formatting layers, etc.

Building connections to commercial software packages Data management decisions Adoption of common standards New user interfaces on top of MetaAPI, Metadata Portal External services

9 feb 2004mikko mäkinen & saija ylönen joint unece/eurostat/oecd work session on statistical...

Documents

metadata services

concept services services

basic classification

basic services

single metadata object

web services

different kinds of metadata

final metadata solutions