synthesis of the technical activity and state of development of the on-line system v1 by dick m.a....

Synthesis of the technical activity and State of development of the on-line system V1

ByDick M.A. Schaap – technical coordinator

Madrid, March 09

SeaDataNet Participating Countries

49 Partners = 40 Data Centres + ICES + IOC-IODE + JRC + specialists in statistics, products and informatics

SeaDataNet Infrastructure objectives

SeaDataNet aims to set up and operate an efficient Pan-European distributed infrastructure for managing marine and ocean data by connecting:

40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europethese Data Centres are mostly divisions of major national marine research institutes and based in 35 countries, surrounding the European seas

SeaDataNet aims to ensure a comparable quality of data sets and to make data sets easily accessible on-line through a unique portal, while the data sets are stored and managed at the Data Centres.

SeaDataNet Infrastructure versions

Version 0 = Continuation and maintenance of existing Sea-Search systems and population of the Common Data Index (CDI) V0 system

Version 1 = Harmonised and upgraded metadatabases adopting ISO 19115 standard; Transparent data access involving all Data Centres via implementation of Common Data Index (CDI) V1 system and downloading services.

Version 2 = Adding OGC viewing services and further virtualisation of data access.

SeaDataNet Technical Task Team

Oversees and coordinates the technical developments

Includes 11 core technical partners

3 Meetings in the third year of SeaDataNet:Paris, France 13-14 May 2008 Hamburg, Germany 20-21 November 2008Madrid, Spain, 24-25 March 2009

Bilateral meeting RIHMI – MARIS concerning DM - RSM:Obninsk, Russia, 18-20 August 2008

SeaDataNet Technical Task Team

Methodology: Agenda with relevant topics, leading speakers, open discussion and brainstorming, possible subgroups for further explorations and analyses, resulting in technical specifications of system architecture and its modules, followed by implementation, evaluation and operation

All presentations and documents are included in the SeaDataNet Extranet in the TTT section. Note: All Project Partners are encouraged to visit this regularly.

SeaDataNet User’s portal schema V1

services for metadata

Data downloading services

Visualization services (WMS) for V2

Ifremer Database

BODC Database

BSH Database

...

Download managers in Data centers

CSR EDIOS

EDMED

CDI

Shopping basket

Requestsstatus

manager

datametadata

CSR

CDI

Project info

Metadata & Data catalogues

Data request Status of request

General request Metadata request

Organisation +data source id

Data download

at BODCat BSH at MARIS+ EDMERP

Entry point for access hits

EDMO

EDIOS EDMED

EDMERP EDMO

Software

Vocabularies

Standards Cross search

AAA

Registr.

My transact.

User Register

Userregistration

SeaDataNet Interoperability

Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via:Using common quality control protocols and flag scale Using common and controlled vocabularies, including international content governanceAdopting the ISO 19115 metadata standard for all metadata directoriesProviding XML Validation Services to quality control the metadata maintenance Providing standard metadata entry toolsUsing harmonised Data Transport Formats (NetCDF, ODV ASCII and MedAtlas ASCII) for data sets delivery Adopting of OGC standards for mapping and viewing servicesUsing SOAP Web Services in the SeaDataNet architectureOrganising training and instruction workshops for transfer of expertise and hands-on training

SeaDataNet Quality Control Guideline

A guideline (V1) of recommended QC procedures, reviewing NODC schemes and other known schemes (e.g. WGMDM guidelines, World Ocean Database, GTSPP, Argo, WOCE, QARTOD, ESEAS,SIMORC, etc.).

QC methods for CTD (temperature and salinity), Current meter data (including ADCP), Wave data, Sea level dataA scheme of QC flags to be used in SeaDataNet.These flags are for assigning to individual data values. They are not for allocating to whole data series, or to accompanying information.

Compiled in discussion with IOC, ICES and JCOM, to ensure an international acceptance and tuning. Important feedback from the joint IODE/JCOMM Forum on Oceanographic Data Management and Exchange Standards (January 2008), joined by SeaDataNet and international experts to consider on-going work on standards and to seek harmonisation, where possible.

Now extending the guideline with QC methods for surface underway data, nutrients, geophysical data, and biological data. V2: June 2009.

SeaDataNet Common Vocabularies

Use of common vocabularies in all metadatabases and data formats is an important prerequisite towards consistency and interoperability.

Set-up and population of Common Vocabularies. The SeaDataNet Vocabulary service is based upon the NERC DataGrid (NDG) vocabulary Web service. For end-users there is a vocabulary Client Interface for searching and browsing and to export selected entries in csv format.

The Web service is compliant to WS Basis Profile 1.1, which is adopted as standard for all Web services in SeaDataNet.

Content governance of the vocabularies is very important and is done by a combined SeaDataNet and MarineXML Vocabulary Content Governance Group (SeaVoX), moderated by BODC, and including experts from SeaDataNet, MMI, MOTIIVE, JCOMMOPS and more international groups. SeaVox operates by mailing list server.

SeaDataNet Common Vocabularies

Upgraded interface in SeaDataNet styleWeb services upgraded to be compliant to WS Basic Profile 1.1Expanding the number of listsWeb services available via BODC and user client via MARIS

SeaDataNet infrastructure V1

Discovery services EDMED - Data SetsEDMERP – Research ProjectsCSR – Cruise Summary Reports EDIOS – Monitoring systemsEDMO – Marine OrganisationsCDI – Common Data Index

SeaDataNet Discovery services

TTT Activities undertaken for:

Reviewing and streamlining the logical formats of each of the Directories

Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services

Defining XML schema’s and formats, using the ISO 19115 metadata standard as basis

Defining and developing maintenance modalities for each of the Directories

SeaDataNet Discovery services

TTT Activities undertaken for:

Defining and developing new User Interfaces for each of the Directories

Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories

Developing Web services for the Directories

Vocabs

EDMO

…

…

…

…

CDIEDMERP

CSR

EDMED

EDIOS

SeaDataNet : Formats review

Review of the format and use of common vocabularies for each of the Directories to achieve harmonisation and integration, and paving the path to data access via the CDI

Data Access

…

SeaDataNet XML Schema’s

The ISO 19115 content model is the basis for the XML formats and exchange schema’s (XSD).

Guidance document has been prepared on how to use XML for SeaDataNet, including how to declare references to Common Vocabularies, EDMO and EDMERP

For each Directory has been prepared:Description of the format and XML tagsXML SchemaXML example file

These documents are available at the SeaDataNet

Extended XML Schema’s have been prepared, using Schematron and OCL to support the checking of mandatory fields, use of codes from the Common Vocabularies and use of organization codes from EDMO. These schema’s are used in the XML Validation Web services

SeaDataNet Maintenance modalities

• Maintenance: Depending on the Directory, the following maintenance modalities are provided:

Online maintenance via online Content Management System (incl master editing options)

XML export from local system

Local XML export can be produced by partners via:own softwareusing upgraded version of MIKADO software, that also will replace the present MS Access tools for EDMED and EDMERPMIKADO can also be used for bulk production of CDI XML files from the partner’s database(s)


Directory Online CMS by partners

XML exchange via use of MIKADO atpartners

XML exchange via local systemat partners

Online CMS bymoderator

XML Validationservice

EDMED(BODC =

authority)

SDN DC’s initial contentconversion freetext => vocabsvia ‘sandbox’

Stand-alone incl. local storage +QC – loop for contentSynchronisation

Generated from in-house system

Validation ofXML syntaxand use ofVocabs, EDMOand EDMERP

EDMERP(MARIS =

authority)

SDN DC’s manage theirnational records +‘sandbox’ forinstitutes

Stand-alone incl. local storage



CSR(BSH =

authority)

Online entries by chief scientists

Stand-alone incl.local storage

Generatedfrom in-housesystem



Directory Online CMS by partners

XML exchange via use of MIKADO at partners

XML exchange via local system at partners

Online CMS by moderator

XML Validation service

CDI(MARIS = authority)

Tool embedded inlocal system



EDIOS(BODC = authority)

SDN DC’s initial revision + new entries via ‘sandbox’

Stand-alone incl. local storage


Validation of XML syntax and use of Vocabs, EDMO and EDMERP

EDMO(MARIS = authority)

SDN DC’s manage their national records

Vocabularies(BODC = authority)

BODC with SeaVox governance mailing list

User Register + AAA service(IFREMER = authority)

SDN DC’s manage their national records

SeaDataNet Content Management Systems

EDMERP

CSREDMED

EDMO

SeaDataNet MIKADO Java tool

Available under multiple environments :Microsoft : Windows 2000, XP, VISTAAPPLEUnix - SolarisLinux

MIKADOJava code

Native DriversMYSQL

ORACLEPOSTGRESSQLServer

Bridge Driversusing Microsoft ODBC

(ACCESS, EXCEL, SQL SERVER)

DATABASE

JDBCJava DataBase Connectivity

EXCEL File

XML filesfor SeaDataNet

catalogues

CSREDMEDEDMERPCDI[EDIOS]

Manual

Automatic

Other DriversDownloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)

SeaDataNet new User Interfaces

EDMERP

CSR

EDIOS EDMO

SeaDataNet Discovery services - statusReviewing and streamlining the logical formats of each of the Directories – READY

Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services – READY

Defining XML schema’s and formats, using the ISO 19115 metadata standard as basis – READY (EDIOS READY SOON)

Defining and developing maintenance modalities for each of the Directories:

EDMO, EDMERP, and CSR – READY VIA ONLINE CMSEDMED upgrading V0 to V1 – READY VIA ONLINE CMS EDMED, EDMERP, CSR and CDI – READY VIA MIKADO V1.5Common Vocabularies and AAA - READY VIA MASTER CMSIMPORT OF XML FOR EDMERP, CSR, EDMED, CDI - READYEDIOS – TO EXTEND MIKADO WITH THE NEW XML

Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories – READY FOR CDI, EDMED, EDMERP and CSR ; EDIOS SOON

SeaDataNet Discovery services - status

Defining and implementing new User Interfaces READY FOR EDMO, CDI, EDMERP, CSR, EDIOS, Common Vocabularies and AAA servicesUNDER DEVELOPMENT FOR EDMED

Content upgrading from V0 to V1 – CONTINUED INPUT AND SUPPORT NEEDED FROM THE NODC’s !!

EDMO READY EDMERP ALMOST READY AND V1 LAUNCHEDCSR WELL UNDERWAY AND V1 LAUNCHED, BUT NEEDS MORE INPUT BY NODC’S EDMED ALMOST READY, SOME PARTNERS ARE BEHIND SCHEDULE EDIOS – SO FAR ONLY DONE FOR UNITED KINGDOM AND BLACK SEA AS PILOTS => OTHER PARTNERS TO START SOON ONCE EXTENDED MIKADO AVAILABLE (END 2009) – COOPERATION WITH EUROGOOS REGIONS!

Note: CDI upgrade is undertaken as part of the Data Access activities;

SeaDataNet Discovery services - status

Implementing Web services READY FOR EDMO, EDMERP, Common Vocabularies and AAA servicesLATER ALSO PLANNED FOR EDMED, CSR, CDI and EDIOS

SeaDataNet : Training and instruction

Training Workshop has been held at IODE in Oostende – Belgium at from 16 to 19th June 2008 for all SeaDataNet data managers:

To train partners in the use of the new V1 formats, interfaces and maintenance tools (MIKADO, online CMS, Web services, Validation services, Vocabularies)

To train partners in performing data quality control and assessments, using ODV software

To train partners in analysing and making data presentations, using ODV – DIVA software

To present and explain the new CDI V1 – data access procedure and system

SeaDataNet Data Access Services

SeaDataNet aims to make data sets easily accessible on-line through a unique portal, connecting 40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europe

SeaDataNet Data Access components

Common Data Index (CDI) system including a Shopping and Transaction mechanism Using harmonised Data Transport Formats for data sets delivery SeaDataNet Data Policy and User LicenceAuthorisation and Authentication Services, based upon a Central User Register and vocabulary of Data Access Restrictions

Authentication, Authorization and AdministrationMetadata systems and website with standards and tools are public domain

Single Sign On system required for access to distributed databases

User must register once in order to get a personal login - passwordWeb form to provide necessary information User agrees with “SeaDataNet User Licence”After processing, login/password sent by email (email check)

Licence is part of the SeaDataNet Data Policy, that is intended to be fully compatible with the Directive of the European Parliament and of the Council on public access to environmental information, the INSPIRE Directive, IOC, ICES, WMO, GCOS, GEOSS and CLIVAR data principles.

Choice in V1 for CAS system (= Centralized) to provide AAA Web services

NODCs intermediate in the registration of their national users to the SeaDataNet central user register

SeaDataNet AAA service

Interface for central log-in

SeaDataNet Common Data Transport Formats

V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA)

The following formats have been defined:SeaDataNet ODV4 ASCII for profiles, time series and trajectoriesSeaDataNet MedAtlas as optional extra format.NetCDF with CF compliance for gridded data sets

ODV4 and MedAtlas have been outfitted with a SeaDataNet semantic header

International cooperation is underway from SeaDataNet with the CF community and UNIDATA for a common NetCDF format (Core Data Model – CDM) for the oceanographic and meteorological domains, including a semantic header

SeaDataNet : Conversion Tools

• NEMO tool ready for conversion of ASCII files to SeaDataNet ODV and MedATLAS

• Med2MedSDN V1.0 ready for conversion of MEDATLAS files without the SeaDataNet extensions to MEDATLAS files with SeaDataNet extensions (SeaDataNet MEDATLAS)

• Transformation Web Service for lat – lon coordinates to Marsden Squares and vice versa

CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). This is in contrast to EDMED, which is an inventory of data sets, each of which comprises a large number of measurements.

CDI XML Schema is based upon the ISO 19115 standard

CDI V0 = Central and homogeneous metadatabase, but for Data Access users are confronted with the different user interfaces of the SeaDataNet Data Centres (which can provide online search and access / ordering facility / e-mail request)

CDI V0 coverage: > 340.000 CDI entries from 36 Data Centres from 29 countries

SeaDataNet Common Data Index (CDI)

SeaDataNet CDI V0 – Data coverage – points

SeaDataNet CDI V0 – Data Access example

RIHMI-WDC – Russia

Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal and downloading services

SeaDataNet Delivery Services – CDI V1

SeaDataNet CDI V1 system components

CDI V1 Metadatabase and User Interface:Searching and browsing of metadata of data sets, managed by SeaDataNet Data Centres

Central User Register - Web service:Details of users, their organizations and addresses, license period, Id-Passwords, SeaDataNet Roles

Shopping Basket – User Interface:Preparing a user request of multiple data sets, handling validation of users and roles via the Central User Register, and routing requests to the Request Status Manager

Request Status Manager – Web service + User Interface:

Processing and administration of all requests and data deliveries (downloads), for users in communication with data centres. For data centres to oversee all transactions.

SeaDataNet CDI V1 system components

Data Centre Download Manager – Java Tool:Handles communication with Request Status Manager and takes care that requested files are made ready for downloading by User (if OK) in the agreed formats via a local website address at the Data Centre

SeaDataNet Download Manager V1.1 FunctionsCommunication with RSM Web service about process progressProcessing order lists => retrieving requested data filesRetrieving data sets from:

file management systemsdatabase systems (e.g. Oracle, MySQL, MS-SQL, Sybase, ….)

If data sets are retrieved from a database => export of data files in SDN ODV4 formatNote: Data sets from a file management system must be pre-processed in SDN transport formats (ODV4 and optional MedATLAS for ASCII type data, or NetCDF (CF) for grid type data)

Combining data set files in daily zip files in a user download directoryGenerating dynamic user download webpage, following SDN template, to give user overview and option of downloadable zip files

SeaDataNet CDI V1 – Data Access example

Search

Results

Include inBasket

Shopping list

Submit + Authentication

RequestConfirmed

Data

Check StatusIn RSM

Ready at DC x

Download

SDN format

SeaDataNet CDI V1 - status

CDI V1 database, interface & transaction system - READY;

Mikado software tool for CDI V1 generation + Validation service – READY

Download Manager (DM) V1.1d – READY

Full system tested and operational – READY

Alternative without DM = V1 Interim - READY

22 Data Centres well underway with installing the DM V1.1 and configuring their infrastructure for being a full V1 data centre. So far 9 Data Centres fully operational with > 380.000 datasets accessable

Other Data Centres are requested to start implementing CDI V1 soon.

SeaDataNet CDI V1 – Quick Search interface

Using an intuitive approach which is easy for users

SeaDataNet CDI V1 – Quick Search interface

With Show / Hide Map option for display of search results

SeaDataNet CDI V1 – Extended Search interface

More search capabilities, but more complicated for first-time users

SeaDataNet CDI V1 shopping principle

SeaDataNet CDI V1 – Some search results

SeaDataNet CDI V1 – Shopping Basket

SeaDataNet CDI V1 – Request Status Manager (RSM) Log-in page

SeaDataNet CDI V1 – RSM: logged in as User

Overview of requests per Data Centre and Status


Status = Ready for Download + Download option


User personal Download webpage at Data Centre

SeaDataNet CDI V1 – Monitoring and Tracking

• All submitted requests, their users, the data sets concerned, and processing of these requests is administered in the Request Status Manager

• SeaDataNet Data centres can also log on to the Request Status Manager and oversee all requests, handle ‘tbd’ requests and prepare a number of online reports.

SeaDataNet CDI V1 – RSM: logged in as data centre

Data centre RSM: overview Standing Orders

SeaDataNet CDI V1 – further implementation

22 Data Centres well underway with installing the DM V1.1 and configuring their infrastructure for being a full V1 data centre. So far 9 Data Centres fully operational with > 380.000 datasets accessable

Other Data Centres are requested to start implementing CDI V1 soon.

Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO softwareInstalling the Download Manager softwareChoosing between pre-processed data files OR conversion via the Download ManagerConverting data files to SeaDataNet formats, in case of choice for pre-processed mode

SeaDataNet CDI V1 interim as temporary alternative

All Data Centres must populate the CDI V1 database:Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software

V1 interim = no installation of Download Manager, but data centres will be informed by the Request Status Manager by e-mail of all requests

Data centres have to process all requests manually using the Request Status Manager

Delivering data sets in SeaDataNet Formats (conversion via new NEMO tool) – pre-processing required.

Effect is, that users will experience a common approach to all data centres. The difference is behind the scenes.

Thereafter centres should move to full V1 status by installing DM

SeaDataNet V1 presentation and analysis tools

Ocean Data View (ODV)General data analysis and visualization software>10,000 registered users

Upgrading ODV software into ODV4 for:Extending ODV‘s graphical display capabilities and interactive controls for automatic and visual quality control and data quality flaggingSeamless connection to SeaDataNet output: SDN ODV4 data formats, quality flag scaleIntegration of DIVA gridding softwareImportant tool for the SeaDataNet regional Data Products

DIVA software (Data-Interpolating Variational Analysis) allows to spatially interpolate observations on a regular grid in an optimal way. The analysis is performed on a finite element grid allowing for a spatial variable resolution and a good representation of the coastline and isobaths.

SeaDataNet V1 presentation and analysis tools

SeaDataNet Website Upgraded

The SeaDataNet Website has recently been upgraded and users can find all presented services including background information.

www.seadatanet.org

Thank you

synthesis of the technical activity and state of development of the on-line system v1 by dick m.a....

Documents