seadatanet

NATO ARW September 2007

SeaDataNetA Pan-European Infrastructure

for Ocean and Marine Data Management

by Dick M.A. Schaap (MARIS)

Technical coordinator


Background

• Data availability of vital importance for marine research

• European data set is fragmented

• Not always quality controlled

• Not always easily accessible

• In 35 countries bordering European seas• more than 1.000 scientific laboratories from government

organizations and private industry collect data

• use various sensors on board of research vessels, submarines, fixed and drifting platforms, airplanes and satellites

• measure physical, geophysical, geological, biological and chemical parameters, biological species, etc.


What is SeaDataNet?

• EU Sixth Framework Programme • Integrated Infrastructure Initiative (I3)• 1 April 2006 to 31 March 2011• Objectives

• develop an efficient distributed pan-European marine data management Infrastructure for managing large and diverse data sets

• network existing professional data centres of 35 countries, active in data collection

• provide integrated databases of standardised quality on-line


SeaDataNet partners

49 partners including:• Major oceanographic

institutes of 35 countries

• National Oceanographic Data Centres (NODCs)

• Divisions of major national marine research institutes

• 3 satellite data centres

• 3 scientific modelling centres

• 2 SMEs experts in software development

• 3 International organisations (IOC, ICES,

JRC)


Building on experience

• Leaders of Data Management Concerted Actions• MEDAR/MEDATLAS II - Mediterranean Data Archaeology

and Rescue, aiming to make available a comprehensive data product of temperature, salinity and bio-chemical data in the Mediterranean and Black Sea through a wide international cooperation

• Sea-Search – The gateway to Oceanographic and Marine Data & Information in Europe, to , to provide an effective navigation tool to oceanographic and marine data and information sources for European marine waters and to centres in Europe with expertise and skills in oceanographic and marine data & information management.

• Data management of scientific projects:• MTPII-MATER, CANIGO, OMEX, MERSEA, etc.

• Bringing in new partners – satellite data centres


SeaDataNet components


Infrastructure, versions & planning

• Version 0Continuation, maintenance and further population of existing Sea-Search systems with minor modifications

• Version 1All data centres involved in upgraded meta-databases (ISO compliant XML metadata-profiles and web services)

Technical Task Team partners involved in transparent data access, security, and monitoring of use = trans-national access via downloading of datasets in standard formats (ODV ASCII; NetCDF; MedAtlas)

Launch by March 2008

• Version 2All data centres involved in trans-national access and upgrading of data access to include spatial data infrastructure features (adopting OGC; ISO and INSPIRE standards) and using XML data profiles

Operational by 2010


SeaDataNet Data Policy

• Metadata• free and open access, no registration required• each data centre should provide the meta-data in standardized

format to populate the discovery services (metadata directories)

• Data and products services • in general free and open access, but mandatory registration• a “SeaDataNet role” (partner, academic, commercial etc.) is

attributed to each user • Each NODC attributes the roles to the users of its of country• Out of the partnership, the roles are assigned by SeaDataNet user-

desk

• each data centre node delivers data according to the user’s role and its local regulation

• each data centre should provide freely the data sets necessary to develop the common products


Discovery services

• Marine environmental data sets (EDMED)

• Marine environmental research projects (EDMERP)

• Observing systems (EDIOS)• Research Cruise Summary

Reports (CSR) • Common Data Index (CDI)• Directory of marine

organisations (EDMO)


Cruise Summary Reports

• Conceived by IOC• ICES acted as focal point • Currently over than 37000

cruises• Documenting data

collected on scientific research cruises operated by European vessels

• Cruise track charts included

• Online entry form• Online searchable

database


European Directory of the Ocean-observing System (EDIOS)

• Initiative of the European Global Ocean Observing System (EuroGOOS)

• Regular repeated measurements

• On-line searchable directory of observing, measuring, and monitoring systems

• Over 10,000 observation entries

• Links to on-line real-time and archive data

• Tool to aid design of observing systems


European Directory of Marine Organisations (EDMO)

• Over 1.000 organisations: addresses, profiles and relations in other directories

• Online maintenance per country by CMS

• On-line user interface with combined alphanumeric – geographical searching

• Export in .csv files• Also available as Web

service (SOAP based)


Common Data Index

• Index to individual data sets

• Now >250000 entries from 20 partners

• Mid Oct 2007 entries from 35 countries

• ISO19115 metadata standard has been adopted

• On-line searchable directory

• Direct access to on-line data sets, on-line order facility or e-mail

request


Technical developments

• Streamlining of metadata formats for interrelationships between the directories

• Establishment and governance of common vocabularies• ISO 19115 content model as basis for the XML formats and

exchange schemas (XSD) with SeaDataNet XML approach• Innovating maintenance modalities:

• XML exchange using local software component (MIKADO) or own application

• XML validation services• Online Content Management System (CMS)

• Introducing Web services• Vocabularies and Directory of Marine Organisations (EDMO)• Coming: other metadata directories


Interrelationships between the directories, using vocabularies

Country

Param

Platf

…

…

…EDMED

EDIOS

EDMERP

CDI

CSR

EDMO

XML


Common Vocabularies

• Controlled vocabularies containing keys, terms, abbreviations and essential (but often missing) definitions

• Lists for platforms, countries, sea areas, etc…

• International content governance by SeaVox list server

• Web service • On-line user client• Export in .csv files


SeaDataNet standards

• Interoperability – key to distributed data management system success• Common understanding of terms used via common

vocabularies• Standard data exchange formats (ODV ASCII,

NetCDF, MedAtlas)• Common quality control protocols and flag scale• Standard Web services • Schemes for handling duplicate data• Ocean Data View software (ODV) adapted for

SeaDataNet


SeaDataNet : Delivery Services V1

Primary function• Users will be able to

download selected data sets

CDI User Interface• Searching and

browsing of metadata of data sets



Shopping basket• Preparing a user request

of multiple data sets Central User Register• Validation of users and

their rolesBusiness matrix• Combining user role and

data access restriction(s) to decide on modality for processing each data request

A B C D

U

V

W


Central User Register

• Single Sign On system required for access to distributed system

• User’s authentication information based on personal login / password

• User must register in order to get login• Web form to provide necessary information • Online user agreement “SeaDataNet General Licence”• Registration processed per country via online CMS • After processing, login/password sent by e-mail (e-mail check)• User gets ‘Role’

• SeaDataNet Data Policy • Data access based upon combination of User role -

Access restriction(s) as indicated in CDI record and business logic



Request Status Manager• Processing of all requests

and data deliveries • Enables users to follow

progress• Enables DC’s to oversee

transactionsDownload manager• Local component for

communication with portal and preparing

data deliveries


SeaDataNet User’s portal schema V1

IFREMER Database

BODC Database

BSH Database

...

Download managers in data centres

CSR EDIOS

EDMED

CDI

Shopping basket

Requestsstatus

manager

datametadata

CSR

CDI

Project info

Metadata & Data catalogues

Data request Status of request

General request Metadata request

Organisation +data source id

Data download

at BODCat BSH at MARIS+ EDMERP

services for metadata

Data downloading services

Visualisation services (WMS) for V2

Entry point for access hits

EDMO

EDIOS EDMED

EDMERP EDMO

Software

Vocabularies

Standards Cross search

AAA

Registr.

My transact.

User Register

Userregistration


Data products

• SeaDataNet will provided added-value products to be created by regional centres• Mediterranean Sea• Black Sea• Baltic Sea• Arctic and North Sea• Atlantic and global ocean

• First products => gridded climatologies • based on already available data-sets (mostly hydrographic data)• updated during the project life

• To make products comparable, all regional data centers will use a common tool for spatial mapping

• For this a new version of DIVA (Data Inverse Variational Analysis) was developed (University of Liege)


SeaDataNet wider involvement

• SeaDataNet partners are involved in many other European and international projects, contributing and safeguarding good data management. Examples:

• Services for the European marine geological institutes

• EDIOS Services for EuroGOOS

• Partners in the MERSEA project (GMES) for MIM

• Partners in the HUMBOLDT project (GMES) for developing a framework for a European Spatial Data Infrastructure

• Partners in SIMORC for oil & gas industry metocean data

• Partners in ECOOP for data provision and management

• Component of the GMES Marine Core Services


Marine geology data

Catalogue of > 300.000 seafloor samples and >2.5 million line kilometres of marine seismic and sonar survey from the ocean basins and continental shelves, held at European institutions

www.eu-seased.net

http://www.eu-seased.net/


Industry metocean datasets - SIMORC

• System of Industry Metocean data for the Offshore and Research Communities• Metadatabase and Database of datasets from major oil & gas companies

www.simorc.orgAt present ca 870 data sets (Shell, BP, Total), more

underway

http://www.simorc.org/


SeaDataNet - BlackSeaScene

• BSS is an Integrating Activity of EU-Research Infrastructures programme and runs from 1 Dec 2005 – 30 Nov 2008

• BSS has 32 partners with 25 from Black Sea region and is coordinated by MARIS

• BSS is a Black Sea Scientific Network• analysing the present quality of Black Sea datasets and applied DQC procedures

• developing a Black Sea regional portal for metadata and data access

• BSS is adopting the SeaDataNet standards and building upon the SeaDataNet basic services

• BSS engages 25 Black Sea institutes, strengthening and expanding the compilation of metadata and access to data for the Black Sea region


SeaDataNet capacity building

The training program has two main objectives:

•to ensure that the data and metadata to be integrated in the SeaDataNet system will be formatted, checked for quality and disseminated according to the common protocols developed

•to transfer expertise and to train IT experts of the data centers in the basics, installation and operation of the SeaDataNet technical components


International Conference IMDIS 2008

• Second international conference “International Marine Data Information Systems” (IMDIS)

• March 31 - April 2, 2008, Athens, Greece • Joint organisers: SeaDataNet, Hellenic Centre for

Marine Research, IOC/IODE • Four sessions:

• Marine environmental data bases: infrastructures and data access systems

• Data Services in ocean science: standards and interoperability

• Visualisation tools• Services for Users and Education


www.seadatanet.org

seadatanet

Documents

transparent data access

biochemical data

mediterranean data archaeology

xml data profiles operational

european marine waters

open access

transnational access

scientific modelling