1CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
SIPAD New Generation:
Dominique HeuletCNES
18, Avenue E.Belin31401 Toulouse Cedex 9
Francee-mail:[email protected]
A multi-field system for accessing scientific data and added-value services
2CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Introduction
SIPAD-NG– What it is:
» a complete operational system for preserving and accessing data,
» implements proven off-the-shelf products and a sophisticated data model,
» meets the requirements of a data center with a long-term mission,
» a centralized system with its own catalogue
– Why it has been developed:
» CNES wants to offer a complete generic software to be used as a basis for building data centers
3CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
SIPAD-NG: follow-on of the SIPAD softwareThe SIPAD system
– Developed in the 1990s for the CDPP» CDPP: Space Plasma Physics Data Centre
» SIPAD enables users to retrieve data and metadata through a web server
– SIPAD software designed to be used by any scientific discipline:» Plasma Physics (CDPP & CASSINI projects),
» Oceanography (MERCATOR project),
» Atmospheric Chemistry & Astronomy
– Advantages of SIPAD:» SIPAD implements all the basic functions of a data access system
» Open-ended Data Model based on a Data Dictionary allowing the control of delivered metadata
– Disadvantages:» Limitations to implement new functions (e.g. interoperability),
» Web user interface not flexible enough (not easily customizable)
4CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
The Data Model: graphs
For each project, definition of the Data Model
First goal of the Data Model: organize the catalogue in order to make data search & selection by navigation easy
– Definition of a data graph, a browse graph and a document graph
» Data graph: storage objects (archived data – e.g. a ‘tar’ file –)data objects (data files seen by end users)data sets (groups of data objects – e.g. instrument level 1
data –)collections (groups of data sets or collections)
» Browse graph: browse products, browse sets, browse collections
» Document graph: documents, document collections
– Definition of links between these graphs
Web server pages are automatically generated from the Data Model
5CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
The Data Model: graphs
Project
Documentcollection
Datacollection
Browsecollection
Data setData
ObjectBrowse set
Browseproduct
Document DocumentData set
Data object Data objectBrowseproduct
Browseproduct
Datacollection
Data setBrowse
collectionBrowse set
Documentcollection
Storageobject
Storageobject
Storageobject
6CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
The Data Model: attributes
Elements of the graphs can be described by attributes
Generic attributes» Size of a data object
» Number of data objects in a data set
Specific attributes according to scientific discipline» Plasma Physics: start time & stop time of an observation
» Earth Observation: geographical coordinates
Use of attributes– To present information associated to data
– To select data by criteria
» The web server provides selection pages for entering criteria values
7CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
The Data Model: attributes
Project
Missiondocuments
Mission
DocumentData set
Data object
ExperimentExperimentdocuments
Data Object Id
Start time
Stop timeVersion
etc …
Title
Author
Name
Description
Measured parameters
etc …
Selectioncriteria
Dataset Id
Level
etc …
Browsecollection
Browseproduct
Browse set
8CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
SIPAD & SIPAD-NG common functions
Data search & selection» Navigation in the Data Model,
» Search on criteria,
» Selection of product browse
User management» Access rights,
» Quotas
Order management» Data extraction from an archive,
» Media production,
» Network transfer,
» User workspace
9CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
New functions implemented in SIPAD-NG
Integration of Added Value Services (AVSs)– SIPAD-NG allows to:
» associate AVSs to Data Model elements in order to propose them to users,
» call AVSs by giving them parameters,
» manage access rights and user quotas
– SIPAD-NG implements a specific interface for:
» Batch AVSs: process data which have already been ordered and deliver the result to the user (e.g. coordinate transformation),
» Interactive AVSs: process data available in the user workspace (e.g. graphic representation of data from several
data sets)
Interoperability– SIPAD-NG provides interfaces to send requests to or to receive requests
from another Data Centre
» Web Services & SOAP protocol
10CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Adaptability
Adaptation of the web server to the project needs– SIPAD-NG web server easily customizable to project needs
– SIPAD-NG web server can be replaced by a project specific web server
Implementation in Data Centres located inside CNES– SIPAD-NG compliant with CNES Computer Centre services:
» SUN SOLARIS operating system,
» interfaces with CNES archiving and media production facilities
– SIPAD-NG compliant with CNES security rules
Implementation in Data Centres located outside CNES– SIPAD-NG can run on a single machine or its components can be
distributed over several machines
– SIPAD-NG can run on both SOLARIS and LINUX operating systems
11CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Architectural design
SIPAD-NGkernel
Enum Type
Objet de stockage
idODtype(local,archive,url)chemin fichiernom fichier
Definition Attribut
idAttidEntitetype(INT,STRING,ENUM)nomstandard(OUI,NON)
1..n
0..1
1..n
0..1
Métadonnées
données
Entité Type
idEntite Typedescritiontype ( OBJDS,JDON,PBR,DOCU,COLLEC )
0..n
1
0..n
1
Attributs_Int
idAttributidAttidEntitevaleur
0..n
1
0..n
1
Jeux de données
idEntitedateDebdateFindroitaccesvolume
Object de données
idEntitedateDebdateFinvolumeradicalreference
1..n
1..n
1..n
1..n
Produit de browse
idEntitedateDebdateFindroitAccestypeBrowse
Collection
idEntitedroitAccesnomFrançaisnomAnglais
Attribut_String
idAttrributidAttidEntitevaleur
1..n
0..1
1..n
0..1Attribut_Enum
idAttributidAttidentitevaleur
1..n
0..1
1..n
0..1
SVA
idSVA
criteres
idCritere
11
11
table
Entite
idEntiteidAttnomdatecreation
0..n1
est de type
0..n1
1..n
1
1..n
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1..n
1
1..n
1
1..n
1
1..n
1
1
1
1
1
1
n
1
n
1
1
1
1
Document
idEntitedroit accesnomFrançaisnomAnglaisgluTagnbrePages
1
1
1
1
Catalogue
Basic services Administration& Supervision
IngestionUser
management
Catalogueconsultation
OrderManagement
User space
AdministrationWeb Server
Administrator
LocalData Centre
services
Clientapplications
On line archive Near line archive Media production
Web ServerAdded Value
ServiceRemote
Data Centre
12CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Client applications provided by SIPAD-NG
The SIPAD-NG software system is composed of:– a kernel:
» the six basic services,
» the Administration Web Server,
» the data base
– several client applications:
» complementary to the basic services (e.g. automatic ingestion),
» main client application: User Web Server
Characteristics of the SIPAD-NG User Web Server:» compliant with the six basic services interfaces,
» easily customizable,
» A project can replace it by its own web server and only reuse the communication layer with the basic services
13CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Time selection
Implementation examples
Internet machine
Geographic selection
Other SIPAD-NG services& Administration Web Server
STAF
SEM
Oracle serverCDPP catalogue
Solaris server
SIPAD-NGUser Web Server &
Catalogue Consultation service
CDPP at CNES Computer Centre
ICAREData Centre
File server
On-line data
SIPAD-NG basic servicesSIPAD-NG user web serverICARE catalogue
Linux server
16CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Technology
User Web Server&
Administration WebServer
Basic services
- JAVA- Apache Tomcat- Struts → MVC2
- Axis → Web Services
- JAVA- RMI → Inter-machines communication
- XML → Ingestion & Basic services interfaces
ORACLE 9i
‘Object-Relational Mapping’layer
‘Business’ layer
‘Application Logic’ layer
‘Client’ layer
‘Database’ layer
17CEOS/WGISS20 – Kyiv – September 13, 2005 Paul
Kopp
Conclusion
Schedule» 2003: requirements, specification
» February 2004: beginning of development
» July 2005: first release
» December 2005: final release
SIPAD-NG contractor: CS SI
SIPAD-NG users– those already using SIPAD
» CDPP, MERCATOR, CASSINI
» Migration of the catalogues from SIPAD to SIPAD-NG
– Future potential users
» ICARE: Data Centre for Clouds & Aerosols
» SSALTO: Altimetry & Orbitography