virtual observatory interfaces reused in the virtual atomic and molecular data centre

Virtual Observatory Interfaces reused in the

Virtual Atomic and Molecular Data Centre

Guy RixonInstitute of AstronomyUniversity of CambridgeSeptember 2009

Rixon: VObs interfaces reused in VAMDC 2

VAMDC reusing VObs IT

Virtual Observatory

VAMDC

Paradigms,Paradigms,Standards,Standards,

ImplementationImplementationss

Toocomplicated?

Start fromscratch?

Toosimplistic?

WHY?


Past, present and future

2009 2010 2011 2012 2013 Time

Use

fuln

es

s

Era of Separateweb-sites

VAMDCproject ???

IT support for use of A & M data does not start and end with the VAMDC project.


Goals Open

Easy to add users (and user communities) Extensible

Easy to add data and applications Composable

Easy to combine with other software Sustainable

Easy (cheap) to maintain Finished in < 3 years


Goal conflicts

Extensible, composable, cheap:

pick any two...

“Good Software Takes Ten Years. Get Used To it” – Joel Spolsky


How does the Virtual Observatory do?

Tries to be open, extensible, composable, etc.

Web services Standard protocols Resource discovery via registry Workflow support via data grids


Open and Composable

GUI Script Workflow Etc.

Vobs service Vobs service Vobs service Vobs service

HTTP callsacross WAN

Web (HTTP) services


Extensible: adding data

VObs clients

Site A Site B Site C

Pu

bli

c

Private


My community's

apps

My community's

apps

The IVOA “API library”

Open & extensible: adding applications

Pu

bli

c

Private“The Virtual Observatory”

My personal

apps

My personal

apps

My personal

apps

My community's

apps

Standard Apps &utilities

Standard Apps &utilities

Generic apps &utilities


Extensible and sustainable: registry

Publish, describe

Discover,choose

Resources registered in on-line directory Scientists can discover meaning of resources Software can discover interface details No need to update software to use new

resources


What can be registered?

Hypothetical extension by VAMDC


Structure of registration document

Resource identity

Curation

Content synopsis

Capability(TAP)

Capability(VAMDC)

Tableset

“Dublin core”: meaningful to science usersUsed to select services.

Each Capability describes a major interfaceLists access URL(s) and support for optional parts of interfaces. Machine readable detail;drives UIs.

Describes the tables available for query.Machine-readable detail; drives querybuilding UIs.


Relational vs. XML storage

Relational: Robust, mature Choice of RDBMS SQL query Hard to add new

schemata

Native XML DB New(ish)

technology Few

implementations XQuery Trivial to add

new schemata

Registries have been built on both relationaland native-XML databases.

XQuery + schema flexibility wins every time.


Publishing, harvesting, searching

“Full” registry

NVO (STSci)

Publishingregistry

VizierPublishing

registry

CASUPublishing

registry

VAMDCregistry

Etc.

All registrations“harvested” to fullregistry

VAMDC providers registerServices here

Virtual Observatory VAMDC

VAMDC users search here


Extensible and composable: data grid

Service A Service B Service C

VOSpace VOSpace

Fast network

My app.

Slower network


UIs: VOSpec Multi-wavelength spectral

analysis tool developed at ESAC by ESA-VO

Create a SED with spectra from: a wide range of

different providers registered in the VO Registry

local data (FITS, ASCII, VOT)

Many fitting/ plotting functions

ISO LWSHST

ISO PHT

EUVE

HUT

HST FOSIUE

INES

http://esavo.esac.esa.int/vospec/





VOSpec cont. VOSpec provides access

to atomic and molecular line Databases (SLAP): NIST Atomic Spectra CHIANTI (X-rays) CIELO (XMM-Newton) IASD (ISO) LERMA (molecular) STSCI (UV)

on-the-fly identification of transitions associated to a given observed feature.


UIs: Taverna


UIs: VODesktop


VODesktop: actions on resources


VODesktop: query builder


VODesktop/SAMP: UI integration


VODesktop: data-grid access


Looter's checklist

Web-service paradigm Registry paradigm Data-grid paradigm VODesktop Other UIs/desktop applications (as

applicable) Service standards (as applicable) Service implementations (as applicable)


IVOA standard services

Standard Purpose Data model Relevance?

Registry Discovery of services Extensible High

Cone search Data access: sky search Astronomical None

SIAP Data access: images Astronomical None

SSAP Data access: spectra Astronomical Low/medium

TAP Data access: tables Generic High

SimDB/SimDAP Data access: simulations Generic ???

SLAP Data access: line lists Astronomical Medium/high

VOSpace Grid-accessible storage Generic Medium

UWS-PA/CEA Grid execution Generic Medium


The data-model issue

Web-service shell

Data-model

Resource adaptor

Requests

Results

What the client needs

Elaborate data-modelmakes it hard to repurpose the service,especially if domain-specific

For reuse, prefer serviceswith little or no data model


TAP: Table Access Protocol

Web-service shell

Data-model

Resource adaptor: async. queries on JDBC

Requests: SQL Results: VOTable/HTML/CSV/???

Relational database

TAP data-model isvery generic:“relational data”

Works with any DB schema


TAP: possible use in VAMDC year 1

TAPservice

TAP

DB

VAMDCApp (desktop)

Describe DB schematain registry

Reuse same implementation on each VAMDC site

Sites provide these

Deal with schemataVariations here

One per VAMDC site


TAP: possible evolution

Web-service shell

Data-model

Resource adaptor: async. queries on JDBC

Requests: XQuery

Results: XSAMS

Relational database

Replace the TAP data-model with something VAMDC-specific

Works with any DB schema

Extend this for VAMDC


VOSpace

Storage accessible to users and services Useful in workflows, grid jobs Can use simple storage or data grid

Site A Site B

iRODS

Site C

iRODS

Site C

iRODS

VOSpace VOSpace VOSpace


SLAP: Simple Line Access Model

Web-service shell

Data-model

Resource adaptor

Requests Results: VOTable

Relational database

SLAP data-model isvery simple: “just enough for astronomy”

Needs a particular schema


CEA & grid

CEA = Common Execution Architecture Describe, register, discover, run server-side apps:

grid “A standard for everything else” Powerful:

CEA UI runs any registered app Allows long-running jobs (asynchronous

execution) Can be connected to the Grids, e.g. EGEE Integrates with VOSpace, Taverna workflow


If all else fails...

...design a new service type! Need to specify:

Service semantics Web-service protocol (e.g. REST) Query language (e.g. XQuery) Representation of results (e.g. XSAMS) XML schema for registration

Consider following IVOA “DAL” architecture


VAMDCstrategy

Adapting to new service types

VODesktop UI

Registry parser

AstroRuntime

SLAPstrategy

SSAPstrategy

TAPstrategy

CEAstrategy VAMDC

strategy


Summary

VAMDC needs adaptable, future-proof software on the cheap

We think the VObs model will work We will copy the VObs ideas We will use their service protocols to

start We will use the ready-made software We will exploit the flexibility of the

system progressively to fit it to VAMDC.

virtual observatory interfaces reused in the virtual atomic and molecular data centre

Documents

vobs interfaces

vamdc extensible

vamdc uis

vamdc past

vamdc publishing

vamdc goalsopeneasy

vamdc relational

vamdc project