2006-11-30, triesteuro-vo dca wp5 kick off euro-vo dca wp4 and gavo and (a little) astrogrid-d...

2006-11-30, Triest Euro-VO DCA WP5 kick off

Euro-VO DCA WP4and

GAVO and (a little) AstroGrid-D

Gerard Lemson

Euro-VO DCA WP5 kick-off meeting

Trieste, 2006-11-30


Overview

• Euro-VO DCA WP4– Goal and motivation– Theory in VO– Theory in IVOA– Theory in Euro-VO DCA: WP4– WP4 and WP5: theory WP5 and/or grid in WP4 ?

• GAVO (& AstroGrid-D)– GAVO I– GAVO II– Theory in GAVO– Grid in GAVO– AstroGrid-D– (GA)VO and AstroGrid-D


Euro-VO DCA WP4• Objectives

– This task aims at assessing the inclusion of theory data and services in the Virtual Observatory

• Description of work– The task is tackled by the Theory Expert Group, which has

yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop.

• Deliverables– WP4-D10: Euro-VO Theory Workshop– WP4-D11: TEG Report: Framework for inclusion of Theoretical

Astronomy Services in the VObs• Expected results

– The final report will describe a framework for inclusion of theoretical data and services in the VObs

– Motivation of the European Theoretical astronomy community to participate in the VObs endeavour.


Reminder

• Broad goal of VObs– Make results of astronomical research, data and

applications, more readily available to larger community, and create value-adding services.

• Facilitate results-– communication– checking– (re)use– comparison– combination

• Organised in IVOA


The observational VObs

• Most VO efforts concentrate on observational data sets– simple observables: photons detected at a certain

time from a certain area on the sky– long history of archiving– pre-existing standards (FITS)– valuable over long time (digitising 80 yr old plates)– long history of comparing different observations:

interoperability• Standards observationally biased

– common sky: cone search, SIAP, region– common objects: XMatch– data models: characterisation of sky/time/energy (/no

polarisation yet)


The same object

John Hibbard

http://www.cv.nrao.edu/~jhibbard/n4038/n4038.html

Radio

NASA/CXC/SAO/G. Fabbiano et al.

X-RayOptical


Theory in VO

• Apply same ideas to theoretical resources (datasets and services).

• Plenty of resources available online.

• Not standardised: – 2004, whitepaper detailing issues related to

standardisation of theory


Theory in the VO: issues

• Simulations not so simple– complex observables– no standardisation (not even HDF5)– archiving ad hoc, for local use

• Moore’s law makes useful lifetime relatively short: few years later can do better


“Moore’s law” for N-body simulations

Courtesy Simon White


Standardisation issues

• Current IVOA standards somewhat irrelevant– assume common sky– common objects– comparison direct

• In theory– no common sky– no common objects– comparison statistical

• requires new data models – content, physics, code


Not the same object

Toomre & Toomre, 1972

Di Matteo, Springel and Hernquist, 2005

Courtesy Volker Springel


So why bother ?

• Simulations are interesting:– For many cases only way to see processes in action– Complex observations require sophisticated models

for interpretation

• Bridging gap in specialisations: not everyone has required expertise to create simulations, though they can analyse them.

• Many use cases do not require the latest/greatest – exposure time calculator– survey design


Theory in the IVOA

• IVOA Interest group on theory– founded May 2004– “horizontal”– use cases

• Current projects:– Theory UCDs/Semantics (Laurie Shaw)– SNAP (Claudio Gheller)– Simulation (meta-) data model (GL)

• Meeting on SNAP in Garching, early 2007


Euro-VO DCA WP4 (repeat)• Objectives

– This task aims at assessing the inclusion of theory data and services in the Virtual Observatory

• Description of work– The task is tackled by the Theory Expert Group, which has

yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop.

• Deliverables– WP4-D10: Euro-VO Theory Workshop– WP4-D11: TEG Report: Framework for inclusion of Theoretical

Astronomy Services in the VObs• Expected results

– The final report will describe a framework for inclusion of theoretical data and services in the VObs

– Motivation of the European Theoretical astronomy community to participate in the VObs endeavour.


WP4: tasks

• Procurement WP4 personnel:• Draft whitepaper “Framework for inclusion

of Theory data and services in the VObs” (End cycle 1)

• EURO-VO Theory Workshop (Deliverable 10, Month 17)

• Whitepaper ready (Deliverable 11, month 26)

• Participation IVOA interop meetings with theory


WP4 first steps

• Formation Theoretical astronomy Experts Group (TEG)

• CNRS Hervé WOZNIAK• ESA Pedro OSUNA• ESO None• INAF Santi CASSISI• INTA Miguel CERVIÑO• MPG Gerard LEMSON (Chair)• NOVA TBD (Joop SCHAYE)• LU TBD (?)

• Census theory data centers (with WP2)• Cooperation in IVOA theory interest group

activities• Coordination with WP 5 (which is why I’m here)


WP4 and WP5: some first ideas

• Grid support for WP4– Grid must be addressed in WP4 whitepaper– Grid can facilitate/is required for implementations of

prototype for standards (see below)• Theory use cases/requirements for WP5.

– Asynchronous, queued querying of large simulation database

• OGSA-DAI? (GAVO+AstroGrid-D)– SNAP on Millennium raw data

• grid services wrapping custom data access modules (GAVO+AstroGrid-D)

– Scientifically useful virtual telescopes– Parameter studies using “small” simulations– Large simulations ??


Observed

Simulated

Thanks to Alexis Finoguenov, Ulrich Briel, Peter Schuecker, MPE)

Thanks to Volker Springel

Example: Virtual Telescope


Virtual telescope configuration


Virtual telescope configuration• Components:

– simulation of source, wrapped with “photon producer” service– spec and implementation for “pre-observation image” (POI)– “foreground” accepting, reemitting POI

• ISM/IGM• lenses

– “software telescope”, simulating realistic observations– comparison software

• Required (VO/Grid) infrastructure supporting this workflow– registry for publishing/discovering components– computational resources – scheduler for executing job– streaming support for flowing POI between components– storage for results– workflow engine?


Discussion

Suggestions very welcome


Intermezzo


GAVO and AstroGrid-D


GAVO and AstroGrid-D

• GAVO I– Funded by German Ministry of – Dec 2002 – March 2005– 3 months extension– 3 FTEs

• AstroGrid-D– 1 of 5 funded “communities” taking part in D-Grid project– O(13) FTEs– Autumn 2005 – 2007/8

• GAVO II– Delayed funding, due to need to distinguish from AstroGrid-D– June 2006-July 2008– 5 FTEs (so far 3, ARI (>~1.5), MPE (1.5), TUM (.5), AIP (<~.5),

Tübingen (1))


GAVO (I + II) projects

• Archives: – ROSAT All Sky Survey

• cone search: photons, sources• SIAP: fields

– ROSAT pointed observations: SIAP• standardised approach to publish FITS images under SIAP• uses FITS metadata repository (SQLServer)• SQL based mapping and SIAP implementation• easily generalised to other datasets and protocols (SSA)

– RAVE– CDFS optical follow up spectroscopy: SSA – Heidelberg catalogues and GAVO archive: SCS/SkyNode

• Data mining– Cross-matcher (also on Grid, TUM)– SED classifier– Cluster finder


GAVO (I + II) projects ctd.• Grid

– evaluation Globus– Gadget 2 on Globus MPI (AIP + MPE)– cluster finder on the grid

• VOQL/SkyNode– VOQL tiger team (Benjamin Gufler)– HiSbase peer-to-peer approach to multi catalogue cross-

matching– Mock catalogues for testing

• Theory– IVOA:

• Initialised Theory Interest Group (chair)• SNAP• Data models• Semantics

– Millennium database– (proto-) Virtual telescopes (planck, MoMaF2, Dolag et al)

http://www.g-vo.org/MyMillennium

http://www.g-vo.org/planck/?

http://www.g-vo.org/mpasims/MoMaf2?

http://www.g-vo.org/hydrosims/?

http://www.g-vo.org/hydrosims/?


AstroGrid-D

• Part of D-Grid

• Community project

• Provide– use cases– middleware– adaptation of existing applications/data to the

(D-)Grid


AstroGrid-D partner projects (incomplete)

• Astrophysikalisches Institut Potsdam– project management– robotic telescopes on the grid– galaxy merger simulations

• MPE:– cluster finder – catalogue cross-matcher (with TUM)– adapt simulation visualisation to the Grid– access to persistent data (in database)

• MPA– Planck Process Coordinator on the Grid

• Technischer Universitat München-Informatik– StarGlobe: streaming data querying/combination


AstroGrid-D partner projects, ctd.

• Zuse Institut Berlin– Grid Application Toolkit– GridSphere web portal– Information Services (“registry”, RDF+SparQL)

• Albert Einstein Institute, Berlin– Cactus tool kit– Grid based simulation monitoring

• Astronomisches Rechen Zentrum, Heidelberg– Stellar dynamics: Grape on the Grid– GaiaGrid– Planck ProC collaboration


GAVO + AstroGrid-D, synergies

• Groups involved in both projects: AIP, ARI, MPE, TUM

• VO offers different kind of use cases:– VObs aims to enable external users to use

resources– Heterogeneous world: Interoperate with

resources outside of AstroGrid-D (but within IVOA standards)

• Grid can help implement VO services


GAVO + AstroGrid-D ctd.

• Planned (from GAVO point of view)– Cluster finder (MPE)– SNAP implementation (MPA)– OGSA-DAI (GSM?, DSA?, CAS Jobs) access

to Millennium DB (MPA)– Real virtual telescopes (MPA+...)– Cross matching (MPE+TUM)

• Possibly– VOEvent and Robotic Telesopes ?– AstroGrid-D Information Services as IVOA

Registry


SNAP using grid services

• Simple Numerical Access Protocol– Project of IVOA theory interest group– Based on simulation data model (input welcome !) – Query interesting simulations– Select and retrieve subset

• Implementation: SNAP on Millennium snapshots– Custom access code deployed near large datasets– Authenticated access– Standardised publication for use in Grid workflows

(ProC? Streaming?)– Data staging and movement


Millenniumsnaphots

getBox()C-module

Grid Service

Globus container

Information Service

SNAP web service

IS awareWorkflow

Engine


Millennium website and OGSA-DAI

• Dataset– postprocessing products of Millennium

simulation (http://www.mpa-garching.pg.de/millennum)

– halo/galaxy catalogues – density fields (environment queries)

• Functions– support generic SQL queries on large (O(5Tb)),

complex database– support mydb– some graphics script based access


Implementation• Microsoft SQLServer database (SkyServer inspired)

– data model for efficient retrieval merger tree histories– spatial indexing

• TOMCAT web server – http://www.g-vo.org/MyMillennium– authentication using RDB– wget based access, IDL+R modules

• Graphics– VOPlot applet (small results, browser)– TOPCAT (plugin)

• Synchronous queries– JDBC– 7 min time limit

• Issues– I/O bottle necks when > 4 users query same table – short maximum query time– no asynchronous access– simplistic user management (compared say to CAS Jobs)

http://www.g-vo.org/MyMillennium


Why Grid interesting?

• Need – Asynchronous access

• Queuing• Improve result management, data delivery

– Bulk uploading into MyDB– More generic authentication (X.509)

• Will try CAS jobs (SDSS/JHU)• Investigate OGSA-DAI

– Map portal to OGSA-DAI functionality – Monitoring ? Logging ? Performance ?

• Will try AstroGrid-UK’s DSA• Other suggestions ?


Thank you !

2006-11-30, triesteuro-vo dca wp5 kick off euro-vo dca wp4 and gavo and (a little) astrogrid-d...

Documents

inclusion of theory

inclusion of theoretical

eurovo dca wp4andgavo

vo dca wp4objectivesthis

overvieweurovo dca wp4goal

eurovo theory workshopwp4d11

theory expert group

theoretical resources