Eur

opea

n G

rid

of S

olar

Obs

erva

tions EGSO

A Virtual Observatory for solar and heliospheric physics

Rob Bentley (UCL/MSSL), Isabelle Scholl (ISU)

and the EGSO Team

Super Computing 2004Pittsburgh, 7-12 November 2004

http://www.egso.org

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

EGSO – European Grid of Solar Observations

EGSO is a Grid test-bed related to a particular application

Addresses the generic problem of linking a distributed heterogeneous data set, and serving a scattered user community Designed to improve access to solar data for the solar physics and other communities by establishing a virtual observatory

Funded under the Information Society Technologies (IST) thematic priority of the EC’s Fifth Framework Program (FP5)

Objectives include:Building enhanced search capability for solar dataSupport of user community scattered around the worldProvide enhanced access to data archives and observatories around the world

o Where possible, provide ability to process data at source

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

The Solar Virtual Observatory family

Partners and collaborators provide expertise in solar physics and IT

UK University College London: UCL-MSSL (PI) & UCL-CS,

Rutherford Appleton Laboratory, University of Bradford

France Institut d’Astrophysique Spatiale (Orsay), Obs. de Paris-Meudon, International Space Univ.

(Strasbourg)Italy Istituto Nazionale di Astrofisica, Politecnico di Torino

INAf includes Obs. of Turin, Trieste, Florence and Naples

Switzerland University of Applied Sciences (Aargau)US Solar Data Analysis Center at NASA-GSFC

(Greenbelt, MD), National Solar Observatory (Tuscon, AZ)

SDAC and NSO are also part of the US VSO

Belgium Royal Observatory of BelgiumNetherlands ESA-ESTEC – Solar Group

US VSO: Stanford University, Montana State University

CoSEC: Lockheed-Martin

VSPO: LEP at NASA-GSFC (Lab. Extraterrestrial Physics)

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Use of solar observations

The appearance of the Sun changes dramatically with wavelength

Emissions originate from different layers in the atmosphere and different physical phenomena

For a complete picture we need to use as wide a range of observations as possible

Mixture of multi-wavelength observations from space- and ground-based platforms

Identifying observations that match some User search criteria and then retrieving them are major problems

Heliosphere

Corona

Chromosphere-TR

Surface Magnetic Field

Photosphere

2x106 K

8x104 K

6x103 K

31JAN0316JAN03

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Need to find ways of tying the data from different communities together

Single data model covering all solar system not practicalIntersecting data models in a general pool should be possible

Linking solar data into a wider context

Increasing desire to use solar data in study problems that span communities

Space weathero heliosphere,

magnetosphere, ionosphere…

Climate physicsPlanetary physicsAstrophysics

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Space Weather affects human activities

Electrical power distributionLong-line telephone communicationsPipeline operationsGeophysical explorationNavigationHF radio CommunicationsHigh-altitude flightsSatellite operationsSpace Shuttle and Space Station activities

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

The Generic Query

Identify suitable observations (many serendipitous)Want to access as many different types of data as are available

o Identification should be possible without accessing the datao Data volumes are increasing rapidly - SDO will produce 2 TB/day

User only wants to know if data addressing a problem exists

Locate the dataData scattered, with differing means of access (some proprietary)

o Large and small data providers, with varying resources

Process the dataInvolves extraction and calibration of a subset of raw data

o Often only need a subset of each data set

Return results to the UserCompare results from different instruments

SolarSoft (IDL) provides a standard platform for analysis

EGSO is placing a lot of emphasis on the first item!

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

The EGSO Search Engine

Enhanced cataloguing describes the data more fullyStandardized versions of observing catalogues (UOC) tie together the heterogeneous data sets

Search Registry, an abstraction of entries in the UOC, enables narrowing of the search in initial stages

New types of catalogue allow searches on events, features and phenomena, not just date & time, pointing, etc…

Solar Event Catalogue (SEC) - derived from published lists

Solar Feature Catalogue (SFC) - generated by feature recognition

Ancillary data used to assist the User in the searchQLK Server provides Phone book access to images, etc.

DSO Server gives Yellow Page information on instrumentsSimilar hierarchical cataloguing techniques are required in other Grid projects handling diverse data sets

EGSO is improving the quality and availability of metadata

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Catalogue relationships

Built from

Database of solar observations- instrument - observatory- EGSO available?- observing location- observing interval- description ….

Time coverages- instrument - observing date start- observing date end- observing parameter name- observing parameter value- data source

Solar Observations - date start- date end- wavelengths- coordinates- ….many more relevant characteristics needed for searches

UOC

SR

Event/Feature Catalogs- catalog name- event name- observing date- description ….

SEC/SFC

DSO Built from

Data Archives

Manually Built

QKL

Objective of the improved metadata, etc. is to be able to pose questions like:

Identify events when a filament eruption occurred within 30° of the north-west limb and there were good observations in H, EUV and soft X-rays

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Simplified Architecture

After R. Linsolas, IAS

Architecture defined in terms of three roles:

Consumer, Broker and Provider

ARCHIVES

Consumer Broker

Broker

Provider

Provider

Provider

Cat.

Consumer

GUI

GUI

GUI

EGSO GRID

API

Special Providers

Results

SEC, SFC, UOC, DSO,

QLK, CoSEC

Archive access can be

by FTP, HTTP,

Web Services, cgi-bin…through adaptor modulesBrokers manage

the metadata and decides and

allocates resources

Consumer supports GUI

and API access

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

The UOC and Search Registry

Unified Observing Catalogue (UOC)Unified form of observing catalogues that tie together the heterogeneous data, leaving the data unchanged

o Increase interoperability by using “standard” coordinate formatsSelf describing, with dependencies on ancillary data or proprietary software removed (and with any errors corrected)Standards defined for future data sets (e.g. STEREO, ILWS, Solar-B)

Search Registry is an abstraction of entries in the UOCIt reduces interactions with Providers without data matching the searchThe Registry allows the Broker to identify instruments that:

o have data properties matching the search - Static SRo probably have observations during search time interval -

Dynamic SR

Static Search Registry is able to support access to different types of data from solar and heliospheric observations

o Describes instrument capabilities & observing objectives o First step in search - later steps handled br dSR and within other

VOso EGSO sSR includes instruments on Ulysses, ACE, Cassini, SDO,

STEREO…

The logic behind the UOC addresses a universal problem - it allows the observations to be described in more interoperable way

Similarities in the way we use data means that the static Search Registry can be used to tie solar and heliospheric data together. Planetary data could easily be added…

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Locating the data

Data could be located anywhere in the worldUser only needs to know observations exist, not where locatedSystem should isolate the user from the intricacies of access

System should be able to optimize use of sourcesChoice of source - most capable, closest, least used, etc.Handles replicated data and aggregated sources Must respect any use policies and ensure data integrityBurden on data providers minimized to encourage participation

In EGSO, data sources are interfaced by the Provider Role

Broker uses Data Registry to formulate requests to data sources

o Which instrument data sets are hosted by each data archiveo Which data archives interfaced to each Provider Role

Provider Role uses adaptor modules to manage access protocols

o Standardizes way data source interface appears to the system and simplifies addition of new data sources

o Also allows access to data “hosted” by other VOs

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Query Work Flow

User specifies query through the GUI or APIStatic Search Registry narrows the search, based on the criteria specified in the query

o Identifies instruments that make the desired type of observations

o Search can includes solar, heliospheric, etc… instruments

Dynamic Search Registry determines (at some granularity) which were actually observing

o Includes pointing, observatory location…

List of instruments identified as possibly having data returned to the User who refines selection

Data Registry used to locate archive holding the instrument data and make data request

List of files returned that can be retrieved directly, used to generate data products, etc.

Convert to processed products if required

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

Summary

EGSO has become a principal player in the virtual observatories covering solar and heliospheric physics

Already able to exchange of information and resources between the solar VO projects

Approaches of the solar VOs very complementaryLinks into the space plasma VOs developing rapidly

The layered approach to metadata greatly simplifies interoperability with other VOs

Similar techniques applicable to similar Data Grid

Demonstration of capabilities at the EGSO stand

http://www.egso.org

Eur

opea

n G

rid

of S

olar

Obs

erva

tions

QuickTime™ and aSorenson Video decompressorare needed to see this picture.