virtual observatories
DESCRIPTION
Virtual Observatories. Wolfgang Voges Max-Planck-Institut für extraterrestrische Physik Garching. Workshop ‘‘Astronomie mit Großgeräten‘‘ Am 17.Oktober 2003 in Potsdam. Virtual Observatories. Overview: Historical roots * What’s happening in the world: IVOA European VO-activities - PowerPoint PPT PresentationTRANSCRIPT
Wolfgang Voges 1
Virtual Observatories
Wolfgang Voges
Max-Planck-Institut für extraterrestrische Physik
Garching
Workshop ‘‘Astronomie mit Großgeräten‘‘Am 17.Oktober 2003 in Potsdam
Wolfgang Voges 2
Virtual Observatories
Workshop ‘‘Astronomie mit Großgeräten‘‘Am 17.Oktober 2003 in Potsdam
Overview:
Historical roots *
What’s happening in the world: IVOA
European VO-activities
German VO-activities (GAVO) Federation of local data-sets
Next-generation search engine
Grid
Theory in GAVO
Outlook* Viewgraphs partly copied from other presentations
Wolfgang Voges 3
Virtual Observatories
1. VO meeting at Caltec in Pasadena (June 2000)
Astronomers mostly from the US
Very enthusiastic talks
Big vision of the future
Foundation of the NVO (US)
Since then similar meeting in Europe (Garching)
Foundation of national European and later
Other VOs
Historical remarks
Wolfgang Voges 4
The exponential growth of data volume (and complexity, quality)
driven by the exponential growth in information
technology …
… But our understanding of the universe increases much more slowly -- Why?
Methodological bottleneck VO is the answer Human wetware limitations … AI-assisted discovery NGVO?
Data Knowledge ?
Wolfgang Voges 5
How and Where are Discoveries Made?
• Conceptual Discoveries: e.g., Relativity, QM, Brane World, Inflation … Theoretical, may be inspired by observations
• Phenomenological Discoveries: e.g., Dark Matter, QSOs, GRBs, CMBR, Extrasolar Planets, Obscured Universe …
Empirical, inspire theories, can be motivated by them
New TechnicalCapabilities
ObservationalDiscoveries
TheoryIT/VO (VO)
Phenomenological Discoveries:
Pushing along some parameter space axis VO useful
Making new connections (e.g., multi-) VO critical!Understanding of complex astrophysical phenomena requires
complex, information-rich data (and simulations?)
Wolfgang Voges 6
Why is VO a Good Scientific Prospect?• Technological revolutions as the drivers/enablers
of the bursts of scientific growth
• Historical examples in astronomy:– 1960’s: the advent of electronics and access to space
Quasars, CMBR, x-ray astronomy, pulsars, GRBs, …
– 1980’s - 1990’s: computers, digital detectors (CCDs etc.)
Galaxy formation and evolution, extrasolar planets, CMBR fluctuations, dark matter and energy, GRBs, …
– 2000’s and beyond: information technology
The next golden age of discovery in astronomy?
VO is the mechanism to effect this process
Wolfgang Voges 7
SurveysObservatories
Missions
Surveyand
MissionArchives Follow-Up
Telescopesand
Missions
Results
Data Services---------------Data Miningand Analysis,
Target Selection
Digital libraries
Primary Data Providers
VOSecondary
DataProviders
A Schematic Illustration of the VO-Based Science
VO as an integral partof the whole system …
Wolfgang Voges 8
The Changing Style of Observational Astronomy
The Old Way: Now: Future:
Pointed, heterogeneous
observations (~ MB - GB)
Large, homogeneous sky surveys
(multi-TB, ~ 106 - 109 sources)
Multiple, federated sky surveys and archives (~ PB)
Small samples of objects (~ 100 - 103)
Archives of pointed observations (~ TB) Virtual
Observatory
Wolfgang Voges 9
This quantitative change in the information volume and complexity will enable the
Science of a Qualitatively Different Nature:
• Statistical astronomy done right – Precision cosmology, Galactic structure, stellar astrophysics …– Discovery of significant patterns and multivariate correlations– Poissonian errors unimportant
• Systematic exploration of the observable parameter spaces (NB: Energy content = Information content)
– Searches for rare or unknown types of objects and phenomena
– Low surface brightness universe, the time domain …
• Confronting massive numerical simulations with massive data sets
Wolfgang Voges 10
Panchromatic Views of the Universe:A More Complete, Less Biased Picture
Radio Far-Infrared Visible
Visible + X-ray
Dust Map
Galaxy Density Map
Wolfgang Voges 11
Examples of Possible VO Projects:• A Panchromatic View of AGN and Their Evolution
– Cross-matching of surveys, radio to x-ray– Understanding of the selection effects– Obscuration, Type-2 AGN, a complete census Evolution and net energetics, diffuse backgrounds
• A Phase-Space Portrait of Our Galaxy– Matching surveys: visible to NIR (stars), FIR to radio (ISM)– A 3-D picture of stars, gas, and dust, SFR …– Proper motions and gas velocities: a 6-D phase-space picture Structure, dynamics, and formation of the Galaxy
• Galaxy Clusters as Probes of the LSS and its Evolution– Cluster selection using a variety of methods: galaxy overdensity,
x-rays, S-Z effect …– Understanding of the selection effects Probing the evolution of the LSS, cosmology
Wolfgang Voges 12
Exploration of new domains of the observable parameter space: the Time Domain
Faint, Fast Transients (Tyson et al.)
Megaflares on normalMS stars (DPOSS)
Existing and Forthcoming surveys: Microlensing experiments (OGLE, MACHO …)
Solar System patrols, GRB patrols …
DPOSS plate overlaps (Mahabal et al.)
QUEST-2 and NEAT at Palomar… and many, many others …
The future: LSST ?
Wolfgang Voges 13
Data Mining in the Image Domain: Can We Discover New Types of Phenomena Using Automated Pattern
Recognition?(Every object detection algorithm has its biases and limitations)
– Effective parametrization of source morphologies and environments– Multiscale analysis (Also: in the time/lightcurve domain)
Wolfgang Voges 14
Exploration of observable parameter spaces and searches for rare or new types of objects
Wolfgang Voges 15
new, more, better, faster, and easier science comparative analysis of multi-instrument data,
permit new approaches to research and multi-wavelength exploration,
opening discovery capabilities not otherwise possible
This is clearly the primary mandate of all VO efforts
minimise redundancy:data collected by a single telescope / instrument can be re-used multiple times by different teams and for different scientific purposes
data integrity: data are archived and documented in a controlled and uniform fashion, ensuring long-term scientific usage
improving calibrations and creating more higher-level data products to make data more science-ready
Advantages of a Virtual Observatory
Wolfgang Voges 16
interoperability of archives:- strengthening connections to other archives, catalogues and abstract
services for broader research parameter space and links to the literature
advancing technologies for computers, networks, data compression, and storage media:- to retrieve and analyse more information more readily at lower cost
efficient serving of data to the public:
- there will be different levels of end-user from professional astronomers to interested (high-school) students and enthusiastic amateurs – many of whom may undertake projects which are simply unrealisable by large
institutes
data-mining with new software tools and new catalogues of object properties:
- to enable higher-order research based on questions posed in scientific terms
Advantages of a Virtual Observatory
Wolfgang Voges 17
improving the preparation, development, building of new ground-based and space-based projects
improving new observation proposals comparison of real data with simulated data – to provide feedback
to new insights, new models, new physics
Advantages of a Virtual Observatory
Wolfgang Voges 18
Korea, Japan, China, Australia, India, Russia, Hungary, Italy, France, Germany, Europe (ESO++), Canada, USA
International Virtual Observatory Alliance
Wolfgang Voges 19
Virtual ObservatoriesWhat’s happening in the world: IVOA
International STANDARDS are needed
Registry
Data-Models
VO-Table
VO-Query
Uniform Content Descriptors (UCD)
Simple Image Access (SID)
GRID-standards
Tools e.g. data-mining
Wolfgang Voges 20
Virtual Observatories
In the AVO (euro-vo.org) under the leadership
of ESO/ESA the following institutes/groups
are collaborating: ESO
ESA/STECF
University of Edinburgh
CDS Strasbourg
University Louis Pasteur
Centre National de la Reserche Scientifique Delegation Paris
The Victoria University of Manchester
GAVO (RDS:MPE,AIP,HS,MPA)
European VO-activities
Wolfgang Voges 21
German Astrophysical Virtual Observatory
GAVO-Team:Wolfgang Voges (PI)
Hans-Martin Adorf, Gerard Lemson, Achim Bohnet, Joachim Paul
Max-Planck-Institut für extraterrestrische Physik, Garching
Matthias Steinmetz (Co-I)Harry Enke, Detlef Elstner
Astrophysikalisches Institut Potsdam
Dieter Reimers, (Co-I)Dieter Engels, Peter Hauschildt
Hamburger Sternwarte
Simon White, Anthony Banday, Volker SpringelMax-Planck-Institut für Astrophysik, Garching
Other institutes are most welcome to join
>>>www.g-vo.org<<<
Wolfgang Voges 22
to remain internationally competitive (proposals, data utilisation, quality of science output)
to make available VO services to everyone and provide support for the science community and public in Germany
to prepare and maintain datasets obtained from German facilities for GAVO and IVO
to establish a network, within which the needs of the German science community and public are coordinated
to obtain financial support from German agencies for such a national task
Why do we need a German AVO?
Wolfgang Voges 23
Activities and responsibilities of partners
Wolfgang Voges 24
Main goal is science driven, but it will drive science, too
- fast access to all kinds of astronomical and related data
- capability to use highly sophisticated software tools for new studies
- GAVO will provide interoperability of distributed archives over a high speed network through a set of interface/infrastructure tools
- GAVO ultimately will be incorporated into larger IVO federation
- Astronomical institutes will require expert data centres of different local character e.g. for providing key data archives, documentations, “simple” analysis-, correlation- and visualisation tools
- computer science groups will develop data handling and novel analysis tools and are responsible for their maintenance
Activities and responsibilities of partners
Wolfgang Voges 25
- university institutes will be able to use GAVO for teaching and will provide a simple gateway to the public and to schools
- the “service community” will be responsible for designing and developing the interface/infrastructure tools necessary for communication between the users
Activities and responsibilities of partners
Wolfgang Voges 26
Archive publication through GAVO
• ROSAT source catalogs published in IVOA compliant manner: – simple cone search– webservices
• RASS Photons stored in PostgreSQL database– Spatial index using HEALPix – Cone search, webservices
• Federation: fast match between ROSAT source catalogues and RASS photons.• Published first proposal for unified datamodel to serve as an ontology for the
IVOA.• Plans:
– Extend query capabilities– Publish ROSAT fields and pointed observations– Federate with SDSS mirror at MPA– Federate ROSAT catalogues with external catalogues for classification of
X-ray sources (in collaboration with ClassX team).
Wolfgang Voges 27
Top priority during initial stage of development federation of local key datasets and provision of key applications
ROSAT, SDSS, Planck, RAVE development of meta-data standards, especially for simulations development of common query tools for the local archives
need ability to query/compare both real sky and simulated data post-processing tools - must be platform-independent installation of visualisation packages
existing software provides a strong foundation to allow
extension to different types of data and archives
The local GAVO activities
Wolfgang Voges 28
Technical challenges and requirements
archive standards: rules for ingestion, data quality, associated meta-data schema, data attributes
archive maintenance/evolution: migration of data with new technology and enhancements in data attributes
meta-data requirements/standards for different data-sets (observations, simulation, calibration)
federation of archives, interoperability high-speed networking, streaming formats for data distributed processing power – GRID concept seeking active cooperation with industry in many of these areas
The local GAVO activities
Wolfgang Voges 29
Next Generation Search Engine
• Download Manager– Retrieves data from multiple distributed databases
• Matcher– Matches sources based on sky-position (astronomical
sources have no unique identifier)
• Classifier– Uses multi-wavelength data for identification purposes
Wolfgang Voges 30
NextGen Search Engine (cntd.)
Multi-Catalogue Multi-Cone Search
"Download Manager"Probabilistic MatcherVOTable Processor
Simple ConeSearch Service #1
ServiceRegistry
Table onLocal Disk
Simple ConeSearch Service #2
VOTables
VOTables
VOTable
BaseURLs
BaseURLs
Simple ConeSearch Service #3
VOTable
MatcherDataSets
Local Disk
VOTable
VOTablesTable
One or moreSCS Queries
Local Disk
InternetTable
Wolfgang Voges 31
Download Manager
• Features– Tool …
• … accesses registry at JHU
– User …• … selects distributed catalogues• … specifies one or more sky-locations
– Tool …• … queries remote catalogues• … retrieves datasets for further processing
Wolfgang Voges 32
Download Manager (cntd.)
Wolfgang Voges 33
Matcher
• Essential for data mining• Prototype features
• Performs “fuzzy” match between pairs of source lists from different catalogues
• Computes probability of real match
• Moving matcher into production use– Collaboration with Canadian Virtual
Observatory (CVO)– Feeding ROSAT source matches to classifier
Wolfgang Voges 34
Matcher (cntd.)
Wolfgang Voges 35
Classification of ROSAT X-ray sources
• ClassX: in collaboration with US-VO
• Requires data from several large sky-surveys
• X-ray: ROSAT (BSC + FSC)
• Optical: SDSS, USNO B1.0
• Infrared: 2MASS
• Radio: FIRST, NVSS, SUMSS
• True multi-wavelength VO-application
Wolfgang Voges 36
Probing the large-scale structure of the universe
with clusters of galaxies Project outline:
(ideally on single photon/galaxy basis) (Schuecker, Boehringer,Voges)
- identify a sample of galaxy clusters using X-ray/optical correlation
>>>>>> see next 3 viewgraphs
- utilise optical multi-colour images (u,g,r,i,z) to derive photometric redshifts
- quantify completeness and selection limits by comparison to simulated cluster data
- search for IR correlation and quantify galaxy evolution in clusters
- determine correlation with radio surveys to identify the frequency of radio galaxies and AGN in clusters, search for radio halos
- optical correlation to identify AGN in clusters
- identify correlations with microwave/sub-mm data to search for the Sunyaev-Zeldovich (SZ) effect (distance measurements, velocity determination)
Correlation of ROSAT and SDSS data
Wolfgang Voges 37
Maximum likelihood contours based on RASS-3 X-ray photons (upper panel, 1, 2 .. contours), SDSS galaxies (middle panel, >10), and the combined maximum likelihood contours of RASS-3 and SDSS data (lower panel, >10). Crosses mark the position of the deepest X-ray clustersamples available sofar (REFLEX-2, X-ray flux limit 1.8 .10-12 erg s-1cm-2). Squares mark the position of the X-ray clusters of the final sample.
Search for clusters of galaxies
Wolfgang Voges 38
Search for clusters of galaxies
Cumulative X-ray cluster number counts of the RASS/SDSSclusters (histograms) for a log-likelihood minimum of 15 applied to SDSS data (continuous line), for 25 (lower dashed line), and for 5 (upper dashed line). The RASS/SDSS cluster counts are compared with results obtained with other surveys (squares: RDCS, REFLEX,REFLEX-2). No corrections for variations of the angular survey-sensitivity (effective survey area) are applied to the RASS/SDSS and REFLEX-2 data. The figure shows that with the combination of RASS and SDSS data a 10 times deeper X-ray flux limit can be obtained compared to traditional X-ray cluster surveys like REFLEX.
Wolfgang Voges 39
General remarks: Our first results are quite important as a guideline for future X-ray missions like ROSITA and DUO. For the latter, about 8,000 X-ray clusters are expected to be detectable with standard methods. The application of the matched-filter technique allows the extraction of about 30,000 X-ray clusters with DUO. Such large numbers of X-ray clusters are needed for precise tests of the dark energy and alternative gravitational theories.
Search for clusters of galaxies
Wolfgang Voges 40
VO functionality required:- federation of relevant datasets including interchange/merging of meta-data
- identification of candidate cluster members by appropriate query applications to optical and/or X-ray catalogues
- acquire multi-colour information to determine photometric redshifts
- identification of candidate radio galaxy cluster members by querying radio catalogues with search criteria (e.g. location) tailored to the derived cluster sample
- identification of associated SZ by specific queries in existing catalogues; if no candidate SZ cluster can be identified apply suitable search algorithms to Cosmic Microwave Background (CMB) sky surveys to determine effect or limits thereon
- visualisation of multi-wavelength cluster data
- deprojection algorithms to allow study of morphology in survey data
- conversion of simulation data to the space of observable parameters
- 3D-interface for visualisation (to schools)
Correlation of ROSAT and SDSS data
Wolfgang Voges 41
Correlation between radio, IR, optical and X-ray sources
Search for SDSS QSO´s with 1 < z < 2, which are variable in one of
the 4 wavelength-bands
search-engine:
- which datasets do exist and in which archive?
- multiple availability? parallel handling on different servers
- data available for different epochs? comparison of fluxes, light curves, period-search
- Source catalogues available? - radio: FIRST, NVSS, …
- IR: IRAS, 2MASS, …
- Optical: SDSS, Tycho-2, HST-GSC, USNO-2…
- X-ray: ROSAT, ASCA, XMM-Newton, Chandra, …
Query example
Wolfgang Voges 42
- if no catalogue entry exists
postage-stamp (pixel-image with/without contour-lines) creation of light curves, fluxes, spectra, etc.
by using original-data
- high demand of CPU?
GRID implementation
- search for publications on derived variable SDSS QSO objects
Query example
Wolfgang Voges 43
Grid Technology for GAVO
GAVO grid :
integration of all GAVO-workstations
at MPE and AIP into a cluster
Basic services on GAVO-grid:
CertificationAuthority provides
single-sign-on/access-all facility via proxy-ca
Resource discovery and runtime information,
network-weather for the grid
Running distributed applications
Running MPI-based applications on the GAVO-cluster
Wolfgang Voges 44
Virtual Observatories
Simulations
Comparison of Simulations and observations
Theory and GAVO
Wolfgang Voges 45
Merging of the Milky Way with the Andromeda galaxy (M31) (3 Mio particles, cluster of 16 CPU’s, 1 week of CPU time) (30 k particles would need 25 minutes of cluster-CPU time)
Simulations in the Virtual Observatory
Wolfgang Voges 46
COMA type cluster of galaxies (>1000 galaxies, 10^15Msolar, 7 Mio particles)
8 CPUs, runtime:2 days; Gravitation, Hydrodynamics, not included: cooling, star formation
(Volker Springel, MPA)
Simulations in the Virtual Observatory
Wolfgang Voges 47
The Role of Datasets fromTheoretical Astrophysics
• Direct Comparisons with Observations– Verification (or not) of Models
• Data Mining for Both Observations and Theory– New Applications– Buried Physics
• Resource for Education and Outreach
Wolfgang Voges 48
Theory and the VirtualObservatory
• Size of Datasets Appropriate to VO– Large Scale Simulations, Parameter Space
Libraries Imply 10GB – 10 TB Datasets
• Rich Complement to Observational Side
• Same/Similar Tools as for Obs. Datasets
• Use of VO Infrastructure– Grid Technology, Portals, etc.
Wolfgang Voges 49
Conclusions
• Theoretical Astrophysics is an Essential Part of the Virtual Observatory Concept
– Provides Benefits to Theorists– Provides Benefits to Observers– Provides Benefits to Education/Outreach
• Drives New Science
Wolfgang Voges 50
GAVO efforts on the TVO
• Published IVOA whitepaper on “Theory in the VO”• Leading theory subgroup in IVOA data modeling effort.• Chair in IVOA special interest group on theory• Plans:
– Publish simulation archives at AIP, LMU-Obs., andMPA – Collaborate with UPitt on publishing services on theoretical
datasets (NSF grant proposal)– Collaboration with Technion Haifa to publish observed and
simulated Ly- forest spectra (GIF proposal)– Collaboration in RTN proposal for comparison of simulated and
observed X-Ray clusters (Boehringer et al@MPE)
Wolfgang Voges 51
Virtual Observatories
Great enthusiasm among astronomical community
that VO will work and will make life easier
BUT still a lot to be done
IVOA is combining all available forces
to attack the manyfold problems
There is still the need to incorporate other fields like mathematics, informatics,computer science, networks
etc. since there is parallel work in progress
GRID paradigm, fast data-links, super-computer access, etc.
MORE MANPOWER NEEDED
Outlook
Wolfgang Voges 52
The end