oas contribution the acds the new xcatdb based on saada
DESCRIPTION
OAS Contribution The ACDS The new XCATDB based on Saada. Laurent MICHEL Christian MOTCH François Xavier PINEAU. Observatoire Astronomique de Strasbourg (Fr) Saada: http://astro.u-strasbg.fr/websaada XCATDB: http://amwdb.u-starsbg.fr/jacds. The ACDS. Delivered Products - PowerPoint PPT PresentationTRANSCRIPT
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
OAS Contribution
• The ACDS
• The new XCATDB based on Saada
Laurent MICHELChristian MOTCHFrançois Xavier PINEAU
Observatoire Astronomique de Strasbourg (Fr)Saada: http://astro.u-strasbg.fr/websaadaXCATDB: http://amwdb.u-starsbg.fr/jacds
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The ACDS
• Delivered Products
– FITS catalogue extractions• X-ray source counterparts• Archival sources in the XMM field of view
– Graphical products (PDF)• Finding charts • Rosat Images• Catalogue plots
– HTML product• Browsing pages for ACDS data with many links to CDS resources
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The ACDS
Vizier at CFA(Harvard USA)
Vizier at Cambridge(UK)
Vizier at CADC(Victoria Canada)
ACDS:• 2 Sparcs 400 Mhz• 1Gb RAM• 8 Agents in //
Vizier at CDS(Strasbourg France)
Simbad(Strasbourg France)
NED(Caltech USA)
Aladin(Strasbourg France)
Pipeline(Leicester)
• Correlation computing• A lot of remote server accesses• A few computation
• File product generation• Some remote server accesses (Aladin)• A lot of computation
• Correlation computing• A lot of remote server accesses• A few computation
• File product generation• Some remote server accesses (Aladin)• A lot of computation
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The ACDS
20%
72%
8%
Cross CorrelationGraphical Products Other Tasks
• Actual Throughput (8 process in //)– 3300 sec /dataset
– (8*3600*24)/3300 = 209 datasets/day
• Most part of the time is pure CPU (graphical prd)– IDL, Distiller….
– Easy to improve
• new hardware
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The ACDS
• Operation issues
– Support the pipeline throughput (but not more)• > 2200 datasets processed since October 13th
– 157 pending this morning due to a 2 days Aladin shutdown• No need to improve the hardware
– CDS database upgrades• Vizier is being moved on PSQL without serious consequences for us• Major Simbad to be release late 2006
– New error definition (no effect for standard users)
– External failures• Remote server shutdown• Corrupted products returned (cache miss-synchronization after crash)
– Internal failures• Empty Rosat images/wrong PPS index• Hardware failure
– Spare available/never used• Miss-configuration• Holidays
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The ACDS
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The ACDS
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The XCATDB
• The XCATDB is not a data repository but a tool to make science
• N to M persistent relationship– Difficult to implement in an efficient way for complex queries– Possibility of doing more than with dynamic cross-matches
• Data-mining feature
X Source 1 Arch Source 1
Arch Source 2
Arch Source 3
Arch Source 4
Arch Source 5
X Source 2
X Source 3
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
Saada in a few Words
• Saada: An Astronomical Database generator– Making automatic the process
of building a database
– Hosting heterogeneous datasets
– Highlighting scientific content
– Publishing personal data into the VO
• Databases are installed on local machines– Any Linux/Windows/Mac box
– Tomcat 5.xx
– PostrgreSQL 8.xx
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
The XCATDB
• Based on Saada– Queries expressed in SaadaQL– Data loader
• How to build quickly millions of correlation links• How to restore archival source uniqueness
– GUI• Layout inspired from the 1XMM XCatDB• Architecture based on native Saada GUI
• Prototype based on RC2 data open for the community– http://amwdb.u-strasbg.fr/jacds
Source X
Source X
Source X
Arch Source 1
Arch Source 2
Arch Source 1
Arch Source 1
Arch Source 2
Source X
Source X
Source X
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
2nd Release of the XMM Catalogue
150,000 X-ray sourcesIn 2800 observations
1,400,000 archival sources
1,000,000 links
Implemented with Saada relationshipsSpecific Module
Native Saada Data-loader
Native Saada Data-loader
Demo
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
A simplified example: Looking for High Mass X-ray Binaries
Hardness ratio range for X-ray binaries
Hardness ratio limit for stars
HR3 = (F2-4.5keV – F1-2keV) / (F2-4.5keV + F1-2keV)HR4 = (F4.5-12keV– F2-4.5keV) / (F4.5-12keV + F2-4.5keV)
HR3 = (F2-4.5keV – F1-2keV) / (F2-4.5keV + F1-2keV)HR4 = (F4.5-12keV– F2-4.5keV) / (F4.5-12keV + F2-4.5keV)
Demo
< 1
.5ke
V
kT =
5 –
50k
eV
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
HR Condition
With bright counterparts
Without WR counterparts
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
Using UCDs in queries
• Selecting X-Rays sources by correlation patterns on 200 classes of X-Ray sources requires to handle a lot of meta-data
select oidsaada from Arch_9037AEntry where ( _Flux > 1e-13 )select oidsaada from Arch_7181AEntry where ( _Flux2 > 1e-16 ) select oidsaada from Arch_9032AEntry where ( _FX > 1e-16 ) select oidsaada from Arch_9015AEntry where ( _fX > 1e-16 ) select oidsaada from Arch_9031AEntry where ( _Fx > 1e-13 )………….
[phot.flux;em.X-ray] > 1e-16 [W/m2]
• Solution: Expressing queries using UCDs and Units
Demo
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
(att1>x1 and att1<x2) or (att2>x1 and att2<x2) or (att1>x1 and att2<x2) or (att2>x1 and att1<x2)
UCD Issue
[UCD]>x1 and [UCD]<x2
Usually makes no sense
Solution: Using unambiguous operators: [UCD]In(x1,x2)
(att1>x1 or att2>x1) and (att1<x2 or att2<x2)
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
UCD Issue (continued)
• Query language issue: using a simple syntax– One operator for one UCD
• Query execution– Editable execution plan
• UCD attribution: a sensible issue– Using multi-words UCDs as often as possible– Limit the columns accessible per UCDs– Take care with units
• Existence• Name: known by the converter
• Result presentation: another sensible issue– The client must understand how resources have been selected and accessed
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
XCATDB: Status and Prospects
• Data loader– Correlation setup works fine– Other products can be loaded with the native data-loader
• GUI– HTML form improvements– Display of constrained values
• Interoperability– External Xmatch in SaadaQL (F.X. Pineau)– Connection with ESAC resources (XSA, VOSpec)– Connections with CDS clients (VO portals)– VO protocols
SAG Meeting – Postdam – November 2006 - The XCATDB: run by Saadahttp://amwdb.u-strasbg.fr/jacds
Thank You for your Attention