sas development, maintenance and future - … · sas development, maintenance and future ... •...

SAS development, maintenance and future | Carlos Gabriel | XMM-Newton Users Group meeting - ESAC, 7/8 June 2016

Carlos GABRIEL (*)

XMM-Newton Science Operations Centre – ESAC / ESA

*on behalf of a lot of people (SAS & PPS Team, SAS WG, ...)

SAS development, maintenance and future


SAS & PPS development and maintenance

MPE: PN s/w - (2) AIP: src det s/w - (1) GOF: esas - (2) IRAP: pps s/w - (1) Strasbourg: ACDS s/w - (1) IFCA: screening s/w - still providing support CEA: MOS s/w - only support since 2009 LUX: ended June 2013 (1)MSSL: OM s/w ended March 2014

SAS CCBSOC (5) + SSC (3)

SAS CAL filesPPS

SAS WGCal Teams

SOC SAS&PPS

SSC SAS&PPS

XSA

Team of 8 (~ 5FTEs) SAS ~ 3 FTEs PPS ~ 2 FTEs


SAS development & maintenance

Main SAS upgrades contain• CAL changes - improvements in instrument calibration• algorithmic changes - and/or better analysis • graphical I/F changes - better I/Fs (or eg. changes in parameters)• changes due to OS / libs / compiler - forced evolution

A typical yearly release implies• many new task versions (~ 102) - despite age of SAS• few new tasks (~ 100) - not growing much anymore • scientific validation - quality assurance always needed• preparation and distribution of SAS - making it as easy and secure as possible to users

Further work• analysis of Helpdesk questions - continuous maintenance, evtl. bugs found• documentation maintenance & upgrade - threads, watch-outs, ...• follow-up measures - patches, watch-outs, individual help


SAS 15.0 released on 4/2/16 - Main changes

GENERAL- correction of the conversion between celestial and detector coordinate systems.

Euler Ψ angle with wrong sign in attcalc >> correctedSeveral tasks affected + new CCF (XMM_BORESIGHT_0026.CCF)

EPIC- new task ebkgreg for determination of optimal background region for a given source (PN)- new task eupper for fast calculation of upper limits- upgraded eboxdetect, accepting now also input in detector coordinates- the metatask eslewchain is now producing .png files for each subimage

RGS- upgraded task rgsrmfgen, able to deliver responses in separated RMF and ARF files

+ many fixes and small improvements: 72 tasks upgraded, with 129 sub-version changes


SAS 15.0 released on 4/2/16 - validation

Thorough validation necessary (“(4+1)way”):

• standard set through procs & chains > IA• X-cal DB fully reduced > comparisons• standard set through test PPS (SAS 15 based)• strict data reduction with the Threads+• dedicated data reduction for new / special S/W

>> several people involved in 4 weeks process+ specific time for validation of new attcalc, special effort by the instrument teams

>> Final report still to be produced

>> CAL-SRN 332 on BORESIGHT_0026

Extra this time: - tests of single released 32bit version in different OSs, both automatic and interactively


SAS validation - conclusions

Everything else validated but:

• use of modified embadpixfind for RGS bad pixel / column finding NOT validated (it needs

parameter space assessment) >> withdrawn from SAS 15

• Cross check of 32bit “single” version from other 32bits OSs OK ... but later Perl problems arising >> diverse 32bit versions released


SAS validation - rgsembadpixfind

Processing of

AB Dor: 35 observationsHR1099: 33 observationsCapella: 27 observationsMrk 421: 64 observations

Default rgsproc reduction with a) rgsbadpixfind b) rgsembadpixfind

Fit model, compare valid bins and fit goodness...

Mrk 421- Continuum - No differences at all

Motivation: SPR on wrong filtering of RGS bright emission lines


SAS validation - rgsembadpixfind

Fitting simultaneously RGS1 and RGS2, 1st and 2nd order in all 35 observations

• In almost all cases, less columns (typically 100) discarded• In almost all cases, smaller Cash/DoF, ie. good fitting of non-discarded columns ... BUT 3 cases with non-discarded hot columns

>> need assessment of full parameter space of new task rgsembadpixfind

☐ ☐

AB Dor data: use of APED model with 3 T’s and free abundances - fitting with ISIS (based on Sanz-Forcada, J.; Micela, G.; Maggio, A. 2007)


Patch 15.0.1 released in May

• preqgti had been kept (wrongly!) as “SOC only”, therefore not part of the public SAS 15 release- also necessary for explaining to users how to deal with multi-pointing mode in a watchout

• esas (v0.9.34) - including missing task merge_comp_xmm

• xmmextractor (v1.4.1) solving various small incompatibilities with SAS 15 tasks

• emosaicproc (v0.4.1) corrects typo in a parameter of emldetect call

• ommergelists (v1.0.5) solves problems with duplicated entries from mosaiced sources

• omsrclistcomb (v2.22.2) OM windows assignment fixed


SAS 15 release - binaries

Linux 32:

Linux 64:

MacOS:

RHEL 5.8 2.6.18 2.5

Ubuntu 14.04 LTS 3.13.0 2.19

Fedora 20 3.19.8 2.18

SuSE 13.1 3.11.6 2.18

MacOS 10.8.3 (M. Lion) Darwin 12.6.0 169.3.0

MacOS 10.10.5 (Yosemite) Darwin 14.5.0 1213.0.0

+ 2 universal SAS-VMs (32 & 64bits) - Ubuntu14.04.1

Already in SAS 14

New in SAS 15

RHEL 5.8 2.6.18 2.5

Ubuntu 14.04 LTS 3.13.0 2.19

Fedora 20 3.11.10 2.18

SuSE 13.1 3.11.6 2.18

RHEL 6.6 2.6.32 2.1

OS Kernel libc


Collecting information with every SAS release - SAS 15.0


GENERAL- migration to gfortran > ending dependency from NAG- graphical I/F for xmmextractor > new type of I/F due to special characteristics of task- study on replacement of PGPLOT by modern package > better output quality (same plotting as Chandra?)

EPIC- esas general refactoring > re-written in F90 + CAL-DB into normal CCFs- task for EPIC internal background event files > image and spectral correction- task for thumbnails incl. src & bck extraction reg. > fast diagnosis- diagnostic X-ray loading images from epproc > assess X-ray loading level

- refinement of pile-up corrections > optimisation / depends on study going on- source detection in slew data processing > slew pipeline providing source detection

- detector map to correct for bad areas in ext. sources > extended source analysis - time dependent PN quiescent background estimation > better gain correction- emosaicproc and emosaicprep upgrades > make data combination more user friendly - alternative stacked source detection task > better src detection on stacked data - time dependent PN quiescent background gain correction > use of instantaneous discarded line rates- different PSFs for different observing modes > to achieve calibration goals for timing / burst modes

Future main SAS upgrades


Future main SAS upgrades

RGS- finish / validate new bad column finding task > avoiding bright lines loses due to misidentification- new RGS1/RGS2 eff. area correction > cross calibration improvement- correct treatment of multi-pointing observations > solve problems of single observations in multipointing- spatial imaging of emission lines from ext sources > narrow energy range images > less background than EPIC

OM- NONE by default ... but some flags should be extracted and added to the catalogue file, output of omcat


Compilers

Migrating to gfortran

> major change, will free us from NAG - main change for SAS 16, it will require substantial tests

Integration tests are running - several tasks already adapted >> our goal is to be ready by September


Remote Interface for Science Analysis (RISA)

JOBS

1

2

4

5

6

7

8

9

SERVICE

REQUEST

SIAP and CDS query

VO

ESP

AC

E

GRID

XSA REQUES

STORAGE REQUEST

SERVICE INFO

RESULTS

VOSpec

10

•Process SOAP (VOTable)•Unique REQID•Perl Script•Grid Template•Job Submission & Status Info

3

Aladin

( )

UNIVA Grid Engine

RISA is ready to run...


RISA upgraded to process “light-services” from XSA:★ Spectra ★ Light curves ★ Images★ Source detection (>> Spectra, Light curves, ...)

RESTful syntax: http://<InternalServer>/RISA_sl? product=spectra& username=<XSAUserName>& obsid=0650560401& inst=EPN& exp=S003& target=Mkn501& srcreg=((X,Y) IN circle(25339.5,23998.5,328.9302))& bkgreg=((X,Y) IN circle(22491.5,26702.5,647.65702))& mail=<XSAMailAddress>

ds9 has been upgraded (there were security problems impeding its usage in XSA)

RISA web services to be used from XSA


Futuredepending on feedback...

• New functionalities• User´s defined filtering• RGS and OM

• New “light services”• Combine ODFs

• Open RESTful services to the world

•…

Reprocessing capabilities

sas development, maintenance and future - … · sas development, maintenance and future ... •...

Documents