hifi user and expert-provided data products in the...

HIFI User and Expert-Provided Data Products in the HSA

D. Teyssier ESAC 28/10/2016

D. Teyssier | Exploiting the Herschel Science Archive: HIFI HPDPs | ESAC | 28/10/2016 | Slide 2

Outline

1.  Reminder: Products from the Standard Product Generation (SPG) 2.  User-Provided Data Products (UPDPs) 3.  Highly-Processed Data Products (HPDPs) 4.  Ancillary Data Products (ADPs) 5.  Product availability status


Standard Product Generation (SPG – aka pipeline data products) – a reminder

Ø  Data Products generated by the respective instrument pipeline software ●  Automatic product generation – no fine tuning ●  State of the art output as of applicable HCSS release ●  Best-ever pipeline products will be achieved by HIPE 14.1 for HIFI

(with about a dozen of observations revisited in HIPE 14.2) Ø  Their data quality, but also limitations and caveats from a science-readiness

perspective, are described in the Legacy Calibration presentation Ø  Served exclusively through the HSA

●  All processing levels are made available, allowing ad hoc reprocessing by interested users (only possible within HIPE)

Ø  See also previous presentation on “HIFI pipeline products and their science-readiness”


User-Provided Data Products (UPDPs)

Ø  Up to now, five Key Programme time holders of HIFI data have delivered UPDPs (more deliveries expected in the future)

●  HIFISTARS (P.I. V. Bujarrabal): high-resolution spectroscopy of the envelope of evolved stars

●  HOP (P.I. P. Goldsmith): survey of the O2 oxygen line in various ISM environments

●  GOT Cplus (P.I. W. Langer): Survey of the [CII] line on the Galactic Plane

●  HEXOS (P.I. E. Bergin): Spectral survey and mapping of outstanding Galactic Sources (Orion and SgrB2 regions)

●  PRISMAS (P.I. M. Gerin): Study of the diffuse ISM Ø  UPDPs can be downloaded as a whole, or on an obsid per obsid basis

(filtered query) ●  Global repository of data and release notes can be accessed at:

http://www.cosmos.esa.int/web/herschel/user-provided-data-products


Beyond SPG: Highly-Processed Data Products (HPDPs)

Ø  As explained on the first workshop day, part of the HIFI pipeline products are not strictly science-ready – we estimate overall that ~20% of HIFI data are still concerned by residual artefacts (~2/3 on point-mode observations)

Ø  Numerous dedicated software tasks and “useful scripts” exist in HIPE in order to circumvent those caveats and allow archival users to improve the products quality to full science-ready level (see e.g. previous presentation and demo script)

Ø  Beside, there exist HIFI products buried deeper inside the Observation Context that could be of interest to the archival user, but would not be served as part of the stand-alone browse products

Ø  To cover both needs, the HIFI team has identified a set of so-called Highly-Processed Data Products (HPDPs), that will be served by the HSA in a similar fashion as the UPDPs

http://www.cosmos.esa.int/web/herschel/highly-processed-data-products Ø  Some of those HPDPs are still in the making - the expectation is that about half

of the non-science-ready products could be covered by that effort


Contemplated HIFI HPDPs

Ø  The Herschel Science Archive should eventually serve the following HIFI Highly-Processed Data Products:

•  Baseline-corrected Spectral Scan observations •  Optimally-regridded and baseline-cleaned cubes for mapping

observations, with priority given to large coverage, bands 6-7, and fields of view observed repeatedly (merging)

•  A database of extracted OFF-position spectra (otherwise available deeper inside the individual observation contexts)

•  A database of flux uncertainty budget tables for each individual observation

•  A line list catalogue of species detected and identified in the HIFI data – priority will be given to Spectral Scans – extension to Single-Point mode observations possible in 2017 – work curated by IRAP (FR)

•  Isolated observations not processable by the standard pipeline, and that will be generated with ad hoc processing by instrument experts


Baseline-corrected Spectral Scans (1)

Ø  Stand-alone browse products for Spectral Scans are deconvolved spectra resulting from the processing of all individual level 2 tunings

Ø  In the deconvolution process, all artefacts still in the data will be propagated to the output products as well – cleaning baselines on wide spectra can be then particularly challenging

Ø  The most efficient baseline cleaning approach consists in correcting the level 2 data first, to feed the deconvolution with a clean input data-set

WHY PROVIDE THESE?

SPG 14.1

SPG 14.1

After baseline cleaning

After baseline cleaning


Baseline-corrected Spectral Scans (2)

Main delivery: dataset (as of pipeline product)

Secondary delivery: baseline (continuum plus artefacts)

baseline-subtracted: dataset – baseline

baseline baseline-subtracted

dataset

WHAT WILL WE PROVIDE AS HPDP ?


Baseline-corrected Spectral Maps (1)

Ø  Stand-alone browse products for Spectral Maps are data cubes resulting from the regridding of all individual level 2 pixels on the sky

Ø  In the regridding process, all artefacts still in the data will be propagated to the output products as well

Ø  The most accurate artefact cleaning approach consists in correcting the level 2 data first, to feed the regridding process with a clean input data-set

Ø  On top of that, merging of H and V polarisation data is best done prior to regridding than by averaging individual H and V cubes, as this is the best treatment of pointing effects

WHY PROVIDE THESE?

ADDED VALUE CUBES

Ø  On spin-off of this effort will consist in merging into super-cubes the data from separate obsids having a meaningful overlap in space and frequency, again aleviating the needs for users to download level 2 products for that purpose



Ø  Cleaning work contemplates baseline correction, Electrical Standing Wave correction revision where applies, H/V polarization merging, optimised regridding – main priority goes to bands 6 and 7

Ø  For strongest lines in each obsid, generate integrated intensity maps Ø  For a limited set of cases, generate cubes with OFF-contamination corrected for

WBS-V (14 Pipeline product)

WBS Merged full range

WBS Merged masked




Ø  High-Processed Data Product pack will also contain merging of overlapping fields (within same observing programme, or not), and of observations repeated over the same field over the mission (mostly calibration data)

Overlapping field merging

OMC-1 [CII] 158 μm

Obsid 1342250415

Obsid 1342250412

Obsid 1342250414

Repeated observation merging

Orion Bar [CII] 158 μm

Goicoechea et al., 2015 P. Morris, priv. com.



Isolated OFF spectra (1)

Ø  OFF-position contaminated data will be frequent in particular in regions of semi-extended emission where the referencing scheme could not reach a strictly blank emission position on the sky

Ø  The most efficient way to correct for this is to reprocess data from level 1 with a modified pipeline algorithm skipping the reference subtraction

Ø  As an alternative, isolated OFF spectra can be computed (average of all reference spectra acquired over the observation), and can be used as approximative solution to correct for the line and continuum contamination (extra baseline cleaning is most often required on those though)

●  Those spectra are part of the standard pipeline output, however they belong to the calibration context and therefore are not part of the stand-alone browse products

Ø  The goal of the HPDP is to alleviate the need to either download level 1 and reprocess within HIPE, or download the calibration-specific products

WHY PROVIDE THESE?


Isolated OFF spectra (2)

Ø  Extract all isolated OFF spectra from calibration context as self-contained set Ø  The dedicated HPDP will also provide level 2.5-equivalent OFF spectra (i.e.

deconvolved for Spectral Scans, regridded cubes for maps when applicable)

Point-mode

SScan

DBS raster cube



Flux Uncertainty budget tables

Ø  Exist inside the HIFI observation context as a calibration product (tables) Ø  The dedicated HPDP simply extracts all tables in a stand-alone data pack

Ø  The tables provide a detailed flux calibration uncertainty budget, breaking down contribution into the total random uncertainty (aka SumUncorrelatedUncertainty)

Ø  It also provides details about whether the uncertainty varies within the HIFI frequency band of a given spectrum (in practice it very rarely does !)

Ø  Overall this is a product rather oriented towards expert users and is less relevant to a large fraction of the archival users

WHY PROVIDE THESE?


HIFI HPDPs: Line catalogues

Line list catalogue postcard

Line transition assignment table

Image credit: Lisa Benamati (IRAP, Toulouse, France)


Ancillary Data Products (ADPs)

Ø  Ancillary Data Products (ADPs) correspond to products not necessarily associated to any observation performed by the Observatory

Ø  Used as Herschel’s long-term preservation storage of items that may be needed for users of the archive, but also as source of information for future missions – as such a lot of engineering-oriented items are contemplated there

Ø  The ADP contemplated for HIFI are: ●  Calibration: Mars (prime calibrator) models, HIFI PSF (2D, 1D azimuthal

averages, Encircled Energy Fraction) ●  Software: on-board software images and release notes ●  Observatory: uplink-related products (mission database, etc) ●  Engineering: trend-analysis data from instrument HouseKeeping ●  Historical: Telemetry and Science data from pre-launch test campaigns

http://www.cosmos.esa.int/web/herschel/ancillary-data-products


Ancillary Data Products: HIFI PSFs

2D beam models 1D azimuthally-averaged profiles

Encircled Energy Fraction

BA

ND

1

BA

ND

7


Ancillary Data Products: pre-launch instrumental measurements

JPL catalogue

Unique dataset collected pre-launch: HIFI spectrum of CH3OH

~6000 spectra from laboratory gas-cell Covers ~22,000 lines of JPL catalogue


HIFI HPDP and ADP availability status

Product Name Timeline

Highly-Processed Data Products

Spectral Scans January 2017 Spectral Maps Optimised cubes November 2016

Merged fields April 2017 Uncertainty Tables Already available OFF spectra Already available Line catalogues December 2016 Ancillary Data Products

HIFI PSFs Already available

Mars Calibrator models Already available

Trend analysis database Already available

Instrument pre-launch measurements December 2016

Software and uplink products December 2016


QUESTIONS ?

Questions

hifi user and expert-provided data products in the...

Documents