firenze, 15-16 september 2006asac meeting- computing ipt1 alma computing ipt science software status...

Firenze, 15-16 September 2006

ASAC Meeting- Computing IPT 1

ALMA Computing IPTScience Software Status Update

Gianni Raffi, ESO

ALMA Science Advisory Committee 15-16 September 2006



Time Now

1.07 Computing Summary Schedule

CDR 4

(Data from IPS as _________)

CPA #1PARRR CPA #2

CDR 7 CDR 8

Computing IPT Management and Science Software Requirements

High Level Analysis and Software Engineering

Common Software and Archive

Executive, Correlator and Control SW

Schedule, Observation Prep, Offline, and Telescope Calibration

Software Integration and Test

3 Antennas Integrated

IR 4.1 (Single antenna commissioning)

IR 5.1 (Interferometer support)

IR 7.0 (Commissioning feedback)

IR 6.0 (CPA punch list)

R6.0R5.1R4.1 R7.0 R9.0

IR9.0 Full Science Support

R4.0

IR 4.0 (Single antenna modes)

R3.0

1st Antenna Integration

Interferometry at ATF

IR3.0

20072006 20092008 20112010 201241 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 2 3 41 2 3

Array Software >> Observatory Software >>



Requirements Management

Goal::

• Evaluate development progress with respect to requirements

• Provide reporting.

=> Done via Central database (DOORS) with requirements derived from all sources (science, technical, operational) that allows us to:– Manage requirements (addition, editing, deletion, prioritization, review)

– Assess the status of software development (reporting)

– Analyze the impact of requirements changes

– Coordinate requirement tracking for all computing subsystems.

• Complete functional team work and release development with integration tests (by ITS team)

• Validate implementation with user tests (by SSR subsystem scientists) following each release.



SSR Testing of Science Software

Extracted from D.Shepherd’s page on TWiki at:http://almasw.hq.eso.org/almasw/bin/view/Usertests/WebHome#Test_PlansRequirements available on SSR TWiki (unless explicitly mentioned below)

All functional subsystems shown for completeness (except ACS framework and ObOps)

• EXEC– R3.1 - Executive internal pre-test complete. – R4.0 - user test scheduled to begin October 20.

• Scheduler– R3.1 - Optical pointing test completed in March 2006 – R4.0 - Testing planned for early fall 2006. This will test the scheduling simulator, particularly

with weather simulation. • TelCal

– R3.1 - Features developed during this release tested August 21, 2006. Optical pointing reduction tests completed March 2006.

– R4.0 – holography reduction tests scheduled late 2006. • Archive

– R3.1 – Basic archive function tested during optical pointing integrated user test in March 2006.

– R4.0 – Tests of archive queries (projects, monitor point data) scheduled for integrated user test late 2006

– Requirements: Use Case development for Science Archive continues. Granular requirements for Archive starting to be produced. ALMA Science Archive (ASA) requirements about ready .

http://almasw.hq.eso.org/almasw/bin/view/Usertests/WebHome#Test_Plans



SSR Testing Status (cont.)

• Correlator– R3.0 - Testing planned for April, June, and July slipped to October 2006.

• Pipeline– R3.1 - End to end user test (from flagging to imaging) is planned for the pipeline in Fall 2006. – Quick Look: R3.0 - Operations test should be possible. It is not yet planned.

• ObsPrep– R3.1 – User test about to conclude.

• Control– R3.1 – Optical pointing features associated tested during integrated user test in March 2006.– R4.0 – Holography features scheduled to be tested late 2006 during integrated user test. – Requirements: adopting EVLA control requirements (modified for ALMA) for ALMA control

granular requirements. Passed System Requirements Review.• Offline

– User Test 4: R2.1 - Single baseline data reduction and analysis - supporting the ATF. Test complete, report written.

– User Focus Group: completed March 2006, provided input on python user interface.– User Test 5: Will test single dish reduction & imaging. Scheduled for Dec 2006 – date

dependent on ACA requirements.

Note: Control and correlator are exercised not only at ATF, but also every day in the Prototype System Integration (PSI) lab.



Function Based Team (FBT) development

In principle planning of R4.0 FBTs

• Works to functionality across subsystems (~2 months)• Based on Technical Use Cases• Interface issues clarified soon • Allows more frequent integration• Some parallelism possible



Latest Developments

• Concluded R3.1 release and integrated in April Includes Optical Pointing e2e functionality (tested in March at ATF) Installed and tested on both prototype antennas Installed also in the Prototype System Integration Laboratory in Socorro Included integrated regression tests

• FBTs so far: Optical Pointing (completed in March) Shared Simulation of Single Field Interferometry (SFI) (April) Monitor DB optimization (July)

On-going FBTs:– ALMA Project Data Model (APDM) re-factoring (Aug)

• Validation almost complete. Currently on hold behind higher priority Holography work.

– Holography (Aug). Hw for tests in the lab early Sept., at ATF late Sept.– Single Field Interferometry II (to be used at ATF just after first fringes- achieved

with PSI scripts)



OPT User Test (mar.’06) (1st integrated user test of ALMA software)

• evaluated by members of the Computing IPT (CIPT) in March 2006.• primary software limitations were found to be jlog instability and

slowness, lack of coherent logging levels across subsystems, archive access (query mechanism not documented at the level of end users), lack of access to archived monitor points, missing CCL offset commands, and uncaught exceptions (e.g. Control, TelCal) that caused software crashes.

• Many of the interfaces and system responses have been designed for the developer.

• Approximately 170 defects, issues, and enhancements were identified in this report. Roughly 40% (70) of those are defects (14 (8%) priority 1 and 56 (32%) lower priority).



Next Developments (*)

• R4.1 ( up to June 2007):– Single Dish (SD) Function Based Team (Feb.) (w/o nutator)

• TP mode 1: OTF mapping in continuum Total Power• TM mode 2: radio pointing measurements using TP

– Test SFI with h/w at ATF (completion of SFI II work)

• Readiness Review (external) by subsystem (May)• R5.0 ( up to Dec 2007)

– SD: Nutator, Spectral line (prepare for equipment in Chile)– Interferometry: WVR, fast switching, Pointed mosaics – Respond to AIV use of software in Chile

• PAR (Dec.07) – Delivery of modes developed thus far to Chile• During all of 2007 support in Chile for antenna:

– Acceptance and integration– System characterization using continuum TP

(*) Development plan will be adapted to availability of h/w CIPT formal user test will occur twice next year



Archive Software

Persistency support for raw data, pipeline results (images and logs), SBs and all other User input, site characteristics, monitor data, calibration data, catalogs, and all other auxiliary information required for ALMA to observe.

Quick look results are temporarily stored in the Archive.

Items not specific to ALMA Science Archive: User Repository: Manage database of registered users, support proprietary periods. • User query, retrieval and monitoring capability. (event data R4.0)• Data replication between OSF, Santiago, ARCs (both via network and physical

media). Use of NGAS supported data replication. Requires ALMA testing.

• Request Handler: Manage packaging, request tracing and delivery of data products to the scientific user community (implementation planned for Q4 2007)

High-availability system with failover (test implementation Q3 2007) Provision for safe backup of all data; when more than one Archive is operational this

may be accomplished via data mirroring between Archives. Support 6 MB/s average and 60 MB/s peak (sustainable for many hours at a time)

data rates.



Archive Replication between SOC and ARCs

Full description on Archive Twiki at:

http://almasw.hq.eso.org/almasw/bin/view/Archive/DatabaseReplication

1. Bulk data replication taken care of by NGAS.

2. Transaction based replication for Oracle DB OSF-SCO

3. Standard DB incremental backups sent via NGAS.

Brian Glendenning

Show a more interesting screen shot



ALMA Science Archive (ASA)

• Scientific view on ALMA observations through ALMA data products (cubes).

– Preview image, possibly via an interface to Science Pipeline.– Science Archive Requirements document was drafted and has been

discussed by the SSR. – Many use cases for the ASA have been drafted and were given to the

developers. (http://almasw.hq.eso.org/almasw/bin/view/Archive/AsaUseCasesList)• Support for storage and association of ‘advanced products’.

– Catalogue Support: offer access to catalogues (calibrators, spectral lines) and calibration database (R4.0)

• Consists of two major parts:– ASA portal: Main access point to all query and retrieval services.– ASA data repositories and ASA services.

• VO aware by sharing technologies and standards.– ASA VO registry (planned)– ASA VO SkyNode (planned)



ALMA Science Archive (ASA)

Operations Archive (Santiago)

Dataset

ALMA Science Archive

Commands

Archive Subsystem

State

Queries

Archive Users, PI, VO, Staff

ALMA Staff

Query Responses

Data

Processed DataQuality DataCorrelator Data

Reduction Pipeline

• Exercise use cases for each user type: • ALMA staff, Archive Users/PI and Reduction Pipeline

• Test interfaces (query structure, ease of use); transfer content (connections between APDM structures in ASA made & distributed correctly); & performance.

• Test data transfer from Operations Archive and ASA.



Support for ASA users Staff & PIs:

User repository development: manage database of registered users

Add a user to the repository:

Manager view of users:

User edits his/her contact information:



Archive table browser prototype

list of ALMA Science Data Model (ASDM) tables

entities table populated by

Correlator (actual contents not shown)



Observing Preparation

• Support creation of Phase 1 proposals and Phase 2 programs, with submission through the ALMA archive.

– “APDM” refactored (R4)– Visual Spatial Editor – first version. Visual Spectral Editor also exists. Final versions of both for R6.– Support for Director’s time (R5)– “Special SBs” for fixed period scheduling now available. (R4)– Phase I “bureaucracy” and textual cases (R5).– Phase I – II transition (creation of Phase II from approved Phase I) (R5)

• Support all ALMA Observing modes– So far optical pointing and holography observatory modes, and Single Field Interferometry

(SFI) observing mode. – infrastructure in place to allow rapid development for further modes, R5-6 and on.

• Provide a mode which operates in terms of science goals with automated generation of SBs.

– Science goal-based input for Phases I and II – First “wizard” now available for SFI. Will expand as modes are developed.

– A generic framework for science input collection and SB generation is in place.– Alternative science goal presentation (“workbooks” - more flexible for users than a wizard)

under development – R4.– Almost complete infrastructure will allow science goal input and SB generation to be

extended to more observing modes. R5-6 and on.



Observing Preparation (cont.)

• Exposure time and data rate calculators shall be provided. – ETC complete – in maintenance. Supports all arrays.– Data rate estimator first version available, requires more user friendly

presentation (R5.1)– Execution likelihood estimator (R5)

• Sensible default values and basic validity checking will be provided. – Infrastructure to support validation and setting of (sensible) defaults is in place. – Needs to be rolled out for all items (R4.1)

• Provide an expert mode with access to technical parameters of interest to observers (e.g., detailed control of correlator configuration).

– Almost complete for main array – in maintenance as other specs emerge. Still details to be determined for ACA/TP arrays. (R5)

• Local disk persistency (provided by MicroArchive). In maintenance.• Source list and catalog (target and spectral) support

– Infrastructure in place with interim catalogs.– To be moved to Observatory Characteristics (R4)– Final catalogs in place by agreement with Operations (R6?)

• Widely portable software. – Supported now, will continue.



Spatial Visual Editor



Observatory Operations Software (ObOps)

Passed its PDR at CDR4 (June 2006). Still at .5 FTE/yr effort

• Infrastructure: Ops Support Database, Data Access Module, etc. (R4-R9)• Shift Log Tool: record all day- and night-time observing activities

– core features (R4) – basic features (R5) – full implementation (R6)

• Observatory Characteristics Database GUI (R4.1)• Phase One Manager: support for proposal review process (R6)• ProjectTracker: control and monitor operations information flow (R6)• User Portal: integrated access to operations info

– internal (ALMA only) access (R7)– public access (R9)

• QualityTool: data quality assurance (R7)• DataPacker

– raw data delivery (R7)– data package production and delivery (R9)

• PublicationsDatabase (R9)



Offline software

• Priority 1 requirements. In particular for the next release:– Single dish analysis, High dynamic range imaging (R4.0)– Standard calibration (R4.0)– CASA framework doc.

• Portable to a range of Unix-like operating systems.• Efficient enough in operations to satisfy user expectations and for use with the Science Pipeline.• GUI ,CLI, and script interfaces (Image viewer & data visualization GUIs available)• Readily installed by end users for processing execution. GUI tools for raw data and image

inspection, flagging, arbitrary plotting.• Filler for ALMA Export Data Format for Interferometric data (single dish ASDM filler to be

developed when ASDM format defined). – ASDM extensions, EDF doc (R4.0)

• Extensive support for multi-channel, multi-polarization, multi-spectral window, multi-pointing flagging, calibration, imaging, and analysis for interferometric, single dish, and combined data from existing telescopes.

• Simulation capability.• Processing history, interactive warnings, and re-execution facilities.• Direct access to the ALMA Archive and the VO. Atmospheric modeling for the ALMA bands• Extensive user documentation



CASA

AIPS++– Glish interface; unknown,

unsupported outside of NRAO– Tasking system based on

Glish– GUI system based on

Glish/Tk; limited widgets, not robust!

– Difficult for external developers to contribute

– Multi-CD binary distribution– Large monolithic libraries with

cross dependencies– No namespace

– Freeze 2006.75

CASA– Python interface (community

standard); IPython extensions– Direct binding to Python; ACS,

other frameworks readily possible

– Hierarchical set of small libraries with clearly defined dependencies

– Namespace protection for integration with other code

– RPM distribution mechanism; auto-updates possible

– Robust– Inherits all application code

improvements in robustness and performance.



DataCapture

DataCapturer

Control

CORBA-IDL Structs

TelescopeCalibration

CORBA-IDL Structs

QuickLookVisualization

CORBA-IDL Structs

Correlator

CORBA-IDL Structs

CORBA-IDL Structs

XML-Entities

ALMA Data Archive

Telescope Domain Science Domain

ASDM data product

Overview of the DataCapturer component and its interactions to other Subsystems in the framework of the ALMA Observing software

DataCapturer

Control

CORBA-IDL Structs


CORBA-IDL Structs


CORBA-IDL Structs

Correlator

CORBA-IDL Structs

CORBA-IDL Structs

XML-Entities

ALMA Data Archive


ASDM data product

DataCapturer

Control

CORBA-IDL Structs


CORBA-IDL Structs


CORBA-IDL Structs

Correlator

CORBA-IDL Structs

CORBA-IDL Structs

XML-Entities

ALMA Data Archive


ASDM data product

Overview of the DataCapturer component and its interactions to other Subsystems in the framework of the ALMA Observing software

Maps telescope, observing information into Alma Science Data Model (ASDM) for post-processing in near-real time.



CASA Timeline

2006– NRAO: NAUG testing (AIPS++)– Community: Project tests– ALMA: external science testing:

+single-baseline commissioning test) (CASA)

– ALMA: external science testing: +single dish reduction (CASA)

– ALMA: commissioning support (CASA)

– ALMA/EVLA: user interface review (CASA)

– EVLA: external science testing: +full polarization imaging; antenna pointing calibration

– 2006.5 AIPS++ frozen2007

– NRAO: User support (CASA)– ALMA: commissioning support

(CASA)

– EVLA: external science testing: + wide band calibration imaging (CASA)

– ALMA: P1 SSRs complete2008

– EVLA: external science testing: RFI (CASA)

– NRAO: user support (CASA)– ALMA: commissioning support– EVLA: P1 SSRs complete

2009-2010 – ALMA/EVLA: commissioning

support– Community: CASA

released/distributed for early ALMA/EVLA science

2011 – Community: CASA

released/distributed; full user support (ALMA/EVLA: P1/P2s)



Offline User Test 4 (report Apr.06)

Test delayed by SSR (not developers) so the first framework changes and python commands could be evaluated in a realistic environment of single baseline interferometry.

CASA is a python binding of the C++ functionality that was in AIPS++. This initial version of CASA had python commands that had a one-to-one correspondence with

AIPS++ Glish commands.

• first test of the CASA infrastructure and python interface.• focus was to exercise and verify functionality for

– editing, visualization, data reduction and analysis for single baseline interferometry –– capabilities that are needed at the ALMA Test Facility and for commissioning support.

• Testers with extensive experience commissioning radio telescopes and/or operating telescopes.

• In general, single-baseline data reduction procedures worked with good stability and accuracy.

• Additional visualization capabilities to support commissioning activities still needed.



Offline Development & Test Schedule

2003 2004 2005 2006 2007

Single FieldVLA & PdBI13mm 1/3mm

User Acceptance

Tests

Small Multi-field Mosaic

Interferometer/TP combo

Single-BL Commissioning at ATF

Incr

ease

d ca

pabi

lity

VLA & BIMA mosaics

Co

mp

lete 1st Python interface, new framework

Single dish reduction & imagingPython tasks, new framework

VLA+GBT & BIMA+12m

Community use, command line pythonLimited release, mostly command line, some GUIs

User Focus groupTools Tasks

Delay: late arrival of ACA requirements.

Delay: SSR decision to allow

testing of 1st python interface.



Offline Key Planned Development for 2006/2007:

• Commissioning Plan shows heavy use of the offline package.– Change development priorities/details (e.g.: single baseline analysis software must be ready

earlier than expected; more plotting capabilities needed). – Completed commissioning support test in February 2006 (1st step before using ATF)

• Single-dish reduction & imaging – will be based on ASAP package– Single-dish user test scheduled for late 2006 (dependent on ACA requirements). – Also evaluate task-based reduction & imaging for interferometric data.

• Complete ALMA interferometric ASDM filler and new viewer (more robust).

Formal tests have been limited to specific datasets to verify development along a defined path. This has been necessary for development but need to expand to more open use once python command line interface is stable –

2007 – 2009: Staged plan to release CASA to astronomers for scientific use.

Not final yet, however, key components are:

• Nov 2006: CASA user point person starts (guide user interface development, documentation & help new users)

• 2007-2008: Increase number of NRAO scientists & a few external scientists reduce own data• Mid 2007: Early release of CASA simulator• Mid-2008: Release to the public.



Science Pipeline Goals

Automated flagging, calibration and imaging with associated quality information for common ALMA observing modes. Pipeline processing

shall be data driven).Main purpose is to produce: calibrated, scientifically useful images.

• Following information will be calculated or available for use in pipeline processing: sideband ratios, temperature scale from an atmospheric model, bandpass calibration, instrumental polarization, phase rms, antenna efficiencies from source with known flux, and time variable amplitude and phase curves. (heuristics computing bandpass, gain, and flux calibrations for single field interferometry data, (R3.1->)

• Choose: whether or not to use WVR corrected if both are available, to reuse previous calibrations if necessary, to select appropriate deconvolution algorithm, to do self-calibration or not, to do continuum subtraction or not (Choices non-existent or hardcoded, (R4.1->).

• Automatically assemble all needed information from the Archive (i.e., assemble appropriate sources from multiple SBs over multiple epochs / configurations). (automatic ASDM retrieval for observing modes with simple observing unit set structures supported, (R2.1->)

• User friendly processing summary and processing script which can be modified and used in later reprocessing. (processing summary logs are written by current pipeline heuristics scripts, automatic script logging may be available from CASAPY, (R4.0->)

• Scripting interface to allow operations to modify processing heuristics without changes in compiled code. (Scripts will be written in Python, (R4.0)

• Manual mode (e.g., for reprocessing particular projects). (R5.0->)• First version of pipeline processing scripts. (E2E for: single field interferometry (R4.0->), for: single

dish: (R4.1->)



Status: Heuristics

• Single field interferometry (prototype e2e heuristics)– calibrator source baseline based flagging heuristics– bandpass, gain, flux calibration solution heuristics– calibrator source imaging heuristics– some quality assessment measures– calibrator source antenna based flagging heuristics– target source flagging and imaging heuristics

• Single dish (baseline fitting heuristics)– including line detection and masking heuristics

• Software– general Python classes for flagging, calibrating, and imaging SFI data– general Python classes for baseline fitting (some cleanup required)

Heuristics development slower than expected (based on 2004 planning)

Reasons: Technical: CASA framework conversion happening at same time;

Developmental: more difficult to get flagging heuristics correct than expected;

Personnel: ramp up to expertise needed to develop interferometric heuristics slow.



Status

Pipeline Infrastructure • Quicklook

– supports parallel observing sessions (independent subarrays)– provides clean separation of processing and display functions– working telescope calibration results monitor and GUI

• Science– working heuristics development framework (IPython, CASAPY)– working set of data processing engines (CASA, ASAP)– heuristics not yet connected to infrastructure– ASDM filler prototype– separate ALMA / CASA build environments

Modules• full set of tool methods available to heuristics from CASAPY for

– table, ms, constants, quanta, autoflag, calibrator, and imager• subset of tool methods available to heuristics from CASAPY for

– image, measures • prototype visibility display tool already available to heuristics from CASAPY

– msplot• Java based table browser available in latest stable

– to be integrated with CASAPY



Heuristics User Test 3 – Browser Snapshots

PdBI Spectral line

From R3 User test conclusions (mar.’06):The quality of the automatic flagging was judged to be good by the majority of the testers and the calibration heuristics were found to be a reasonable first implementation



Pipeline - What is ahead for next year ?

• Science

– E2E single field interferometry heuristics running in the ALMA environment

– E2E single dish heuristics

– prototype cluster computing environment

• QuickLook

– user friendly telescope calibration monitor and display

– real time visibility display using real time filler

– some bad data detection heuristics integrated with alarm system



Dynamic Scheduling

Software to automatically choose best SB given scientific priority, current environmental conditions, current and future configurations, and the total

pool of possible observations

• Complete interactive - and queue - scheduling (R4.0) (R4.0). • Planning mode simulator (incl. weather data and quality - R4.0) (R4.0)• SB interruption and re-execution. (Special SBs for fixed-time scheduling - R4.0) (R5-6)• Manual override of automated choices. (R4.1)• Mechanisms for operations to change the weight of factors within the dynamic

scheduler without significant programming. (R4.1)• Scheduler reporting facilities. (GUI for master scheduler, Exec plug-in - R4.1) (R4.1)• Heuristics delivered by CIPT will require optimization by operations. (R5)• Multiple sub-arrays supported (including taking antennas off-,on-line - R4.1) (R4.1 )• ACA projects operating independently (R5-6)• Medium term scheduling (R5-6) • Long term scheduling (R7-8) • Receiver setup optimizations (R7-9) • [Breakpoint facility – low priority.] (R7-8)• Accounting of time spent for various purposes (source, programs, observatory

calibrations). (R?)



Operator Master Client

Scheduler



TelCal Online (and Offline) Calibrations

Telescope Calibration takes scans (mainly special calibration scans) and performs data reduction on them.

Results are used by Control for further observations with improved parameters and by the Quick Look Pipeline to display and calibrate quick look data.

Some array calibrations are not required to be performed on-line (e.g. Holography, pointing models).

• Holography (Pointing & focus offset, illumination pattern, surface err.) (R4.0)• Atmospheric calibration (Temperature scale) (R4)• WVR values (R5)• Sideband ratios (R4)• Phase calibration (phase rms, calibration curves) (R4)• Amplitude calibration (calibration curve) (R4)• Bandbass calibration (R5)• Reference pointing (publish/reduce scan results) (done)• Focus (partly done)• Delay (publish/reduce delay scans) (R4.0)• Pointing Model (optical done) • Baseline measurement (positions from delays) (R4.0)• Skydip• Interferometry solvers (done)• Automatic activation (according to intent) (R4.0)

firenze, 15-16 september 2006asac meeting- computing ipt1 alma computing ipt science software status...

Documents

end user test

user tests

offlineuser test

operations test

optical pointing test

impact of requirements

requirements addition

computing subsystems