National Radio Astronomy ObservatoryJune 13/14, 2005 EVLA Phase II Proposal Review
EVLA Phase IIComputing Development
Bryan Butler (EVLA System Engineer for Software)
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
EVLA Software
The primary goals of EVLA software are:
• maximize the scientific return of the EVLA;
• be easy to use, for all astronomers;
• provide a look-and-feel and functionality which is consistent with all NRAO telescopes.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
EVLA Phase II Computing
The EVLA Phase II computing effort:
• Develops software, by the end of the construction project, which can be used for accessing or operating those parts of the EVLA telescope included in Phase II (E configuration; NMA; use of WIDAR for VLBA correlation);
• turns this software over to Operations - there is therefore a close relationship between software development (by the EVLA Phase II project) and operation, maintenance, and upgrade (by Operations);
• Mostly a simple extension of EVLA Phase I software effort.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
EVLA Software - Methodology
1. Scientists deliver use cases and requirements;
2. Overall design developed based on these;
3. Each subsystem with the overall design then developed in
greater detail;
4. For all three of the above, iterate with short cycles (as short
as a week, depending on the particular item).
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Overall NRAO Design (e2e)
Telescope Data Model
Export Data Format
Science Data Model
Feedback to telescope
Proposal SubmissionAnd Handling
Observation Preparation
EVLA VLBA ALMA
GBT
EVLASched
EVLAControl
ALMASched
ALMAControl
GBTSched
GBTControl
DataCapture
Archive
Telcal
Offline VO
ObserverDomain
Mostly Telescope-Independent
Common Software
VLBASched
VLBAControl
Quick Look
Pipeline
GBT Postproc
TelescopeDomain
ScienceDomain
Mostly Telescope-Specific
Project Software
Mostly Telescope-Independent
Common Software
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Overall EVLA Design (1)
Observation Preparer
Ast
rono
mer
Default Program Block(with ‘suggestions’ filled in)
One Program
EVLA ObservingHeuristics
Program Block(Set of Scheduling Blocks for one Program)
Refinements
Proposal Preparer
To Observation Scheduler
ObserverDomain
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Overall EVLA Design (2)
TelescopeDomain
ObservationScheduler
ObservationExecutor
Hardware M&C
Next SBExecutionState
Equipment State
Metadata to DCAF
Operator
Environment
From Observation Preparer
Results from TelCal
Sequence of ConfigurationsAntenna Delays
Archive
Archive
Operator
Raw Visibility Data
Properties
Data Addressing InfoEquipment State
Archive
Operator
Heuristics
Metadata to DCAF
To DCAF To Archive
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Overall EVLA Design (3)
ScienceDomain
Quick Look Pipeline
Astronomer
Observation Monitor
Image Cubes
SDM
ArchivePost-Processing
Image Cubes
Astronomer
Default Image Pipeline
Image Cubes
Data CaptureAnd Format
From M&C
TelCal
To DCAF & Executor
SDM
Archive
Archive
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Phase II Specific Developments
The major developments for EVLA Phase II are:• Additional hardware to support much higher
archive data rates;• Software to schedule the WIDAR correlator to
support VLBA recorded media, and simultaneously correlate EVLA, NMA, and/or VLBA;
• Post-processing system development in support of the much higher resolution full EVLA and NMA.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Archive Data RatesExpected data rates are determined by computing the dump time necessary to avoid time-bandwidth smearing. We know however that we cannot support the highest rates immediately (in 2014), so we have a staged plan, to allow storage (and processing) speeds to catch up:
We do not expect these rates to be difficult to support (they are quite conservative), given current storage capabilities and expected increases, and in fact we may increase at a faster rate.
DateMax Data Rate
(MB/s)
Total Volume
(TB/yr)
2008 (Phase I) 25 75
2014 500 1500
2017 1600 4500
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
SchedulingAt the completion of Phase II we will have 37 antennas which must be flexibly scheduled in various combinations. The software must support this, which will require changes to most of the subsystems. All are minor, except the Observation Scheduler.
In addition, we plan to use the WIDAR correlator to correlate VLBA disk-recorded observations, and to concurrently correlate combinations of all of the possibilities. This will require development of new software, including a Correlator Scheduler subsystem.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Post-ProcessingEffort in several areas of post-processing is needed, but much of this development is common with either ALMA or EVLA Phase I or both. The primary issue is the sheer size of the databases and the computational load. We estimate the following CPU power necessary:
This is not trivial, but we do not see it as a major problem.
DateAverage CPU rate
TFlopnormalized*
compute power
2008 (Phase I) 0.5 1
2014 10.0 0.6
2017 32 0.5 * - normalized by assuming doubling every 18 months (Moore’s Law), and
scaling to the 2008 compute power. This implies 125 GFlop right now.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Post-Processing
Recent changes:• major reorganization;• rewrite of code base (CASA) and interface
(Python);• development driven by ALMA and EVLA
deliverables;• ALMA already incorporating outside users
in testing, EVLA planning to do so soon.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Post-Processing - DetailsUse of a common post-processing package is the baseline plan for all three of
ALMA, EVLA Phase I, and EVLA Phase II. We understand that there is some skepticism that NRAO can successfully develop such a post-processing package, given recent past performance. We point out, however, the following salient features of the new post-processing package:
• the project has undergone a major reorganization, including the disbanding of the AIPS++ consortium and replacement of top level management;
• the code base is undergoing a massive rewrite (the CASA libraries are the result), with a revised interface based on Python instead of glish;
• development is now based upon project needs for ALMA and EVLA, including a coherent testing plan (for both ALMA and EVLA) which incorporates NRAO staff scientists and external users, and responds to deliverables based on requirements provided by the projects.
Because of these changes, we are confident that the post-processing package can deliver what is necessary for our processing needs. This is not to say that there are no remaining problems - just that they are not insurmountable.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Personnel (1)61.5 FTE years budgeted:• Observer Domain - Total = 7.5
• Proposal - 0.5• Observation Preparation - 2.5• Telescope and Correlator Scheduling - 4.5
• Telescope Domain - Total = 21.5• E configuration - 0.5• Other (MIBs, Executor, VLBA antenna rewrite, etc…) - 11.0• Correlator Backend - 1.0• Using WIDAR for VLBA - 9.0
• Science Domain - Total = 29.5• Post-processing - 24.0• Data Archiving - 2.0• Pipelines - 3.5
• Computing Infrastructure - Total = 3.0
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Personnel (2)
We can do this with only 61.5 FTE years (which would seem too small for a project of the size of EVLA Phase II) because of:
• heritage from EVLA Phase I - with ~100 FTE years of effort;
• overlap with ALMA - with ~250 FTE years of effort.
In this way, just as we are leveraging EVLA Phase I hardware (which leveraged VLA hardware) we are also leveraging the software.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Hardware BudgetIn addition to the 61.5 FTE years of effort, there is ~$2M budgeted for computing hardware. This includes:• networking upgrades;• pipeline hardware;• M&C (CBE and CALC) hardware;• Post-processing hardware:
• Development cluster;• Final cluster;• Observers workstations;• Data storage and distribution media.
National Radio Astronomy Observatory
13/14 June, 2005 EVLA Phase II Proposal Review
Summary
• EVLA software is designed to:• maximize the science derived from the instrument;• be easy to use, for all astronomers;• provide a look-and-feel and functionality which is consistent with
all NRAO telescopes.
• It builds extensively on the foundation of Phase I;• There are three main areas specific to Phase II:
• Scheduling - telescope and correlator;• Archive (mostly hardware);• Post-processing.
• 61.5 FTE years and ~$2M hardware budgeted, with ~100 FTE years and $2M hardware leveraged from Phase I; some part of ~250 FTE years leveraged from ALMA.