Archive, Data Products, & Operationsfor the Murchison Widefield Array

Ron Remillard (MIT); MWA Project Meeting, June 6, 2011

Outline

Resources for the MWA Archive at MIT NSF MRI Program

Standard Data Products Incomplete definitions in early 2011 Strategy for progress Information Flow Diagram

Commissioning Phase Needs for Commissioning Plan Needs for more sophisticated Monitoring

Operations Phase Operations Plan and Needs for Update

Plans for remainder of 2011

MWA Archive at MIT

NSF MRI Program ($868 K: NSF + MIT contribution) Large equipment budget Advanced spending on archive requirements, data products Need to set aside majority of funds until late 2012

Archive Tasks (based on EoR, systematic error analyses, student projects) Capture raw data (in short bursts) Intermediate data capture (non-calibrated images; less resolution (, t)) High timer resolution calibrated data

Determine calibrations in parallel with RTS Calibrate data offline Study ionosphere parameters at 2 s

Service MWA EoR Consortium, other EoR users, other outside projects

MWA Archive at MIT

Gaps in Plan Redefine SOW for 128T (retain EoR focus; gain approval for NSF extensions) Complete Archive Requirements Document (project-wide scope) Method to obtain data from Curtin Method to provide data access for Users

Standard Data Products

Science Data Eor Data Products (calibrated raw images) defined in MRI proposal

SHI Needs raw data (pro-Correlator) and UV data from outlier tiles

Transients & GEG groups need Images and other modes

(light curves, transient searches (image diffs), wider images (low t res.), others?)

“Meta Data” RTS {calibrator catalogs, calibrator solutions, ionospheric params., “peeled” sources}

M&C {configuration parameters, hardware status, housekeeping data}

Common RTS/M&C Database

EOR Data Writing Requirements for 512T

Images: HEALPix files, instrumental images (from back end of RTS): 25o radius ; 4 polarizations; 768 channels

UV data samples: (from front end of RTS): visibilities for 100[TBR] Tile pairs

Foundation for MWA Calibrations and Data Processing

High-Quality Calibrations and Supression of Systematic ErrorsCalibrator gain measurements; Ionosphere refraction data; Faraday rotation; beam fit parameters

EOR Data Products are Basis for 3 Other Science Packages• Imaging mode with variable {time, frequency-binning; spatial selection} for Transients Imaging (TLA mode) and GEG Imaging• SHI: UV Data Samples + IPC Mode to be written Directly from Correlator• Stand-alone processing for Transients discovery (Transient ASM mode) and other data products (e.g., flux-calibrated images; light curves)

Standard Data Products

Correlator

Digitized Data

RTS/RTC

Calibration Subtract Calibrated Staging & Solutions Bri. Sources Images Processing

MWA Data Archive (Perth)

MWA Information Processing of EOR Data

UV Data (40 Gbs)

CommandWriting mode

Science Data:EOR Images

Catalogs:Calibration SourcesSources Subtracted

Calibration Data:Beam Parameters

Ionosphere Params.

Raw Data:UV Data [~100 tiles]

Additional MWA Archives (MIT, Melbourne)

Secondaryprocessing

Flux ImagesLight Curves

(study systematics)

M&C

Housekeeping Configuration

M&C MetaData:ConfigurationsHousekeeping

Receivers

Correlator

Digitized Data

RTS/RTC

Calibration Subtract Calibrated Staging & Solutions Bri. Sources Images Processing

MWA Data Archive (Perth)

MWA Information Processing and Data Products

UV Data (40 Gbs)

CommandWriting mode

Science Data:EOR Images

TLA Images GEG Images

Catalogs:Calibration SourcesSources Subtracted

Calibration Data:Beam Parameters

Ionosphere Params.

Raw Data:SHI: IPS Data [512]

UV Data [~100](for SHI & EOR)

Additional MWA Archives (MIT, Melbourne)

Secondaryprocessing

Flux ImagesLight Curves

Transient ASM Data

M&C

Housekeeping Configuration

M&C MetaData:ConfigurationsHousekeeping

Receivers

Archive & Standard Data Products for 128T

Save Correlator output (larger Archive)?

Built-in Capacity for UV Data analyses, Iterative Calibrations,

Comparative Tests, Specialized Tests, Secondary Data Products, etc.

Standard Data Products: No Significant Changes

Next efforts: Finalize design and track development of

standard data products

Commissioning Phase (Checkout to Verification)

Requirements & Verification Document Colin & RR first draft March 2011 Needs Adaptations for 128T Needs Attention and Inputs from Judd, Frank, etc.

Gaps and Risk Verification Details are complicated; plans incomplete Insufficient strategy for automation and scripts

Upgrade Plan for Monitoring Functions Status screens for hardware & housekeeping data for Installation/Checkout Well-defined metrics and useful error logs Take next step to rudimentary Interferometry monitoring Add smart scripts to monitor for Intermittent problems Examine all data

Who? (Curtin Archive & Superscience help? ; MIT student project?)

Operations Phase

Operations Management Plan Last update: Nov 2010 (Wayne Arcus) Operations succeeds end of Commissioning phase

Engineering component – Dave Emrich (Commissioning Engineer) Science Performance Verification – Judd Bowman (Project Scientist) Initial Operations Manager – Tom Booler (Deputy Project Manager for Australia)

Assumptions: move toward 24/7 operations ; proposals ; review ; schedule ;

observe ; trend analyses ; archive verification ; TBD user support

Gaps and Risk Currently No Operations funds Details for Proposals, Target Selection, Scheduling

Plans during remainder of 2011 Revise Plan after work on:

schedule, cost, monitoring, automation, data products, commissioning plan