NRAO VLA Archive SurveyNRAO VLA Archive SurveyNational Radio Astronomy ObservatoryNational Radio Astronomy Observatory

2008-06-25 SPIE: Observatory Operations: Strategies, Processes, and Systems2008-06-25 SPIE: Observatory Operations: Strategies, Processes, and Systems

Jared CrossleyLoránt SjouwermanEdward FomalontNicole Radziwill

Jared CrossleyLoránt SjouwermanEdward FomalontNicole Radziwill

NRAO VLA Archive SurveyNRAO VLA Archive Survey(NVAS)

•Survey of the VLA Archive generated by an automated data pipeline.

•Under development since 2006.

•Data products include: images, calibrated data, diagnostics, logs, …

•Project ongoing, but data is already available on the Web.

http://archive.nrao.edu/

(NVAS)

•Survey of the VLA Archive generated by an automated data pipeline.

•Under development since 2006.

•Data products include: images, calibrated data, diagnostics, logs, …

•Project ongoing, but data is already available on the Web.

http://archive.nrao.edu/

Very Large ArrayVery Large Array

• Maximum baseline: 0.3 to 35 km• Frequencies: 0.07 to 50 GHz• Produces over 150 papers per

year• Outputs “visibility” measurements

• Maximum baseline: 0.3 to 35 km• Frequencies: 0.07 to 50 GHz• Produces over 150 papers per

year• Outputs “visibility” measurements

• Synthesis array of 27 antennas

• Located in New Mexico, USA• Operational for 28+ years

• Synthesis array of 27 antennas

• Located in New Mexico, USA• Operational for 28+ years

“Typical” VLA Data Reduction

• Project scientists are responsible for all parts of the data reduction process.

• Download VLA raw data from the VLA Archive.

• Making images requires:– Careful editing and calibration– Fast Fourier transform processing– Deconvolution methods

• To make data more easily accessible we have created the VLA Archive Survey

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Present State of Survey

Number of Images 72839

Unique Sky Positions

15590

Dates 1991 to 2003

Frequencies 1 to 50 GHz

Observing Modes Continuum

Array Configurations

D,C,B, some A

Sky Coverage

NVAS Pipeline System

ProcessProcess2.

AcquireAcquire1.

ArchiveArchive3.

ValidateValidate4.

• Makes use of preexisting components:– Raw data archive and interface– AIPS synthesis imaging software

package

• Perl and shell scripts connect components to form the pipeline.

• NVAS data is incorporated in the raw data archive. One interface One interface provides access to raw and provides access to raw and processed data.processed data.

Acquire Data

• Perl script automates communication with Archive server by HTTP requests.

• Input start and stop dates for processing1. Script queries the server2. Algorithm selects one or more data files3. Download to local disk4. Data is loaded into AIPS, split by frequency,

and prepared for processing

NVAS Pipeline Stages

AcquireAcquire1.

Data Processing

• AIPS “runfile” (script) performs standard synthesis imaging steps(VLARUN)

• Output data products:– Images– Calibrated data– Diagnostic plots– Log files– Metadata for Web interface

NVAS Pipeline Stages

AcquireAcquire

ProcessProcess

1.

2.

Example image and diagnostic plots:

Crab Nebula at 5 GHz

Archiving

• Shell and Perl scripts • Move data products to

archive server• Generate HTML for survey

Web interface

NVAS Pipeline Stages

AcquireAcquire

ProcessProcess

ArchiveArchive

1.

2.

3.

Validation

• Human-interactive process• A private Web interface is

used to remove low quality data.

• All NVAS data is reviewed by a human.

• Less than 5% of images have been removed during data validation.

NVAS Pipeline Stages

AcquireAcquire

ProcessProcess

ArchiveArchive

ValidateValidate

1.

2.

3.

4.

Pipeline System PerformancePipeline System Performance

• Only data validation requires human interaction.

• All other pipeline stages are limited by computer and network data rates.– Data processing requires the most time;

high-resolution imaging may take many hours

• Survey data is 1.4 TB and growing

Access the survey from http://archive.nrao.edu/

Data Vault:

7019-47

Future Development

1. Finish imaging of VLA continuum dataAt the same time, continue algorithm

upgrades:• Self-calibration• Better data editing• Mosaic imaging• Composite imaging of long-timescale

projects

2. Apply new algorithms to all survey data3. Process spectral line data

Expect to finish these stages end of 2010.

SummarySummary

• Automated survey of the full VLA archive

• Makes VLA data more accessible to all astronomers

• Provides reference images, calibrated data, diagnostics, logs, …

• Ongoing survey, expected completion in 2010

• Currently available at:

http://archive.nrao.edu/

• Automated survey of the full VLA archive

• Makes VLA data more accessible to all astronomers

• Provides reference images, calibrated data, diagnostics, logs, …

• Ongoing survey, expected completion in 2010

• Currently available at:

http://archive.nrao.edu/

NRAO VLA Archive Survey is…NRAO VLA Archive Survey is…