integrated data cycle systems harvey e. rhody chester f. carlson center for imaging science

Integrated Data Cycle Systems

Harvey E. Rhody

Chester F. Carlson Center for Imaging Science

The Problem

An expensive facility with many customers

The customers have different needs

Expert knowledge is needed to operate the facility

Operate the facility at low cost with good service

The Need

Increased Productivity

Fast Response

Continuous Improvement

Required Functions

Job Specification

Job Planning

Job Execution

Delivery of Results

Archive

Kinds of Knowledge

• Technical knowledge

• Systems knowledge

• Project management knowledge

• Business knowledge

All are necessary for a successful high-value project

Astronomical Observatories

Complex and Expensive System

Remote Location

Shared Resource

Leading-edge Technology

Long System Life

Spirex Telescope with Abu Camera in Antarctica

Winterover on arrival at the pole….

Six Months Later….

Abu Pipeline

Astronomer

ScienceCenter

AnalysisCenter

DataArchive

ObservationCenter

NOAO/RIT

SouthPole

White Sands/RIT

RIT

IDL, C,...XMLFITS, TIFF, ...

Pipeline

Database

ImageData

CAMLDocuments

Algorithms

CAMLController

Java, ORB

OperatorInterface

NetworkInterface

Large Magellanic Cloud

30 Doradus nebula

Star formation

NGC 6334

Abu/Spirex 1998

Productivity Challenge

960 observing hours per year

160 observing flights per year

Variety of instruments

Large community of observers

Conclusion: The traditional “PI” approach is unworkable. A new system for efficiency and relentless observation is needed.

The DCS Mission

The DCS shall provide the infrastructure to support the operation of facility instruments on SOFIA using best current knowledge and tools, and supporting continuous improvement in an efficient, extensible, and modular architecture.

SOFIA Facility Instruments

Instrument Institution Characteristics

AIRES NASA-Ames Echelle Spectrometer17 to 210 micronsR=10,000

HAWC U. ChicagoRITNASA-Goddard

Far Infrared BolometerCamera40-300 microns

FORCAST Cornell Mid IR Camera5-40 microns

FLITECAM UCLA Near IR Test Camera1-5 microns

SOFIA Team Players and Strengths

IPAC

IRAS, ISO, 2Mass, WIRE, SIRTF

Archiving, Observation planning

ARC

Smart scheduling activities for instruments and telescopes for HST, Mars Rover, Deep Space 1

GSFC

HST and NGST development

HAWC and SAFIRE instrument architecture

Object Oriented development

RIT

South Pole Data Pipelining

HAWC development

UCLA

Keck

FLITECAM

SOFIA Data CycleD ata

M is s io nP lan

O b s erver'sP ro p o s al &Exp erim ent

S im ulatio n

Q uic kLo o k

A nalyzedD ata

A rc hive

O b s erver'sP lan

PR

DataCo l l e c t i o n

P l an

P ub lic atio n

A nalys isP lan

Science Center Activities

Community Development and Support

Maintain Science Data Archives

General Investigator Support

Maintain Center Information Infrastructure System Documentation , Observatory Performance Information, Observation Constraints

Calibration Information, Atmospheric Models

Maintain FI Tools Facility Software Management, Proposal Development Tools, Proposal Evaluation Tools

Observation Planning Tools , Observing Tools, Flight Planning Tools

Data Reduction Tools -- Pipeline analysis modules

Coordinate with Observatory Operations

User Needs

• Archival Research • Get Information on Observatory Capabilities • Get Information on Instrument Capabilities • Build and submit a Proposal • Build an Observation Plan • Build a Data Reduction Pipeline • Access Observation Results • Interpret Experimental Results • Publish

User Access Requirements

User access to all user functions shall be supported via the Web

1.Access to archived science products 2.Access to supported external archives 3.Access to observatory capabilities information 4.Access to instrument capabilities information 5.Proposal development, submission and tracking 6.Observation timing estimation 7.Atmospheric modeling resources 8.Access to observation data 9.Access to flight and housekeeping data 10.Access to flight logs 11.Data processing 12.Published document archiving

System Requirements

User Access

Administrator Access

Data Storage and Retrieval

Data Processing

Planning and Scheduling

Documentation Support

Data Reduction Resources

FSI & MCS

Interfaces

Task Library

DCS Data Storage

User Interaction

Layer

prefs, context

requests, results

run an experiment

raw and processed

data

AOTs, exp info, raw data, obs logs..

various searches

run a pipelineSSMOC

Internet

DCS Data Storage

DCS DataStorageInterface

DB LocatorAnd Selector

User DB

Accounts Groups

Privileges Contexts

Preferences

Experiment DB

Templates Planned Accepted

Completed Reviews

Comments

Data DB

Raw Instr DataOBS Logs

Processed Data

Instr DB

GSFC’s IMLFSI Chain ListsHistory of Instr

Use

Provides common DCS interface to all storage,

translates DCS data access to standard DB

interface methods Convenient switch for criteria such as: load

balancing, fault tolerance

The RIT Astronomy Pipeline Team

Harvey Rhody, Scott Lawrence, Erika Tolar, Stephen Schaeffer, James DeFelice, Adith Chandrasekhar, J. Fernando Naveda, Bob Krzaczek (standing) and Ian Gatley. Not shown: Jim Chauvin, Brian Perry, Joel Kastner.