nasa’s goddard space flight center lro mission operations review (mor)slide - 1 crater soc...

LRO Mission Operations Review (MOR) Slide - 1NASA’s Goddard Space Flight Center

CRaTER SOC Operations

Mr. Michael Golightly

CRaTER Deputy Project Scientist

Boston University

LRO MOR Section 10


CRaTER SOC Overview

MOCR/T

MOCDMP

TCP

Rawarchive

SCP PDIST

PipelineProcessing

Archivefor PDS

MOCCMD

SCPUplinkarchive

SSH

Housekeeping monitorsRealtime

archive

PDSPPI

node

Pipelinearchive

SCP

Archivemirror

Secureweb

server

SCP

firewall


CRaTER Software Component Status

Component Function Status

C++ Object library Object-oriented interface to Level 0, 1, and 2 data products

100% writtenReleased

CRaTER Pipeline Time ordering and gap detection in raw (EDR) files; Generation of Level 0, 1, and 2 products

100% writtenReleased

Higher level products Model-dependent generation of LET spectra 0% written

Data distribution TCP/IP packet distribution server and clients 100% writtenUnder review

Engineering display TCL/Tk graphical interface to display and analyze secondary instrument science and housekeeping

100% writtenUnder review

Science display IDL graphical interface to display and analyze primary and secondary science data products

90% written

Private web interface HTTPS/SSL server to provide secure access to CRaTER data products and processing information

10% written

Public web interface HTTP server to provide access to CRaTER status 50% written


CRaTER SOC Functions

• Uplink command generation (SCP to MOC, as needed)• Continuous real-time data collection (TCP from MOC) and display• Daily receipt of recorded data (SCP from MOC)• Daily pipeline processing• Daily quick-look analysis and trending; examine data gaps for existence of

anomalous data loss• Daily report to SOC lead and MOC• Daily distribution to non-secure hosts (CRaTER internal server)• Daily review of system logs for evidence of attempted intrusions• Daily/weekly data back-ups to offline storage• Weekly SOC team meetings• Weekly processing & trending report and security inspection• Monthly archive collection and rearrangement• 3-monthly archive generation and transfer to PDS• Yearly security analysis and report• Anomaly response, e.g., replication on engineering unit or flight unit #2


CRaTER SOC Organization

Principal Investigator(Spence)

Project Manager1

(Foster)

Project Scientist(Kasper)

DeputyProject

Scientist(Golightly)

ProjectEngineer3

(Goeke)

• Pipeline software• Document preparation• LDWG/PDS Interfaces• Archive generation• IT security and

hardware maintenance

• Calibration algorithms• Conversion algorithms• Science display• On-orbit commanding

• GSE control S/W• Engineering display• Document reviews• Code reviews

1 MIT project management ends Dec 2007

2Golightly takes over as SOC Lead Jan 2008

3Support reduced to 25% beginning Jan 2008

SOCLead2

(Ford)


CRaTER SOC Staffing

FY08 FY09

O N D J F M A M J J A S O N D J F M A M J J A

PrincipalInvestigator Spence

Proj. Scientist,Science Lead Kasper

Deputy PS, SOC Lead Golightly

TemporarySoc Lead Ford

InstrumentScientist Case

OperationsLead Bradford

ScienceSoftware Wilson

Student

Student

SOC Operations

S/W Development/Maintenance

Integration/Test

Science Algorithms

Science AlgorithmImprovements/Evolution


CRaTER Key Documentation Status

Document Description/Purpose Need Date Current Status

32-01209 SOC Requirements Document — Plans for SOC component verification

12/01/2006 Rev B: 11/02/2006Delivered / signed off

32-02080 CRaTER SOC to PDS PPI Node ICD — Defines the process of data transfer from SOC to PDS

12/01/2006 Rev B 11/21/2006Delivered / signed off

32-01210 Data Management and Archive Plan — Describes the generation, format, and archival of CRaTER pipeline products

12/01/2006 Rev A: 10/25/2006Delivered / signed off

32-01213 SOC Risk Assessment — Identifies security risks and their proposed mitigations

04/30/2007 Rev B: 06/01/2007Delivered / signed off

32-01208 IT Security Plan and Contingency Plan — Describes the controls to apply to minimize IT risks

06/30/2007 Rev A: 07/01/2007Delivered / signed off

32-01212 SOC Test Plan — Describes IT testing methodology and test suites traceable from IT requirements

10/30/2007 Rev 01: 08/31/2007Draft released

32-01211 Standard Software Interface Specification — Describes the raw (EDR) and pipeline (CDR) products and PDS archive organization

05/31/2007 Rev A: 05/31/2007Rev B: under review

32-01207 Calibration Plan — Describes how detector counts are converted to energy and lineal energy transfer

10/01/2007 Rev 01: 08/12/2007Draft released

32-01214 Special Software Interface Specification — Describes the Level 3 and 4 spectral data products

02/29/2008 Not started


CRaTER SOC Hardware Facilities

SOC-A SOC-B

UPS

mux

console

Crater-A

Crater-B

2x500 GB

2x500 GB

Locked cabinet in key-coded room

CRaTER LAN (BU)

A CRaTER Team Member’s LAN

Printer

Printer

Recorder

Logger

BU NETWORK

switch

“Secure” LAN

“Semi-open” LANs

Disk accessExternal networks

Power supply Multiplexed consoles

To backup server

RAID15x50

0GB

Crater-Devel


CRaTER Engineering Unit

• The SOC will maintain a CRaTER engineering unit† and associated 1553 interface (S/C simulator) to assist in the diagnosis of post-delivery problems with the flight unit.

• SOC staff will interact with the simulator via a UDP interface to the following set of software programs

– Graphical command interface– Output capture to disk– Analog housekeeping display– User status display– Primary science stats display– Output conversion to ASCII

CRaTEREngineering

Unit1553B-UDP

Emulator

Internet(UDP)

2 x 1553,

1 Hz tick

Example: send commands to the engineering unit and monitor the resulting housekeeping channels

[1] Start the graphical command interface:

host# CCmd &

[2] Start the graphical housekeeping interface

host# CHouse &

† the CRaTER flight spare can also be used in place of the engineering unit, e.g., to assist in flight-unit recalibration

28VDC


CRaTER Software Maintenance

• Operating system maintenance– SOC-A, SOC-B, Crater-A, Crater-B, and Crater-Devel– Redhat Linux—updates provided under 3-year maintenance contract with Redhat– SOC Operations Lead responsible for applying updates/patches

• Pipeline code development/modifications– Through start of instrument I&T: MIT/Ford, BU/Bradford– After start of instrument I&T: BU/Bradford, BU/Wilson

• Configuration Control– Code version control maintained under Subversion version control system (

http://subversion.tigris.org/)– Copies of all implemented version will be maintained under version control in case it is

necessary to roll-back to an earlier version• System stability/integrity carefully controlled during all upgrades

– Code development/testing done outside SOC on Crater-Devel– User manuals, installation guides, code reference manuals, etc must be updated and released

with new versions/changes to existing software– Changes to operating system, pipeline, or science software implemented in sequence of

distinct steps• Install changes/updates on Crater-Devel--test for compatibility, stability, & functionality;• Install on the appropriate backup machine (e.g., machine-B if machine-A prime, etc)--test for

compatibility, stability, & functionality;• Switch backup machine to prime, and prime to backup (e.g., machine-B → prime, machine-A → backup)• Install on remaining machine--test for compatibility, stability, & functionality.

• NO FLIGHT INSTRUMENT SOFTWARE


CRaTER Software Testing

• The Test Plan is directly derived from the following documents:– 32–01209 CRaTER SOC Requirements Document– 32–01208 CRaTER IT Security and Contingency Plan

subject to the constraints identified in the following:– 32–01210 CRaTER Data Management and Archive Plan– 32–01211 CRaTER Standard Product Software Interface Specification– 32–02080 CRaTER Science Team and the PDS PPI Node ICD– 431–ICD–000049 External Systems ICD for the LRO Ground System

• The Traceability Matrix identifies 40 separate requirements:– 13 functional — the basic operations expected from the SOC– 10 design — requirements imposed by external constraints– 10 performance — ability of the s/w and human interfaces to do the job– 7 security — factors that protect the system from unintended behavior

which are verified by a set of 10 tests, using various methodologies:– 5 by explicit test — did the system pass the prescribed procedural test?– 3 by demonstration — did the system produce the expected results?– 1 by analysis — are the algorithms adequate to the task?– 1 by inspection — are the planned resources available?


CRaTER SOC Development Status

Current Status

• Hardware (5 processors, RAID storage, etc.) is being assembled

• Documentation deliveries are up-to-date

• Pipeline Software (Levels 0, 1, 2) is written and is under review

• Standard product SIS is under review by the PDS PPI node

• Engineering Unit interface is built and tested

• Draft SOC test plan is out; individual test scripts are being written

• Draft calibration plan has been released

• SOC is fully staffed (except for students)

Future Tasks

• Place SOC software under configuration control

• Erect network firewall around secure SOC hardware

• Complete SOC testing, including SOC-MOC and SOC-PDS tests

• Conduct SIS peer review by PDS PPI node


Backup Charts



• SOC computers– Host data pipeline process– Receive data pushed from MOC– Process data into specified L0, L1, L2 (, L3, L4) products– Push processed data to RAID storage, PDS, and Crater computers– CONNECTIVITY IS ONE WAY— will not be able to access SOC

machines except via computer’s console in controlled access facility– A & B machines mirror each other for redundancy

• Crater computers– Receive CRaTER data products– Serves data for public and internal web sites, nominal displays, team

scientists’ applications, etc (link between CRaTER data stores and anything outside of the SOC firewall)

– No connectivity from Crater computers back to the SOCs– A & B machines mirror each other for redundancy

SOC-A

Crater-B



Crater-A

Crater-B

2x500 GB

2x500 GB

CRaTER LAN (BU)

A CRaTER Team Member’s LAN

Printer

Printer

Recorder

BU NETWORK

Secure MOC→SOC Ingest Connection

CRaTER Data Via Restricted Access Network Connections

Disk AccessExternal networks

To backup server

RAID15x50

0GB

Crater-Develswitch

SOC-A SOC-B

SOC→Crater, PDS Push Connection



• SOC RAID Storage– RAID storage selected for short-term reliability

• derived both from the SOC requirements and risk assessment plans• prevent a disk I/O error from crashing SOC-A/B while receiving real-time

data from MOC

– Sized for 4 years of data storage• Uncompressed Level 0, 1, and 2 products → 789 GB/y

– Serves as backup to CRaTER PDS data submission

nasa’s goddard space flight center lro mission operations review (mor)slide - 1 crater soc...

Documents

crater soc operations

crater data products

written slide

crater status50

soc lead

monthly archive generation

moc daily pipeline processing

moc daily distribution