ESPC CIP Update

November 7, 2008

COPC Action Item 2008-1.1

NESDIS is currently in the process of reviewing and planning upgrades for the critical infrastructure protection in the Environmental Satellite Processing Center (ESPC) (including the portions related to the Shared Processing Program). This includes various backup capabilities for priority Level 1 and 2 products within the ESPC. To date, only partial backup tests have been conducted. There is concern within the OPCs that NESDIS should evaluate full backups (i.e., “hot” backups). NESDIS is asked to provide a full plan to the other OPC directors on the backup procedures for the ESPC CIP. This plan would include, e.g., a “HOT” backup, and working with the other OPCs, determine the degradation expected, and length of test. Briefing of this plan is due at the next COPC. CSAB 22 October, 2008. CSAB recommends closing this action

with the briefing to review the CIP implementation at Fall COPC.

CIP concept of operations is to operate as a “warm backup” as opposed to “hot backup” All equipment is on but in an idle state, not producing products for

distribution Priority 1 and 2 products to be generated within 24 and 48 hours of

management decision to activate CIP Once fully spun up CIP provides all critical products to customers

with latencies comparable to the primary production system at NSOF

Re-engineering CIP as a hot backup Additional costs arise from

• continuous synchronization with primary system at NSOF• increased O&M activities at CIP to monitor and maintain continuous

data flows• Increased licensing costs

COPC Action Item 2008-1.1 (cont)

Project Overview

ESPC/CIP is a phased implementation Phase 1 – CEMSCS (POES) Phase 1.5 – Metop Phase 2 – SATEPS (GOES) Phase 3 – SATEPS and Metop (IASI, ASCAT)

Provide backup capability at Wallops for all Priority 1 & 2 “critical products” in the event ESPC at NSOF becomes non-operational Priority 1 Products within 24 hours after decision to activate Priority 2 Products within 48 Hours after decision to

activate

CIP Phases

Phase 1 Capabilities The CIP Phase 1 capabilities focus on providing

POES products that have no dependence on data from Metop sources

• Does not include NOAA-18 Blind GAC data

Connectivity• NWS (Telecom Gateway)• Suitland/NSOF• FCDAS (Fairbanks AK)• WCDAS (Wallops Main Facility)• NOC (Silver Spring, MD) – Internet Access

CIP Phases (Cont.)

Phase 1.5 Capabilities Backup of Metop Data Processing System

• Create and distribute products produced by NOAA instruments on board the Metop satellite (IASI and ASCAT Phase 3)

Receive Generic File Transfer Data (GOME and GRAS products)

Ability to Process Blind GAC from NOAA-18 Connectivity

• NWS (Telecom Gateway)• Suitland/NSOF• FCDAS (Fairbanks AK)• Darmstadt (Germany)• WCDAS (Wallops Main Facility)

CIP Phases (Cont.)

Phase 2 Backup of ESPC SATEPS (GOES) Phase 1 QuickSCAT LRIT MTSAT Family of Services

Phase 3 Backup of ESPC SATEPS (GOES) Phase 2

• Primarily a SW effort IASI ASCAT Okeanos (Ocean Color)

Current Status

Project Reviews Phase 1 Operational Readiness Review (May 28) Phase 1 Commissioning Review (July 31) Phase 1.5 Operational Readiness Review (September 8)

• Acceptance Testing (July 28 to Sept 3)• Operational Exercise (Aug 25 – Sept 3)• User Test (October 27 – November 3)

Phase 1.5 Commissioning Review (November 18) January 2009 Phase 2 Operational Readiness Review (March 2009) May

2009 Phase 2 Commissioning Review (April 2009) June 2009

Current Status (cont)

Systems Servers operational Network established locally and to NSOF AFEP configured to ingest NOAA-KLM, N-18 BGAC

and MetOp data Communications

External (Silver Spring, NWS, NSOF, FCDAS, Darmstadt, Internet)

Internal (WCDAS)• Interconnectivity between main facility and OBF

National Weather Service Telecommunications Gateway(NWSTG) Connections

Current Status (cont)

Facility Power and HVAC operating as planned Security System operating as planned Fire Suppression System operating as planned Quiet Room Occupancy Underway

• SARSAT Backup occupancy date is Nov ‘08• ESPC space Occupancy delayed

• additional phone lines and electrical circuit

Current Status (cont)

Phase 1.5 User Test -- Complete

Results: Successfully tested the transfer and receipt of data at CIP Did not meet all test objectives because of data receipt problems

Test System State Schedule Report

CIP Activation Test October 27 13:00 EDT

DOD User Testing Begins Test October 28 13:00 EDT

NCEP User Testing Begins Test October 28 10:00 EDT

DOD User Testing Ends Test October 30 10:00 EDT

NCEP User Testing Ends Test November 3 10:00 EDT

CIP Deactivation Stand-by November 3 10:00 EDT

User Test Reports Stand-by November 7 17:00 EDT

Current Status (cont)

Based on CIP Activation and User Test results 1.5 Commissioning Review delayed until completion of the following corrective actions Complete all CIP 1.0 Commissioning Review Requests for Actions

(RFA’s) Complete all CIP 1.5 Commissioning Review Requests for Actions

(RFA’s) Ensure all computer operator monitoring procedures and start-up

procedures are complete and up to date Write procedures for METOP data connection until Global patch is ready Test NOAA-18 Blind GAC Schedule pre-activation test Schedule another phase 1.5 CIP Activation and User Test

Next Steps

Phase 2 Operational Readiness Review Requirements have been identified and draft document generated for review

• Hardware identified and procurements initiated• Build-out of hardware in progress

User Testing for Phase 2 tentatively scheduled for March 16-24 Gather requirements for Phase 3

Back Up

User 1.5 Testing -- Results

Test Results: NCEP

• User testing occurred without MetOp data for 24 hours• from 10:51 am 10/ 28 to10:33 10/29

• User testing occurred without NOAA-18 Blind GAC until 10/31/08 DOD

• Testing commences on schedule• User testing occurred without MetOp data for 24 hours

• from 10:51 am 10/ 28 to10:33 10/29• User testing occurred without NOAA-18 Blind GAC until 10/31/08• All required products were available

• FNMOC received the following non-optimal products:• ATOVS Soundings Data• MSPPS

• AFWA received the following non-optimal products:• ATOVS Soundings Data• Level 1b MetOP-A FRAC data

• Scheduled Deactivation (10:00 am on 10/30) schedule was not met:• 11:41 – AFWA distribution to CIP Turned off • 13:02 – sppgw1_cip to diamond_cip turned on• 14::35 – AFWA and NAVO data forwarded from sppgw1 to sppgw1_cip• 15:50 – FNMOC distribution to CIP Turned off • 19:28 – FNMOC data forwarded from sppgw1 to sppgw1_cip