jpss direct broadcast support to cspp...1 jpss direct broadcast support to cspp christie best noaa...

1

JPSS Direct Broadcast Support to CSPP

Christie BestNOAA JPSS Program Office

[email protected]

CSPP/IMAPP User GroupMadison, Wisconsin27-29 June 2017

2

▪ JPSS J1 is launching 9/21/17 and is the first Next Gen JPSS polar orbiting satellites extending continuity of polar satellite observations until 2038.

▪ CSPP, funded by NOAA JPSS, is a collection of software packages for processing direct broadcast (DB) data from polar-orbiting meteorological satellites. NOAA/NESDIS contributes standalone JPSS science algorithms for CSPP distribution to create L1B and higher level products and images including Level 2 Environmental Data Records (EDRs).

▪ NOAA/NESDIS is transitioning from individual, stove-piped, mission-centric science product ground processing to an Enterprise approach which is cost effective, agile and sustainable. Instead of developing standalone science algorithms per instrument, NOAA/NESDIS/Satellite Applications and Research (STAR) is developing Enterprise algorithms (EA)s supporting multiple instrument types.

▪ JPSS has transitioned the EAs for Level 2 EDRs and will no longer be dependent on the Raytheon Algorithm Development Library (ADL) for this EDR software, firmware and documentation.

▪ JPSS is offering CSPP NOAA/NESIDS endorsed JPSS EAs to be distributed within the Data Algorithm Package (DAP) from STAR to enable CSPP users to create Level 2 EDRs used with S-NPP and JPSS data processing.

Introduction

3

Enterprise Algorithms (EA)

▪ An enterprise algorithm is a NOAA/NESDIS endorsed algorithm that uses the same scientific methodology (i.e., physical basis, including assumptions) and software base to create the same classification of product from differing input data (satellite, in-situ, or ancillary).

▪ An endorsed algorithm has the following characteristics– It meets the JPSS Level 1 requirements.– It is well vetted and peer reviewed by independent reviewers. – It is well documented in an Algorithm Theoretical Basis Document (ATBD) and in the scientific literature.– It includes test and validation procedures that are well vetted. – Its performance is well demonstrated and characterized to the extent possible using available proxy and/or real

data.

▪ An Enterprise algorithm applies singular scientific algorithm methodology and common software to generate consistent products. Each time an unfamiliar observation / source is acquired, interfaces (metadata, data structures and formats) have to be written. But over time, as new observations are introduced into NOAA, the process of adding external sources of data will become easier because of the tools and interfaces already established.

4

Contain:– Science Enterprise Algorithm source

code, including make files and scripts. – Test plans, test description, test

procedures, and detailed performance testing results.

– Test input data, temporary files, and expected output data.

– Coefficient files and/or look-up tables. – Quality monitoring information (quality

flags, quality flag values). – Production rule-set definitions. – Product file specifications – layout,

content, and size.

Delivery Algorithm Packages (DAP)– Data flow diagrams– List of exit codes and their associated

messages. – List of expected compiler warnings – Estimates of resources required for

execution. – SPSRB documents – Including: Algorithm

Theoretical Basis Documents (ATBDs), System Maintenance Manual (SMM), and External Users Manual (EUM)

– Delivery Memo. – README text file

Standards for Algorithm Delivery and Integration Using DAPs available upon request.

5

▪ Typical Individual Enterprise Algorithm – Reads in multiple types of satellite data (one at a time) into one executable to create one set of products

▪ Examples of Individual Enterprise Algorithms are:–AVHRR Clear-Sky Processor for Oceans (ACSPO) – SST for polar satellites, currently being updated to process

AHI data–Microwave Integrated Retrieval System (MiRS) – Microwave product suite for most microwave instruments–NOAA-Unique Combined Atmospheric Processing System. (NUCAPS) – The NUCAPS algorithm runs on

CrIS/ATMS and IASI/AMSU data and is a sounding product suite created using Suomi NPP and Metop-A/Metop-B sounders.

Individual Enterprise Algorithms

GOES, AVHRR, VIIRS, MODIS, SEVIRI Data

Sea Surface Temperature

Algorithm

GOES SST, AVHRR SST, VIIRS SST, MODIS SST,

SEVIRI SST Products

ExecutableInput Output

Credit: Walter Wolf/Nadia Smith

6

▪ Typical Multiple Enterprise Algorithms reads in multiple types of satellite data (one at a time) into one program to create multiple products. The algorithms can also be run individually where any product precedence data is input from a file.

▪ Examples of multiple Enterprise Algorithms are:– JPSS Cloud, Aerosol, Cryosphere and Volcanic Ash products – STAR Enterprise Processing Framework

▪ An executable that keeps common data in memory– Individual algorithm teams create their own subroutines to plug into the program– Algorithm teams are dependent upon products from other algorithm teams– Redundancy removed that is common amongst algorithms

Multiple Enterprise Algorithms

Credit: Walter Wolf

6GOES, AVHRR,

VIIRS, MODIS, AHI, ABI Data

STAR Enterprise Processing Framework

GOES Cloud, Aerosol, Snow, Fog, etc., VIIRS Cloud, Aerosol, Snow,

Ice, Vol Ash, etc., ABI Cloud, Aerosol, Rain Rate, Snow, Ice, Vol

Ash, Fog, etc. Products, ……..

ExecutableInput Output

7

S-NPP Enterprise ProductsEnterprise

Aerosol Detection (VIIRS) Global Surface Type (VIIRS) Rainfall Rate (ATMS)

Active Fires (VIIRS) Green Vegetation Fraction (VIIRS) Sea Ice Characterization (AMSR-2)

Aerosol Optical Depth(VIIRS) Ice Age/Thickness (VIIRS) Sea Surface Temperature (AMSR-2)

Aerosol Particle Size (VIIRS) Ice Concentration (VIIRS) Sea Surface Temperature (VIIRS)

Albedo (Surface) (VIIRS) Ice Concentration (ATMS) Sea Surface Wind Speed (AMSR-2)

AMSR Calibrated Sensor Data (AMSR-2) Ice Surface Temperature (VIIRS) Snow Cover/Depth (AMSR-2)

Atmospheric Vertical Moisture Profile (CrIS/ATMS) Imagery (AMSR-2) Snow Cover (ATMS)

Atmospheric Vertical Temperature Profile (CrIS/ATMS) Imagery (ATMS) Snow Cover (VIIRS)

Carbon Dioxide (CO) (CrIS) Infrared Ozone Profile (CrIS) Snow Water Equivalent (ATMS)

Carbon Monoxide (CO2) (CrIS) Land Surface Emissivity (ATMS) Snow Water Equivalent (AMSR-2)

Cloud Cover/Layers (VIIRS) Land Surface Temperature (VIIRS) Soil Moisture (AMSR-2)

Cloud Height (Top and Base) (VIIRS) Land Surface Temperature (ATMS) Surface Reflectance (VIIRS)

Cloud Liquid Water (AMSR-2) Methane (CH4) (CrIS) Surface Type (AMSR-2)

Cloud Liquid Water (ATMS) Moisture Profile (ATMS) Temperature Profile (ATMS)

Cloud Mask (VIIRS) Ocean Color/Chlorophyll (VIIRS) Total Precipitable Water (AMSR-2)

Cloud Optical Depth (VIIRS) Outgoing Longwave Radiation (CrIS) Total Precipitable Water (ATMS)

Cloud Particle Size Distribution (VIIRS) Ozone Nadir Profile (OMPS-N) Vegetation Indices (VIIRS)

Cloud Phase (VIIRS) Ozone Total Column (OMPS-N) Vegetation Health Index Suite (VIIRS)

Cloud Top Pressure (VIIRS) Polar Winds (VIIRS) Volcanic Ash Detection And Height (VIIRS)

Cloud Top Temperature (VIIRS) Precipitation (Type/Rate)(AMSR-2) J1 has similar products

8

JPSS has L1 requirements to provide software, documentation and updates to the direct readout community.

– 6.2.13 JPSS shall provide the Direct Readout community with software, documentation, and periodic updates to enable civilian and military agencies to produce data products from JPSS, using their own hardware to receive the JPSS High Rate Data (HRD) broadcast. [Priority 3]

CSPP Benefits of Receiving EA/DAPs– NOAA/NESDIS endorsed EA/DAP software and documentation increasing product quality

through configuration management, timeliness and functionality based on STAR enterprise product life cycle (EPL)

– Support from STAR Algorithm Scientific Software Integration and Transition Team (ASSISTT) improving coordination between science teams and users.

– Ensuring continuity of NOAA products between current and future satellites– EA reduce maintenance costs for algorithms due to fewer algorithms to maintain– Comprehensive documentation from the DAPs

CSPP Benefits of EA/DAPs

9

Science Product TimelinessSTAR Enterprise Product Lifecycle (EPL)

JPSS DB SUPPORT EFFORT Add delivery of Preliminary DAP

to CSPP at the Unit Test Readiness Review Decision Gate

Add delivery of Final DAP to CSPP at the Algorithm Readiness Review Decision Gate

• Consists of process steps that take a product from initial concept through development of a pre-operational system ready for transition to operations.

• Delivered Algorithm Packages (DAPs) include the EA, processing code, test data and results, and comprehensive documentation.

Credit Diagram: Bonnie Reed

10

Science Product Quality Assurance (QA) & Configuration Management (CM)

STARBuilds

EA/DAPs for NDE & Users

CSPP Delivers SW Algorithm

PackagesTo Users

JPSS DB SUPPORT EFFORT Increase the quality of the JPSS algorithm

documentation by providing DAPs Ensure consistent deliveries of updates and

patches of EA/DAPs

Configuration Management

Configuration Identification

Change Management

Version Control

Communication

JPSS DB SUPPORT EFFORT Identify JPSS EDR algorithms to be updated

in CSPP Monitor changes to STAR delivered EA/DAP

science products for CSPP applicability Keep CSPP apprised of new EA/DAP

versions and/or upcoming updates

Quality Assurance

11

Current CSPP Software Packages

CSPP Software Supporting Suomi NPP Last Updated JPSS ADL VersionCSPP

Algorithm Version NOAA IDPS ComparableNext Planned

Update

VIIRS, ATMS and CrIS calibration and geolocation software (Raw Data Records (RDRs) to Sensor Data Records (SDRs)) 07/04/14 4.2.11 2.2 Mx8.11 TBD

VIIRS EDRs including a subset of Land, Ocean and Atmosphere Products 12/13/16 4.2.4 2.0.1 Mx8.4 TBDVIIRS, MODIS and AVHRR Imager reprojection software for the creation of GeoTIFFs and/or AWIPS NetCDF files 04/27/17 N/A 2.1 N/A TBD

NOAA/NESDIS/STAR NOAA Unique CrIS/ATMS Processing System (NUCAPS) EDR Hyperspectral Sounding Retrieval Software 12/14/15 N/A 1.1 N/A TBDCrIS, AIRS and IASI University of Wisconsin dual regression single Field-of-View (FOV) Temperature, Moisture, Surface and Cloud Retrieval EDR 04/29/14 TBD TBD TBD TBD

S-NPP VIIRS, ATMS,CrIS and EOS Aqua and Terra HYDRA2 multispectral data analysis toolkit 02/22/16 N/A 1.3 N/A TBD

NOAA/NESDIS/STAR Microwave Integrated Retrieval System (MiRS) supporting S-NPP ATMS, NOAA-18, 19 and Metop-A, B AMSU-A and MHS instruments 12/2/16 N/A 2.0 N/A TBDVIIRS Imagery EDRs1 12/13/16 Unk 2.0.1 Mx8.4 TBD

VIIRS, MODIS and AVHRR (POES and Metop) Cloud and Land Surface Retrievals from CLAVR-x 04/07/16 N/A 2.0 N/A TBDInternational ATOVS Processing Package (IAPP) Retrieval Software, supporting POES and Metop HIRS, AMSU-A and MHS Instruments. 03/03/17 N/A 1.1 N/A TBDNOAA/NESDIS/STAR ACSPO Advanced Clear-Sky Processor for Oceans software supporting VIIRS, AVHRR and MODIS imagers. 02/08/17 N/A 1.1.2 N/A TBD

Sounder Quicklook Software for display of NUCAPS, HSRTV, MIRS and IAPP Atmospheric Retrievals 04/08/14 N/A 1.3 N/A TBD

Yellow Highlight: CSPP SW packages that could benefit from EA/DAP EDR updates1 VIIRS Imagery EDRs are Key Performance Parameters and will continue to be produced by JPSS IDPS using ADLs

12

JPSS/CSPP VIIRS EDR EA/DAPComparison (1)

JPSS Multiple Enterprise Algorithm (DAP): VIIRS S-NPP/JPSS-1JPSS Cloud, Aerosol, Cryosphere and Volcanic Ash2 CSPP Products in the CSPP VIIRS EDR Package

L1RD Product NameData Record

Type JPSS PriorityImplementation

(System) CSPP SW Package Product Name JPSS EA Availability StatusActive Fires1 EDR 3 ESPC Active Fires JPSS Individual EAAerosol Detection EDR 3 ESPC Suspended Matter JPSS Multiple EAAerosol Optical Depth EDR 4 ESPC Aerosol Optical Thickness JPSS Multiple EA

Aerosol Particle Size EDR 4 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EAAlbedo (Surface)2 EDR 4 ESPC Currently Not Available in VIRS EDR CSPP Package JPSS Individual EACloud Cover/Layers EDR 3 ESPC Cloud Cover/Layers JPSS Multiple EA

Cloud Height (Base and Top) EDR 3 ESPC Cloud Height (Base and Top) JPSS Multiple EA

Cloud Particle Size Distribution EDR 3 ESPC Cloud Particle Size Distribution JPSS Multiple EACloud Mask EDR 3 ESPC Cloud Mask JPSS Multiple EACloud Optical Depth EDR 3 ESPC Cloud Optical Depth JPSS Multiple EA

Cloud Particle Size Distribution EDR 3 ESPC Cloud Particle Size Distribution JPSS Multiple EACloud Phase EDR 3 ESPC Cloud Phase JPSS Multiple EA

JPSS DB SUPPORT EFFORT Identify products not in the JPSS Multiple EA but currently contained in the CSPP VIIRS EDR Package For those that are not, CSPP to determine if they want to add JPSS Single EAs or offer individually.

JPSS is proposing updating the VIIRS EDR CSPP package with a Multiple EA DAP named JPSS Cloud, Aerosol, Cryosphere and Volcanic Ash currently in operations.

Gaps

1 Products highlighted in red text may be available as Single EA/DAPs but not currently contained in the initial JPSS Cloud, Aerosol, Cryosphere and Volcanic Ash EA/DAP.

2 Land products to be included in Operational EA/DAP delivery once released to operation.3 Not all JPSS products can be integrated into a multiple EA.

13

JPSS/CSPP VIIRS EDR EA/DAPComparison (2)

JPSS Multiple Enterprise Algorithm (DAP): VIIRS S-NPP/JPSS-1JPSS Cloud, Aerosol, Cryosphere and Volcanic Ash CSPP Products in the CSPP VIIRS EDR Package

L1RD Product NamesData Record


(System) CSPP VIIRS EDR SW Package Product JPSS EA Availability Status

Cloud Top Pressure EDR 4 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Multiple EA

Cloud Top Temperature EDR 4 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Multiple EACloud Type - non L1RD EDR N/A ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EA

Ice Age/Thickness EDR 3 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EA

Ice Concentration EDR 3 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EAIce Surface Temperature EDR 4 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EA

Land Surface Temperature2 EDR 4 ESPC Land Surface Temperature JPSS Multiple EA

Vegetation Index EDR 4 ESPC Normalized Difference Vegetation Index (NDVI) JPSS Multiple EA

Vegetation Index EDR 4 ESPC Enhanced Vegetation Index (EVI) JPSS Multiple EASea Surface Temperature1 N/A 3 ESPC Sea Surface Temperature JPSS Multiple EASnow Cover EDR 3 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EASurface Reflectance EDR 3 ESPC Surface Reflectance JPSS Individual EASurface Type EDR 3 ESPC Surface Type JPSS Individual EASuspended Matter EDR 3 ESPC Suspended Matter JPSS Individual EAVolcanic Ash Detection and Height EDR 3 ESPC Currently Not Available in VIIRS EDR CSPP Package JPSS Individual EA

JPSS DB SUPPORT EFFORT Identify products not in the JPSS Multiple EA but currently contained in the CSPP VIIRS EDR Package For those that are not, CSPP to determine if they want to add JPSS Single EAs3 or offer individually.

Gaps

1 Products highlighted in red text may be available as Single EA/DAPs but not currently contained in the initial JPSS Cloud, Aerosol, Cryosphere and Volcanic Ash EA/DAP.

2 Land products to be included in Operational EA/DAP delivery once released to operation.3 Not all JPSS products can be integrated into a multiple EA.

14

JPSS/CSPP NUCAPS EA/DAPComparison

Single DAPs CSPP Products in the JPSS ATMS/CrIS NUCAPS Package



(System) CSPP NUCAPS Product Names JPSS EA Availability Status

Atmospheric Vertical Moisture Profile EDR 3 ESPC Moisture Profile JPSS Individual EA

Atmospheric Vertical Temperature Profile EDR 3 ESPC Temperature Profile JPSS Individual EACarbon Dioxide EDR 4 ESPC Carbon Dioxide JPSS Individual EACarbon Monoxide EDR 4 ESPC Carbon Monoxide JPSS Individual EAInfrared Ozone Profile EDR 3 ESPC Infrared Ozone Profile JPSS Single EAMethane EDR 4 ESPC Methane JPSS Single EAMicrowave Imagery EDR 3 ESPC Microwave Imagery JPSS Single EAOutgoing Long Wave Radiation EDR 3 ESPC Cloud Liquid Water JPSS Single EA

JPSS DB SUPPORT EFFORT Identify EA/DAP needs for NUCAPS Identify CSPP needs for IASI & AIRS to STAR

1 NUCAPS was "NOAA-Unique CrIS ATMS Product System" but since NUCAPS now runs on Metop-A/B soundings as well as for the future JPSS sounders, it was changed to "NOAA-Unique Combined Atmospheric Processing System"

NOAA-Unique Combined Atmospheric Processing System (NUCAPS)1 is a joint infrared and microwave algorithm, and the retrieved soundings are IR+MW running on CrIS/ATMS and IASI/AMSU data.

15

JPSS/CSPP MiRS EA/DAP Comparison

Single EA/DAP CSPP Products in the JPSS ATMS/CrIS MiRS Package



(System) CSPP Microwave Integrated Retrieval System (MiRS) JPSS EA Availability Status

Atmospheric Vertical Moisture Profile EDR 3 ESPC Moisture Profile JPSS Multiple EA

Atmospheric Vertical Temperature Profile EDR 3 ESPC Temperature Profile JPSS Multiple EACloud Liquid Water EDR 3 ESPC Cloud Liquid Water JPSS Multiple EAMicrowave Imagery EDR 3 ESPC Imagery JPSS Multiple EALand Surface Temperature EDR 4 ESPC Land Surface Temperature JPSS Multiple EALand Surface Emissivity EDR 2 ESPC Surface Type Classification JPSS Multiple EA

Rainfall Rate EDR 3 ESPC Rainfall Rate JPSS Multiple EA

Sea Ice Concentration EDR 3 ESPC Sea Ice Concentration JPSS Multiple EASnow Cover Depth EDR 3 ESPC Snow Cover Extent JPSS Multiple EASnowfall Rate TBD 3 TBD Snowfall Rate JPSS Multiple EA

Snow Water Equivalent EDR 3 ESPC Snow Water Equivalent JPSS Multiple EA

Total Precipitable Water EDR 3 ESPC Total Precipitable Water JPSS Multiple EAJPSS DB SUPPORT EFFORT

STAR’s MiRS team delivered to CSPP an EA/DAP, need to identify if CSPP needs additional software versions or patches.

Microwave Integrated Retrieval System (MiRS) is for retrieval of atmospheric profile and surface parameters from ATMS, AMSU-A, and MHS microwave sensor data.

16

Establish JPSS DB Support Working Group

PRIMARY JPSS DB SUPPORT EFFORT Conduct monthly meetings with

STAR/CSPP/NJO Leadership and NOAA users

Also a JPSS Forum for CSPP NOAA User Feedback

17

JPSS DB Support Potential Schedule

Phase 1 InceptionIdentify DAP products to be made available to CSPP, deliver JPSS VIIRS Multiple EA pathfinder, along with GCOM and Active Fire EA/DAP, document lessons learned

Phase 2 ElaborationDevelop integrationprocesses based on EA/DAP pathfinders, implement CSPP into STAR EPL deliverySchedule, identify other EA/DAPs

Phase 3 ConstructionImplement STAR processes for repeated delivery of other EA/DAPs, Assess feasibility of providing other types ofEA/DAPs.

Phase 4 TransitionUpdate CSPP Website, Notify Users, Receive Feedback

JPSS DB Support is now focusing on building best practices for JPSS operational DAP delivery.

Potential Timeline

FY18 Q3

FY18 Q4

FY19Q1

FY19Q2

FY19Q3

FY19Q4

FY20Q1

FY20Q2

18

▪ JPSS is offering NOAA/NESDIS endorsed Individual and Multiple EA/DAPs to CSPP. By ensuring CSPP has the most recent JPSS EDR algorithms versions available, the closer the end user will have to JPSS operational products.

▪ JPSS will leverage NOAA/NESDIS best practices to develop repeatable processes from NOAA/NESDIS/STAR Enterprise Product Lifecycle (EPL), Configuration Management (CM) and Quality Assurance (QA) processes to standardize EA/DAP deliveries to CSPP and create a JPSS Direct Broadcast Support Working Group for NOAA users

▪ JPSS will leverage lessons learned from STAR’s EA/DAP delivery to CSPP for MIRS, offer CSPP a Multiple EA/DAP for VIIRS data, and create a project plan for future STAR releases of Individual and multiple EA/DAPs based on NOAA operational releases and CSPP needs.

▪ JPSS will work with CSPP to understand the needs and timeline for implementing these changes.

Summary

19

Questions/Answers

20

Backup

21

6.1.1.10 JPSS-1 shall provide a real-time X-band direct broadcast of instrument data to the direct readout community (i.e., High Rate Data (HRD)). [Priority 3]

6.1.2.10 JPSS-2 shall provide a real-time X-band direct broadcast of instrument data to the direct readout community (i.e., High Rate Data (HRD)). [Priority 3]

6.2.13 JPSS shall provide the Direct Readout community with software, documentation, and periodic updates to enable civilian and military agencies to produce data products from JPSS, using their own hardware to receive the JPSS High Rate Data (HRD) broadcast. [Priority 3]

▪ Direct Broadcast: A spacecraft capability to broadcast satellite mission data via a High Rate Data (HRD) downlink directly to users equipped with suitable field terminals.

▪ Direct Readout: Refers to the user community on the ground that receives and processes HRD data in their own field terminals and to the support provided to those users by the JPSS program.

JPSS Level 1 Requirements

22

EDRs: Cloud Liquid WaterIce ConcentrationImageryLand Surface EmissivityLand Surface TemperatureMoisture Profile

Rainfall RateSnow CoverSnow Water EquivalentTemperature ProfileTotal Perceptible Water

ATMS (11 EDRs)

JPSS Science Data Provided to the NOAA Line Offices & Around the World

23

JPSS Multi Mission System

Sustain • Enable • Create – OSGS

GEARS ArchitectureNESDIS Ground Enterprise (NGE)

Fairbanks GroundStation – POES

GOES/GOES-R(West)

GOES/GOES-R(East)

TDRS(Satellite)

White Sands

TrollSatAntarctic

METOP-A,B (AM)Secondary

BackupNOAA POES

(AM/PM)

CBU Fairmont

SNPP/JPSS-1 (PM)

Non-NOAA

Storage and Access (CLASS)

Satellite Operations

Data Products

Ingest, Process and Distribute

Non–Satellite Data

CDACDALEGEND

Commanding & Telemetry

Commanding or Telemetry

GOES Re-Broadcast

Direct Readout

User Community

NSOF

CDA

Svalbard Ground

Station –JPSS

McMurdo GroundStation–JPSS D/L

Wallops GroundStation

GOES / POES

User Products

Non–NOAASatellite Data

Apr 2016

24