page 1 ocean peate fred patt npp sds peer mor building 8 auditorium august 27, 2008

Ocean PEATE

Fred Patt

NPP SDS Peer MOR

Building 8 Auditorium

August 27, 2008

Ocean PEATE Agenda

• Ocean PEATE Overview

• Implementation Status

• Documentation

• Support Staff

• EDR Assessment

• Issues/Concerns

LandLand

SD3ESD3E

PSOEPSOE

OceanOcean

AtmosphereAtmosphere

OzoneOzone

SounderSounder

I&TSE

NICSE

Ocean PEATE Overview

● The NPP Ocean PEATE will be implemented within the framework and facilities of the current NASA Ocean Data Processing System (ODPS)

● This system has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions.

Ocean PEATE Functions

• Acquire VIIRS RDRs, SDRs, and Ocean EDRs from the SD3E and ADS/CLASS

• Assess the quality of the NPP Ocean EDRs for accomplishing NASA’s climate research requirements

• Provide suggested algorithm improvements to the IDPS via the Project Science Office Element (PSOE)

• Process selected data subsets in support of Evaluation and Validation activities

Ocean PEATE Interface Diagram

PSOESD3E

I&TSE

NICSE

VOST

CLASS(ADS)

AncillaryData

Providers

Ocean Science

Community

Casa-NOSA

xDRs, IPs, Ancillary Data

Alternate Ancillary Data

Management Direction

Calibration Updates and Evaluations

Interaction

xDR Eval. Results, Algorithm Updates

Pre-flight

Algorithms,

Data, Info

Software, Data

In Situ Data

OceanPEATE

Algorithm Updates, Test Requests &

Results

xDRs, IPs, Ancillary Data(if unavailable from

SD3E)

Analysis Results, Proposed

Algorithm Updates

RSMASSeaBASS

In Situ Data

Matchups

Ocean PEATE External Interfaces (1 of 2)

• SDS Science Data Distribution and Depository Element (SD3E)– Provides NRT access to raw data– Primary source of RDRs– Provides selected SDRs and EDRs

• SDS Integration and Test System Element (I&TSE)– Build and test updates to operational code in mini-IDPS– Run tests on selected data per request of PEATE

• Archive Distribution Segment (ADS) – Primary source for archived data

• xDRs, IPs, Ancillary Data, Operational Algorithm/Source Code and Calibration Products

• Ancillary Data Providers (ADP)– Provides alternate ancillary data sets (e.g., ozone, meteorological data sets)

• CasaNOSA– Serves as the NPP pre-flight repository of Government held data for distribution to

Government user teams– Place to acquire pre-launch NPP algorithms and supported data files

Ocean PEATE External Interfaces (2 of 2)

• NASA VIIRS Ocean Science Team (VOST)– Coordinate activities with PEATEs and PSOE on xDR and recommended

algorithm improvements. Supports Independent Calibration Validation Activities• NPP Instrument Calibration Support Element (NICSE)

– Provides alternative calibration LUTs and recommended improvements to calibration algorithms

– PEATE provides results of LUT and algorithm tests• Project Science Office Element (PSOE)

– Provides management direction– Accepts algorithm update recommendations

• SeaBASS/ODPS– Provides Ocean Color in situ data

• RSMAS/U. Miami– Provides SST in situ locations– PEATE provides SST EDR matchups

• Ocean Science Community– Relies on Ocean PEATE to provide evaluation products and results

Roles & Responsibilities of Science Team / PEATE

PEATE Science Team

Acquire, catalog, and distribute xDRs

Perform automated matchups with SeaBASS data

Subscribe, browse, download xDRs

Evaluate Ocean EDRs using matchups

Generate Level-3 products from Ocean EDRs

Compare VIIRS Ocean Level-3 files with heritage sensors

Acquire test LUTS from NICSE Evaluate VIIRS calibration using test LUTS

Deliver/install recommended software updates at I&TSE

Generate recommended algorithm changes

Upload RDRs to I&TSE, input test run requests and download SDRs and/or EDRs

Develop algorithm test cases and scenarios and evaluates test products

Interface with SDS, I&TSE, ADS, … Provide feedback to Project Scientist

Ocean PEATE Agenda

LandLand

SD3ESD3E

PSOEPSOE

OceanOcean


OzoneOzone

SounderSounder

I&TSE

NICSE



• Documentation

• Support Staff

• EDR Assessment

• Issues/Concerns

ODPS Design Overview

• Fully automated, distributed data system for acquiring, processing, archiving, and distributing scientific data

• Highly scalable

• Easily adaptable to support multiple concurrent missions

• Graphical user interfaces for controlling and monitoring system functions and activity

• Non-platform specific

ODPS COTS and Freeware

• Linux OS (CentOS 4.x)• Solaris OS• Sybase RDBMS• Subversion (source code management)• Pro-active DBA• Interactive Data Language (IDL)• Generic Mapping Tool (GMT)• Netpbm (graphic image toolkit)• HDF5 Library• Languages: C, PERL, SQL

ODPS Architecture: Hardware

• Processing Servers• Intel-based dual Xeon / AMD-based dual Opteron• 8 GB RAM• Five 72 GB SCSI drives

• Storage Servers• Intel-based P4 / AMD-based single Opteron• 1 GB / 2 GB RAM• 1.5 TB IDE RAID 5 (3ware) / 9.6 TB SATA RAID 6

(Areca)• 2 hot spare drives per RAID5

• Database Server• Sun V880• 8-16 GB RAM• 6-12 70 GB SCSI HDD

ODPS Current Components

Processing Cluster 45 processing nodes

12.4 TB

Ingest Servers2 SeaSpace ground stations

2 processing nodes5.6 TB

Distribution Servers (FTP)1 processing node

6 storage nodes48 TB Distribution Servers (web)

1 large server8 processing nodes

68 storage nodes599 TB

Network Support Systems

Database Server2 servers

18 TB

Backup Servers2 storage nodes

18 TBCritical data mirrored

Extreme NetworksBlack Diamond 6816

Gigabit Ethernet switch

Development Servers1 processing node

2 storage nodes2.4 TB

User Desktops

Cal/Val & QC Systems

Mission Operations Systems

Building 28 Room W220 Computing Facility

Reliability and Redundancy

• Critical components (database server, network systems) have full maintenance contracts to ensure rapid response to problems

• Multiple-server components (ingest, processing, storage, distribution) have substantial redundancy to maintain full capability; spares maintained for rapid replacement.

• Testing nodes are separated from mainstream production components.

Technology Refresh

• Hardware technology advances (CPU, storage) are continuously monitored to select new components for evaluation.– Typical upgrade threshold is a doubling in capacity (~18 months).

– Two generations of hardware are generally in use.

• Candidate components are procured, installed in testing cluster and rigorously evaluated in a production-like environment.

• Following successful evaluation, multiple copies are procured, installed, tested and swapped in for older components.– ODPS design allows new components to be rapidly added to resource

tables without interrupting system operations.

• Performance of new components is closely evaluated following installation in operations environment.

• Critical components are run in parallel with existing system to ensure reliability under production loading.

Existing Software Reuse

• ODPS Components– Database– VDC/Scheduler– Data Acquisition and Ingest– Level-3 Scheduler– File migration and management– Archive Device Manager– Data distribution

• Level-2 multi-mission software (vicarious calibration)• Level-3 multi-mission software (long-term trends and

comparisons• Level-2 to Level-3 comparison software (residual sensor

errors)• SeaBASS (in situ data management)• Matchup/extraction software

Ocean PEATE New Capabilities (1 of 2)

• Acquire, ingest and catalog NPP VIIRS data products: RDRs, SDRs and Ocean EDRs (Data Acquisition & Ingest, Device Manager and File Migration and Management).– Status: Initial interface test run with SD3E.

• Process selected Ocean EDRs (SST and OCC) to Level-3 to support data product and algorithm evaluations (Level-3 Scheduler, VDC and Level-3 binner).– Status: Prototype Level-3 processing has been demonstrated using

sample IDPS Build 1.4 OCC and SST EDRs.

• Perform VIIRS OCC EDR matchups with SeaBASS Ocean Color in situ data (extract code).– Status: Pending re-test of acquisition and ingest capabilities with

latest IDPS build.

Ocean PEATE New Capabilities (2 of 2)

• Incorporate VIIRS SDR processing for vicarious calibration analysis.– Status: Pending re-test of acquisition and ingest capabilities with

latest IDPS build.

• Produce VIIRS proxy data using VOST-developed software (VDC/Scheduler).– Status: Under development

• Acquire SST in situ data from RSMAS and perform matchups with SST EDRs– Status: SST acquisition in place for MODIS; matchups pending re-

test of acquisition and ingest capabilities with latest IDPS build.

• Support browse and distribution of data products for team members (Data Distribution).– Status: Pending re-test of acquisition and ingest capabilities with

latest IDPS build.

Ocean PEATE Data Storage Estimate

Data Type Daily 1 Year 5 Years

RDR 150 GB 53.5 TB 268 TB

SDR (M-band) 242 GB(2) 8.6 TB(1,2) 43 TB(1,2)

OCC EDR 84 GB 3 TB(1) 15 TB(1)

SST EDR 19 GB 0.7 TB(1) 3.4 TB(1)

Inter. Products ~70 GB N/A N/A

Ancillary Data 0.1 GB .04 TB .2 TB

Total 565 GB 66 TB 330 TB

Assumptions: (1) Long-term storage is sized for 100% of RDRs and 10% of SDRs and EDRs(2) SDR volume includes geolocation

New Hardware for the Ocean PEATE

• 7 Storage Servers @ 9.6 TB – first-year VIIRS data storage

• Additional servers acquired post-launch to handle years 2 – 5

• No new processing or network capacity required; technology refresh cycle to be continued within the ODPS as described.

Ocean PEATE Build Schedule

Build 1 (L-18 months)• All interfaces fully

implemented and tested• Verify initial versions of

operational code ported and running in I&TSE

• L-3 product code developed and tested

• Prelaunch VIIRS test data storage and SDS interface testing support with existing ODPS storage capacity

• Initial test products generated for review by VIIRS Ocean Science Team

Build 2 (L-12 months)• Routine exercise of interfaces to

acquire proxy, surrogate (Aqua?) and/or simulated data

• Verify pre-launch version of operational code running in I&TSE

• Browse and distribution capability developed and tested

• Test products routinely acquired as available and posted for access by VIIRS Ocean Science Team

• Data storage for one year

Ocean PEATE Testing Status (1 of 6)

Req. # Requirement Support Verification Status

3.4.1.1 PEATEs xDR Ingest from the SD3E New Data Acquisition and Ingest Interface Test X

3.4.1.2 PEATEs IP Ingest from the SD3E New Data Acquisition and Ingest Interface Test

3.4.1.3PEATEs Official Ancillary Data Ingest from the SD3E

New Data Acquisition and Ingest Interface Test

3.4.1.4PEATEs Calibration Product Ingest from the SD3E


3.4.1.5 PEATEs Request Listings New Data Acquisition and Ingest Interface Test X

3.4.1.6PEATEs Product Subscriptions from the ADS

PEATE Personnel Interface Test

3.4.1.7PEATEs Ad-hoc Requests to the ADS


3.4.1.8 PEATEs xDR Ingest from the ADS New Data Acquisition and Ingest Interface Test

3.4.1.9PEATEs Official Ancillary Data Ingest from ADS


3.4.1.10PEATEs Calibration Product Ingest from ADS




3.4.1.11 PEATEs IP Ingest from the ADS New Data Acquisition and Ingest Interface Test

3.4.1.12PEATEs ADS Query and Request Status Ingest


3.4.1.13PEATEs Alternate Ancillary Data Ingest

Existing ODPS Capability Inspection X

3.4.1.14 PEATEs Pre-launch Data Ingest Existing ODPS Capability Inspection X

3.4.1.15 PEATEs Engineering Service Ingest PEATE Personnel Interface Test

3.4.2.1 Store/Catalog Data New Data Management I&TSE Functional Test X

3.4.3.1 Manage Software Configuration Existing Subversion Repository Inspection X

3.4.4.1Operational Algorithms Ingest from the SD3E

I&TSE Functional Test

3.4.4.2PEATEs Management Direction Ingest


3.4.4.3PEATEs Instrument Service Report Ingest




3.4.4.4 PEATEs NICSE Cal LUT Ingest I&TSE Interface Test

3.4.4.5 PEATEs I&TSE Algorithm Status PEATE Personnel Interface Test

3.4.4.6 Pre-Launch Assessment New PEATE Tool Development Functional Test

3.4.4.7 xDR Implementation Assessment I&TSE Functional Test

3.4.4.8 EDR Algorithms I&TSE Functional Test

3.4.4.9 SDR and EDR Testing Environment I&TSE Functional Test

3.4.4.10 Comparison Tools Development Existing ODPS Capability Inspection X

3.4.4.11 Comparing against Other Data New Level-3 Processing Functional Test

3.4.4.12 Comparing against Other Products New Level-3 Processing Functional Test

3.4.4.13Analyzing Algorithm Improvement/ Enhancement

I&TSE/IngestInterface/ Functional Test



3.4.4.14 Generating SDR I&TSE Functional Test

3.4.4.15 Generating EDR I&TSE Functional Test

3.4.4.16 Test DataVOST Simulation Software / ODPS New Data Processing

Design

3.4.4.17Developing SDR Algorithm Improvement/Enhancement


3.4.4.18Developing EDR Algorithm Improvements/Enhancements


3.4.4.19 Testing Algorithm I&TSE/Acquisition and IngestFunctional/ Interface Test

3.4.4.20 Documentation ODPS Web Site Inspection X

3.4.4.21 PEATEs Management Report PEATE Personnel Interface Test

3.4.4.22 PEATEs Instrument Service Request PEATE Personnel Interface Test

3.4.4.23 PEATEs Quality Control Response I&TSE Functional Test



3.4.4.24 PEATEs xDR Ingest from the SD3E New Data Acquisition and Ingest Interface Test

3.4.5.1 PEATEs Science Data Export New Product Distribution Interface Test

3.4.5.2 PEATEs Data Compression Existing ODPS Capability Inspection X

3.4.8.1 Ocean Product AssessmentNew and Existing ODPS Capabilities

Functional Test

3.4.8.2 Ocean EDR EvaluationNew and Existing ODPS Capabilities

Functional Test

3.7.1 Mission Support All New CapabilitiesAggregate of All Testing

3.7.3 Availability Existing ODPS Capabilty Inspection X

3.7.4 System Failure Existing ODPS Capabilty Inspection X

3.7.5 Operational Duty Cycle Existing ODPS Capabilty Inspection X

3.7.6 Design Life New Data Product Storage Inspection



3.7.7 Maintainability Existing ODPS Capability Inspection X

3.7.8.1 Unclassified Security Restriction Existing ODPS Facility Inspection X

3.7.8.1Complying with NASA Security of Information. Technology1

Code 600 Security Plan Inspection X

3.7.8.1 Operators Access and Privilege Existing ODPS Capability Inspection X

3.7.8.1 SDS Data Verification New Data Acquisition and Ingest Interface Test

3.7.9 Data Integrity New Data Acquisition and Ingest Interface Test

Ocean PEATERequirements Implementation

• 98% of requirements (55) implemented by Build 2; add additional storage capacity before launch

Requirements Met

16

5454 5555 56

0

10

20

30

40

50

60

MOR Build 1 Build 2 Launch

# o

f R

eq

s

Build 1Build 2Launch

Sustaining Operations and Maintenance

• ODPS runs 24x7 with on-site support 8x5.• Support is shared across all projects.

• Staffing needs are covered by existing OBPG support personnel.

Ocean PEATE Agenda

LandLand

SD3ESD3E

PSOEPSOE

OceanOcean


OzoneOzone

SounderSounder

I&TSE

NICSE



• Documentation

• Support Staff

• EDR Assessment

• Issues/Concerns

Documentation

• Ocean PEATE Level 4 Requirements and Operations Concept

• ICDs– SD3D to PEATEs

• MOU with RSMAS• ODPS Project Data Management Plan (review)• ODPS Software Management Plan• Network IT Security Plan

Ocean PEATE Agenda

LandLand

SD3ESD3E

PSOEPSOE

OceanOcean


OzoneOzone

SounderSounder

I&TSE

NICSE



• Documentation

• Support Staff

• EDR Assessment

• Issues/Concerns

Ocean PEATE Team

• Ocean PEATE Management– Gene Feldman– Fred Patt

• Validation Support– Sean Bailey– Jeremy Werdell

• Software Support– Bryan Franz– Joel Gales

• Data System– John Wilding– Dan Knowles– Muriel Taylor

• Systems Management– Paul Smith– Joe McSorley– Chris Moellers

Ocean PEATE Staffing Levels

VOST PEATE VOST SDS VOST SDS VOST SDSManager 0.7 0.7 0.7 0.7Sr. Sci. Prgmr/Mgr 1 1 1 1System Administrator 1 1.5 1.5 1.5Software Engineer 1.5 2 2 2Sci. Prgmr/Analyst 1.5 1.5 2 2.5Database Manager 0.5 0.5 0.5 0.5Totals 1 5.2 1 6.2 1 6.7 1 7.2Grand total

FY09 FY10 - FY14FY07 FY08

6.2 7.2 7.7 8.2

• All staffing needs will be met with existing OBPG staff• Increases will be accommodated as other projects (e.g., MODIS Aqua) wind

down.

Ocean PEATE Agenda

LandLand

SD3ESD3E

PSOEPSOE

OceanOcean


OzoneOzone

SounderSounder

I&TSE

NICSE



• Documentation

• Support Staff

• EDR Assessment

• Issues/Concerns

VIIRS Ocean Products for Evaluation

• All VIIRS Ocean products to be evaluated will be acquired from the IDPS, ADS/CLASS, or the I&TSE.

• Testing and evaluation of algorithm changes will require regenerating product time series in the I&TSE, to be downloaded to the Ocean PEATE.

• The Ocean PEATE will design changes to the code in the I&TSE for the purpose of algorithm improvement or problem resolution, develop appropriate test cases and request runs to verify and evaluate the changes.

VIIRSRDR

Previous VIIRSGridded Products

NAAPSTOD

MODISLand/Water

Mask

NCEPGeopotential

HeightAncillary

Files

DEM

NDT

AncillaryFiles


ProcessingModule

VIIRSProduct

DynamicAncillary

Data

StaticAncillary

Data

IDPS VIIRS Ocean EDR Data Flow


VIIRS_SDR_01RDR Decompression

VIIRS_GEO_01Geolocation

VIIRS_SDR.IM375m SDR

VIIRS_SDR.MOD750m SDR

VIIRSRDR


NAAPSTOD

MODISLand/Water

Mask

NCEPGeopotential

HeightAncillary

Files

DEM

NDT

AncillaryFiles


VIIRSSDRVIIRS

SDR

VIIRSGeolocation

ProcessingModule

VIIRSProduct

DynamicAncillary

Data

StaticAncillary

Data


VIIRS_GD_28Surface Pressure

Adjustment

VIIRS_GD_08750m Granulation

VIIRS_GD_25NAAPS Granulation

VIIRS_GD_27L/W Mask Granulation

VIIRS_GD_09GFS Granulation

VIIRS_GD_12BathymetryGranulation

VIIRS_GD_13TemperatureGranulation

ALL_GD_01Time Interpolation





VIIRSRDR


NAAPSTOD

MODISLand/Water

Mask

NCEPGeopotential

HeightAncillary

Files

DEM

NDT

AncillaryFiles


VIIRS_GD_11Ancillary Profile

VIIRSSDRVIIRS

SDR

VIIRSGeolocation

ProcessingModule

VIIRSProduct

DynamicAncillary

Data

StaticAncillary

Data


VIIRS_LN_06Active Fires

VIIRS_CM_01Cloud Mask



Adjustment












VIIRSRDR


NAAPSTOD

MODISLand/Water

Mask

NCEPGeopotential

HeightAncillary

Files

DEM

NDT

AncillaryFiles


VIIRSSDRVIIRS

SDR

VIIRSGeolocation

ProcessingModule

VIIRSProduct

DynamicAncillary

Data

StaticAncillary

Data

VIIRS_OC_01Ocean Color /

Chlorophyll

VIIRS_ST_02 Surface Temp

VIIRS_SN_03Ice Concentration

VIIRS_ST_01Sea SurfaceTemperature

VIIRS_AR_01Aerosol Type

VIIRS_SN_02Ice Quality

VIIRS_LN_06Active Fires

VIIRS_CL_01Cloud Optical

Properties

VIIRS_CM_01Cloud Mask



Adjustment












SSTEDR

OCCEDR

VIIRSRDR


NAAPSTOD

MODISLand/Water

Mask

NCEPGeopotential

HeightAncillary

Files

DEM

NDT

AncillaryFiles


ProcessingModule

VIIRSProduct

DynamicAncillary

Data

StaticAncillary

Data

VIIRSSDRVIIRS

SDR

VIIRSGeolocation


ODPS MODIS Ocean Product Data Flow

ProcessingModule

MODISProduct

DynamicAncillary

Data

StaticAncillary

Data

MOD_PR01Level-0 to 1A

MOD_PR03Geolocation

MSl12Level-1B to 2

MOD_PR02Level-1A to 1B

MODISLevel-0

Land/WaterMask

PlatformATTEPH

Data

OzoneAncillary

Files

METAncillary

Data

MODIS1 km

Level-1B

MODISGeolocation

MODISLevel-1A

MODISSST

MODISOceanColor

Ocean Team Algorithm Change Process

ST member identifies

algorithm change

ST analyzesproducts andtest results

ST member develops evaluationstrategy

ST member andPEATE staff

develop code change

PEATE staffrequest ITSE

granule-level test

PEATE retrievestest products

ST providesalgorithm changerecommendation

to PSOE

PEATE and ITSEstaff install and

build code change

ITSE performs test run

ST member verifies test

products

PEATE staffupload test inputdata and request

evaluation test run

Expected results?

Productsimproved?

N Y

ITSE performs test run

PEATE retrievestest products

ST member evaluates test

products

N

Y

PEATE and STmember checkcode change

Code correct?

N

Y

Evaluation vs. Product Level

Granule-based Evaluations• Level-1 (SDR) Evaluations

– Onboard calibration analyses– Vicarious calibration

• Level-2 (EDR) Evaluations– Matchup analyses– Flag analyses

Time Series Evaluations• Level-2 (EDR) Evaluations

– Residual detector (striping) and scan (RVS) dependence• Level-3 Product Evaluations

– Sensor and algorithm comparisons– Temporal anomaly evaluations

Vicarious Calibration

• Vicarious calibration using surface measurements (e.g., MOBY) gives constant gain correction to TOA radiances over the mission

• Analyses of calibration results provide insight into temporal and geometric sources of variation.

Vicarious Calibration

• Strengths:– Fine-tuning of sensor absolute radiometric response; allows for

correction of prelaunch radiometric calibration

• Limitations:– “System-level” calibration that incorporates atmospheric

correction into measurements, convolving algorithm and sensor effects.

– One-time gain correction per band for the entire sensor lifetime; assumes sensor stability has already been established through other means; not a substitute for lack of instrument characterization.

– Observation limitations and rigorous data quality control severely restrict the number of useful measurements (typically 1 or 2 per month per site).

– Specific to ocean algorithms, not applicable to other surface types.– Availability of MOBY or equivalent is unknown for VIIRS.

Matchup Analysis

• Ocean data granules in ODPS catalog are automatically matched with in situ data

• SeaWiFS Bio-optical Archive and Storage System (SeaBASS) stores and manages in situ holdings from field programs and supported investigators.

• Ocean staff acquire, QC and analyze new data samples.

• Over 60,000 in situ profiles stored covering SeaWiFS/ MODIS time frame.

Matchup Process Flow Chart

Matchup Analysis

• Strengths:– The only truly independent validation of the science data products

using ground truth measurements.

• Limitations:– Quality of in situ data is highly variable and difficult to assess.– Coverage for OC in situ data is limited, both geographically and

temporally.– Assumes that highly localized (~meters) measurements are

representative of pixel (km) area.– In situ measurements require discipline expertise to analyze and

compare with satellite values– Generally useful only for assessing static biases in final products.– Availability of in situ data in VIIRS timeframe is unknown.

Flag Analyses

50-meter depth Sensor zenith

• Determine loss of coverage resulting from each flag

• Verify effectiveness of flags at removing invalid or questionable data

Residual Detector Dependence

• Level-3 multi-day products (same or different sensor) are resampled to generate values at Level-2 resolution.

• Comparisons are performed by detector and mirror side to evaluate residual detector response errors

Residual Scan Dependence

• Level-3 multi-day products (same or different sensor) are resampled to generate values at Level-2 resolution.

• Comparisons are performed by pixel number and mirror side to evaluate residual response-vs-scan (RVS) errors

Residual Detector/Scan Dependence

• Strengths:

– Provides evaluation of response vs. detector, scan angle and mirror side independent of onboard calibration.

– Can also be used to evaluate errors in other sensor characteristics

• Limitations:

– Works well mainly in temporally stable areas.

– Noise levels may be large compared with residual errors.

Level-3 Product Generation

• Sensor, algorithm and interannual comparisons require Level-3 binned (equal-area) products.

• The Ocean PEATE is implementing software to process VIIRS EDRs to Level 3 binned products in current ODPS (SeaWiFS-like) format.

– Use current binning code with new input functions to read EDRs

• This will automatically provide the additional capabilities to produce multi-temporal composites, standard mapped image (SMI) products and Level-3 browse files.

• Time-series analyses of recommended algorithm changes will require reprocessing EDRs in the I&TSE.

Sensor and Algorithm Comparisons

• Level-3 parameters (e.g., nLw) compared for common spectral bands

• Common bins extracted and compared over the period of overlap between the sensors

• Comparisons are performed globally (deep water, oligotrophic, mesotrophic, eutrophic), zonally and for specified regions.

Zonal Cross-Comparisons

Chlorophyll Image Comparisons

SeaWiFS Deep-Water Chlorophyll 2002 193

SeaWiFS Deep-Water Chlorophyll 2006 365

MODIS Deep-Water Chlorophyll 2002 193

MODIS Deep-Water Chlorophyll 2006 365

Level-3 Comparisons

• Strengths:

– Sensitive to small differences in products from different sensors or algorithms.

– Excellent coverage available, both temporal and geographic.

– Can assess continuity among data sets (Climate Data Records)

• Limitations:

– No obvious truth in comparisons.

– Sensitive to band-pass differences.

– May be affected by time-of-observation differences.

– Availability of other sensors for comparison (SeaWiFS or MODIS) is unknown; climatology is available as an alternative

Temporal Anomaly Evaluations

• Level-3 global averages for the entire mission are fit to a periodic function to remove natural annual variability.

• The differences between the global averages and the annual cycle are then plotted over the mission.

• The results show both geophysical variations and any unexpected changes due to uncharacterized instrument effects or algorithm artifacts.

Temporal Anomaly Example

Temporal Anomaly Evaluations

• Strengths:– Very sensitive to small changes in sensor

performance

• Limitations:– Difficult to distinguish sensor from real

geophysical changes.– Can be affected by sampling variations.

Issues/Challenges/Concerns

Issue/Challenge/Concern Action/MitigationLimits of available test data• Ability to adequately test data flows across interfaces• Verification of EDR evaluation processes

VIIRS Ocean Science Team will develop an independent simulation of the VIIRS ocean bands

Extent of prelaunch end-to-end (observatory & ground system) data flows

Develop/maintain system readiness for testing opportunities

ADS to PEATE interface and test planning Communication initiated, TIM planned

Complexity of IDPS internal processes and data flows impacts ability to diagnose software and algorithm problems in the I&TSE

Exercise in the I&TSE as possible

Version synchronization of operational products with IDPS software in the I&TSE

Coordinate software updates with I&TSE support staff

SD3E to Ocean PEATE data transfers Performance improvements being worked on both sides

IDPS file naming convention Attempt to get patch for I&TSE; develop workarounds for testing with SD3E

page 1 ocean peate fred patt npp sds peer mor building 8 auditorium august 27, 2008

Documents

selected data

data subsets

sds science data distribution

distributed data system

situ data rsmasu

npp ocean edrs

test system element

calibration algorithmspeate