management of amsr-e and glas data by nsidc melinda marquis amsr/amsr-e data management team lead...

Management of AMSR-E and GLAS Data by NSIDC

Melinda MarquisAMSR/AMSR-E Data Management Team Lead

GLAS/ICESat Data Management Team LeadNational Snow and Ice Data Center

PoDAG Meeting April 25 & 26, 2001

Outline

• Data Flows• Data Distribution Services• Documentation• Post-launch communications• Validation• Operations• Testing• Progress & Milestones• Subsetting• Products tables

AMSR-E Data Flow

EDOSL0

L0

GLAS Data Flow

EOS Data Gateway (EDG)

User

Data Management Subsystem

Data Server Subsystem

Data Archive(Powderhorn)

Science Data Server Database

Distribution of data by ECS

Search

and Order

Granules Distributed

Data Distribution Services

• Data centers provide different levels of service for different products

• NSIDC distributes data at the same granularity as it is archived

1. We plan to limit distribution of level 0 AMSR-E and GLAS data

2. We intend to offer additional services, such as subsetting

Data Support Services Provided by NSIDC

• DIFs (submitted to the Global Change Master Directory)

• Guide Documents (User guides)• Web Site• Newsletter• DAAC Yearbook• Outreach

Directory Interchange Formats (DIFs)

• A DIF is a metadata document

• Submitted to the NASA’s Global Change Master Directory (GCMD) http://gcmd.gsfc.nasa.gov/

• GCMD is a comprehensive directory of descriptions of data sets relevant to global change research.

• A DIF is a collection of fields that allow user to decide whether a particular data set would be useful to them.

• Compatible with U.S. federally mandated Federal Geographic Data Committee’s (FCDC) Content Standard on Digital Geospatial Metadata.

Guide Documents

• Is a technical document (user’s guide) for data products funded by NASA.

• Must follow a specific template

• Template at (http://harp.gsfc.nasa.gov/v0ims/DOCUMENTATION/GUIDE-VALID/GUIDES/TemplateECS2.htm).

• Contains detailed information about the data product, allowing an uninitiated user to understand and use the data fully and accurately.

Post-Launch Communications between SIPS/SCF/Science Team and NSIDC

• Communication will continue to be important after launch

• Either document protocol in Operations Agreement (OA) between NSIDC and SIPS or in Memorandum of Understanding (MOU) between NSIDC and SCF


• NSIDC’s web site will provide information on:– Availability of products

– Status of Aqua and of AMSR-E and of ICESat and of GLAS

– Known errors

• Any changes to processing will need to be conveyed so our documentation can be updated– So we can inform users, including registered users


USER

NSIDC USO

NSIDC IN-HOUSE RESOURCES

Programmers Ops Science Staff System Engineering/Admin.

OUTSIDE RESOURCES

Science Team Others

Documentation Specialist

Documentation

Science Operations Support Staff

As needed

SIPS

Validation – AMSR-E

• Richard Armstrong, Jim Maslanik and Julienne Stroeve are actively involved in validation efforts in support of the AMSR-E science team

• Therefore, they will perform very high level QA of AMSR-E data products

Archive Migration

• Old drives and media will no longer be supported by Storage Tek.

• Installation of new drives and media began in April 2001.

• All data, including ASMR-E and GLAS will be archived on the new media, which will be the 9940 tapes from Storage Tek.

PDS (Product Distribution System)

• Installation completed by the second week in April 2001.

• The PDS system capabilities:

– CD-R

– 8MM tape

– DV

• Automated QC of all media (not data)

• Automated label printing

Mission Operations and Science Systems (MOSS) Tests – Aqua/AMSR-E

• Formal test exercise to demonstrate Aqua Science Operations Concurrent Aqua test load with nominal Terra operations

Mission Operations and Science Systems (MOSS) Tests

Aqua science systems end-to-end test from EDOS through science data processing facilities (AMSR-E SIPS) to assess readiness for:

• Demonstration of system performance and concurrent operations ( ingest, archive, processing, user access, and distribution) for Terra, Landsat-7 and Aqua

• Verify compatibility of AMSR data products with ECS data types

• Ability to support early post-launch science operations with DAAC/SIPS production using simulated data

• Demonstrate subscriptions, user pull and distribution (ftp and tape)

• Demonstrate nominal operations personnel and procedures

• Identify critical liens, problems, or defects that must be resolved prior to Aqua launch

MOSS 1Summary Totals

• Data Provider Data Type Granules Ingested• EDOS AE_PMSCI 21• PM1GBAD1 19• PM1GBAD4 2• PM1GDAB8 2• GHCC AE_L2A 47• AE_Ocean 48• AE_Land 48• AE_Rain 48• AE_RnGd• AE_DyOcn• AE_WkOcn• AE_M0Ocn• AE_DySno 2• AE_5DSno 2• AE_MoSno 2• AE_S16 2• AE_SI12 2• AE_S125 2• AE_Land3 2• Browse 46

• PO.DAAC AMSREL1A 53

MOSS 2Summary Totals

• Data Provider Data Type Granules Ingested

• EDOS AE_PMSCI 32• PM1GBAD1 33• PM1GBAD4 6• PM1GDAB8 6•• GHCC AE_L2A 56• AE_Ocean 56• AE_Land 56• AE_Rain 56•• PO.DAAC AMSREL1A 56••

Future MOSS Tests (Aqua/AMSR-E)

• MOSS 3 : May 14 - May 18

• MOSS 4 : July 16 - July 20

• MOSS 5 : September 17 - September 21

GLAS Test Schedule

• April 24: ICT -- went well. We received PDRs and PDSs for 17 L0 APIDs. All successfully ingested.

• May 14-16: MOSS 3 (part of Aqua MOSS)

• July 16-18: MOSS 4 (part of Aqua MOSS)

• July 2-31: IGST-4

• July 10: IGST-6

• Sept. 3- Oct. 5: IGST-7

Test Schedule: AMSR-E and GLASDate AMSR-E GLASDec. 4-7, 00

MOSS-1

X

Feb. 20-22, 01

MOSS 2

X

May 14-18

MOSS 3

X X

July 2-31

IGST-4

X

July 10

IGST-6

X

July 16-20

MOSS 4

X X

Sept. 17-Sept. 21

MOSS 5

X

Sept. 3- Oct. 5

IGST-7

X

Target launch date NET Dec., 2001

Dec. 15, 2001

Progress – AMSR-E

•ECS System Upgrades needed for AMSR-E

–ECS 5B and 55 Integration and Test•Install and check out Release 5B (this was completed by July 2000, i.e., before FY01)

•Install and check out Release 55 Release 55 has been split into 2 parts:

–a) 5B06 (Dec. 2000)

–b) IRIX upgrade/SGI (2nd quarter FY01)

• 5B07 (March 12 in TS1; March 19 in OPS)

oSystem Tests–Perform Regressions Tests on Release 55 (2nd quarter FY01)

–ECS 55 Site Readiness Acceptance (2nd quarter FY01)

–System Confidence Tests (SCTs) (One already during 1st quarter; maybe one or more in future.)

–MOSS (Mission Operations and Science Systems) Tests (First MOSS was in Dec., 2000. Second was in Feb., 2001, i.e., 2nd quarter FY01)

•Interface Documents

–OAs and ICDs are completed, except for those with NASDA.

Milestones – AMSR-E•1st Quarter (December 31, 00)

–1. Installation and Checkout 55:

–a. 5B06 (Dec. 2000)

–2. System Confidence Test

–3. MOSS Test (Dec. 2000)

•2nd Quarter (March 31, 01)

4. IRIX upgraded

5. Second MOSS Test (Feb.)

•3rd Quarter (June 30, 01)

–6. DIFs completed

–7. Add additional Hardware (getting PDS, another rimage system, RAID)

•4th Quarter (September 30, 01)

–8. Guide documents (summaries) completed

–1st Quarter (December 31, 01)

–9. Operations Readiness Review (Oct. 15)

–10. Launch (NET 12/01)

Level 0 ingest begins

Milestones -- GLAS

• 1st Quarter (by December 31)– 1. Write DIFs (drafts)

• 2nd Quarter (by March 31)– 2. Polish/finish (have reviewed) DIFs

• 3rd Quarter (by June 31)– 3. ICD b/w ECS at NSIDC DAAC and ISIPS

– 4. OA b/w NSIDC DAAC and ISIPS

– 5. OA b/w NSIDC DAAC and EDOS

– 6. OA or MOU b/w NSIDC DAAC and SCF ?

– 7. Interface Confidence Tests

Milestones – GLAS (cont.)

• 4th Quarter (by September 30)– 9. Guide doc – L1 Alt. (GLA01, 05 & 06)– 10. Guide doc – L2 Alt. (GLA12-15)– 11. Upgrades to ECS for support of ICESat – 12. Installation and Checkout – 13. Acceptance and Regression Tests – 14. Installation and Checkout – 15. Regression Tests – 16. Interface Tests– 17. Mission End-to-End Tests (IGSTs)

• Launch (Dec. 15, 01)• FY02, 1st Quarter

– Guide doc – L1 &L2 Atmos. (GLA02, 7-11)– Guide doc – GLA16

Subsetting

• ESDIS is working to provide subsetting for data managed in the ECS– MODIS– AMSR-E– GLAS

• For data in HDF-EOS format, the University of Alabama’s HEW subsetter may be used

• GLAS data products will be in binary format, with exceptions (higher level products)– NSIDC is pursuing becoming an rSCF– NSIDC hopes to develop subsetter tool unique to GLAS

Backup Slides

AMSR-E Products

Short Name

Data Level

Collection Description Delivery Frequency

Data Volume uncompressed

PM1GBAD1

Level-0

Ground-Based Attitude Determination Data for EOS Aqua in 1 second intervals

Every two hours

small

PM1GBAD4

Level-0


Every two hours

small

PM1GBAD8

Level-0


Every two hours

small

AE-PMSCI

Level-0

AMSR-E Science and Engineering Data Every two hours

1.3 GB/day

AMSR_L1A

L1A ADEOS-II AMSR L1A Raw Observation Counts

~28 half-orbits / day

1.2 GB/day

AMSREL1A

L1A Aqua AMSR-E L1A Raw Observation Counts


1.2 GB/day

Products

Short Name

Data Level



AE_L2A Level-2A

Aqua AMSR-E global swath Brightness Temperatures are resampled at resolutions of 57 km, 36 km, 21 km, 11 km, and 5.5 km.


2.489 GB/day

AE_Ocean Level-2B

Aqua AMSR-E global swath ocean wind speed at 36 and 21 km res., water vapor over ocean at 21 km res., cloud liquid water at 11 km res., and sea surface temperature at 57 and 36 km res. generated using the Wentz Algorithm and Level-2A product.

~28 half-orbits /day

277.1 MB/day

AE_Land Level-2B

Aqua AMSR-E swath surface soil moisture and interpretive parameters including surface type, vegetation water content, surface temperature, and QC parameters are generated from Level-2A AMSR-E TBs spatially resampled to a nominal 25 km equal area earth grid


11.9 MB/day

AE_Rain Level-2B

Aqua AMSR-E global swath rain rate and rain type products are generated using the Level-2A spatially resampled TBs as input. Over ocean the Goddard Profiling Algorithm produces rain rates and types; over land the Ferraro Algorithm is used.


497.7 MB/day

AE_RnGd Level-3 Aqua AMSR-E global monthly global rainfall accumulations are 5 x 5 degree grids generated using the Wilheit Algorithm and Level-2 rain products as input.

1 / month .005 MB/month

AE_DyOcn Level-3 Aqua AMSR-E global ocean Level-3 daily products are .25 x .25 degree ascending and descending grids generated using the six Level-2B ocean products as input.

1 / day 14.5 MB/day

Products

Short Name

Data Level



AE_WkOcn

Level-3 Aqua AMSR-E global ocean Level-3 weekly products are .25 x .25 degree ascending and descending grids generated using the six Level-2B ocean products as input.

1 / week 12.4 MB/week

AE_MoOcn

Level-3 Aqua AMSR-E global ocean Level-3 monthly products are .25 x .25 degree grids generated using the six Level-2B ocean products as input.

1 / month 12.4 MB/month

AE_DySno

Level-3 Aqua AMSR-E Level-3 daily product contains global snow water equivalent EASE grids. Snow depth EASE grids are included as research products.

1 / day 4.2 MB/day

AE_5DSno

Level-3 Aqua AMSR-E Level-3 product contains 5-day global snow water equivalent EASE grids. Snow depth EASE grids are included as research products

1 / day 4.2 MB/ 5-day

AE_MoSno

Level-3 Aqua AMSR-E Level-3 product contains monthly global snow water equivalent EASE grids. Snow depth EASE girds are included as research products

1 / month 4.2 MB/ month

AE_WkOcn

Level-3 Aqua AMSR-E global ocean Level-3 weekly products are .25 x .25 degree ascending and descending grids generated using the six Level-2B ocean products as input.

1 / week 12.4 MB/week

Products

Short Name

Data Level



AE_SI6 Level-3 Aqua AMSR-E Level-3 products at 6.25 km contain polar stereographic grids of 89.0 GHz brightness temperatures . Grids are daily averages, daily ascending averages, and daily descending averages.

1 / day 46.3 MB/day

AE_SI12 Level-3 Aqua AMSR-E Level-3 products at 12.5 km are polar stereographic grids of sea ice concentration, snow depth over ice, 18 - 89.0 GHz TBs. The sea ice concentration and TBs are daily, daily asc, and daily desc grids. The snow depth is a 5-day grid.

1 / day 64.6 MB/day

AE_SI25 Level-3 Aqua AMSR-E Level-3 products at 25 km contain polar stereographic grids of sea ice concentration, sea ice temperature, 6.925, 10.65, 18.7, 23.8, 36.5, and 89.0 GHz TBs. Grids are daily averages, daily ascending averages, and daily descending averages.

1 / day 22.4 MB/day

AE_Land3 Level-3 Global surface soil moisture with vegetation water content, surface temperature, and TB parameters are generated on a nominal 25-km equal area earth grid by time-compositing the Level-2B parameters, separately for ascending and descending passes.

1 / day 58.3 MB/day

GLAS Data Products and Granularity

Data_type Short Name

ECS File Type Data level

Collection

Description

Granule Size

(MB)

Frequency per Day

Daily Transfer

(MB) GLA01 Science /

Metadata Level 1 A Altimetry Data

File 34 56 1904

GLA02 Science / Metadata

Level 1 A Atmosphere Data File

703 7 4921


Level 1 A Engineering Data File

13 1 13


Level 1 A SRS and GPS Data File

610 4 2440


Level 1 B Waveform-based Range Corrections

File

21 56 1176


Level 1 B Elevation File 9 1 9


Level 1 B Backscatter File 828 7 5796


Level 2 A Boundary Layer and Elevated

Aerosol Layer Heights File

2 1 2


Level 2 A Cloud Height for Multiple Layers

File

81 1 81


Level 2 A Aerosol Vertical Structure File

301 1 301


Level 2 A Thin Cloud/Aerosol

Optical Depth File

12 1 12


Level 2 A Ice Sheet Products File

49 1 49


Level 2 A Sea Ice Products File

22 1 22


Level 2 A Land Products File 273 1 273


Level 2 A Ocean Products File

240 1 240

GLA16

HDF-EOS / Metadata

Level 2 / 3

Composite File

117

1

117

GLA17- GLAxx

TBD Level 3 TBD TBD TBD 5072

GLAS Data Granularity and Potential User Issues

• User A:– GLA12 is 49MB– 49MB/day * 183 days = 8,967 MB– Greenland Ice Sheet occupies 15% of GLA12– 0.15*8967MB = 1345MB

• User B:– GLA16 is 117MB– 117MB/day * 183 days = 21,411 MB– Greenland Ice Sheet occupies 5% of GLA16– 0.05*21411MB = 1071MB

GLAS Data Granularity and Data Delivery Issues

• A CD-ROM holds approx. 650 MB of data, an 8mm Tape holds approx. 5 GB of data.

• Our typical GLA12 user would require 2 x 8mm tapes or 15 CD-ROMs.

• Our GLA16 user would require 5 x 8mm tapes or 36 CD-ROMs.

• If the data were staged to an ftp server, then the user would be required to download large amounts of data, which would require large periods of time.– @56 kb/sec, the GLA12 user would take ~44.5 hrs– @56 kb/sec, the GLA16 user would take ~106.2 hrs

GLAS Data Granularity and Data Delivery Issues

• However, if the data were subsetted for the Greenland Ice Sheet alone, then the GLA12 user would require– 1x8mm tape, 2 CD-ROMs or ~6.6 hrs of @download time– similarly the GLA16 user would require 1x8mm tape, 2

CD-ROMs or ~5.3 hrs of download time.

• This reduction in media use will reduce media distribution costs and can encourage use of ftp to deliver the data to the user. Subsetting becomes essential for efficient data distribution and is being addressed.

Spatial search via NOSE

EDGECS Science Data Server (SDSRV)

Query Parameters AMSR-E

Granule Inventory

Spatial Indicies

Granule List

Spatial Query Server

Implementation of Nominal Orbital Spatial Extent (NOSE) for AMSR-E

ECS Science Data Server (SDSRV)

Spatial Query Server

AMSR-E Granule Inventory

AMSR-E NOSE Table

AMSR-E DLL

AMSR-E pre-defined orbit data

Insertion of AMSR-E granule into ECS

ECS Science Data Server (SDSRV)

AMSR-E granule metadata Spatial

Query Server

AMSR-E Granule Inventory

Spatial Indicies

AMSR-E DLL

Subscriptions

Communications Subsystem (CSS)

NSIDC User Services

Data Server Subsystem (DSS)

2. Event occurs

1. USO enters subscription for user through GUI

Ingest of data

3. DSS tells subscriptionServer about each event

4. Event is recognized as “belonging” to this user

5. Email and/or Granule push to user

QA at NSIDC

• Automatic QA:

– If can’t ingest -> no good

– We plan to exercise media: pull data off randomly to inspect quality of media – still readable (new PDS does this, too)

• Manual QA:– Inspect media when we distribute data on media (not via ftp)

• We will support QA updates if provider wishes to utilize that functionality (hope to make it automated)

AMSR-E L1A Data -- availability

Proposal:• One orbit of data will be available to team members only, three weeks after the AMSR-E

instrument is turned on.

• One (1) day's worth of data will be available to team members only, four weeks after the AMSR-E instrument is turned on.

• All Level 1A data will be continuously available to team members after 2 months.

management of amsr-e and glas data by nsidc melinda marquis amsr/amsr-e data management team lead...

Documents

particular data set

ice data centerpodag

management of amsr

sipsscfscience team

amsre science teamtherefore

distribution of level

scfpostlaunch communications

high level qa of amsr