QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Status of the GHRSST GDAC (Global Data Assembly Center) at the JPL PO.DAAC

Ed Armstrong, Jorge Vazquez and Andrew Binghamhttp://ghrsst.jpl.nasa.gov

28 Nov 2005 US GHRSST meeting, RSMAS

Overview of GDAC Tasks

Ingest project data filesL2P, L4, HR-DDS, MMR and MDB files from RDACs

Stage on public FTP site for 40 daysDeliver data and metadata to LTSRF (NODC) for 30 day old dataMaintain metadata in a relational database (MMR)Provide web-base interface for data discovery and metadata queryFill ancillary fields for all L2P productsSupport GHRSST User Services and ApplicationsMaintain a match-up database of in situ/satellite obs RDAC: MODIS L2P production

. . . .data management and distribution center for all GHRSST products

28 Nov 2005 US GHRSST meeting, RSMAS

Recent Activities

Public GHRSST data release in August 2005Interacted with OBPG and RSMAS to design the MODIS L2P processing architecture New Linux-based web and database severs, and development workstation integrated into PO.DAACGDAC web pages enhancedAccess to MMR improvedInput for GDS rev 1.7 including metadata revisionIDL and C readers for L2P Implemented FGDC metadata and transfer to LTSRFIngested NWP and model data streams for L2P ancillary fieldsDeveloping code and methodology for realtime ancillary filling Implemented user support through a customer management tool (RightNow)First draft of a user requirements document

28 Nov 2005 US GHRSST meeting, RSMAS

GDAC System Architecture

GDAC (JPL U.S.)

RDACs

OCEANIDS(Data Management)

MMR/MDB

L4PC(MIST – U.S.)

Archive(NODC – U.S.)

NRTUsers

Users

Medspiration (Europe)

RSS(U.S.)

BlueLink(Australia)

PO.DAAC(U.S.)

NGSST(Japan)

NAVOCEANO(U.S.)

Data Tools

28 Nov 2005 US GHRSST meeting, RSMAS

GDAC Web Page

28 Nov 2005 US GHRSST meeting, RSMAS

Master Metadata Repository

28 Nov 2005 US GHRSST meeting, RSMAS

Data Access

FTP: podaac.jpl.nasa.gov/pub/GHRSSTData public since late August 2005Organised by:

datatype / sensor / RDAC / year / DOY

Search interface through the MMR

Future delivery mechanismsPOET (L2 subsetting)OpenDAPLive Access ServerSubscriptionData casting

28 Nov 2005 US GHRSST meeting, RSMAS

Data IngestRDAC Datastream StatusEUR L2P TMI

L2P SEVRIL2P NAR 16 & 17 & 18L2P AVHRR LAC (16 & 17)L2P AVHRR GAC (16 & 17)L2P ATSRL2P AMSR-EL4 UHFndHR-DDS

GoodGoodGoodGoodGoodGoodGoodGoodNo data since 25 Mar

REMSS L2P AMSR-EL2P TMI

Good, sporadic in Oct

NAVO L2P AVHRR LAC (18)L2P AVHRR GAC (18)

Expected Dec 2005 ?

RSMAS/OBPG/JPL

L2P MODIS_A In preparation

NOAA NESDIS L2P GOES Unknown

AUS (Bluelink)

L2P AVHRR HRPT Unknown

28 Nov 2005 US GHRSST meeting, RSMAS

Data Latencies

Datastream Avg (hr)

Std Dev (hr)

Max (hr)

EUR L2P TMIEUR L2P SEVRIEUR L2P NAR 16EUR L2P NAR 17EUR L2P AVHRR LAC 16EUR L2P AVHRR LAC 17 EUR L2P AVHRR GAC 16EUR L2P AVHRR GAC 17EUR L2P AATSREUR L2P AMSR-EEUR L4 UHFnd

20.77.27.47.08.68.57.58.19.422.211.8

40.416.815.112.013.312.211.213.814.648.113.4

137639216998108971033883731377108

28 Nov 2005 US GHRSST meeting, RSMAS

Oceanids ingest statistics

1 2 3 4 5 6 7 8 9 10 11

S1

02000

4000

6000

8000

10000

12000

14000

16000

18000

month - 2005

granules

L2P/L4/HRDDS granules ingested77000 total

Granules

Volume (GB)

1 2 3 4 5 6 7 8 9 10 11

S1

0

1

2

3

4

5

6

7

month -2005

volume - GB

Data volume ingested47 GB total

28 Nov 2005 US GHRSST meeting, RSMAS

Ancillary fields JPL GDAC will fill the L2P ancillary fields of wind, solar insolation, sea ice and aerosols

Some RDACs cannot fill these fields in an operational environmentGDAC has accepted this additional responsibility at last GHRSST ST meeting

Ingest numerical weather prediction and model data, i.e., NCEP, NAAPS, FNMOC sea ice and SST

A backup data stream will be implemented for each ancillary field (e.g., ECMWF model output)Ancillary filling methodology and management has been conceptualized.

• Readers for input fields being built

28 Nov 2005 US GHRSST meeting, RSMAS

NOAA LTSRF Interface Status

After 30 days data are placed on a rolling stage for NOAA to pull FGDC metadata created “on the fly” through access to the MMR

Formatted for the NOAA accession record (daily collection of same product files)

Currently NOAA has implemented an automated retrieval of L2P data and FGDC metadataFGDC records for all L2P Medspiration data created through Nov 2005

Other products to follow

28 Nov 2005 US GHRSST meeting, RSMAS

MMR Status

MMR (Master Metadata Repository)Fully transactional MySQL relational databaseSchema finalized and implemented for FR and DSD metadata in Jan 2005

• New database server implemented in Jun 2005

Over 80,000 DSD and FR records ingested since Feb 2005

• No operational failures

Web search interface improved, access is very fast.Automated translation to FGDC metadata through XML style sheet conversion

• Other standards can also be supported

28 Nov 2005 US GHRSST meeting, RSMAS

User services and applications

Updated US GDAC User Requirements document

GHRSST requirementsGDAC system requirementsRDAC requirementsUser requirements

• E.g., OOSes

28 Nov 2005 US GHRSST meeting, RSMAS

MODIS L2P

Joint effort between Univ. of Miami RSMAS, NASA Ocean Biology Processing Group (OBPG) and JPL

This is a new RDAC activityInitially concentrate on Aqua SST

RDAC Division of LaborRSMAS: Algorithm and error statisticsOBPG: 1 km L2 SST production with some ancillary fieldsJPL GDAC: L2P integration with error statistics and ancillary fields

Current StatusOBPG has produced a realtime L2 SST stream for MODIS Aqua (and Terra) RSMAS has developed a MODIS error “hypercube”JPL has tested throughput and acquisition from OPBG server

28 Nov 2005 US GHRSST meeting, RSMAS

Future activities . . .

Ancillary fields fillingAccept delivery of Linux 8 node clusterInstalled in Jan 2006Implement code for realtime filling

• Investigating parallelization of processing

MODIS L2PFinalize transfer between three partiesImplement error statistics and L2PSchedule for Aqua and Terra SST L2P

Interface to new RDACsNAVONOAA NESDISOBPGUpdate ICD

28 Nov 2005 US GHRSST meeting, RSMAS

. . . Future activities

FGDC metadataOperational transfers for all GHRSST products (L2P, L4) to LTSRFs

User ServicesGHRSST product supportApplications development

Data deliveryL2P subsetting in POETOpenDAPImproved search and delivery via MMR

New technology infusionGDAC will be a test bed for the Earth Science Data Casting Project

• a recently funded NASA ROSES activity for XML-based data management and delivery

Implement the Matchup Database (MDB) with similar functionality as MMR