earth science data types (esdts)

Earth Science Data Types(ESDTs)

A.K.Sharma

ESDT Process: Creation through OPS Installation

IT

EDG

ECSSCI

SE

SS EDG

SDS OPS

CMT CCB

ECS

Create Release ScheduleDefine ESDT RequirementsDirect ECS and ECS SCI

Desired Release Date

ES

DT

Prio

ritie

s

Distribute Release Sched.Receive Checklist ReportsSummarize Checklist ReportsReports to CMT and CCBConfirm Valids Submission

ECS Elements Review Reports

ECS Approval

DA

AC

Mgm

t. A

ppro

val DAAC

Mgmt.Review

Schedule ESDT Install to Support Public Release

For TS2, TS1, OPS:Receive ESDTInstall ESDTTestChecklist Report to ECS SCI

ESDT ReviewODL PreparationTestChecklist Report to ECS SCISubmit Valids to EDG

Datasets Available To Public On Release Date

External to DAAC DAAC ECS Review/Approval

Monitor Delivery StatusUpdate DAAC ScheduleProcess Related TTs

Coordination

Deliver ESDT

4 Weeks Prior to Release

Obtain ESDTConfirm Arrival of Correct ESDTCopy ESDT to EDGDDMT Valids to EDGTest & AnalyzeChecklist Report to ECS SCI

Submit Valids2 Weeks Prior to Release

Reports

Rel

ease

Sch

edul

e P

lan

Friday 3am Eastern

ESDTC

heck

list R

epor

ts

ES

DT

& D

DM

T V

alid

s

START

END

To Support Public Data Release

What are Metadata?

– Metadata are descriptive information about the data, employing a common set of terms and definitions

– An Earth Science Data Type (ESDT) is the interface between the science data and the ECS system

How are Metadata Used in ECS?

•Search and access of data

– Inventory searches submitted by users via the ECS Client

– Collection discovery via the Advertising Service

•Other uses

– Subscriptions based on metadata

– Staging of input files based on metadata for Data Processing (e.g., time, location, version)

• Representation of data products from the Earth Scientist’s perspective

– Earth Science Data Types (ESDTs) embody the core science data knowledge for data products contained in the EOS Data and Information System (EOSDIS)

– Mechanism used to inform the Science Data Server about

• Core and product-specific metadata attribute values for each collection and data granule

• EOSDIS Core System (ECS) Data Server services (e.g., Insert, Search, Acquire) which can be invoked for the collection

• One ESDT per collection

• Before a new collection can be added to ECS

- An ESDT must be created and submitted to Science Data Server (SDSRV), a component of the Data Server Subsystem (DSS)

Overview: Role of ESDTs

Overview: Role of ESDTs (Cont.)

– An ESDT, added to the SDSRV, defines to the ECS subsystems

• Collection level metadata attributes and values

• Granule metadata attributes, source of attribute values and attribute valid values

• Product-Specific attributes and their valid values and ranges

• Subscribable events

• Services

Each ESDT consists of two components

•Descriptor file

-ASCII file in Object Description Language (ODL) format describing metadata and services

»ODL is an object-based language with “keyword = value” (keyword/value) statements that allows the class and attributes of an entity to be specified

»ODL is used to convey relations between attributes and their characteristics within the descriptor files

•Dynamic Link Library (DLL)

-Shared object file coded in C++

-Handles the implementation of the specified services on each collection described in the Descriptor file

Overview: Role of ESDTs (Cont.)

Descriptor File

• All collection-level attributes are known ahead of time so their values can be specified in the descriptor file

• The ESDT descriptor has six basic sections:

– Collection level metadata information

– Granule/Inventory level metadata information

– Unparsed/Archive metadata information

– Structure information which holds the CSDT (Computer Science Data Type) mapping - (the actual physical file organization and storage implementation)

– ESDT service descriptions

– ESDT supported event handlers

Product Specific Attributes

• Optional Additional Attributes or Characteristics which the data provider makes available to further describe the Collection or Granule, e.g.,

NUMBADDATA = INTEGER

NUMSPECIALDATA = INTEGER

NUMPROCESSDATA = INTEGER

NUMMISSINGDATA = INTEGER

• note: A listing of the PSAs defined by other Instruments can be found at the following URL: http://ecsinfo.gsfc.nasa.gov/ECS_metadata/PSA/

Group = SERVICES

– The ACQUIRE object specifies the characteristics of the Acquire service

• Acquire service allows users to retrieve data granules

– The INSERT object specifies the characteristics of the Insert service

• The Insert service is used to store data granules in the archive

– The UPDATEMETADATA object allows changing of already inserted values with new values

– The BROWSE object allows access to browse images associated with a particular granule

– The GETQUERYABLEPARAMETERS object allows access to a list of collection and inventory attributes defined within the ESDT descriptor file

– The INSPECT object allows users to look at values from the Inventory metadata attributes

• The INSPECTCL object allows users to look at values from the Collection metadata attributes

•The DELETE object allows the removal of granules from the archive as well as the metadata

•The DELETEFROMARCHIVE object allows the deletion of data granules from the archive but does not delete the associated metadata

•Subsetting services are included as applicable for extracting information by:

-Altitude (only)

-Area (only) by bounding box

-Area (only) by WRS Row, Path

-Parameter (only)

-Time (only)

-Rows (Swath products only)

-multiple means, area by WRS Row, Path

-multiple means, area by Bounding Box

-Apply Mask

-Extract Narrow

Group = SERVICES Cont..

Collection Metadata

DLL Name libDsESDTSyBASIC.001Sh.soSpatial Search Type = Rectangle

Collection Description Class Short Name = AIRAASCI

Long Name = AIRS/Aqua AMSU-A geolocated science countsCollection Description = AIRS/Aqua AMSU-A1 and AMSU-A2 combined geolocated counts for scene

footprintsVersion ID = 1ECS Collection

Revision Date = 2001-09-24Processing Center = GSFC

Archive Center = GSFCVersion Description = Beta Delivery

Single Type Collection Collection State = In Work

Maintenance and Update Frequency = DailySpatial

Spatial Coverage Type = HORIZ&VERTSpatial Domain Container Vertical Spatial Domain

Vertical Spatial Domain Container Vertical Spatial Domain Type = Minimum Altitude

Vertical Spatial Domain Value = SFC

Vertical Spatial Domain Container Vertical Spatial Domain Type = Maximum AltitudeVertical Spatial Domain Value = TOAHorizontal Spatial Domain Container Bounding Rectangle West Bounding Coordinate = -180.000000North Bounding Coordinate = 90.000000East Bounding Coordinate = 180.000000South Bounding Coordinate = -90.000000Coordinate System Container Horizontal Coordinate System Container Geographic Coordinate System Latitude Resolution = 0.250000Longitude Resolution = 0.250000Geographic Coordinate Units = Decimal DegreesTemporal Time Type = UTCDate Type = JulianTemporal Range Type = Continuous RangePrecision of Seconds = 4Endsat Present Flag = Y

Range Date Time Range Beginning Date = 1998-06-08Range Beginning Time = 00:00:00Range Ending Date = 1998-06-08Range Ending Time = 23:59:59Contact Contact Person Contact Person Container Role = InvestigatorContact Instructions = Send email if there is a problem.Contact Job Position = AIRS AnalystContact First Name = DenisContact Middle Name = N/AContact Last Name = ElliottContact Person Address Contact Person Address Container Street Address = 4800 Oak Grove DriveCity = PasadenaState Province = CaliforniaPostal Code = 91109-8099Country = USA

Telephone Telephone Container Telephone Number = 818-354-2117Telephone Number Type = VoiceEmail Electronic Mail Address = [email protected] Organization Contact Organization Container Role = InvestigatorContact Instructions = Send email if there is a problem.Contact Organization Name = JPL AIRS Project OfficeContact Organization Address Contact Organization Address Container Street Address = 4800 Oak Grove DriveCity = PasadenaState Province = CaliforniaPostal Code = 91109-8099Country = USAOrganization Telephone Organization Telephone Container Telephone Number = 1-818-354-2117Telephone Number Type = Voice

Organization E-Mail Electronic Mail Address = [email protected] Topic Parameters

Discipline Topic Term Variable

1 Earth Science Radiance or Imagery Microwave Sensor Counts

DisciplineTopicParameters

Processing Level Processing Level Description = Radiometric CountsProcessing Level ID = 1APlatform Platform Container Platform Short Name = AquaPlatform Long Name = First EOS Polar Orbiting Satellite, 1:30 PM Ascending Equator CrossingPlatform Type = Spacecraft

Platform Characteristic Platform Characteristic Container Platform Characteristic Name = EquatorCrossingTimePlatform Characteristic Description = Local time of the equator crossing and direction (ascending or descending)Platform Characteristic Data Type = varcharPlatform Characteristic Unit = Local Mean TimePlatform Characteristic Value Class Platform Characteristic Value = 1:30 PM, ascendingPlatform Characteristic Container Platform Characteristic Name = OrbitInclinationPlatform Characteristic Description = inclined orbitPlatform Characteristic Data Type = floatPlatform Characteristic Unit = degreesPlatform Characteristic Value Class Platform Characteristic Value = 98.2Platform Characteristic Container Platform Characteristic Name = OrbitalPeriodPlatform Characteristic Description = orbital period for a 16-day, 233-orbit repeat cyclePlatform Characteristic Data Type = floatPlatform Characteristic Unit = minutes

Platform Characteristic Value Class Platform Characteristic Value = 98.88Instrument Instrument Container Instrument Short Name = AMSU-AInstrument Long Name = Advanced Microwave Sounding UnitInstrument Technique = Cross-Track Scanning Multichannel Microwave SoundingNumber of Sensors = 15Operation Mode Class Operation Mode = 1. Normal2. Calibration3. Test Instrument Characteristic Instrument Characteristic Container Instrument Characteristic Name = IFOVInstrument Characteristic Description = instantaneous field of viewInstrument Characteristic Unit = degreeInstrument Characteristic Data Type = floatInstrument Characteristic Value Class Instrument Characteristic Value = 3.3

Sensor Short Name Long Name Technique

1 AMSU-A Channel 1 AMSU-A Chan 1 center freq 23.800 GHz bandpass 0.270 GHz

Broadband Microwave Radiometry







5 AMSU-A Channel 5 AMSU-A Chan 5 center freq 53.5960.115 GHz bandpass

2x0.170 GHz

Split Bandpass Microwave Radiometry




7 AMSU-A Channel 7 AMSU-A Chan 7 center freq 54.940 GHz bandpass 0.400

GHz


8 AMSU-A Channel 8 AMSU-A Chan 8 center freq 55.500 GHz bandpass 0.330

GHz


9 AMSU-A Channel 9 AMSU-A Chan 9 center freq 57.290344 GHz bandpass

0.330 GHz


10 AMSU-A Channel 10 AMSU-A Chan 10 center freq 57.2903440.217 GHz

bandpass 2x0.078 GHz


11 AMSU-A Channel 11 AMSU-A Chan 11 center freq 57.2903440.32220.048 GHz bandpass 4x0.036 GHz


12 AMSU-A Channel 12 AMSU-A Chan 12 center freq 57.2903440.32220.022 GHz bandpass 4x0.016 GHz

Doubly Split Bandpass Microwave

Radiometry


13 AMSU-A Channel 13

AMSU-A Chan 13 center freq

57.2903440.32220.010 GHz bandpass 4x0.008

GHz

Doubly Split Bandpass

Microwave Radiometry


AMSU-A Chan 14 center freq

57.2903440.32220.0045 GHz bandpass

4x0.003 GHz

Doubly Split Bandpass

Microwave Radiometry


AMSU-A Chan 15 center freq 89.000 GHz

bandpass 6.000 GHz


Sensors

Class Name Description Type

1 NumBadData Number of data points in granule that cannot be

processed.

int

2 NumSpecialData Number of data points in granule for which the

instrument is in a special calibration mode.

int

3 NumProcessData Number of data points in granule whichare present

and can be processed.

int

4 NumMissingData Number of data points in granule with missing data.

int

5 NumLandSurface Number of data points within the granule with

land fraction greater than 90%.

int


6 NumOceanSurface Number of data points within the granule with land fraction less

than 10%.

int

7 NumGeoQA Number of data points in granule in which a geolocation error was

detected.

int

8 OrbitPath Orbit number within the repeating set of 233 orbits.

int

9 SP_STARTING_PATH Starting WRS path for this product.

int

10 SP_STARTING_ROW Starting WRS row for this product.

int


11 SP_ENDING_ROW Ending WRS row for this product.

int

12 ScanLineCount Number of scan lines within the granule, corresponding to

GeoTrack.

int

13 NumTotalData Number of data points in granule. int

14 AIRSGranuleNumber Granule number within the set of 240 possible within a day (or 4

possible if RaObs).

int

15 AIRSGranuleCycleNumber A counter which indicates one less than the number of times the granule has been processed with the current PGEVersion at the

ProductGenerationFacility. It is also the last three digits of

LocalGranuleId and is used to assure uniqueness of that field.

int

Class Name Description Type16 NodeType String denoting whether granule is

completely Ascending or Descending, or transitional with value NorthPole or SouthPole.

varchar

17 ProductGenerationFacility A single letter which denotes the location where this granule was processed. This will help mark granules not produced at the

GDAAC. Valids to be the same as for ProcessingCenter.

varchar

18 NumFpe Number of profiles in granule with floating point exception in retrieval.

int

19 NumSunGlint Number of data points in granule with possible sun glint.

int

20 LonGranuleCen Longitude of the center of the granule in degrees.

int

Class Name Description Type21 LatGranuleCen Latitude of the center of the

granule in degrees.int

22 LocTimeGranuleCen Local time at the center of the granule in minutes from

midnight.

int

Additional Product Specific Attributes

Inventory Metadata

ECS Data Granule

Size MB ECS Data Granule = DOUBLELocal Granule ID = STRINGDay Night Flag = STRING

Production Date Time = DATETIMELocal Version ID = STRING

Measured Parameter Measured Parameter Container Parameter Name = STRINGQA Flags Automatic Quality Flag = STRINGAutomatic Quality Flag Explanation = STRINGOperational Quality Flag = STRINGOperational Quality Flag Explanation = STRINGScience Quality Flag = STRINGScience Quality Flag Explanation = STRINGQA Stats QA Percent Missing Data = INTEGEROrbit Calculated Spatial Domain Orbit Calculated Spatial Domain Container Orbit Number = INTEGERStart Orbit Number = INTEGERStop Orbit Number = INTEGEREquator Crossing Longitude = DOUBLEEquator Crossing Time = TIMEEquator Crossing Date = DATE

Collection Description Class Short Name = AIRAASCIVersion ID = 1Input Granule Input Pointer = STRINGSpatial Domain Container Horizontal Spatial Domain Container Zone Identifier Class Zone Identifier = Other Grid SystemBounding Rectangle West Bounding Coordinate = DOUBLENorth Bounding Coordinate = DOUBLEEast Bounding Coordinate = DOUBLESouth Bounding Coordinate = DOUBLERange Date Time Range Beginning Time = TIMERange Ending Time = TIMERange Beginning Date = DATERange Ending Date = DATEPGEVERSIONCLASS PGE Version = STRING

Sensor Platform Name Platform Name Instrument Name Instrument Name Sensor Name

M Aqua STRING AMSU-A STRING STRING

Sensors Associated With This Product

Associated Platform Instrument Sensor

Product Specific Metadata NUMBADDATA = INTEGERNUMSPECIALDATA = INTEGERNUMPROCESSDATA = INTEGERNUMMISSINGDATA = INTEGER

NUMLANDSURFACE = INTEGERNUMOCEANSURFACE = INTEGERNUMGEOQA = INTEGERORBITPATH = INTEGERSp Starting Path = INTEGERSp Starting Row = INTEGERSp Ending Row = INTEGERSCANLINECOUNT = INTEGERNUMTOTALDATA = INTEGERAIRSGRANULENUMBER = INTEGERAIRSGRANULECYCLENUMBER = INTEGERNODETYPE = STRINGPRODUCTGENERATIONFACILITY = STRINGNUMFPE = INTEGERNUMSUNGLINT = INTEGERLONGRANULECEN = INTEGERLATGRANULECEN = INTEGERLOCTIMEGRANULECEN = INTEGER

Metadata File (.met)

• Metadata File (.met)

– Built around the 'parameter = value' form

– Provides the inventory level and unparsed metadata attribute values for each data granule file; this information is used by the SDSRV to populate the inventory metadata database

– Each inventory and unparsed metadata element in the .met is fully described in the MCF

– If the attribute is not defined in the MCF, it can not be set in the .met.

– .met File Naming Convention

• Ought to end with “.met”

• Example: AIRS_L1b20000524190442v1.met

Ingest Granule Metadata Generation

ECS IngestProcessing

DataGranule

ScienceData Server

Inventory Tables

MetadataFile

InventoryMetadata

(ASCII File)

SourceMetadata

ConfigurationFile

Target MetadataConfiguration

File

Processing Granule Metadata Generation

SDP Toolkit Inventory Metadata

Scientific Data Set

Inputs

ProcessControl

File

Product GenerationExecutable

Product

InventoryMetadata

(ASCII File)

Archive Metadata

(Generated by Science Data Server)

(Generated by Processing)

Target Metadata

ConfigurationFile

(HDF-EOS)

Scientific Data Set

ECS Data Distribution

DataGranule

Metadata

(ASCII File with.met extension)

Data ServerSubsystem

MetadataTables

StorageManagement

ToUser

earth science data types (esdts)

Documents

eos data

data processing

science data server

ecs systemhow

core science data knowledge

earth science data typesesdtsa

ecs scisubmit valids

ecs clientcollection