edg valids activities

1

EDG Valids Activities

Patrick Agbu & Frank Corprew EOS Data Gateway Science and Operations

[email protected]

CEOS Subgroup Meeting – CINTEX Split

September 9-13, 2002

Global Science & Technology, Inc.

2

Introduction

• To enable and maintain interoperability among EOS Data Gateway (EDG) and other catalog systems, e.g., ECS Data Model, ECHO, GCMD, ECS Data Model, ECHO, GCMD, and CINTEX among others.and CINTEX among others.

• To this end, the science and operations team have established a procedure for managing attribute keywords (Valids).


3

Justification

• Valids are the vital components to the operation of the EDG.

• Utilized for the building of search queries.

• Must be resubmitted on a regular basis by a data provider every time their data holdings change.


4

Guidelines

• Proper management of Valid keywords ensures interoperability with other catalog systems by using standard and agreed upon terms.

• Using and sharing keywords from reputable sources, e.g., the GCMD.

• Controlling some keywords and checking for compatibility with established ones.


5

Clients• Local EDG clients are hosted at data centers both

nationally and internationally.

• The Public and Hidden Operational clients are the two clients that require the most frequent update of Valids and Valid-related information.

• A regular and frequent schedule of Valid processing is needed to maintain and synchronize the operational EDG client and the various EDG servers.


6

Operational Requirements

• Components used in the EDG Valids ingest process include: – Submission interface – Various computer systems – Processing utilities – Valids files – Related files and information.


7

Processing Tool• The first component of the EDG Valids ingest process is

the EOS Data Gateway (EOS) Valids Processing (EVP) interface.

• The EVP interface is a web-based ingest/processing and testing tool for EDG Valids, and Valid-related files.

• Valids and Valid-related information can be submitted independent of one another.

• EVP handles updates to dependent Valids, definitions, package information, and extended criteria.


9

EVP Access

• There is only one username and password assigned to each data center/server. If more than one person will be accessing EVP from a particular data center/server, please coordinate among staff.

• The address of the operational EDG Valid Processing (EVP) interface is:

http://pilsner.gsfc.nasa.gov/evp-doc/index.html.


10

Processing Streams

• There are 2 Valids processing streams:

1) Operation stream (performed twice weekly)

2) Test stream (1-2 day turnaround, perhaps longer for a NEW Data Center) or custom processing

• Flow: TEST --> PREOPS --> OPS • Clients: Pre-Ops, Public, Pre-Ops Hidden, Hidden


11

Processing Modes

• Handles all modes of Valid processing and testing, consisting of:

- public modes (ops and pre-ops),

- hidden modes (ops and pre-ops), and

- test modes (ECS-TS1, ECS-TS2, ECS-OPS, VATC-TS1, PVC-TS1, PVC-TS2, and PVC-OPS).

• EVP handles complete replacement of Valids, deletions, and additions.

VATC = Verification Test Center

PVC = Pretest Verification Center


16

Testing and Checking

• EVP can be used for preliminary testing of Valids against your server before a need to promote.

• EVP will check any Valid or Valid related files for syntax, semantics, and content.


17

Schedule

• Processing of Valids is now done on a twice weekly basis, rather than a 2-week cycle.

• Valids received between Thursday and COB Monday will be processed and released by the following Wednesday.

• Valids received between Tuesday and Wednesday COB, will be processed and released by the following Friday.


18

Updates

• EVP can only handle one deletion action, or one addition action per processing cycle.

• Replacement can be performed repeatedly, so long as each successive replacement incorporates previous modifications.

• Updates to a Data Center’s dependent Valids in RCS occur after final ingest processing for a particular cycle. At that time, updated files are then accessible for further submission actions (deletion, addition, and replacement) in EVP.


19

Test client

• The EDG client invoked by EVP is a test instance of the client, visible only to the data submitter, after Valids and Valid related information have passed syntax, semantic, and content checks by EVP.

• Files to be uploaded, either from UNIX platforms, or Pc's should be text formatted files only, in ODL. Microsoft word format, or other non text formats will not work.


20

Promotion

• The use of EVP to upload and test Valids does not make them directly visible in the various EDG clients (hidden, public, test).

• Conversely, if one desires to test only one’s Valids, but do not wish to have them processed operationally by the EDG Science and Operations group at that time, ‘Request for Promotion’ should not be chosen within the EVP interface.


21

Last Chance for Correction

• When the ‘Request for Promotion’ is made, a template with prepopulated information is presented. Additional explanation of any additions, deletions, or changes made to any Valid and Valid-related files since the last submission is appended below the prepopulated information.

• EVP can be used to test Valid submission against either preoperational or operational server. If a request for promotion of Valids for either preoperational or operational mode is submitted for an operational data center, the submission will be treated as an operational submission.


22

File format

• Complete replacement of dependent Valids, in Object Description Language (ODL), are to be submitted in their entirety.

• Dependent Valid additions should be submitted in complete odl groups for each data product.

• Dependent Valid deletions should be submitted as a text file with one dataset on each line, and each dataset name enclosed in parenthesis, and prefixed with

DATASET_ID =


23

File Formats (cont)

• Packaging information, in ODL, is to be submitted in its entirety as a text file, regardless of the type of change made since last submission.

• Definition files, in ODL, are submitted separately as a text file. Only new or modified definitions are submitted for processing.

• Extended Attributes (Extended Valids) information, in ODL, are submitted separately as a text file. Only new or modified extended search attributes are submitted for processing.


24

ODL Requirements

• Required fields and character limits– Dataset_id (85 chars)– Parameter (80 chars)– Sensor (30 chars)– Source (30 chars)– Processing_level (2 chars)– MD_ENTRY_ID (31 chars)


25

ODL Requirements (cont)


• Optional fields and character limits– Campaign (80 chars)– Specialized_criteria_avail (None)

26

Definition Guidelines

• Syntax includes ATTRIBUTE, KEYWORD, and DEFINITION fields.

• Definitions should be clear and concise; maximum length should be from 500 to 1000 characters; most definitions are much shorter.

• Include units in parameter definition when appropriate.

• Acronyms may be used in definitions, but they must be defined on first usage.

• Only new or changed definitions should be submitted.


27

Extended Criteria

• In the EDG, search fields that are common across all (or most) Data Sets are supported as core search attributes. Many data centers may also want to provide searching on additional attributes, which may be common across several datasets, or specific to one.

• The implementation provides for extended search criteria as defined by individual data centers, and allows data centers to coordinate in designing specialized search criteria that apply to some subset of the EDG datasets.


28

Extended Criteria (cont)

• Because the implementation calls for the EDG client to construct a custom search screen for each extended search criterion, we rely on the Valids information provided in the SPECIALIZED_CRITERIA group.

• When a user selects standard search criteria, only the extended search criteria applicable to those datasets making use of such extended criteria will be available.

• If there are conflicts within the user's final search criteria, the client will notify the user of the incompatibilities before the search is sent.


29

Extended Search

• Path/Row is treated as a geographic specification, not an extended search. Search by Granule ID is implemented as a separate search type. Thus, all extended search criteria will be compatible with geographic search criteria.

• Unique search criteria (i.e., not variants) will be the only members of their categories.

• Extended search definitions are submitted by data providers as text file, using ODL objects for each SPECIALIZED_CRITERIA group corresponding to each extended search criteria.


30

Advantages

• Alerts by email indicate that Valids have been submitted for processing in EVP.

• Gives us a record of submission in the event of system or disk failures.

• Indicates the changes from last submission and assists us in verifying that processing is done correctly.

• Gives us a record of input file name along with system generated file name for proper mode of processing.


31

Summary• Established a procedure for managing attribute keywords

or Valids. • Evaluate system Valids for consistency, scientific validity; Evaluate system Valids for consistency, scientific validity;

ensure compatibility, maintain interoperability, and ensure compatibility, maintain interoperability, and populate keyword definitions. populate keyword definitions.

• Valids are submitted via the EDG Valids Processing (EVP) tool.

• Data provider is responsible for ensuring proper ODL syntax, semantics, and content.

• Final promotion to operational status (visible to world) is performed by the Science & Operations staff.


32

Finally• We would like to suggest that INFEO tap into

ECHO


edg valids activities

Documents

valids processing streams

eos data gateway edg

modes of valid processing

dependent valids

ops clients

attribute keywords valids

hidden modes ops

data holdings