esdis metadata architecture study discussion
DESCRIPTION
ESDIS Metadata Architecture Study Discussion. Presented by Kevin Murphy and the Metadata Architecture Study Team Members 2013 ESIP Summer Meeting | Chapel Hill, NC http://bit.ly/13D7r5A. Outline. Background MAS Phase I MAS Phase II Community Feedback Open Discussion. Background. - PowerPoint PPT PresentationTRANSCRIPT
ESDIS METADATA ARCHITECTURE STUDY DISCUSSIONPresented by Kevin Murphy and the Metadata Architecture Study Team Members2013 ESIP Summer Meeting | Chapel Hill, NC
http://bit.ly/13D7r5A
2
Outline• Background• MAS Phase I• MAS Phase II• Community Feedback• Open Discussion
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
3
BACKGROUND
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
2013 ESIP Summer Meeting | Chapel Hill, NC 4
EOSDIS Mission
The Earth Science Data and Information System (ESDIS) project provides access to data through the development and operation of the science systems of the Earth Observing System (EOS) Data and Information System (EOSDIS).
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 5
Metadata Architecture Study (MAS)ESDIS initiated a study of EOSDIS metadata capabilities/processes in the spring of 2012 to evaluate opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity.
Goals:• Unify several disparate metadata formats (DIF, ECHO, EMS).• Provide consistent results, performance, and access to users
regardless of the mechanism through which they access EOSDIS metadata.
• Improve the consistency and quality of EOSDIS metadata holdings.• Provide a platform capable of expanding EOSDIS metadata
systems to incorporate services, tools, documents, etc.
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 6
Ground Rules Metadata Harmony Study
• EOSDIS metadata are currently a widely used resource which must remain available with at least the same level of service during system definition, development and implementation.
• Enhancements to metadata systems must focus on improving user interactions with the system (End Users, International Users, Data Providers), sustaining engineering of the system, and the integrity of metadata.
July 11, 2013
7
METADATA ARCHITECTURE STUDY (PHASE I)
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
8
MAS I Research Study• Objective:
• Identify opportunities for improved data access and use, reduce efforts by data providers, and improve metadata integrity. Of interest were the overlapping roles and interfaces between the GCMD and ECHO. The five-member study team included representatives from ESDIS, GCMD, ECHO, EMS, and the DAACs.
• Results - Eight recommendations in four categories:
Common Approaches to
Metadata (2)
Improved Communications
(4)
Keywords (1)
Looking Forward (1)
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
9
MAS I Recommendations• Move to a single, shared NASA Earth Science Metadata repository for submission
of collection level metadata.• ESDIS should define a consistent dataset identifier policy to be used across
DAACs and central data systems (ECHO, GCMD, EMS). • Consider a regular communication forum to address and clarify ESDIS
requirements. • Schedule periodic technical telecons between GCMD and the DAACs to provide
insight into new or planned capabilities.• GCMD should implement a mechanism to notify DAACs of DIF and SERF
structure modifications.• Both ECHO and GCMD should develop a DAAC-accessible mechanism for
tracking implementation plans and status for requested changes.• GCMD should roll out the new Keyword Management System and its RESTful
APIs to the DAACs as soon as practical. • ESDIS should continue to monitor emerging metadata needs and related
technologies, and work with the community to define conventions for these new metadata.
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
2013 ESIP Summer Meeting | Chapel Hill, NC 10
METADATA ARCHITECTURE STUDY (PHASE II)
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 11
MAS II Architecture Development• Objectives:
• Develop a Unified Metadata Model (UMM) that unifies the EOSDIS metadata assets into one cohesive model. • Good position for future ISO compatibility.
• Create a Concept of Operations document that provides an overview of how the Unified Metadata Model (UMM) and Common Metadata Repository (CMR) will be used.
Unified Metadata
Model (UMM)
Common Metadata
Repository (CMR)
Metadata Harmony
July 11, 2013
UNIFIED METADATA MODEL (UMM)
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC 12
2013 ESIP Summer Meeting | Chapel Hill, NC 13
Overview• Unification Process
• What was in scope, what was not in scope• Vocabulary
• UMM Core Vocabularies• Extendable Vocabularies
• Knowledge Base System• Documentation
July 11, 2013
14
Unification Process• Initial focus is to unify the following models:
• Current conversions are “lossy”.• By unifying these models ESDIS can latter focus on creating one mapping to
ISO utilizing the upcoming NASA “flavor”.• The ISO model was mapped where possible, and will be in scope for a future
phase.
Met
adat
a
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
15
Unifying the ModelsDIF ECHO
• Data Set Citation• Personnel• Instrument• Platform• Temporal Coverage• Paleo-Temporal Coverage• Spatial Coverage• Location• Data Resolution• Project• Quality• Access Constraints• Use Constraints• Distribution• Data Set Language• Data Set Progress• Related URL• DIF Revision History
• CitationForExternalPublication• ProcessingLevelId• Price• SpatialKeywords• TemporalKeywords• Temporal• Contact• ScienceKeywords• Platform• Instrument• Sensor• Campaigns• TwoDCoordinateSystem• OnlineAccessURL• OnlineResource• AssociatedDIFs• Spatial• ArchiveCenter• AdditionalAttributes• AssociatedBrowseImages• SpatialInfo• CollectionProgress• RestrictionComments
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
EMS Not Shown
16
Unified Fields
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC
2013 ESIP Summer Meeting | Chapel Hill, NC 17
Vocabularies
• UMM Core Vocabularies• Extendable Vocabularies
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 18
Benefits of Core Vocabularies
• Provides a classification system for metadata.
• Prevents zero-hit queries.• Improves metadata accuracy and
completeness.• Improves the usability of earth
science metadata.
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 19
Core Vocabularies - Unique Keyword Identifiers
• Controlled vocabularies are referenced in the UMM using keyword identifiers (UUID’s) to SKOS concepts in the GCMD Keyword Management System Service.
• Updates to keywords referenced in metadata happen automatically since keywords are only “referenced” using these identifiers.
• Additions and/or changes to the vocabularies can be made through the existing governance process.
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 20
UMM Core Vocabulary Types• Science Keywords• Platforms• Instruments• Sensors• Projects• Organizations• Organization Types• Locations• Temporal Resolution Range Keywords• Horizontal Resolution Range Keywords• Vertical Resolution Range Keywords
• Chronostratigraphic Units• Dataset Languages• Related URL Types and Protocols• ISO Topic Categories• DataSet Progress• DataSetLanguage• Temporal Coverage Types• Processing Level Identifiers• Units for Resolutions• Units for Geographic Coordinates
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 21
Extendable VocabulariesAncillaryKeyword Class
Provides a way to reference controlled vocabularies external to the CMR. This UMM class has a resource attribute associated with it, allowing the metadata author to reference an external RDF concept with its own namespace identified through a URI.
Example:<AncillaryKeyword resource="http://mmisw.org/ont/cf/parameter/wind_speed.rdf">Wind Speed</AncillaryKeyword> The example above links to a CF standard_name keyword provided by the Marine Metadata Interoperability Project (MMI).
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 22July 11, 2013
Knowledge Based System(Many Interconnections)
2013 ESIP Summer Meeting | Chapel Hill, NC 23
UMM Documents (Finalized by Early August)
• UMM Overview Document• High level UML diagrams of key parts of the model and descriptions of how
metadata can be represented in the model.
• UMM Crosswalk Study Document• Analysis of each model element, similarities, differences, and provides
recommendations.
• UMM Crosswalk Spreadsheet• Detailed mapping between DIF, ECHO, EMS.• Maps UMM to ISO (where possible).
• UMM Schema created from the model• UMM is not intended to be a format, but can be useful to describe in this
way.
• Online Javadoc style navigation of the UMM Schema
July 11, 2013
COMMON METADATA REPOSITORY (CMR)
July 11, 2013 2013 ESIP Summer Meeting | Chapel Hill, NC 24
252013 ESIP Summer Meeting | Chapel Hill, NC
URSGIBSEMSEOSDIS
Discovery Tools
DAACsNear Real
Time (LANCE)
???CMR
July 11, 2013
262013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
272013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
28
Metadata Concept Support
2013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
292013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
Extensibility
302013 ESIP Summer Meeting | Chapel Hill, NC
ECHO 10
GCMD DIF
EMS ISO19115UMM
Ingest Adapter
VisualizationDocumentation
ServicesParameters
July 11, 2013
1 2
3
31
Improving Metadata Quality with Assessment and Scoring
2013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
322013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
332013 ESIP Summer Meeting | Chapel Hill, NC July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 34
How Will This Affect Me?
July 11, 2013
http
://st
ore.
met
mus
eum
.org
/con
tent
/ebi
z/th
emet
stor
e/in
vt/8
0010
981/
8001
0981
_01_
l.jpg
2013 ESIP Summer Meeting | Chapel Hill, NC 35
Metadata Providers
• You won’t have to reconcile multiple systems• Most systems will continue to work
• ECHO FTP ingest going away• Current formats will continue to function• You will need to be engaged in the adaptation process• You will need to support controlled vocabularies• You will have a voice through the Technical Committee
July 11, 2013
Streamlined Processes, More Engagement
2013 ESIP Summer Meeting | Chapel Hill, NC 36
Client Developers
More Metadata, Better Quality
• Richer, higher quality metadata will be available• Existing APIs will be supported• Standards-compliant APIs will be available• Metadata will be available in an ISO19115 compliant format
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 37
End Users
More Metadata, Better Quality
• Richer, higher quality metadata will be available• Unified view of EOSDIS holdings • Easier access through a variety of clients, data casting, etc• Metadata will be available in an ISO19115 compliant format
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 38
COMMUNITY FEEDBACK
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 39
Early ReviewersNASA Standards
Interest Group (SIG) Review
StakeholderReview/Interviews
Review Groups
July 11, 2013
April 2013 May – June 2013 June – July 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 40
Community Feedback Process• Reviewers submitted feedback and received comments using the EOSDIS
Issue Tracking Software – JIRA• 334 “Issues” created: 206 Resolved/Closed, 68 for Future Study
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 41
Summary of Feedback Received• Provide consistent product identifiers (i.e. UUID’s)• Address ISO and NASA flavor of ISO• Include more required fields• Provide common values for project, platform, instrument• Offer more controlled vocabulary…such as processing level• Contain more structured fields...such as resolution units• Provide types and descriptions for any URL• Include provenance tracking
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 42
Next Steps
July 11, 2013
•Wrap-Up/Document Delivery (Early August)
1
•Requirements, Prototyping, and Design2
•Implementation and Testing
3
•Data Cleanup/Transition
4
2013 ESIP Summer Meeting | Chapel Hill, NC 43
MAS II Team• Lead: Kevin Murphy ([email protected]) • ECHO Representatives
• Katie Baynes ([email protected]) • John Farley ([email protected]) • Dan Pilone ([email protected])
• EMS Representative• Lalit Wanchoo ([email protected])
• GCMD Representatives• Thomas Cherry ([email protected]) • Tom Northcutt ([email protected]) • Scott Ritz ([email protected])
July 11, 2013
2013 ESIP Summer Meeting | Chapel Hill, NC 44
OPEN DISCUSSION
July 11, 2013