Harvesting Metadata for Use by the geodata.gov

Portal

Doug NebertFGDC Secretariat

Geospatial One-Stop Team

2

NSDI: a Foundation for GOSGeospatial One-Stop (GOS) implements the principles of the NSDI spelled out in Executive Order 12906 (recently revised)GOS builds on the resources already available in the NSDI – FGDC and endorsed external Framework Standards, NSDI Clearinghouse NetworkOperated as project coordinated through FGDC Secretariat with many agency partners

3

Geodata.gov and the NSDI Clearinghouse

Over 150 collections of metadata exist within searchable NSDI Clearinghouse Nodes in the US All FGDC metadata records are searchable by fields and full-text through six replicated search gateways geodata.gov will speed and simplify search and browse of these and other metadata through its portal interface

6

GOS Expectations of PortalSearch for geospatial data is 'easy-to-use‘ and would replace the primary search facilities of the NSDI Clearinghouse GatewaysMetadata are harvested from remote collections into a single metadatabaseBrowsing for geospatial data is made possible through common ISO Categories and resource type assignmentsDescriptions of map and data services are also indexed from FGDC metadata entriesData descriptions are linked to an online map viewer where map services exist

7

Portal at geodata.govInitial release of the GOS portal is online with startup content selectively gathered from key providers and entered ‘by hand’New content will be harvested from the Web or individual metadata files may be uploaded as XML or via form inputOGC Web Map Services (WMS) and ArcIMS ‘image’ services supported in viewerData, map services, static maps, and applications are among the resource types indexed

8

9

Current Publication Options Users may contribute metadata one of three ways: Enter FGDC metadata into a form at

geodata.gov and they are stored and indexed there

Upload metadata as FGDC XML to the geodata.gov site

Register their existing metadata collection to be harvested into the GOS metadata repository

10

metadatacache

geodata.gov

map viewer

metadata

metadata

metadatametadatadata

data

data

data

formentry

XMLupload

search

map services

11

Harvesting capabilities Harvesting is a scheduled retrieval of metadata from a remote collection, such as an existing Clearinghouse Node The harvester detects new content and brings it into a metadata ‘cache’ at geodata.gov for inclusion in search Scheduling can be established in the registration profile of each provider

12

13

14

15

Harvesting options A provider wanting to have content harvested into in geodata.gov can register: Z39.50 “GEO” Metadata server (NSDI

Clearinghouse Node) ArcIMS Metadata Server OAI-PMH (Open Archives Initiative,

Protocol for Metadata Harvest) Web Accessible Folder

16

Publishing content to geodata.gov

OAI-PMH Service

ArcIMS Service

Z39.50 Service

Web folder

Metadata

GOSMetadata

cache

XML

GOSSubset

Repository

Metadata

Metadata

Entry Form

XML upload XML

VALIDATE

validation

ok reject/inform

Schedule/registry

Metadataharvester

Searchinterface

no

yes

ok reject/inform

noyes

geodata.gov

17

Portal expectations on metadata

GOS Portal requires tagging of metadata content using the ISO 19115 Category values for browse and search assistance

Collections must also be tagged as to the provider scope (Federal, State, Local, Commercial, Academic, etc)

Online Linkage field is used to store links to OGC Web Mapping Services or (soon) ArcIMS image services or to dataset-specific download/ordering page

18

Suggested ResolutionMetadata collections will be registered with the Portal along with: Related service information pulled from

Clearinghouse Registry, where available Reference to an ISO Category mapping

table to translate Theme Keyword values to Category equivalents – or add ISO Category as a controlled vocabulary in existing metadata

Classification as to Provider type

19

20

Automated TaggingThe following tag values will be derived from the FGDC metadata and stored in the metadata cache at geodata.gov:

Geographic extent (Global, National, Regional, Local) will be determined by the area referenced in the Bounding Coordinates

Resource type will be determined heuristically from the Geospatial Data Presentation Form and clues in the Online_Linkage and Resource_Linkage URLs

Specific guidance will be given to providers

21

To participate in One-Stop:Provide FGDC XML form of CSDGM V2 (1998) or ISO 19139Areas to focus on:

Assure minimum mandatories (pass mp as best you can) Populate online_linkage with valid URL to the data being

described Fix dates (use Metadata_Date to reflect metadata changes) Review content of text fields

Select publishing method (XML upload, form, harvest of Z39.50, ArcIMS, Web, or OAI)If a collection is to be harvested:

Develop a mapping table of Theme Keywords to ISO Category values – or add ISO Categories into metadata

Register collection with GOS Portal

22

Who is being test-harvested?

Z39.50: NOAA Coastal Services Center Alamo Area COG FEMA Multi-hazard Mapping Initiative Alaska Geospatial Data Clearinghouse

ArcIMS: Oregon State University New Jersey Office of GIS

23

How can Z39.50 Nodes participate in geodata.gov?

Update your Z39.50 software to allow: Query on “Metadata Date” as date-last-

modified Retrieval of each metadata record by a

unique and persistent ID

Build a mapping table from local theme keywords to ISO Categories Register your collection with geodata.gov and schedule your harvest

24

Alternatives to Z39.50

ArcIMS Metadata Server can be harvested using ‘native’ AXL requests – just register your Metadata Server If you don’t have a metadata service (ArcIMS+SDE or Z39.50) consider creating a Web-accessible folder that allows directory browsing, then register its existence with geodata.gov

25

What's Next?GOS Team is refining harvesting and publishing services to more automatically process remote metadata

FGDC staff will enhance testing tools and perform audits and metadata counseling to improve metadata content

Guidance will be provided as to how to construct the mappings from local Theme Keywords to ISO Categories

Instructions on setting up registered Web Folders and OAI-PMH providers to follow

26

Interface EnhancementsClarify the options for publishing (XML upload, form entry, collection via Z39.50, OAI, web folders)Clarify the search and display modes of the interfaceDisplay search results rectangle on mapElimination of the Primary/Secondary/Tertiary classification in favor of more legible and derivable resource typesAllow discovery and loading of a single layer (not all layers from a service by default)

27

Research Topics for FY 2004

Following USGS example, evaluate spatial ranking to improve geographic relevance of resultsProvide developers with technical information on how applications can connect to catalog, data access, and mapping services identified through GOSRegister geodata.gov organizations and services to the public UDDI as another method to promote GOS resourcesInvestigate a common thematic thesaurus for GOS/NSDI useImplement transition strategy from FGDC to full ISO 19115 metadata