clearinghouse wg telecon october 7, 2003 (updated after call)
TRANSCRIPT
Clearinghouse WG Telecon
October 7, 2003(updated after call)
2
AgendaIntroductionsStatus Gateways, Registration, Service
testing Software (Isite, others)
Geospatial One-Stop Portal Overview Integration with Clearinghouse
3
Attendees
USGSDenver BLMMinnesotaWisconsinOak Ridge NLATS IncDept Homeland SecurityUSDA FSA
IntergraphCornell UnivWyomingWashingtonGt. Lakes CommissionArmy CorpsNew MexicoSARIS
James Madison Univ (VA)NASA GCMDFGDC StaffIllinoisNRCSGeoConnectionsESRI
4
GatewaysOperating Blue Angel Gateway 3.6.2 at all six locationsUpdated service templates and a javascript OGC Web Map Viewer (multiviewer) included in the packageGateways should be at same revisionNo migration to MetaStar 6 planned
5
Gateway Actions FGDC and EDC have updated their Gateway instances as per recent guidance Be sure Gateway admins are getting the notices Find status of NOAA-CSC, ESRI, and Alaska GDC Gateways
6
Status checks on server quality
Existing Clearinghouse Status page is being renovated to provide more information for providers and administrators New page will be set up on the new Linux host at EDC (server consolidation) May add more checks to help in assessing quality for One-Stop use
7
8
1. Quick Status Line• X means successful• 0 means 0 records returned• Timed Out means after 60 seconds the connection was broken• Error means an error occurred2. Test that returned error3. Text of error message (if any)4. Information currently stored in the database for testing5. Contact information for node administrator
10
Summary of Recent ActivitiesNumerous bug fixes and several new features, including new Z39.50 configuration optionsRelease available this week at ftp.awcubed.com, FGDC to follow Binaries for Linux, Solaris, Windows
(Cygwin)
Updated documentation (now and forever)
11
Isearch2 Release 2.0.5Spatial Ranking now implemented and undergoing testing CGI and Z39.50 demos will be available
shortly atwww.awcubed.com/FGDCsearch.html
Official release this week Binaries for Linux, Solaris, Windows/Cygwin
Isearch-cgi now includes full support for search engine capabilities numeric, temporal & spatial searching,
remote and virtual collections
12
Coming NextNew GUI administration program for configuring and running Isearch2 and Isite2 Both Windows and Unix Based on FLTK GUI toolkit
Investigation into how to improve harvesting capabilities – OAI, RSSCGI interface to support CQL, SRW
13
Isite Admin Console Redesign
Similar to Internet Information Server (IIS) control interfaceNodes established through wizard interfaceNodes most visible component that actions are performed againstActions or Right-Click to perform: Re-index Node Properties Server tests (full text, field and spatial)
14
Mock-up
15
Other Software Status Intergraph SMMS/GeoConnect Capability to mass export XML files GeoConnect works with Oracle, SQL-
Server Scripts to store metadata will be
posted to website
ESRI ArcIMS Metadata Server Will be modified to support harvesting
by the OAI-PMH (Library) protocol
16
ISO 19115 and 19139 Status
19115 is now a published standard 19139 is a draft ISO Technical Specification to publish an XML Schema for 19115 – soon to be Committee Draft Will allow a conversion engine to convert FGDC metadata to ISO form with some configuration support to fill-in missing source or destination info
17
Geospatial One-Stop (GOS)Module 1: Develop logical data models and encoding for Framework data exchangeModule 2: Prepare and publish metadata on existing (primarily) Framework data holdingsModule 3: Prepare and publish metadata for planned data acquisitionModule 4: Set up authoritative and reliable Web services for maps and dataModule 5: Construct an easier-to-use search and access Portal for geospatial data and services
18
NSDI: a Foundation for GOSGeospatial One-Stop (GOS) implements the principles of the NSDI spelled out in Executive Order 12906 (recently revised)GOS builds on the resources already available in the NSDI – FGDC and endorsed external Framework Standards, NSDI Clearinghouse NetworkOperated as project coordinated through FGDC Secretariat with many agency partnersNSDI and GOS activities provide no significant new funding
19
What's different about One-Stop?
GOS has a timeline for implementation of the various modulesStandards are being developed with multi-sectoral stakeholders as national (ANSI) standards, not FGDC onesVisibility of the initiative as one of 24 e-gov initiatives from the AdministrationGoals include measures of costs and savings through cost-sharing in data acquisition, processing, and service of geospatial data
20
What else is different?More explicit focus on including State and Local government as primary stakeholdersDecisions in GOS are made by a governmental (non-FGDC) Board of Directors who advise the Executive DirectorGOS as an initiative is intended to sunset when all Modules are operational (2005+)Management partners from DOI and OMB are watching performance and capabilities (and budget effects) very closely
21
GOS Expectations of PortalSearch for geospatial data is 'easy-to-use'Metadata are harvested from remote collections into a single metadatabase for search efficiencyBrowsing for geospatial data is made possible through common ISO Categories and resource type assignmentsDescriptions of map and data services are also indexedData descriptions are linked to an online map viewer where map services exist
22
Portal at geodata.govFirst release of a GOS portal is online with a few metadata collections plus individual files uploaded as XML or via form inputDeveloped and temporarily hosted by ESRI through a DOI contract and managed through BLM, DenverOGC Web Map Services (WMS) and ArcIMS supported in viewerData, map services, static maps, and applications are among the resource types indexed
23
24
Current Publication ProcessUser enters FGDC metadata into a form at geodata.gov and they are stored and indexed thereUser uploads FGDC metadata as XML into a the metadata store at geodata.govAdmin staff negotiates with existing Z39.50 or ArcIMS metadata collections to harvest metadataNo current facility to nominate the existence of a metadata collection (Z39.50) to harvest
25
Current ProcessBLM validates metadata, inserts special GOS-specific tags for: Provider type Resource type Data extent
Validation and acceptance process is quite tediousMetadata that are uploaded by XML or form are mixed in with harvested metadata
26
Observations
FGDC Metadata reviewed by GOS Team have many problems: Metadata is not being parsed by 'mp'
and fixed Online_linkage not holding a valid URL Dates violate the strict YYYYMMDD
format Some mandatory fields are populated
with instructional content (e.g. “This field is required”)
27
Additional info needed by PortalGOS Portal requires tagging of metadata
content using the ISO 19115 Category values for browse and search assistance
Records must also be tagged as to the provider type (Federal, State, Local, Commercial, Academic, etc)
Also needed: Information on the geographic scope (Global,
National, Regional, Local) Resource type (map graphic, map service,
data, ref) – need feedback on an approach
28
Suggested ResolutionMetadata collections will be registered with the Portal along with: Related service information pulled from
Clearinghouse Registry, where available Reference to an ISO Category mapping
table to translate Theme Keyword values to Category equivalents – or add ISO Category as a controlled vocabulary in existing metadata
Classification as to Provider type
29
Automated TaggingThe following tag values will be derived from the FGDC metadata and stored in the metadata at geodata.gov:
Geographic extent (Global, National, Regional, Local) will be determined by the area referenced in the Bounding Coordinates
Resource type will be determined heuristically from the Geospatial Data Presentation Form and clues in the Online_Linkage and Resource_Linkage URLs
Specific guidance will be given to providers
30
Getting Content into GOSMetadata may currently be harvested from existing Z39.50 and ArcIMS (Metadata) servers by the GOS TeamFuture options for publishing to GOS include:
Z39.50 GEO Profile on ISO or FGDC metadata ArcIMS Metadata Server Open Archives Initiative Protocol for
Metadata Harvesting (OAI-PMH) Web-accessible folder of FGDC XML
metadata Form entry of metadata on geodata.gov site XML upload of FGDC or ISO metadata
31
Anticipated Publishing Threads for GOS
OAI-PMH Service
ArcIMS Service
Z39.50 Service
Web folder
Metadata
GOSMetadatabase
XML
GOSSubset
Repository
Metadata
Metadata
Entry Form
XML upload XML
VALIDATE
validation
ok reject/inform
Schedule/registry
Metadataharvester
Searchinterface
no
yes
ok reject/inform
noyes
32
To participate in One-Stop:Provide XML form of CSDGM V2 (1998)Metadata must pass mp without errorsAreas to focus on:
Assure minimum mandatories Populate online_linkage with valid URL to the data being
described Fix dates Review content of text fields
Select publishing method (XML upload, form, harvest of Z39.50, ArcIMS, Web, or OAI)If a collection is to be harvested:
Develop a mapping table of Theme Keywords to ISO Category values – or add ISO Categories into metadata
Register collection with GOS Portal
33
IssuesPolicy decision is needed for harvesting big collections. There are about 25 image collections with between 10,000 and million+ records each. Choices: Require export of XML representations of
metadata Publish only a collection-level metadata record
with links for further search Provide a Z39.50 cascaded search of the deep
servers, as in Geography Network Set up an OAI-PMH Provider on each collection Not include deep collections in geodata.gov
34
Discussion: Big collectionsOne-Stop metaphor shouldn’t require one to search again on image collections – incorporate them somehowImage collections should be includedCollection-level metadata record could be provided instead of full inventoryPassthrough distributed search might work well enough on a few collectionsNeed to ask the Big providers their preferences
35
What's Next?GOS Team will refine harvesting and publishing services to more automatically process remote metadata
FGDC staff will enhance testing tools and perform audits and metadata counseling to improve metadata content
Guidance will be provided as to how to construct the mappings from local Theme Keywords to ISO Categories
Instructions on setting up registered Web Folders and OAI-PMH providers to follow
36
Interface Enhancements
Clarify the options for publishing (XML upload, form entry, collection via Z39.50, OAI, web folders)Clarify the search and display modes of the interface
Display search results rectangle on map
Elimination of the Primary/Secondary/Tertiary classification in favor of more legible and derivable resource typesAllow discovery and loading of a single layer (not all layers from a service)
37
Research Topics (6-12 months)
Following USGS example, evaluate spatial ranking to improve geographic relevance of resultsProvide developers with technical information on how applications can connect to catalog, data access, and mapping services identified through GOSRegister organizations and services to the public UDDI as another method to promote GOS resourcesInvestigate a common thematic thesaurus for GOS/NSDI useImplement transition strategy from FGDC to full ISO 19115 metadata
38
Discussion What are the timelines for testing and deploying geodata.gov enhancements? Unknown, pending assessment of
workflow and assignments
What are the priorities? ArcIMS harvesting underway Z39.50 and Web Folder next OAI provider connections
39
Who are good candidates?The following sites could be good
Z39.50 harvest candidates for GOS: Cornell University (CUGIR) has metadata in good shapeUSGS Geosciences collectionSome EDC International collections