cookbook - imos · logo contributors should add a logo to their geonetwork instance. this will be...
TRANSCRIPT
Cookbook
Version 2.0
Table of Contents
Table of ContentsIntroductionOGC Web Map Service
IntroductionLayer Appearance and SymbologyGetFeatureInfo ResponseLinking to Metadata RecordsLayer FilteringMonitoring and NotificationVM Image
Metadata and GeoNetworkIntroductionHarvesting metadata directlyLogoKeywords and Other Metadata
Serving DataTHREDDS Data ServerNetCDFMatlab ToolboxData Storage Options
Appendix A Required MetadataAppendix B Sample FTLAppendix C OGC Web Map ServiceAppendix D WMS to ISO Metadata Translation
Introduction
The AODN cookbook describes how to contribute data to the AODN.
The vision of the AODN was articulated in 2005 by the Australian Ocean Data Centre JointFacility (AODCJF), a joint venture between the six Commonwealth Agencies[1] with primaryresponsibility for marine data. This was “to put in place, by June 2011, an interoperable, onlinenetwork of marine and coastal data resources, including data from the six AODCJF partneragencies, supported by standardsbased metadata, which will serve data to support Australia'sscience, education, environmental management and policy needs: Australia's digital oceancommons.”
This AODN portal (http://portal.aodn.org.au) is the primary access point for search, discovery,access and download of data collected by the Australian marine community. Primary datasetsare contributed by the Integrated Marine Observing System (IMOS) an Australian GovernmentResearch Infrastructure project, and the six Commonwealth Agencies with responsibilities in theAustralian marine jurisdiction (Australian Antarctic Division, Australian Institute for MarineScience, Bureau of Meteorology, Commonwealth Science and Industrial Research Organisation,Geoscience Australia and the Royal Australian Navy). You can obtain full information about theAODN via the AODN Webpage.
Custodians of marine data are encouraged to contribute to the AODN. The AODN portal requiresmetadata so that users can find the data, and web map services so that users can visualise thedata. There are 3 ways in which the AODN can harvest metadata from external parties:
1. Metadata can be harvested from OGC compliant web map services (WMS).2. Metadata can be harvested from OGC compliant web map services (WMS) and
augmented with additional metadata from linked ISO19115 MCP metadata records.3. Metadata can be harvested from ISO19115 MCP metadata records that have links to
WMS layers.
The cookbook describes these options.
If you would like to contribute to the AODN, please contact [email protected] and provide detailsof your WMS server, and, if you have one, your metadata catalogue.
Please refer to the AODN help site for information about using the AODN portal:http://portalhelp.aodn.org.au/Portal2_help/
OGC Web Map Service
IntroductionContributors to the AODN must provide an OGC compliant Web Map Service (WMS). WMS is astandard protocol for serving georeferenced map images over the Internet that are generated bya map server using data from a GIS database. The AODN portal allows users to add map layersthat visually represent available data to the portal map. These map layers are provided by theservers at each contributor’s site. The WMS servers are regularly scanned by the AODNsystem.
Please see Appendix C for a short summary of the WMS API.
There are a number of opensource implementations of OGC WMS. The AODN recommendsGeoserver. http://docs.geoserver.org/stable/en/user/
Some AODN contributors are using MapServer. http://mapserver.org/documentation.html. Werecommend against this however, as the the getFeatureInfo responses are not compatible withthe AODN.
ncWMS is an effective option for gridded data stored as CF compliant netCDF.http://ncwms.sf.ne
Data providers are also welcome to implement their own WMS server. The implementationspecification is available on the OGC website. http://www.opengeospatial.org/standards/wms/.
Layer Appearance and SymbologyMap layers should be more visually descriptive than a collection of dots. See Figure 1 for anexample of a layer showing ocean gliders. Data providers may choose to make use of theAODC symbol library: http://data.aad.gov.au/aadc/symbology/. Please note that this librarycontains symbols used by the Australian Antarctic Division and whilst it is a repository designedto manage symbols used in the AODN there has not yet been any ratification by the communityof AODN users regarding the use of the contained symbols.
Figure 1 Ocean Glider Layer
GetFeatureInfo ResponseThe AODN portal displays information in a popup when a user clicks on a map feature such asa sensor. See Figure 2. The content of the popup is retrieved from the remote WMS via agetFeatureInfo request. The request is designed to retrieve data, including geometry andattribute values, for a pixel location. The operation also allows an html output that is defined onthe server side. The AODN portal displays this html response in the popup window when a userclicks on a feature. For this reason, it is important that each getFeatureInfo response includesdescriptive information that will be clear to a wide range of users.
It is also important that contributing WMS servers deliver html getFeatureInfo responses that fitnaturally into the popup. The window size is 435x325 pixels.
GeoServer allows getFeatureInfo responses to be generated using Freemarker templates.Please refer to the GeoServer tutorial on getFeatureInfo templates for information on how tocreate the html output. An FTL example is provided in Appendix B.http://docs.geoserver.org/stable/en/user/tutorials/GetFeatureInfo/index.html#tutorialsgetfeatureinfo
nb. The MapServer implementation returns text/plain rather than text/html by default. Thetext/plain responses are not currently compatible with the AODN.
Figure 2 Popup displaying GetFeatureInfo response from WMS
Linking to Metadata RecordsIt is recommended that the WMS layers include references to ISO19115 Marine CommunityProfile (MCP) metadata URLs. This allows more complete metadata to be stored than if a WMSserver is provided on its own. The additional metadata is important because it is used by theportal’s faceted layer navigation feature. The WMS getCapabilities response should includesomething similar to the following for each layer:
<MetadataURL type="TC211"><Format>text/xml</Format><OnlineResource xlink:type="simple"xlink:href="http://imosmest.aodn.org.au/geonetwork/srv/en/iso19139.xml?uuid=744ac2a9689c40d3b2620df6863f0327"/></MetadataURL>
n.b. The http link must link to the actual xml document and not to an html document suchas a URL in GeoNetwork that shows the metadata in a web page. In this case the linkmust contain iso19139.xml, not iso19139.html
Layer FilteringAODN contributors can log in to the portal’s administration area and add filters. Filters provide away for users to select features within a layer. It might be useful, for example, to filter ship tracksby date rather than needing to click on each track to find out when the voyage occurred.
To add a filter:
1. Click on the administration link at the top right of the portal. You will not see the link if youare not a server owner or an administrator. Please contact [email protected] for anaccount.
2. Select the “Layer” link from the menu at the top left.3. Click on the layer to which you would like to add the filter.4. Click the “Create Filter” link.5. Fill out the form and click Create.
You can add as many filters as you like. Once created the filtering options will appear in theportal’s right hand panel when a user selects a filtered layer.
Monitoring and NotificationThe AODN will be adding a feature to notify data owners when the WMS scanner detectsoutages at the provider’s WMS. At this stage, however, data providers will need to monitor theirown WMS instances, and/or keep an eye on the AODN portal to ensure that layers remainavailable.
VM ImageThe AODN will provide a virtual machine image containing a preconfigured environment forcontributing data to the AODN. This will include GeoServer, Geonetwork, Postgres andTHREDDS. The VM image will include sample NetCDF files, WMS layers, and MCP metadata.The image will also include an instance of the AODN portal running under Tomcat for testing orinternal use.
The National eResearch Collaboration Tools and Resources project (NeCTAR) provides freecloud computing for research purposes in Australia. The VM will be available in the NeCTARresearch cloud to help get providers up and running quickly.
http://www.nectar.org.au/researchcloud/
Metadata and GeoNetwork
IntroductionIn most cases, each WMS layer will have a corresponding ISO19115MCP compliant metadatarecord. As described in the previous section, the location of the metadata record is indicated inthe WMS getCapabilities document. The metadata records are used by the portal for searching,and to provide links to abstracts and online resources. The AODN harvests WMS layers andtransforms them into MCP records (See appendix D). If, as is recommended, there are links toexisting MCP records, this metadata will also be retrieved and merged with the metadataretrieved from the WMS getCapabilities document.
The linked metadata must conform to the marine community profile (MCP) standard. Forinformation on the MCP, please refer to the documentation at:http://www.aodn.org.au/redmine/projects/mcp/wiki/Marine_Community_Profile_Version_14There is also a wiki dedicated to maintaining, governing and updating this profile at:http://www.aodn.org.au/redmine/projects/mcp.
The MCP records are important because the portal relies on these for searching. For thisreason the records must include consistent metadata for searching (e.g. keywords) and forreference (e.g. point of contact). See Keywords and other Metadata below. The mapping ofGetCapabilities metadata to iso19115 metadata is described athttp://trac.osgeo.org/geonetwork/wiki/ISO19119impl. It is expected that Marine CommunityProfile specific mappings will be made available as required.
It is suggested that AODN data providers publish their metadata using GeoNetwork, although it ispossible to provide metadata by any means that suits. GeoNetwork is a catalog application thatmanages spatially referenced resources. The GeoNetwork user guide is available here:http://geonetworkopensource.org/docs.html. The central AODN metadata repository is itself aninstance of GeoNetwork, which is why the most straightforward approach is for data providers toto publish their metadata using Geonetwork. The AODN catalog can, however, harvestmetadata from any catalog that supports OGCCSW or OAIPMH.
Harvesting metadata directlySo far we have discussed how the AODN can harvest WMS layers and augment with linkedMCP metadata. It is also possible for the AODN to harvest the metadata directly. In this casethe harvested metadata must contain links to the corresponding WMS layers. The metadatarecords should include a section similar to the following XML fragment.
<gmd:onLine>
<gmd:CI_OnlineResource> <gmd:linkage> <gmd:URL>http://geoserver.imos.org.au/geoserver/wms</gmd:URL> </gmd:linkage> <gmd:protocol> <gco:CharacterString>OGC:WMS1.1.1httpgetmap</gco:CharacterString> </gmd:protocol> <gmd:name xmlns:gmx="http://www.isotc211.org/2005/gmx" xmlns:xlink="http://www.w3.org/1999/xlink"> <gmx:MimeFileType type="text/html">imos:soop_sst_mv</gmx:MimeFileType> </gmd:name> <gmd:description> <gco:CharacterString>SOOP SST (all tracks)</gco:CharacterString> </gmd:description>
</gmd:CI_OnlineResource></gmd:onLine>
nb. In this example the WMS server ishttp://geoserver.imos.or.au/geoserver/wms and the name of the layer isimos:soop_sst_mv.
LogoContributors should add a logo to their GeoNetwork instance. This will be displayed in the AODNportal search results. The logo can be added under logo management in the GeoNetworkadministration section.
Where metadata is harvested from the WMS only, or from a nonGeoNetwork source, a logo willneed to be supplied to the AODN so that it can be manually attached.
Keywords and Other MetadataSearching within the AODN portal is metadata based. The usefulness of the search for findingdatasets will be related directly to the relevance and consistency of the metadata supplieddescribing those datasets and of the search terms included in the metadata in particular. Thevocabularies used within the MCP metadata are currently under review and it is anticipated thatthe Profile will be soon be updated. The MCP wiki(http://www.aodn.org.au/redmine/projects/mcp) is the forum for discussion papers on suggestedchanges.
The AODN portal currently allows datasets and map layers to be found using free text searching,‘keyword’ searching and searches by ‘organisation’ and by ‘parameter’. The free text search is
performed on all text fields in the iso19115.mcp metadata record describing the dataset and/ormap layer.
‘Keyword’ searching is enabled only on metadata keywords selected from the MarineCommunity Profile thesauri to provide a consistent set of search terms. The Marine CommunityProfile includes (keyword) thesauri for ‘theme’, ‘geographic location’ and ‘collection method’search terms reflecting the common requirements for finding datasets. Specifically, GCMD5.3.8 is the currently supported thesauri for searching by theme (however, this requirement touse GCMD vocabulary scheme 5.3.8. is currently not specified in the Marine Community Profiledocumentation).
‘Organisation’ searching is performed on organisations specified in the metadata as points ofcontact for the provided resource. ‘Parameter’ searching is performed on the long parametername specified for ‘associated data parameters’ in the Marine Community Profile ‘dataparameters’ section of the metadata. There is currently no recommended thesuari supportingthe selection of ‘parameter’ names, nor are ‘parameters’ listed in the MCP 1.4 documentation,however an XML schema example is providedat:http://schemas.aodn.org.au/mcp1.4/schema/extensions/mcpExtensions.xsd.
To ensure advanced searching, facilities in the AODN portal can be used to find the Contributor’sdatasets and map layers, contributors should ensure that their metadata includes relevantsearch terms, and that the search terms are selected from thesauri specified in the MarineCommunity Profile.
In addition to providing a basis for searching for datasets and associated map layers, themetadata provided should allow the fitness for purpose of the dataset to be determined, and itshould provide relevant contact details.
See Appendix A for required fields
Serving Data
This section contains information about how data can be stored and served. The AODN has nospecific requirements for data storage, but the following information may be useful to somecontributors.
Data FilteringIf data is stored in a database and served by Geoserver/WFS, it can be filtered and downloadedthrough the portal. The same filtering mechanisms that are used for layer filtering can be appliedto the WFS.
THREDDS Data ServerMany AODN data providers serve their data through THREDDS. THREDDS Data Server (TDS)serves the contents of datasets and provides catalogs and metadata for them. The TDS usesthe Common Data Model to read datasets in various formats, and serves them throughOPeNDAP, OGC Web Coverage Service, NetCDF subset, and bulk HTTP file transfer services.The first three allow the user to obtain subsets of the data, which is crucial for large datasets.The TDS has the ability to aggregate many files into virtual datasets, which insulates users fromthe details of file storage and naming, and greatly simplifies user access to large collections offiles.
Please see the following thredds server for an example of a typical folder structure:http://opendapvpac.arcs.org.au/thredds/catalog.html
NetCDFMany AODN contributors store their data using the netCDF (network Common Data Form)system, a set of software libraries and machineindependent data formats that support thecreation, access, and sharing of arrayoriented scientific data.
The following documents describe the IMOS NetCDF format and the the format of filenames thatwill be used to distribute IMOS data in netCDF format. AODN contributors are not required tofollow these standards but may find these useful.
File naming conventions:http://imos.org.au/fileadmin/user_upload/shared/IMOS%20General/documents/Facility_manuals/IMOS_netCDF_filenaming_convention_v1.4.pdf
IMOS NetCDF User Manual:
http://imos.org.au/fileadmin/user_upload/shared/IMOS%20General/documents/Facility_manuals/IMOS_netCDF_usermanual_v1.3.pdf
Matlab ToolboxThe IMOS Matlab Toolbox aims to provide an automated, easy to use interface for convertingraw instrument data into IMOS compatible files, ready for handover to the IMOS way todayeMarine Information Infrastructure facility (eMII). The first stage of this project is underway, and isspecifically targeting 'delayed mode' data; in other words, data which is manually retrieved fromlongterm mooring sites. Real time data will be addressed in a later stage of the project.
The Matlab Toolbox may be useful to AODN providers and can be found at this link:
http://code.google.com/p/imostoolbox/
Data Storage OptionsContributors who require data storage are welcome to contact RDSI. http://rdsi.uq.edu.au/. RDSIprovides free data storage for research data collections of national significance.
Appendix A Required Metadata
Required information for AODN MEST
Title M A clear and concise description of thecontent of the resource.
Date & Type M Date is commonly when themetadata record was created. Typeis selected from a controlled codelist,and is the method used to indentifythe event associated with the date,e.g. creation publication and revision.
Cited Responsible Party/ies for the dataset O Are the IP owners, usually theprimary researcher or organisationinvolved in the research. The addressdetails refer to where the researchwas conducted and the position heldat that time. Role for each personselected from a controlled codelist.
Abstract M Describes the content of theresource. For example, whatinformation was collected (e.g.variables measured), how it wascollected (equipment used andmethodology), purpose of the study,and a brief statement about thelocation and/or temporal extent.
Credits O Recognition of the contributors to theresource, for example financialsupport.
Status of data O Ongoing, complete or historical.
Point of Contact Details O Full CURRENT contact details forperson/s and/or organisation(s)associated with the resource,including their role.
Resource maintenance (update frequency ifplanned)
O Choices: continual, daily, weekly,fortnightly, monthly, quarterly,biannually, annually, asNeeded,irregular, notPlanned, unknown.
Descriptive keywords O IMOS utilise: Global Change MasterDirectory (GCMD), MarineCommunity Profile (MCP) collectionmethods, MCP Geographic Extentand for all other keywords, thesehave been included in our AODNthesaurus.
Resource Constraints O Provides information about anyconditions and contraints which applyto the resource access and use.Information is provided here aboutany data licences, e.g. CreativeCommons.
Aggregation information O If applicable, can aggregate togetherassociated metadata records.Requires a title for the aggregateddatasets, and aggregating to whichrecords?
Topic Category C Values are selected from a controlledcodelist. The default value is oceans.
Geographic extent of data C Geographic description and/or pointcoordinates or coordinates togenerate a bounding box or polygon.
Temporal Extent C Start and end date/time
Vertical extent of data C Minimum and maximum value inmetres.
Supplemental information O Any additional information that willassist the users understanding of thedata. For example, publicationsgenerated from the resource.
Sampling frequency O Choices: continual, hourly, daily,
weekly, fortnightly, monthly, quarterly,biannually, annually, asNeeded,irregular, notPlanned, unknown.
Data parameter and units of measure O IMOS are utilising the CF conventionwhere possible. Name of dataparameter is recognised astype=longName, and the format that itappears in the dataset istype=shortName.
Distribution format (name and version) M e.g. Microsoft Excel, 2003
Distributor Contact O The contact associated with thedistribution of the resource
Online resource O Where files (data, documentation,reports etc) are uploaded (preferredprotocols: data file for download orother file for download) or linked(preferred protocols: web address(URL) or data for download (URL) tothe metadata record. WMS pathdetails.
Data Quality information O Used to describe a number ofaspects about the resource such assampling equipment (collectionhardware), methodology, precisionand resolution. For experimentaldata, it is recommended that youhave at least two occurences of thiselement: the first to describe thecollection hardware and the secondto the describe the dataset moregenerally. Information is enteredunder lineage statement.
Metadata contact M The contact associated with thecreation of the metadata record,commonly the organisations datamanager, and can include the currentpoint of contact (with abbreviated
contact details: name, organisation,position, email and role).
Note: M = mandatory, C = conditional, O = optional. C are optional fields that become M undercertain circumstances.
Topic category is M if hiearchy level="dataset". One of geographic, temporal or vertical is M ifhierarchy level="dataset".
Appendix B Sample FTL
The FTL code sample in this appendix is used to generate the ACORN coastal radargetFeatureInfoResponse shown below.
<#import "config.ftl" as my>
<h3>ACORN Australian Coastal Ocean Radar Networks</h3>
<#list features as feature><div class="feature">
<p>$feature.station_name.value is part of the $feature.site_name.value group.</p>
<ul><li><b>Frequency</b> $feature.frequency.value</li><li><b>Bandwidth</b> $feature.bandwidth.value</li><li><b>Max Range</b> $feature.max_range.value</li></ul>
<#if feature.radar_system.value?trim == "Phased Array System"><#if feature.opendap.value?trim != ""> Radial Data: <a rel="external"href="http://opendapqcif.arcs.org.au/thredds/catalog/IMOS/ACORN/radial_quality_controlled/$feature.platform_code.value/catalog.html" target="_blank" class="h3" >QC</a><a rel="external"
href="http://opendapqcif.arcs.org.au/thredds/catalog/IMOS/ACORN/radial/$feature.platform_code.value/catalog.html" target="_blank" class="h3" >nonQC</a><br>1 Hour Average Gridded Data: <a rel="external"href="http://opendapqcif.arcs.org.au/thredds/catalog/IMOS/ACORN/gridded_1havgcurrentmap_QC/$feature.site_code.value/catalog.html" target="_blank" class="h3" >QC</a><a rel="external"href="http://opendapqcif.arcs.org.au/thredds/catalog/IMOS/ACORN/gridded_1havgcurrentmap_nonQC/$feature.site_code.value/catalog.html" target="_blank" class="h3">nonQC</a></#if></#if>
<#if feature.radar_system.value?trim == "Direction Finding System"> Radial Data: <a rel="external"href="http://opendapqcif.arcs.org.au/thredds/catalog/IMOS/ACORN/radial/$feature.platform_code.value/catalog.html" target="_blank" class="h3" >nonQC</a><br>1 Hour Average Gridded Data: <a rel="external"href="http://opendapqcif.arcs.org.au/thredds/catalog/IMOS/ACORN/seastate/$feature.site_code.value/catalog.html" target="_blank" class="h3" >nonQC</a></#if><BR><#if feature.station_uuid.value?trim != ""><ahref="http://imosmest.emii.org.au/geonetwork/srv/en/metadata.show?uuid=$feature.station_uuid.value" target="_blank" rel="external" class="h3"title="http://imosmest.emii.org.au/geonetwork/srv/en/metadata.show?uuid=$feature.station_uuid.value">Station Information</a><BR></#if><#if feature.site_uuid.value?trim != ""><ahref="http://imosmest.emii.org.au/geonetwork/srv/en/metadata.show?uuid=$feature.site_uuid.value" target="_blank" rel="external" class="h3"title="http://imosmest.emii.org.au/geonetwork/srv/en/metadata.show?uuid=$feature.site_uuid.value">Site Information</a><BR></#if>
</div></#list>
Appendix C OGC Web Map Service
The full WMS specification can be found here: http://www.opengeospatial.org/standards/wms/
The AODN makes requests to WMS servers provided by data providers. The requests includegetCapabilities, getMap, and getFeatureInfo.
getCapabilitiesThe getCapabilities request retrieves metadata about the service, including supported operationsand parameters, and a list of the available layers. Once harvested, the layers (which representdatasets) described in the getCapabilities response can be accessed through the AODN.Here is an example of a request for the capabilities provided by the IMOS Geoserver instance:
http://geoserver.imos.org.au/geoserver/wms?VERSION=1.1.1&REQUEST=GetCapabilities&SERVICE=WMS
getMapThe getMap request retrieves a map image for a specified area and content. WMS maps areviewable through the AODN portal. Here is an example of a request for a map of the IMOS argolayer:
http://geoserver.imos.org.au/geoserver/imos/wms?service=WMS&version=1.1.0&request=GetMap&layers=imos:argo_float&styles=&bbox=179.85,68.326,179.998,19.754&width=1348&height=330&srs=EPSG:4326&format=image/png
getFeatureInfoThis retrieves the underlying data, including geometry and attribute values, for a pixel location ona map. Here is an example of a request for feature information about a particular argo floatrepresented on the IMOS argo map requested above:
http://geoserver.imos.org.au/geoserver/imos/wms?REQUEST=GetFeatureInfo&EXCEPTIONS=application%2Fvnd.ogc.se_xml&BBOX=236.779078%2C82.26932%2C236.927078%2C33.69732&SERVICE=WMS&VERSION=1.1.1&X=467&Y=270&INFO_FORMAT=text%2Fhtml&QUERY_LAYERS=imos%3Aargo_float&FEATURE_COUNT=50&Layers=imos%3Aargo_float&WIDTH=1348&HEIGHT=330&format=image%2Fpng&styles=&srs=EPSG%3A4326
Appendix D WMS to ISO Metadata Translation
The AODN metadata catalogue is an instance of GeoNetwork. GeoNetwork is able to harvestmetadata from WMS servers. Details of the default mapping from WMS to ISO19115 can befound here: http://trac.osgeo.org/geonetwork/wiki/ISO19119impl.
It is possible to define iso19115.mcp specific mappings (an xsl to use in place of the defaults)and to add this to available mappings in the harvester by including the mapping in the mcp profileplugin. It is also possible to do this on a site by site base (i.e. custom mcp mappings) byadjusting the supplied stylesheets. This is how the AODN can 'inherit' metadata (in our casekeywords) from a linked metadata record.