ipy data management a brief summary Øystein godøy
TRANSCRIPT
![Page 1: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/1.jpg)
IPY Data management
A brief summaryØystein Godøy
![Page 2: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/2.jpg)
IPY Organisation
ICSU/WMOJoint Committee
Data Policy & Management
Subcommittee
ObservationsSubcommittee
Education, Outreach & Communication Subcommittee
ProgrammeOffice
Space TaskGroup
![Page 3: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/3.jpg)
Twelfth WMO Congress, Resolution 40 (1995)
Thirteenth WMO Congress, Resolution 25 (1999)
ICSU General Assembly Resolution (1996)
ICSU Assessment on Scientific Data and Information (2004)
Article III-1c from the Antarctic Treaty (1959)
the Intergovernmental Oceanographic Commission Data Exchange Policy (2003)
International consensus on free and open access
![Page 4: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/4.jpg)
The primary purpose should be to serve the objectives of IPYInternational exchange (>60 countries)Interdisciplinary science (>50,000
participants representing physical, life, and social scientists)
Building a legacyIncrease the value of observations
Overarching purpose
![Page 5: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/5.jpg)
Data used by IPY
Data generated by IPY
Special Cases:• Human subjects• Intellectual property of LTK• Where data release may cause harm
“…the IPY Joint Committee requires that IPY data, including operational data delivered in real time, are made available fully, freely and on the shortest feasible timescale.”
http://www.ipy.org/Subcommittees/final_ipy_data_policy.pdf
IPY Data Policy—Defining IPY Data
![Page 6: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/6.jpg)
Implications
Publish and share your dataIncreased data valuePromote interdisciplinary useHave someone measured something you
need?
Preserve and archive your dataGenerate a legacyThe basis for the next IPY
![Page 7: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/7.jpg)
What has been done?
The Data committee has defined a data policy
Workshop on data management http://classic.ipy.org/international/joint-committee/data-management.h
tm
IPY metadata profile http://ipydis.org/metadata.html
Data coordinators in US, Canada, UK and Norway
Data released for IPY use by ECMWF, ESA and others
Some data management projects have been funded
But not enough...IPY Layer in Google Earth available at
http://ipy.org
![Page 8: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/8.jpg)
IPY data policy - metadata
“All IPY data must be accompanied by a full set of metadata that completely document and describe the data.”
“Regardless of any data access restrictions or delays in delivery of data itself, all IPY projects must promptly provide basic descriptive metadata in an internationally recognized, standard format to an appropriate catalog or registry.”ISO19115/ISO19139
![Page 9: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/9.jpg)
“All data registries and repositories collecting data and metadata from IPY projects are required to collect and share sufficient information to adhere to the IPY Metadata Profile”
Basic who, what, where, when in either FGDC, DIF, THREDDS (ISO coming, but could use some help), plus some information on metadata provenance.
Controlled vocabulary from GCMD for some fields.
The “bare minimum of information necessary to allow simple discovery across disciplines and to ensure we can track the heritage of the metadata in a broadly distributed data management environment.”
Details are available at http://ipydis.org/
IPY Metadata Profile (and crosswalk)
![Page 10: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/10.jpg)
• Entry ID (controlled)• Data set title• Data set progress• Data set summary• Data set citation
information including Online Resource
• Parameters• Locations• ISO topic categories• Temporal coverage• Spatial coverage
• Data center contact information
• Access restrictions• Use constraints• Data Set Language• Metadata contact
information• Metadata authority• Metadata version• Last revision• IPY flag• IPY Project ID
IPY Metadata profile
![Page 11: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/11.jpg)
Formats:• Negotiate common formats and conventions
• ASCII is useful but not really a precise format• avoid proprietary formats • some suggestions: netCDF is popular for some, OGC (WMS/WCS/WFS)
compatibility is nice• Archives and users may have different needs
Access:• Integrate with many systems to allow increased user discovery (register
with the IPYDIS)• Use open source software when possible, use open standards everywhere.
Preservation• Open Archive Information System Reference Model• Attribute and provide info for attribution readily through all gateways
The Data
![Page 12: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/12.jpg)
What is out there?
![Page 13: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/13.jpg)
![Page 15: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/15.jpg)
ELOKA The Exchange for Local
Observations and Knowledge of the Arctic
works to provide data management and user support to facilitate the collection,
preservation, exchange, and use of local observations and knowledge of the
Arctic.
http://nsidc.org/eloka PI: Shari Gearheard
![Page 16: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/16.jpg)
18 May 2006
![Page 18: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/18.jpg)
• Assist on compliance with standards, identification of archives, development of the union catalogue, and other data management requirements for IPY.
• Visibly track the data flow for IPY.
• In collaboration with the IPO, develop a data registry that will continue throughout the IPY.
• Survey the planned projects and the data they intend to collect and identify existing archives, portals, experts, and significant gaps in the IPY data infrastructure.
Mark Parsons—Overall, US
Øystein Godøy—Operational Data, Norway
Canadian Coordinator—Overall, Canada
National coordinators in Netherlands, China, UK
Data coordinators
![Page 19: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/19.jpg)
International Operational Data Coordinator (1)
Fist version based upon DAMOCLES systemPostGRES metadatabaseOpeNDAP ⇒ THREDDS (OpeNDAP/WCS
support) or early 2008Adding WMS support for selected
fields/datasetsExpected late 2007
Second version based upon SIMDAT and first versionExpected 2008
![Page 20: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/20.jpg)
International Operational Data Coordinator (2)
Final version should be WIS DCPCExpected 2009
Have focused on ECMWF data so farDirect access at ECMWF disks if using WIS
DCPC software
Interface for metadata submissionAs well as notifications on interesting
datasets
Import/export of metadataMethod and format open issue
![Page 21: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/21.jpg)
International Operational Data Coordinator (3)
How can this service help promoting remote sensing data during IPY?Access to metadata?
Provide metadata information on available products (??)
Users get forwarded to data catalogues at the space centres(??)
OGC access to selected datasets(??)
![Page 22: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/22.jpg)
![Page 23: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/23.jpg)
![Page 24: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/24.jpg)
![Page 25: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/25.jpg)
National recommendations (1)
NorwayNational IPY-projects should relate to the
relevant standards forData collectionProcessingArchivalDescriptionTransferMetadata (ISO19115, ISO19139)OpenGIS
![Page 26: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/26.jpg)
National recommendations (2)
NorwayData and metadata shall be freely
available with minimal time delay
Metadata shall be transmitted to relevant national and international data centres
Data shall be transmitted to relevant national and international data centres for long term archival
Delivery of data and metadata is the responsibility of the project coordinator
![Page 27: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/27.jpg)
National recommendations (3)
NorwayA minimum of standardised file formats
should be used
Existing national and international infrastructure shall be utilised for long term archival
Data used in publications shall be correctly cited
The use of licences will be evaluated if this can improve data access
![Page 28: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/28.jpg)
National requirementsNorway
IPY projects with NFR-contractsshall submit a data management plan within
3 months of contract signingDescribing who is responsible for overall data
managementWho is responsible for datasetsWhere datasets will be stored and finally archived
shall submit yearly updates of the data management planWhat have been collectedWhere is it stored
![Page 29: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/29.jpg)
National IPY-data serviceNorway
DokIPYFunded by NFRShall connect existing metadata- and
databases in NorwayExtract and exchange metadataUtilise existing infrastructure and national
mandates atInstitute of Marine ResearchNorwegian Polar InstituteNorwegian Meteorological Institute
Robust/simple exchange of metadata
![Page 30: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/30.jpg)
METNO version
Relies on the same software as DAMOCLES and International Operational Data Coordination
New feature – metadata exchangeExamining international
requirements/specifications, not much found yet
Funding situation require simple and robust solutions
![Page 31: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/31.jpg)
Special issue
Remote Sensing of EnvironmentSpecial issue on Remote Sensing of the
CryosphereRemote sensing of Environment, Vol. 111,
Issues 2-3, 30 November 2007.Available online
![Page 32: IPY Data management A brief summary Øystein Godøy](https://reader036.vdocuments.site/reader036/viewer/2022062315/56649f3a5503460f94c57f49/html5/thumbnails/32.jpg)
How can we interface?