EMODnet Thematic Lot n° 4 - Chemistry

EMODnet Phase III

Updated guidelines for SeaDataNet ODV production

M. Lipizer, M. Vinci, A. Giorgetti, L. Buga, M. Fichault, J. Gatti, S. Iona, M. Larsen, R. Schlitzer, D. Schaap, M. Wenzer, E. Molina

Date: 12/04/2018

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Index

EMODnet Phase III............................................................................................................................1

Updated guidelines for SeaDataNet ODV producton......................................................................1

Updated guidelines for SeaDataNet ODV producton..........................................................................1

Introducton......................................................................................................................................1

SeaDataNet ODV import format.......................................................................................................1

How to check your SeaDataNet ODV fle format?............................................................................6

Vocabulary........................................................................................................................................7

How to choose the correct P01?......................................................................................................8

Flagging of Data Below Detecton Limits and Data Below Limit of Quantfcatoni.......................13

Guidelines for SeaDataNet ODV producton for sediment data and contaminants in biota.............14

Contaminants in the Sediment.......................................................................................................14

Contaminants in Biota....................................................................................................................18

Partcular case in biota datai......................................................................................................18

Other general issues and inventory of common errors.................................................................21

Acknowledgements:We acknowledge the fundamental contribution of the British Oceanographic Data Centre(BODC) for the design, development and continuous management and update of theVocabularies and for the provision of the webservices.

How to cite this document: M. Lipizer, M. Vinci, A. Giorgetti, L. Buga, M. Fichault, J. Gatti, S. Iona, M. Larsen, R.Schlitzer, D. Shaap, M. Wenzer, E. Molina, 2018, EMODnet Phase III - Updated guidelinesfor SeaDataNet ODV production, 12/04/2018, 23 pp., DOI: 10.6092/259c43eb-4ba4-419b-bb38-df00e189bd35

0

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Updated guidelines for SeaDataNetODV production

IntroductionEMODnet Chemistry uses SeaDataNet infrastructure (https://www.seadatanet.org/) for thetechnical set-up. In particular, it adopts:

A set of Standards for metadata description and data formats:

SeaDataNet (SDN) Standards for metadata (https://www.seadatanet.org/Standards/Metadata-formats)

Common Vocabularies (i.e. standardised terms that cover a broad spectrum of disci-plines) to allow consistency and interoperability (https://www.seadatanet.org/Standards/Common-Vocabularies)

Common Data Index mechanism (CDI) to access data with data policy (https://www.seadatanet.org/Standards/Metadata-formats/CDI)

Ocean Data View format (ODV) for data exchange (https://www.seadatanet.org/Standards/Data-Transport-Formats)

Data Quality Control procedures (https://www.seadatanet.org/Standards/Data-Quality-Control)

A set of software tools specifically developed for metadata and data formatting, data exchangeand visualization:

MIKADO to prepare XML metadata files NEMO to enable conversion from any type of ASCII format to the SeaDataNet ODV and

Medatlas ASCII formats as well as the SeaDataNet NetCDF (CF) format OCTOPUS to convert files in a given SeaDataNet format to another SeaDataNet format

(e.g.: ODV to NetCDF, MedAtlas to NetCDF, MedAtlas to ODV) and to check the compli-ancy of SeaDataNet MedAtlas and ODV files format

ODV as the fundamental data analysis and visualisation software. DIVA to spatially interpolate (or analyse) observations on a regular grid in an optimal

way

SeaDataNet ODV import formatAll data entering EMODnet Chemistry must be converted to SeaDataNet ODV import format,which is the common standard format. Delivery of data to users requires common data trans-port formats, which interact with other SeaDataNet standards (Vocabularies, Quality FlagScale) and SeaDataNet analysis & presentation tools (ODV, DIVA). Detailed guidelines on Datatransport formats area for data in the water column and specific for biological data are avail -able on: https://www.seadatanet.org/Standards/Data-Transport-Formats (Fig. 1). All SDNODV files are .txt files.

1

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 1: SeaDataNet datafile formats documentation (https://www.seadatanet.org/Standards/Data-Transport-Formats)

From: “SeaDatanet DATAFILE FORMATS”:

The fundamental data model underlying the format is the ASCII spreadsheet (ODV spread-sheet files may contain comments, and the column separation character may be TAB or semi-colon, https://odv.awi.de/fileadmin/user_upload/odv/misc/odv4Guide.pdf ) : i.e. a collectionof rows each having the same fixed number of columns. There are three different types of col-umn:

Metadata columns Primary variable data columns (one column for the value plus one for the qualifying

flag) Data columns (one column for the value plus one for the qualifying flag)

The metadata columns are stored at the left hand end of the row, followed by the primary variable columns and then the data columns.

There are three different types of rows:

Comment rows

2

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Semantic header row Column header row Data row

Comment lines start with two slashes //as first two characters of the line and may contain ar -bitrary text in free format. Comment lines may, in principle, appear anywhere in the file, mostcommonly, however, they are placed at the beginning of the file and contain descriptions ofthe data, information about the originator or definitions of the variables included in the file.

The Semantic header consists in a set of ‘special’ mandatory comment lines that must appearbefore the column header row. Their function is to map the text strings used to label the meta-data and data columns to standardized SeaDataNet concepts, which is necessary if data filesfrom different sources are to be combined in a meaningful way (Fig. 2, Fig. 3).

Fig. 2: Example of SDN ODV file, with depth as primary variable (in column 10):

3

Important: there MUST be exact correspondence between Semantic header and Header. The order of the first 9 metadata column MUST be respected (ie. Cruise; Station; Type; date-time; Long.; Lat.; Local_CDI_ID; EDMO_Code; Bot.Depth.).

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

The metadata values are followed by the primary variable value (on the 10th col-umn), its flag and then the data value plus qualifying flag pairs for each parameter. Flag valuesare taken from the SeaDataNet vocabulary for qualifying flags (L20 at http://vocab.ner-c.ac.uk/collection/L20/current).

Fig. 3: Example of SDN ODV file showing correspondence between SDN_parameter_map-ping lines and measured parameters.

The primary variable depends on the type of dataset:

Time series: point time series have row_groups made up of measurements from a given in-strument at different times. The primary variable (column 10) is time. In case datacome from samples taken in the same position and sampled at a fix frequency, thedataset type can be considered as “time series” and the SDN ODV file must providetime as primary variable in the 10th column. This is particularly suitable for regu-lar monitoring stations and is commonly used for mooring data (eg. oceanographicbuoy).

Vertical profile: profile data have row_groups made up of measurements at different depths.The primary variable (which is indicated in column 10) is the ‘z co-ordinate’, whichis either depth in meters or pressure in decibars (in case of water column data), ordepth below seabed (in case of sediment data). It is the choice for stations usuallynot repeated regularly over time, for which several depths are sampled. Typical ex-amples: CTD profiles, bottle stations (chemical data, chlorophyll, plankton,…)

However, this distinction is not always clear and in cases of a same position, sampled overtime at different depths, some data providers prefer to consider these as “repeated profiles”stations.

4

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Lastly, when data are not available for the same position at more or less regulartime periods, the dataset type is regarded as “vertical profiles” even if there is only 1 samplingdepth and it is not a real profile. This can be the case of sediment samples taken duringoccasional surveys, for example.

In order to produce correctly SDN ODV datasets for EMODnet Chemistry, the use NEMOsoftware is STRONGLY RECCOMENDED. User Manual, NEMO presentation and NEMOexamples are available on: https://www.seadatanet.org/Software/NEMO and lessons onNEMO given during EMODnet Chemistry 3 first training (18-19 May 2017, Trieste) areavailable on: http://www.emodnet-chemistry.eu/trieste_16-19_May_2017.html.

SeaDataNet ODV Dataset templates for water column data (eg. CTD, nutrients, dissolvedoxygen,…) are available on: https://www.seadatanet.org/Standards/Data-Transport-Formats(Fig. 4, 5, 6)

Fig. 4: SeaDataNet ODV Dataset templates for water column data.

5

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 5: Example of time series of nutrient data in the water column. NB: Tofacilitate reading, the above example shows the Column header (CH) displayed in 2lines, but the standard SeaDataNet ODV file in .txt has the header on 1 single row.

Fig. 6: Example of vertical profile of CTD data in the water column. NB: To facilitatereading, the above example shows the Column header (CH) displayed in 2 lines, but thestandard SeaDataNet ODV file in .txt has the header on 1 single row.

How to check your SeaDataNet ODV file format?- SeaDataNet import of ODV software- OCTOPUS

OCTOPUS (https://www.seadatanet.org/Software/OCTOPUS) allows to check the compliancyof your SeaDataNet ODV files format. Once the directory of files has been chosen, it is possibleto check the format of the file(s) by clicking on the "Check the input format" button (Fig. 7).

6

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 7: Example of Octopus working window.

VocabularyUse of common vocabularies in all metadatabases and data formats is an importantprerequisite towards consistency and interoperability. Common vocabularies consist of listsof standardised terms that cover a broad spectrum of disciplines of relevance to theoceanographic and wider community (https://www.seadatanet.org/Standards/Common-Vocabularies). The British Oceangraphic Data Center (BODC) manages and updates allvocabularies used by EMODnet Chemistry(http://seadatanet.maris2.nl/v_bodc_vocab_v2/welcome.asp).

The basic term to describe parameter is P01 (BODC Parameter Usage Vocabulary) which ismade up by 3 main elements:

The property observed (e.g. “concentration”)

7

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

The entity observed (e.g. a chemical substance like “phosphate” or a physicalphenomenon like “waves” or a biological entity like “Skeletonema costatum”)

The matrix (e.g. water body and its various phases, sediments and its variouscomponents, atmosphere and its various components, etc.)

Sometimes P01 contains also information on sampling, filtration and analytical method (Fig.8).

Fig. 8: Example of P01.

How to choose the correct P01?NEMO allows to find the correct P01 starting from the more general P02 Vocabulary (SDNParameter Discovery Vocabulary, Fig. 9), or from P09 (MEDATLAS Parameter UsageVocabulary).

Fig. 9: Example: P02 to P01, http://seadatanet.maris2.nl/v_bodc_vocab_v2/vocab_relations.asp?lib=P02

The user can also use the ad hoc P35 Vocabulary (EMODnet Chemistry aggregated parameternames, http://seadatanet.maris2.nl/v_bodc_vocab_v2/vocab_relations.asp?lib=P35 andhttp://vocab.nerc.ac.uk/collection/P35/current/, Fig. 10) developed for EMODnet Chemistryand find the correct P01 from the aggregated terms (Fig. 11).

8

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 10: Extract from P35 vocabulary (EMODnet Chemistry aggregated parameternames).

Fig. 11: Example: from P35 to P01

9

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

The British Oceanographic Data Centre provides a useful and user-friendly tool forVocabulary search available at:https://www.bodc.ac.uk/resources/vocabularies/vocabulary_search/ for simple andadvanced search for vocabulary and within a vocabulary (Fig. 12-14).

Fig. 12: Example of “Vocabulary search” tool(https://www.bodc.ac.uk/resources/vocabularies/vocabulary_search/).

10

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 13: Example of “Simple search within a vocabulary”, example: nitrate in P01Vocabulary tool.

Lastly, the user can also search for the correct P01 term with the P01 vocabulary - facet search on semantic components (http://seadatanet.maris2.nl/bandit/browse_step.php ) (Fig. 14a) and with (https://www.bodc.ac.uk/resources/vocabularies/vocabulary_search/ ) (Fig.14b).

11

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 14a: Output of search of “nitrate” in P01 Vocabulary.

12

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Fig. 14b: Output of search of “nitrate” in P01 Vocabulary.

Flagging of Data Below Detection Limits and Data Below Limit of Quantification:The SDN L20 (Seadatanet measureand qualifier flags) Vocabulary which includes QualityFlags concepts and definitions has been updated with a new Quality Flag needed to flag datawith concentrations “below limit of quantification (LOQ)”. The new flag “Q” (value below limitof quantification) is used when “The level of the measured phenomenon was less than thelimit of quantification (LoQ). The accompanying value is the limit of quantification for theanalytical method or zero if that value is unknown.”

This must not be confused with QF 6 (value below detection) which indicates that “the level ofthe measured phenomenon was less than the limit of detection (LoD) for the methodemployed to measure it. The accompanying value is the detection limit for the technique orzero if that value is unknown.”

Data originators have to be sure to use the correct QF.

13

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Guidelines for SeaDataNet ODVproduction for sediment data and con-taminants in biota

Contaminants in the Sediment

For the management of data in the sediments, the protocols proposed within the EU projectGeo-Seas Pan-European infrastructure for management of marine and ocean geological andgeophysical data (http://www.geo-seas.eu/) have been taken into consideration and adoptedand previous experience gained with EMODnet phase 1 and 2 have been used. An example of aSDN ODV template provided by Geo-Seas is attached to this report.

Samples can be collected in the sediment with several devices, and data may refer to differentdepth layers below the seabed (ex. Fig. 15 c).

Fig. 15: Examples of sampling devices: Grab sampler (a), box corer (b), corer (c) andmulticorer (d)

The “depth” variable represents a depth INSIDE the seabed and should be indicated as“COREDIST” which is the distance of a sensor or sampling point below the floor of a waterbody (Fig. 16). The preferred unit for COREDIST is “m (= meters)”.

14

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

T

Fig. 16: Example of a vertical profile in the sediment. For SeaDataNet ODV depth mustbe coded as “COREDIST” and expressed in “m (meters)“.

The following table contains several terms used to describe the depth where sedimentsamples were collected (Tab. 1).

The depth of the sample is indicated as:

EntrytermP01entrykey

Definition

Depth below surface ofthe bed

COREDIST The distance of a sensor or sampling point below thefloor of a water body

Sample length SEGMLENG The physical length of a sample upon whichmeasurements have been made

Minimum depth belowsurface of the bed

MINCDIST The distance between the top of a core sample andthe seabed.

Maximum depth below MAXCDIST The distance between the base of a core sample and

15

Important: DO NOT use depth as ADEPZZ01, which corresponds to “The distance of a sensor or sampling point below the sea surface.

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

surface of thebed

the seabed. For an unsegmented core with its topcoincident with the bed this is equivalent to the corelength.

Table 1: List of P01 terms and definitions for “depth” parameters.

Example: COREDIST = 0 in case of surface sediment data

COREDIST ≠ 0 when samples are taken below the sea floor (ex. Depth of sample A, B,…, Fig. 17in core cut into sections and depth of the sample in the unsegmented core, Fig. 18)

Fig. 17: Example of core cut into sections.

Fig. 18: Example of unsegmented core.

P01 terms:

16

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

COREDIST: http://vocab.nerc.ac.uk/collection/P01/current/COREDIST/ - Thedistance of a sensor or sampling point below the floor of a water body MANDATORY

MINCDIST: http://vocab.nerc.ac.uk/collection/P01/current/MINCDIST/ - The distancebetween the top of a core sample and the seabed. OPTIONAL (STRONGLYRECOMMENDED)

MAXCDIST: http://vocab.nerc.ac.uk/collection/P01/current/MAXCDIST/ - The distancebetween the base of a core sample and the seabed. For an unsegmented core with its topcoincident with the bed this is equivalent to the core length (SEGMLENG) OPTIONAL(STRONGLY RECOMMENDED)

SEGMLENG: http://vocab.nerc.ac.uk/collection/P01/current/SEGMLENG/ - The physicallength of a sample upon which measurements have been made OPTIONAL

As a general rule:

COREDIST = MINCDIST + (MAXCDIST-MINCDIST)/2

and SEGMLENG = MAXCDIST - MINCDIST.

The information of depth of the seafloor should be put in the metadata "Bot.Depth" (9 th

column in SDN ODV format, see Tab. 2).

Table 2: Template for SDN ODV dataset of sediment profile. NB: To facilitate reading,the above example shows the Column header (CH) displayed in 2 lines, but thestandard SDN ODV file in .txt has the header on 1 single row.

17

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Contaminants in Biota

According to experience obtained during the previous phase of EMODnet Chemistry,contaminants in biota are mostly regarded as time series. This consideration is based on thefact that, most of the time, no depth information is linked to biota data. In this case, the SDNODV file must provide time as primary variable which is indicated in the 10th column (seeexample in Table 3).

Particular case in biota data:

Sometimes there are several measurements taken in the same position and time, but they aredistinguished according to their sample_id (case of ICES datasets). This additional variablecould be then used for products generation.

THIS ISSUE NEEDS MORE CLARIFICATION AND EXPERIENCE.

Please find also attached examples of time series files from Ifremer with Time as a variable inthe 10th column (Tab.3: 01950406_ODV_biota_TS.txt).

18

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Table 3: Template of SDN ODV dataset of time – series of contaminants in biota (IFREMER example 01950406_ODV_biota_TS.txt).NB: To facilitate reading, the above example shows the Column header (CH) displayed in 2 lines, but the standard SDN ODV filein .txt has the header on 1 single row.

19

EMODnet Thematic Lot n° 4 - ChemistryUpdated guidelines for SDN ODV production

Table 4: Template of SDN ODV dataset of vertical profile of contaminants in biota (note however that in this example value ofdepth is missing and QV is 9 = missing value). NB: To facilitate reading, the above example shows the Column header (CH)displayed in 2 lines, but the standard SDN ODV file in .txt has the header on 1 single row.

20

EMODnet Thematic Lot n° 4 – ChemistryUpdated guidelines for SDN ODV production

Other general issues and inventory of common errors

A list of common errors and suggestions to avoid them is listed in Tab. 5.

Table 5: List of common errors and suggestions to avoid them.

Missing values:

In case of missing value, the field MUST be left EMPTY, and the associated QF is 9 (=missing value).