© crown copyright met office inter-comparison and validation task team matt martin, fabrice...

© Crown copyright Met Office

Inter-comparison and Validation Task Team

Matt Martin, Fabrice Hernandez

GOVST meeting, Paris, November 2011.


Agenda

1. 14:00 – 14:30. Introduction:

1. Overview of IV-TT objectives.

2. Existing workplan of IV-TT.

3. Summary and actions from Santa Cruz workshop.

2. 14:30 – 16:30. Recent progress on various initiatives:

1. Routine class 4 inter-comparison (Matt).

2. QC inter-comparison project (Jim).

Coffee break (30 minutes).

3. Discussion of GOV multi-model ensemble system (Matt).

4. Discuss inter-comparison of climate indices with GSOP (all).

3. 16:30 – 17:00. Summary and report for GOVST.

4. + XBTs and other “core” validation data-sets.


Overview of IV-TT objectives

• To coordinate and promote the development of scientific validation and inter-comparison of operational oceanography systems.

• Activities include:

• The definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skills), for physical and biogeochemical parameters.

• Setting up specific global and regional intercomparison experiments.

• Define metrics related to specific applications.

• Liaise with the JCOMM ET-OOFS team for operational implementation.

• Develop cooperation with CLIVAR/GSOP for climate issues.


Work plan of IV-TT

1. Provide a demonstration of inter-comparison and validation in an operational framework:• write proposal for production of routine class 4 metrics. • implement at participating OOFS.• routinely upload metrics from OOFS to the US GODAE server.• after a few months, meet up to discuss progress.• once mature, report to ET-OOFS for implementation operationally.• continue to test new metrics etc in GOV and regularly update ET-OOFS.

2. Improve inter-comparison and validation methodology for operational oceanography:• Obtain feedback from GOVST members participating in dedicated

meetings organized by specific communities• New metrics outlined during the workshop organized by the task team• Limited inter-comparison projects between some OOFS, in order to test

new metrics.

Membership of IV-TT

• Existing membership of the IV-TT:

• Fabrice Hernandez

• Matt Martin

• Alistair Sellar

• Hal Ritchie

• Greg Smith

• Gary Brassington

• Jim Cummings

• Clemente Tanajura

• Peter Oke

• Pavel Sakov

• Marina Tonani

• Japan contribution (to be discussed with Masa Kamachi)

• Others?

Outcomes from Santa Cruz workshop

Main outcomes/decisions at the workshop w.r.t IV-TT:

• Changes to implementation of Class 4 inter-comparison:

• The need to provide files with common information was emphasised.

• It was therefore agreed that UK Met Office produce the class 4 files 7-days behind real-time and send them to the US GODAE server.

• The other groups will retrieve those files and produce the same information at the same locations/times.

• Proposed GOV multi-model ensemble

• GOV contributions to GSOP climate indices inter-comparisons.


Actions from Santa Cruz wokshop

Action 3.1 Martin and Hernandez to update the class 4 inter-comparison proposal document

Action 3.2 Martin to distribute code to produce class 4 files and to calculate statistics from them

Action 3.3 All GODAE OceanView system representatives to begin sending class 4 files to the US GODAE server routinely.

Action 3.4 Martin and Hernandez to produce a document describing the proposed multi-model ensemble system.

Action 3.5 Martin, Hernandez and system representatives to work out the data size requirements for the Class 4 files and multi-model ensemble fields and send information to Jim Cummings

Action 3.6 Martin, Hernandez and Wilmer-Becker to produce a twiki page to share information on the class 4 and ensemble work.

Action 3.7 Martin, Hernandez and system representatives to investigate how GODAE OceanView systems can contribute to the climate indices initiative of GSOP – please also see action 4.7

Action 3.8 Martin and Hernandez together with Balamaseda to think about how the GSOP systems can be included within the inter-comparison of assimilation statistics and QC.

Action 3.9 Martin and Hernandez to update the IV-TT work plan to reflect the new initiatives on the multi-model ensemble and production of climate indices

Action 3.10 Martin to discuss including the plan for GODAE OceanView/CLIVAR community to work towards multi-model ensemble to be added to the GEO sub-task on “promoting the use of ensemble forecasting outside the weather forecasting community”.


Agenda

1. 14:00 – 14:30. Introduction:

1. Overview of IV-TT objectives.

2. Existing workplan of IV-TT.

3. Summary and actions from Santa Cruz workshop.

2. 14:30 – 16:30. Recent progress on various initiatives:

1. Routine class 4 inter-comparison (Matt).

2. QC inter-comparison project (Jim).

Coffee break (30 minutes).

3. Discussion of GOV multi-model ensemble system (Matt).

4. Discuss inter-comparison of climate indices with GSOP (all).

3. 16:30 – 17:00. Summary and report for GOVST.


Class 4 inter-comparisonOverview

• Aim is to develop a system for routine inter-comparison of the operational ocean forecasts contributing to GOV.

• In order to facilitate improvements to the systems and to help understand where our systems should focus development.

• Comparisons of forecasts against observations was chosen for this so that we don’t have to deal with excessive data volumes.

• These data will be made available for inter-comparison:

• in common formats;

• for a core sub-set of the various data types which the systems assimilate;

• to enable assessment of forecasts.


Class 4 inter-comparisonOverview

• Data types agreed on and now being implemented are:

• SST from in situ drifting buoys, using the US GODAE data-set.

• SST from L3 AATSR data-set.

• In situ T & S profile data from MyOcean (produced by Coriolis).

• Altimeter SLA data from MyOcean (CLS/Aviso).


Class 4 inter-comparisonRecent progress

Updated the proposal document, distributed to the IV-TT and put it on the GOV web-site.

• Appendix B gives a step-by-step guide of how to be do the inter-comparison – see next slides.

• Class 4 files for the Met Office system will be put on the USGODAE server within 5 days of the observation time, using the agreed observation datasets.

• Other participating centres should download these files using ftp and use them as a basis for their own class 4 files.

• This ensures that all centres use exactly the same observations.


Class 4 inter-comparisonProcedure (1)

1. Contact [email protected] to confirm your participation in the inter-comparison and to arrange the appropriate access and password.

2. Download the Met Office files from the USGODAE server: ftp.usgodae.org, directory (/pub/outgoing/class4). The files will be compressed with gzip and encrypted with openssl, and will have names of the format: class4_YYYYMMDD_FOAM_orca025_Version_DataType.nc.gz.enc

3. Decrypt the files using the following command:

4. % openssl enc -d -aes-256-cbc –pass pass:<passwd> -in file.gz.enc > file.gz

5. Decompress the file with gunzip, then rename the file to reflect your system, following the naming convention in section 4.

6. In Fortran (or another data analysis tool), open the NetCDF file and replace the model values with the equivalents from your system.

mailto:[email protected]

ftp://ftp.usgodae.org/


Class 4 inter-comparisonProcedure (2)

7. Modify the following global attributes of the class 4 file to reflect your system (see table 2, section 4):

• institution, contact, creation_date, system, configuration, best_estimate_description, sst_generation_method

• There is no need to alter the following global attributes: title, version, obs_type, obs_files, time_interp

8. Ideally all participants will obtain the model counterparts using daily mean fields, with no time interpolation. If you do not have daily mean fields available, then modify the global attribute time_interp to describe how you have temporal processed of the fields (e.g. linear interpolation between 6-hourly instantaneous fields).

9. Compress the file with gzip and encrypt it with the following command, using the same password as was used to decrypt the original file:

% openssl enc -aes-256-cbc -a –salt –pass pass:<passwd> -in file.txt -out file.enc

10. Upload the file via ftp to the USGODAE server: ftp.usgodae.org, directory: /pub/incoming/class4


Class 4 inter-comparisonRecent progress

• Production of SST files at UKMO now being done routinely and uploaded onto the US GODAE server.

• Other data types:

• code written and tested for T & S profiles, and altimeter SLA data.

• Will get T & S profiles, and SLA transferred to US GODAE server soon (profiles in next week or so, SLA before end of 2011).

• Setting up web-pages to display information from these class 4 files.

• Could be used as a basis for an external web-page to display inter-comparison information. If so this would be password protected.

Class 4: FOAM results

In Situ SST

Profile Temperature

RM

SE

oC

RM

SE

oC

Class 4: Web pages


Class 4 inter-comparisonOutstanding issues

• Participation of other groups:

• Canadians already started downloading data. Greg Smith: “I've been able to access the UK met file, and have successfully downloaded, unlocked and viewed its contents. We've not got going on setting up the collocalization code to provide model equivalents from our forecasts yet. That being said, we are keen to participate and I anticipate that we'll be up and running in the next few months.”

• Mercator plan to participate.

• Others? At the workshop, BLUELink, NRL and NCEP agreed to participate.


GOV multi-model ensembleDiscussion

A proposal was made at the Santa Cruz workshop to begin generation of a GODAE OceanView multi-model ensemble. There was a general consensus that this should be done. Practical aspects were discussed including:

• It was agreed to provide information on native grid

• It was agreed to provide analyses and daily mean forecasts out to 6-days of SST, SSS, SSH, surface u and v, and sea-ice concentration and thickness. MLD.

• When producing an ensemble we discussed whether the common grid should be at the highest or lowest resolution? It was agreed that the highest resolution grid would be most appropriate in most cases.

• Groups interested in ensembles in particular areas could develop a regional version of the ensemble in that region, e.g. Gulf of Mexico, EAC.

• The ensemble mean/median should be compared with each forecast product to diagnose where they differ significantly.



Suggestions from Fabrice:

• every one produces its own native surface data on its own ftp server

• password protected to a limited number of partners well identified (the GOV partners who decide to participate), in order to cope with e.g. MyOcean requirements on users.

• no dedicated task for assembling forecast at the moment: let ask the partners to start whenever they can, for the area they need. Imagine that we have a Oil Spill simulation in Europe, then UKMO and Mercator could "collect" from the different servers what they need, and do their "cooking"

• If a partner does something, he tells the others that it is starting, and get committed in a 6 month time, or for the next IV-TT meeting to report on his work.


Other questions:

• Which OOFS products are already available routinely in a suitable way?

• How should we share information about access to products?

• Do we need to define access mechanisms? Ftp, opendap, …

• Should we have an on-going standard “global” ensemble product?

• Will these ensemble products be generally available?

• Should we define formats for the ensemble products?

• Change notification for contributing systems, format etc.

• Feedback to contributing systems from ensemble producers.

Climate indicesDiscussion

• The production of climate indices from GODAE OceanView systems was discussed in Santa Cruz.

• Mercator are involved in the GSOP inter-comparisons and produce various climate indices from their 1/4- and 1/12-resolution model, which has proved useful.

• For a number of the GODAE OceanView systems, the priorities are the class 4 and ensemble inter-comparison projects discussed before.

• However, it was agreed that we should produce the surface climate indices as part of the multi-model ensemble as a minimum (and interested groups could provide other information if desired).

• Any progress from any of the GOV systems?

• Any further comments about this activity?

Core validation data-sets

Should we expand the definition of a core set of data which is used for validation/inter-comparison of all systems?

e.g. XBTs, gliders

Should we withhold some of those data from assimilation?

Should we formalise this/get agreement from all OOFS?

© crown copyright met office inter-comparison and validation task team matt martin, fabrice...

Documents