govst-iii meeting, esa hq, paris, 14-18 november 2011 inter-comparison and validation task team...

GOVST-III Meeting, ESA HQ, Paris, 14-18 November 2011

Inter-comparison and Validation Task Team (IV-TT)

Report for GOVST

Matt Martin, Fabrice Hernandez


Contents

• Introduction– Objectives of IV-TT– Membership– Main elements of work-plan

• Santa Cruz workshop outcomes for IV-TT.• Recent progress:

– Routine class 4 inter-comparison project.– GOV multi-model ensemble project.– Links to GSOP inter-comparison of climate indices.– Other validation work.

• Summary and future plans.


IntroductionObjectives of IV-TT

• To coordinate and promote the development of scientific validation and inter-comparison of operational oceanography systems.

• Activities to include:– The definition of metrics to assess the quality of analyses and

forecasts (e.g. forecast skill), for physical and biogeochemical parameters.

– Setting up specific global and regional inter-comparison experiments.

– Define metrics related to specific applications.

• Liaise with the JCOMM ET-OOFS team for operational implementation.

• Develop cooperation with CLIVAR/GSOP for climate issues


IntroductionMembership

Existing members of IV-TT:

– Fabrice Hernandez (France)– Matt Martin (UK)– Alistair Sellar (UK)– Hal Ritchie (Canada)– Greg Smith (Canada)– Gary Brassington (Australia)– Jim Cummings (US)

– Clemente Tanajura (Brazil)– Peter Oke (Australia)– Pavel Sakov (Norway)– Marina Tonani (Italy)– Avichal Mehra (US)

Other members welcome.


IntroductionIV-TT work plan

1. Provide a demonstration of inter-comparison and validation in an operational framework:• write proposal for production of routine class 4 metrics. • implement at participating OOFS.• routinely upload data from OOFS to the US GODAE server.• generate statistical inter-comparisons on a web-page.• once mature, report to ET-OOFS for implementation operationally.• continue to test new metrics etc in GOV and regularly update ET-

OOFS.

2. Improve inter-comparison and validation methodology for operational oceanography:• Obtain feedback from GOVST members participating in dedicated

meetings organized by specific communities• New metrics outlined during workshops organized by the task team• Limited inter-comparison projects between some OOFS, in order to

test new metrics.


Santa Cruz workshop IV-TT objectives

– GODAE OceanView – CLIVAR-GSOP Workshop on

Observing system evaluations and Inter-comparison

13 – 17 June 2011, UCSC, Santa Cruz, CA, USA

– Specific goals of the workshop were to:• Demonstrate the value of in situ and satellite observations to short-

term, seasonal, and decadal forecast systems;• Move towards routine monitoring of the global ocean observing

system (explore possibilities for “Observation Impact Statements”);• Review inter-comparison of class 4 metrics from operational short-

range forecast systems and seasonal prediction systems; and • Linking inter-comparison and observing system evaluation

monitoring activities within GODAE OceanView to those of CLIVAR.


Santa Cruz workshop Inter-comparison and validation presentations

• Fabrice Hernandez GODAE OceanView IV Work Plan

• Yan Xue* Comparative Analysis of Upper Heat Content Variability from Ensemble Operational Ocean Analyses

• Alexandre Kurapov* The data assimilation component of the real-time coastal ocean circulation forecast system off Oregon

• Tong Lee Inferring meridional heat transport through meridional volume transport of the AMOC

• Clemente Tanajura A Sensitivity Study of short-range Forecasts over the Atlantic METAREA V with HYCOM and the Cooper Haines Scheme using different sea surface heights

• Mario Adani Quality Assessment of a 1985 - 2007 Mediterranean Sea Reanalysis

• Stephanie Guinehut Combination of in-situ and satellite observations to monitor the Global Ocean State

• Gary Brassington OceanMAPS version 2 - what did we get from the upgrade?

• Andrea Storto The CMCC experience in the context of Global Ocean reanalyses

• Oscar Alves An ensemble ocean data assimilation system for seasonal prediction and its comparison with other state-of-the-art ocean reanalysis

• Balakrishnan Nair Real-time and delayed mode validation for the Indian Ocean Forecasting System (INDOFOS)

• Gregory Smith Evaluation of CONCEPTS Ice-Ocean Forecasting System

• Santha Akella The GMAO Ocean Retrospective Analysis


Santa Cruz workshop Validation techniques

Some commonly used techniques from GOV and coastal groups:– Innovation statistics (bias, RMS, stdev, ACC), for common variables (T,

S, SSH, SST)– Time average increments for diagnosing model biases.– Assessing forecast skill using observations and analyses.– SSH variability and position of western boundary currents.

Some commonly used techniques from CLIVAR/GSOP groups:– Assessing meridional overturning circulation (AMOC);– Heat transport diagnostics;– Comparisons of the depth of the 20C isotherm; – Upper ocean heat content for various depth ranges.


Santa Cruz workshop Validation techniques

Less commonly used or new assessment techniques (for ocean forecasting systems) include:• Assessment of tides in a global system using tide gauges as reference;• Altimetry-based analysis of the internal tide (sampling the model in the same way as the

data);• Comparisons with HF radar data as an independent reference (especially useful when the

available velocity data is assimilated);• Forecast skill assessments in ensemble forecasting systems, e.g. by comparing spread

in the ensemble with the RMSE of the mean;• Calculating EOFs of assimilation increments to ascribe various dynamical processes to

the model errors;• Using Lyapunov exponents to show the scales which are reconstructed by the analysis;• Statistical verification of the location of features (e.g. eddies);• Verification of Arctic Ocean circulation and investigation of Arctic heat budget;• Assessment of volume-integrated heat content in a particular 3D box and calculation of

various contributing terms (advective and surface fluxes, data assimilation impact);• Validation of geostrophic currents at 1000m depth using the ANDRO data-base (based

on Argo trajectories);• Comparison of currents with ADCP data from ships and from moored buoys;• Comparison of modeled ice thickness with IceSat thickness data..


Santa Cruz workshop Inter-comparison talks

Hernandez presented an overview of the IV-TT work plan.

Xue presented a detailed comparison of upper ocean heat content (over the top 300m, HC300) from various ocean reanalyses. The questions addressed by the study included:

• How well is the mean HC300 analysed?• What are the impacts of changes in the observing system?• How well do the reanalyses capture inter-annual and multi-decadal

variability?• What climate indices should be monitored?• What is the role of HC300 in predictability of ENSO, PDO, NAO, …?

Also inter-comparisons carried out by the PEODAS group for assessing their system.


Yan XueClimate Prediction Center/NCEP

Mean Heat Content in 1985-2009

obs1985

obs2008


Santa Cruz workshop IV-TT breakout

1. Discussions of the routine class 4 inter-comparison project

=> changes to the proposed method to ensure consistency of observations and format.

2. Initial discussion of a proposed project to produce a GOV multi-model ensemble, which we decided to pursue.

3. Discussions of ways in which the GOV groups could get involved in the CLIVAR/GSOP inter-comparison of climate indices.

These three initiatives were discussed further yesterday…


1. Class 4 inter-comparisonOverview

– Aim is to develop a system for routine inter-comparison of the operational ocean forecasts contributing to GOV.

– This will be done in order to facilitate improvements to the systems and to help understand where our systems should focus development.

– Comparisons of forecasts against observations was chosen for this so that we don’t have to deal with excessive data volumes.

– These data will be made available for inter-comparison:• in common formats;• for a core sub-set of the various data types which the systems

assimilate;• to enable assessment of forecasts.



– The core data types agreed on for use in the inter-comparison, which are now being implemented are:

– SST from in situ drifting buoys, using the US GODAE data-set.– SST from L3 AATSR data-set.– In situ T & S profile data from MyOcean (produced by Coriolis).– Altimeter SLA data from MyOcean (CLS/Aviso).

– Model data exchange will be done through the US GODAE server. All files will be encrypted.

– The project proposal document has been updated to include a step-by-step guide of how groups can contribute to the inter-comparison.

– The document was distributed to the IV-TT and put on the GOV web-site.



UKMO download obs files and produce model analysis, forecast, persistence and climatological information at the obs locations in netCDF format.

US GODAE server

Each OOFS downloads the UKMO files and replaces the model data with those from their system.

Any participating group can download files and inter-compare the results.

Produce a core set of plots for the GOV web-site (password protected).

Observation inputs

~5 days behind real time


1. Class 4 inter-comparisonCurrent status

• Production of SST files at UKMO now being done routinely and uploaded onto the US GODAE server – set up by Andy Ryan and Alistair Sellar.

• Other data types: – code written and tested for T & S profiles, and altimeter SLA data.– Will get T & S profiles, and SLA transferred to US GODAE server soon

(profiles in next week or so, SLA before end of 2011).

• Canadians already started downloading data and plan to start producing the files soon. Mercator, Bluelink, NRL, NCEP and REMO also plan to participate soon.

• Now setting up web-pages (currently internally) to display information from these class 4 files.

• These could be used as a basis for an external web-page to display inter-comparison information. If so this would be password protected.


1. Class 4 inter-comparison Example results from FOAM

In Situ SST

Profile Temperature

RM

SE

o CR

MS

E o C


1. Class 4 inter-comparisonPrototype web pages


2. GOV multi-model ensembleOverview

– In the past, for particular incidents (e.g. oil spills), groups have requested sharing of ocean forecasts from GOV partners in order to provide ensemble information.

– A proposal was made at the Santa Cruz workshop to begin generation of a GODAE OceanView multi-model ensemble.

– There was a general consensus that this should be done and some practical aspects were discussed at the workshop. Yesterday we had further discussions.

– A number of the OOFS now have model data available externally and routinely on OpenDAP or ftp servers.


2. GOV multi-model ensembleProposal

– In order to make the exchange of the data an easier process in the case of emergencies, and to perform some scientific studies on the use of multi-model ensembles, it was agreed that some pilot studies will be carried out:• In the Australian region by CSIRO (Peter Oke).• Globally by NCEP (Hendrik Tolman).

– It was agreed to use analyses and daily mean forecasts out to 6-days of SST, SSS, SSH, surface u and v, and sea-ice concentration and thickness. Also proposed to include mixed layer depth, although difficulties with common definition.

– First step is to collect information on access for this data and distribute to the participating groups.


3. GOV/GSOP climate indices inter-comparison

– The production of climate indices from GODAE OceanView systems was discussed at the Santa Cruz workshop.

– Mercator are involved in the GSOP inter-comparisons and produce various climate indices from their 1/4- and 1/12-resolution models, which has proved useful.

– For a number of the GODAE OceanView systems, the priorities are the class 4 and ensemble inter-comparison projects discussed before.

– However, it was agreed that we should produce the surface climate indices as part of the multi-model ensemble as a minimum (and interested groups could provide other information if desired).

– In addition, it was agreed that some of the systems would produce upper ocean heat content estimates on a monthly basis on a 1 degree grid, and provide them to Yan Xue.


Other validation issues discussed yesterday

The XBT community would like to know the impact of their data on the GOV systems.

Some groups have agreed to examine particular repeat lines (e.g. near Brazil, near Australia) and to inter-compare model data with the XBTs, e.g.:

• REMO will compare results to XBT lines from Brazil.• CSIRO will compare results to XBT lines near Australia.

Tide gauges. • Nicolas Ferry presented interesting results from a validation

exercise with Mercator. • It was agreed that this would be useful to include in the framework

of the class 4 inter-comparison project. But on a slower time-scale (once per year).

• The data-set used so far can be distributed to interested groups.• Will need to distribute a list of the location of the tide gauges so

that the model information can be calculated routinely.


Summary and future work

• Workshop at Santa Cruz helped to:

• Provide overview of validation done at the various centres.

• Refine plans for class 4 inter-comparison.

• Propose and discuss GOV multi-model ensemble.

• Make stronger links with GSOP groups.

• Class 4 inter-comparison is progressing.

• General agreement on the GOV multi-model ensemble for surface fields, and a way forward with some pilot studies (globally and around Australia).

• Agreement from some systems to begin contributing to the inter-comparison of upper ocean heat content organised within GSOP.

govst-iii meeting, esa hq, paris, 14-18 november 2011 inter-comparison and validation task team...

Documents

intercomparison of class

esa hq

govst members

statistical intercomparisons

introduction ivtt work

et oofs

govst matt martin

validation work