© crown copyright met office cost benefit studies for observing systems stuart goldstraw, met...

© Crown copyright Met Office

Cost benefit studies for observing systemsStuart Goldstraw, Met Office, CBS-RA3-TECO-RECO, 13th September 2014


Contents

This presentation covers the following areas

• Background

• The cost – benefit chain

• Introducing the FSO Tool

• Some results for the Met Office

• Discussion and Next Steps


Background

• CBS-14 established IPET-OSDE as part of the structure of OPAG-IOS.

• WIGOS Network Design Principles, as discussed at the CBS-Ext Session include the following:

5. DESIGNING COST-EFFECTIVE NETWORKS

Observing networks should be designed to make the most cost-effective use of available resources. This will include the use of composite observing networks.

• This presentation is based on the paper presented by John Eyre to IPET-OSDE-1 in March 2014 and the subsequent work undertaken John Eyre and Rebecca Reid.


Why study cost benefit for observing systems?

• Society benefits from the Services we provide too them

• These Services are developed using the outputs obtained from 12 recognised WMO Application Areas

• Observing Systems are developed and operated to meet the requirements of these Application Areas

• Observing Systems have costs associated with them

• The requirements of Society, our Services and Application Areas have evolved and will continue to evolve with time

• Therefore we need to optimise the investment in observing systems to ensure we meet the changing needs of Society.

© Crown copyright 2007

The cost-benefit chain

Observing Systems

Impact per cost

ServicesApplication

Areas Society

Indirect users of observations

M$ cost

M$ societal benefit

Benefit per

impact

Direct users of observations

Direct users of Services

Direct users of Application

outputs

Remote users of

observations

12 WMO Application Areas

• Global NWP

• Regional NWP

• Nowcasting and VSR Forecasting

• Sub-seasonal and longer range forecasting

• Aeronautical Meteorology

• Atmospheric Chemistry

• Ocean Applications

• Agricultural Meteorology

• Hydrology

• GCOS

• Climate Applications

• Space Weather



The four stages to cost benefit impact

• The costs of Observing Systems

• The impact of observations on Application Areas

• The impact of Application Areas on Services

• The benefit of Services to Society


Focus on the first two components

• The costs of Observing Systems

• The impact of observations on Application Areas




Simplify the challenge further

• The costs of UK Observing Systems

• The impact of observations on Global NWP




Considering the Global NWP Application Area

• Developments in Data Assimilation methodologies have allowed the community to develop new observations impact tools.

• The Forecast Sensitivity to Observations (FSO) tool can provide an indication of the relative impact of observations on forecast skill

• We also understand the costs of the observing systems we operate

• Therefore we can determine a relative impact per cost of the observing system, the first two steps in the benefit chain.


The Adjoint-based FSO Method

The change in forecast error at T+24 is entirely due to the assimilation of observations at T+0.

Slide courtesy of Richard Marriott


FSO evaluation – total impact

Total impact of observations by type, period April-July 2013

Negative impacts show an improvement in forecast skill

© Crown copyright Met Office© Crown copyright 2007

FSO Evaluation – impact per observation

Impact per observation by type, period April-July 2013

Negative impacts show an improvement in forecast skill


Costs of Met Office Observing Programmes in 2013 in US$

• AMDAR*• ASAP*• Drifting Buoys• Moored Buoys• Ship Obs• Surface Synoptic• Wind Profilers^• GPS IWV• Radiosonde^

$0.47M

$0.33M

$0.07M

$1.53M

$0.87M

$6.77M

$0.27M

$0.19M

$1.75M


Impact of global observations and use of Met Office costs


Firstly considering the total impact

Radiosonde, AMDAR and Synops are showing largest impact


Secondly considering the impact per observation

Moored buoys, Ship Obs and Synops showing similar impact


Finally considering impact per cost

AMDAR and Drifting Buoys show high impact per $

Distribution of buoy observations


Distribution of synop and ship obs



Interpretation of results

• Impacts must be taken in the context of the total current ‘system’: the observing system; the Data Assimilation system and the Model version.

• Impacts should be interpreted as "the effective impact which those observations are having in the presence of all other observations" (RM 2013)

• Impacts only represent the change in error of our chosen forecast metric – i.e. the global moist energy norm on 24-hour forecasts.

• Impacts should be averaged over many cases, in this case 4 runs a day for 3 months.


• Considering the three layers of detail together provides a fuller evaluation of the value of networks

• Important to avoid misinterpretation of results: e.g. ‘high impact per cost’ does not imply ‘invest here’

• Results may help to advise evolutionary changes in network design, if they are considered in full context.

• They can provide a ‘top level’ analysis, and be considered alongside individual characteristics of the observation types of interest (e.g. data quality, location, etc)

• This analysis is presented as a first, tentative step at evaluating ‘cost benefit’ of observing networks – remembering the specificity of the metric

Discussion Points


• Improved estimates of the global, or regional costs of component observing systems are required.

• Results need to be generalised across multiple Global NWP centres.

• Development of methodologies to determine impacts across the other 11 WMO Application Areas.

• Development of methodologies to distribute costs of observing systems that serve more than one application area in a weighted manner.

• Further development of the end to end cost to benefits methodology – building on existing work, some of which has been highlighted this week.

What are the next steps


Acknowledgement

This presentation could not have been made with the ongoing work being undertaken by John Eyre and Rebecca Reid and the previous work and advice of Richard Marriot.


Thank you for your time

© crown copyright met office cost benefit studies for observing systems stuart goldstraw, met...

Documents

impact of application

observing systems

benefit of services

systems impact

met office discussion

societal benefit benefit

following areas

cost benefit chain