using fluxnet data to evaluate land surface models ray leuning and gab abramowitz 4 – 6 june 2008

Using FLUXNET data to evaluate land surface models

Ray Leuning and Gab Abramowitz

4 – 6 June 2008

CSIRO. Using FLUXNET data to evaluate land surface models

Land surface model evaluation framework

Reto Stockli’s ‘Model farm’


Schematic diagram of model components from asystems perspective

Liu, Y. Q. and Gupta, H. V. (2007). Uncertainty in Hydrologic Modeling: Toward an Integrated Data Assimilation Framework. Water Resources Research 43, W07401, doi:10.1029/2006/WR005756.

1. system boundary, B2. inputs, u3. initial states, x04. parameters, θ5. model structure, M6. model states, x7. outputs, y

Errors in each component affects model performance


Parameter estimation Multiple objective functions possible


Parameter estimation Multiple criteria possible, e.g. λE, NEE

The dark line between the two criteria’s minima, α and β, represents the Pareto set


Comparing RMSE of models of varying complexity across sites after parameter optimization

Hogue, T. S., Bastidas, L. A., Gupta, H. V., and Sorooshian, S. (2006). Evaluating Model Performance and Parameter Behavior for Varying Levels of Land Surface Model Complexity. Water Resources Research 42, W08430, doi:10.1029/2005WR004440.

Models

Sites

Ideal result (0,0)

λE

H


SOLO neural network - cluster analysis

Abramowitz, G., Gupta, H., Pitman, A., Wang, Y.P., Leuning, R. and Cleugh, H.A. (2006). Neural Error Regression Diagnosis (NERD): A tool for model bias identification and prognostic data assimilation. Journal of Hydrometeorology, 7:160-177.


Poor model performance not just due to poor parameter estimation

CABLE with 4 different parameter sets

SOLO – cluster analysis

observedcablesolo


No model or single performance measure is best for all fluxes

NEE of CO2 (µmol/m2/s) Latent heat flux (W/m2) Sensible heat flux (W/m2)

mean (obs)

rmse grad int rsq mean (obs)

rmse grad

int rsq mean (obs)

rmse grad

int rsq

CAB -2.3 (-0.1) 5.48 0.55 -2.2 0.52 49 (52) 55.9 0.71 12.0 0.61 38 (54) 58.3 0.79 -5.0 0.82

ORC -0.5 (-0.1) 5.16 0.47 -0.4 0.50 46 (52) 58.1 0.73 8.1 0.59 14 (54) 118.1 0.16 6.0 0.78

CLM -1.2 (-0.1) 6.13 0.24 -1.6 0.35 26 (52) 72.9 0.38 6.5 0.40 56 (54) 84.5 0.83 10.9 0.64

MLR -1.5 (-0.1) 4.40 0.55 -1.4 0.70 45 (52) 47.6 0.59 14.5 0.74 43 (54) 54.9 0.69 5.2 0.87

T u m b a r u

ANN -1.3 (-0.1) 4.65 0.52 -1.2 0.64 42 (52) 49.9 0.54 14.1 0.74 41 (54) 51.4 0.81 -3.0 0.86

CAB -1.7 (-0.9) 6.13 0.61 -1.2 0.49 43 (47) 64.5 0.89 1.8 0.55 -0.1 (26) 71.8 0.62 -16.1 0.32

ORC -0.4 (-0.9) 9.07 0.68 0.2 0.29 30 (47) 48.0 0.84 -9.4 0.70 3.2 (26) 52.2 0.69 -14.5 0.56

CLM -1.2 (-0.9) 7.92 0.12 -1.1 0.08 33 (47) 50.5 0.63 3.0 0.62 30 (26) 57.9 1.01 4.3 0.59

MLR -0.7 (-0.9) 7.38 0.21 -0.5 0.20 38 (47) 45.8 0.53 13.6 0.73 26 (26) 47.7 1.08 -2.0 0.71

B o n d v i l

ANN 0.0 (-0.9) 7.69 0.20 0.2 0.15 39 (47) 45.7 0.53 14.4 0.72 27 (26) 49.0 1.14 -2.2 0.73

CABLE, ORCHIDEE, CLM,

MLR multiple linear regression,

ANN artificial neural network


Model comparisons - average seasonal cycle

NEE

λE

H

Global default parameters for each PFT used


Model comparisons - average daily cycle

NEE

λE

H

Global default parameters for each PFT used


PDF’s for NEE, λE & H across 6 sites


NEE Perturbed-parameter ensemble simulations

Monthly averages Average diurnal cycle


λE Perturbed-parameter ensemble simulations



H Perturbed-parameter ensemble simulations



Partitioning climate space into 9 SOM nodes

S↓ Tair qair

night

night

night

S↓ Tair qairS↓ Tair qair


NEE PDFs at nodes 7 -9 at Tumbarumba

night

S↓ Tair qair S↓ Tair qair S↓ Tair qair

7 8 9


Suggested set of discussion topics

• Primary objectives• Establish a framework that provides standardised data sets

and an agreed set of analytical tools for LSM evaluation• Analytical tools should provide a wide range of diagnostic

information about LSM performance

• Datasets specifically formatted for LSM execution and evaluation

• Specific objectives• To detect and eliminate systematic biases in several LSMs in

current use• To obtain optimal parameter values for LSMs after biases have

been diminished or eliminated• To evaluate the correlation between key model parameters and

bioclimatic space


Tasks for meeting 1

• Discuss what form the LSM evaluation framework should take• PILPS style?• What will be asked of data providers?• What will be asked of LS modellers?

• Agree on a minimal set of LSM flux performance measures (model vs observations vs benchmark):

• Average diurnal cycle?• Average annual cycle (monthly means)?• Some type of frequency analysis (wavelet, power spectrum etc)?• Conditional analysis (SOM node analysis):

• Overlap of pdfs

• Multiple criteria cost function set (mean, rmse, rsq, regression gradient and intercept)

• Discuss other LSM outputs and datasets useful for process evaluation

• Discuss ways to include parameter uncertainty in LSM evaluation (c.f. Abramowitz et al., 2008)


Tasks for meeting 2

• Discuss options for the most effective way to provide these services

• Will individual groups do benchmarking, evaluation of model states?

• Preference for an automated web-based interface and data server• Automatic processing through a website?• Abramowitz suggests automation of basic LSM performance measure

plots, including benchmarking (as in Abramowitz, 2005). • Uploaded output from LSM runs in ALMA format netcdf could return

standard plots to the user and/or post on website.

• Model detective work and improvement to be done by individual groups


Data analysis will use:

• Several current LSMs• Quality controlled Fluxnet datasets• SOFM (Self-organizing feature maps) analysis

• to classify bioclimatic data into n2 nodes

• to evaluate model biases for each node to help the ‘detective work’ of identifying areas of model weaknesses

• to identify upper-boundary surfaces for stocks of C and N and P in global ecosystems as a function of the n2 climate nodes

• Benchmarking • to compare model predictions at each climate node against

multiple linear regression (MLR) estimates


Tools currently available from Abramowitz

• SOLO (SOFM + MLR) software (Fortran)• LSMs ‘Model Farm’ of Reto Stöckli plus CABLE• CSV to ALMA netcdf conversion routine (Fortran)• Plotting routines in R• Fluxnet database in CSV and netcdf formats

using fluxnet data to evaluate land surface models ray leuning and gab abramowitz 4 – 6 june 2008

Documents

sitesinsert presentation

ehinsert presentation

model states

model structure

footerno model

footerpoor model performance

model bias identification

parameter behavior