calibration guidelines
DESCRIPTION
Calibration Guidelines. Model development. Model testing. 9. Evaluate model fit 10. Evaluate optimal parameter values 11. Identify new data to improve parameter estimates 12. Identify new data to improve predictions 13. Use deterministic methods 14. Use statistical methods. - PowerPoint PPT PresentationTRANSCRIPT
Calibration Guidelines
1. Start simple, add complexity carefully2. Use a broad range of information3. Be well-posed & be comprehensive4. Include diverse observation data for
‘best fit’5. Use prior information carefully6. Assign weights that reflect ‘observation’
error7. Encourage convergence by making the
model more accurate 8. Consider alternative models
9. Evaluate model fit 10. Evaluate optimal
parameter values
11.11. Identify new data to improve parameter estimates
12.12. Identify new data to improve predictions
13. Use deterministic methods14. Use statistical methods
Model development Model testing
Potential new dataPotential new data
Prediction uncertainty
Guideline 11: Identify New Data to Improve Model Parameter Estimates and Evaluate Potential for
Additional Estimated ParametersHere we consider measurements related to observations because the
connection between hydrologic and hydrogeologic data is direct and needs no special statistics.
Hydrologic and hydrogeologic dataRelate to model inputs
Dependent variable ObservationsRelate to model outputs
Ground-Water Model -- Parameters
Predictions
Prediction uncertainty
Societal decisions
Potential new observations to improveparameter estimates
• Goal: evaluate worth of the type and location of potential observations. No observed value yet, so need statistics that don’t depend on this value. Use fit-independent statistics
• In the context of reducing parameter uncertainty and increasing uniqueness, use the statistics:A. Dimensionless scaled sensitivities. B. One-percent scaled sensitivities. Often plotted in map form.C. Parameter correlation coefficientsD. Leverage statisticsE. Influence statistics (not discussed here)
A. Dimensionless scaled sensitivities (dss): • Can be calculated for any potential observation type or location• Account for the expected accuracy of the measurements because dss
include the observation weight. • Larger values identify observations that are likely to reduce parameter
uncertainty.
B. Maps of one-percent scaled sensitivities
• Conveniently shows spatial relations.
• One map for each parameter, for each model layer, for each time step: potentially a huge number of maps!.
• Inconvenient because can not determine the effect on the entire set of parameters.
• Does not reflect the expected accuracy of observations in the different locations.
Parameter pair for each correlation coefficient
0.5
0.6
0.7
0.8
0.9
1.0
1 (Heads) 2 (+Flow) 3 (+ADV)
Observation set
Abs
olut
e va
lue
of th
e co
rrel
atio
n co
effi
cien
t
K-RCH K-Qn
K-Qb K-GHB
RCH-Qn RCH-Qb
RCH-GHB Qn-Qb
Qn-GHB Qb-GHB
• Clearly illustrates that collecting flow and advective transport data can radically reduce the extreme parameter correlations that occur when only head are used.
Example from Cape Cod (Anderman and others
1996; Anderman and Hill 2001)
C. Parameter correlation coefficientsCalculate without and with the potential new data. If correlations with absolute values close to 1.00 become smaller, the new data will help attain unique parameter estimates.
D. Leverage StatisticsIndicate potential effect of an observation on a set of parameter estimates. Do not indicate the particular parameter(s) to which an observation is important
Dimensionless scaled sensitivities (dss) for each parameter. Leverage
HK_2 VK_CB K_RB RCH_2
Potential head observation
-3.5 8.010-3 -0.105 54.8 0.988
Potential flow observation
-3.210-
5 1.110-6 -0.3510-
5 -4.50 0.491
css for existing observations
3.1 0.22 0.20 25.3
Table 13.1, p. 331
0
2
4
6
8
10
12
K1
K2
K3
K4
K5
K6
(el)
K7
(Nw
flt)
K8
(dr)
K9
(fm
tn)
AN
IV1
AN
IV3
RC
H0
RC
H1
RC
H2
RC
H3
ET
M
GH
Ba
m
GH
Bg
s
GH
Bo
GH
Bfc
GH
Bt Q Q
Po
ros
Parameter label
Com
posi
te s
cale
d se
nsiti
vity
0
50
100
150
200
250
K1
K2
K3
K4
AN
IV3
AN
IV1
RC
H
ET
M
P arameter names
Com
posi
te s
cale
d se
nsit
ivit
y
First model
Final model
What parameters could be supported in more detail, given the information in the observations? Use css
Evaluation of possible additional estimated parameters
Calibration Guidelines
1. Start simple, add complexity carefully2. Use a broad range of information3. Be well-posed & be comprehensive4. Include diverse observation data for
‘best fit’5. Use prior information carefully6. Assign weights that reflect ‘observation’
error7. Encourage convergence by making the
model more accurate 8. Consider alternative models
9. Evaluate model fit 10. Evaluate optimal
parameter values
11.11. Identify new data to improve parameter estimates
12.12. Identify new data to improve predictions
13. Use deterministic methods14. Use statistical methods
Model development Model testing
Potential new dataPotential new data
Prediction uncertainty
Guideline 12: Identify New Data to Improve Model Predictions
Two categories of potential new data: Measurements related to observations and hydrology and hydrogeology
Hydrologic and hydrogeologic dataRelate to model inputs
Dependent variable ObservationsRelate to model outputs
Ground-Water Model -- Parameters
Predictions
Prediction uncertainty
Societal decisions
Potential new data to improve predictions
– What existing or new observations are important to predictions?• Observation-Prediction Statistic (opr)
– Which parameters are important to predictions? Infer important hydrologic and hydrogeologic data• Prediction scaled sensitivities (pss) with
composite scaled sensitivies (css) and parameter correlation coefficients (pcc)
• Parameter-Prediction Statistic (ppr)
Potential new observations to improve predictions
Hydrologic and hydrogeologic dataRelate to model inputs
Dependent variable ObservationsRelate to model outputs
Ground-Water Model -- Parameters
Predictions
Prediction uncertainty
Societal decisions
Approach: OPR
Potential new observations to improve predictions
Observation-Prediction (opr) Statistic
• Which existing observations are important to predictions?
– opr indicates the percent increase in prediction uncertainty caused by omitting an existing observation
• What new observations would be most valuable to predictions?
– opr indicates the percent decrease in prediction uncertainty caused by adding a new observation
• Advantages:– Combines dssdss, csscss,, and psspss into one statistic that also accounts
for parameter correlation– is independent of model fit – is computationally manageable
Predictions of Interest in the Death Valley Model
• Resource managers are interested in long-term, regional transport from selected sites, including all processes – advection, dispersion, reactions, adsorption and desorption
• The regional model can be used to address advection.
• Advective transport considered -- consistent with regional-scale model
• Track movement in 3 coordinate directions – here, north-south, east-west, and vertical
Which existing observations are important(or not) to predictions?
Use opr(-1) to rank the 501 existing observation locations by their importance to predictions
• Averaged values of opr(-1) for all the predictions are used, to obtain a measure indicating the importance of a single observation to all the predictions of interest.
• Calculate opr(-100) by removing the 100 least important observations
• opr(-100) = mean prediction uncertainty increase = 0.6%
Consider potential new head observations in layer 1.
Calculate opr(+1) for each cell in the layer.
What new observations would be important(or not) to predictions?
Potential new data to improve predictions
– What existing or new observations are important to predictions?• Observation-Prediction Statistic (opr)
– Which parameters are important to predictions? Infer important hydrologic and hydrogeologic data• Prediction scaled sensitivities (pss) with
composite scaled sensitivies (css) and parameter correlation coefficients (pcc)
• Parameter-Prediction Statistic (ppr)
Potential new system property information to improve predictions
Two approaches:
A. Prediction scaled sensitivities (pss) together with composite scaled sensitivities (css) and parameter correlations (pcc)
B. Parameter-prediction (ppr) statistic
Hydrologic and hydrogeologic dataRelate to model inputs
Dependent variable ObservationsRelate to model outputs
Ground-Water Model -- Parameters
Predictions
Prediction uncertainty
Societal decisions
A. Prediction Scaled Sensitivities (pss): What parameters are important to predictions?
Vertical advective-transport distance
0.0001
0.001
0.01
0.1
1
10
100
1000
K1 K2 K3 K4 K5
K6(el)
K7(Nwflt)
K8(dr)
K9(fmtn)
ANIV1
ANIV3
RCH0
RCH1
RCH2
RCH3
ETM
GHBam
GHBgs
GHBo
GHBfc
GHBt Q1
Q2
POROS
Parameter label
Per
cen
t ch
ang
e
pss-desired model
complexity
Here, pss are scaled to equal percent change in prediction
caused by 1% change in parameter value
0
2
4
6
8
10
12
K1
K2
K3
K4
K5
K6(
el)
K7(
Nw
flt)
K8(
dr)
K9(
fmtn
)
AN
IV1
AN
IV3
RC
H0
RC
H1
RC
H2
RC
H3
ET
M
GH
Bam
GH
Bgs
GH
Bo
GH
Bfc
GH
Bt Q Q
Por
os
Parameter label
Com
posi
te s
cale
d se
nsiti
vity
Vertical advective-transport distance
0.0001
0.001
0.01
0.1
1
10
100
1000
K1 K2 K3 K4 K5
K6(el)
K7(Nwflt)
K8(dr)
K9(fmtn)
ANIV1
ANIV3
RCH0
RCH1
RCH2
RCH3
ETM
GHBam
GHBgs
GHBo
GHBfc
GHBt Q1
Q2
POROS
Parameter label
Per
cen
t ch
ang
ecss – css –
supported supported model model
complexity complexity
pss – pss – desired desired model model
complexity complexity
B. Parameter-Prediction (ppr) Statistic
• Which parameters are important to predictions?
– ppr indicates percent decrease in prediction uncertainty caused by a decrease in parameter uncertainty.
• The decrease in parameter uncertainty is implemented by increasing the weight on prior information for the parameter(s).
• This increase in weight represents the increased certainty that would result from collection of additional field data about the parameter or associated system property.
Predictions of Interest in the Death Valley Model(simulations from the 3-layer model of D’Agnese +, 1998)
Apply ppr statistic to
one prediction on
Yucca Flat
East-WestR4K2
K1K3
0
5
Vertical R1
0
6
12
North-SouthR4K2
K1
K3
0
5
Parameter with Improved Information
Hydraulic Conductivity Recharge
Which individual parameters (and associated system features) would be most beneficial to further
characterize in the field?
Ppr statistic(percent
decrease in prediction
uncertainty)
A. Prediction scaled sensitivities (pss) together with composite scaled sensitivities (css) and parameter correlations (pcc)
– PRO: pss, css, and pcc are each conceptually easy to understand and convey to others
– PRO: independent of model fit and computationally manageable– CON: Can be cumbersome to evaluate the three measures to
determine the value of new system property data
B. Parameter-prediction (ppr) statistic – PRO: Combines csscss, pss, pss, andand pcc pcc into one statistic– PRO: independent of model fit and computationally manageable– CON: More conceptually difficult to understand and explain to
others. Best so far -- express in terms of percent changes in prediction uncertainty
Pros and cons of A (pss+) and B (ppr)