Model Calibration and Sensitivity Analysis
A Framework for Model Applications 1
Calibration Criteria
1. Mean Error (ME)
( )MEn
cal obsi ii
n
= −=∑1
1
2. Mean Absolute Error (MAE)
MAEn
cal obsi ii
n
= −=∑1
1
3. Root Mean Squared (RMS)
( )RMSn
cal obsi ii
n
= −
=∑1 2
1
1 2
4. Correlation Coefficient (ϒ)
( )( )
( ) ( )γ =
− −
− −
=
= =
∑
∑ ∑
cal cal obs obs
cal cal obs obs
i ii
n
ii
n
ii
n1
2
1
2
1
A Framework for Model Applications 2
Presentation of Calibration Results
• qualitative comparison of calculated and observed contour maps
• tabulated results with summary statistics
• scatter diagrams • comparison of observed
and calculated breakthrough curves
• spatial distribution of residual errors
• comparison of observed and calculated mass distributions
A Framework for Model Applications 3
Observed concentrations superimposed on calculated contours
A Framework for Model Applications 4
A Framework for Model Applications 5
61 62 63 64 65
Observed Head (m-amsl)
61
62
63
64
65
Pre
dict
ed H
ead
(m-a
msl
)
A Framework for Model Applications 6
Comparison of calculated and observed discharge from 4 pumping wells
A Framework for Model Applications 7
Scatter Diagram
Comparison of breakthrough curves
A Framework for Model Applications 8
NATS Site, Columbus, MS
(Julian et al., 2001)
-30 -20 -10 0 10 20x (m)
-10
0
10
20
30
40
y (m
)
Source Trench
lnK (cm/s)
-9 -7 -5 -3 -1
-100 -50 0 50 100x (m)
-50
0
50
100
150
200
250
300
y (m
)
MLS Boundary
SourceTrench Groundwater
Flow
Abandoned Meander
K (cm/s)
Source Trench -5 -4 -3 -2 -110 10 10 10 10
-50 0 50 100 150 200 250 300y (m)
62
60
58
56
54
z (m
-am
sl)
A Framework for Model Applications 9
-10
0
10
20
30
40
50
y (m
)
-30 -20 -10 0 10 20x (m)
-10
0
10
20
30
40
50
y (m
)
-30 -20 -10 0 10 20x (m)
SourceTrench
MLS
(a)
(b)
Observed Predicted
Bromide(ppm)0 1 2 5 10 20
Observed and Calculated Bromide Concentrations
A Framework for Model Applications 10
Scatter diagram of observed versus calculated bromide peak concentrations. Snapshots 3 and 4 correspond to observation times of 152 and 278 days after source emplacement.
0
2
4
6
8
10
12
14
0 2 4 6 8 10 12 14
Observed Concentration (ppm)
Cal
cula
ted
Con
cent
ratio
n (p
pm)
Snapshot 3
Snapshot 4
A Framework for Model Applications 11
Methods for Model Calibration
1 by trial-and-error procedures a) select one or more calibration criteria b) adjust one input parameter at a time c) compare values of calibration criteria
2 by “automated” procedures a) parameter values b) parameter structures c) available codes
3 points to ponder a) perform transient calibration, if possible at all b) calibrate flow rates, if possible c) much can be gained from calibration against transport data
A Framework for Model Applications 12
Model Calibration as an Optimization Problem
( )Minimize objective function
subject to one or more specified constraints
S obs cali i i= −∑ω2
Example of simple objective function
A Framework for Model Applications 13
Example of more complex objective function with multiple local optima
A Framework for Model Applications 14
Illustrative Example
0 2000 4000 6000
X Axis (m)
0
2000
4000
6000
Y A
xis
(m)
No-flow boundary
No-flow
boundary
Specified-flow
boundary (Q=0.25 m
/day)
Constant head boundary (h=100 m)
Zone 1
Zone 2
Zone 3
Observation wellPumping well
P1
P2
P3
Figure 4-1. Configuration of the two-dimensional test problem. The solid dots and open circles indicate the locations of pumping and monitoring wells where hydraulic heads and/or concentrations are used to estimate hydraulic conductivity in the three zones.
A Framework for Model Applications 15
0 2000 4000 6000
X Axis (m)
0
2000
4000
6000
Y A
xis
(m)
Zone 1
Zone 2
Zone 3
0
20
40
60
80
100
0 200 400 600 800 1000
Time (days)
Con
cent
ratio
n (p
pm)
P1 P2 P3
True head distribution
True concentration data at 3 wells
A Framework for Model Applications 16
Objective Functions for Parameter Estimation
Head data only
( ) 2
1
ˆ NH
i i ii
Minimize S h hα=
= − ∑
Both head and concentration Data
( ) ( )2 2
1 1
ˆ ˆ NH NC
i i i j j ji j
Minimize S h h C Cα β= =
= − + − ∑ ∑
K1 (m/d) K2 K3 Obj. Func. # Generations
True 150 50 200
H only 150 50 200 0 28
H & C 150 50 200 0 20
A Framework for Model Applications 17
Sensitivity Analysis
• sensitivity coefficient
measure of the effect of change in one factor on another factor
X ya
yai k
i
k
i
k,
$ $= ≈∂∂
∆∆
normalized with respect to ak,
X ya a
ya ai k
i
k k
i
k k,
$ $= ≈∂
∂∆
∆
dimensionless,
X y ya a
y ya ai k
i i
k k
i i
k k,
$ $ $ $= ≈∂∂
∆∆
A Framework for Model Applications 18
Procedure for Sensitivity Analysis
• base case: calibrated model
• change a parameter by a certain percentage from the base case (∆a/a)
• run the model again
• calculate the change in model response (∆y) which could be any variable of interest, such as head, flow rate, concentration at a receptor, etc.
• calculate sensitivity coefficient and evaluate the results
A Framework for Model Applications 19
Examples of Sensitivity Analysis
A Framework for Model Applications 20
A Framework for Model Applications 21
A Framework for Model Applications 22
Observed plumes of BTEX and electron acceptors at the Hill Air Force Base (Lu et al., 1999)
Results of sensitivity analyses for total BTEX mass and BTEX plume front (Lu et al., 1999)
Degradation rate constants Sensitivity Coefficient K Recharge Long.
Disp. Retard. Factor
DO Nitrate Ferrous iron
Sulfate methane
Total BTEX mass
-0.202 -0.047 -0.036 -0.058 -0.031 -0.047 -0.084 -0.111 -0.054
Front of BTEX plumea
+0.563 -0.088 -0.419 -0.077 -0.331 -0.287 -0.188 -0.265 -0.003
a Sensitivity analysis performed on 0.05 mg/l contour line of BTEX plume
10.00
1.005.00
0.05
2.00 0.50
0.005
60.0 40.0
60.02.0
40.020.0
60.0
N
0.05
1.008.00100 m
300 ft
(a) BTEX
4.00
2.001.00
(b) Oxygen
5.001.00
0.05
(c) Nitrate (f) Methane
(e) Sulfate
(d) Ferrous iron
A Framework for Model Applications 23
Prediction and Uncertainty
• simulation of future conditions evaluation/assessment vs. prediction
• sources of uncertainty
⇒conceptual related to the mathematical model
⇒geological harder to quantify
⇒stress and other conventional parameters targets of most uncertainty analysis studies
A Framework for Model Applications 24
Methods for Uncertainty Analysis
• sensitivity analysis ⇒simple, straightforward, but cannot examine correlation between parameters
• first-order error analysis ⇒in the simplest form
[ ] [ ][ ]Var y Var x y xii
N
i==∑
1
2∂ ∂
⇒approximation
• Monte Carlo simulation ⇒most general, but computationally intensive
A Framework for Model Applications 25
A Framework for Model Applications 26
A Framework for Model Applications 27
Procedure for Monte Carlo Analysis
A Framework for Model Applications 28
Example of Monte Carlo Analysis Woldt et al. (1992)
A Framework for Model Applications 29
Only K as random variable
Both K and Initial Plume as random variables
Only Initial Plume as random variable