uncertainty analysis and model validation
DESCRIPTION
Uncertainty analysis and Model Validation. Final Project. Summary of Results & Conclusions. In a real-world problem we need to establish model specific calibration criteria and define targets including associated error. Calibration Targets. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/1.jpg)
Uncertainty analysis
and
Model Validation
![Page 2: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/2.jpg)
Final Project
Summary of Results &
Conclusions
![Page 3: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/3.jpg)
Calibration Targets
calibration value
associated error
20.24 m
0.80 m
Target with relativelylarge associated error.
Target with smaller associated error.
In a real-world problem we need to establish model specific calibration criteria and define targets including associated error.
![Page 4: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/4.jpg)
Smith Creek Valley (Thomas et al., 1989)
Calibration Objectives
1. Heads within 10 ft of measured heads. Allows forMeasurement error and interpolation error.
2. Absolute mean residual between measured and simulated heads close to zero (0.22 ft) and standard deviation minimal (4.5 ft).
3. Head difference between layers 1&2 within 2 ft of field values.
4. Distribution of ET and ET rates match field estimates.
![Page 5: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/5.jpg)
Also need to identify calibration parameters and theirreasonable ranges.
![Page 6: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/6.jpg)
Calibration Prediction
Group ARM h ARM ET (x10e7)
ARM h (at targets)
ARM h(at pumping wells)
1 0.92 1.38 1.60 4.16
2 0.73 1.11 1.99 3.03
3 0.69 0.51 0.95 1.76
4 1.34 1.27 1.46 2.57
5 1.56
0.89
2.79
1.43
6 1.29
0.16
2.58
2.92
![Page 7: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/7.jpg)
Calibration to Fluxes
When recharge rate (R) is a calibration parameter, calibrating to fluxes can help in estimating K and/or R.
![Page 8: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/8.jpg)
H1H2
q = KI
In this example, flux information helps calibrate K.
![Page 9: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/9.jpg)
In this example, discharge information helps calibrate R.
![Page 10: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/10.jpg)
All water discharges to the playa.Calibration to ET merely fine tunesthe discharge rates within the playaarea.
1 2 3 4 5 6 7 8 9 10 11
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
In our example, total recharge is known/assumed to be 7.14E08 ft3/year and discharge = recharge.
![Page 11: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/11.jpg)
Calibration Prediction
Group ARM h ARM ET (x10e7)
ARM h (at targets)
ARM h(at pumping wells)
1 0.92 1.38 1.60 4.16
2 0.73 1.11 1.99 3.03
3 0.69 0.51 0.95 1.76
4 1.34 1.27 1.46 2.57
5 1.56
0.89
2.79
1.43
6 1.29
0.16
2.58
2.92
![Page 12: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/12.jpg)
00.5
1
1.52
2.53
3.54
4.5
0 0.5 1 1.5 2
Calibrated ARM
Pre
dic
ted
AR
M-t
arg
ets
Includes results from 2000, 2001, 2003
![Page 13: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/13.jpg)
0
2
4
6
8
10
12
0 0.5 1 1.5 2
Calibrated ARM
Pre
dic
ted
AR
M-p
um
pin
g w
ells
Includes results from 2000, 2001, 2003
![Page 14: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/14.jpg)
Group P1 P2 P3 P4 P5 P6 P7
1 2320 PW1 3970 PW2 2310 playa 1920 PW4
1500 PW4 4810 PW2 684 PW2
2 74,000 PW2
39,000 PW2 393,000playa 3.93E6 PW4
252 PW4 1084 playa
1576 playa
3 1.21E6 PW1
2.15E6 PW2
3.90E6 playa 1110 PW4
1.58E6 playa
1860 playa
893 playa
4 1200 PW1 1900 PW2 6.7E6 PW3 290 PW4
2800 PW5
760 PW1 1200 PW2
5 1295 PW1 3160 PW2 503 playa 986 PW4
605 PW4 316 PW1 3100 PW2
6 3100 PW1
982 PW1
4.9E5 playa 603 PW4
1450 PW4
2000 PW1
1380 PW2
Truth 802 PW1 1913 playa
620 playa 310 PW4
1933 PW5
690 playa
2009 PW2
Particle Tracking
![Page 15: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/15.jpg)
Predicted ARM > Calibrated ARM
Predicted ARM at pumping wells > Predicted ARM at nodes with targets
Flow predictions are more robust (consistent among different calibrated models) than transport (particle tracking) predictions.
Observations
![Page 16: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/16.jpg)
Conclusions
• Calibrations are non-unique.
• A good calibration (even if ARM = 0) does not ensure that the model will make good predictions.
• Need for an uncertainty analysis to accompany calibration results and predictions.
• You can never have enough field data.
• Modelers need to maintain a healthy skepticism about their results.
![Page 17: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/17.jpg)
Uncertainty in the Calibration
Involves uncertainty in:
Parameter values
Conceptual model including boundary conditions,zonation, geometry, etc.
Targets
![Page 18: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/18.jpg)
Ways to analyze uncertaintyin the calibration
Sensitivity analysis
Use an inverse model (automated calibration) to quantify uncertainties and optimize the calibration.
![Page 19: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/19.jpg)
Uncertainty in the Prediction
Involves uncertainty in how parameter values(e.g., recharge) will vary in the future.
Reflects uncertainty in the calibration.
![Page 20: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/20.jpg)
Stochastic simulation
Ways to quantify uncertaintyin the prediction
Sensitivity analysis
![Page 21: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/21.jpg)
MADE site – Feehley and Zheng, 2000, WRR 36(9).
![Page 22: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/22.jpg)
![Page 23: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/23.jpg)
A Monte Carlo analysis considers 100 or more realizations.
![Page 24: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/24.jpg)
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7
Drawdown at pumping well
nu
mb
er o
f re
aliz
atio
ns
![Page 25: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/25.jpg)
Stochastic modeling option in GW Vistas
![Page 26: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/26.jpg)
Stochastic simulation
Ways to quantify uncertaintyin the prediction
Sensitivity analysis
Scenario analysis
![Page 27: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/27.jpg)
How do we “validate” a model so thatwe have confidence that it will makeaccurate predictions?
![Page 28: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/28.jpg)
Modeling Chronology
1960’s Flow models are great!
1970’s Contaminant transport models are great!
1975 What about uncertainty of flow models?
1980s Contaminant transport models don’t work. (because of failure to account for heterogeneity)
1990s Are models reliable? Concerns overreliability in predictions arose over efforts to modela geologic repository for high level radioactive waste.
![Page 29: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/29.jpg)
“The objective of model validation is to determine how well the mathematical representation of the processes describes the actual system behavior in terms of the degree of correlation between model calculations and actual measured data”(NRC, 1990)
![Page 30: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/30.jpg)
Oreskes et al. (1994): paper in Science
Calibration = forced empirical adequacy
Verification = assertion of truth (possible in a closed system, e.g., testing of codes)
Validation = establishment of legitimacy (does not contain obvious errors), confirmation, confidence building
What constitutes validation? (code vs. model)
NRC study (1990): Model validation is not possible.
![Page 31: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/31.jpg)
How to build confidence in a model
Calibration (history matching) steady-state calibration(s) transient calibration
“Verification” requires an independent set of field data
Post-Audit: requires waiting for prediction to occur
Models as interactive management tools
![Page 32: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/32.jpg)
HAPPY MODELING!
![Page 33: Uncertainty analysis and Model Validation](https://reader030.vdocuments.site/reader030/viewer/2022032804/56812a48550346895d8d8c97/html5/thumbnails/33.jpg)