two models of the gulf stream - oceancolor.gsfc.nasa.gov€¦ · what is truth? data model ε d ε...
TRANSCRIPT
![Page 1: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/1.jpg)
Two models of the Gulf Stream
Stommel, 1948
0=!
0!"
![Page 2: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/2.jpg)
A third model of the Gulf StreamA third model of the Gulf Stream
Simulation courtesy ofMat Maltrud, Los Alamos National Laboratory
Temperature at 100m
Simulation duration: 1 year
![Page 3: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/3.jpg)
An eddy-resolving model of the North AtlanticAn eddy-resolving model of the North Atlantic
Temperature log (New Production)
![Page 4: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/4.jpg)
Skill Assessment for Coupled Biological/Physical Models of Marine Systems
Lynch, McGillicuddy, Werner, Haidvogel
Sponsor: NOAA
Goals: 1. To assess the state-of-the-art in quantitative evaluation of coupled physical-biological models (journal issue)2. Provide recommendations for future progress in this area in support of NOAA’s Ecosystem Based Management and Ecological Forecasting initiatives.
http://www-nml.dartmouth.edu/Publications/internal_reports/NML-06-Skill/
![Page 5: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/5.jpg)
Skill Assessment Workshop Attendees
Icarus Allen
Enrique Curchister
Brad de Young
Scott Doney
Geoff Evans
Wolfgang Fennel
Peter Franks
Marjorie Friedrichs
Watson Gregg
Dale Haidvogel
Jason Jolliff
John Kindle
Ivan Lima
Daniel Lynch
Dennis McGillicuddy
Roger Proctor
Allan Robinson
Kenny Rose
Don Scavia
Rainer Schlitzer
Peter Sheng
Keston Smith
Dougie Speirs
John Steele
Charles Stock
Craig Stow
Keith Thompson
Shelly Tomlinson
Elizabeth Turner
Phil Wallhead
Francisco Werner
![Page 6: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/6.jpg)
Timeline
July ‘06 Authors' Workshop 1Vocabulary Rev. 1Working Groups: DA, Metrics
Dec ‘06 Working Group Reports to Editors
Feb ‘07 Vocabulary Rev. 2Working Group Report Distribution
March '07 Authors’ Workshop 2
April ‘07 MS Submission; Peer Review Starts
April ‘08 Publication in Journal of Marine SystemsReport goes to NOAA
![Page 7: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/7.jpg)
OrganizationScientific Applications
Carbon CycleHarmful Algal BloomsEcosystem Dynamics and FisheriesEstuarine/Coastal Water Quality
Cross -Cutting ThemesSkill VocabularyMetricsData Assimilation
![Page 8: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/8.jpg)
What is Truth?
Data Model
εd εm
Misfit δ
Predictionεp
Skill:Misfits
Small, Noisy
Deduced InputsSmall, Smooth
Processes, FeaturesRealistic
Truth real but unknowable
Errors unknowable
Prediction a credible blend:
Data + Model
Invokes statistics of εd , εm
Prediction Error: blend of εd , εm
Misfit: δ = εd - εm
Truth
![Page 9: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/9.jpg)
RationaleSystematic model evaluation requires a hierarchy of performance metrics.
DefinitionsBiasRMS differenceCentered RMS difference (“pattern similarity”)Correlation coefficientCoherenceModel Efficiency (gamma squared - 1)
Misfit Metrics
![Page 10: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/10.jpg)
Taylor Diagram
Azimuthal position: correlationRadial distance: standard deviation, normalized by std dev of dataPerfect model: (1,0)Mean model: (0,0)Centered RMS difference proportional to distance from (1,0)
Doney and Lima
![Page 11: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/11.jpg)
Quantifies skill as a function of threshold using a binary discriminator
Temperature Chlorophyll
Sensitivity=fraction of true positives Specificity=(1 – fraction of true negatives)
In sensitivity vs. specificity spaceBelow data range (1,1) all TP, no TNAbove data range (0,0) no TP, all TNPerfect model (0,1) all TP, TNRandom number generator 1:1 line
Appealing characteristics for HAB and water quality applications with critical thresholds such as dissolved oxygen and toxin concentration.
Receiver-Operator Characteristic (ROC)
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
1 - Specificity
Sen
sit
ivit
y
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
1 - Specificity
Sen
sit
ivit
y
D MTP + +TN - -FP - +FN + -
Icarus Allen
![Page 12: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/12.jpg)
Problem: How does one assess theskill of models in which data have
been assimilated?
![Page 13: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/13.jpg)
Fig. 4.2. Assimilation model chlorophyll (mg m-3), SeaWiFS mean chlorophyll, and the difference (Assimilation-SeaWiFS, inchlorophyll units) for March 2001. From Gregg (2007).
![Page 14: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/14.jpg)
Annual Error (Bias and Uncertainty)
Assimilation vs. SeaWiFS Chlorophyll
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 15 20 25 30 40 50 60 90 183
Assimilation Frequency (days)
Err
or
(%)
Free-Run Model Uncertainty (65.3%)
Free-Run Model Bias (21.0%)
Uncertainty
Bias
Figure 4.3. Annual bias and uncertainty for assimilation as a function of assimilation frequency (days of assimilationevents, i.e., 1 is every day, 2 is every other day, etc.) assimilation is performed). The annual bias and uncertaintyfor the free-run model is shown. From Gregg (2007)
![Page 15: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/15.jpg)
Problem: More complex modelscontain more degrees of freedom.
How do we determine whether a morecomplex model has statistically moresignificant explanatory power than a
simpler one?
![Page 16: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/16.jpg)
Hindcast Simulations in the Western Gulf of Maine
Model: ECOM 3-DMY 2.5 Closure
Forcing: Wind, Heat Fluxes,Tides,River discharge
Stock et al.(2005)
![Page 17: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/17.jpg)
4/13 4/29 5/11 5/25 6/5
1993Obs
BestModel
log10(C)
log10(C)
![Page 18: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/18.jpg)
)1ln()1ln( mod,, +!+=iiobsi
cc"
)det(2
)2
1exp(
);....(2/
1
1
!!
!!
"
!!
!##C
C
LM
T
n
$%$
=
Model-data misfit of concentration (c)
Likelihood function for a model with parameters θn:
Maximum likelihood ratio test: null hypothesis (L0 vs. alternative L1)
1,..,...1
,...1
);(
);(
L
L
L
Ll
o
mn
n
==!"""
!""
Likelihood ratio (l) has chi-square distribution with m-n degrees of freedom
Maximum Likelihood Methodology
Stock et al., 2005
![Page 19: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/19.jpg)
Confidence limits setbased on properties ofmaximum likelihoodparameter estimates
Reject nullhypothesis:mortality≠0
Models with and without mortality
90%99%
![Page 20: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/20.jpg)
Models with and withoutnutrient dependence
Reject nullhypothesis: KN ≠0
90%99%
KN = half saturationconstant for nutrientuptake
![Page 21: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/21.jpg)
Nutrient Dependence or Mortality?
•Reject baseline formort. + DIN
•Cannot determineif loss limitation isbest imposed bymean mortality,DIN or somecombination.
• Avoidederroneousrejection!
99%
90%
![Page 22: Two models of the Gulf Stream - oceancolor.gsfc.nasa.gov€¦ · What is Truth? Data Model ε d ε m Misfit δ Prediction ε p Skill: Misfits Small, Noisy Deduced Inputs Small,](https://reader034.vdocuments.site/reader034/viewer/2022050323/5f7cf4ec3b2572106a6ce929/html5/thumbnails/22.jpg)
SummaryNeed to move beyond qualitative phenomenological evaluation
science (hypothesis testing)management (prediction)
Methods for quantitative skill assessment of coupled models are in theirinfancy
Special volume underwayfirst ms submitted April, 2007additional submissions welcome – cutoff summer 2007publication in Journal of Marine Systems – spring 2008
Potential interagency partnerships?
Development of an implementation plan for Model Intercomparison andEvaluation Projects (MIEPs)?
http://www-nml.dartmouth.edu/Publications/internal_reports/NML-06-Skill/