![Page 1: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/1.jpg)
Development, implementation, and application of an improved model performance evaluation and diagnostics approach
Byeong-Uk Kim*, William Vizuete, and Harvey E. Jeffries
Department of Environmental Sciences & Engineering, University of North Carolina at Chapel Hill
*Georgia Department of Natural Resources
5th Annual CMAS ConferenceOctober 17, 2006
![Page 2: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/2.jpg)
Typical SIP modelingBase case
Future case
Future control case
Base case emissions
Proposed controls
Preset controls
Base case meteorology
Future controlcase emissions
Future projected emissions
Preset controls
= +
More controls until passing attainment demonstration
Model Performance Evaluation
If attainmentdemonstrationfailed
Model performance evaluation (MPE) is the process for assessing the “reliability of model predictions.”
Future caseemissions
![Page 3: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/3.jpg)
Issues with typical MPE practice
MPE for only (if not, mostly) ozone signals No systematic evaluation about if models get right answers
for likely right reasons No evaluation of winds with respect to chemical signals
“Waterfall” procedures and no explicit consideration of the impact of model performance on policy choices
No further MPE for model inputs/outputs with respect to proposed policy options once a MPE is done by following the EPA guidance literally
Probable diagnostic evaluation after many ad hoc analyses
Over-dependence on statistical tests No acceptance for partially useful modeling results No systematic analysis for graphical measures
Needs for investigation of possible causes of poor performance
![Page 4: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/4.jpg)
The expected outcomes of MPE
Is the formulation of a model scientifically acceptable in general? (i.e. what is the adequacy and quality of model formulation for this use?) Concerning if models simulate general causes
Does a model replicate the observations adequately? (i.e. does it make predictions that match history?) Examining if models get right answers for right reasons
Is a model usable for answering specific (e.g. policy) questions? (i.e. does the model fulfill the designed task?) Assessing if models are usable for target purposes
Modified from the original questions in Beck, 2002
![Page 5: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/5.jpg)
Protocol for Regulatory Ozone Modeling Performance Tests (PROMPT)
PROMPT is a meta-protocol; it is a protocol for protocols.Four phases of evaluation procedures
Does this model show or have all necessary components to produce the phenomena that I can expect from the current best perceptual/conceptual model? (Evaluation Phase One)
Can this model distinguish which precursor(s) to control for ozone reduction? (Evaluation Phase Two)
How precisely can the model estimate control requirements? (Evaluation Phase Three)
What are the possible biases in the prediction and the impact of biases on the policy choice? (Evaluation Phase Four)
Performance measures will be examined in a “progressive” manner.
In later evaluation phases, more information will be investigated than earlier phases of evaluation.
PROMPT emphasizes “day-by-day” and “site-by-site” performance analyses and requires evaluators to examine meteorological inputs, ozone, NOx and VOCs as well as geographical features.
![Page 6: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/6.jpg)
Importance of consideration of control options in MPE
Given two winds, A and B, control options for R can be evaluated if the target emission source is the grey area.
Assuming the emission intensity in the grey area is homogeneous in time and space
![Page 7: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/7.jpg)
Illustration of PROMPT application
Houston-Galveston-Brazoria 8-hour Ozone SIP Modeling (“base1b” was used for this example although “base1c” is the newest case)
Modeling period: 2000-08-16 ~ 2000-09-06 Extensive observational data available through TexAQS 2000
campaign Almost same period of Houston-Galveston Mid-Course Review (MCR)
modeling for the 1-hour ozone (“base5b”) In general, this episode shows a very Houston-specific ozone problem;
Transient High Ozone Events (THOEs) that are often characterized by hourly ozone concentration changes more or equal than 40 ppb.
THOEs are often caused by epidemic highly reactive volatile organic compounds (HRVOCs) emission events under ozone-conducive conditions.
No official HRVOC emission event record available for 2000 The possible existence of event emissions in 2000 can be inferred
from a study conducted by UT researchers.
![Page 8: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/8.jpg)
Transient High Ozone Event in Houston
>10,000 lbs/hr ethylene release at La Porte, (6700 lbs between 11:00 AM and 11:25 AM) 3/27/2002
![Page 9: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/9.jpg)
Modeling domain
36 km 12 km 4 km 1 km 1 km
Galveston Bay
Ship Channel
36 km
![Page 10: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/10.jpg)
Modeling with improved model inputs
Issues in base5b (1-hour MCR): Poor surface wind predictions PBL height and vertical mixing Over-predictions in NOx, CO, and HRVOCs at surface
monitors Insufficient ozone formation
Major changes in inputs Meteorology, Emissions, Chemistry, Boundary conditions Yet, questionable 4-km grid resolution
Does the set of new inputs make base1b (8-Does the set of new inputs make base1b (8-hour SIP) more “useful” for assisting hour SIP) more “useful” for assisting decision makers in choosing control options decision makers in choosing control options between NOx control and VOCs control (or between NOx control and VOCs control (or both)?both)?
![Page 11: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/11.jpg)
RegEvnt1 base5b with CMAQ
base5b (1km) base5b
![Page 12: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/12.jpg)
Older 1-h model Newer 8-h model
![Page 13: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/13.jpg)
Base1b Vertical Kv Profile
0
100
200
300
400
500
600
700
800
900
1000
0 50 100 150 200 250 300 350 400
Kv
CA
Mx L
ayer
(m)
(020:029) (021:029) (020:030) (021:030)
Base5b Vertical Kv Profile
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
0 50 100 150 200 250 300 350 400
Kv
CA
Mx L
ayer
(m)
(020:029) (021:029) (020:030) (021:030)
Base1b (8-hr SIP)
Base5b (MCR)Clinton (C35C)
![Page 14: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/14.jpg)
NOx emissions
Bush Airport Only 6AM~5PMDecrease ~4 ppb/h downtown/west Houston; increase perimeter counties
![Page 15: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/15.jpg)
CO emissions
Decrease ~50 ppb in downtown/west Houston
![Page 16: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/16.jpg)
ETH emissions
Only 7AM on 25, 29, and 31Some increases in downtown/west Houston
![Page 17: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/17.jpg)
NO2 Barchart and time series
![Page 18: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/18.jpg)
Drops to model boundary condition at night.
Level 4 in modelCO timeseries
![Page 19: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/19.jpg)
ISOP timeseries
![Page 20: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/20.jpg)
ETH timeseries
![Page 21: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/21.jpg)
O3 peak
8/25
![Page 22: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/22.jpg)
Aloft NO2 and CO
base1b
base5b
![Page 23: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/23.jpg)
Summary of Process Analysis
Overall, OH distribution of reaction with NO2, CO, CH4 ranges 41%~ 48%; very similar new OH radical source strength across HG domain
This is somewhat low compared to other PA results in other areas.
A significant portion of the total OH reaction (=new OH x chain length) is with NO2, CO, CH4, and other non-NO oxidizing paths. The absolutely maximum amount of O3 that can be formed at the four sites ranged from 127 ppb to 150 ppb minus the emitted NO which ranged from 22 to 123 ppb, thus limiting chemical ozone to values between 36 and 103 ppb of ozone.Thus, the chemical production of O3 is inversely proportional to the NOx at these four sites.PAN is predicted to be very low at these sites, so is RNO3.
![Page 24: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/24.jpg)
•What are the implications from What are the implications from insufficient radical source?insufficient radical source?
–The deficient radical sources result in insensitivity to VOC precursors and inhibited due to elevated levels of NOx.–With current model configuration, VOC controls will have little to no effect in future control strategies.
Major SIP-related question
![Page 25: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/25.jpg)
AcknowledgementThe Houston Advanced Research Center and the 8-hour ozone Coalition Group
Texas Commission on Environmental Quality: Dr. Jim Smith for base1b and base5b files
University of Houston: Dr. Daewon Byun and Dr. Soontae Kim for Q20 files
![Page 26: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/26.jpg)
Missing FORM?
Observational Evidence
![Page 27: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/27.jpg)
Two potential sources of HCHOFlares
98-99% combustion assumed, 1% to 2% emitted VOC composition is assumed same as that fed to flare; rest assumed to be CO2. We assumed that HCHO emitted was equal to VOC emitted.
“Flare case” - Assumed that VOCs fed to flares were partially converted to HCHO and that an amount equal to another 1% was emitted as HCHO. This added a total of 55, 58, and 59 tons on 25th, 30th and 31st. to 13 flares located mainly in the eastern part of Houston
Mobile sources New data (SWRI, 2005) on Heavy Duty Diesel show that HCHO is 23%
of VOC and ethene is 18% of THC. HCHO was 5% of CO. We added HCHO at 4% of low level CO
“Mobile case” - Based on AC obs, assumed that MV emissions did not have enough HCHO. An appropriate factor appeared to be 4% of CO. This added 167, 156, and 145 tons on 25, 30, and 31.
![Page 28: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/28.jpg)
DeltaOzone,ppbO8/2513-16hFlare case
![Page 29: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/29.jpg)
DeltaOzone,ppbO8/2509-12hMobile case
![Page 30: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/30.jpg)
SummaryFlare imputation caused >30 ppb increase in ozone concentrationsCO ratio caused >18 ppb increase ozone concentrations, more distributedIncreased peak ozone at almost every monitor causing 4 monitors to match observations~20% increase in new OH and ~30% in ozone productionStill did not match observed HCHO.
![Page 31: Development, implementation, and application of an improved model performance evaluation and diagnostics approach Byeong-Uk Kim *, William Vizuete, and](https://reader030.vdocuments.site/reader030/viewer/2022032709/56649eb65503460f94bbfc6b/html5/thumbnails/31.jpg)
OH + (VOCs + CO + CH4 + NO2)
NO225%
CO19%
FORM14%
ALD210%
MGLY1%
OPEN0%
CH44%
PAR8%
MEOH1%
ETOH0%
ETH3%
OLE2%
TOL1%
XYL2%
CRES1%
ISOP
ISPD4%
25 Aug. 2000Bayland Park
Total OH reacted:105.29 ppb
VOC + CO + CH4:75.9 ppb
VOC:52.63 ppb
TCEQ b1b.psito2n2base1b