FAIRMODE meeting, Norrkoping, June Institute for Environment and Sustainability SG4: Benchmarking of air quality models FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Ensemble Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June SG4 Presentations on the Questionnaire Short presentations (max 10 min) by: Ana Miranda (PT) David Carruthers (UK) Hans Backstrm (SE) Helge Olesen (DK) Marcus Hirtl (AT) Mihaela Mircea, Guido Pirovano (IT) FAIRMODE meetind, Norrkoping, June Background FAIRMODE meetind, Norrkoping, June Model results DELTA JRC USER Data Extraction Facility BENCHMARKING service Model performance Evaluation reports The benchmarking procedure FAIRMODE meetind, Norrkoping, June Since the Oslo meeting (sep. 2010) - Document DELTA concepts sent to SG4 participants (March 11) - Distribution of DELTA tool & utilities (about 20 users) - SG4 Web page created (http://aqm.jrc.it/DELTA)http://aqm.jrc.it/DELTA - Use of DELTA on different datasets (POMI, Madrid, London) - User feedback questionnaire FAIRMODE meetind, Norrkoping, June Intended for rapid diagnostics by single users (at home) Focus mostly on surface measurement- model pairs independence of scale Focus on AQD related pollutants on a yearly period (but AQ related input data also checked) Exploration and benchmarking modes The DELTA tool FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble) Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June Outline 1. Content of the performance report 2. Links between Target and more traditional indicators (an analysis based on 3 datasets) 3. A first-guess for criteria/goals 4. Comparison with RDE 5. Observation uncertainty 6. Proposed update to the report template FAIRMODE meetind, Norrkoping, June Content of the performance report FAIRMODE meetind, Norrkoping, June Should include a set of statistical indicators and diagrams complete enough to capture the main aspects of model performance but limited enough to fit in one page summary Keep a similar template for all pollutants and spatial scales (but differences in terms of criteria/gals). Restricted to AQD needs. Currently proposed for O3 8h daily max, NO2 hourly and PM10 daily. Developed (at least first) for assessment purposes Should include performances criteria and goals Content of the performance report (1) Constraints FAIRMODE meetind, Norrkoping, June Criteria: Acceptable performance for a given type of application (e.g. PM: MFE=75%, MFB=+/-60%) Goal: Best performance a model should aim to reach given its current capabilities (e.g. PM: MFE=50%, MFB=+/-30%) Content of the performance report (2) RMSE/ o R=0.7 OU MEF < 0 FAIRMODE meetind, Norrkoping, June MFB=0.67 Checks on data availability for each stations 75% for time averaging (e.g. 18h at least per day) 90% available on total (e.g. >328 days/year) Content of the performance report (3) 90% concept for indicators FAIRMODE meetind, Norrkoping, June Links between Target and more traditional indicators (an analysis based on 3 datasets) FAIRMODE meetind, Norrkoping, June Links between Target and more traditional indicators (1) Bias SigM/SigO R FAC2 CRMSE Target indicator = RMSE / SIgO FAIRMODE meetind, Norrkoping, June monitoring sites suburban, urban and rural background 5 models: CHIMERE, TCAM, CAMX, RCG, MINNI Year:2005 Domain resolution:6x6km 2 O3 PM10 10 monitoring sites urban background 1 model: WRF-CMAQ Year: 2007 Domain resolution: 1x1 km2 O3 NO2 107 monitoring sites suburban/urban background, kerbside and roadside 1 model: ADMS Year: 2008 NO2 O3 PM10 Po - Valley Madrid London Examples on 3 datasets FAIRMODE meetind, Norrkoping, June Based on real datasets, start by analysing how the bias criteria (MFB) proposed by Boylan and Russel (2005) compares to Target. Fix a criteria for the target indicator which is consistent with the MFB criteria Fix values for the other statistical indicators (R, StdDev ratio, FAC2) to be consistent with the assigned criteria on the Target value Methodology to fix first guess criterias Links between Target and more traditional indicators (3) FAIRMODE meetind, Norrkoping, June T: 58% RDE: 83% How to connect Target to more accessible indicators? (3) EXAMPLE 1: Po_valley (PM10) Crit Target=1 FAIRMODE meetind, Norrkoping, June T: 32% RDE: 95% How to connect Target to more accessible indicators? (3) EXAMPLE 2: Po_valley (PM10) Crit Target=1 FAIRMODE meetind, Norrkoping, June T: 96% RDE: 100% How to connect Target to more accessible indicators? (3) EXAMPLE 3: London (PM10) Crit Target=1 FAIRMODE meetind, Norrkoping, June T: 70% RDE: 96% How to connect Target to more accessible indicators? (3) EXAMPLE 4: Po_valley (O3) Crit Target=0.8 FAIRMODE meetind, Norrkoping, June T: 66% RDE: 100% How to connect Target to more accessible indicators? (3) EXAMPLE 5: Madrid (O3) Crit Target=0.8 FAIRMODE meetind, Norrkoping, June T: 77% RDE: 94% How to connect Target to more accessible indicators? (3) EXAMPLE 6: London (NO2) Crit Target=1 FAIRMODE meetind, Norrkoping, June T: 60% RDE: 100% How to connect Target to more accessible indicators? (3) EXAMPLE 7: Madrid (NO2) Crit Target=1 FAIRMODE meetind, Norrkoping, June A first-guess for criteria/goals FAIRMODE meetind, Norrkoping, June MFBTargetRFAC2SM/SO PM10Daily60% 1.0~0.5050% 50% O38h30% 0.8~0.6550% 50% NO21h30% 1.0~0.5550% 50% A first-guess for criteria/goals (1) NOTE: Boylan and Russel MFB criteria is proposed based on urban to regional scale modelling (from 4 to 36 km spatial resolution) addresses only O3 and PM10 FAIRMODE meetind, Norrkoping, June A first-guess for criteria/goals (2) Different criteria are currently proposed for O3-8h, PM10-daily and NO2-hourly. Although spatial-scale and time average dependency are possible, they are not considered up to now (point of discussion) Scale is intended in terms of spatial resolution, linked to monitoring station type: Regional Rural background Urban Urban & suburban background Local All urban stations (incl. roadside & kerbside) Criteria probably need to be developed for yearly averaged values Performance goals have arbitrarily been fixed to a 20% more stringent value 3 datasets is not ENOUGH! FAIRMODE meetind, Norrkoping, June How do these criteria compare to RDE? FAIRMODE meetind, Norrkoping, June RDE Target R FAC2 SM/SO MFB FAIRMODE meetind, Norrkoping, June RDE Target R FAC2 SM/SO MFB FAIRMODE meetind, Norrkoping, June RDE Target R FAC2 SM/SO MFB FAIRMODE meetind, Norrkoping, June Station Osio Sotto, POMI (NO2 - RCG) Target: 2.19 MFB:73% FAC2:41% R:0.39 SigM/SigO:1.49 How these criteria compares to RDE? (2) RDE=11% FAIRMODE meetind, Norrkoping, June Station EA1, London (NO2 - ADMS) Target: 0.82 MFB:8% FAC2:89% R:0.73 SigM/SigO:1.14 How these criteria compares to RDE? (3) RDE = 56% FAIRMODE meetind, Norrkoping, June About observation uncertainty FAIRMODE meetind, Norrkoping, June S0RMSE / SO O3 PO LOND MAD NO2 PO LOND MAD PM10 PO LOND MAD About observation uncertainty (1) RMSE/S0 S0 PollutantOU O3, NO215% PM1025% AQD FAIRMODE meetind, Norrkoping, June NO2 O3 PM10 ADMS, London, 2008 About observation uncertainty (2) FAIRMODE meetind, Norrkoping, June Proposed update to the report template FAIRMODE meetind, Norrkoping, June SigO/SigM SigO > SigM SigO < SigM Proposed update to the report template FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble) Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June Exploration : Time selection (period, averaging time, season, day/night-time, max/min/mean) Information overlay (models, scenarios, variables, stations) Spatial analysis (color codes vs. 2D maps) DELTA: Exploration mode (1) FAIRMODE meetind, Norrkoping, June DELTA: Exploration mode (2) FAIRMODE meetind, Norrkoping, June Model V1 vs. Model V2 DELTA: Exploration mode (3) Upgrade FAIRMODE meetind, Norrkoping, June DELTA: Exploration mode (4) Upgrade FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble) Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Inclusion of planning application Benchmarking service Others (Ensemble) Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Benchmarking service Others (Ensemble) Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June DELTA developments Short term (Autumn 2011) Flexible use of benchmarking mode and production of pdf or postscript reports On-click mouse information Windows/Linux portability Station grouping mode Longer term ( ) Inclusion of planning applications Extension of benchmarking for annual averages (?) Inclusion of PM2.5 FAIRMODE meetind, Norrkoping, June Require a series of simulations with fixed emission reductions for main precursors (NOx, VOC, NH3, SO2, PPM) and analyze difference in behavior. Problem: No observations available Reference model ? Joint exercices 2.Analysis of spatio-temporal emission patterns in provided data (e.g week vs. week-end day, DEFRA 2011) DELTA expl. Mode (Links with SG3) Model responses to emission reductions depend on the geographical location, the model scale, meteorological year DELTA developments FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Benchmarking service Others (Ensemble) Evaluation Datasets Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June Model results DELTA JRC USER Data Extraction Facility BENCHMARKING service Model performance Evaluation reports DELTA Benchmark Case & Report DB Model info Deadline: end 2012 FAIRMODE meetind, Norrkoping, June Agenda Status DELTA: Template for performance reporting DELTA: Exploration mode Users feedback & discussion Users presentations Discussion around the questionnaire Future Plans & deliverables for TSAP DELTA Benchmarking service Others (Ensemble) Evaluation Datasets Review of possible TSAP Deliverables FAIRMODE meetind, Norrkoping, June Deliverables for TSAP review Common template for reporting model performances for O3, NO2 and PM10 as complement to the current RDE indicator. Recommendations on quality objectives (performance criteria) to be achieved Benchmarking procedure including the following elements: DELTA Benchmarking service ENSEMBLE + data preparation facility FAIRMODE meetind, Norrkoping, June Annexes FAIRMODE meetind, Norrkoping, June SpeciesMetricCriteriaGoal Boylan and Russel, 2005, EPA report 2007 Main PM constituents (> 30% total mass), PM2.5 MFE MFB 75% 60% 50% 30% Minor PM constituents (< 30% total mass) Exp variations to reach 100% / 200% at 0 concentrations OzoneMFE MFB 35% 15% Evaluating the Performance of Air Quality Models, AEA (2009) Any pollutantFAC2 NMB Half points within -0.2 < MFB < 0.2 Air quality model performances evaluation, Chang et Hanna (2004) NOx, CO, PM10FAC2 FB NMSE Half points within -0.3 < FB < 0.3 NMSE < 4 Air Quality (Regional scale modelling) Criteria & goals FAIRMODE meetind, Norrkoping, June RDE takes closest to exceedance values. In the case the max value does not exceed the threshold, it is relative to the closest one. Lack of comparability of this measure among stations MFB can be taken as 60%,30% No change because low values are within range of stable MFB No assumption on station representativness Screen monitoring data and drop extreme artificial events (e.g. fireworks) Adapt templates for annual values How these criteria compares to RDE? (2) FAIRMODE meetind, Norrkoping, June SG4 participants - EoI PersonInstitutePersonInstitute Marco BEDOGNIAMAT (IT)Jana MATEJOVICOVAHMI (SK) Bertrand BESSAGNETINERIS (FR)Anna MIRANDAUniversity of Aveiro (PT) David CARRUTHERSCERC (UK)Dietmar OETTLGovernment of Styria (AT) Koen DE RIDDERVITO (BE)Christakis PAPADOPOULOSDLI (CY) Dick DERWENTAEA (UK)Guido PIROVANOERSE (IT) Hilde FAGERLIMet Inst (NO)Peter ROBERTSCONCAWE (BE) Alenka FRITZELMinistry of Environment (SI)Martijn SCHAAPTNO (NL) Stefano GALMARINIEC-JRC, IspraCamillo SILIBELLOARIANET (IT) Emilia GEORGIEVAEC-JRC, IspraAlexandros SIRAKOSUOWM (GR) Maria GONCALVESBSC (ES)Philippe THUNISEC-JRC, Ispra Pernige GRENNFELTIVL (SE)Keith VINCENTAEA (UK) Ari KarppinenFMI (FI)Rahela ZABKARUniversity of Ljubljana (SI) Fernando MARTINCIEMAT (ES)John BARTZISUOWM (GR) Jose BALDASANOBSC (ES)Nicolas MOUSSIOPOULOSAUT (GR) John DOUROSAUT (GR)Leonor TARRASONNILU (NO) Bruce DENBYNILU (NO)H. BECKSTROEMSMHI (SE) Joanna STRUZEWSKAWOT (PL)Nutthida KITWIROONHERTS (UK) Marcus HirtlZAMG(AT)Ruben BeijkRIVM (NL) Joost WesselingRIVM (NL) FAIRMODE meetind, Norrkoping, June User Name Giudo Pirovano Mihaela Mircea Marcus Hirtl Addo Van Paul Dietmar Oettl Rafael Borge David Carruthers Suzanne Yongen Kevin Delaney Stefan Andersson Claudio Carnevale Ana Miranda Helge Olesen User affiliation RSE ENEA ZAMG RIVM Graz University UPM CERC VITO Irish EPA SMHI Brescia University Aveiro University DMU QuestionnaireV V V v x ? v v v v v V x PresentationV V V x x ? v x ? v x V v