aqd ipr pilot study programme on e-reporting - … · aqd ipr pilot study programme on e-reporting...

AQD IPR pilot study programme on e-Reporting Up-to-date data flow

Jaume Targa 4th pilot meeting

Copenhagen 15-16 March 2012

European Environment Agency


Introduction

• Aim

• Timeline for reporting

• Example from a data provider (data model, schema & codelists)

• Testing phase at EEA

• Q&A


Aim

•  To continue sharing “near real time” (now, UP-TO-DATE) air quality information across Europe

•  UTD data flow is now managed by Article 10 of the 2011/850/EC

•  Provision of UTD data + update of UTD following further quality control (Re-submission!)


Timeline for reporting AQD dataflow E2

Continuous sharing of UTD

On-going throughout year X, at hourly frequency appropriate to the assessment method

Continuous updates on UTD


The AQD dataflow E2

Purpose: Provision for reporting of NRT information exchange for public covering information and alert thresholds

Timeline: On-going throughout year X, at hourly frequency appropriate to the

assessment method Legal instruments: Dir. 2008/50/EC Art. 26 expanding upon EoI decision 97/101/EC

NATIVE

UTD-XML


AQD data model for dataset E – primary data

•  It is based on the O&M Observation type

•  It contains relevant info on measurement data + result values

•  It links to both •  AQD Process -> methodology used •  AQD Feature of Interest -> exact location of measurement


AQD data model for dataset E – primary data

Let’s work through an example


Country examples - UK

•  Aprox. 136 sites

•  Stored as MySQL database

•  Most stations measure several pollutants


•  What are we exchanging? Latest 24-48 hours of AQ data!!

•  How? Data blocks within XML as…

What information are we sharing?

2012-02-28T13:00:00,1,0,23


How is the data encoded?

2012-02-28T13:00:00,1,0,23

•  Corresponding “time” of the result •  YYYY-MM-DDThh:mm:ss



2012-02-28T13:00:00,1,0,23


•  Validity 1: valid 2: valid, but number replaced by 0.5*detection limit 3: not valid due to station maintenance or calibration not valid or missing 4: not valid or missing



2012-02-28T13:00:00,1,0,23


•  Validity

•  Verification

1: verified 2: preliminary verified 3: no verification



2012-02-28T13:00:00,1,0,23


•  Validity

•  Verification

•  Value

European Environment Agency European Environment Agency

RE

SU

LTS

2012-02-28T13:00:00,1,0,23

What does it look like in the XML?

Number of hourly values included for this sampling

point


RE

SU

LTS

2012-02-28T13:00:00,1,0,23

What does it look like in the XML?


point

•  What else? -> how do we tell where the measurement was taken!


AQD data model for dataset D - Station

•  Provider (data type ‘Contact Details’) •  Change documentation •  Zone code •  Zone name •  Zone type •  Zone delimitation (data type ‘Spatial Extent’) •  Zone history: application start and end date •  Predecessors (link to B) •  Resident population •  Resident population reference year •  Code of designated pollutant •  Protection Target •  Exemption or postponement according to Article 22 of Directive 2008/50/EC


UTD XML


Unique process ID

Unique Sampling Point + Unique feature of interest (INLET)


point

RE

SU

LTS

2012-02-28T13:00:00,1,0,23


Time period of data included

TimeInstant -> time file has been generated


Unique process ID

Unique Sampling Point + Unique feature of interest (INLET)


point

RE

SU

LTS

2012-02-28T13:00:00,1,0,23


General view on XML


XML file including observations from different sampling points (measured at several monitoring stations)

Observation gml:id uses the EoI code where SamplingPoint (Inlet) belongs to

The station name is followed by a simple sequence to diferenciate between different Processes (i.e. Instruments)

In this example, GB is using a combination of EoI code and number for the unique id required for gml:id


Interim xlink conventions

•  In the pilot testing programme there will be a need for interim xlink conventions

•  Metadata cited by xlink conventions in one data flow may not exist elsewhere (yet) e.g. within UTD data flow

</om:parameter><om:observedProperty xlink:href="#PM10"/><om:featureOfInterest xlink:href="SampleFeature_GB0729A_sp2"/>

•  So in this case you will need to refer to the gml:id and location where you expect it to be when you have finished the job

Doesn’t exist yet !


Interim xlink conventions - assumptions

•  FeatureOfInterest SampleFeature_EoICode_INLET sp1 = gases inlet sp2 = PM10 automatic inlet sp3 = PM2.5 automatic inlet

•  Process Process_UniqueValue process_6 = PM10 TEOM FDMS process_9 = NO2-NOx (general) process_5 = O3

(No. of processes will increase as GB meta data is re-structured)

PROCESS ID need to be re-used when necessary (i.e. same measurement config. will use same ID)


Interim xlink conventions - assumptions

•  SamplingPoint SamplingPoint_EoICode_INLET sp1 = gases inlet sp2 = PM10 automatic inlet sp3 = PM2.5 automatic inlet


Naming conventions proposals for UTD

•  Interim file naming conventions for the pilot

•  Data flow deliveries should be prepared for with a 3-part notation

•  Using this convention

dataFlowSuffix_countryCode_genericTimeReference.xml

E2a_GB_2012022912.xml - relates to UTD dataflow for the 24-hour period specified


Transformation mechanism

•  TB??

Communication

•  Extraction of AQ data from stations

Database

•  MySQL

AQD schema

•  Load to AQD.xsd with GOPublisher

Transmission

•  FTP upload of UTD data


Testing phase at EEA

UTD XML delivered by pilot MS

Operational delivery via FTP

Upload to database (EEA)

Perform QA/QC checks & calculate initial statistics (EEA)

Publish via AQ viewers (EEA)


Common issues – issues to discuss

•  Metadata cited by xlink DO NOT EXIST

•  Current database not structured in relation to Dataset D

•  For current NRT, we only use the EoI code to allocate station


Important issues – issues to discuss

•  Agreement of generation of gml:id

•  Need to agree common field and record separator

•  Agreement on generating Process_IDs ?? Useful question from BE: the Process_ID for NO2 is 9, for PM10 it is 6. What are the id's for the other pollutants ?


Any Questions ?

aqd ipr pilot study programme on e-reporting - … · aqd ipr pilot study programme on e-reporting...

Documents