Architecture and Technologies for an Agile, User-Oriented Air Quality Data System

Rudolf B. HusarWashington University, St. Louis

Presented at the workshop The User and the GEOSS Architecture

Applications for North AmericaJuly 30, 2006, Denver

Outline

• Highlight Trends of Air Quality Sensing and Management• Describe an Agile IS Architecture for Air Quality Decision Support • Show Their Application Through Two Use Cases

Smoke Event ppt, flashAQ Policy ppt, flash

Changes in Air Quality Management

Command & Control

Weight of Evidence

Flexible NAAMS

Rigid Monitoring

Real-time Air Pollution Sensing and Reporting

High Resolution Satellite DataSurface PM25 and Ozone Data

Smoke Plumes

Generic Decision Support for Air Quality Decisions

GEOSS Architecture Framework

Knowledge into the Minds of

Regulatory Analysts

Knowledge into the Minds of

Technical Analysts

Observations

Reports:Model Forecasts,

Obs. Evidence

Models

DecisionsKnowledge

into the Minds of

Decision- making managers

Decision Support System

Key Technical Challenge: Characterization

• Pollutant characterization requires many different instruments and analysis tools.

• Each sensor/network covers only a fraction of the 6-8 dimensional data space.

• Other sensors provide only integral measures of the pollution, e.g. satellite - vertical integral.

Satellite-Integral

• Data are distributed geographically by autonomous providers

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

• Data includes emissions

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

Information Providers: Geography, Content, Agency, Form

• Data includes emissions, ambient data,

Ambient

Ambient

Ambient

Ambient

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

• Data includes emissions, ambient data, satellite data

Satellite

Satellite

SatelliteSatellite

Ambient

Ambient

Ambient

Ambient

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

• Data includes emissions, ambient data, satellite data and model output

Model

Model

ModelModel

Satellite

Satellite

SatelliteSatellite

Ambient

Ambient

Ambient

Ambient

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

• Data are provided by multiple agencies: EPA, NOAA, NASA and others

NASAMission

NOAAGASP

NASAIDEA

NASA DAACs

NOAA ASOS

EPA-AQSDataMart

EPA AIRNow

RPO VIEWS

FS FireInv

State/LocalEmission

EPA NEISGEI

EPA NEI

NOAA WeaMod

EPAAQModel

NOAA Forecast

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

NASA DAACs

NOAA GASP

NASAIDEA

NASA Missions

EPA NEI

EPA NEISGEI

FS FireInv

State/Local Emission

NOAA ASOS

RPO VIEWS

EPA AIRNow

EPA-AQS AIRS

NOAA WeaMod

EPA AQModel

NASA GloModel

NOAA Forecast

• Furthermore, data are provided in varied formats and access protocols

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

• Data on Internet are geography-independent and can be ‘linearized’

Internet

NASA DAACs

EPA R&DModel

EPA AIRNow

others

• Users are distributed geographically

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

Policy

Policy

Policy

• Users includes policy makers

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

• Users includes policy makers, the public

Policy

Policy

Policy PublicPublic

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

• Users includes policy makers, the public, AQ managers

Policy

Policy

Policy PublicPublic

Manager Manager

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

and scientist

Policy

Policy

Policy PublicPublic

Manager ManagerScientist Scientist

Scientist

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

• Users are affiliated with multiple agencies: EPA, NOAA, NASA, as well as others

Policy

Policy

Policy PublicPublic

Manager ManagerScientist Scientist

Scientist

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

Users: By Types, Agency, Info Needs

• Furthermore, users need various types of information provided in multiple formats

Policy

Manager

Policy

Scientist

ManagerScientistScientist

Policy

Public Public

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

• Since the users are also on the Internet, their geographic location is irrelevant

Public

Manager

Scientist

Internet

other

• The data life cycle consists of the acquisition and the usage parts

Usage ActivitiesData Acquisition

Data Acquisition and Usage Activities(Select View Show, click to step through PPT)

• The acquisition part processes the sensory data by firmly linked procedures

The focus is on data usage activities

• The usage activities are more iterative, dynamic procedures

• The collected and cleaned data are stored in the repository

Data Repository

• The usage cycle transform data into knowledge for decision making

Decisions

ScientistScience

DAACs

• Current info systems are project/program oriented and provide end-to-end solutions

Info UsersData Providers Info System

AIRNowPublicAIRNow

ModelCompliance

Manager

‘Stovepipe’ and Federated Usage Architectures Landscape

• Part of the data resources of any project can be shared for re-use through DataFed

• Through the Federation, the data are homogenized into multi-dimensional cubes

• Data processing and rendering can then be performed through web services

• Each project/program can be augmented by Federation data and services

The Network Effect:Less Cost, More Benefits through Data Multi-Use

ProgramPublic

Data Organization

Data

Data Program

Program

OrganizationData

Data

ProgramData

Orgs Develop Programs

Programs ask/get Data Public sets

up Orgs

Pay only once Richer content

Data Re-Use Network Effect

Data are costly resource – should be reused (recycled) for multiple applications

Data Reuse

Less Prog. Cost More Knowledge

Data reuse saves $$ to programs and allows richer knowledge creation

Less Soc. Cost More Soc. Benefit

Data reuse, like recycling takes some effort: labeling, organizing, distributing

Providers

NASA DAACs

EPA R&DModel

EPA AIRNow

others

Public

Manager

Scientist

Users

other

• The info system transforms the data into info products for each user • In the first stage the heterogeneous data are prepared for uniform access

Uniform Access

Agile Information System: Data Access, Processing and Products

• The second stage performs filtering, aggregation, fusion and other operations

Data Processing Web Service Chain

Custom Processing

SciFlo

DataFed

Info Products Reports, Websites

Forecasting

Compliance

Other

Sci. Reports

• The third stage prepares and delivers the needed info products

Decision Support System

Event Knowledge into the Minds of

EPA Analysts

Knowledge into the Minds of

State Analysts

DSS for Exceptional Event Decisionsapping of

Observations

Event Reports:Model Forecasts,

Obs. Evidence

Models

DecisionsEvent Knowledge into the Minds of

EPA Regulators

Decision Support System

Data Sharing

Std

. In

terf

ace

Data

Obs. & Models

Characterization

Std

. In

terf

ace

ReportingDomain Processing

ControlReports

Stages of AQ Data Flow and Value-Adding Processes

Domain ProcessingData Sharing

Std

. In

terf

ace

Gen. ProcessingS

td.

Inte

rface

Data

Control

Reports

Reporting

Obs. & Models Decision Support System

Analyzing

Filter/IntegrateAggregate/FuseCustom Analysis

Organizing

DocumentStructure/FormatInterfacing

Characterizing

Display/BrowseCompare/Fuse Characterize

Valu

e-A

dd

ing

P

rocesses

Reporting

Inclusiveness Iterative/Agile Dynamic Report

Loosely Coupled Data Access through Standard Protocols

The next three slides describe the key technologies used in the creation of an adaptable and responsive air quality information system.

OGC data access protocols and standard formats facilitate loose coupling between data on the internet and processing services.

For air quality, the Web Coverage Service (WCS), provides a universal simple query language for requesting data as where, when, what. That is: geographic (3D bounding box), time range and parameter.

The Web Map Service (WMS) and Web Feature Service (WFS) are also useful.

The use of standard data physical data formats and naming conventions elevates the syntactic and semantic interoperability.

Within DataFed all data access services are implemented as WCS or WMS and optionally WFS. General format adapter components permit data request in a variety of standard formats.

GetCapabilities

GetData

Capabilities, ‘Profile’

Data

Where? When? What? Which Format?

Server

Back End S

td.

Inte

rface

Client

Front EndS

td.

Inte

rface

Query GetData Standards

Where? BBOX OGC, ISO

When? Time OGC, ISO

What? Temperature CF

Format netCDF, HDF.. CF, EOS, OGC

T2T1

Domain ProcessingData Sharing

Std

. In

terf

ace

Gen. ProcessingS

td.

Inte

rface

Data

Control

Reports

Reporting

Obs. & Models Decision Support System

Web Services and Workflow for Loose Coupling

Service Broker

Service Provider

PublishFind

BindServiceUser

Web Service Interaction Service Chaining & Workflow

Domain ProcessingData Sharing

Std

. In

terf

ace

Gen. ProcessingS

td.

Inte

rface

Data

Control

Reports

Reporting

Obs. & Models Decision Support System

Web Services Triad:Publish – Find – Bind

Workflow Software:Dynamic Programming

Collaborative Reporting and Dynamic Delivery

Co Writing - Wiki

ScreenCast

Analysis Reports:

Information supplied by manyNeeds continuous program feedbackReport needs many authorsWiki technologies are for collaborative writing

Dynamic Delivery:

Much of the content is dynamicAnimated presentations are compellingMovies and screencasts are for dynamic delivery

Domain ProcessingData Sharing

Std

. In

terf

ace

Gen. ProcessingS

td.

Inte

rface

Data

Control

Reports

Reporting

Obs. & Models Decision Support System

Summary

The current challenges for air quality information systems include data delivery in real time, pollution characterization through the integration of multi-sensory data and providing agile support to regulatory management. The talk describes the architecture and implementation of a standards based system for accessing and processing air quality data to satisfy the above needs. The agile federated data system, DataFed, system has been in use for science and management support since 2005.

DataFedis composed of distributed data and web services and integrated by user-configurable workflow software.The use of DataFed is illustrated through two use cases: (1) A real time monitoring example of a smoke event uses surface,satellite and model forecast information to inform air quality managers and the public. (2) Hemispheric aerosol transport model is compared to surface monitoring data to estimate the uncertainty and to improve themodel estimates.

Links

Powerpoint Flash

Background 800/1200

DataFed Architecture 800/1200

Smoke Event ppt 800/1200

AQ Policy ppt 800/1200

Acknowledgements

The presentation on Air Quality Background and Information Architecture benefited greatly from ideas, and challenges posed by a number of experienced individuals, from EPA (Rich Scheffe, Steve Young, Terry Keating), NASA (Lawrence Friedl, Kathy Fontaine).

The participation in the NASA Information Technology Infusion workgroup (Karen Moe, Bran Wilson, Liping Di and others) was an intense collective learning experience.

At CAPITA, Kari Hoijarvi engineered and implemented DataFed; Stefan Falke contributed datasets and application software. This presentation was prepared with help from Erin Robinson.