051101 epa event seminar 050603 current future idaq black

ESIP AQ Cluster, [email protected] June 6, 2005

Air Quality ClusterTechTrack

Earth Science Information PartnersPartners

• NASA• NOAA• EPA

• (?)• USGS• DOE• NSF• Industry…

Agile Data Integration Infrastructure

Flow of DataFlow of Control

Air Quality Data

Meteorology Data

Emissions Data

Informing Public

AQ Compliance

Status and Trends

Network Assess.

Tracking Progress

Data to Knowledge Transformation

Draft, June 2005(intended as background for AQ Cluster discussions)

http://datafed.net/AQCluster/

http://www.esipfed.org/

http://www.esipfed.org/

DRAFT June 6, 2005 ESIP AQ Cluster, [email protected]

• The data life cycle consists of the acquisition and the usage parts

Usage ActivitiesData Acquisition

Data Acquisition and Usage Activities(Select View Show, click to step through PPT)

• The acquisition part processes the sensory data by firmly linked procedures

The Federation focuses on data usage activities and presumes repositories

• The usage activities are more iterative, dynamic procedures

• The collected and cleaned data are stored in the repository

Data Repository

•The usage cycle transform data into knowledge for decision making

Decisions


ScientistScience

DAACs

• Current info systems are project/program oriented and provide end-to-end solutions

Info UsersData Providers Info System

AIRNowPublicAIRNow

ModelCompliance

Manager

Sample AQ Information Usage Landscape

a

• Part of the data resources of any project can be shared for re-use through DataFed

• Through the Federation, the data are homogenized into multi-dimensional cubes

• Data processing and rendering can then be performed through web services

• Each project/program can be augmented by Federation data and services


• Data are distributed geographically by autonomous providers

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other

Content | Agency | Form

• Data includes emissions

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other


Information Landscape: Providers Geography, Content, Agency, Form

• Data includes emissions, ambient data,

Ambient

Ambient

Ambient

Ambient

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other


• Data includes emissions, ambient data, satellite data

Satellite

Satellite

SatelliteSatellite

Ambient

Ambient

Ambient

Ambient

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other


• Data includes emissions, ambient data, satellite data and model output

Model

Model

ModelModel

Satellite

Satellite

SatelliteSatellite

Ambient

Ambient

Ambient

Ambient

Emission

Emission

Emission

Emission

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other


• Data are provided by multiple agencies: EPA, NOAA, NASA and others

NASAMission

NOAAGASP

NASAIDEA

NASA DAACs

NOAA ASOS

EPA-AQSDataMart

EPA AIRNow

RPO VIEWS

FS FireInv

State/LocalEmission

EPA NEISGEI

EPA NEI

NOAA WeaMod

EPAAQModel

NOAA Forecast

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other


NASA DAACs

NOAA GASP

NASAIDEA

NASA Missions

EPA NEI

EPA NEISGEI

FS FireInv

State/Local Emission

NOAA ASOS

RPO VIEWS

EPA AIRNow

EPA-AQS AIRS

NOAA WeaMod

EPA AQModel

NASA GloModel

NOAA Forecast

• Furthermore, data are provided in varied formats and access protocols

Emission

Ambient

Satellite

Model

EPA

NOAA

NASA

Other


• Data on Internet are geography-independent and can be ‘linearized’

Internet

NASA DAACs

EPA R&DModel

EPA AIRNow

others


• Users are distributed geographically

EPA

NOAA

NASA

Other

Stakeholder | Agency | Form

Policy

Manager

Public

Scientist

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

Policy

Policy

Policy

• Users includes policy makers

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

• Users includes policy makers, the public

Policy

Policy

Policy PublicPublic

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

• Users includes policy makers, the public, AQ managers

Policy

Policy

Policy PublicPublic

Manager Manager

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

and scientist

Policy

Policy

Policy PublicPublic

Manager ManagerScientist Scientist

Scientist

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

• Users are affiliated with multiple agencies: EPA, NOAA, NASA, as well as others

Policy

Policy

Policy PublicPublic

Manager ManagerScientist Scientist

Scientist

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

Information Landscape: UsersTypes, Agency, Info Needs

• Furthermore, users need various types of information provided in multiple formats

Policy

Manager

Policy

Scientist

ManagerScientistScientist

Policy

Public Public

EPA

NOAA

NASA

Other


Policy

Manager

Public

Scientist

• Since the users are also on the Internet, their geographic location is irrelevant

Public

Manager

Scientist

Internet

other


Providers

NASA DAACs

EPA R&DModel

EPA AIRNow

others

Public

Manager

Scientist

Users

other

• The info system transforms the data into info products for each user • In the first stage the heterogeneous data are prepared for uniform access

Uniform Access

Information Landscape: Info System Data Access, Processing and Products

• The second stage performs filtering, aggregation, fusion and other operations

Data Processing Web Service Chain

Custom Processing

SciFlo

DataFed

Info Products Reports, Websites

Forecasting

Compliance

Other

Sci. Reports

• The third stage prepares and delivers the needed info products


Current Delivery and Integration SystemsData:

Emission, Surface Satellite, Model

Single Datasets

• Currently distributed are web-accessible as files, through data warehouses and web portals

• Virtually all processing, analysis and synthesis is custom-programmed by people

Reports

• The resulting information products are also hand-crafted by analysts

ESIP AQ Cluster, [email protected] June 6, 2005

• The challenge is to design a general supportive infrastructure• Simply connecting the relevant provides and users for each info product is messy

Integrated Data System for Air Quality-IDAQ

ESIP AQ Cluster 050510 Draft [email protected]

• The info system infrastructure needs to facilitate the creation of info products

AQ Compliance

Nowcast/Forecast

Status & Trends

Find Data Gaps

ID New Problems

………

Info Needs

Reports

• Providers supply the ‘raw material’ (data and models) for ‘refined’ info products

EmissionSurface Satellite

Model

Single Datasets

Providers

Wrappers

Where?

What?

When?

Federate Data

Structuring

• Structuring the heterogeneous data into where-when-what ‘cubes’ simplifies the mess

Slice & Dice

Explore Data

Viewers

• The ‘cubed’ data can be accessed and explored by slicing-dicing tools

Programs

Integrate

Understand

• More elaborate data integration and fusion can be done by web service chaining• This infrastructure support for IDAQ can be provided by the ESIP Federation

Non-intrusive Linking & Mediation Data UsersData Providers


A Sample of Datasets Accessible through ESIP – DataFed MediationNear Real Time (~ day)

It has been demonstrated (project FASTNET) that these and other datasets can be accessed, repackaged and delivered by AIRNow through ‘Consoles’

MODIS Reflectance

MODIS AOT TOMS Index

GOES AOT

GOES 1km Reflec

NEXTRAD Radar

MODIS Fire Pix

NRL MODEL

NWS Surf Wind, Bext

http://webapps.datafed.net/dvoy_services/datafed_catalog.aspx

http://datafed.net/

http://datafed.net/projects/FASTNET/FASTNET_Links.htm

051101 epa event seminar 050603 current future idaq black

Technology