070416 egu vienna husar

Architectures and Technologies Enabling the Diffusion of

Atmospheric Science Information

Rudolf B. Husar and Erin Robinson Washington University, St. Louis

Presented at EGU, Vienna April 18, 2007

Data

Data

Data

Data

Data

Application

Application

Application

Application

Application

Think Networking!

The Transformational Effect of Networking

“Networking has led to an unprecedented surge of productivity” Time Magazine, Person of the Year 2006, YOU

• These are opportunities to enable Earth Science through more networking

• But many resistances to networking exist that need to be overcome

• Information has become the main driver of progress• Time and place are no longer barriers to participation and interaction • The Web has become a medium participation - ‘Web 2.0’ phenomenon

Networking Multiplies Value Creation

ApplicationData

1 User Stovepipe Value = 1 1 Data x 1 Program = 1

Enclosed Value-Creating Process - ‘Stovepipe’

ApplicationData

Application

Application

Application

Application

Stovepipe


5 Uses of Data Value = 5 1 Data x 5 Program = 5


Merging data may creates new, unexpected opportunities

Not all data are equally valuable to all programs


5 Uses of Data Value = 5 1 Data x 5 Program = 5

Open Network Value = 25 5 Data x 5 Program = 25

Data

Data

Data

Data

Data

StovepipeApplication

Application

Application

Application

Application


Agile Information System: Data Access, Processing and Products

Public

Manager

Scientist

Users

other

Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Organizing

DocumentStructure/FormatInterfacingV

alu

e A

dd

ing

P

rocesses


Uniform Access

Public

Manager

Scientist

Users

other

Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Organizing

DocumentStructure/FormatInterfacingV

alu

e A

dd

ing

P

rocesses

Homogenizing

Format profile Standard accessData as Service


Uniform Access

Data Processing Web Service Chain

Custom Processing

SciFlo

DataFed

Public

Manager

Scientist

Users

other

Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Organizing

DocumentStructure/FormatInterfacing

Characterizing

Display/BrowseCompare/Fuse CharacterizeV

alu

e A

dd

ing

P

rocesses

Homogenizing



Uniform Access


Custom Processing

SciFlo

DataFed

Products Reports

Forecast

Compli.

Other

Science

Public

Manager

Scientist

Users

other

Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Analyzing

Filter/IntegrateAggregate/FuseCustom Analysis

Organizing


Characterizing

Display/BrowseCompare/Fuse CharacterizeV

alu

e A

dd

ing

P

rocesses

Homogenizing


Value-Adding Processes

Uniform Access


Custom Processing

SciFlo

DataFed

Products Reports

Forecast

Compli.

Other

Science

Public

Manager

Scientist

Users

other

Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Analyzing

Filter/IntegrateAggregate/FuseCustom Analysis

Organizing


Characterizing

Display/BrowseCompare/Fuse Characterize

Reporting

Inclusiveness Iterative/Agile Dynamic ReportV

alu

e A

dd

ing

P

rocesses

Homogenizing



Uniform Access


Custom Processing

SciFlo

DataFed

Products Reports

Forecast

Compli.

Other

Science

Public

Manager

Scientist

Users

other

Control

Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Data

Control

Seeking Information

Providing Information

Negotiating & Market Space

System of SystemsGlobal Earth Observing System of Systems - GEOSS

Characteristics of System of Systems (SoS)

• Autonomous constituents managed/operated independently• Independent evolution of each constituent• SoS displays emergent behavior

Must recognize, manage, exploit the characteristics:

• No stakeholder has complete SoS insight• Central control is limited; distributed control is essential• Users, must be involved throughout the life of a SoS

Lets agree onSpace-Time-Parameter

Data Access Query Protocol

Interoperability Stack: Key concept of the Web Connecting Machines and People

IP – Internet Protocol

Service Orientation Open Architecture

Data Standards

Amplify Individuals Connect Minds

System components have to be interoperable at each layer

Uniform Access


Custom Processing

SciFlo

DataFed

Products Reports

Forecast

Compli.

Other

Science

Public

Manager

Scientist

Users

other

Control

Data

Acq

uis

itio

n Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Standard Data Query Language:

Where? When? What? (Space-time query - WMS, WCS)

GetCapabilities

GetData

Capabilities, ‘Profile’

Data

Where? When? What? Which Format?

Server

Back End S

td.

Inte

rface

Client

Front EndS

td.

Inte

rface

Query GetData Standards

Where? BBOX OGC, ISO

When? Time OGC, ISO

What? Temperature CF

Format netCDF, HDF.. CF, EOS, OGC

T2T1

Loosely Coupled Data Access through Standard Protocols

Standard Messaging

What data you have?Give me this data

Uniform Access


Custom Processing

SciFlo

DataFed

Products Reports

Forecast

Compli.

Other

Science

Public

Manager

Scientist

Users

other

Control

Data

Acq

uis

itio

n Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Web Services and Workflow for Loose Coupling

Service Chaining & Workflow

Workflow Software:Dynamic Linking

Software Mashups

Software Mashup:Coarse-grain Linking

SeaWiFS Satellite

SeaWiFS Satellite

Aerosol Chemical

Air Trajectory

Map Boarder

VIEW by Web Service Composition

Uniform Access


Custom Processing

SciFlo

DataFed

Products Reports

Forecast

Compli.

Other

Science

Public

Manager

Scientist

Users

other

Control

Data

Acq

uis

itio

n Provider

NASA DAACs

EPA Model

EPA AIRNow

others

Data

Collaborative Reporting and Dynamic Delivery

Co Writing - Wiki

ScreenCast

Collaborative Analysis and Writing

Wiki, Blogs, Group Annotations

Dynamic Content Delivery:

GoogleEarth, Screencasting…

ScientistScience

DAACs

• Current info systems are project/program oriented and provide end-to-end solutions

Info UsersData Providers Info System

AIRNowPublicAIRNow

ModelCompliance

Manager

‘Stovepipe’ and Federated Usage Architectures Landscape

• Part of the data resources of any project can be shared for re-use through DataFed

• Through the Federation, the data are homogenized into multi-dimensional cubes

• Data processing and rendering can then be performed through web services

• Each project/program can be augmented by Federation data and services

DataFed: 100+ Datasets Non-intrusively Federated

• Data are accessed from autonomous, distributed providers• DataFed ‘wrappers’ provide uniform geo-time referencing• Tools allow space/time overlay, comparisons and fusion

Near Real Time Data IntegrationDelayed Data Integration

Surface Air Quality AIRNOW O3, PM25 ASOS_STI Visibility, 300 sitesMETAR Visibility, 1200 sitesVIEWS_OL 40+ Aerosol Parameters

SatelliteMODIS_AOT AOT, Idea ProjectGASP Reflectance, AOTTOMS Absorption Indx, Refl.SEAW_US Reflectance, AOT

Model OutputNAAPS Dust, Smoke, Sulfate, AOTWRF Sulfate

Fire DataHMS_Fire Fire PixelsMODIS_Fire Fire Pixels

Surface MeteorologyRADAR NEXTRADSURF_MET Temp, Dewp, Humidity…SURF_WIND Wind vectorsATAD Trajectory, VIEWS locs.

Sample of Federated Datasets

Sulfate in the Northeast

Sahara Dust in the Gulf

Fires in the Southeast

Time Series Console: Southeast

Analyst Console Applications:

Sulfate Episode: 8/ 27/ 04

A Sample of Datasets Accessible through ESIP MediationNear Real Time (~ day)

It has been demonstrated (project FASTNET) that these and other datasets can be accessed, repackaged and delivered by AIRNow through ‘Consoles’

MODIS Reflectance

MODIS AOT TOMS Index

GOES AOT

GOES 1km Reflec

NEXTRAD Radar

MODIS Fire Pix

NRL MODEL

NWS Surf Wind, Bext

Summary Grand ConvergenceWill we make use of it?

• Third-party mediation can homogenize distributed ES data• Agile SOA-based IS can deliver diverse info products to users

• Since 2005, one such IS, DataFed is used by EPA and in research

• However, more data need to be federated by the community

Parting thoughts

Think outside the stovepipe – Think networking

Divide and Conquer, NO! Connect and Enable, YES!

Thank you

070416 egu vienna husar

Technology