Integrated Decision Support:A Tale of Two Systems

“It was the best of times, it was the worst of times…”

Charles Dickens, A Tale of Two Cities, 1859

“Actually, right now, I’d say the times are just OK…”

Shawn McClure, EPA Data Summit, 2008

“On second thought, the times must be pretty damn good. After all, we’re at a data summit convened by the EPA to talk about data systems. Woo-hoo!!…”

Shawn McClure, EPA Data Summit, 2008

VIEWSThe Visibility Information Exchange Web System

http://vista.cira.colostate.edu/views

Developed by the five Regional Planning Organizations (RPOs) in 2002

Provides easy online access to a wide variety of air quality data

Offers user-friendly tools for exploring, visualizing, and analyzing data

The primary source for IMPROVE Regional Haze Rule data

Over 1100 registered users

Over 300 organizations, institutions, and companies represented

Developed for RPOs, States, Tribes, Federal Land Managers, and local agencies

Used by researchers, analysts, planners, regulators, stakeholders, and students

TSSWRAP Technical Support System

http://vista.cira.colostate.edu/tss

Built upon the database and software infrastructure of VIEWS

Developed initially for Western States, Tribes, and air quality agencies

Provides a access point for regional technical data, guidance, and results

Offers tools and support for the development of SIPs and TIPs

Documents the technical methods used in implementation plans

Helps assess the impact of other areas on local Class I Areas

Provides ongoing tracking and assessment of emissions control strategies

Will adapt to and serve the future regional technical needs of WRAP members

DataCollection

PrimaryStorage

Aggregation, Integration, Processing,

Modeling

Analysis, Visualization,

Reporting

Decision Support

EndUsers

Draft Data Value Chain Diagram

Analysis, Visualization and ReportingApplications

Decision SupportApplications

Group A2: Data

Access

Group B2:Integration

Consolidation

Group C2:Data Visualization and

Analysis

Group A2: Data

Access

DataCollection

PrimaryStorage

Aggregation, Integration, Processing,

Modeling

Analysis, Visualization,

Reporting

Decision Support

EndUsers

Draft Data Value Chain Diagram

Analysis, Visualization and ReportingApplications

Decision SupportApplications

Group A2: Data

Access

Group B2:Integration

Consolidation

Group C2:Data Visualization and

Analysis

Group A2: Data

Access

Data

Data Collection

Standardization

Transformation

Aggregation

Normalization

Verification

Validation

Processing

Primary StorageAggregation, Integration,

Processing, Modeling

Modeling

Reanalysis

Analysis

Comparison

Assessment

Forecasting

Analysis

Analysis, Visualization, Reporting

Recommendations

Conclusions

Regulations

Decisions

Decision Support End Users

VIEWS/TSS Data Value Chain

Standardization

Transformation

Aggregation

Normalization

Modeling

Reanalysis

Analysis

Comparison

Assessment

Forecasting

Recommendations

ConclusionsVerification

Validation

Regulations

Data Processing Analysis Decisions

Data Collection Primary StorageAggregation, Integration,

Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users

VIEWS/TSS Data Inventory

VIEWS/TSS Data Inventory

VIEWS/TSS Data Inventory

Standardization

Transformation

Aggregation

Normalization

Modeling

Reanalysis

Analysis

Comparison

Assessment

Forecasting

Recommendations

ConclusionsVerification

Validation

Regulations

Data Processing Analysis Decisions

Data Collection Primary StorageAggregation, Integration,

Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users

VIEWS/TSS Data Processing: Import & Validation

AIRDATA_OLTP

Data Acquisition

System

AIRDATA_SOURCE

Metadata Import System

Data Import System

Validation

Validation

· Emailed updates from data providers· Automatic HTTP extraction of source

data files· Database replication· Imports data “as-is” into the source

database· Technologies: ASCII text files, Excel

spreadsheets, Access databases, etc.

· Facilitates the entry of new metadata

· Validates new metadata entries· Detects overlap with existing

metadata· Technologies: VB .Net, ASP .Net,

and SQL Server stored procedures

· Extracts data from the source database· Scrubs data and performs conversions· Maps source metadata to integrated metadata· Transforms the data into an integrated schema· Verifies and validates imported data· Loads data into the back-end OLTP system· Technologies: Microsoft SQL Server DTS,

stored procedures, SQL scripts, and Visual Basic routines

Metadata Import System Data Import SystemData Acquisition System

VIEWS/TSS Data Processing: Integration

OLTP:OLTP:

• Functions as the “back-end” database• Fully relational and in 3rd normal form• Used for data import, validation, and

management• Technologies: Microsoft SQL Server

Data Warehouse Generation System:Data Warehouse Generation System:

• Extracts data from the OLTP• De-normalizes and transforms data• Loads data into the Data Warehouse• Builds table indexes• Archives “snapshots” of the database• Technologies: VB, stored procedures

Data Warehouse:Data Warehouse:

• Functions as the “front-end” database• Uses a de-normalized “star schema”• Used for querying and archiving data• Automatically generated from the OLTP• Technologies: Microsoft SQL Server

Standardization

Transformation

Aggregation

Normalization

Modeling

Reanalysis

Analysis

Comparison

Assessment

Forecasting

Recommendations

ConclusionsVerification

Validation

Regulations

Data Processing Analysis Decisions

Data Collection Primary StorageAggregation, Integration,

Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users

VIEWS/TSS Analysis Tools: Dynamic Contour Maps

VIEWS/TSS Analysis Tools: Dynamic Data Maps

VIEWS/TSS Analysis Tools: Network Inter-comparison

Parameter: Nitrate Ion Concentrations

Location: Bondville, IL

Networks: IMPROVE, STN, and CASTNet

Graphs: Time Series and Scatter Plot

VIEWS/TSS Tools: Model Performance Evaluation

CMAQ Model Performance vs. Monitored Worst 20% Days in 2002

VIEWS/TSS Analysis Tools: Source Apportionment

Mass source apportionment by source category and region

From regional photochemical model with comprehensive emissions inputs

Species mass for various time periods – directly comparable to monitoring data

VIEWS/TSS Analysis Tools: Glide Slope

VIEWS/TSS Analysis Tools: Multidimensional Analysis

Multiple NAAQS indicators, multiple time periods, multiple locations

Displayed with consistent, comparable graphics in the same tool

VIEWS/TSS Analysis Tools: Emissions Review Tool

• Available Parameters:– Sulfur Dioxide– Sulfur Oxides (gas and

particulate)– Nitrogen Oxides (gas)– Nitrogen Oxides (gas and

particulate)– Other species from SMOKE

• Multiple dimensions:– Parameters– Emissions Scenarios– Source Categories– Regions of Interest

• Future plans:– Display regional summaries as

well as state-level emissions

VIEWS/TSS Tools: Raw Data

VIEWS/TSS Tools: Connections to Other Systems

VIEWS/TSS Interoperability

Import and output data in common formats, by standard protocols

K.I.S.S. – Because users like to do odd things with data

Practice a “Service Oriented Architecture” approach

Develop reusable, “component” tools

Make interoperability an important goal, while resisting the urge to develop the “Ultimate All-Things-to-All Users” type of system

Adhere to sound software and system architecture principles

Allow direct, read-only access to our database

Support and promote an open data and software architecture

Standardization

Transformation

Aggregation

Normalization

Modeling

Reanalysis

Analysis

Comparison

Assessment

Forecasting

Recommendations

ConclusionsVerification

Validation

Regulations

Data Processing Analysis Decisions

Data Collection Primary StorageAggregation, Integration,

Processing, ModelingAnalysis, Visualization, Reporting Decision Support End Users

VIEWS/TSS Decisions: Benefactors & Benefits

Federal Land Managers:

• Better air quality evaluation• More accurate source apportionment• More meaningful impact assessment• Informed permitting and regulation• Improved air quality in ecosystems

EPA and RPOs:

• Effective emissions control evaluation• Practical and equitable standards• Leveraging of existing efforts• Improved synergy with States and Tribes• Valuable case studies and precedents• More effective application of resources

States, Tribes, and Local Agencies:

• Effective Implementation Plans• Defensible and achievable control

strategies• Uniform and reasonable progress• Better air quality in Class I Areas

General Air Quality Community:

• Better insight into the decision process• Real life evidence of successes &

failures• Incorporation of feedback into next

generation products• Increasing expertise in developing

decision support tools

VIEWS/TSS Team: Future Plans

Add satellite data in conjunction with NASA ROSES contract

Add additional NAAQS (ozone, for example) and global data

Develop more tools that offer region- and locale-specific analyses

Develop appropriate new functionality as web services

Refine decision tools for making interpretations and recommendations

Expand and refine guidance documentation

Establish case studies from the user community as guidance

Allow users to upload and immediately work with their data

Develop modular, reusable, components for other developers

Establish reusable precedents for air quality decision support

Have a few beers every once (or twice) in a while…

Decisions(interpretations, conclusions, guidelines, regulations

Science(knowledge, examples,

expertise, theory)

Analysis(Tools, reanalysis, modeling,

inputs from other systems)

Processed data(calculated, RHR,

normalized, value-added)

Raw data(observations, methods)

The End…

Thanks!