surface water quality monitoring information system (swqmis) cindi atwood tetra tech, inc. (703)...
TRANSCRIPT
Surface Water Quality Monitoring Information System
(SWQMIS)
Cindi AtwoodTetra Tech, Inc.(703) [email protected]
Nancy RaglandTCEQ(512) [email protected]
TCEQ’s New Enterprise Data System for Managing Ambient
Water Quality Data
Benefits
Benefits
Consolidate and centralize dataConsolidate and centralize data Increase data consistency and Increase data consistency and
integrityintegrity
Assist your agency with internal Assist your agency with internal and external data requestsand external data requests
Help management to turn around Help management to turn around urgent, high priority requests urgent, high priority requests quicklyquickly
Save time and moneySave time and money
One Week vs One Day
44 Hours down to 8 Hours 44 Hours down to 8 Hours
2 person task down to a 1 2 person task down to a 1 person taskperson task
System Deployment
System Deployment
Deployed the system successfully in early Deployed the system successfully in early 20072007
Developed the system as:Developed the system as: Web-basedWeb-based J2EE and OracleJ2EE and Oracle
Integrated the system with web-based GIS Integrated the system with web-based GIS functionalities using ESRI ArcIMS and functionalities using ESRI ArcIMS and ArcSDEArcSDE
Core System Modules
Monitoring StationsMonitoring Stations
EquipmentEquipment
Geographical AreasGeographical Areas
ConstituentsConstituents
SamplingSampling
Comparison Comparison InformationInformation
ProjectsProjects
Reports and ExtractsReports and Extracts
Maps (GIS)Maps (GIS)
AssessmentAssessment
AdministrationAdministration
Manage Ambient Water Quality Data
Manage Ambient Water Quality Data
Organize data in a way that enhances Organize data in a way that enhances business processesbusiness processes
Protect data integrity through clean Protect data integrity through clean system design system design
Involve stakeholders at all levels of Involve stakeholders at all levels of organization in design and testorganization in design and test
Keep system flexible to meet future Keep system flexible to meet future needsneeds
Consolidate & Centralize Databases
Consolidate & Centralize Databases Allow core data submittal from various data Allow core data submittal from various data
owners owners
Consolidate and centralize databases Consolidate and centralize databases containing :containing : Field measurements Field measurements
Toxics and organics in a variety of mediaToxics and organics in a variety of mediaBiological Biological Habitat parameters, etc. Habitat parameters, etc.
Spatial dataSpatial data
Various Data Sources
Various Data Sources Supply custom interfaces for manual entrySupply custom interfaces for manual entry
AdministratorsAdministrators Data ManagersData Managers Field PersonnelField Personnel AssessorsAssessors
Provide mass data loaders for various data Provide mass data loaders for various data sources including sources including Continuous data from data loggers Continuous data from data loggers LIMSLIMS USGSUSGS River and municipal authorities in TexasRiver and municipal authorities in Texas
Data Integrity
Data Integrity
Provide custom interfaces for manual entry Provide custom interfaces for manual entry Enforce consistent data entry of all types of Enforce consistent data entry of all types of
data data Typographical errorsTypographical errors Outlier checksOutlier checks
Validate data from manual and mass loaded Validate data from manual and mass loaded datadata
Review of submitted data by data managers Review of submitted data by data managers to approve or denyto approve or deny
Mass Loaded Data Validation
The automated validation checks are all run, without stopping after The automated validation checks are all run, without stopping after the first “failure.” In this way, a complete list of errors can be the first “failure.” In this way, a complete list of errors can be shown. shown.
Over 50 validation checks run on mass loaded dataOver 50 validation checks run on mass loaded data
Check for outliers. Check for outliers. Check measurements against predefined Check measurements against predefined ranges (min/max). Compares verify flag in the Results file to ranges (min/max). Compares verify flag in the Results file to see if the outlier value has already been verified by the see if the outlier value has already been verified by the collector.collector.
If sample If sample collected when rainingcollected when raining or has rained within the last or has rained within the last 24 hours, report a value of <1. In other words, is days since 24 hours, report a value of <1. In other words, is days since precipitation (72053) < 1?precipitation (72053) < 1?
Check for duplicatesCheck for duplicates in data set based on station ID, sample in data set based on station ID, sample date, sample time, and parameter code.date, sample time, and parameter code.
GIS Integration
GIS Integration
Allow map-based creation of sitesAllow map-based creation of sites
Automatically assign geographic Automatically assign geographic attributes such as basin, county and attributes such as basin, county and ecoregion to a siteecoregion to a site
Provide map-based search and Provide map-based search and reporting featuresreporting features
Synchronize geographic data with Synchronize geographic data with tabular data tabular data
GIS Integration
GIS Integration
Public Access
Public Access
Allow the public and other organizations Allow the public and other organizations to retrieve water quality data to retrieve water quality data
Public Access
Public Access – Drill Down100 x 100 Mile Square
Public Access – Drill Down25 x 25Mile Square
Public Access – Drill Down6.25 x 6.25Mile Square
Public Access – Select Stations
Public Access -- Reports
Public Access – Text Reports
Reports & Data Extractions
Reports & Data Extractions
Avaliable in:Avaliable in: HTMLHTML CSV (Comma Separated Values)CSV (Comma Separated Values) Pipe delimited “|” (text) formatPipe delimited “|” (text) format
Reports & Data ExtractionsStation Inventory Report
Reports & Data Extractions
Sampling History Report
Assessments
Assessments
Integrates SWQMIS with the statistical Integrates SWQMIS with the statistical SAS tools in order to:SAS tools in order to: Summarize data Summarize data Calculate various statisticsCalculate various statistics Provide reports for assessors Provide reports for assessors
Assessments
Assessments – Business Logic
Identify and convert preferred Identify and convert preferred parameters to correct unitsparameters to correct units
Handle profile samples appropriately Handle profile samples appropriately Identification of the mixed surface layerIdentification of the mixed surface layer
Summarize data and calculate statistics Summarize data and calculate statistics Mean, geometric mean, median, standard Mean, geometric mean, median, standard
deviation, percentiles, confidence intervalsdeviation, percentiles, confidence intervals
Assessments – Business Logic (cont.)
Determine appropriate criteria and Determine appropriate criteria and identify exceedancesidentify exceedances Basin-specificBasin-specific Segment-specificSegment-specific Site-specific Site-specific Formulas (acute & chronic metals)Formulas (acute & chronic metals)
Ascertain waterbody’s support statusAscertain waterbody’s support status
Provide full documentation for all steps Provide full documentation for all steps of the processof the process
Data Flow
Data Flow
Transfer of core data to EPA Transfer of core data to EPA automatically on a periodic basis automatically on a periodic basis through web services and the Central through web services and the Central Data Exchange (CDX via the Water Data Exchange (CDX via the Water Quality Exchange (WQX)Quality Exchange (WQX)
Data Flow
Actively flowing data to the EPA since Actively flowing data to the EPA since September, 2007September, 2007
Fully automated processFully automated process
Currently transfer physical and chemical Currently transfer physical and chemical data data
Will transfer biological data at a later Will transfer biological data at a later datedate
Administration
Administration
Track the “history” of data changesTrack the “history” of data changes
Provide interfaces that allow a database Provide interfaces that allow a database administrator to manage: administrator to manage: UsersUsers Security Security Reference data Reference data
Lessons Learned
Allow design time that is adequate for Allow design time that is adequate for the scope the scope
Employ tried technologyEmploy tried technology
Use agile development techniquesUse agile development techniques
Start third party integration Start third party integration communication and testing earlycommunication and testing early