Lessons Learned from Getting Lessons Learned from Getting Data into STORET:Data into STORET:

A State’s PerspectiveA State’s Perspective

Mike Beauchene, State of Connecticut Department of Environmental Protection;Mellony Hoskinson, Dwane Young, RTI; Dave Wilcox, Gold Systems

STORET SuccessSTORET Success

OverviewOverview

Connecticut’s Data Management SystemConnecticut’s Data Management System Re-formatting data to STORET Re-formatting data to STORET

specificationsspecifications– Column MappingColumn Mapping– Crosswalk TablesCrosswalk Tables– QueriesQueries

Challenges/BenefitsChallenges/Benefits

Cro

ssw

alk

The ProcessThe ProcessCHAOS

Database

Queries$$$

Step 1. Create CT DEP Database

Step 2. Populate the database

Step 3. Data AnalysisUpdate database and create crosswalks

Step 4. Build Queries

Step 5. Load to STORET

Now that the process is developed, data can be loaded to the CT DEP database, and easily migrated to STORET.

SIM

Connecticut - 2001Connecticut - 2001

Move data system from hard copy to electronic systemMove data system from hard copy to electronic system Use STORET data structure as guideline for setting up Use STORET data structure as guideline for setting up

Connecticut’s databaseConnecticut’s database– ProjectsProjects– TripsTrips– StationsStations

Key to this process is implementing required STORET data Key to this process is implementing required STORET data elementselements– Analytical methodsAnalytical methods– Gear TypeGear Type– Sample Collection Procedures, etc.Sample Collection Procedures, etc.

Step 1 – CT DEP creates a database to store monitoring data that meets DEP’s needs

Database ElementsDatabase Elements

Projects

Trips

Stations

Samples

Connecticut’s Connecticut’s Data Management SystemData Management System

Assigned Project IDs and appropriately Assigned Project IDs and appropriately tagged all related data under projecttagged all related data under project

Project managers selected from STORET Project managers selected from STORET lists which methods, gear, collection lists which methods, gear, collection procedures, etc. were used on projectprocedures, etc. were used on project

Projects are maintained separately, but all Projects are maintained separately, but all contain same data elements and can be contain same data elements and can be imported into STORETimported into STORET

Step 2 – Populate CT DEP Database

From State to STORET

AnalysisAnalysis

Identify and map CT DEP data elements to the Identify and map CT DEP data elements to the corresponding STORET data elementscorresponding STORET data elements– ProjectsProjects– StationsStations– TripsTrips– ActivitiesActivities

Identify missing data elements that would be Identify missing data elements that would be required by STORETrequired by STORET

Step 3 – STORET Comparison

Match CT DEP values with STORET valuesMatch CT DEP values with STORET values– Accounts for capitalization, abbreviations, etc.Accounts for capitalization, abbreviations, etc.– Provides a means for accounting for data Provides a means for accounting for data

elements that are required by STORET, but is elements that are required by STORET, but is not in DEP’s databasenot in DEP’s database

Can be easily modified to reflect changes Can be easily modified to reflect changes made in current systemmade in current system

Crosswalk Tables are the keys to creating Crosswalk Tables are the keys to creating new STORET formatted queriesnew STORET formatted queries

Crosswalk TablesCrosswalk Tables

Step 3 cont. – Build Crosswalk Tables

Example of how a characteristic in CT DEP’s Database could be broken down in a crosswalk table to provide for the necessary STORET data elements.

Data Select QueriesData Select Queries

Query designQuery design– State data tablesState data tables– Crosswalk tablesCrosswalk tables

Queries take care of conditional situations (i.e. Queries take care of conditional situations (i.e. adding *Non-Detect when necessary)adding *Non-Detect when necessary)

Since the queries run directly on CT DEP Since the queries run directly on CT DEP database, when DEP data changes, so do the database, when DEP data changes, so do the queriesqueries

Provide a means for flagging data in CT DEP Provide a means for flagging data in CT DEP database that has been loaded into STORETdatabase that has been loaded into STORET

Step 4 – Build Queries that use DEP database and crosswalk tables to create SIM ready files for loading into STORET

Load the Data to STORETLoad the Data to STORET

Using SIM and the data queries, DEP can now Using SIM and the data queries, DEP can now load the data into STORETload the data into STORET

Using these same queries, DEP then can flag all Using these same queries, DEP then can flag all of the data (Stations, Activities) that have been of the data (Stations, Activities) that have been loaded to STORET so that they would be excluded loaded to STORET so that they would be excluded from future data loadsfrom future data loads

Step 5 – Load the data into STORET

Challenges to OvercomeChallenges to Overcome

Some data elements discovered during the Some data elements discovered during the data analysis that are required by STORET, data analysis that are required by STORET, and therefore required changes to CT DEP’s and therefore required changes to CT DEP’s database.database.

Crosswalks have to be kept up to date. If Crosswalks have to be kept up to date. If CT DEP adds a new characteristic, then that CT DEP adds a new characteristic, then that characteristic needs to be added to the characteristic needs to be added to the crosswalk.crosswalk.

Benefits to StateBenefits to State

Maintain original data management system Maintain original data management system If there is a need to change the format for the If there is a need to change the format for the

SIM-ready files, then it’s relatively easy to SIM-ready files, then it’s relatively easy to change the queries.change the queries.

DEP doesn’t need to adopt all of the STORET DEP doesn’t need to adopt all of the STORET codes, but rather just needs to keep the codes, but rather just needs to keep the crosswalks up to date.crosswalks up to date.

Now that the data analysis is done, and the Now that the data analysis is done, and the process is established, future data loads to process is established, future data loads to STORET are relatively painless.STORET are relatively painless.