using fme to compile, validate and maintain a 4 million oil and gas well database
DESCRIPTION
Approximately 32 states and more than four million well bores have been drilled in the United States. For its well data, each state agency must deal with an uncoordinated, autonomous data collection process, data model, and distribution methods. This session discusses how Whitestar uses FME to build an extensive set of dataflow models that regularly ingest the raw data, compute locations, verify elevations, perform data validation checks, and standardize the schema nationwide. We'll also highlight how FME is used to output data to a series of open source version 8.4 postgreSQL database structures.TRANSCRIPT
Using FME to Compile, Validate, and Maintain a Four Million Oil and Gas Well Database
Robert C. White, Jr. President, WhiteStar Corporation
2010:An FME
Odyssey
The Problem
4 Million Wells, 32 States, 7 Provinces Building a Consistent the Data Structure Pre-processing and Loading Data Data Validation Challenges Update Challenges Export Challenges
The Tools
Documentation from States and Provinces. FME, particularly FME Workbench. POSIX Tools Various Datasets, Custom Programs PostgreSQL – Open Source Database. Last but not Least…
Varieties of Source Data
File Type Occurrences
CSV Files 7
Excel 5
Access Tables 4
Web Site Scrapes 2
Manual Data Entry 4
dBase Files 5
Card Records, EBCDIC 3
Card Records, ASCII 5
Shape Files 3
Arc Export 1
Open Records Appeal 1
Inventing a Data Model
Gathered Available Documentation Scanned all Data Fields
Grouped by Survey Type, Name, Size
Looked at PPDM.
Building the Data Structure
Used PostgreSQL to Input the Structure FME Readers to Determine Field Lengths PostgreSQL Field Types Helped Determine
Validity. Discovered Use of StringSearcher.
E.g., 660FNL 660FWL should be 4 fields.
Pre-Processing Data
POSIX Type Tools, i.e. Linux Tools Convert EBCDIC to ASCII (dd) Edit Large Files to Delete “Junk” (vi) Unzip Files (unzip, uncompress) Untar Files (tar) Pattern Processing (awk)
Data Validation Problems
Dates are Problems… 19000000 02/31/2010 Jan 29-30, 1995 14 Days
Need for some Robust Date Validation Transformers.
Data Validation Challenges
Missing Coordinates Consistently calculate them to a Land Grid using an
External Program.
County Name Lookup (ValueMapper) Non Unique API Numbers Missing API Numbers Check for “Reasonable” Values. Check if You’re in the Geography you Expect.
Am I Within My Boundary?
Calculate a Well Elevation
Updating the Data
Update with Minimal Intervention Save the previous database. Compare New to Old Set the Update Date.
Export Challenges
Software Clients Require Specific Formats GeoGraphix, Petra, etc.
Used Text writer to meet these challenges. Sorted on Formation Top Depth Non-Unique Identifiers.
Summary
4 Million Wells, 32 States, 7 Provinces Building a Consistent Data Structure Pre-processing and Loading Data Data Validation Challenges Update Challenges Export Challenges
Thank You!
Questions?
For more information: Robert White – [email protected] WhiteStar Corporation http://www.whitestar.com
dd.exe – http://www.chryscosome.net/dd Other POSIX tools
http://www.cs.nmsu.edu/~jeffery/win32