Integration through Data Integration through Data ManagementManagement

The “SEACOOS experience”Accomplishments and Obstacles

SEACOOS approach to data integration Accomplishments and contributions to

IOOS interoperability Some “lessons learned” Coordination with nascent Regional

Association

SEACOOS was initiated in 2002 with ONR funding

to develop a coastal ocean information system

for FL, GA, SC and NC.

SEACOOS Over-Arching Goal: To significantly increase the quantity and quality of environmental information from the coastal ocean of the SE U.S. and facilitate its use in a wide range of societal, scientific, and educational applications.

Observational Platforms Contributed by SEACOOSPartners

Observational Platforms Contributed by SEACOOSPartners

Real-Time Observations: SE Coastal Ocean

SEACOOS Modeling Coordinating model

simulations forthe entire region

SEACOOS Information SEACOOS Information ManagementManagement

Data Management Coordinating Committee (DMCC)

CoordinationMadilyn Fletcher (USC), Dwayne Porter (USC),

P.I. representatives Ed Kearns (U Miami), Mark Luther (USF), Harvey Seim (UNC),

Nick Shay (U Miami), Jim Nelson (SkIO)People who get it done

Charlton Purvis (USC), Jeremy Cothran (USC), Vembu Subramanian (USF),

Jeff Donovan (USF), Sara Haines (UNC), Jesse Cleary (UNC), Liz Williams (U Miami), Tom Cook (U Miami)

Web site documentation /portalChris Calloway (UNC), Claire Eager (UNC)

SEACOOS achievements in SEACOOS achievements in regional Data Managementregional Data Management

“Data commons” -- Protocols for data providers to make their information available. A set of standards for use with a given file format

Methods for data aggregation and display Methods for data aggregation and display

TogetherTogether – a demonstration of how an – a demonstration of how an RCOOS can function to aggregate and RCOOS can function to aggregate and display informationdisplay information

Data “commons”Data “commons”

netCDF adopted as the common data language(“SEACOOS CDL”, currently version 2)

-- well supported (UCAR, Unidata, Boulder, CO)

-- supported under DODS/OPeNDAP-- interfaces to programming and analysis packages (e.g., Perl, Matlab)-- a flexible system for a variety of platforms/sensor types

(scalar time series, vector time series, vector profiler time series,scalar gridded maps, vector unstructured maps)

Data “commons”Data “commons”(cont.)(cont.)

“Data Dictionary”-- table that registers known standards

with each other-- provides an English language description-- “down-the-road” advantages:

options for data providers; flexibility in presentation and interpretation of information(i.e., if format is registered, the providers’ data can be represented)

Data Management:Data Management:the Nuts & Boltsthe Nuts & Bolts

Data is aggregated and storedData is aggregated and stored Data is normalizedData is normalized Data is visualizedData is visualized Data is disseminatedData is disseminated

Data Aggregation & Data Aggregation & StorageStorage

Aggregation format flavorsAggregation format flavors netCDF (in situ & RS data; model netCDF (in situ & RS data; model

output)output) PNG (RS imagery)PNG (RS imagery)

StorageStorage Relational database (in situ, model Relational database (in situ, model

output, some RS)output, some RS) Files (RS imagery)Files (RS imagery)

TechnologyTechnology PerlPerl PostgreSQLPostgreSQL

& PostGIS& PostGIS

SEACOOS Data Train A central aggregation site as opposed to distributed network (e.g.,

LAS) Link to software community developing GIS-type applications Powerful visualization tool without limits to numbers of layers

NormalizationNormalization Reporting time Reporting time

variesvaries In-situ dataIn-situ data

E.g. daily, hourly,E.g. daily, hourly,half-hourly, every 10 half-hourly, every 10 minutesminutes

Remotely-sensed dataRemotely-sensed data E.g. twice dailyE.g. twice daily

Reporting area Reporting area variesvaries Remotely-sensedRemotely-sensed

data passesdata passes

Round-the-clock Round-the-clock updating is updating is resource resource intensiveintensive Balance the Balance the

server loadserver load

Visualization : Example 1Visualization : Example 1 Production siteProduction site

http://www.seacoos.orghttp://www.seacoos.org

Visualization : Example 2Visualization : Example 2 Development siteDevelopment site

http://nautilus.baruch.sc.edu/rshttp://nautilus.baruch.sc.edu/rs

Main enginesMain engines PHPPHP

PHP-MapScriptPHP-MapScript PerlPerl MapServerMapServer

Visualization Fun : Visualization Fun : Graphs & AnimationsGraphs & Animations

Ad-hoc time-series Ad-hoc time-series graphsgraphs

Ad-hoc animationsAd-hoc animations

Hurricane/Tropical Storm Jeanne25 - 27 September, 2004

Merged wind observations and shore radar

DisseminationDissemination

OPeNDAP (DODS) data accessOPeNDAP (DODS) data access OGC-friendly: WMS, WFSOGC-friendly: WMS, WFS

pick a layer, any layer pick a layer, any layer http://nautilus.baruch.sc.edu/http://nautilus.baruch.sc.edu/seacoos_misc/show_sea_coos_obs_time_ranges.phpseacoos_misc/show_sea_coos_obs_time_ranges.php

Dissemination : example Dissemination : example 11

NC OneMapNC OneMapViewer Viewer http://gisdata.usgs.net/http://gisdata.usgs.net/website/NC_OneMap/website/NC_OneMap/viewer.aspviewer.asp

Dissemination : example Dissemination : example 22

IntegratedIntegratedOceanOceanObservingObservingSystemSystemhurricanehurricanedemo demo http://www.openioos.orghttp://www.openioos.org

Some Lessons :Some Lessons :Interoperability Interoperability DemonstrationsDemonstrations Communication is key : an open source attitude

Mailing lists, bulletin boards, and wiki’s proved invaluable across internal SEACOOS projects and external projects, e.g. Interoperability Demonstrations

Successfully accomplished (capabilities were demonstrated), but had “fire-drill” aspects from perspective of technical personnel Better coordinated in Summer 2004 than previously, a more

mature effort Better adaptation of OGC WMS and WFS technologies Additional layers A better product “under the hood”

However, the push to get out products in the short-term was done at the expense of thoroughness

Process to add new products, identify sources seen as somewhat haphazard

Not the way to build an operational system

Other Obstacles / Issues QA/QC of real-time data and archives

Needs further development; compliant with national standards

Resource availability and allocation Many personnel have multiple roles Crucial expertise in a small number of people Time for documentation Need for redundancy in comm./processing streams

Data from national providers not always in a readily accessible form (often requires “screen

scrapes”)

Engaging users / getting feedback “In-reach” within the regional program as well as with

external users Track users and map results

Other SEACOOS regional DM activities

Coordination with NDBC, NWS e.g., data push from COMPS, SABSOON Stations part of NDBC network

Coordination with developing Regional Association (SECOORA) Sensor/equipment inventory Data dictionary (documentation)

Development of metadata tool/metadata system for the RCOOS (“MetaDoor” & documentation, USC)

Coordination with State Agencies

Equipment Inventory

Full Search

Query Returns: DB records

Query Returns: Map

User Interface ComponentsQuick Search – by variable measured

Equipment InventoryAdministrative Interface

1. Institution: Institution Name, Abbreviation, Observing Group URL, Affiliation

2. Contacts: Contact Name, Phone Number, Email Address, Contact level

3. Stations: Institutional Station ID, Station Name, Lat/Long

4. Equipment: Package Description, Manufacturer, Model, Equipment Type, Vertical Position, Power Requirement, Communication Type, Near Real-time Status, Variables Measured, Change/Calibration History, Active Status, Comments

This interface provides database access to equipment managers to add and edit the following information.

Data input and involvement from federal data Data input and involvement from federal data providers for a more accurate picture of the providers for a more accurate picture of the regional observation system (assets, gaps, regional observation system (assets, gaps, performance)performance)

Connecting equipment data as metadata to support Connecting equipment data as metadata to support observations and the QA/QC process. Exploration observations and the QA/QC process. Exploration of SensorML (XML) and MetaDoor (metadata tool of SensorML (XML) and MetaDoor (metadata tool developed at USC).developed at USC).

Improved data intake capability: automated batch Improved data intake capability: automated batch processing for new equipmentprocessing for new equipment

Expanded interactive map functionality: Zoom/PanExpanded interactive map functionality: Zoom/Pan

Equipment inventoryNext Steps

Metadoor : metadata Metadoor : metadata creationcreation

Metadoor : entry context Metadoor : entry context featuresfeatures

- Collapsible forms organized into tabbed sections

- Fields color coded by requirement need

- Help context link for each field- Listbox selections and calendars- Form presents only subfields needed

based on listbox selection

Metadoor : planned Metadoor : planned metadatametadata

support forsupport for marineXMLmarineXML sensorMLsensorML

Platforms and SensorsPlatforms and Sensors Observations and MeasurementsObservations and Measurements OGC Sensor WebOGC Sensor Web

Other metadata harvestersOther metadata harvesters OGC Catalog servicesOGC Catalog services geodata.govgeodata.gov

A friend in the businessA friend in the business Development site: Development site:

http://nautilus.baruch.sc.edu/rshttp://nautilus.baruch.sc.edu/rs Production site: http://www.seacoos.orgProduction site: http://www.seacoos.org Very active listserv dealing with mainly remote-Very active listserv dealing with mainly remote-

sensing issues: remotesensing@caro-coops.orgsensing issues: remotesensing@caro-coops.org mailto: majordomo@caro-coops.orgmailto: majordomo@caro-coops.org message text: subscribe remotesensingmessage text: subscribe remotesensing

Who am I?Who am I? Charlton Purvis, University of South Carolina, Charlton Purvis, University of South Carolina,

SEACOOSSEACOOS cpurvis@sc.educpurvis@sc.edu Happy to help and share.Happy to help and share.