SSDS: Data for Science

A Walkthrough of Proposed

SSDS Capabilities

4 April 2002John Graybeal

2002.04.04

2

Topics

What you want to hear:• What data is in SSDS• How to access data• How to display data• How to command

instruments

What else you should know:• How easy to use is it?• Are we sure the data’s OK?

– Raw data always available?

– Is it reliable? Is time right?

• What if there’s a problem?– Can we tell what happened?

– Can we gracefully recover?

• Is data distributable/secure? • What aren’t you getting?

2002.04.04

3

What Data is Available?

1. All data produced by MOOS instruments• Data is available ‘right away’ if sent to shore, or• Data could be loaded later, directly from device

2. Other data which has been submitted to SSDS• Submitted data must follow basic ISI/SSDS guidelines• Can be brand new (e.g., calibrations), or derived (e.g.,

from other SSDS data)

3. “Metadata” (descriptive info) about the above Notes

– SSDS should not replicate external data stores– Someday could re-process existing MBARI data– Operational data can also be sent to SSDS and ingested

2002.04.04

5

How To Access the Data?

• Ask (catalog) for data of interest (search by device, date, data item name, or combination)

• Choose a data set (sets?) of interest, click to access– Probably multiple text formats—what’s important?

(ASCII CSV? ODV? netCDF? other?) – Do you need to monitor or process ‘streaming’ data?

• What more advanced features are needed? Desired?– Displaying same item across multiple data sets?– Selecting specific items or times within data set? – Processed data products…Sub-setting or interpolating

data by time or item? Averaging? Filtering? …?– Combining 2 data sets using time as reference?

2002.04.04

6

How to Display the Data?

• Basic plots will be available via web interface– Quick look in the truest sense– We don’t want to create yet another plotting program

• Data will be available to existing tools– Minimum capability is usable files (ASCII, netCDF, ?)– Ideal is to embed SSDS data access directly into tools

• In this model, software within Matlab (for example) can open anything in the archive

• Browsing from within application would be a big plus• Some (many) tools may do this for free; others we can ‘help’

• Before discussing further, you should understand the way we want SSDS (and MOOS) to work

2002.04.04

7

MOOS Data Architecture

Devices

ObservingPlatform

Shore Side Data System

User

Applications

Applications

(User Tools)

DataPresentation

Communications

Archiving

Applications/ Interfaces

101110110011

110234999

Data line 1more datalast data

OceanOceanSideSide

ShoreShoreSideSide

Cataloging

2002.04.04

8

How to Access Instrument(by the way, it’s not an SSDS task)

Devices

ObservingPlatform

Shore Side Data System

User

Applications

Applications

(User Tools)

DataPresentation

Communications

Archiving

Applications/ Interfaces

101110110011

110234999

Data line 1more datalast data

OceanOceanSideSide

ShoreShoreSideSide

Cataloging

2002.04.04

9

How Data Access Works

Devices

ObservingPlatform

Shore Side Data System

User

Applications

Applications

(User Tools)

DataPresentation

Communications

Archiving

Applications/ Interfaces

101110110011

110234999

Data line 1more datalast data

OceanOceanSideSide

ShoreShoreSideSide

Cataloging

2002.04.04

10

How Data Access Works1. SSDS automatically notified of instrument information

– Instrument qualification and installation on MOOS– Instrument configuration (default settings, changes)– Data record descriptions (syntactic and semantic)– Arrival of new data records

2. SSDS automatically catalogs, archives all arriving data 3. Users search catalog for data of interest

– References to archived data returned with search results– Source data can be accessed via the references

4. User can then view (or subscribe to?) the source data– Various formats provided, including basic plots– Connections to advanced presentation packages supported

2002.04.04

11

Topics

What you want to hear:• What data is in SSDS• How to access data• How to display data• How to command

instruments

What else you should know:• How easy to use is it?• Are we sure the data’s OK?

– Raw data always available?

– Is it reliable? Is time right?

• What if there’s a problem?– Can we tell what happened?

– Can we gracefully recover?

• Is data distributable/secure?

• What aren’t you getting?

2002.04.04

18

Conclusions• SSDS should improve data management for all users

– At minimum, easier access to your own data and plots– Straightforward access to all MBARI data (references)– More reliable data storage, time references, metadata links– Better long-term usability (gives us more time)

• Development will be incremental– Full-featured release targeted for MSE 2003– Prototypes will exist before then (soon!), but may evolve– Features will grow with third-party solutions

• Many questions about first-order science priorities– Which general-purpose functions do you really need?– What are most useful data formats? application interfaces?– How important is fine-grained access security?