epic tools for in-situ data collections

EPIC ToolsEPIC Toolsfor for in-situin-situ data collections data collections

Donald W. Denbo, NOAA/PMELWilla H. Zhu, NOAA/PMEL/JISAO

John Osborne, OceanAtlas SoftwareChristopher Moore, NOAA/PMEL/JISAO

NOAA Coastal Storms Initiative MeetingNOAA Coastal Storms Initiative MeetingMay 6-8, 2003May 6-8, 2003

EPIC ComponentsEPIC Components Data Archive and RetrievalData Archive and Retrieval

Multiple file formats supported with “epslib”Multiple file formats supported with “epslib” Metadata database supports subsetting of large data collectionsMetadata database supports subsetting of large data collections

““Classic EPIC” Classic EPIC” Unix programs for desktop display, analysisUnix programs for desktop display, analysis Gateways to Matlab, PPLUS, Ferret, IDL, etc.Gateways to Matlab, PPLUS, Ferret, IDL, etc.

EPIC Web BrowserEPIC Web Browser Search, graph, list, download from the WebSearch, graph, list, download from the Web Toolkit for building custom web applicationsToolkit for building custom web applications

Java applicationsJava applications Java OceanAtlas - Sophisticated browse and data explorationJava OceanAtlas - Sophisticated browse and data exploration Climate Data Portal - distributed data access (CORBA/OPeNDAP)Climate Data Portal - distributed data access (CORBA/OPeNDAP) OceanShare - Network data collaboration toolOceanShare - Network data collaboration tool ncBrowse - Generalized browse tool for data/attributes (local files, ncBrowse - Generalized browse tool for data/attributes (local files,

network files, OPeNDAP files)network files, OPeNDAP files)

EPIC Archive and RetreivalEPIC Archive and Retreivalfor Classic EPIC, Web, and Java Applicationsfor Classic EPIC, Web, and Java Applications

Data collections may include thousands Data collections may include thousands or even millions of profiles or time seriesor even millions of profiles or time series

Typically the researcher will use EPIC Typically the researcher will use EPIC tools to:tools to: Select data by lat, lon, time, keyword, etc.Select data by lat, lon, time, keyword, etc. Refine the initial data selection Refine the initial data selection Display and/or analyze the selected dataDisplay and/or analyze the selected data

Pacific

NODC

Alaska

Meta DataData

File Collections

Data LoaderUtility

Selecteddatasets pointers

to data files

Data SearchLat, long, timekeyword, etc

creates

DataSearch

andSelection

“Classic EPIC”

programs

MatlabFerret

IDLPPLUS

EPIC Web

Java OceanAtlasClimate Data Portal

OceanSharencBrowse

MySQLdatabase

EPIC Data Management EPIC Data Management StrategyStrategy

EPIC data usageEPIC data usage

EPIC Data File Input/OutputEPIC Data File Input/Output

Applications programs

API

File Independent Layer

FDL

Existing Planned

Data file formatData file format Multiple data file formats Multiple data file formats

are supportedare supported New formats are readily New formats are readily

addedadded

Applications software Applications software Independent of file formatIndependent of file format API for most API for most

programming languagesprogramming languages

Gateways Gateways to Matlab and other to Matlab and other

graphics/analysis graphics/analysis packagespackages

EPS Library

Classic EPICClassic EPICSuite of Unix programs for the desktopSuite of Unix programs for the desktop

Over 100 EPIC programs are available for:Over 100 EPIC programs are available for: Data selection, metadata and file editing...Data selection, metadata and file editing... GraphicsGraphics

Time series, multivariable plots, section plots...Time series, multivariable plots, section plots...

Parameter calculationParameter calculation Dynamic height, geostrophic velocity...Dynamic height, geostrophic velocity...

Statistical, spectral, tidal harmonic analyses...Statistical, spectral, tidal harmonic analyses...

EPIC’s modularity allows a researcher to EPIC’s modularity allows a researcher to easily add new analysis techniques or use easily add new analysis techniques or use a favorite graphics or analysis program.a favorite graphics or analysis program.

EPIC Web BrowserEPIC Web Browser

EPIC Web BrowserEPIC Web BrowserData selection, graphics, downloadData selection, graphics, download

EPIC Web Browser CapabilitiesEPIC Web Browser Capabilities

PlotsPlots User customized plots for profiles and User customized plots for profiles and

time series time series (T, S, Sigma-t, Ox) vs Pressure/Depth(T, S, Sigma-t, Ox) vs Pressure/Depth Temperature vs SalinityTemperature vs Salinity One or two user-selected Properties vs One or two user-selected Properties vs

Pressure/DepthPressure/Depth Contoured section plotsContoured section plots

List/download data, metadataList/download data, metadata New plots and functions readily includedNew plots and functions readily included

EPIC tools for the Web EPIC tools for the Web for building customized web access to datafor building customized web access to data

EPIC Web ToolkitEPIC Web Toolkit Interface design Backend graphics Map data selector

Example applicationsExample applications EPIC Web Browser (general) TAO (El Nino buoys) Unaami (Arctic time series) OSCAR (Satellite ocean

currents) Multibeam data

(hydrography) http://www.epic.noaa.gov

EPIC Java Tools and ApplicationsEPIC Java Tools and Applications

Distributed data accessDistributed data accessand and

Advanced data browsing Advanced data browsing

Java ToolsJava Tools

EPIC Java EPIC Java ApplicationsApplications

for locating and selecting for locating and selecting from distributed data from distributed data

collectionscollections

ChallengeChallenge

Locating and selecting individual Locating and selecting individual observations from large, distributed observations from large, distributed collections of collections of in situin situ data data observationsobservations Examples: Individual oceanic or

atmospheric profiles, time series, buoy tracks, etc.


Some solutionsSome solutions

Tools and ApplicationsTools and Applications LDAP Directory Services NdEdit for subsetting large in-situ collections Interactive Java graphics Intensive exploration of multiple data types OPeNDAP (DODS) clients and servers for in-situ data Access from Desktop and Web

Assembled from OpenSource componentsAssembled from OpenSource components http://www.epic.noaa.gov/epic/software/ep_java.htm


LDAP Directory ServiceLDAP Directory ServiceFinding distributed data collectionsFinding distributed data collections

Search on geographic region, observed quantities or Search on geographic region, observed quantities or keywordkeyword Automatically populated from data serversAutomatically populated from data servers Metadata instantly known to LDAP directoriesMetadata instantly known to LDAP directories Access control for experimental data productsAccess control for experimental data products

Open SourceOpen Source Standards-basedStandards-based Platform independent Platform independent Scalable, extensible Scalable, extensible Supports desktop or web clientSupports desktop or web client


X-Y map view

Y-TY-T time range

NdEdiNdEditt

Investigate space-time distribution

Select data in any View

Explore Data Locations and Subset Data Collections

Interactive, linked windows

View any 2D sliice xy, xt, xz, yt, yz, yt Select

time rangeSee data in range

X-Y map view

Y-TY-T time range


Section Tool - select stations along a depth contour

NdEdiNdEditt


Selected stations are in a narrow time band

NdEdiNdEditt


Polygon tool – select an oblique section line

See time range of stations

NdEdiNdEditt


NdEdiNdEditt

NdEdit is an interactive Java toolNdEdit is an interactive Java tool NdEdit is being re-engineered for the WebNdEdit is being re-engineered for the Web


SGT: SGT: A Scientific Graphics ToolkitA Scientific Graphics Toolkit Java toolkit for scientific graphicsJava toolkit for scientific graphics

On-line documentation, examples, tutorials, streaming On-line documentation, examples, tutorials, streaming videovideo

JavaBeans in next releaseJavaBeans in next release FeaturesFeatures

Contour, line and vector plots, Contour, line and vector plots, Axes user settable or automatically scaledAxes user settable or automatically scaled multiple Time Axis formats multiple Time Axis formats mouse-movable, customizable Labels, line keys, iconsmouse-movable, customizable Labels, line keys, icons Legend generation automaticLegend generation automatic

Widely used, freely availableWidely used, freely available scientific and commercial applications use SGTscientific and commercial applications use SGT 14,782 downloads by 5,151 unique sites from 70 14,782 downloads by 5,151 unique sites from 70

countriescountries http://www.epic.noaa.gov/java/sgt/http://www.epic.noaa.gov/java/sgt/EPIC Java EPIC Java

ApplicationsApplications

Java ApplicationsJava Applications

for the Desktop and the for the Desktop and the NetworkNetwork


ncBrowsencBrowseJava tool for data/metadata browseJava tool for data/metadata browse

NetCDF and full OPeNDAP supportNetCDF and full OPeNDAP support Desktop Java application Axis mapping, parametric plots, flexible mapping of

independent variables to axes Line, vector and contour plots Animations, 3D graphics via VisAD

Popular, freely availablePopular, freely available 8,187 downloads by 3,415 unique sites in 59 countries OPeNDAP (DODS) client, Unidata netCDF client, WOCE

data CD, etc. http://www.epic.noaa.gov/java/ncBrowse/


Java OceanAtlasJava OceanAtlasfor browsing ocean datafor browsing ocean data

Intensive Java desktop browsing of ocean dataIntensive Java desktop browsing of ocean data Property-property plots with up to 7 x axes, waterfall diagrams, Property-property plots with up to 7 x axes, waterfall diagrams,

Contour plots/residual contour plots, Map plots with optional Contour plots/residual contour plots, Map plots with optional "bubble" isosurface plots, Linked browsing of all plot types, Built-in "bubble" isosurface plots, Linked browsing of all plot types, Built-in and custom calculations, NdEdit for data location/selectionand custom calculations, NdEdit for data location/selection

Reads WOCE, EPIC, OPeNDAP netCDF data, some ARGO formatsReads WOCE, EPIC, OPeNDAP netCDF data, some ARGO formats Extended for wide range of data typesExtended for wide range of data types

Profile, time series, drifting buoy, gridded analyses, model outputsProfile, time series, drifting buoy, gridded analyses, model outputs Widely used, freely availableWidely used, freely available

http://www.epic.noaa.gov/epic/software/JavaJOA.htmhttp://www.epic.noaa.gov/epic/software/JavaJOA.htm http://odf.ucsd.edu/joa/jsindex.htmlhttp://odf.ucsd.edu/joa/jsindex.html


Climate Data PortalClimate Data PortalDistributed ocean observationsDistributed ocean observations

Major Major in situin situ data data collections:collections: TAO El Nino buoy data

(realtime/PMEL), Global sea level data (realtime/U of Hi), GTSPP (NODC), PMEL hydrographic data and time series collection, WOCE ADCP (U of Hi), IPRC/APDRC in situ data, US Coastal Ocean Time Series database (NODC)

OPeNDAP: in-situ data server and clientOPeNDAP: in-situ data server and client LDAP directory services NdEdit for data selection/subsetting Data Servers, Clients (Java and Web) – focus on in

situ data http://www.epic.noaa.gov/cdp/cdpjava.htmEPIC Java EPIC Java

ApplicationsApplications

OceanShareOceanShareNetwork collaboration toolNetwork collaboration tool

Collaboration featuresCollaboration features Local, remote data file access and Java graphicsLocal, remote data file access and Java graphics Standard collaboration tool features (annotation, Standard collaboration tool features (annotation,

whiteboard, etc) whiteboard, etc) Secure LDAP Document RepositorySecure LDAP Document Repository

Proposals, papers in progress, data, imagesProposals, papers in progress, data, images Access control by document or by folderAccess control by document or by folder Accessible from web and desktop Accessible from web and desktop

OPeNDAP compatible soonOPeNDAP compatible soon http://www.epic.noaa.gov/collab/http://www.epic.noaa.gov/collab/


EPIC Java Tools and Applications EPIC Java Tools and Applications

Modular Web and Open Source Modular Web and Open Source componentscomponents

Java interactivityJava interactivity Low cost development/maintenance Low cost development/maintenance OpenSource, standards-basedOpenSource, standards-based Scalable, extensibleScalable, extensible OPeNDAP supported or plannedOPeNDAP supported or planned

http://www.epic.noaa.govhttp://www.epic.noaa.gov http://www.epic.noaa.gov/epic/software/http://www.epic.noaa.gov/epic/software/

ep_java.htmep_java.htm

epic tools for in-situ data collections

Documents

situ data collectionsdonald

situ data observationsexamples

epic web browsersearch

epic toolsfor

customized web access

epic componentsdata

file editing

retreivalfor classic