a data access framework for esmf model outputs roland schweitzer steve hankin jonathan callahan...

A Data Access Framework A Data Access Framework for ESMF Model Outputsfor ESMF Model Outputs

Roland SchweitzerRoland Schweitzer Steve HankinSteve HankinJonathan CallahanJonathan Callahan

Kevin O’BrienKevin O’BrienAnsley MankeAnsley Manke

20 July 200520 July 2005 2005 ESMF Workshop2005 ESMF Workshop 22

Some MotivationSome Motivation

Model Components produce … Model Components produce … Model Runs which produce … Model Runs which produce … Data and Metadata whichData and Metadata which should/could produce …should/could produce … A data portal which allows user to:A data portal which allows user to:

Search, browse, compare, validate and Search, browse, compare, validate and analyze results …analyze results …

which in turn helps produce insightwhich in turn helps produce insight


Some More MotivationSome More Motivation

ESMF and the ESCurator manage ESMF and the ESCurator manage components, runs and outputcomponents, runs and output

LAS is a candidate to provide a data LAS is a candidate to provide a data portalportal


The Live Access Server (LAS)A highly configurable Web server designed to provide

flexible access to geo-referenced scientific data


LASLAS LAS is an information broker that:LAS is an information broker that:

• ingests descriptions of dataingests descriptions of data data setsdata sets variablesvariables (optional reshuffling of above into new categories) (optional reshuffling of above into new categories) domains: 1d-4ddomains: 1d-4d special constraints (ensemble index)special constraints (ensemble index) rendering hints (smoothing, palettes)rendering hints (smoothing, palettes)

• presents the descriptions to the user in a presents the descriptions to the user in a coherent fashioncoherent fashion

• asks other software to perform tasks for the asks other software to perform tasks for the useruser

• and gives the results to the user and gives the results to the user


Architecture ComponentsArchitecture Components

User Interface Server (UIS)

ProductServer (PS)

Backend Server (BES)

FerretData

(OPeNDAP)Server(FDS)


Data


LAS OffersLAS Offers

Independence from underlying dataIndependence from underlying data Uniform interface for browse Uniform interface for browse

(visualize), compare, validate and (visualize), compare, validate and analyze results for many data typesanalyze results for many data types

Easy way to make connections to Easy way to make connections to metadata and documentation metadata and documentation

Customizable user interfaceCustomizable user interface


Example (Multiple Hierarchies)Example (Multiple Hierarchies)


Example (Collapse Hierarchies)Example (Collapse Hierarchies)

Experimental Development Server


Example (Custom Headers)Example (Custom Headers)

The NVODS Server (http://www.ferret.noaa.gov/nvods)


Example (Adopt a Drifter)Example (Adopt a Drifter)


Return to the MotivationReturn to the Motivation





Data BrowsingData Browsing


Analysis and ComparisonAnalysis and Comparison

Compute Compute differencesdifferences

Can define new Can define new variables that are variables that are the result of some the result of some analysis operation analysis operation (done in delayed (done in delayed mode so only data mode so only data that is needed is that is needed is computed) computed)


Analysis and ComparisonAnalysis and Comparison

Can use FDS to define regriddings (or Can use FDS to define regriddings (or other analysis) to allow more complex other analysis) to allow more complex comparisonscomparisons


ValidationValidation

Use FDS to create samplings of in Use FDS to create samplings of in situ observations which can be situ observations which can be compared to model datacompared to model data

(This is a planned for a future (This is a planned for a future release.)release.)


Detailed AnalysisDetailed Analysis

Can jump to local tool (Matlab, IDL, Can jump to local tool (Matlab, IDL, Ferret, etc.) by requesting scripts for Ferret, etc.) by requesting scripts for that application that application

Scripts encode access to remote LAS Scripts encode access to remote LAS data via FDSdata via FDS

Can take advantage of FDS analysis Can take advantage of FDS analysis to compute some data reductions to compute some data reductions and transfer only the result to the and transfer only the result to the local applicationlocal application


Additional Flexibility beyond Additional Flexibility beyond ConfigurationConfiguration

Use LAS framework with different Use LAS framework with different backend engine (new Product Server backend engine (new Product Server is designed to make this as easy as is designed to make this as easy as possible)possible)

Web-services interfacesWeb-services interfaces• To Product Server from other clientsTo Product Server from other clients• Will allow implementation of “well-Will allow implementation of “well-

known” service like WMSknown” service like WMS



User Interface Server (UIS)

ProductServer (PS)


FerretData

(OPeNDAP)Server(FDS)


Data


Access Control and the GRIDAccess Control and the GRID

The new architecture makes participating as a The new architecture makes participating as a GRID partner easierGRID partner easier

Web services-based authentication schemes can Web services-based authentication schemes can be used by the Product Server and Backend be used by the Product Server and Backend Server to get authentication information (if not Server to get authentication information (if not supplied by a browser certificate).supplied by a browser certificate).

Implemented as another “filter” in a already-Implemented as another “filter” in a already-existing mechanism for filtering on IP-address, existing mechanism for filtering on IP-address, number of accesses, etc.number of accesses, etc.

Configuration metadata can be augmented to Configuration metadata can be augmented to identify allowed access “roles” by dataset and/or identify allowed access “roles” by dataset and/or variable (and maybe by output product).variable (and maybe by output product).


Return to the MotivationReturn to the Motivation





More InformationMore Information

www.ferret.noaa.govwww.ferret.noaa.gov Subscribe to the LAS User’s mailing Subscribe to the LAS User’s mailing

list list


The EndThe End


browser

UI (java/script)

gif,netCDF,etc.

servers

UIS

PS

LAS XML configuration metadata

browserinterface

BES

HTTP Request(later SOAP)

scripts &/orcommands

products

HTTP Response

data

LAS Architecture with OPeNDAPLAS Architecture with OPeNDAP

LegacyOPeNDAP

OPeNDAP Server

browser

OPeNDAP Server

BESBES

LegacyOPeNDAP

UIS = User Interface Server

PS = Product Server

BES = Backend Server

FDS = Ferret Data Server

SOAP

addXMLFDS



User Interface Server (UIS) – keeps User Interface Server (UIS) – keeps browser up-to-date with relevant browser up-to-date with relevant interface widgetsinterface widgets

Product Server (PS) – takes users Product Server (PS) – takes users requests and returns resultsrequests and returns results

Backend Server (BES) – runs legacy Backend Server (BES) – runs legacy application to fulfill requestapplication to fulfill request

Ferret Data Server (FDS) – General Ferret Data Server (FDS) – General OPeNDAP server for LAS dataOPeNDAP server for LAS data


The UI Server ArchitectureThe UI Server Architecture

User Interface ServerUser Interface Server• Installation process translates LAS Installation process translates LAS

configuration into User Interface Widgets need configuration into User Interface Widgets need to present datasets, variables, geo-spatial and to present datasets, variables, geo-spatial and time extents in HTMLtime extents in HTML

• Allows dataset collection to be easily organized Allows dataset collection to be easily organized into any hierarchies the installer choosesinto any hierarchies the installer chooses

• Communicates requests to the Product Server Communicates requests to the Product Server and returns responses (images, files, data and returns responses (images, files, data tables, etc.) to the Web Browsertables, etc.) to the Web Browser

• Many other customization featuresMany other customization features


The Product ServerThe Product Server Product ServerProduct Server

• Has knowledge of the LAS configurationHas knowledge of the LAS configuration• Makes requests from one or more Back-end ServersMakes requests from one or more Back-end Servers• Fully implemented Web-services interface will be Fully implemented Web-services interface will be

available to client writersavailable to client writers• Can interact with a series of Backend-servers to fulfill a Can interact with a series of Backend-servers to fulfill a

single request from the user client single request from the user client Back-end serverBack-end server

• Can produce product itself Can produce product itself oror

• Interact with a single legacy application (like CDAT or Interact with a single legacy application (like CDAT or Ferret) to produce productFerret) to produce product

Requests are communicated via Request and Requests are communicated via Request and Response XML that is exchanged via SOAP.Response XML that is exchanged via SOAP.


FDS (Ferret Data Server)FDS (Ferret Data Server)

FDS is an OPeNDAP serverFDS is an OPeNDAP server Automatically installed with LASAutomatically installed with LAS Therefore every gridded LAS dataset Therefore every gridded LAS dataset

is served via OPeNDAPis served via OPeNDAP Has extended analysis capabilities Has extended analysis capabilities

beyond basic OPeNDAP serversbeyond basic OPeNDAP servers

a data access framework for esmf model outputs roland schweitzer steve hankin jonathan callahan...

Documents

data reductions

data access framework

configurable web server

remote las data

data typeseasy way

advantage of fds analysis

flexible access

new variables