opendap 4 data server – hyrax

OPeNDAP Developer’s Workshop Feb 21-23 2007

OPeNDAP 4 Data Server – Hyrax

James Gallagher and Nathan Potter

21 Feb 2007


But first…

• These are Hyrax

“Despite the unexceptional appearance of modern hyraxes, the order has a remarkable prehistoric lineage and hyraxes may be the closest living relatives of the elephant.” WikiPedia.


Overview

• Goals of the Hyrax Project

• Introduction to customization

• Discussion topic


Goals of the Hyrax Project

• Integrate THREDDS catalogs with DAP • Improve security of our data server• Reduce work to maintain the server• Reuse the existing server software• Integrate work with/for Earth System

Grid (ESG) back into the OPeNDAP software

• Improve the server’s performance


Integrate THREDDS

• Requirements:– Standard representation for catalogs– Separation of the logical from the physical data

organization– A no-hands mode » Fully automatic catalogs– Graceful transition from custom to automatic

catalogs– Full THREDDS features – not stripped down– Must use Unidata’s THREDDS software

• Hyrax 1.0.2 has met these.


Improve the Server’s Security• The original data server design was hard to

make ‘hack proof’– Common Gateway Interface (CGI)– Data handlers were run as a subprocess in a

protected shell– Main CGI modules written in Perl

• As the server grew in complexity, these became more pressing issues.

• Improvements– Perl/CGI » Java/Servlet– Subprocess/shell » Specialized IPC


Reduce Maintenance and Reuse Existing Software

• The Perl software was becoming unwieldy – Reimplementation was needed whether in Perl, Java, …

• Considerable effort has gone into writing our data handlers for netCDF, …, and we could not rewrite them.

• Additional code for the handlers had to be minimal (for the most complex handler - asciival, it’s size increased by 21%, format handlers were just half that value)


Integrate ESG Work

• ESG II had developed a flexible server for the DAP– Its capabilities far out stripped the CGI server– Several extensions to OPeNDAP software were

needed

• The changes/extensions to the OPeNDAP software were merged back into our copies of the code and both projects resumed working from the same version of the software.


Improve Performance

• The new server architecture eliminates basic design problems.– Fewer subprocess created, issues with Perl, …

• Response handlers in the BES pass binary objects, not the objects’ network representation

• Better use of HTTP (e.g., persistent connections now supported)


Recap: Goals of Hyrax

• A higher performance server

• Provides true THREDDS catalogs

• A high level of data-handling code reused

• A more secure design

• A more maintainable design that can evolve gracefully


Introduction to Customization

• The OPeNDAP Data Server was designed to be extended.

• Hyrax - can be Extended by:– Writing a handlers for the front end– Writing a handlers for the back end– Directly access the back end


Hyrax Architecture

OLFS

Java Servlet Engine

BES

Unix Daemon

BES Commands

XML- encapsulated object

File system with data files,SQL Database, …

DAP2

THREDDS

HTML

Optional THREDDS catalogs


Adding a new BES Handler

• Add support for a new data format or data source (relational data base)

• Build a gateway to another data system (such as a WCS server)

• Support new response types (such as netCDF 3 files)

• Add new commands• Patrick West will talk about this next.


Direct access to the BES

• Use this to build a connection-oriented system

• Build new front end systems

• And more…

• Jose Garcia will talk about this following Patrick.


Adding a new OLFS Handler

• OLFS – Dispatch activity abstracted– Dispatch figures out what they want– Uses a handler to figure out how to do it


Inside OLFS

Dispatch DispatchHandler


Inside OLFS

Dispatch

GET

POST

DispatchHandler

Network Interface


Inside OLFS

Dispatch

GET

POST

DispatchHandler

Network Interface Uniform Dispatch Interface


Inside OLFS

Dispatch

GET

POST

DispatchHandler

HttpDispatchHandler

ThreddsDispatchHandler

SoapDispatchHandler



Inside OLFS

Dispatch

GET

POST

DASDDScatalog

SOAP

DispatchHandler

HttpDispatchHandler


SoapDispatchHandler



Inside OLFS

Dispatch

GET

POST

DASDDScatalog

SOAP

DispatchHandler

HttpDispatchHandler


SoapDispatchHandler

Network Interface Uniform Dispatch Interface Various sources of information

BES

BES

Crawlable…


Summary

• BES Handler: Add support for a new data format, response type, commands or to build a gateway

• OLFS Handler: Build support for a new network protocol while (re)using the existing software and/or interact with a new data source

• Use the BES directly: Build new front–ends (web portals, LAN-based systems)


Recap: Hyrax Architecture

OLFS

Java Servlet Engine

BES

Unix Daemon

BES Commands

XML- encapsulated object

File system with data files,SQL Database, …

DAP2

THREDDS

HTML

Optional THREDDS catalogs


Discussion Topic: How Could Hyrax Support the WCS Protocol?

• A BES Handler could be written to support new response formats (i.e., the well-known binary formats)

• A BES command set could be written for operations unique to WCS

• An OLFS DispatchHandler could be built to use the new BES commands

opendap 4 data server – hyrax

Documents

data system

new data format

data files

data serverreduce work

physical data organizationa

format handlers

response handlers

flexible server