WP-8, ZIB

WP-8: Data Handling And Visualization

Review Meeting Report

Felix Hupfeld, Andrei Hutanu, Andre Merzky, Thorsten Schütt, Brygg Ullmer

Zuse-Institute-Berlin

WP-8, ZIB

Overview

Formal Matters

Visualization

Tangible Devices

Remote File Access

Replica Management / File Movement

Metadata Service

WP-8, ZIB

Formal Matters

Deliverables: most right on time, some delivered earlier

Milestone 8.2 (M24): Dynamic data access techniques are designed and implemented

Replica catalog (Location independant file access)Remote File Access (Pitfalls and GridFTP)Visualization service

Plan for the rest of this year:Further integration with other WPsImplement additional features (user management)

WP-8, ZIB

Formal Matters (II)

B. Ullmer, A. Hutanu, W. Benger, H.-C. Hege. (2003). Emerging Tangible Interfaces for Facilitating Collaborative Immersive Visualization. Workshop on Collaborative Virtual Reality and Visualization, October 2003.

H.-C. Hege, A. Hutanu, R. Kähler, A. Merzky, T. Radke, E. Seidel, B. Ullmer. (2003). Progressive Retrieval and Hierarchical Visualization of Large Remote Data. Workshop on Adaptive Grid Middleware, September 2003.T. Schütt, A. Merzky, A. Hutanu, F. Schintke, Remote Partial File Access Using Compact Pattern Descriptions, IEEE CCGrid 2004.

WP-8, ZIB

Visualization context

Grid = “virtualization” of resources

Broad progress toward virtualization of hardware resources (compute power, disk space, network)

Less common: virtualization of graphics hardware, software

“Virtualization” of visualization resources

Scientific visualization has demanding hardware and software requirements, unavailable to many users

Software is frequently complex and “heavy-weight”

Our approach:Automation for core visualization operations

Provision of grid-based “visualization services”

Develop simplifying mobile services & devices

WP-8, ZIB

Visualization and mobile devices

Mobile devices• Access to viz services

from mobile phones, PDAs

• Development of new grid-enabled “mobile”viz devices

Visualization• Access to grid resources

from existing viz applications

• Access to existing viz applications from the grid

WP-8, ZIB

Visualization service

1D

2D

3D

WP-8, ZIB

Mobile + viz = “viz artifacts”

MotivationAdvantages of stereo, collaborative interactionImportance of the meeting room, teleconferencingSimplifying interaction + gaining real estate by breaking out into the physical world

WP-8, ZIB

Data cards

Represent online data, user credentials, remote users, parameters, etc.

WP-8, ZIB

Interaction pads

targ

auth

sens?

net

source target

source

targ

auth

sens?

net

upper left upper right

center

lower left lower right

targ

auth

sens?

net

targ

auth

sens?

net

param 1 param 2

“binding pad” “placement pad”

“parameter pad” “control pad”

WP-8, ZIB

Remote File Access (I)

Needed forVisualization of remote files (intermediate results of large jobs)Job input files (extract and transmit only data of interest)…

Existing mechanismsGridFTP remote partial file access (read, seek, read, seek, .. )

I/O wrapper for local and remote access methods (to be used in GAT)

But does not fit Grid requirements, because

too many small remote operations!

WP-8, ZIB

Remote File Access (II)

Need to execute high-level operations on the server side

File format specific approach(HDF5)HDF5 – structured binary file format.

Datasets, Groups, Attributes

Forwards HDF5 calls to the remote side

Plugin for the Globus GridFTP server

Generic approach (much better ) next slide

WP-8, ZIB

Remote File Access (III)

Generic approach

Compact access pattern description: Pitfalls

Speedup of several orders of magnitude achieved!

Example: Access part of a Cube: (339, 65874, 262144, (0, 511, 2048, 32,

(0, 511, 2048, 32, (0, 3, 16, 32)))

just 1 (in words: one!) remote call

WP-8, ZIB

Replica Management Components

Replica catalogSupports hierarchical data organization

POSIX filesystem like interface

Uses C++ with SOAP/GSI interface

Adaptor inside of GAT

File movement serviceUsed for replication

Based on GridFTP

Adaptive service (WP6)Ranks replica locations

Optimizes GridFTP parameters

WP-8, ZIB

StorageBox Metadata Service

Unified storage for metadata of persistent Grid objects (archived files, job input/output, messages/notifications)

Any number of namespaces, configurable via pluggable access policies as public or private archives, group collaboration spaces or private annotations for public data

Public Sensor Data Archive

User ‘A’: Sensor Data Annotations and Private Messages

Project ‘X’: Collaboration

Space

User ‘B’: Sensor Data Annotations and Private Messages

AB

WP-8, ZIB

StorageBox Metadata Service

Portlet as user interface

Integrated with WP12’s MessageBox service

SOAP web service interface

Pluggable database backendswith PostgreSQL implementation

Portable C++ implementation Win32, Linux, Mac OS X

Planned: Improved query language / processing

Remote administration interface

More access policies

WP-8, ZIB

Continuing plans

Visualization, mobile devicesFurther integration with portal

Implement additional viz operations

Deploy, test both services & devices w/ end users

Data managementProvide a flexible namespace for organizing files

Remote File Access: Generic vs. File Format Dependant

WP-8, ZIB

Questions?

WP-8, ZIB

Visualization and mobile devices

Mobile devices• Access to viz services from mobile phones, PDAs

Completed + demoable from mobile phones, iPAQCompleted + demoable from mobile phones, iPAQ• Development of new grid-enabled “mobile” viz devices

Demoable: use of new devices to load remote content, Demoable: use of new devices to load remote content, activate and control video conference sessions activate and control video conference sessions

Visualization• Access to grid resources from existing viz applications

Completed: remote data access from AmiraCompleted: remote data access from Amira• Access to existing viz applications from the grid

Demoable: grid automation, presentation of reports Demoable: grid automation, presentation of reports integrating content from gnuplot, OpenDX, Amira;integrating content from gnuplot, OpenDX, Amira;grid access to streaming Amira contentgrid access to streaming Amira content

WP-8, ZIB

Videoconferencing example

24.02.04, Berlin-Munich, surgical planning