manchester computing supercomputing, visualization & e-science realistic modelling of complex...

Man

ch

este

r C

om

pu

tin

gSup

erc

om

puti

ng,

Vis

ualiz

ati

on &

e-S

cien

ce Realistic modelling of complex Realistic modelling of complex problems on Gridsproblems on Grids

John Brooke (University of Manchester) Peter Coveney PI RealityGrid (University College London) Stephen Pickles (University of Manchester)

Thanks also to the other RealityGrid co-InvestigatorsJohn Darlington (Imperial College)Steve Kenny and Roy Kalawsky (Loughborough University)John Gurd (University of Manchester)Mike Cates (University of Edinburgh)Adrian Sutton (University of Oxford)

http://www.realitygrid.org2

The RealityGrid project

Mission: “Using Grid technology to closely couple high performance computing, high throughput experiment and visualization, RealityGrid will move the bottleneck out of the hardware and back into the human mind.”

Scientific aims: to predict the realistic behavior of matter using diverse

simulation methods (Lattice Boltzmann, Molecular Dynamics and Monte Carlo) spanning many time and length scales

to discover new materials through integrated experiments.


Partners

Academic University College London Queen Mary, University of London Imperial College University of Manchester University of Edinburgh University of Oxford University of Loughborough

Industrial Schlumberger Edward Jenner Institute for

Vaccine Research Silicon Graphics Inc Computation for Science

Consortium Advanced Visual Systems Fujitsu


RealityGrid

User with laptop/PDA (web based portal)

VR and/or AG nodes

HPC resources

Scalable MD, MC, mesoscale modelling

“Instruments”: XMT devices, LUSI,…

Visualization engines

Steering

ReG steering API

Storage devicesGrid infrastructure (Globus, Unicore,…)

Moving the bottleneck out of the hardware and into the human mind…Moving the bottleneck out of the hardware and into the human mind…

Performance control/monitoring


RealityGrid Characteristics

Grid-enabled (Globus, UNICORE) Component-based, service-oriented Steering is central

– Computational steering– On-line visualisation of large, complex datasets– Feedback-based performance control– Remote control of novel, grid-enabled, instruments (LUSI)

Advanced Human-Computer Interfaces (Loughborough) Everything is (or should be) distributed and collaborative High performance computing, visualization and networks All in a materials science domain

– multiple length scales, many "legacy" codes (Fortran90, C, C++, mostly parallel)


Three dimensional Lattice-Boltzmann simulations

Code (LB3D) written in Fortran90 and parallelized using MPI.

Scales linearly on all available resources. Fully steerable. Future plans include move to parallel data

format PHDF5. Data produced during a single large scale

simulation can exceed hundreds of gigabytes or even terabytes.

Simulations require supercomputers High end visualization hardware and parallel

rendering software (e.g. VTK) needed for data analysis. 3D datasets showing snapshots from a

simulation of spinodal decomposition: A binary mixture of water and oil phase separates. ‘Blue’ areas denote high water densities and ‘red’ visualizes the interface between both fluids.


Exploring parameter spacethrough computational steering

Initial condition: Random water/ surfactant mixture.

Self-assembly starts.

Rewind and restart from checkpoint.

Lamellar phase: surfactant bilayers between water layers.

Cubic micellar phase, low surfactant density gradient.

Cubic micellar phase, high surfactant density gradient.


Computational Steering - Why?

Terascale simulations can generate in days data that takes months to understand

Problem: to efficiently explore and understand the parameter spaces of materials science simulations

Computational steering aims to short circuit post facto analysis– Brute force parameter sweeps create a huge data-mining problem– Instead, we use computational steering to navigate to interesting

regions of parameter space– Simultaneous on-line visualization develops and engages

scientist's intuition– thus avoiding wasted cycles exploring barren regions, or even

doing the wrong calculation


Computational steering – how?

We instrument (add "knobs" and "dials" to) simulation codes through a steering library

Library provides:– Pause/resume

– Checkpoint and windback

– Set values of steerable parameters

– Report values of monitored (read-only) parameters

– Emit "samples" to remote systems for e.g. on-line visualization

– Consume "samples" from remote systems for e.g. resetting boundary conditions

Images can be displayed at sites remote from visualization system, using e.g. SGI OpenGL VizServer, or Chromium

Implemented in 5+ independent parallel simulation codes, F90, C, C++


Philosophy

Provide right level of steering functionality to application developer

Instrumentation of existing code for steering– should be easy

– should not bifurcate development tree

Hide details of implementation and supporting infrastructure– eg. application should not be aware of whether communication with

visualisation system is through filesystem, sockets or something else

– permits multiple implementations

– application source code is proof against evolution of implementation and infrastructure


Steering and Visualization

Simulation

Visualization

Visualization

data transfer

Client

Steering library

Steering library

Steering library

Display

Display

Display


ArchitectureCommunication modes:• Shared file system• Files moved by UNICORE daemon• GLOBUS-IO• SOAP over http/https

Simulation

Visualization

Visualization

data transfer

Client

Steering library

Steering library

Steering library

Data mostly flows from simulation to visualization.

Reverse direction is being exploited to integrate NAMD&VMD into RealityGrid framework.


Steering in the OGSA

Steering client

Simulation

Steering library

Visualization

Visualization

Registry

Steering GS

Steering GS

con

nect

publish

find

bind

data transfer

publish

bind

Client

Steering library

Steering library

Steering library


Steering in OGSA continued…

Each application has an associated OGSI-compliant “Steering Grid Service” (SGS)

SGS provides public interface to application– Use standard grid service technology to do steering

– Easy to publish our protocol

– Good for interoperability with other steering clients/portals

– Future-proofed next step to move away from file-based steering or Modular Visualisation Environments with steering capabilities

SGSs used to bootstrap direct inter-component connections for large data transfers

Early working prototype of OGSA Steering Grid Service exists– Based on light-weight Perl hosting environment OGSI::Lite

– Lets us use OGSI on a GT2 Grid such as UK e-Science Grid today


Steering client Built using C++ and Qt library – currently

have execs. for Linux and IRIX Attaches to any steerable RealityGrid

application Discovers what commands are supported Discovers steerable & monitored

parameters Constructs appropriate widgets on the fly

Web client (portal) under development


program lbe

use lbe_init_module

use lbe_steer_module

use lbe_invasion_module

RealityGrid-L2: LB3D on the L2G

VisualizationSGI Onyx

Vtk + VizServer

SimulationLB3D with RealityGrid

Steering API

LaptopVizserver Client

Steering GUIGLOBUS used to

launch jobs

SGI OpenGL VizServer

SimulationData

GLOBUS-IOSteerin

g (XML)

File based communication via

shared filesystem: Steerin

g GUI

X output is tu

nnelled back using

ssh.

ReG steering GUI


Performance Control

application

component1

component2

component3

application performanc

e steerer

component performance steerer




Advance Reservation and Co-allocation:Summary of Requirements

Computational steering + remote, on-line visualization demand:– co-allocation of HPC (processors) and visualization (graphics pipes and processors)

resources– at times to suit the humans in the loop

• advanced reservation

For medium to large datasets, Network QoS is important– between simulation and visualization,– visualisation and display

Integration with Access Grid– want to book rooms and operators too

Cannot assume that all resources are owned by same VO Want programmable interfaces that we can rely on

– must be ubiquitous, standard, and robust

Reservations (agreements) should be re-negotiable Hard to change attitudes of sysadmins and (some) vendors


Steering and workflows

Steering adds extra channels of information and control to Grid services.

Steering and steered components must be state-aware, underlying mechanisms in OS and lower-level schedulers, monitors, brokers must be continually updated with changing state.

How do we store and restore the metadata for the state of the parameter space search?

Human factors are built into our architecture, humans continually interact with orchestrated services. What implications for workflow languages?


Collaborative aspects

Multiple groups exploring multiple regions of parameter space.

How to record and restore the state of the collaboration? How to extend the collaboration over multiple sessions? What are the services and abstractions necessary to

bootstrap collaborative sessions? How do we reliably recreate the resources required by the

services, in terms of computation, visualization, instrumentation and networking.


Integration with Access Grid?

Service for Bootstrappingsession

Contains “just enough”Information to start otherServices, red arrows indicate bootstrapping

Virtual Venues ServerMulticast addressingBridges

Visualization WorkflowWorkflows saved from Previous sessions or Created in this session

Simulation WorkflowWorkflows saved fromPrevious sessions orCreated in this session

Data Source WorkflowWorkflows saved from Previous sessions or Created in this session

Process RepositoryCollaborative processesCaptured using ontologyCan be enacted byWorkflow engines

Application RepositoryUses application specific ontology to describe what in silico processes need To be utilised for the session

Participants location and access rights

Application data, computation and visualization requirements

Who participates?

What do they use?


How far have we got? Linking US Extended Terascale Facilities and UK HPC resources via a Trans-

Atlantic Grid

We used these combined resources as the basis for an exciting project

– to perform scientific research on a hitherto unprecedented scale

Computational steering, spawning, migrating of massive simulations for study of defect dynamics in gyroid cubic mesophases

Visualisation output was streamed to distributed collaborating sites via the Access Grid

Workshop presentation with FZ Juelich and HLRS, Stuttgart on the theme of computational steering.

At Supercomputing, Phoenix, USA, November 2003 TRICEPS entry won “Most Innovative Data-Intensive Application”


Summary

All our workflow concepts are built around the idea of Steerable Grid Services.

Resources used by services have complex state, may migrate, may be reshaped.

Collaborative aspects of “Humans in the loops” are becoming more and more important.

The problems of allocating and managing the resources necessary for realistic modelling are very hard, they require (at present) getting below the Grid abstractions.

Clearly the Grid abstractions are not yet sufficiently comprehensive and in particular lack support for expression of synchronicity.


London University Search Instrument

LUSI is located at and developed by Queen Mary College, University of London

Aim: Find ceramics (e.g. rare earth metal oxides) with interesting / valuable properties (e.g. high temperature superconductivity)

Motivation: theory cannot indicate how to construct a compound with a particular property. Established methodology in pharmaceutical industry uses automated sample generation and testing. Let's apply the same idea in materials science, exploring properties that are difficult to predict: superconductivity, luminescence, dielectric response…

Furnace XY Table Instruments Printer


LUSI - schematic

Database

New materials c

cc

c

Predictions

Neural network

Measured data

Robot

manchester computing supercomputing, visualization & e-science realistic modelling of complex...

Documents

high water densities

days data

data analysis

cubic micellar phase

surfactant bilayers

water layers

lamellar phase

binary mixture of water