dr chris maynard epcc [email protected] +44 131 650 5077 grid@epcc joining the dots

Dr Chris MaynardEPCC

[email protected]+44 131 650 5077

Grid@epcc

Joining the dots

mailto:[email protected]

Introduction

Edinburgh - Tsukuba Workshop 2

The ideas of grid computing are everywhereeven if the actual grids are not as pervasive

22/02/2009

The are many grid middleware packages with overlapping functionality

No universal solutionEach project requires some glue to tie components together

Outline

• Three example projects at EPCC

• OGSA-DAI

• BEinGRID

• ILDG

22/02/2009 Edinburgh - Tsukuba Workshop 3

Challenges

• Diversity– Data resource types, vendors, middleware, schema, meta

data

• Scale– Collections, formats, volumes, geographical, political and

social distance

• Ownership– On individual, group, and organisational levels

• Security– Client, service and data owners

22/02/2009 4Edinburgh - Tsukuba Workshop

Sharing data


• Convert data into information• Reveal new insights

– Scientific knowledge– Business advantage

• Data mining across distributed data resources– Exploit public and private data

• Open or closed communities– Scientific collaborations– Business partnerships

OGSA-DAI

• OGSA-DAI – 02/2002 – 07/2003– EPCC, NeSC, IBM, Oracle, NEReSC, eSNW

– DTI/EPSRC via UK e-Science Grid Core Programme

• DAIT (DAI-Two) – 10/2003 – 10/2005– EPCC, NeSC, IBM, NEReSC, eSNW

– DTI/EPSRC via UK e-Science Grid Core Programme 2 as part of the OMII-UK project

• OMII-UK – 11/2005 – 04/2009– EPCC, NeSC

– EPSRC

• OMII-UK extension – 04/2009 – 04/2010– EPCC, NeSC

– EPSRC


Pays Capital

l'Espagne Madrid

l'Italie Rome

Pays Capital

Grande-Bretagne Londres

France Paris

Convert query from French to English

Convert query from French to English

Convert query from French to Spanish

Convert query from French to Spanish

Run SQL query

Run SQL query

Convert data from English to

French

Convert data from English to

French

Join the data

Join the data

País Capital

España Madrid

Italia Roma

Country Capital

UK London

France Paris

SELECT Country, Capital FROM Countries

SELECT País, Capital FROM Países

SELECT Pays,Capital FROM Pays

Run SQL query

Run SQL query

Convert data from

Spanish to French

Convert data from

Spanish to French

Pays Capital

Grande-Bretagne Londres

France Paris

l'Espagne Madrid

l'Italie Rome

Activity

Activity input

Activity outputTarget data resource

Workflows


http://www.33ff.com/flags/worldflags/United-Kingdom_flag.html

http://www.33ff.com/flags/worldflags/Spain_flag.html

• Advanced Data Mining and Integration Research for

Europe– EU 7th Framework program project– EPCC, NeSC and European partners

• Infrastructure for data integration and mining– Large scale enterprise systems

• Applications– Flood modelling and simulations– Customer relationship management

ADMIRE


GEOGrid

• Global Earth Organisation (GEO) Grid– National Institute of Advanced Industrial Science and Technology,

Japan

• Geo-spatial data and services– Disaster mitigation– Environmental monitoring– Natural resource exploration– Virtual integration and access control

• Data– Satellite imagery– Geological data– Ground-sensed data


GLS Portal

Deliver

Deliver

Transform

Transform

JoinJoinGetGet

GetGet

Maps

1: GLSQuery submited via

portal e.g. “Leeds population

distribution by census output

area”

4: URL of image is returned to portal – avoids costly SOAP/HTTP transfer of image

5: Portal gets image using URL

Image Creation Service

MIMASCensus

UK

BORDERS

OGSA-DAI

2: Workflow is populated with query parameters and run

3: Image is placed on a map

server

SEE-GEO – geo-linking portal



BEinGRID

• Type of project: Integrated Project

• Project coordinator: ATOS ORIGIN

• Project start date*: 1st June 2006

• Duration: 42 months

• Max EC contribution: 15.7 M euros

• Consortium: 99 partners

http://www.beingrid.eu/

http://www.it-tude.com/

22/02/2009


BEinGRID Vision

• Typical Technology Transfer project:– 2 waves of 18+7 Business Experiments involving:

– SMEs in various industry-sections– Technical and Business experts

– Set up a repository of Grid solutions, available free/at cost to the respective sectors

– Prove that businesses will benefit from the adoption of Grid technologies

22/02/2009


BE02 – FilmGrid

• “Movie post-production workflow”

• Reviewing data flow in the industry– Current data movement tied into celluloid shooting– What is the effect of digital capture?– How useful is Sohonet other than for email?

• The FilmGrid prototype proves:– Grid technology is highly appropriate for movie post-production– Potentially large gains in:

– Efficiency– Reliability– Accountability– Accessibility

• http://tinyurl.com/filmgrid

22/02/2009


Asset Manager

Global AssetsLocal Files Transfer Status

22/02/2009


Database Triggers

• Procedure to be executed when a modification is

made to a table– INSERT, UPDATE or DELETE

• Various use cases– Log changes– Execute business rules (e.g. email a

manager when online orders push stock

levels below a specified threshold )– Enforce business rules (e.g. all invoices

must be associated with a valid customer)

• How to set-up a trigger is dependent on DB implementation

22/02/2009


OGSA-DAI Trigger

• Uses database triggers to call an OGSA-DAI

workflow upon modification to a database

• Extends single-database trigger functionality to:– Span several, heterogeneous databases– Execute powerful OGSA-DAI workflows

• Many possible use cases– Synchronising databases– Logging to an external database– Ensuring or executing business logic across partners

http://tinyurl.com/ogsadaitrigger

22/02/2009


BE24 – GRID2(B2B)

• “Grid technologies for affordable data synchronization and SME integration within B2B networks”

• Empowering existing B2B networks by electronically connecting suppliers at an affordable price– Webservices-based add-on to allow data exchange at database

level– Uses OGSA-DAI Trigger to automate synchronization

• The GRID2(B2B) prototype demonstrates:– Easy integration with multiple B2B platforms– User in total control of what data is sent– Automated synchronization:

• Fast and frequent data transfer• Remove the need to enter data twice

• http://tinyurl.com/grid2b2b

22/02/2009


How does it work?

DBMS

DBMS

Ducati - Starter Bentivogli - Partner

MaNeM – B2B Platform

GRID2(B2B) Data Federation Agent

GRID2(B2B)Data Service

New orders generated by Ducati software

DBMS

Orders written to an internal database

GRID2(B2B)Data Service

OGSA-DAI Trigger used to monitor for new data

Data Service communicates the new information to the Data Federation Agent

Data Service and Data Federation Agent are configured using the GRID2(B2B) Configurator

Data Federation Agent inserts information into B2B database.

Data Federation Agent also monitors for new data in the B2B platform and propagates it on to the correct member of the network

22/02/2009

International Lattice Data Grid


• Sharing Lattice QCD data

• ILDG has no formal role– groups collaborate informally– working groups for metadata and middleware

• Individual groups were already starting to build data grid

infrastructures– UKQCD – QCDgrid, later DiGS– German groups combined into LATFOR, grid arm is LDG– US groups formed USQCD– Japanese – JLDG– Australia – Web portal

• Middleware often dictated by national considerations– ILDG is an aggregation of existing grids– Interoperable

ILDG WG

• Edinburgh and Tsukuba personnel

• Metadata Working Group– Tomoteru Yoshie Previous Convener– Chris Maynard Current Convener

• Middleware Working Group– George Beckett, Daragh Byrne, Eilidh Grant, Radek Ostrowski, and

James Perry– Mitsuhisa Sato, Toshiyuki Amagassa, Osamu Tatebe

• Example of Tsukuba and Edinburgh active collaboration


Three requisite conditions

• Trust– already established in the community

– known community

• Altruism– political will to make data available– effort to build infrastructure– effort actually making data available

• Reward– how to credit those making data available– data users should cite a designated paper


Three ideas to make this work

• Standard data format– Doesn’t really matter what, as long as one can read and write– configurations: SciDAC LIME record is 3x3 NERSC data layout

• Standard metadata– Semantic description of the data – Can be processed by an application

• Standard interfaces to services– Queries to metadata catalogues (MDC)– Queries to File Catalogue Web services (FC)– Authentication and authorisation


Architecture


Summary

• Rise in data complexity– doing things by hand is no longer scalable– we need tools to automate logistics and glue systems and data

together

• Grid architecture sits on top of existing systems– can access remote data with local tools– Many different middleware stacks

– Effort required to ensure interoperability

• Tsukuba and Edinburgh already collaborated successfully on

ILDG


Lunch


dr chris maynard epcc [email protected] +44 131 650 5077 grid@epcc joining the dots

Documents

data integration

data owners22022009

japangeospatial data

paysrun sql queryconvert

nesc epsrcomiiuk extension

omiiuk projectomiiuk

nesc epsrc22022009

grid middleware packages