Everything you always wanted to know about the Grid and

never dared to ask

Tony Hey and Geoffrey Fox

Outline• Lecture 1: Origins of the Grid – The Past to

the Present (TH)• Lecture 2: Web Services, Globus, OGSA and

the Architecture of the Grid (GF)• Lecture 3: Data Grids, Computing Grids and

P2P Grids (GF)• Lecture 4: Grid Functionalities – Metadata,

Workflow and Portals (GF)• Lecture 5: The Future of the Grid - e-Science

to e-Business (TH)

Lecture 1

Origins of the Grid –

The Past to the Present

[Grid Computing Book:

Chs 1,2,3,4,5,6,36]

Lecture 5

The Future of the Grid –

e-Science to e-Business

[Grid Computing Book:

Chs 38, 39, 40,41,42,43]

Lecture 5

1. e-Science Research and the Future of Scientific Research

2. Computer Science Research Issues

3. A Business Case for the Grid

4. Concluding Remarks

e-Science and the Future of Scientific Research

‘e-Science will change the dynamic of the way science is undertaken.’

John Taylor, 2001

Integrated e-Science Environment

Framework for distributed scientific computing and experimentation

local remote

Computers

local remote

Data storage

local remote

Experiments

Grid services middleware

Computing Grid

service

Data discovery

Grid service

Data visualisation Grid service

“Problem Solving Environments” Domain-specific application interfaces for scientists

Authentication

AuthorisationAccounting

Experiment control Grid

service

e-Science Examples

• Particle Physics

• Virtual Observatories

• e-Engineering

• e-Chemistry

• Bioinformatics

• High-Throughput Applications

• e-Health

In flight data

Airline

Maintenance Centre

Ground Station

Global Networkeg: SITA

Internet, e-mail, pager

DS&S Engine Health Center

Data centre

DAME Project

Comb-e-Chem

Structures DB

Properties DB

Simulation andcalculation

Structure + Properties Knowledge + Prediction

Combinatorial Chemistry

• Parallel synthetic approach– create hundreds of materials

– screen properties to find those that fit the bill

• Typically requires several passes– find chemical structure of the best

candidates

– create new batches of similar materials for subsequent passes

• Leads to explosive growth in:– volume of data generated

– potential to exploit this data

MeOH EtOH PrOH BuOH

R1COOH

R2COOH

R3COOH

R4COOH

same reaction sequencefor all combinations

Mon

itor

& A

naly

sis

Dat

a

Inte

rfac

e to

Gri

d

AArray production of different chemical speciesrray production of different chemical species

Mass Spec

x-ray

Raman

Well plate with typically 96 or 384 cells

Str

uctu

re a

nd p

rope

rtie

s an

alys

is

databases

High throughputsystems

Librarysynthesis

100,000’s compounds at a time analysis100,000’s compounds at a time analysisProduces huge amounts of complex dataProduces huge amounts of complex data

Users Users Users

Experiment Expert

Data & control links

Access Grid links

ExperimentRemote (Dark) Laboratory

Remote equipment, multiple users, few experts

Model for National crystallographic Service NCSModel for National crystallographic Service NCS

NCS Workflow

X-Ray e-LaboratoryStructuresDatabase

ComputationService

Send sample Send sample material to material to

NCS serviceNCS service

Search materials database Search materials database and predict properties using and predict properties using

Grid computationsGrid computations

Download full Download full data on materials data on materials

of interestof interest

Collaborate in e-Lab Collaborate in e-Lab experiment and experiment and obtain structureobtain structure

NCS Portal Access

NCS Experimental Services

NCS Lab Service

Samples andSchedules

StatusMonitor

Collaboratory Interface Data Access Interface

Proxy Proxy Proxy

ControlGUI

(FilteredVNC)

Chat AudioRaw

ImagesResultsAccess

StructAccess

Schedules

SampleManage-

ment

ScheduleManage-

ment

Raw Data(Files)

ProcessedData (DB)

StructureData (DB)

Admin

Auth

Scheduling Expt Control HKL Calculation Struct Calc

UI

Middleware

Backend

Control

myGrid Project

• Imminent ‘deluge’ of data

• Highly heterogeneous• Highly complex and

inter-related• Convergence of data

and literature archives

myGrid: Generic Technologies

1. Database access from the Grid

2. Process enactment on the Grid

3. Personalisation services

4. Metadata services

5. Development of Agent Services

Ultimate goal is to put Grid Services together

with Ontologies to develop ‘Semantic Grid’

Workflow

• Know how.• Associate base resources with derived data.• Keep, describe, find, compare, protect,

share.• Repeat/reuse/re-enact• Specialise/Customise/Personalise• Evolution – notification, knowledge• Quality & best practice

– Need the workflows to be effective good experimental practice.

1

2

3

4

Personalisation

• Dynamic creation of personal data sets

• Personal views over repositories• Personalisation of workflows • Personal notification • Annotation of datasets and

workflows• Personalisation of service

descriptions – ‘what I think the service does’

1

2

3

4

Provenance

• Who, what, where, why, when, how?• The traceability of knowledge as it is

evolves and as it is derived.• Identity – the Life Sciences ID• Lab Books, Methods in papers.• Immutable Metadata• Migration – travels with its data but

may not be stored with it.• Private vs Shared provenance records.• Ownership/credit

1

2

3

4

Using Distributed Resources

ScientificInformationScientific

InformationScientific Discovery

In Real Time

LiteratureLiterature

DatabasesDatabases

OperationalData

OperationalData

ImagesImages

InstrumentData

InstrumentData

Discovery Net Project

Real Time Integration

Dynamic ApplicationIntegration

Workflow Construction

Interactive Visual Analysis

Discovery Process Management• Workflow = Service Composition +

Discovery Pathway

• Towards a Standard Workflow Representation for Discovery Informatics: Discovery Process Markup Language (DPML):

– Discovery Pathway Construction: Recording and managing a collaboratively-built discovery process

– Distributed Service Composition: Components organsied by the workflow can be executing anywhere

– Discovery Pathway as Key Intellectual Property: Discovery Processes can be stored, reused, audited, refined and deployed in various forms D-Net Workflow for Genome Annotation :

16 services executing across Internet

Dynamic Integration Services• Dynamic Application Integration =

On-demand access and composition of remote analysis components

• Towards a Dynamic Component Integration:

– Knowledge Servers: allow users to register, locate and remotely execute components

– Execution Servers: allow users to control the execution of components distributed environments

– Easy Maintenance: New components can be added through a clean API

Text analysisText analysis

ClusteringClustering ClassificationClassification

Gene function perdition

Gene function perdition

Homology Search

Homology Search

Promoter PredictionPromoter

Prediction

D-NET APID-NET API

Case Study: SC2002 HPC Challenge

blastgenscan

RepeatMasker

grail

genscanE-PCR

Identify

Genes

Gene markers

tRNAs, rRNAs

Non-translatedRNAs

RegulatoryRegions

RepetitiveElements

SegmentalDuplication

SNPVariations

LiteratureReferences

…..

3D-PSSMblast

MotifSearch

PFAM

DSCpredator

InterPro

InterPro

SMARTSWISSPROT

Identify

FunctionalCharacteisation

Homologues

Domain 3-D Structure

Fold PredictionSecondary structure

LiteratureReferences

…..

ProteinsClassify into

Protein Families

Identify

OrganismChromosomes

Organism’sDNA

Relate

CellCycle

Metabolism

DrugsBiologicalProcess…..

Cell deathEmbryogenesis

LiteratureReferences

…..

Ontologies

PathwayMaps

GeneMapsAmiGO

GenNav

virtual chip

High ThroughputSequencers

Nucleotide-level Annotation

Protein-level Annotation

Process-level Annotation

NCBIEMBL

TIGR SNP

GO CSNDB

GKKEGG

15 DBs 21 Applications

D-Net based Global Collaborative Real- Time Genome Annotation

Genome Annotation

Nucleotide Annotation Workflows

How It Works

Download sequence

from Reference

Server

Save to Distributed Annotation

Server

InteractiveEditor &

Visualisation

Execute distributed annotation workflow

NCBIEMBL

TIGR SNP

InterPro

SMART

SWISSPROT

GO

KEGG

500 Web access1800 clicks200 copy/paste 3 weeks work in 1 workflow and few second execution

eDiamond Applications of SMF

Training and Differential Diagnosis “Find one like it”

Teleradiology and QC VirtualMammo

Epidemiology SMFcomputed breast density

?

Advanced CAD SMF-CAD workstation

Image guided interventions Images CourtesyDerek HillGuy’s Hospital

Image guided interventions (2)Images CourtesyGuy’s Hospital

Surgical verificationAccuracy of surgical placement against plan

• Surgeon plans on X-ray or CT, uses database of prostheses• Operation takes place using plan as guidance• Post operative X-ray evaluated for accuracy of placement• Data stored and used for short term assessment and long

term evaluation studies

Courtesy of Ian RevieDepuy International

• UK e-Science projects emphasize data federation and integration as much as computation

• Metadata and ontologies key to higher level Grid services

• e-Science projects will produce a deluge of scientific data that will need to be annotated and curated in scientific data ‘digital libraries’

Summary

Databases in the Grid

Computational Complexity

DataComplexity

OGSA – DAI Project

• Key middleware project for UK Program- Total Budget £3M (CP £1.5M)

• Three Centres involved: - Edinburgh, Manchester and Newcastle • Industrial partners:

- IBM US, IBM Hursley and Oracle UK

Goal is to develop high-quality data-centric middleware

OGSA – DAI Project

• Design Specification completed– Papers for GGF WG on Database Access and

Integration Services• Alpha versions delivered:

– Distributed Query Service– XML Database Interface– Relational Database Interface

• Beta versions by April 2003– Integrate with Globus GT3 release

e-Science and the Future of Scientific Research

‘e-Science will change the dynamic of the way science is undertaken.’

John Taylor, 2001 Need to break down the barriers between the Victorian

‘bastions’ of science – biology, chemistry, physics, …. Develop ‘permeable’ structures that promote rather than

hinder multidisciplinary collaboration Engage Computing Services and Libraries in developing a

new e-Science support service on Campus

e-Science and Computer Science

• The lesson of the Web

• The Semantic Grid

– The myGrid project

– The Discovery Net Project

• Computer Science Research and the Grid

Error 404: Page not found

‘If you want the Web to scale,

You must allow the links to fail’

Wendy Hall after Tim Berners-Lee

HTML as the ‘Fortran’ of Hypertext!

Semantic Web

Metadata & Ontologies• Metadata – computationally

accessible data about the services

• Ontologies – the shared and common understanding of a domain– A vocabulary of terms– Definition of what those terms

mean.– A shared understanding for

people and machines– Usually organised into a

taxonomy.

• Consistency — check if knowledge is meaningful

• Subsumption — structure knowledge, compute classification

• Equivalence — check if two classes denote same set of instances

• Instantiation — check if individual instance of class C

• Retrieval — retrieve set of individuals that instantiate C

Reasoning in DAML+OIL

Computer Science Challengesfrom e-Science

UK CS Team led by Tom Rodden identified 4 major research challenges arising from e-Science:

- Developing a Semantic Grid- Trusted Ubiquitous Systems- Rapid Customized Assembly of Services- Autonomic Computing

Towards a Semantic Grid

• Trace provenance from initial data to information and knowledge structures

• Techniques to allow scalable reasoning over uncertain/incomplete knowledge

• Tools for design, development and deployment of large-scale ontologies

• Support for semantic-directed knowledge discovery to complement data-mining

• Development of flexible network-based reasoning and decision support services

Trusted Ubiquitous Systems

• New theories to model, specify and analyse trust in distributed ubiquitous systems

• New quality of service and service-based models for ubiquitous systems

• New design guidelines and practices to enable the development of reusable trusted components

• New understanding of the practical engineering trade-offs required to realise trusted ubiquitous systems

Rapid Customised Assembly of Services

• New theories to describe and reason about semantics and behaviour of services and compositional effects

• Agent and service representations that promote adaptability and emergent, opportunistic and implicit arrangement of services

• New tools to support the discovery, composition and use of services based on high-level description of requirements

• Techniques to support directed automatic composition, decomposition and recomposition of services

Autonomic Computing

• Techniques to analyze, describe and reason about adaptive systems

• Management of semi-autonomous systems with policies, services and software agents

• Interoperability and reasoning across and between different autonomous domains

• Modeling and measurement of performance of QoS for autonomic structures

• Techniques to capture and represent history, context and environment

IBM Autonomic Computing Vision

Self-Healing Discover, diagnose, and react to disruptions

Self-Healing Discover, diagnose, and react to disruptions

Self-OptimizingMonitor and tune resources automatically

Self-OptimizingMonitor and tune resources automatically

Self-Protecting Anticipate, detect, identify, and protect against attacks from anywhere

Self-Protecting Anticipate, detect, identify, and protect against attacks from anywhere

Self-ConfiguringAdapt automatically to the dynamically changing environments

Self-ConfiguringAdapt automatically to the dynamically changing environments Self-

ConfiguringSelf-

Configuring

Self-Healing

Self-Healing

Self-Optimizing

Self-Optimizing

Self-Protecting

Self-Protecting

A Business Case for the Grid

• Total Cost of Ownership – TCO

• Value of Open Standards

• Industrial Applications

• Time to exploitation

• e-Utilities

Gateway

Hub Server Group

Local

Director

Network

Business LogicPresentation

WebSphereApplication

Server

NetscapeEnterprise

Server WebSphereApplication

Server

JDBC

HTTP

MQ

ProfileCapture

DatabaseServers

SecurityGateways

DB2SecurityServers

MQMQ

ApplicationLogging

GatewayLogging

MQ MQ

SecurityClient

SNA

SNA

SNA

Back-endSystems

ComplexIMSData

CICS

SysplexIMSData

SysplexIMSData

SysplexIMSData

TPF

Front-end for Web presence for financial services

Typical Financial Subsystem Configuration

Current IT EnvironmentDistributed, Heterogeneous, Complex

Gateway

Hub Server Group

Local

Director

Network

Business LogicPresentation

WebSphereApplication

Server

NetscapeEnterprise

Server WebSphereApplication

Server

JDBC

HTTP

MQ

ProfileCapture

DatabaseServers

SecurityGateways

DB2SecurityServers

MQMQ

ApplicationLogging

GatewayLogging

MQ MQ

SecurityClient

SNA

SNA

SNA

Back-endSystems

zSeries

ComplexIMSData

CICS

zSeries

SysplexIMSData

zSeries

SysplexIMSData

zSeries

SysplexIMSData

zSeries

TPF

Front-end for Web presence for financial services

Typical Financial Subsystem Configuration

Current IT EnvironmentDistributed, Heterogeneous, Complex

Complexity, TCO

Tech. Cost, Utilization

Server / Storage Utilization

52%N/AN/AStorage

2-5%5-10%30%Intel-based

<10%10-15%50-70%UNIX

60%70%85-100%Mainframes

24-hour Period Utilization

Prime-shift Utilization

Peak-hour Utilization

Source: IBM Scorpion White Paper: Simplifying the Corporate IT Infrastructure, 2000

Total Cost of Ownership: TCO

Integration32.0%

Hardware10.0%

Software12.0%

Personnel16.0%

Maintenance30.0%

IT Budgets

HardwareSoftwarePersonnelMaintenanceIntegration

32%

30%

Integration32.0%

Much More than Hardware and Software Costs

16%Personnel

16.0%

Grid Computing Sales Pitch

Storage

ApplicationsApplicationsProcessingProcessing

Operating System

DataData

I/O

Distributed Computing Over a Network, Using Open Standards to Enable Heterogeneous Operations

Grid Technology Enables

Increased Server Utilization Workload Management and Consolidation Reduced Cycle Times

Collaboration and Access to Data Federation of Data Global Distribution

Resilient/Highly Available Infrastructure Business Continuity Recovery and Failover

Supporting Heterogeneous Resources Through Open Standards….

Increased Server Utilization• Exploit distributed resources to provide

capacity for high-demand applications– Existing applications that cannot be run

effectively on a single processor

– New large scale application that provide strategic business advantages

• Reduce infrastructure cost associated with over-provisioned resources– Balance workload based on policies

– Optimize for cost or throughput

• Reduce the cost of manpower to manage and configure resources– Fewer resources to manage for the same

workload

Collaboration and Access to Data

• Enable collaboration across applications to integrate results – Leverage Distributed Data and Resources

• Support large multi-disciplinary collaborations– Link Business Processes

– Federation of Data

• Both within a single organization and between partners– Exploit Replication Services Across

Enterprises

Simulation

PricingDesign

Design

Design Analytics

The Value of Open Standards

Networking:The Internet

(TCP/IP)

Communications:e-mail

(pop3,SMTP,Mime)

Information:World-wide Web

(html, http, j2ee, xml)

Applications:Web Services

(SOAP, WSDL, UDDI)

Distributed Computing:Grid

(Globus -> OGSA)

Operating System: Linux

Sun and the Grid: ‘Grid Computing is one of the three next big

things for Sun and our customers’

Ed Zander, COO

Microsoft and the Grid: ‘The alignment of OGSA with XML Web

services is important because it will make Internet-scale, distributed Grid Computing possible’

Robert Wahbe, General Manager of Web Services

Grid Infrastructure

Industry Applications

DerivativesAnalysis

Statistical Analysis

Portfolio Risk

Analysis

Batch Throughput

Product Design

Process Simulation

FiniteElement Analysis

Failure Analysis

Cancer Research

Drug Discovery

Protein Folding

Protein Sequencing

Collaborative Research

Weather Analysis

HPC

Unique by Industry with Common Characteristics

Seismic Analysis

Reservoir Analysis

Bandwidth Consumption

Digital Rendering

Multiplayer Gaming

Primary Focus

Energy

Financial Services

Manufacturing

LS/

BioinformaticsTelco & Media

Gov’t & Education

Unlimited Numbers of Players

Distributed Artificial Intelligence

Multiple Concurrent Players

1,000 downloads of developer’s kit per week

Hot-swappable Components

Developers, Publishers, ESPs

Globalization Grid: Butterfly.net

HP, the Grid and e-Utilities

The Grid fabric for e-Utilities will be:

• Soft – malleable, multi-purpose

• Dynamic – resources will be constantly changing

• Federated – global structure not owned by any single authority

• Heterogeneous – from supercomputer clusters to PCs

John Manley, HP Labs

Timescales for Exploitation?• IBM see ‘early adopters’ of Grid technology

coming from pharmaceutical, engineering and petrochemical sectors

UK program confirms this picture (AstraZeneca, GSK, Merck, Pfizer, Rolls Royce, BAESystems, Schlumberger)

• IBM see Grid middleware being adopted by more mainstream commerce and industry in 2003/2004 timeframe

Status of the Grid• Today - ‘early adoption’ phase - just like the Web in

the early days– Industry now selling ‘IntraGrid’ solutions– Genuine Virtual Organisation ‘InterGrid’

middleware not yet mature• ‘Tomorrow’ - sophisticated combinations of services

to locate information, applications to process it, and computer systems to run them

Autonomic Middleware infrastructure capable of supporting Virtual Organisations, c-Commerce and e-Utilities will take time!

e-Government and the Grid

‘[The Grid] intends to make access to computing power, scientific data repositories and experimental facilities as easy as the Web makes access to information.’

Tony Blair, 2002

Acknowledgements

With thanks to:

Gerd Breiter, Phillipe Bricard, David Boyd,

Jens Jensen, Daron Green, Mike Brady,

Derek Hill, Carole Goble, Yike Guo,

Jeremy Frey, Bill Johnston, Ray Browne,

Jim Fleming, Anne Trefethen and many others