grid computing an optimistic introduction...

Bucharest, 2010

Grid ComputingGrid ComputingAn Optimistic IntroductionAn Optimistic Introduction

N Tapus, A Iosup, A Muraru

PARALLEL AND DISTRIBUTED SYSTEMS GROUP

Agenda

1. About2. Grid definition3. Grid characteristics4. Grid@Work5. Grid benefits6. Conclusions

i

Computing and Communication Technologies Evolution: 1960-2010!

* Sputnik

1960 1970 1975 1980 1985 1990 1995 2000

* ARPANET

* Email* Ethernet

* TCP/IP* IETF

* Internet Era * WWW Era

* Mosaic

* XML

* PC Clusters* Crays * MPPs

* Mainframes

* HTML

* W3C

* P2P

* Grids

* XEROX PARC worm

CO

MP

UTI

NG

Com

mun

icat

ion

* Web Services

* Minicomputers * PCs

* WS Clusters

* PDAs* Workstations

* HTC

2010

* e-Science

* Computing as Utility

* e-Business

* SocialNet

ControlCentralised Decentralised

Computer Systems: Single -> GlobalComputer Systems

Distributed SystemsSingle System

PC/Workstation SMP/NUMA Vector Mainframe

Client ServerClusters Grids Peer-to-Peer

(multiple systems)

Centralised DecentralisedControl and Management

Performance, Capability, Value of ICT asdefined by the three Laws of Computing

• Moore’s Law. – Transistors on a single

chip doubles ~ every 18–24 months.

• Gilder’s Law. – Aggregate bandwidth triples ~

every year.

• Metcalfe’s Law. – The value of a network may

grow exponentially with the number of participants.

1,000,000,000,000100,000,000,000

1970

Moore/Transistors

Gilder/Bandwidth

Metcalf/NetworkNodes

10,000,000,0001,000,000,000

100,000,00010,000,0001,000,000

100,00010,0001,000

100101

1975 1980 1985 1990 1995 2000 2005 20102,300 6,000 29,000 275,000 1.2 mil 5.5 mil 42 mil 252 mil 1.344 bil

50 50 56 1,544 45,000 145,000 10 mil 2.43 bil 200.49 bil

4 111 200 10,000 300,000 1 mil 140 mil 3.5 bil 300 bil

Source: Cambridge Energy Resource Associates

Bucharest, 2010

Mainframe

Vector Supercomputer MPP

WorkstationPC

Mini Computer(hitting wall soon)

(future is bleak)

2100

Bucharest, 2010

2100 2100 2100 2100

2100 2100 2100 2100

2100

2100 2100 2100 2100

2100 2100 2100 2100

Personal Device SMPs or SuperComputers

LocalCluster

GlobalGrid

SERVICES

+PERFORMANCE

Inter PlanetGrid

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe

Administrative Barriers

EnterpriseCluster/Grid

Computing is Scaling: Towards Inter-Planetary Level

Alessandro Volta in Paris in 1801 inside French National Institute shows the battery while in the

presence of Napoleon I

Fresco by N. Cianfanelli (1841) (Zoological Section "La Specula" of National History Museum of Florence University)

….and in the future, I imagine a WorldwidePower (Electrical) Grid …...

What ?!?!This is a mad man…

Oh, monDieu !

2010 - 1801 = 209 Years

4 Essential Utilities and Delivery Networks

(1) WaterWater

Distribution Network

(4) TelephoneTelecom Networks

(3) Gas

(2) Electricity

Power Grid

Power Grid Inspiration: Seamlessly delivering electricity as a utility to users

” I think there is a world market for about five computers.”Thomas J. Watson Sr., IBM Founder, 1943

Can we Predict its Future ?

Agenda


i

Grid definition

2

“ A Grid is a large, heterogeneous, system that allows sharing and coordinating resources

in a dependable and pervasive manner “

“ A Grid is a large, heterogeneous, system that coordinates resources spread over wide ares ““ A Grid is a heterogeneous system that allowsmultiple entities to share and use resources,

under various administrative policies,offering a transparent access to the user “

“A Grid is a heterogeneous system spreadover a wide geographical area, which allowsmultiple entities to share and use resources,

under various administrative policies,offering a transparent access to the user, through

the use of consistent access protocols and interfaces “ “ A Grid is a large system that allows

sharing and coordinating resources

in a pervasive manner, under various

administrative policies “

What is Grid, after all!?Over 20 definitions

Grid definition

2

“ A Grid is a large, heterogeneous, system that allows sharing and coordinating resources

in a dependable and pervasive manner “

“ A Grid is a large, heterogeneous, system that coordinates resources spread over wide ares “

“ A Grid is a heterogeneous system that allowsmultiple entities to share and use resources,

under various administrative policies,offering a transparent access to the user ““ A Grid is a heterogeneous system spread

over a wide geographical area, which allowsmultiple entities to share and use resources,

under various administrative policies,offering a transparent access to the user, through

the use of consistent access protocols and interfaces “

“ A Grid is a large system that allows sharing and coordinating resources

in a pervasive manner, under variousadministrative policies “

“From a hardware perspective, a Grid is a collection of distributed resources connected by a network.From a user perspective a Grid gathers together

resources and makes them accessible in a secure manner to users and applications.“

A. Grimshaw, Industry view

“ A Grid is a system that coordinates resources that are not subject to centralized control using standard, open, general-purpose protocols and

interfaces to deliver nontrivial qualities of service “I.Foster, Research view

Grid Computing

• The Grid is a service for sharing computer power and data storage capacity over the Internet. The Grid goes well beyond simple communication between computers, and aims ultimately to turn the global network of computers into one vast computational resource.

• The essence of Grids is that they utilise highly flexible network architectures, and that they involve the sharing of all computing resources, not just data.

Agenda


i

Some Characteristics of GridsNumerousresources

Different securityrequirements

& policies

Resources areheterogeneous

Geographicallydistributed

Different resourcemanagementpolicies

Connected byheterogeneous, multi-level networks

Owned by multiple organizations &

individuals

Unreliable resources and environments

Slide by Hiro

Grid ChallengesSecurity

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Computational Economy

Application Construction

Some Grid Initiatives Worldwide• Australia

– Nimrod-G– Gridbus– GrangeNet.– APACGrid– ARC eResearch

• Brazil– OurGrid, EasyGrid– LNCC-Grid + many others

• China– ChinaGrid – Education– CNGrid - application

• Europe– UK eScience– EU Grids..– and many more...

• India– Garuda

Japan– NAREGI

• Korea...N*Grid

• Singapore RomaniaNGP RoGRID

• USA– Globus– GridSec– AccessGrid– TeraGrid– Cyberinfrasture– and many more...

• Industry Initiatives– IBM On Demand Computing– HP Adaptive Computing– Sun N1– Microsoft - .NET– Oracle 10g– Infosys – Enterprise Grid– Satyam – Business Grid– StorageTek –Grid..– and many more

• Public Forums– Open Grid Forum– Australian Grid Forum– Conferences:

• CCGrid• Grid• HPDC• E-Sciencehttp://www.gridcomputing.com

1.3 billion – 3 yrs

1 billion – 5 yrs

450million – 5 yrs486million – 5 yrs

1.3 billion (Rs)

27 million

2? billion

120million – 5 yrs

Grid characteristics

The five big ideas

• Resource Sharing• Secure Access• Resource Use• The Death of Distance• Open Standards

4


The five big ideas

• Resource Sharing• You enter the Grid to use remote resources, which allows

you to do things that you cannot do with the computer you own, or the computer centre you normally use

• Secure Access• Resource Use• The Death of Distance• Open Standards

5

Grid characteristicsThe five big ideas

• Resource Sharing• Secure Access

• Access policy - resource providers and users must define clearly and carefully what is shared, who is allowed to share, and the conditions under which sharing occurs;

• Authentication - mechanism for establishing the identity of a user or resource;

• Authorization - mechanism for determining whether an operation is consistent with the defined sharing relationships.

• Resource Use• The Death of Distance• Open Standards

6


The five big ideas

• Resource Sharing• Secure Access• Resource Use

• optimal allocation of resources

• The Death of Distance• Open Standards

7


The five big ideas

• Resource Sharing• Secure Access• Resource Use• The Death of Distance

• high-speed connections

• Open Standards

8


The five big ideas

• Resource Sharing• Secure Access• Resource Use• The Death of Distance• Open Standards

• more than 5000 individual researchers and practitioners participating in Global Grid Forum (GGF)

9

Grid characteristics• Large System

Many resources Computing Data storage Interconnection others

• Wide geographical distribution• Heterogeneous• Resource sharing• Multiple admin policies• Resource coordination• Transparent access• Dependable• Consistent• Pervasive

10

Grid characteristics• Large System• Wide geographical distribution

Bigger and bigger locations Laboratory Campus Metropolitan Continental World

• Heterogeneous• Resource sharing• Multiple admin policies• Resource coordination• Transparent access• Dependable• Consistent• Pervasive

11

Grid characteristics• Large System• Wide geographical distribution

• HeterogeneousMany resource types Clusters Supercomputers Computing centers others

• Resource sharing• Multiple admin policies• Resource coordination• Transparent access• Dependable• Consistent• Pervasive

12

Grid characteristics• Large System• Wide geographical distribution• Heterogeneous

• Resource sharingShare various types of resources Owner Normal use Special use Other roles

• Multiple admin policies• Resource coordination• Transparent access• Dependable• Consistent• Pervasive

13

Grid characteristics• Large System• Wide geographical distribution• Heterogeneous• Resource sharing

• Multiple admin policiesMany policies Security Control rights Operation rights others

• Resource coordination• Transparent access• Dependable• Consistent• Pervasive

14

Grid characteristics• Large System• Wide geographical distribution• Heterogeneous• Resource sharing• Multiple admin policies

• Resource coordinationCoordinate all resources Single-user Multi-users Reservation others

• Transparent access• Dependable• Consistent• Pervasive

15

Grid characteristics• Large System• Wide geographical distribution• Heterogeneous• Resource sharing• Multiple admin policies• Resource coordination• Transparent access

Single virtual system• Dependable

Many QoS rules• Consistent

Open, public access protocols• Pervasive

Fail-safe system

16

Agenda


i

Grid Computing

• Decompose across network• Clients integrate dynamically

– Select & compose services– Select “best of breed” providers– Publish result as a new service

• Decouple resource & service providers

FunctionResource

Data Archives

Analysis tools

Discovery toolsUsers

Fig: S. G. Djorgovski

Beyond Science Silos:

Service-Oriented Architecture

Grid@Work

17

Virtualization Virtual organization Virtual enterprise

A Sample Grid Computing Environment: Resource Sharing & Aggregation

Grid Resource Broker

Resource Broker

Application

Grid Information Service


databaseR2 R3

RN

R1

R4

R5

R6

Grid Information Service

2

Grid Node N

Grid Architecture for Computational Economy

Grid Consumer

Prog

ram

min

gEn

viro

nmen

ts


Grid Service Providers

Grid Explorer

Schedule Advisor

Trade Manager

Job ControlAgent

Deployment Agent

Trade Server

Resource Allocation

ResourceReservation

R1

Misc. services

Information Service

R2 Rm…

Pricing Algorithms

Accounting

Grid Node1

…

Grid Middleware Services

…

…

HealthMonitor

Grid Market Services

JobExec

Info ?

Secure

Trading

QoS

Storage

Sign-on

Grid Bank

App

licat

ions

Data Catalogue

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service

Grid Market Directory

GSP(Accounting Service)

GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer

Application CodeExplore

data1

36

45

Resu

lts

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

Using Gridbus + Nimrod-G:Brokering and On Demand Assembly of Grids

Data Source

(Instruments/distributed sources)

Data Replicator(GDMP) ASP Catalogue

Grid Info Service



GridBank

Data

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus/CPM) Grid

ServiceGS

GTS

Cluster Scheduler

Grid Service Provider (GSP)

(e.g., CERN)

PECluster Scheduler

Job

8

(Data) GridResource Broker

2

Visual Parametric Tool


data1

46

35

Resu

lts

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

33

On Demand Assembly of Services: Putting Them All Together

ASP Catalogue

Grid Info Service



GridbusGridBank

GSP(e.g., UofM)

PEGSP

(e.g., VPAC)

PE

GSP(e.g., IBM)

CPUorPE

Grid Service (GS)(Globus) Alchemi

GS

GTS

Cluster Scheduler

Job

8

GridResource Broker

2

Visual Application Composer


data1

36

45

Resu

lts

9 7

Results+

Cost Info

10

11

Bill

12Data Catalogue

Protocols, services and interfacesLanguages/Frameworks

Fabric Layer

Applications

Local Access APIs and protocols

Grid Service APIs and SDKs

Grid ServicesGrid Service Protocols

Resource APIs and SDKs

Resource ServicesResource Service Protocols

User Service ProtocolsUser Service APIs and SDKs

User Services

Connectivity APIs Connectivity Protocols

Networked Resources across Organizations

Computers Networks Data Sources Scientific InstrumentsStorage Systems

Local Resource Managers

Operating Systems Queuing Systems Internet ProtocolsLibraries & App Kernels

Distributed Resources Coupling Services

Information QoSProcess

Application Development and Deployment Environment

Languages/Compilers Libraries Debuggers Web tools

Resource Management and Scheduling:

Applications and Portals

Prob. Solving Env.Scientific …CollaborationEngineering Web enabled Apps

Trading

…

…

…

…

FABRIC

APPLICATIONS

SECURITY LAYER

Security Data

CORE MIDDLEWARE

USER LEVEL MIDDLEWARE

Monitors

Layered Grid ArchitectureA

daptive Managem

ent A

utonomic/ G

rid Econom

y

Trends

• IBM Computing on Demand

• Oracle DataGrid solution (Oracle 11g)/ Sun Grid Utility Computing : pay-per-use service

• Hewlett Packard grid-enabled software solutions

• Microsoft Research Project BigTop: software as a service

Agenda


i

Gridbus Project: Some Applications and Users

Gridbus Project: Gridbus Project: Some Applications and UsersSome Applications and Users

http://www.gridbus.org

BioGrid: Molecular docking for Drug-discovery

BioGrid: Molecular docking for Drug-discovery

High Energy Physics: Particle Discovery

High Energy Physics: Particle Discovery

Melbourne University

NeuroScience: Brain Activity Analysis

NeuroScience: Brain Activity Analysis

Natural Resource ModelingNatural Resource Modeling

CSIRO Land and Water, Austraila.

Large Scale document processing

Large Scale document processing

Tier Technologies, USA.

Detection of patterns of transcription factors in mammalian genes

Detection of patterns of transcription factors in mammalian genes

2: [Grid in Data Sharing] Physics Collaboration

(fundamental investigation on the origin of mass)

LHC Grid Computing Model

Tier2 Centre ~1 TIPS

Online System

Offline Processor Farm

~20 TIPS

CERN Computer Centre

Asia Pacific Centre ~4 TIPS

France Regional Centre

US Regional Centre

Italy Regional Centre

InstituteInstituteInstituteMelbourne~0.25TIPS

Physicist desktop computers

~100 MBytes/sec

~100 MBytes/sec

~622 Mbits/sec

~10 to 100 Mbits/sec

There is a “bunch crossing” every 25 nsecs.There are 100 “triggers” per secondEach triggered event is ~1 MByte in size

Physics data cache

~PBytes/sec

~622 Mbits/sec




Australian Centre ~1 TIPS

~622 Mbits/sec

1 TIPS is approximately 25,000 SpecInt95 equivalents

Tier 4

Tier 0

Tier 1

Tier 2

Tier 3

3. [Grid Use in Business] Enterprise Computing Application

• Traditional Model • Grid-based Model

Email server

Webserver

Databaseserver

Appsserver

Upgrade to a new serverto handle

more users

Utilise IT infrastructure effectively

Service Virtualization Layer & Load Balancing

Grid benefits• Efficient use of already installed

computing power• Ease of resource partitioning• Processing larger data volumes• Sharing software licenses• Ready-to-use solutions

18

industryindustry--backedbacked

IBM, HP, Sun, Oracle, Avaki

Grid benefits• Efficient use of already installed

computing power Fast Return of Investment (RoI) 75% computer time unused 90% server, 95% PC time unused (IBM)

• Ease of resource partitioning• Processing larger data volumes• Sharing software licenses• Ready-to-use solutions

19

Grid benefits• Efficient use of already installed computing

power

• Ease of resource partitioning Smaller cost of ownership (CoO)Service persistence sharing, scheduling utilization, proper use management facilitated more resources getting shared

• Processing larger data volumes• Sharing software licenses• Ready-to-use solutions

20


power• Ease of resource partitioning

• Processing larger data volumesSmaller cost of ownership (CoO)Secured services summed resources better than sum of resources outsourcing security

• Sharing software licenses• Ready-to-use solutions

21


power• Ease of resource partitioning• Processing larger data volumes

• Sharing software licensesBetter investment use summed resources better than sum of resources outsourcing security

• Ready-to-use solutions

22


power• Ease of resource partitioning• Processing larger data volumes• Sharing software licenses• Ready-to-use solutions

Big companies make it work! financial services automotive/aerospace design & assembly lines national-scale database systems simulations for chemistry and life sciences data visualization for petroleum fields virtual systems with massive simultaneous uses

23

Agenda


i

Conclusions

• The Grid is to computing whatInternet is to information sharing resources is the key 10 characteristics 5 industry-backed benefits lots of uses

• Problems? We’ll leave them for later – after all,

this is an optimistic introduction…

21

Thank you!

Questions? Observations? All welcome…

GridGrid

GridGridGridGrid

GridGrid

Grid

Grid

GridGrid

Grid

GridGrid

GridGridGrid

grid computing an optimistic introduction...

Documents