Convergence Characteristics for Clusters, Grids, and P2P
networks
Rajkumar BuyyaGrid Computing and Distributed Systems (GRIDS) Lab. The University of MelbourneMelbourne, Australiawww.gridbus.org
WW Grid
Panel at the P2P conference, Linkoping, Sweden
2
Where Did I come From ?
*
*
*
*
Melbourne
Stockholm+Linkoping
Singapore
London
3
GRIDS Lab @ the U. of Melbourne, The Gridbus Project:
www.gridbus.org Conduct R&D in Distributed Computational Economy for
Service-Oriented Computing: Architecture, Specification, and Open Source Reference
Implementation in in Collaboration with Any Global Peers. Platform: Cluster, Grid, P2P for diverse applications.
Distributed Application Composition Tools Grid Economy & Scheduling (via Nimrod-G Broker) GridSim Toolkit: Grid Modeling and Simulation (Java based): Libra: Economic Cluster Scheduler Grid Bank: Accounting and Transaction Management World Wide Grid (WWG) – A Global Testbed P2P/Jxta-based Compute Power Market Application Enabler Projects:
Virtual Laboratory Toolset for Drug Design High-Energy Physics and the Grid Network (HEPGrid) Brain Activity Analysis on the Grid Future: GridEmail for Spam Management
—read only if sender pays!
4
Books
5
Technology Evolution: Cluster, Grid, P2P
6
2100
2100 2100 2100 2100
2100 2100 2100 2100
Desktop SMPs or SuperComputersLocal
ClusterGlobal
Cluster/Grid
PERFORMANCE
Inter PlanetaryGrid!
•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Galaxy
Administrative Barriers
EnterpriseCluster/Grid
?
Scalable Seamless Computing: Breaking Administrative Barriers
7
Basic ElementsSecurity
Resource Allocation & Scheduling
Data locality
Network Management
System Management
Resource Discovery
Uniform Access
Computational Economy
Application Development Tools
8
Cluster Architecture
Sequential Applications
Parallel Applications
Parallel Programming Environment
Cluster Middleware(Single System Image and Availability Infrastructure)
Cluster Interconnection Network/Switch
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
Sequential Applications
Sequential Applications
Parallel ApplicationsParallel
Applications
9
Cluster Architecture
Sequential Applications
Parallel Applications
Parallel Programming Environment
Cluster Middleware(Single System Image and Availability Infrastructure)
Cluster Interconnection Network/Switch
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
PC/Workstation
Network Interface Hardware
CommunicationsSoftware
Sequential Applications
Sequential Applications
Parallel ApplicationsParallel
Applications
10
A Typical Grid Computing Environment
Grid Resource Broker
Resource Broker
Application
Grid Information Service
Grid Resource Broker
databaseR2 R3
RN
R1
R4
R5
R6
Grid Information Service
2
11
A Typical Grid Computing Environment
Grid Resource Broker
Resource Broker
Application
Grid Information Service
Grid Resource Broker
databaseR2 R3
RN
R1
R4
R5
R6
Grid Information Service
2
12
A Typical P2P Computing Environment
Peer Agent
Peer Agent
Application
Peer Discovery Service
Peer Agent
P1
P3
pN
P2
R7
Who ca
n help
?
Peer
P 2, P 7 c
an he
lp!
pM
Request
Response
p4 p5
RequestSorry, I am busy.
13
A Typical P2P Computing Environment
Peer Agent
Peer Agent
Application
Peer Discovery Service
Peer Agent
P1
P3
pN
P2
R7
Who ca
n help
?
Peer
P 2, P 7 c
an he
lp!
pM
Request
Response
p4 p5
RequestSorry, I am busy.
14
Cluster, Grid, P2P: CharacteristicsCharacteristic Cluster Grid P2P
Population Commodity Computers
High-end computers Edge of network (desktop PC)
Ownership Single Multiple MultipleDiscovery Membership
ServicesCentralised Index & Decentralised Info
Decentralized
User Management Centralised Decentralised DecentralisedResource management
Centralized Distributed Distributed
Allocation/Scheduling
Centralised Decentralised Decentralised
Inter-Operability VIA based? No standards yet No standardsSingle System Image Yes No NoScalability 100s 1000? Millions?
[@Home]Capacity Guaranteed Varies, but high VariesThroughput Medium High Very HighSpeed(Lat. Bandwidth)
Low, high High, Low High, Low
15
Nimrod-G: Modeling Distributed Computing in Economic Terms
Grid Middleware
Nimrod/G Client Nimrod/G ClientNimrod/G Client
Grid Information Server(s)
Schedule Advisor
Trading Manager
Nimrod-G Engine
GridStore
Grid Explorer
GE GISTM TS
RM & TS
Grid Dispatcher
RM: Local Resource Manager, TS: Trade Server
Globus, Legion, Condor, etc.
G
G
CL
Globus enabled node. Legion enabled node. G
L
Condor enabled node.
RM & TSRM & TS
C L
16
On-Demand Computing
CostCostDeadlineDeadline
Legion hosts
Globus Hosts
Bezek is in both Globus and Legion Domains
Arlington
Alexandria
Richmond
HamptonNorfolk
Virginia BeachChesapeakePortsmouth
Newport News
RoanokeApp om a toxRive r
Ja m esRive r
Shena ndo ahRive r
Rap pa hanno c kRive r
Potom acRive r
VIRGINIA77
81
64
64
66
85
17
Active Sheet Nimrod-G:Transparent Spreadsheet Processing from PCs to
Clusters to World-Wide Grids
NimrodNimrodProxyProxy
Nimrod-GNimrod-G
World-Wide Grid
18
MEG(MagnetoEncephaloGraphy) Data Analysis on the Grid: Brain Activity
Analysis
Life-electronics laboratory,AIST
Data Analysis
•Provision of expertise in the analysis of brain function•Provision of MEG analysis
Data Generation
Nimrod-G
64 sensors MEG
Results
Analysis All pairs (64x64) of MEG data by shifting the temporal region of MEG data over time: 0 to 29750: 64x64x29750 jobs
World-Wide Grid•[deadline, budget, optimization preference]
1
5
4
3
2
[Collaboration with Osaka University, Japan]
Backup Slides
20
Evolution Time Line
* Sputnik
1960 1970 1975 1980 1985 1990 1995 2000
* ARPANET
* Email* Ethernet
* TCP/IP* IETF
* Internet Era * WWW Era
* Mosaic
* XML
* PC Clusters* Crays * MPPs
* Mainframes
* HTML* W3C
* P2P
* Grids
* XEROX PARC wormCOM
PUTI
NG
NET
WO
RKIN
G
* Web Services
* Minicomputers * PCs
* WS Clusters
* PDAs* Workstations
* HTC
21
What is Cluster/Grid ?
A type of parallel and distributed system that enables the sharing, selection, & aggregationof resources distributed in _____/_______ administrative domains depending on their availability, capability, performance, cost, and users quality of service requirements.
A SingleMultiple
Cluster Grid
22
CPM: DC Economy-based P2P Computing
(Jxta based Implementation)
Accounting
Bill
Resources(Provider)
User (Consumer)
CPM Agent
Trader
Market Server
Market Repository
Job Management
- Discovery- Membership
23
Drug Design: Data Intensive Computing on Grid
It involves screening millions of chemical compounds (molecules) in the Chemical DataBase (CDB) to identify those having potential to serve as drug candidates.
Protein
Molecules
Chemical Databases(legacy, in .MOL2 format)
[Collaboration with WEHI for Medical Science, Melbourne]
24
Virtual Drug DesignA Virtual Lab for “Molecular Modeling for Drug Design” on P2P Grid
“Screen 2K molecules in 30min. for $10”
Grid Market Directory
ResourceBroker
Grid Info. Service
GTS
GTS
GTS
GTS
“Give me list PDBs sourcesOf type aldrich_300?”
“service co
st?”
(GTS - Grid Trade Server)
PDB2
“get mol.10 from pdb1 & screen it.”
Data Replica Catalogue
“service providers?”
GTSPDB1
“mol.10 please?”
“mol.5 please?”
(RB maps suitable Grid nodes and Protein DataBank)