ececs lecture 18 grid computing citation: b.ramamurthy/suny-buffalo
TRANSCRIPT
ECECS Lecture 18
Grid Computing
Citation: B.Ramamurthy/Suny-Buffalo
Globus Material
The presentation is based on the two main publications on grid computing given below:
1. The Physiology of the Grid, An Open Services Architecture for Distributed Systems Integration, by Ian Foster, Carl Kesselman, Jeffrey Nick, and Steven Tuecke, 2002.
2. The Anatomy of the grid, Enabling Scalable Virtual Organization, Ian Foster, Carl Kesselman, Steven Tuecke, 2001.
3. URL:http://www.globus.org/research/papers.html
Grid Technology
• Grid technologies and infrastructures support the sharing and coordinated use of diverse resources in dynamic, distributed “virtual organizations”.
• Grid technologies are distinct from technology trends such as Internet, enterprise, distributed and peer-to-peer computing. But these technologies can benefit from growing into the “problem space” addressed by grid technologies.
Virtual Organization: Problem Space
• An industrial consortium formed to develop a feasibility study for a next generation supersonic aircraft undertakes a highly accurate multidisciplinary simulation of the entire aircraft.
• A crisis management teams responds to a chemical spill by using local weather and soil models to estimate the spread of the spill, planning and coordinating evacuation, notifying hospitals and so forth.
• Thousands of physicists come together to design, create, operate and analyze products by pooling together computing, storage, networking resources to create a Data Grid.
Resource Sharing Requirements
• Members should be trustful and trustworthy.• Sharing is conditional.• Should be secure.• Sharing should be able to change dynamically over
time.• Need for discovery and registering of resources.• Can be peer to peer or client/server.• Same resource may be used in different ways.• All these point to well defined architecture and
protocols.
Grid Definition• Architecture identifies the fundamental system
components, specifies purpose and function of these components, and indicates how these components interact with each other.
• Grid architecture is a protocol architecture, with protocols defining the basic mechanisms by which VO users and resources negotiate , establish, manage and exploit sharing relationships.
• Grid architecture is also a services standards-based open architecture that facilitates extensibility, interoperability, portability and code sharing.
• API and Toolkits are also being developed.
Grid Services Architecture
Grid FabricLayer
Applications
Transport Multicast
Instrumentation Control interfaces QoS mechanisms
Grid ServicesLayer
Information Resource mgmt
Security Data access Fault detection
. . .
. . .
High-energyphysics data
analysis Regionalclimate studies
Collaborativeengineering
Parameterstudies
On-lineinstrumentation
ApplicationToolkit Layer
Highthroughput
Data-intensive
Collab.design
Remoteviz
Remote control
Architecture
Application
Collective
Resource
Connectivity
Fabric
Application
Transport
Internet
Link
GRIDInternet
Fabric Layer• Fabric layer: Provides the resources to which shared
access is mediated by Grid protocols.• Example: computational resources, storage
systems, catalogs, network resources, and sensors.• Fabric components implement local, resource
specific operations.• Richer fabric functionality enables more
sophisticated sharing operations.• Sample resources: computational resources,
storage resources, network resources, code repositories, catalogs.
Connectivity Layer
• Communicating easily and securely.• Connectivity layer defines the core
communication and authentication protocols required for grid-specific network functions.
• This enables the exchange of data between fabric layer resources.
• Support for this layer is drawn from TCP/IP’s IP, TCL and DNS layers.
• Authentication solutions: single sign on, etc.
Resources Layer• Resource layer defines protocols, APIs, and SDKs for
secure negotiations, initiation, monitoring control, accounting and payment of sharing operations on individual resources.
• Two protocols information protocol and management protocol define this layer.
• Information protocols are used to obtain the information about the structure and state of the resource, ex: configuration, current load and usage policy.
• Management protocols are used to negotiate access to the shared resource, specifying for example qos, advanced reservation, etc.
Collective Layer
• Coordinating multiple resources.• Contains protocols and services that capture
interactions among a collection of resources.• It supports a variety of sharing behaviors without
placing new requirements on the resources being shared.
• Sample services: directory services, coallocation, brokering and scheduling services, data replication service, workload management services, collaboratory services.
Applications Layer
• These are user applications that operate within VO environment.
• Applications are constructed by calling upon services defined at any layer.
• Each of the layers are well defined using protocols, provide access to useful services.
• Well defined APIs also exist to work with these services.
• A toolkit Globus implements all these layers and supports grid application development.
Globus Toolkit Services
• Security (GSI)– PKI-based Security (Authentication) Service
• Job submission and management (GRAM)– Uniform Job Submission
• Information services (MDS)– LDAP-based Information Service
• Remote file management (GASS)– Remote Storage Access Service
• Remote Data Catalogue and Management Tools– Support by Globus 2.0 released in 2002
High-level services
Part II
Sample of High-Level Services
• Resource brokers and co-allocators– DUROC, Nimrod/G, Condor-G, GridbusBroker
Communication & I/O libraries– MPICH-G, PAWS, RIO (MPI-IO), PPFS, MOL
• Parallel languages– HPC++, CC++, Nimrod Parameter Specification
• Collaborative environments– CAVERNsoft, ManyWorlds
• Others– MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow
• A resource broker for managing, steering, and executing task farming (parameter sweep/SPMD model) applications on the Grid based on deadline and computational economy.
• Based on users’ QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability.
• Key Features– A single window to manage & control experiment– Persistent and Programmable Task Farming Engine– Resource Discovery– Resource Trading – Scheduling & Predications– Generic Dispatcher & Grid Agents– Transportation of data & results– Steering & data management– Accounting
• Uses Globus – MDS, GRAM, GSI, GASS
The Nimrod-G Grid Resource Broker
Condor-G: Condor for the Grid• Condor is a high-throughput scheduler• Condor-G uses Globus Toolkit libraries for:
– Security (GSI)– Managing remote jobs on Grid (GRAM)– File staging & remote I/O (GSI-FTP)
• Grid job management interface & scheduling– Robust replacement for Globus Toolkit programs
• Globus Toolkit focus is on libraries and services, not end user vertical solutions
– Supports single or high-throughput apps on Grid• Personal job manager which can exploit Grid resources
Production Grids & Testbeds
• Production deployments underway at:– NSF PACIs National Technology Grid– NASA Information Power Grid– DOE ASCI– European Grid
• Research testbeds– EMERGE: Advance reservation & QoS– GUSTO: Globus Ubiquitous Supercomputing Testbed
Organization– Particle Physics Data Grid– World-Wide Grid (WWG)
Production Grids & Testbeds
NASA’s Information Power Grid The Alliance National Technology Grid
GUSTO Testbed
World Wide Grid (WWG)
GMonitor
@ SC 2002/Baltimore
Grid MarketDirectory
Australia
Melbourne+Monash U:
VPAC, Physics
Solaris WS
Gridbus+Nimrod-G
Europe
ZIB: T3E/OnyxAEI: Onyx CNR: ClusterCUNI/CZ: OnyxPozman: SGI/SP2Vrije U: ClusterCardiff: Sun E6500Portsmouth: Linux PCManchester: O3KCambridge: SGIMany others
Asia
AIST, Japan: Solaris ClusterOsaka University: ClusterDoshia: Linux clusterKorea: Linux cluster
North America
ANL: SGI/Sun/SP2NCSA: ClusterWisc: PC/clusterNRC, CanadaMany others
InternetWW Grid
MEG Visualisation
Example Applications Projects (via Nimrod-G or Gridbus)
• Molecular Docking for Drug Discovery– Docking molecules from chemical databases with
target protein
• Neuro Science– Brain Activity Analysis
• High Energy Physics– Belle Detector Data Analysis
• Natural Language Engineering– Analyzing audio data (e.g., to identify emotional state
of a person!)
Example Application Projects
• Computed microtomography (ANL, ISI)– Real-time, collaborative analysis of data from X-Ray
source (and electron microscope)
• Hydrology (ISI, UMD, UT; also NCSA, Wisc.)– Interactive modeling and data analysis
• Collaborative engineering (“tele-immersion”)– CAVERNsoft @ EVL
• OVERFLOW (NASA)– Large CFD simulations for aerospace vehicles
Example Application Experiments
• Distributed interactive simulation (CIT, ISI)– Record-setting SF-Express simulation
• Cactus– Astrophysics simulation, viz, and steering– Including trans-Atlantic experiments
• Particle Physics Data Grid– High Energy Physics distributed data analysis
• Earth Systems Grid– Climate modeling data management
The Globus Advantage
• Flexible Resource Specification Language which provides the necessary power to express the required constraints
• Services for resource co-allocation, executable staging, remote data access and I/O streaming
• Integration of these services into high-level tools– MPICH-G: grid-enabled MPI– globus-job-*: flexible remote execution commands– Nimrod-G Grid Resource broker
– Gridbus: Grid Business Infrastructure– Condor-G: high-throughput broker– PBS, GRD: meta-schedulers
Resource Management
• Resource Specification Language (RSL) is used to communicate requirements
• The Globus Resource Allocation Manager (GRAM) API allows programs to be started on remote resources, despite local heterogeneity
• A layered architecture allows application-specific resource brokers and co-allocators to be defined in terms of GRAM services
GRAM GRAM GRAM
LSF EASY-LL NQE
Application
RSL
Simple ground RSL
Information Service
Localresourcemanagers
RSLspecialization
Broker
Ground RSL
Co-allocator
Queries& Info
Resource Management Architecture
GRAM Components
Globus SecurityInfrastructure
Job Manager
GRAM client API calls to request resource allocation
and process creation.
MDS client API callsto locate resources
Query current statusof resource
Create
RSL Library
Parse
RequestAllocate &
create processes
Process
Process
Process
Monitor &control
Site boundary
Client MDS: Grid Index Info Server
Gatekeeper
MDS: Grid Resource Info Server
Local Resource Manager
MDS client API callsto get resource info
GRAM client API statechange callbacks
A simple run
• [raj@belle raj]$ globus-job-run belle.anu.edu.au /bin/date
• Mon May 3 15:05:42 EST 2004
Resource Specification Language (RSL)
• Common notation for exchange of information between components– Syntax similar to MDS/LDAP filters
• RSL provides two types of information:– Resource requirements: Machine type,
number of nodes, memory, etc.– Job configuration: Directory, executable, args,
environment
• API provided for manipulating RSL
RSL Syntax
• Elementary form: parenthesis clauses– (attribute op value [ value … ] )
• Operators Supported:– <, <=, =, >=, > , !=
• Some supported attributes:– executable, arguments, environment, stdin, stdout,
stderr, resourceManagerContact,resourceManagerName
• Unknown attributes are passed through – May be handled by subsequent tools
Constraints: “&”
• globusrun -o -r belle.anu.edu.au "&(executable=/bin/date)"
• For example:
& (count>=5) (count<=10)
(max_time=240) (memory>=64)
(executable=myprog)
“Create 5-10 instances of myprog, each on a machine with at least 64 MB memory that is available to me for 4 hours”
Disjunction: “|”
• For example:
• & (executable=myprog)
• ( | (&(count=5)(memory>=64))
• (&(count=10)(memory>=32)))
• Create 5 instances of myprog on a machine that has at least 64MB of memory, or 10 instances on a machine with at least 32MB of memory
Multirequest: “+”
• A multi-request allows us to specify multiple resource needs, for example
+ (& (count=5)(memory>=64)
(executable=p1))
(&(network=atm) (executable=p2))– Execute 5 instances of p1 on a machine with at least
64M of memory– Execute p2 on a machine with an ATM connection
• Multirequests are central to co-allocation
Co-allocation
• Simultaneous allocation of a resource set– Handled via optimistic co-allocation based on free
nodes or queue prediction
– In the future, advance reservations will also be supported
• globusrun and globus-job-* will co-allocate specific multi-requests– Uses a Globus component called the Dynamically
Updated Request Online Co-allocator (DUROC)
DUROC Functions
• Submit a multi-request• Edit a pending request
– Add new nodes, edit out failed nodes
• Commit to configuration– Delay to last possible minute– Barrier synchronization
• Initialize computation– Bootstrap library
• Monitor and control collection
DUROC Architecture
ControllingApplication
ControlledJobs
RSL multi-request
Job 1
RM1
Job 4
Job 5
RM4
Job 2
RM2
Job 3
RM3
Edit request
Subjob status
Barrier
RSL Creation Using globus-job-run
• globus-job-run can be used to generate RSL from command-line args:globus-job-run –dumprsl \
-: host1 -np N1 [-s] executable1 args1 \ -: host2 -np N2 [-s] executable2 args2 \ ... > rslfile
– -np: number of processors– -s: stage file– argument options for all RSL keywords– -help: description of all options
Job Submission Interfaces
• Globus Toolkit includes several command line programs for job submission – globus-job-run: Interactive jobs– globus-job-submit: Batch/offline jobs– globusrun: Flexible scripting infrastructure
• Other High Level Interfaces– General purpose
• Nimrod-G, Condor-G, PBS, GRD, etc
– Application specific• ECCE’, Cactus, Web portals
globus-job-run
• For running of interactive jobs• Additional functionality beyond rsh
– Ex: Run 2 process job w/ executable stagingglobus-job-run -: host –np 2 –s myprog arg1 arg2
– Ex: Run 5 processes across 2 hostsglobus-job-run \
-: host1 –np 2 –s myprog.linux arg1 \
-: host2 –np 3 –s myprog.aix arg2
– For list of arguments run:
globus-job-run -help
globus-job-submit
• For running of batch/offline jobs– globus-job-submit Submit job
• Same interface as globus-job-run• Returns immediately
– globus-job-status Check job status– globus-job-cancel Cancel job– globus-job-get-output Get job stdout/err– globus-job-clean Cleanup after job
globusrun
• Flexible job submission for scripting– Uses an RSL string to specify job request – Contains an embedded globus-gass-server
• Defines GASS URL prefix in RSL substitution variable:
(stdout=$(GLOBUSRUN_GASS_URL)/stdout)
– Supports both interactive and offline jobs
• Complex to use– Must write RSL by hand– Must understand its esoteric features– Generally you should use globus-job-* commands
instead
Simultaneous start
co-allocator
InformationService
“Run SF-Expresson 300 nodes”
"Run SF-Expresson 256 nodes”
“Run adistributed interactive
simulation involving100,000 entities”
“80 nodes on Argonne SP,256 nodes on CIT Exemplar300 nodes on NCSA O2000”
“Supercomputers providing 100 GFLOPS, 100 GB, < 100 msec latency”DIS-Specific
Broker
" . . ."
“Performa parameter studyinvolving 10,000separate trials”
Parameter studyspecific broker
Supercomputerresource broker
NCSAResource Manager
ArgonneResource Manager
CITResource Manager
Resource Brokers
" . . ."
“Create ashared virtual space
with participantsX, Y, and Z”
Collaborativeenvironment-specific
resource broker
"Run SF-Expresson 80 nodes”
Brokering via Lowering
• Resource location by refining a RSL expression (RSL lowering):
(MFLOPS=1000) (& (arch=sp2)(count=200)) (+ (& (arch=sp2) (count=120)
(resourceManagerContact=anlsp2))
(& (arch=sp2) (count=80)
(resourceManagerContact=uhsp2)))
Remote I/O and Staging
• Tell GRAM to pull executable from remote location
• Access files from a remote location
• stdin/stdout/stderr from a remote location
What is GASS?
(a) GASS file access API– Replace open/close with
globus_gass_open/close; read/write calls can then proceed directly
(b) RSL extensions – URLs used to name executables, stdout, stderr
(c) Remote cache management utility
(d) Low-level APIs for specialized behaviors
GASS Architecture
CacheCache
GASS Server
HTTP Server
FTP Server
% globus-gass-cache
(c) Remote cache management
GRAM
(a) GASS file access API
&(executable=https://…)
(b) RSL extensions
(d) Low-level APIs for customizing cache & GASS server
main( ) { fd = globus_gass_open(…) … read(fd,…) … globus_gass_close(fd)}
GASS File Naming
• URL encoding of resource nameshttps://quad.mcs.anl.gov:9991/~bester/myjob
protocol server address file name
• Other exampleshttps://pitcairn.mcs.anl.gov/tmp/input_dataset.1
https://pitcairn.mcs.anl.gov:2222/./output_data
http://www.globus.org/~bester/input_dataset.2
• Supports http & https• Support ftp & gsiftp.
GASS RSL Extensions
• executable, stdin, stdout, stderr can be local files or URLs
• executable and stdin loaded into local cache before job begins (on front-end node)
• stdout, stderr handled via GASS append mode
• Cache cleaned after job completes
GASS/RSL Example
&(executable=https://quad:1234/~/myexe) (stdin=https://quad:1234/~/myin) (stdout=/home/bester/output) (stderr=https://quad:1234/dev/stdout)
Example GASS Applications
• On-demand, transparent loading of data sets
• Caching of data sets
• Automatic staging of code and data to remote supercomputers
• (Near) real-time logging of application output to remote server
GASS File Access API
• Minimum changes to application• globus_gass_open(), globus_gass_close()
– Same as open(), close() but use URLs instead of filenames
– Caches URL in case of multiple opens– Return descriptors to files in local cache or
sockets to remote server
• globus_gass_fopen(), globus_gass_fclose()
GASS File Access API (cont)
• Support for different access patterns– Read-only (from local cache)– Write-only (to local cache)– Read-write (to/from local cache)– Write-only, append (to remote server)
Remove cachereference
Upload changes
Modified no
yes
globus_gass_open()/close()
Download Fileinto cache
open cached file,add cachereference
URL in cache? no
yes
globus_gass_open()
globus_gass_close()
GASS File API Semantics
• Copy-on-open to cache if not truncate or write-only append and not already in cache
• Copy on close from cache if not read only and not other copies open
• Multiple globus_gass_open() calls share local copy of file
• Append to remote file if write only append: e.g., for stdout and stderr
• Reference counting keeps track of open files
globus-gass-server
• Simple file server– Run by user wherever necessary– Secure https protocol, using GSI– APIs for embedding server into other programs
• Exampleglobus-gass-server –r –w -t
– -r: Allow files to be read from this server– -w: Allow files to be written to this server– -t: Tilde expand (~/… $(HOME)/…)– -help: For list of all options
1. Derive Contact String2. Build RSL string3. Startup GASS server4. Submit to request5. Return output
jobmanager
gatekeeper
program
GRAM & GASS: Putting It Together
stdout
GASS server
3
4
globus-job-run
Host name
Contactstring
1
RSLstring
2CommandLine Args
4
4
55
55
Example: A Simple Broker
• Select machines based on availability– Use MDS queries to get current host loads– Look at output and figure out what machines
to use
• Generate RSL based on selection– globus-job-run -dumprsl can assist
• Execute globusrun, feeding it the RSL generated in previous step
GRAM & GASS
• Using RSL with globusrun
• Running globus-gass-server
• Modifying a program to use globus_gass_open() to read files remotely from a GASS server
Globus Components In ActionLocal Machine
mpirun
globusrun
GRAM
ClientGSI
GRAM
ClientGSI
Remote Machine
AppNexus
AIX
PBS
MPI
grid-proxy-initX509UserCert
UserProxyCert
Machines
GRAM Gatekeeper
GSI
GRAM Job Manager
GASS Client
Remote Machine
AppNexus
Solaris
Unix Fork
MPI
GRAM Gatekeeper
GSI
GRAM Job Manager
GASS Client
RSL string
RSL multi-request
RSL single requestDUROC
GASS Server
RSL parser
GRAM Components
Globus SecurityInfrastructure
Job Manager
GRAM client API calls to request resource allocation
and process creation.
MDS client API callsto locate resources
Query current statusof resource
Create
RSL Library
Parse
RequestAllocate &
create processes
Process
Process
Process
Monitor &control
Site boundary
Client MDS: Grid Index Info Server
Gatekeeper
MDS: Grid Resource Info Server
Local Resource Manager
MDS client API callsto get resource info
GRAM client API statechange callbacks
MDS: Monitoring and Discovery Service
• Learn how to use the MDS to locate and determine characteristics of resources
• Locate resources– Where are resources with required
architecture, installed software, available capacity, network bandwidth, etc.?
• Determine resource characteristics– What are the physical characteristics,
connectivity, capabilities of a resource?
The Need for Information
• System information is critical to operation of the grid and construction of applications– How does an application determine what resources
are available?– What is the “state” of the computational grid?– How can we optimize an application based on
configuration of the underlying system?
• We need a general information infrastructure to answer these questions
Using Information forResource Brokering
“10 GFlops, EOS data,20 Mb/sec -- for 20 mins”
MetacomputingDirectoryService
GRAMGRAMGRAM
ResourceBroker
Info service:location + selection
Globus ResourceAllocation Managers
GRAM
ForkLSFEASYLLCondoretc.
“What computers?”“What speed?”“When available?”
“50 processors + storage from 10:20 to 10:40 pm”
“20 Mb/sec”
Examples of Useful Information
• Characteristics of a compute resource– IP address, software available, system
administrator, networks connected to, OS version, load
• Characteristics of a network– Bandwidth and latency, protocols, logical
topology
• Characteristics of the Globus infrastructure– Hosts, resource managers
Grid Information Service
• Provide access to static and dynamic information regarding system components
• A basis for configuration and adaptation in heterogeneous, dynamic environments
• Requirements and characteristics– Uniform, flexible access to information– Scalable, efficient access to dynamic data– Access to multiple information sources– Decentralized maintenance
MDS
• Store information in a distributed directories– Directory stored in collection of LDAP servers– Each server optimized for particular function
• Directory can be updated by – Information providers and tools– Applications (i.e., users)– Backend tools which generate info on demand
• Information dynamically available to – Tools– Applications
Directory Service Functions
• White Pages– Look up the IP number, amount of memory, etc., associated
with a particular machine
• Yellow Pages– Find all the computers of a particular class or with a
particular property
• Temporary inconsistencies are often considered okay– In a distributed system, you often do not know the state of a
resource until you actually use it– Information is often used as “hints”– Information itself can contain ttl, etc.
MDS Approach
• Based on LDAP– Lightweight Directory Access Protocol v3 (LDAPv3)– Standard data model– Standard query protocol
• Globus specific schema– Host-centric representation
• Globus specific tools– GRIS, GIIS– Data discovery, publication,…
SNMP
GRIS
NIS
NWS
LDAP
LDAP API
Middleware
…
Application
GIIS…
MDS Components
• Uses standard LDAP servers– OpenLDAP, Netscape, Oracle, etc
• Tools for populating & maintaining MDS– Integrated with Globus Toolkit server release, not of
concern to most Globus users– Discover/update static and dynamic info
• APIs for accessing & updating MDS contents– C, Java, PERL (LDAP API, JNDI)
• Various tools for manipulating MDS contents– Command line tools, Shell scripts & GUIs
Anonymous Grid info search
• grid-info-search -x -h belle.anu.edu.au….Mds-Computer-isa: i686Mds-Computer-platform: i686Mds-Computer-Total-nodeCount: 1Mds-Cpu-Cache-l2kB: 512Mds-Cpu-features: fpu vme de pse tsc msr pae mce cx8 apic sep
mtrr pge mca cmo v pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tmMds-Cpu-Free-15minX100: 400Mds-Cpu-Free-1minX100: 400Mds-Cpu-Free-5minX100: 400Mds-Cpu-model: Intel(R) Xeon(TM) CPU 2…
Summary
• MDS provides the information needed to perform dynamic resource discovery and configuration– Critical component of resource brokers
• MDS is base on existing directory service standards (LDAPv3)