globus grid tutorial part 3: information services
DESCRIPTION
Globus Grid Tutorial Part 3: Information Services. Goals of this Tutorial. Learn how to use the Globus Metacomputing Directory Service to locate and determine characteristics of resources Locate resources - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/1.jpg)
Globus Grid TutorialPart 3:
Information Services
![Page 2: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/2.jpg)
Goals of this Tutorial
Learn how to use the Globus Metacomputing Directory Service to locate and determine characteristics of resources
Locate resources Where are resources with required
architecture, installed software, available capacity, network bandwidth, etc.?
Determine resource characteristics What are the physical characteristics,
connectivity, capabilities of a resource?
![Page 3: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/3.jpg)
The Need for Information
System information is critical to operation of the grid and construction of applications How does an application determine what
resources are available? What is the “state” of the computational
grid? How can we optimize an application based on
configuration of the underlying system? We need a general information
infrastructure to answer these questions
![Page 4: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/4.jpg)
Using Information forResource Brokering
“10 GFlops, EOS data,20 Mb/sec -- for 20 mins”
MetacomputingDirectoryService
GRAMGRAMGRAM
ResourceBroker
Info service:location + selection
Globus ResourceAllocation Managers
GRAM
ForkLSFEASYLLCondoretc.
“What computers?”“What speed?”“When available?”
“50 processors + storage from 10:20 to 10:40 pm”
“20 Mb/sec”
![Page 5: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/5.jpg)
Examples of Useful Information
Characteristics of a compute resource IP address, software available, system
administrator, networks connected to, OS version, load
Characteristics of a network Bandwidth and latency, protocols, logical
topology Characteristics of the Globus infrastructure
Hosts, resource managers
![Page 6: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/6.jpg)
Information Infrastructure
Provide access to static and dynamic information regarding system components
A basis for configuration and adaptation in heterogeneous, dynamic environments
Requirements and characteristics Uniform, flexible access to information Scalable, efficient access to dynamic data Access to multiple information sources Decentralized maintenance
![Page 7: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/7.jpg)
The GlobusMetacomputing Directory Service
Store information in a distributed directory Directory stored in collection of servers
Directory can be updated by Globus system Other information providers and tools Applications (i.e., users)
Information dynamically available to Tools Applications
![Page 8: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/8.jpg)
Directory Service Functions White PagesWhite Pages
Look upLook up the IP number, amount of memory, and so forth, associated with a particular machine
Yellow PagesYellow Pages FindFind all the computers of a particular class
or with a particular property Tuned for high-volume lookup or search Temporary inconsistencies are considered
okay
![Page 9: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/9.jpg)
The Metacomputing Directory Service
Standard interface and protocol Lightweight Directory Access
Protocol (LDAP) Add gateways and pragmas
Globus specific data model Host-centric representation
Data discovery & update SNMP
MDS Client Library
NIS
NWS
LDAP
LDAP API& represent.
Middleware
Datamodel
…
Application
![Page 10: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/10.jpg)
MDS Components
Standard LDAP server Netscape Directory Server v4
Tools for populating & maintaining MDS Integrated with Globus server release, not of
concern to most Globus users Discover static info, update dynamic info
APIs for accessing & updating MDS contents C, Java, PERL (LDAP API, JNDI)
Various tools for manipulating MDS contents Command line tools, Shell scripts & GUIs
![Page 11: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/11.jpg)
LDAP Directory Service
Lightweight Directory Access Protocol Stripped down version of X.500 DAP protocol Supports distributed storage/access (referrals) Supports replication Becoming de facto standard
Defines: Network protocol for accessing directory contents Information model defining form of information Namespace defining how information is
referenced and organized
![Page 12: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/12.jpg)
Directory Structure
Directory contents What information is stored in directory Group related information into entries
Directory organization Objects organized into tree structure Position of object in tree uniquely names
entry
![Page 13: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/13.jpg)
Entries and Object Classes
Directory entry consists of related set of attributes and values
Entries are typed by their Object Class Object class specifications define
Required and optional attributes Attribute types Single inheritance on object classes
![Page 14: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/14.jpg)
Sample Object Classes
Compute Resources Operating System Memory Hierarchy Health and Status
Network Interfaces IP address Interface types
Performance Data Schedule Jobs CPU Loads Network Traffic
Resource Managers Contact strings Scheduled jobs Free nodes
Software Configuration Version Control Contact information
Organizations People
![Page 15: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/15.jpg)
Object Class SpecificationGlobusHost OBJECT CLASS SUBCLASS OF GlobusPhysicalResource RDN = hn (hostName) CHILD OF { organization } MUST CONTAIN { type:: cis, vendor :: cis, model :: cis, ostype :: cis, osversion :: cis } MAY CONTAIN { networkNode :: dn, totalMemory :: cis, totalSwap :: cis, . . . }
GlobusPhysicalResource OBJECT CLASS SUBCLASS OF GlobusTop RDN = rn (resourceName) MUST CONTAIN { imageObject :: dn } MAY CONTAIN { manager :: dn, provider :: dn, technician :: dn, description :: cis, documentation :: cis }
![Page 16: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/16.jpg)
The Directory Information Tree
Directory entries organized into tree Called Directory Information Tree (DIT) Subtrees can be distributed or replicated
Position in tree uniquely names entry Each object is uniquely determined by its
distinguished name (DN) List of unique attribute names and values
along path from root of DIT to object, e.g.:<hn=sp2.sdsc.edu, ou=SDSC, o=NPACI, o=Globus, c=US>
![Page 17: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/17.jpg)
MDS Data Model
Carl Steve
Switch
Ethernet
Ian Gregor SteveWarren
sunny
hot
IBMSP
dark coldLAN
LANWAN
USC/ISI ANL/MCS
c=US
o=globus
o=USC o=ANL
ou=MCSou=ISI
nn=WAN
cn=Ian
cn=Gregor
cn=Warren
cn=Steve
nn=SP-switch
nn=SP-ether
nn=MCS-lan
hn=sp1.mcs.anl.gov
hn=spN.mcs.anl.gov
hn=cold.mcs…
hn=dark.mcs…
cn=Carl
cn=Steve…
…
……
Physical StructureDirectoryInformationTree
<hn= dark.mcs.anl.gov, ou= MCS, o = Argonne National Laboratory, o = Globus, c = US>
Distinguished Name
![Page 18: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/18.jpg)
MDS Tools
Web-based browsers and displays MDS Object Class Browser MDS Explorer
Various APIs and search tools Translators from “Globus Object Definition
Language” Commented LDIF LDAP schema definition language HTML
![Page 19: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/19.jpg)
MDS Object Class Browser
![Page 20: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/20.jpg)
![Page 21: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/21.jpg)
MDS Access/Update Commands
LDAP defines a set of standard commands
ldapsearch, ldapmodify, ldapdelete, etc. We also define MDS-specific commands
grid-info-search, grid-info-create, grid-info-update, grid-info-remove
Routines to ensure data consistency and to insert metadata
APIs are defined for C, Java, etc.
ldap_search_s, ldap_modify_s, ….
![Page 22: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/22.jpg)
Searching an LDAP Database
grid-info-search [options] filter [attributes]grid-info-search [options] filter [attributes]
Default Default grid-info-searchgrid-info-search options-h mds.globus.org MDS server-p 389 MDS port-b “o=Globus, c=US” search start point-T 30 LDAP query timeout-s sub scope = subtree
alternatives: base : lookup this entry one : lookup immediate children
![Page 23: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/23.jpg)
grid-info-search: “Filter”
Filters allow selection of object based on relational operators (=, ~=,<=, >=) grid-info-search “cputype=*”
Compound filters can be construct with Boolean operations: (&, |, !) grid-info-search “(&(cputype=*)(cpuload1<=1.0))” grid-info-search “(&(hn~=sdsc.edu)(latency<=10))”
Hints: white space is significant use -L for LDIF format
required
![Page 24: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/24.jpg)
Example 1: grid-info-search “(mn~=sdsc.edu)”
cn=sp097.sdsc.edu-loadleveler, ou=SDSC, o=NPACI, o=Globus, c=USobjectclass=GlobusResourceManagercn=sp097.sdsc.edu-loadlevelermn=sp097.sdsc.edu-loadlevelerhn=sp097.sdsc.educontact=sp097.sdsc.edu:754:/C=US/O=Globus/O=NPACI/OU=SDSC/ \ CN=sp097.sdsc.edu-loadlevelerapiversion=0.1apitype=loadlevelermanagedresources=hn=sp097.sdsc.edu, ou=SDSC, o=NPACI, \ o=Globus, c=USgramversion=1.53gramversiondate=1998/06/1921:17:23gramsecurity=ssleayfreenodes=49totalnodes=117lastupdate=Sat Jun 27 18:23:34 GMT 1998scheduledjob=none
![Page 25: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/25.jpg)
Example 2:grid-info-search -L “(sw=Globus)”
dn: sw=Globus, hn=host.domain.edu, o=My Organization, o=Globus, c=USsw: Globusobjectclass: GlobusSoftwarereleasemajor: 1releaseminor: 1releasepatch: 0configuration: --prefix=/cpc/globus/service/sun4-sparc2 \ --with-ssl-path=/cpc/ssl/sun4-sparc2 configuration: --prefix=/cpc/globus/tools/sun4-sparc2 \ --with-ssl-path=/cpc/ssl/sun4-sparc2 installprefix: /cpc/globus/serviceprefix: /cpc/globus/servicetoolsprefix: /cpc/globus/toolsdeployprefix: /opt/globus/userid: globusobjectname: sw=Globus, hn=host.domain.edu, o=My Organization, \ o=Globus, c=USlastupdate: Wed Jan 20, 06:06:13 GMT 1999
![Page 26: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/26.jpg)
grid-info-search: “Attributes”
Projections on the data objects are performed by supplying a list of attributes
Retrieve the DN for all objects in the MDS:% grid-info-search -L “(objectclass=*)” dn
dn: o=Globus, c=US
dn: o=University of Southern California, o=Globus, c=US
dn: ou=MCS, o=Argonne National Laboratory, o=Globus, c=US
dn: o=NASA Ames, o=Globus, c=US
dn: ou=SDSC, o=NPACI, o=Globus, c=US
![Page 27: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/27.jpg)
Multiple Attribute Selection
% grid-info-search -L “(objectclass=*)” dn hn Returns the distinguished name (dn) and hostname
(hn) of all objects
Objects without hn fields are still listed DNs are always listed
dn: o=CalTech, o=Globus, c=US
dn: hn=neptune.cacr.caltech.edu, o=CalTech, o=Globus, c=UShn: neptune.isi.edu
dn: ou=SDSC, o=NPACI, o=Globus, c=US
dn: o=University of Southern California, o=Globus, c=US
![Page 28: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/28.jpg)
Efficient Lookups
Naïve searches retrieve lots of information % grid-info-search “(objectclass=*)” | grep contact
Whenever possible, reduce search domain Specify a search base (-b) % grid-info-search -b “o=NASA Ames, o=Globus, C=US”
Specify the scope of the search (-s)options: base, one, and sub
Narrow the amount of objects selected, i.e., use specialized filters
Specify which attributes you want
![Page 29: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/29.jpg)
Authentication to MDS
Authentication is required to perform certain operations (e.g., write operations)
Each site has a Directory Manager cn=Directory Manager, o=Organization, c=US
Users registered with the MDS have a DN cn=Jane Doe, o=Organization, c=US
% grid-info-search -D “cn=Jane Doe, o=Organization, c=US” \
-w <passwd>
![Page 30: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/30.jpg)
Updating MDS Entries
To update an existing database entry:grid-info-modify -D "cn=Directory Manager, o=Globus, c=US” \
-w <PASSWORD> -f sample.ldif
dn: hn=tuva.mcs.anl.gov, ou=MCS, o=Argonne National Laboratory, o=Globus, c=US
cpuload1: 3.11cpuload5: 2.64cpuload15: 2.57
Metadata is inserted by grid-info-updategrid-info-update -D "cn=Directory Manager, o=Globus, c=US” \
-w <PASSWORD> -f sample.cldif
![Page 31: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/31.jpg)
Example: Discovering CPU Load
Retrieve CPU load fields of compute resourcesgrid-info-search -L “(objectclass=GlobusComputeResource)” \
dn cpuload1 cpuload5 cpuload15 dn: hn=lemon.mcs.anl.gov, ou=MCS, o=Argonne National Laboratory, o=Globus, c=UScpuload1: 0.48cpuload5: 0.20cpuload15: 0.03 dn: hn=tuva.mcs.anl.gov, ou=MCS, o=Argonne National Laboratory, o=Globus, c=UScpuload1: 3.11cpuload5: 2.64cpuload15: 2.57
![Page 32: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/32.jpg)
Exercise 3Extended Exercises
Explore the MDS via MDS object code Browser
http://www.globus.org/mds MDS Navigator
http://www.globus.org/testbeds Globus View
http://www.globus.org/testbeds
Write a set of MDS/LDAP queries% grid-info-search “hn~=isi.edu”
![Page 33: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/33.jpg)
MDS Search Exercises
1. List all hosts that have a “lsf” jobmanager type.
2. List all hosts that have a CPU load greater than 5.
3. List all hosts with 16 or more CPUs.
4. Determine the current versions of Globus installed at Argonne.
5. Determine the deployment directory on flash.isi.edu.
6. Determine the bandwidth between pitcairn.mcs.anl.gov and bolas.isi.edu.
![Page 34: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/34.jpg)
MDS Search Answers (1)
1. List all hosts that have a “lsf” gatekeeper. grid-info-search “apitype=lsf” hostname
2. List all hosts that have a CPU load greater than 5. grid-info-search “cpuload1>=5” hostname
3. List all hosts with 16 or more CPUs. grid-info-search “cpucount>=16” hostname
4. Determine the current versions of Globus installed at Argonne.
grid-info-search \ -b ‘o=Argonne National Laboratory, o=Globus, c=US’ \ ‘sw=Globus’ releasemajor releaseminor releasepatch
![Page 35: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/35.jpg)
MDS Search Answers (2)
5. Determine the deployment directory on flash.isi.edu. grid-info-search \
‘(&(objectclass=*ResourceManager)(hn=flash.isi.edu))’ \ deploydir
6. Determine the bandwidth between pitcairn.mcs.anl.gov and bolas.isi.edu.
grid-info-search \ “(|(&(sourcehostname=pitcairn.mcs.anl.gov) \ (destinationhostname=bolas.isi.edu)) \ (&(sourcehostname=bolas.isi.edu) \ (destinationhostname=pitcairn.mcs.anl.gov)))” bandwidth
![Page 36: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/36.jpg)
Summary
MDS provides the information needed to perform dynamic resource discovery and configuration Critical component of resource brokers
MDS is base on existing directory service standards (LDAPv3)
![Page 37: Globus Grid Tutorial Part 3: Information Services](https://reader034.vdocuments.site/reader034/viewer/2022051416/56813bbc550346895da4ec1d/html5/thumbnails/37.jpg)
Changes from 1.0 to 1.1
Tools are renamed
globus-mds-{search,add,delete,modify} became
grid-info-{search,add,delete,modify}
Configurable LDAP search timeout (-T)