manchester computing supercomputing, visualization & e-science course instructors: mark mc keown...

Course Instructors: Mark Mc Keown (UoM) Mike Jones (UoM) Matt Viljoen (RAL)

Installing Globus GT2 and joining the L2G

UK Grid Support Centre

Supercomputing, Visualization & e-Science2

Explain what a Grid is and provide some background information. Describe the components of the Globus Toolkit Version 2 and its

release mechanism. Install from the source distribution;

– Showing dependant software such as compilers/interpreters.

Setup and start all services;– Show common housekeeping tasks such as user addition, log file interpretation etc.

Configuration of more in depth tasks;– Hierarchal MDS setup.– Alternative job manager setup.– Recognition of multiple certification authorities.

Introduction

Grid Support Centre

Part of the e-Science Core Programme

Team of 6 FTE posts based at CCLRC (RAL+DL), and Universities of Edinburgh and Manchester

Exists to help users install and apply Grid software quickly, easily and productively

(2003) Two additional posts to cover support for OGSA-DAI and the ETF

Grid Support Centre

Partnership between CCLRC University of Edinburgh University of Manchester

Supports UK e-Science programme

www.grid-support.ac.ukGlobus

Installationsupport

Softwaredownload

Evaluationreports

Links to otherresources

Documentation

NationalDirectoryService

Referencesystems

System admintraining

NationalCertificateAuthority

CondorSRB

Webservices

Network

Web InformationResource

www.grid-support.ac.uk

Helpdesksupport@grid-support.ac.uk

Technical Support Team

UK Grid Support CentreCLRC Rutherford Appleton and Daresbury Laboratories

and Universities of Edinburgh and Manchester

Research CouncilPilot Projects

Regional e-ScienceCentres

Research Labs

Universities

Industry

GSC provides

Helpdesk support@grid-support.ac.uk– first point of contact for requests and queries

– personally contactable during office hours

– provides access to technical expertise at all sites

Web information resource http://www.grid-support.ac.uk– tutorials

– evaluation reports

– links to other resources

Grid starter kit– Globus, Condor, SRB, OGSA-DAI

– downloadable software

– installation support

– documentation

... and other services

Certificate Authority for the UK e-Science programme– Issues X.509 digital certificates (usable with Globus inter alia)– Uses network of registration authorities (RA) to validate users– Monthly training course for RA operators– See http://ca.grid-support.ac.uk

National resource directory service– based on Globus MDS 2– holds published information on Grid-enabled resources

Training for system administrators– to help with setting up local installations

Reference software installations on supported platforms– Linux, AIX, Solaris, IRIX, Tru64

Liaison with software development teams

The Grid…

"…is the web on steroids." "…is Napster for Scientists" [of data grids] "…is the solution to all your problems." "…is evil." [a system manager, of Globus] "…is distributed computing re-badged." "…is distributed computing across multiple administrative domains"

– Dave Snelling, senior architect of UNICORE

[…provides] "Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource"

– From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”

"…enables communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of central location, central control, omniscience, existing trust relationships."

The roots of Grid computing

The roots of Grid computing are in scientific High Performance Computing (HPC), informed by both capacity (high-throughput) and capability (large-scale) considerations.

The Grid is descended from:– Metacomputing

– Distributed computing

– Load balancing and scheduling

It is enabled by increasingly ubiquitous and reliable infrastructure.

It has outgrown its HPC roots.

The power grid analogy

"Computational Grid" was coined by analogy with power grids

• In power grids, plug in your appliance and draw current, without caring where the power is generated

• In computational grids, plug in your application and draw cycles

The Grid Book

2nd edition, now in press, will be more data-oriented– Reflecting a similar evolution in the Grid itself.

Virtual Organisations

What is a Virtual Organization?

• Facilitates the workflow of a group of users across multiple domains who share (some of) their resources to solve particular classes of problems.

• Collates and presents information about these resources in a uniform view.

The UK e-Science community is effectively a Virtual Organization made up of real institutions – the Universities involved.

An organization can of course be part of lots of Virtual Organizations.

Common principles

Single sign-on– Often implying Public Key Infrastructure (PKI)

Standard protocols and services Respect for autonomy of resource owner Layered architectures Higher-level infrastructures hiding heterogeneity of lower

levels Interoperability is paramount

Grid Middleware

Middleware Globus UNICORE Legion and Avaki

Scheduling Sun Grid Engine Load Sharing Facility (LSF)

– from Platform Computing

OpenPBS and PBS(Pro)– from Veridian

Maui scheduler Condor

– could also go under middleware

Data Storage Resource Broker (SRB) Replica Management OGSA-DAI

Web services (WSDL, SOAP, UDDI) IBM Websphere Microsoft .NET Sun Open Net Environment (Sun

PC Grids

Peer-to-Peer computing

Globus Toolkit™

A software toolkit addressing key technical problems in the development of Grid enabled tools, services, and applications

– Offer a modular “bag of technologies”– Enable incremental development of grid-enabled tools and applications – Implement standard Grid protocols and APIs– Make available under liberal open source license

The Globus Toolkit™ centers around four key protocols– Connectivity layer:

• Security: Grid Security Infrastructure (GSI)

– Resource layer:• Resource Management: Grid Resource Allocation Management (GRAM)

• Information Services: Grid Resource Information Protocol (GRIP)

• Data Transfer: Grid File Transfer Protocol (GridFTP)

Summary of Globus features

"Single sign-on" through Grid Security Infrastructure (GSI) Remote execution of jobs

– GRAM, job-managers, Resource Specification Language (RSL)

Grid-FTP– Efficient file transfer; third-party file transfers

MDS (Metacomputing Directory Service)– Resource discovery– GRIS and GIIS– Today, built on LDAP

Co-allocation (DUROC)– Limited by support from scheduling infrastructure

Other GSI-enabled utilities– gsi-ssh, grid-cvs, etc.

Commodity Grid Kits (CoG-kits), Java, Perl, Python Widespread deployment, lots of projects

GT3 and the Future of Grids- well maybe not!

Globus GT3 is based on Grid Services – Grid Services are an extension of Web Services. Grid Services are defined by the Open Grid Service Infrastructure (OGSI) specification – GT3 & OGSI are now obsolete!!

Work is progressing at the Global Grid Forum (GGF) to define the Open Grid Service Architecture (OGSA) which will be used to develop Grids in the future – OGSA is still valid

The UK production Grid will use GT2 based technology until March 2005 – various “testbed” Grids are being deployed in the UK to develop experience with OGSI/OGSA.

OGSI and OGSA

Network

OGSA Enabled

Storage

OGSA Enabled

Servers

OGSA Enabled

Messaging

OGSA Enabled

Directory

OGSA EnabledFile

Systems

OGSA Enabled

Database

OGSA EnabledWorkflo

OGSA Enabled

Security

OGSA Enabled

Web Services

OGSI – Open Grid Services Infrastructure

Web Services

OGSA Architected Services

Applications

Level 2 Grid Project (L2G)

Ambitious project Persistent grid (8 hours per day at 12 centres) Globus toolkit version 2 Simple jobs only (no parallel processing across grid) Simple grid information system No dynamic resource discovery Large variety of resources Large variety of security policies Large variety of usage policies Large variety of naming policies Simple accounting to be provided Simple monitoring to be provided Authentication with X509 certificates

National Grid Service (NGS)

The L2G is built on a best efforts basis – the NGS will be a more production orientated Grid with Service Level Agreements etc.

The NGS which goes live on April 1st 2004, supersedes the L2G

The NGS initially consists of CSAR, HPCx and the four JISC nodes located at RAL, Oxford, Leeds (White Rose Grid) and Manchester.

The JISC nodes consist of two compute nodes with 128 CPUs each and two data nodes with 18TB of storage each.

The JISC nodes are free to use – http://www.ngs.ac.uk/

VDT is a distribution of Globus from the University of Wisconsin, it also includes Condor and some EDG (European Data Grid) software.– http://www.lsc.group.phys.uwm.edu/vdt/

It is much easier to install than Globus, using a packaging tool called pacman – however it may be a Linux only distribution.

People are interested in VDT because they are more willing to include patchs than Globus, this is the case with EDG.

The VDT distribution generally lags the Globus distribution – they use the same version numbers.

The Globus Toolkit

The Globus Toolkit, an Overview

Three component structure (pillars) of the toolkit

Resource management• Allocation of available resources

Information services• Provides information about grid resources

Data management• Accessing and managing data in a grid environment

Resource Management

Resource Management involves the allocation and management of Grid resources. It includes GRAM, DUROC and GASS.

– GRAM: Globus Resource Allocation Manager– DUROC: Dynamically-Updated Request Online

Coallocator – GASS: Global Access to Secondary Storage

Job Submission Architecture

LSF PBS None

How do we write a Batch Submission Script for our job?

Resource Specification Language

Common notation for exchange of information between components– Syntax similar to MDS/LDAP filters

RSL provides two types of information:

– Resource requirements: Machine type, number of nodes, memory, etc.

– Job configuration: Directory, executable, args, environment

Globus Toolkit provides an API/SDK for manipulating RSL

Example RSL file

& (executable = /bin/date)

(directory =/home/globus )

(arguments = arg1 "arg 2") (count = 1)

RSL Syntax

Elementary form: parenthesis clauses– (attribute op value [ value … ] )

Operators Supported:

– <, <=, =, >=, > , !=

Some supported attributes:– executable, arguments, environment, stdin, stdout,

stderr, resourceManagerContact,resourceManagerName

Unknown attributes are passed through – May be handled by subsequent tools

Constraints: “&”

For example:

& (count>=5) (count<=10)

(max_time=240) (memory>=64)

(executable=myprog) “Create 5-10 instances of myprog, each on a

machine with at least 64 MB memory that is available to me for 4 hours”

Disjunction: “|”

For example:

& (executable=myprog)

( | (&(count=5)(memory>=64))

(&(count=10)(memory>=32))) Create 5 instances of myprog on a machine that

has at least 64MB of memory, or 10 instances on a machine with at least 32MB of memory

Information Services

Information Services provide information about Grid resources. This includes MDS, GIIS and GRIS.– MDS: Monitoring and Discovery Service

• GIIS: Grid Index Information Service Usually one per Organisation

• GRIS: Grid Resource Information Service One per machine.

(The GRIS and the GIIS use the same executable with differing configurations)

A MDS Structure

GRIS GRIS GRIS GRISGRIS

The MDS software is based on OpenLDAP – the arguments to grid-info-search are the same as ldapsearch.

The GRIS collects information about the resource by periodically running scripts that query how much free memory the machine has, the amount of free disk space etc. This is then passed to the GIIS.

When a user makes a query at the top level GIIS it can prompt the GRIS’s to refresh the data for their resource.

We might move to bdii in the future.

Data Management

Controls the ability to access and manage data in a grid environment. This includes components:– GridFTP - This is used to move files between grid-enabled storage

systems. – Replica Catologue Mechanism

GridFTP Why FTP?

– Ubiquity enables interoperation with many commodity tools

– Already supports many desired features, easily extended to support others

– Well understood and supported

We use the term GridFTP to refer to– Transfer protocol which meets requirements

– Family of tools which implement the protocol

Supports “Third Party Transfer”. Supports parallel streams which can have a significant

increase in performance.

Third Party Transfer

Control Control

Replica Management

GASS – Global Access to Secondary Storage

The Globus Toolkit provides services for file and executable staging and I/O redirection that are used by GRAM. This is known as Globus Access to Secondary Storage (GASS). (It is part of the resource management pillar)

When files are moved using FTP their permissions are lost – scripts are not longer executable – so we use GASS which retains file permissions.

GASS uses GSI-enabled HTTP as the protocol for data transfer, and a caching algorithm for copying data when necessary.

The globus_gass, globus_gass_transfer, and globus_gass_cache APIs provide programmer access to these capabilities, which are already integrated with the GRAM job submission tools.

What are we installing?

Reference implementation of Grid Protocols GRAM

– globus_gatekeeper (runs as part of xinetd)

MDS– gris (Started as SXXgris)

GridFTP– in.ftpd (runs as part of xinetd)

GSISSH– gsisshd

ce Package Management and Installation

Package Management and Installation

Generally recommended to use the Grid Packaging Toolkit– Comes with various user tools for:

• Installation of binary bundles• Building of source bundles• Updating already installed bundles• Querying installation tree

– For the developer this also has tools for constructing source and binary bundles for further distribution

A similar idea to RPM

GPT and What Is It?

A set of PERL libraries/modules Requires Perl 5.005 or later to be installed Can install independently of Globus

– GPT_LOCATION

Ability to manage post installation setup of packages

Can be used to patch an installation Ensures that no package will be reinstalled

unnecessarily

Method of Toolkit Distribution

Source and binary bundles– E.g. globus-information-services-client-2.4.0-src_bundle.tar.gz– Contains all the information for package

• Dependancies• Post-install setup• Uninstall information

Each pillar has one bundle for each of:– Server– Client– Software development kit

Source Distribution

Advantages:– If you build the code on your own machine then dependencies

during build should be found then.– Stops strange unknown errors occurring due to other installed

applications and their dependant library versions.– Easier to ultimately debug problems within the toolkit should any

occur. Disadvantages:

– Slow due to compilation etc.– Can lead to space problems due to object files etc.

Flavours of Building

Flavours are used to encapsulate compile-time choices– Compiler choice, gcc or vendorcc– 32 or 64bit memory model– Debugging information inclusion?– Type of threading, pthreads, other?

We will use two flavours:– gcc32dbg, gcc32dbgpthr

Allow multiple build types for a single installation– Libraries are tagged with the flavour name

• libXXX_gcc32dbg.so

Header files– Moved into labelled directories

• include/gcc32dbg• Include/gcc32dbgpthr

Building 64 bit Globus

It is generally not necessary to build the Globus gatekeeper, GridFTP or GRIS servers using 64 bit.

Some applications may need to build against 64 bit versions of the Globus libraries – so generally you should only need to build the SDK 64 bit.

Building 64 bit Globus can be difficult – generally problems arise when building the underlying OpenSSL libraries that Globus use.

Additional Packages Available

Job Manager Scheduler Support: This is to include support for different job managers beyond the default installed Fork. The currently supported job managers from Globus are:– Condor– LSF– PBS

Reporter Scheduler Support: This allows job manager information to be passed into the GRIS system, this is not done by default for any job manager including fork. All of the above are supported

GSI-SSH: There is a GSI compatible version of SSH and SCP available. This comes from NCSA though and so can have moments of incompatibility with the main Globus release.

Overview of Process of Building and Installing

Break Copy GPT and Globus bundles to your machine Define locations for GPT and Globus Create Globus destination directory Install GPT Install bundles Do post installation tasks Full details on the process can be found at

– http//www.globus.org/gt2.4/admin/

Installing GPT

Create directory /usr/local/globus2 Set environmental variable $GLOBUS_LOCATION to point

to /usr/local/globus2

export GLOBUS_LOCATION=/usr/local/globus2 (Set this environmental variable in your start up scripts.) Create directory /usr/local/globus2/src Download the required files into the directory

/usr/local/globus2/src from

http://www.globus.org/gt2.4/download.html

Required Files…

Get GPT 3.0.1 Under “Source Download” download

Resource Management (Client, Server and SDK)

Information Services (Client, Server and SDK)

Data Management (Client, Server and SDK) globus_gram_reporter-2.0.tar.gz globus_gram_reporter_setup_fork-1.0.tar.gz

Building GPT

Make directory /usr/local/GPT Set environmental variable GPT_LOCATION to

/usr/local/GPT

export GPT_LOCATION= /usr/local/GPT Change directory to /usr/local/globus2/src

cd /usr/local/globus2/src Unzip and untar the GPT bundle

tar zxvf gpt-XXX.tar.gz

cd gpt-XXX

./build_gpt

Building Globus GT2

Building Globus Toolkit 2.X

From with the /usr/local/globus2/src directory run the following command;

$GPT_LOCATION/sbin/gpt-build <bundle_name> <flavour>

eg. $GPT_LOCATION/sbin/gpt-build globus-resource-management-client-2.4.3-src_bundle.tar.gz gcc32dbg

Repeat this for all bundles; Nominally in the order:

– SDK– Server– Client

flavours

NOTE:– Some of the bundles use threading and some do not– Important to get this correct– These are the flavours as previously discussed

• See next table

Bundle Build Flavours

Bundle FlavourData Management bundles gcc32dbg

Information Services bundles gcc32dbgpthr

Resource Management bundles gcc32dbg

GRAM Job Manager Scheduler gcc32dbg

GRAM Reporter gcc32dbg

GSI-SSH gcc32dbg

Note – we have not downloaded the GSI-SSH bundle yet.

Post Installation Operations

Source the general user environment setup script:– source $GLOBUS_LOCATION/etc/globus-user-env.csh– . /usr/local/globus2/etc/globus-user-env.sh– (add to your start up scripts eg .bashrc)

Each package that makes up the Globus Toolkit requires some form of post installation setup:– $GPT_LOCATION/sbin/gpt-postinstall– This performs creation of some of the services setup scripts as

well as creating and installing the Globus configuration files in $GLOBUS_LOCATION/etc

Post Installation Operations

Now Globus Security Infrastructure (GSI) needs initialisation, creating /etc/grid-security directory and contents– $GLOBUS_LOCATION/setup/globus/setup-gsi

Finally verify the installation coherance– $GPT_LOCATION/sbin/gpt-verify

For details on these steps see:– http://www.globus.org/gt2.4/install.html

Break You should have GPT and Globus compiled. All the automatic configuration should be done.

Manual Installation Steps

The addition of globus-gatekeeper and gridftp configuration files for xinetd.d/inetd

Adding port definitions to /etc/services file Setup soft links for MDS server. Removing trust for Globus certificates and adding UK E-

Science CA trust Installation of host and LDAP certificates Add users Distinguishing Name definitions to /etc/grid-security/grid-mapfile

Adding globus-gatekeeper to /etc/xinetd.d

service gsigatekeeper { socket_type = stream protocol = tcp wait = no user = root env = LD_LIBRARY_PATH=GLOBUS_LOCATION/lib server = GLOBUS_LOCATION/sbin/globus-gatekeeper server_args = -conf GLOBUS_LOCATION/etc/globus-gatekeeper.conf disable = no }

NB – replace GLOBUS_LOCATION with the actual directory Globus has been installed in e.g. /usr/local/globus2

Adding grid-ftp to /etc/xinetd.d

service gsiftp{ instances = 1000 socket_type = stream wait = no user = root env = LD_LIBRARY_PATH=GLOBUS_LOCATION/lib server = GLOBUS_LOCATION/sbin/in.ftpd server_args = -l -a -G GLOBUS_LOCATION log_on_success += DURATION USERID log_on_failure += USERID nice = 10 disable = no}

NB – replace GLOBUS_LOCATION with the actual directory Globus has been installed in eg. /usr/local/globus2

Adding files to /etc/xinetd.d

You can either type the files out by hand – be careful as any typos will cause problems with the service that may not be immediately obvious.

Or you can cut ’n’ paste them from the Globus Web Site, they are located at:– http://www.globus.org/gt2.4/admin/guide-startup.html

Adding entries to /etc/inetd.conf file

RedHat does NOT use inetd.conf so we can skip this step. Add the following statements one per line to the file

/etc/inetd.conf globus-gatekeeper gsigatekeeper stream tcp nowait root

/usr/bin/env env LD_LIBRARY_PATH=GLOBUS_LOCATION/lib GLOBUS_LOCATION/sbin/globus-gatekeeper -conf GLOBUS_LOCATION/etc/globus-gatekeeper.conf

grid-ftp gsiftp stream tcp nowait root /usr/bin/env env LD_LIBRARY_PATH=GLOBUS_LOCATION/lib

GLOBUS_LOCATION/sbin/in.ftpd -l -a -G GLOBUS_LOCATION Warning – if the entry in inetd.conf gets to long you can have

problems.

To restart service:killall -HUP inetd

Adding entries to /etc/services

Entries must be added to /etc/services for the gatekeeper and the gridftp server because they are being started by xinetd (We do not have to add entries for MDS or GSI-SSH to /etc/services since they are not run by xinetd – but it can be worth doing)

Adding port definitions to /etc/services file – run the following commands:

echo "gsigatekeeper 2119/tcp #Globus Gatekeeper" >> /etc/servicesecho "gsiftp 2811/tcp #GSIFTP" >> /etc/services

The names of the services in /etc/services must match the names used in the files in /etc/xinetd.d.

Setting up MDS

Setup soft links for MDS server.

ln -s $GLOBUS_LOCATION/sbin/SXXgris /etc/rc.d/init.d/gris

ln -s /etc/rc.d/init.d/gris /etc/rc.d/rc3.d/SXXgris

ln -s /etc/rc.d/init.d/gris /etc/rc.d/rc5.d/SXXgris

You can replace XX in SXXgris with a number if you want but it is not necessary – in the rc.d directories the services are started in alphabetical order ie S90gris will start before S95gsisshd.

GSI-SSH

Issued by NCSA Separate to Globus, though dependant on GSI security Enables secure login based on digital certificates Defined within the L2G as operating on port 2222 Downloaded as a GPT source bundle. Installed in the same way as Globus bundles – does not

need threads so can be built with gcc32dbg flavour.

Installing GSI-SSH

Download source bundle from web– http://grid.ncsa.uiuc.edu/ssh/

Build using gpt-build$GPT_LOCATION/sbin/gpt-build gsi_openssh_bundle-X.X-

src.tar.gz gcc32dbg

Do post-install and gpt-verify

$GPT_LOCATION/sbin/gpt-postinstall

$GPT_LOCATION/sbin/gpt-verify

Configuring GSI-SSH

GSI-SSH should use port 2222 rather than 22 for the UK e-Science Grid.

Alter configuration as follows:– Set port numbers to 2222 in /etc/services

– Run the following command:

echo "gsisshd 2222/tcp #GSI-SSH" >> /etc/services

Set port number in ssh configuration files:

$GLOBUS_LOCATION/etc/ssh/ssh_config

$GLOBUS_LOCATION/etc/ssh/sshd_config

Create Symbolic Links for GSI-SSHD

ln -s $GLOBUS_LOCATION/sbin/SXXsshd /etc/rc.d/init.d/gsisshd

ln -s /etc/rc.d/init.d/gsisshd /etc/rc.d/rc3.d/SXXgsisshd

ln -s /etc/rc.d/init.d/gsisshd /etc/rc.d/rc5.d/SXXgsisshd

Break You should have Globus and GSI-SSHD installed and

configured (but not started). Next step is to add the Certificates that will be used for

authenication.

Certificates – Why and Wherefore

Three main uses for certificates within Globus– Gatekeeper/GridFTP certificate – the Gatekeeper and GridFTP daemons

use the same certificate.

– LDAP certificate (MDS)

– User certificates

These are acquired through the UK E-Science Grid Support Centre or another of the trusted CAs defined through EDG.

They may need format conversion depending on where they have come from

Installation of Certificates

Gatekeeper/GridFTP certificates will have to moved to:

/etc/grid-security– hostkey.pem and hostcert.pem

LDAP certificates will have to be moved to: – /etc/grid-security/ldap (you will have to create this

directory)– ldapkey.pem and ldapcert.pem

key files must be only owner readable while cert should be world readable

Gatekeeper and user commands will not work if incorrect permissions are present on key files

Certificating Authority Trust

CA Certificates are located in

/etc/grid-security/certificates Three files associated with each Certificate Authority (CA):

– xxxxxxxx.0 – PEM formatted CA certificate– xxxxxxxx.r0 – Certificate Revocation List URL– xxxxxxxx.signing_policy – defines which user certificates this CA

will authorise

Remove these files to remove CA trust and add copies of them to add trust

Adding the UK e-Science CA certificate

Normally the UK e-Science certificate can be downloaded from http://ca.grid-support.ac.uk/

We will get it from http://vermont.mvc.mcc.ac.uk/01621954.0 and http://vermont.mvc.mcc.ac.uk/01621954.signing_policy

Copy the two files into /etc/grid-security/certificates Optionally download the Certificate Revocation File from

http://ca.grid-support.ac.uk/ and store it as 01621954.r0 in /etc/grid-security/certificates. The CRL has a lifetime of about a month, if the file is not replaced within the month Globus will NOT accept any certificates issued by that CA. (remember to chmod 444 the file).

It is possible to set up a CRON job to update the CRL each night.

Removing the Globus CA

The Globus CA certificate is installed by default during the Globus installation – you should remove this CA as the quality of the certificates can be questionable.

To remove the Globus CA delete the following files from /etc/grid-security/certificates:– 42864e48.0– 42864e48.signing_policy

The Certificate “hash”

The “01621954” is the “hash” of the certificate subject name of the UK e-Science CA certificate – each certificate should have a different hash.

The hash is used in OpenSSL to form an index to allow certificates in a directory to be looked up by subject name.

To find the hash of a certificate use

openssl x509 –in 01621954.0 –noout -hash

Host and LDAP Certificates

On each of your machines are certificates named:– host.p12 (Used by gatekeeper,gsiftp,gsisshd)– ldap.p12 (Used by MDS)

You will need to use OpenSSL to change the format from p12 to PEM:

openssl pkcs12 -in host.p12 -clcerts -nokeys –out hostcert.pem

openssl pkcs12 -in host.p12 -nodes -nocerts -out hostkey.pem

openssl pkcs12 -in ldap.p12 -clcerts -nokeys –out ldapcert.pem

openssl pkcs12 -in ldap.p12 -nodes -nocerts -out ldapkey.pem

You will be prompted for a password each time you run the command.

Host and LDAP Certificates

Permissions must now be change for the generated files:– chmod 400 hostkey.pem– chmod 444 hostcert.pem– chmod 400 ldapkey.pem– chmod 444 ldapcert.pem

Move the files to their final locations– mv host*.pem /etc/grid-security/.– mv ldap*.pem /etc/grid-security/ldap/.

User Certificates

Normally to get a user certificate you would visit:

http://www.grid-support.ac.uk/ca/useful.htm For the purposes of the course we will use the “host”

certificate as user certificate. Create a test user – “useradd test” Set password for account – “passwd test” Create a directory /home/test/.globus (To become that user you can use the command “su test”)

Creating User Certificate As root create the user certificate from the host certificate:openssl pkcs12 -in host.p12 -clcerts -nokeys –out usercert.pem

openssl pkcs12 -in host.p12 -nocerts -out userkey.pem

When creating userkey.pem you will be prompted for a new password to protect the private key – you will need this password latter when creating a proxy certificate.

Change the permissions for the files:– chmod 400 userkey.pem

– chmod 444 usercert.pem

Move the certificates to the users directory and change the ownership of the files:– mv user*.pem /home/test/.globus

– chown –R test:test /home/test/.globus

Using the grid-mapfile

User access is controlled through the mapping of authenticated grid users to local users with the grid-mapfile.

Located in /etc/grid-security – this file will have to be created.

Specific format as shown below:“/o=grid/L=London/u=Joe Bloggs” JBloggs13

The first part is the “Distinguished Name” or DN of the user’s certificate, the second part is the user account they should be mapped to.

To find the DN of a user certificate use openssl x509 –in usercert.pem –noout -subject

grid-mapfile

“/C=UK/O=eScience/OU=Edingburgh/L=NeSC/CN=lab-01.nesc.ed.ac.uk/Email=support@nesc.ac.uk” test

Create a grid-mapfile with entries for everyone.

Find out your “Distinguished Name” using– openssl x509 –in usercert.pem –noout –subject

– Ignore the “subject= “ part

Add your entry to your grid-mapfile. Add an entry for everyone in your grid-mapfile (from your

DN you should be able to guess what everyone else’s DN is)

Break Installed Globus Toolkit version 2.4.X. Installed GSI-SSH Configured services for Globus gatekeeper and gridFTP Configured GSI-SSH Added UK E-Science CA Created Test Account & grid-mapfile Next we will start all the services and test them.

Starting Services

To restart xinetd– /etc/init.d/xinetd restart or use – service xinetd restart

To start MDS– /etc/init.d/gris start or use– service gris start

To start GSI-SSHD– /etc/init.d/gsisshd start or use– service gsisshd start

Testing the Gatekeeper

Log in as the “test” user.– As root you can use “su test” to become the test user

Run #>grid-proxy-init

– You should be prompted for a password– This will create a file in /tmp that contains your

proxy certificate.– You can run grid-proxy-info to check your proxy is

Run #>globus-job-run <machine_name> /bin/date Successful output will be the same as running /bin/date

from a shell

Command line options

For the globus-job-run command the following options are allowed:globus-job-run {[-:] <contact string> [-np N] <executable> [<arg>...]}..

Where the options are:

<contact string> the specific location where the job will run including possible service name if not default

-np The number of different processing elements the job will run on.

<executable> & <args> the job and its arguments that you wish to run on the grid.

The file modifiers -l and -s specify different filespaces:

-l[ocal] file is relative to working directory of job (DEFAULT)

-s[tage] file relative to job request is staged to job host

Full descriptions available from globus-job-run -help

Contact Strings

Used to specify the names of machines and available services and their locations:

Valid forms of the contact string are:– hostname– hostname:port– hostname:port/service– hostname/service– hostname:/service– hostname::subject– hostname:port:subject– hostname/service:subject– hostname:/service:subject– hostname:port/service:subject

MDS Verification

Make sure MDS is running: – service gris start

Run as a test user

#>grid-proxy-init Submit a test query

#>grid-info-search– This will search for information stored on the local

machine– If you have a problem try “grid-info-search –x” this

will do an un-authenticated search

Command line options

For the grid-info-search command the following options are allowed:grid-info-search [ options ] <search filter> [attributes]

Where the options are:

-mdshost host (-h) The host name on which the MDS server is running the default is hostname

-mdsport port (-p) The port number on which the MDS server is running. The default is port 2135

-mdsbasedn branch-point (-b) Location in DIT from which to start the search. The default is "Mds-Vo-name=local, o=Grid"

-anonymous (-x) Use anonymous binding instead of GSSAPI.

An example is to query the national information server:

grid-info-search –h ginfo.grid-support.ac.uk –p 2135 -b "Mds-Vo-name=UK e-Science, o=Grid“ -x

GridFTP testing

Run as a normal user

#>grid-proxy-init Create a file /tmp/file1 with some garbage in it. Run

#> globus-url-copy gsiftp://<hostname>/tmp/file1 gsiftp://<hostname>/tmp/file2

file2 should now be the same as file1 For more details on globus-url-copy use “globus-url-copy –

help”

Current Status of Your Machines

Each machine is now able to run jobs submitted through Globus.

MDS will use authenticated queries to show local system information

All services are now running and have been successfully tested

(tomorrow we will see how the L2G group has created a set of automated tests to automate those procedures)

Break Everyone should have Globus up and running. Next we will:

– Install advisories from Globus

– Logfile checking

Installation of Globus Advisories

Security and bug fix advisories are released by Globus as soon as they become essential

These are locate on the web at:

http://www-unix.globus.org/toolkit/advisories.html

Classified to be either a bug-fix, security patch or an enhancement.

Generally each time there is an OpenSSL security alert a Globus advisory will appear.

Before installation of an updated package use gpt-query to find out which flavours it was installed with (eg gcc32dbg – gcc32dbgpthr).

To install update packages use: – gpt-build –update <package_name> flavour

Additional specification for ‘noflavour’ is not needed

EXAMPLE – globus_openssl_module

[prompt]# $GPT_LOCATION/sbin/gpt-query globus_openssl_module4 packages were found in /usr/globus that matched your query:packages found that matched your query globus_openssl_module-gcc32dbg-dev ver: 0.2 cmp id: 0.2.0 globus_openssl_module-gcc32dbg-rtl ver: 0.2 cmp id: 0.2.0 globus_openssl_module-gcc32dbgpthr-dev ver: 0.2 cmp id: 0.2.0 globus_openssl_module-gcc32dbgpthr-rtl ver: 0.2 cmp id: 0.2.0

– Therefore to update this module you would run

gpt-build -update globus_openssl_module-0.2.tar.gz gcc32dbg gcc32dbgpthr

Globus Advisories

Add the advisories from the Globus website to your installation.

Remember to restart the services again:– “service gris stop” followed by “service gris start”

– service gsisshd restart

– service xinetd restart (not strictly needed)

Done so far….

So far you have:– Installed Globus 2.4.X on a bare system– Installed all relevant Advisories from Globus– Configured Globus services and run simple jobs across

our ‘Grid in a room’– Installed and used GSI-SSH for certificate based system

To Do….

Show further tasks for integration of the system into the L2G– Configure our GRIS to report to the UK GIIS through a local GIIS.

– Install and use the GITS script.

Only used simple fork jobmanager – need to know about more complicated job managers

ce Configuration and Setup Changes for connection to the L2G

– Common errors with Globus Setup– Install necessary extra packages from MDS reporting– Advanced Job manager setup– Advanced GridFTP setup– Configure MDS to run to a central server in the

classroom– Change from MDS reporting to machine here to actual

machine on the L2G

The Most Common Errors

The most common errors when using Globus have the following root causes and as such should be investigated first (The number in brackets are the number of associated Errors with this solution from the Globus Website page)– Services not correctly being started (1)– User proxies too old or not created (3)– Username and DN correctly added to grid-mapfile (5)– Date and time synchronisation (3)– User/host/service certificates validity problems due to CA

certificates not being loaded correctly into /etc/grid-security/certificates (3)

http://www.globus.org/about/faq/errors.html

It doesn’t work – what to do

Can you “globusrun –a –r hostname”. Can you telnet to the port the service is running on (port 2119 for the

gatekeeper and 2811 for the gridftp server). If you cannot telnet to the port – is there a firewall. If there is no firewall then xinetd may not be starting the gatekeeper

properly – check the system logs. If the gatekeeper is starting properly check the gatekeeper log for

errors –

$GLOBUS_LOCATION/var/globus-gatekeeper.log If the permissions on the hostcert and hostkey are wrong Globus will

NOT work. globus-url-copy with –dbg is better for debugging than globusrun

Firewalls

If your Globus installation is behind a firewall then certain ports need to be open – 2119 (gatekeeper), 2811 (gridftp) and 2135 (GRIS)

Globus will also need a range of ports open for such things as GASS – to inform Globus of the port range you need to set the environmental variable GLOBUS_TCP_PORT_RANGE. This should be set in the xinetd files and user start up scripts.

The size of the port range depends on how much use you expect of the services – generally a range of couple of thousand should be OK.

GLOBUS_HOSTNAME

Some systems have more than one hostname – if Globus picks the wrong hostname during the install it can cause problems.

Problems can also arise during the Globus authentication process – Globus compares the hostname in the host certificate with the hostname returned by the DNS server for the host’s IP address.

These problems can sometimes be solved by setting the environmental variable GLOBUS_HOSTNAME in the xinetd files and the user’s start up scripts. (This is often necessary on clusters where the head node may have more than one interface)

Adding GLOBUS_TCP_PORT_RANGE

service gsigatekeeper { socket_type = stream protocol = tcp wait = no user = root env = LD_LIBRARY_PATH=GLOBUS_LOCATION/lib env += GLOBUS_TCP_PORT_RANGE=65000,65256 env += GLOBUS_HOSTNAME=blah.foo.ac.uk server = GLOBUS_LOCATION/sbin/globus-gatekeeper server_args = -conf GLOBUS_LOCATION/etc/globus-gatekeeper.conf disable = no }

NB – the systems we are using for the course don’t have firewalls so we do not need to do this step.

Job Manager and Reporter Packages

These are available for download from:– http://www.globus.org/gt2.4/download.html– To install reporters the reporter package is also needed.– Reporters available for the following

• LSF• PBS• Condor • Fork

– These packages are all installed using the gcc32dbg flavour

Installing Job Manager and Reporter Packages

There are 2 steps to this: adding gram-reporter, this publishes jobmanager information into MDS and installing the jobmanager setup package.

Install the following; – globus-gram-reporter package

– jobmanager-specific setup package (in our case fork)

– Use gpt-build, gpt-postinstall and gpt-verify.

Edit $GLOBUS_LOCATION/etc/globus-job-manager.conf and add "-publish-jobs" and "-job-reporting-dir" options– “-publish-jobs” does not take any parameters, “-job-reporting-dir” can be set

to /tmp

Addition of Further Job Managers

Adding scheduler-setup-packages The first job manager scheduler package installed will be the

default job manager service (e.g. $GLOBUS_LOCATION/etc/grid-services/jobmanager).

Additional job manager scheduler packages installed will be installed using the convention "jobmanager-<scheduler-name>" (e.g. $GLOBUS_LOCATION/etc/grid-services/jobmanager-pbs).

Changing the Default Jobmanager

The default job manager when installed is fork. To change the default job manager:

setup-globus-job-manager-xxx -service-name jobmanager The Jobmanager consists of a Perl script that effectively

converts the RSL into the appropriate batch script file and submits it to the batch system. The JobManager scripts often need to be customised.

Located in:

$GLOBUS_LOCATION/lib/perl/Globus/GRAM/JobManager

Advanced GridFTP setup

Advanced GridFTP setup consists of only one change to allow third party transfers.

Involves editing $GLOBUS_LOCATION/etc/ftpaccess file to add the lines:

port-allow all 0.0.0.0/0

pasv-allow all 0.0.0.0/0

on both sides of the third-party connection. You should find this has already been done – sometimes

a system may already have an ftpaccess file in /etc.

Break Install the reporter and job manager specific setup

packages.– For this example download the Fork jobmanager package.

Perform GridFTP advanced configuration. Check the behaviour of the job manager under MDS, is the

jobmanager being reported to the MDS. (Remember you will have to restart the GRIS)

MDS Reporting

The Monitoring and Discovery Service uses the LDAP protocol for passing machine and service information around such as:– Processing load– Physical Resources (ie disks etc)– Software installations

GRIS – GIIS Configuration

Both ends of the GIIS-GRIS chain need to be aware of the other

Server GIIS

Machine 1 GRIS Machine 2 GRIS Machine 3 GRIS

Creating a Hierarchical MDS System

It is necessary within a unified Grid to have all systems reporting to a single server to provide a single place where all information on resources on the Grid can be found.

Each Organisation in the Virtual Organisation must register all its resources under the one name.

MDS Server

Within the UK E-Science community the main GIIS server is defined as:– Machine=ginfo.grid-support.ac.uk – Port=2135 – Mds-VO-name=UK e-Science

In order to register ourselves though we must use an intermediate VO-name.– Machine=lab-18.nesc.ed.ac.uk– Port=2135– Mds-Vo-name=nesc-demo

Configuration Files for MDS

All files located in $GLOBUS_LOCATION/etc– grid-info-resource-register.conf

• Lists the GIIS servers to which a GRIS will register directly. – grid-info-site-giis.conf

• Initializes the data structure for a GRIS registering to a GIIS. – grid-info-site-policy.conf

• Controls the acceptance of registration messages by a GIIS.– grid-info-slapd.conf

• Designates the GIIS and GRIS provider components to OpenLDAP.

– grid-info.conf • Sets the default values for the arguments to the grid-info-search

command

MDS Configuration Changes

The only changes to make the reporting to another MDs server are needed in– $GLOBUS_LOCATION/etc/grid-info-resource-register.conf

The changes are that you need to copy the existing block describing ‘local’ to the name of your own VO – see

http://vermont.mvc.mcc.ac.uk/grid-info-resource-register.conf For our example:

– reghn= lab-18.nesc.ed.ac.uk– Port=2135

– Mds-VO-name=nesc-demo

Now stop and restart you GRIS services and then test using: grid-info-search –h lab-18.nesc.ed.ac.uk –p 2135 –b “Mds-

VO-name=nesc-demo, o=Grid” –x

MDS Changes for GIIS

By default the GIIS will only accept registrations from itself. Need to change the file:

$GLOBUS_LOCATION/etc/grid-info-site-policy.conf

Adding a “MDS-Service-hn=*” entry. Also need to edit grid-info-slapd.conf and grid-info.conf,

replacing “site” with “nesc-demo” These steps are only required when creating a GIIS – you

do not need to do them. For examples of these files log onto lab-18.

Checking for you machine in the National GIIS

grid-info-search –h ginfo.grid-support.ac.uk –p 2135 –b “Mds-VO-name=UK e-Science, o=Grid” –x

Break Configured MDS to report to single server within the ‘grid in

a room’ setup.

Adjusted machine configurations to connect to L2G GIIS server.

Conclusions

Shown how to install the Globus Toolkit. Added Globus advisories to bring it up to date. Registered all the machines with the UK e-Science top

level GIIS. Shown how to configure extra job managers and how to

change default jobmanager.

Useful Information

http://www.globus.org http://www.grid-support.ac.uk http://esc.dl.ac.uk http://www.edg.org The Globus lists

ce The Grid Integration and Testing Script - GITS

The GITS script is written and maintained by David Baker (D.J.Baker@soton.ac.uk) from the University of Southampton.

It was written at the request of the EFT (Engineering Task Force) to provide a script that could be used to monitor the status of a Grid.

As part of the Level 2 Grid requirements the ETF requires each site to run and publish results from the GITS script nightly.

The script is available from:

http://www.soton.ac.uk/~djb1/gits.html

http://www.soton.ac.uk/~djb1/gits.1.4.tgz Current version is 1.4.1 The script is realised under a GPL license. David welcomes contributions to the script. It is probably one of the most comprehensive Grid

monitoring scripts available!!

The script is written in Perl, it requires one extra module Proc::Reliable

There are two GUI versions one written with Perl/Tk ( http://www.soton.ac.uk/~djb1/tkgits ), the other a C++/Qt executable which acts as a front end to the script ( http://vermont.mvc.mcc.ac.uk/qgits ).

The script can output its results in HTML format which can be put on a Web Server.

Running the Script

Before running the script some site specific parameters should be set within script:

$timeout

$local_giis

$local_vo_name

These can also be set using command line arguments.

Command Line Arguments

gits [-t abcdefghijklmn] [-h htmlfile] [-x XMLfile] contact1 [contact2 ...]

contact1 is of the format… vermont.mvc.mcc.ac.uk:fork

vermont.mvc.mcc.ac.uk:fork:lsf .

Tests -1

• a Ping Test - globusrun –a –r contact

(if this test fails no other tests are run)

b RSL-Hello - globusrun –o –r contact ‘&(executable=/bin/echo)(arguments=“Hello World”)’

c Hello World – globus-job-run contact /bin/echo “Hello World”

Tests - 2

d Stage – globus-job-run contact –s testscript

e RSL-Shell – globusrun –o –r contact ‘&(executable=$GLOBUS_LOCATION/bin/globus-sh-exec)(arguments= -e ${GLOBUS_SH_UPTIME})’

Tests - 3

f batch tests (Batch-Submit, Batch-Query and Batch-Cancel)

globus-job-submit contact /bin/sleep 600

globus-job-status

globus-job-clean –force g Batch-Retrieve

globus-job-submit contact /bin/echo hello

globus-job-get-output

Tests - 4

h GASS – globusrun –s –r contact

‘&(executable=$GLOBUS_LOCATION/bin/globus-url-copy) (arguments= $GLOBUS_GASS_URL/blah file:/tmp/blah) (environment= (LD_LIBRARY_PATH $GLOBUS_LOCATION/lib))’

Tests - 5

i GridFTP – globus-url-copy file:/tmp/blah gsiftp://contact/tmp/blah

j GRIS – grid-info-search –h contact –x

k UK GIIS – grid-info-search –nowrap ginfo.grid-support.ac.uk –x –b “UK e-Science,o=grid” –s sub “(objectclass=MdsComputer)”

Tests - 6

l Local GIIS - grid-info-search –nowrap $local_giis –x –b “$local_vo_name,o=grid” –s sub “(objectclass=MdsComputer)”

m Jobmanagers – checks for jobmanagers reported in UK GIIS

n Comparison – compares results from UK GIIS and Local GIIS

Script Output

STDOUT/STDERR XML format (using –x option). Required to upload

results to GITS Web Service HTML – should be put on a Web Server with a link

https://www.grid-support.ac.uk/etf/wg/integration-tests.html

GITS Web Service

http://vermont.mvc.mcc.ac.uk/gqec/ Written by Mark Mc Keown, University Manchester

(mark.mckeown@man.ac.uk)

Provides a mechanism for sites to store results from the GITS script in a central location.

Uses Web Services technology to store and retrieve results to a database.

Uses Apache/Perl/SOAP::Lite/MySQL on the server side.

GITS Web Service Clients

WSDL - http://vermont.mvc.mcc.ac.uk/gqec/GITSquery.WSDL

Perl client, gqec.pl, requires Perl modules SOAP::Lite installed on client machine.

http://vermont.mvc.mcc.ac.uk/gqec/

CGI interface: http://vermont.mvc.mcc.ac.uk/gqec/gqec_cgi.html

C++/Qt/gSOAP http://vermont.mvc.mcc.ac.uk/qgits/

Access Control

Requires a UK eScience certificate to upload results to the database, downloading results is open to everyone.

The gqec.pl can use a proxy certificate for authentication.

Perl Client

gqec.pl usage gqec.pl wsdl (returns WSDL of service) gqec.pl method parameters

gqec.pl QueryHostJob vermont.mvc.mcc.ac.uk fork

gqec.pl QueryHostJobDate vermont.mvc.mcc.ac.uk fork 2002-11-15

Uploading Results

gqec.pl UpLoader file1 file2

gqec.pl XMLtoHTML file1 file2 ....

gqec.pl QueryHostJob vermont.mvc.mcc.ac.uk fork | gqec.pl XMLtoHTML

Acknowledgements

Thanks to David Wallom (ex RAL now at Bristol) for slides. Thanks to Stephen Pickles (Manchester) for slides.

manchester computing supercomputing, visualization & e-science course instructors: mark mc keown...

Documents

financial management e10 by keown

birdville schools › cms › lib › tx01000797...item...

davis mc keown cap 1

selected creative 006: abe viljoen

manajemen keuangan keown fmch07

onderwêreld- fanie viljoen · onderwêreld- fanie viljoen...

viljoen marketing 2009

indigenous spiritual rica viljoen slideshare

piet viljoen on asset allocation

11th ed keown ch 1

chapter 1 myfinancelab solutions -...

redpaper - uom

l&f brando grant viljoen

02 resume - summarised - jo keown july 15

keown ch02

university of mysore [uom] yuvaraja’s college of uom

credit and inventory management keown

#virtualdesignmaster 3 challenge 3 - steven viljoen

patrick mc keown rhinosinusitis poster

ch01 akuntansi menengah keown