grid technology: the rough guide

December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide

Grid Technology: The Rough Guide

Ashok Adiga (TACC) &Victor Bolet (Georgia State

University)


Outline• Introduction• Workshop Objectives• Grids 101• Grids in action• Workshop Outline• Open Discussions


Introduction• Workshop announcements

– Wireless access available– Dinner on Thursday– Ride back to hotel after Workshop on

Friday?• Introductions

– Speaker Introductions– Attendee introductions

• What would you like to get out of this workshop?


Workshop Objectives• Grid Technology is gaining acceptance

– Still evolving; but many success stories– New middleware/tools are becoming

available all the time• Target audience is technical people who

are interested in building grids but not sure how best to begin

• Objective is to provide an overview of current grid technology landscape via– Presentations & demonstrations– Hands-on session building “classroom grid”– Detailed coverage of selected topics,

overview of other topics with links to details


What is a Grid? “Resource sharing & coordinated

problem solving in dynamic … virtual organizations”

1. Enable integration of distributed service & resources2. Using general-purpose protocols & infrastructure3. To achieve useful qualities of service

“The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001


What is a Grid? (cont’d) 1. Integrate and coordinate resources not

subject to centralized control …– addresses the issues of security, policy,

payment, membership, etc.2. … using standard, open, general-purpose

protocols and interfaces …– for authentication, authorization, resource

discovery, and resource access.3. … to deliver nontrivial qualities of service.

– relating to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands,


Grid Applications

Grid Technologies for Resource Integration &

Management

Grid Resources

inte

grat

ioninteroperability

DB Access

PDB portal

App Scheduler

PSE

portalUser-level Middleware and Tools

System-level Common Infrastructure


Grid Applications

Grid Services Architecture• Agree on interfaces, services

– Common infrastructure services act like a “grid OS”– Users interact with the Grid through higher-level, user-

friendly middleware layer

User-focusedmiddleware & tools

(commercial opportunities)

Grid Resources

DB Federation

PDB portal

App Scheduler

PSE

Chem portal

Common infrastructure

services(many open source)

Authentication, information, resource

access, resource mgmt, negotiation,

scheduling, monitoring, data transfer, etc., etc.


Available Grid Technologies• Several Grid solutions & toolkits exist

– Globus Toolkit (www.globus.org)• A toolkit of grid services and tools jointly developed by the Globus

Alliance in the US.– Gridlab (www.gridlab.org)

• A set of application-oriented Grid services and toolkits jointly developed by academic & commercial companies primarily based in Europe

– Gridbus (www.gridbus)• Grid computing and Business technologies developed at the University

of Melbourne.– Unicore (www.unicore.org)

• Grid solution created by the Unicore Forum, a developed by leading European Computing Centers and supporting hardware vendors

– Open Middleware Infrastructure Institute (OMII) (www.omii.ac.uk)• UK e-Science Grid solution

– Condor (www.cs.wisc.edu/condor)• Complete grid solution developed at the University of Wisconsin

– Grid MP (www.ud.com)• Commercial grid solution developed by United Devices

• We will primarily discuss the use of the Globus Toolkit & Condor


What Is the Globus Toolkit?• A Grid development environment

– Develop new OGSA-compliant Web Services– Develop applications using Java or C/C++ Grid APIs– Secure applications using basic security mechanisms

• A set of basic Grid services– Job submission/management– File transfer (individual, queued)– Database access– Data management (replication, metadata)– Monitoring/Indexing system information

• Tools and Examples• The prerequisite for many Grid community tools


“Standard Plumbing” for the Grid

• Not turnkey solutions, but building blocks and tools for application developers and system integrators.– Some components (e.g., file transfer) go

farther than others (e.g., remote job submission) toward end-user relevance.

• Since these solutions exist and others are already using them (and they’re free), it’s easier to reuse than to reinvent.– And compatibility with other Grid systems

comes for free!


WSAuthenticationAuthorization

Pre-WSAuthenticationAuthorization

Data MgmtSecurity CommonRuntime

Execution Mgmt

Information Services

ReliableFile

Transfer(RFT)

GridFTP

OGSA-DAI[Tech Preview]

Python WS Core[contribution]

C WS Core

CommunitySchedulerFramework

[contribution]

GridResource

Allocation Mgmt(WS GRAM)

Monitoring& Discovery

System(MDS4)

Java WS Core

Web ServicesComponents

Non-WS Components

CredentialManagement

ReplicaLocationService(RLS)

GridResource

Allocation Mgmt(Pre-WS GRAM)

Monitoring& Discovery

System(MDS2)

C CommonLibraries

XIO

DelegationService

CAS

GT2

GT3

GT3GT4

GT4

Globus Toolkit Components


Areas of Competence• “Connectivity Layer” Solutions

– Service Management (WSRF)– Monitoring/Discovery (WSRF and MDS)– Security (GSI and WS-Security)– Communication (XIO)

• “Resource Layer” Solutions– Computing / Processing Power (GRAM)– Data Access/Movement (GridFTP, OGSA-DAI)– In development: Telecontrol (NTCP/GTCP)

• “Collective Layer” Solutions– Data Management (RLS, MCS, OGSA-DAI)– Monitoring/Discovery (MDS)– Security (CAS)


Evolution of the GridIn

crea

sed

func

tiona

lity,

stan

dard

izat

ion

Time

Globus Toolkit

Open GridServices Arch

GGF: OGSI, WSRF, …(leveraging OASIS, W3C, IETF)

Multiple implementations,including Globus Toolkit

Defacto standardsGGF: GridFTP, GSI(leveraging IETF)

Customsolutions

X.509,LDAP,FTP, …

Web services

App-specificServices


Standards: Open Grid Services Architecture

• Define a service-oriented architecture…– the key to effective virtualization

• …to address vital Grid requirements– AKA utility, on-demand, system

management, collaborative computing, etc.• …building on Web service standards.

– extending those standards when needed


Standards Compliance• Web services: WS-I compliance

– All interfaces support WS-I Basic Profile, modulo use of WS-Addressing

• Securitya) WS-I Basic Security Profile (plaintext)b) IETF RFC 3820 Proxy Certificate

• GridFTP– GGF GFD 020

• Others in progress & being tracked– WSRF (OASIS), WS-Addressing (W3C), OGSA-DAI

(GGF), RLS (GGF)


Grid and Web Services Convergence

The definition of WSRF means that the Grid and Web services communities can move forward on a common base.


Grid Planning Considerations• All Grid technology is evolving rapidly.

– Web services standards– Grid interfaces– Grid implementations– Grid hosting services (ASP, SSP, etc.)

• Community is important!– Best practices (GGF, OASIS, etc.)– Open source (Linux, Axis, Globus, etc.)

• Applying community standards is vital.– Increases leverage– Mitigates (a bit) effects of rapid evolution– Paves the way for future integration/partnership


What End Users NeedSecure, reliable, on-demand access to data,software, people, and other resources(ideally all via a Web Browser!)


How it Really Happens

WebBrowser

ComputeServer

DataCatalog

DataViewer

Tool

Certificateauthority

ChatTool

CredentialRepository

WebPortal

ComputeServer

Resources implement standard access & management interfaces

Collective services aggregate &/or

virtualize resources

Users work with client applications

Application services organize VOs & enable

access to other services

Databaseservice

Databaseservice

Databaseservice

SimulationTool

Camera

CameraTelepresence

Monitor

RegistrationService


How it Really Happens• Implementations are provided by a mix

of– Application-specific code– “Off the shelf” tools and services– Tools and services from the Globus Toolkit– Tools and services from the Grid community

(compatible with GT)• Glued together by…

– Application development– System integration


How it Really Happens (without the Grid)

WebBrowser

ComputeServer

DataCatalog

DataViewer

Tool

Certificateauthority

ChatTool

CredentialRepository

WebPortal

ComputeServer







Databaseservice

Databaseservice

Databaseservice

SimulationTool

Camera

CameraTelepresence

Monitor

RegistrationService

A

B

C

D

E

Application Developer

10

Off the Shelf 12

Globus Toolkit 0

Grid Community

0


How it Really Happens (with the Grid)

WebBrowser

ComputeServer

GlobusMCS/RLS

DataViewer

Tool

CertificateAuthority

CHEF ChatTeamlet

MyProxy

CHEF

ComputeServer







Databaseservice

Databaseservice

Databaseservice

SimulationTool

Camera

CameraTelepresence

Monitor

Globus IndexService

GlobusGRAM

GlobusGRAM

GlobusDAI

GlobusDAI

GlobusDAI

Application Developer

2

Off the Shelf 9

Globus Toolkit 4

Grid Community

4


Examples of Grids in Action• Campus level grid projects

– UT Grid (University of Texas)– MGrid (University of Michigan)– …

• Regional initiatives– SURAgrid

• National/International Grids– NSF Teragrid– OSG (Open Sciences Grid)– EGEE (Enabling Grids for E-Science in Europe)

• Volunteer grids– SETI@home– World Community Grid– Grid.org


NSF TeraGrid• 40+ teraflops

compute • 1+ petabyte

online storage• 10-40Gb/s

networking

• Heterogeneous compute, storage, visualization resources• Globus-based grid middleware• Any US researcher can apply for allocations• http://www.teragrid.org for more information

http://www.teragrid.org/


The TeraGrid Strategy• Creating a unified user

environment…– Single user support

resources.– Single authentication point– Common software

functionality– Common job management

infrastructure– Globally-accessible data

storage• …across heterogeneous

resources– 7+ computing architectures– 5+ visualization resources– diverse storage technologies

• Create a unified national HPC infrastructure that is both heterogeneous and extensible


Current TeraGrid Usage Scenarios

• “Traditional” massively parallel jobs– Tightly-coupled interprocessor communication– storing vast amounts of data remotely– remote visualization

• Thousands of independent jobs– Automatically scheduled amongst many

TeraGrid machines– Use data from a distributed data collection

• Multi-site parallel jobs– Compute upon many TeraGrid sites

simultaneouslyTeraGrid is working to enable more!


SURAgrid• A “beyond regional” initiative in support of

SURA regional strategy“Mini-About” SURA– SURA region: 16 states & DC, from Delaware to Texas– SURA membership: 62 research universities, mostly

within the region– SURA mission: Foster excellence in scientific research,

strengthen capabilities, provide training opportunities• Evolved from the NMI Testbed Grid

project, an outgrowth of SURA’s management of the NMI Integration Testbed Program– http://www1.sura.org/3000/NMI-Testbed.html


SURAgrid GoalsSURAgrid: Organizations collaborating to bring

grids to the level of seamless, shared infrastructure

Goals:To develop scalable infrastructure that

leverages local institutional identity and authorization while managing access to shared resources

To promote the use of this infrastructure for the broad research and education community

To provide a forum for participants to gain additional experience with grid technology, and participate in collaborative project development


SURAgrid Participants• University of Alabama at

Birmingham*• University of Alabama in

Huntsville*• University of Arkansas*• University of Florida*• George Mason University*• Georgia State University* • Great Plains Network• University of Kentucky*• University of Louisiana at

Lafayette*• Louisiana State University*• University of Michigan• Mississippi Center for

SuperComputing Research*

• University of North Carolina, Charlotte

• North Carolina State University*

• Old Dominion University*• University of South Carolina*• University of Southern

California• Southeastern Universities

Research Association (SURA)• Texas A&M University*• Texas Advanced Computing

Center (TACC)*• Texas Tech• Tulane University* • Vanderbilt University* • University of Virginia*

*SURA member


SURAgrid Resources


SURAgrid Applications• Multiple Genome Alignment (GSU, UAB, UVA) • Task Farming (LSU)• Muon Detector Grid (GSU)• BLAST (UAB)• ENDYNE (TTU)• SCOOP/ADCIRC (UNC, RENCI, MCNC,

SCOOP partners, SURAgrid partners)• … Potential applications…


Workshop Agenda• Introduction & Overview• Configuring Resources for the Grid• Authentication, Authorization, &

Identity Issues in Grids• Level Grid Services• High Level Grid Services• Grid Packages• User Interfaces• Grid Application Toolkits• Hands-on Session


Hands-on Session• Build Classroom Grid during workshop

– Using some of the technologies described in the presentations

– Grid services pre-installed or installed during sessions

– Laptop “servers” installed & configured during hands-on session

• Try out some basic grid operations on this grid– Security, data & job management, resource

monitoring, grid portals


Questions?

grid technology: the rough guide

Documents