promoting and standardizing grid computing defining the grid: open grid services architecture...
TRANSCRIPT
Promoting and Standardizing Grid Computing
Defining the Grid: Open Grid Services Architecture
Current and Future Generation Grid Technology
Summer School 2006
24th - 28th July, Bonn (Germany)
Hiro Kishimoto, Ph.D.
OGF OGSA-WG, co-chair
Research Fellow, Fujitsu
Visiting Professor, National Institute of Informatics
OGSA is a trademark of the Open Grid Forum
July 24, 2006 Open Grid Forum - www.ogf.org 2
Agenda• Open Grid Services Architecture
Overview
• A Closer Look at OGSA
• OGSA and the Standards Landscape
• The OGSA Working Group
• Summary
July 24, 2006 Open Grid Forum - www.ogf.org 3
The Open Grid ServicesArchitecture
Open Grid Forum - www.ogf.org 4July 24, 2006
Three Generations of Grid
Source: Charlie Catlett
StandardizationStandardization is key for third-generation grids! is key for third-generation grids!StandardizationStandardization is key for third-generation grids! is key for third-generation grids!
• Local “metacomputers“– Distributed file systems– Site-wide single sign-on
• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept
• Local “metacomputers“– Distributed file systems– Site-wide single sign-on
• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept
• Utilize software services and communications protocols developed by grid projects:
– Condor, Globus, UNICORE, Legion, etc.
• Need significant customization to deliver complete solution• Interoperability is still very difficult!
• Utilize software services and communications protocols developed by grid projects:
– Condor, Globus, UNICORE, Legion, etc.
• Need significant customization to deliver complete solution• Interoperability is still very difficult!
• Common interface specifications support interoperability of discrete, independently developed services
• Competition and interoperability among applications, toolkits, and implementations of key services
• Common interface specifications support interoperability of discrete, independently developed services
• Competition and interoperability among applications, toolkits, and implementations of key services
We are here!
We are here!
Open Grid Forum - www.ogf.org 5July 24, 2006
The Open Grid Services Architecture
• An open, service-oriented architecture (SOA)− Resources as first-class entities− Dynamic service/resource creation and destruction
• Built on a Web services infrastructure
• Resource virtualization at the core
• Build grids from small number of standards-based components− Replaceable, coarse-grained− e.g. brokers
• Customizable− Support for dynamic, domain-specific content…− …within the same standardized framework
Open Grid Forum - www.ogf.org 6July 24, 2006
• Logical view of capabilities
• Relatively coarse-grained functions
• Reusable and composable behaviors
• Encapsulation of complex operations
• Naturally extendable framework
• Platform-neutral– machine and OS
Why Use an SOA?
Open Grid Forum - www.ogf.org 7July 24, 2006
SOA & Web Services: Key Benefits
SOA• Flexible
− Locate services on any server− Relocate as necessary− Prospective clients find services using
registries
• Scalable− Add & remove services as demand
varies
• Replaceable− Update implementations without
disruption to users
• Fault-tolerant− On failure, clients query registry for
alternate services
SOA• Flexible
− Locate services on any server− Relocate as necessary− Prospective clients find services using
registries
• Scalable− Add & remove services as demand
varies
• Replaceable− Update implementations without
disruption to users
• Fault-tolerant− On failure, clients query registry for
alternate services
Web Services• Interoperable
− Growing number of industry standards
• Strong industry support• Reduce time-to-value
− Harness robust development tools for Web services
− Decrease learning & implementation time
• Embrace and extend− Leverage effort in developing and
driving consensus on standards− Focus limited resources on
augmenting & adding standards as needed
Web Services• Interoperable
− Growing number of industry standards
• Strong industry support• Reduce time-to-value
− Harness robust development tools for Web services
− Decrease learning & implementation time
• Embrace and extend− Leverage effort in developing and
driving consensus on standards− Focus limited resources on
augmenting & adding standards as needed
Open Grid Forum - www.ogf.org 8July 24, 2006
Virtualizing Resources
Resources
Webservices
AccessAccess
StorageStorage SensorsSensors ApplicationsApplications InformationInformationComputersComputers
Resource-specific InterfacesResource-specific Interfaces
Common Interfaces
Type-specific interfaces
Open Grid Forum - www.ogf.org 9July 24, 2006
A Service-Oriented Grid
Virtualized resources
Grid middleware
services
Brokering Service
Brokering Service
Registry Service
Registry Service
DataService
DataService
CPU ResourceCPU ResourcePrinter Service
Printer Service
Job-Submit Service
Job-Submit Service
ComputeService
ComputeService
No
tify
Ad
vertise
ApplicationService
ApplicationService
July 24, 2006 Open Grid Forum - www.ogf.org 10
A Closer Look at OGSA
Open Grid Forum - www.ogf.org 11July 24, 2006
OGSA Capabilities
Security• Cross-organizational users• Trust nobody• Authorized access only
Security• Cross-organizational users• Trust nobody• Authorized access only
Information Services• Registry• Notification• Logging/auditing
Information Services• Registry• Notification• Logging/auditing
Execution Management• Job description & submission• Scheduling• Resource provisioning
Execution Management• Job description & submission• Scheduling• Resource provisioning
Data Services• Common access facilities• Efficient & reliable transport• Replication services
Data Services• Common access facilities• Efficient & reliable transport• Replication services
Self-Management• Self-configuration• Self-optimization• Self-healing
Self-Management• Self-configuration• Self-optimization• Self-healing
Resource Management• Discovery• Monitoring• Control
Resource Management• Discovery• Monitoring• Control
OGSAOGSA
OGSA “profiles”OGSA “profiles”
Web services foundation Web services foundation
Open Grid Forum - www.ogf.org 12July 24, 2006
CDLCDL
3. Select from or deployrequired resources
3. Select from or deployrequired resources
Execution Management
• The basic problem− Execute and manage jobs/services in the grid− Select from or provision required resources
• The basic problem− Execute and manage jobs/services in the grid− Select from or provision required resources
2. Submit the job2. Submit the job
1. Describe the job1. Describe the job
JSDLJSDL
Job
4. Manage the job4. Manage the job
Open Grid Forum - www.ogf.org 13July 24, 2006
Describing a Job Submission: JSDL• Job Submission Description Language (JSDL)
− A language for describing the requirements of jobs for submission− Declarative description
• A JSDL document describes the job requirements − Job identification information − Application (e.g., executable, arguments)− Required resources (e.g., CPUs, memory)− Input/output files
JobJob
IT InfrastructureIT Infrastructure
JSDLJSDL
Open Grid Forum - www.ogf.org 14July 24, 2006
Configuration & Deployment: CDL• Prepare the infrastructure so that the job can execute
− Provide a right-shaped slot to fit the job
• Main parts:− Configuration Description Language (CDL) provides declarative definition of
system configuration− Deployment service carries out configuration requests to deploy and
configure the system
IT InfrastructureIT Infrastructure
CDLCDL
Prepare
Open Grid Forum - www.ogf.org 15July 24, 2006
IssuesIssues
Find Describe
Access
DataDataDataData
Formats
ProtocolsProtocols
Use casesUse cases
DataDataDataData
DataData
DataDataMove/Copy/ReplicateMove/Copy/Replicate
Metadata Metadata
DataData
ManageManage
Common
accessCommon
access
Data Services
The basic problem
• Manage, transfer and access distributed data services and resources
The basic problem
• Manage, transfer and access distributed data services and resources
Derived dataCatalog
Sensor Data stream
Text file
Relational database
Open Grid Forum - www.ogf.org 16July 24, 2006
Basic Data Services
Data Resources
Data Resources
Managed StorageManaged Storage
Data Resources
Data Resources Transfer ProtocolsTransfer Protocols
Storage Management
Storage Management
Data Management
Data Management
Other Data Services
Other Data Services
TransferTransfer RegistriesRegistries
Other OGSA services and non-OGSA client APIsOther OGSA services and non-OGSA client APIs
Serviceinterface
Resourceinterface
Open Grid Forum - www.ogf.org 17July 24, 2006
Data Services
Data Service n
Data Service n
Data Service 1
Data Service 1
Data Service 2
Data Service 2
Composite Data Services
ReplicationReplication
CacheCache
FederationFederation
Open Grid Forum - www.ogf.org 18July 24, 2006
Basic Data Interfaces
• Storage Management− e.g. Storage Resource
Management (SRM)
• Storage Management− e.g. Storage Resource
Management (SRM)
• Data Access− ByteIO− Data Access & Integration
(DAI)
• Data Access− ByteIO− Data Access & Integration
(DAI)
• Data Transfer− Data Movement Interface
(DMI)− Protocols (e.g. GridFTP)
• Data Transfer− Data Movement Interface
(DMI)− Protocols (e.g. GridFTP)
• Replica management
• Metadata catalog
• Cache management
• Replica management
• Metadata catalog
• Cache management
Open Grid Forum - www.ogf.org 19July 24, 2006
Resource Management
• Provides a framework to integrate resource management functions− interfaces, services, information models, etc.
• Enables integrated discovery, monitoring, control, etc.
• Provides a framework to integrate resource management functions− interfaces, services, information models, etc.
• Enables integrated discovery, monitoring, control, etc.
High-levelmanagementservices(GGF)
Domain-specific capabilitiesDomain-specific capabilities
OGSA
Access tomanageability(OASIS, DMTF)
Informationmodels (DMTF,SNIA, etc.)
Resources
WSDM, WS-ManagementWSDM, WS-Management
WSRF/WSN, WS-Transfer/EventingWSRF/WSN, WS-Transfer/Eventing
Dataservices
Securityservices
ExecutionManagement
services
Application-specific
Open Grid Forum - www.ogf.org 20July 24, 2006
Self-Management
Self-
Management
Self-
Management
MonitoringMonitoring
ProjectionProjection
AnalysisAnalysisActionAction
PolicyPolicyPolicyPolicyPolicyPolicy
PolicyPolicyPolicyPolicySLASLA
• Self-configuration: Automatically adapt to changes in the environment:− e.g. Deploy/undeploy resources as load changes
• Self-configuration: Automatically adapt to changes in the environment:− e.g. Deploy/undeploy resources as load changes
• Self-optimization: Automatically tune system to best meet user or business needs− Uses service-level agreements (SLAs)
• Self-optimization: Automatically tune system to best meet user or business needs− Uses service-level agreements (SLAs)
• Self-healing: Automatically detect & correct problems− Component failures− Security violations− etc.
• Self-healing: Automatically detect & correct problems− Component failures− Security violations− etc.
Open Grid Forum - www.ogf.org 21July 24, 2006
Information Services
Executionmanagement
Resourcereservation
Problemdetermination
Accounting
Applicationmonitoring
Loadbalancing
Servicediscovery
Consumers ConsumersInformationServices
InformationServices
• Reliable
• Secure• Efficient
Provide management and access facilities for information about applications and resources in the grid environment
Provide management and access facilities for information about applications and resources in the grid environment
ProducersProducers
Asynchronous notification
Retrieval
RegistryRegistry
LoggerLogger
Open Grid Forum - www.ogf.org 22July 24, 2006
Security Services• Authorization, roles, and access privileges
− Locally (site) managed− Based on SAML and XACML security standards− Implementations provide credential mapping
• Working with GGF Security Area groups− Authorization attributes for grids
• Developing OGSA basic security profiles
PKIcertificate
PKIcertificate
WS-SecurityWS-Security
WS-AddressingWS-Addressing
OGSA
security profilesOGSA
security profiles
Open Grid Forum - www.ogf.org 23July 24, 2006
OGSA Profiles• The normative definition of OGSA
− Styled on WS-I profiles to promote interoperability− Define specific usage patterns
• e.g. execution management
− Basis for claims of conformance• “My scheduler conforms to the OGSA Execution Management Profile…”
• Include specifications developed by OGF and by other bodies− Issue: How mature and widely adopted?− OGSA Profile Definition document provides guidelines
In the futureEarly stagesIn the pipeline
Data ProfileData Profile
OGSA WSRF Basic Profile OGSA WSRF Basic Profile
OGSA Basic Security Profile – CoreOGSA Basic Security Profile – Core
OGSA Basic Security Profile – Secure ChannelOGSA Basic Security Profile – Secure Channel
Execution Mgmt ProfileExecution Mgmt Profile
Others…Others…
HPC ProfileHPC Profile
July 24, 2006 Open Grid Forum - www.ogf.org 24
OGSA and theStandards Landscape
Open Grid Forum - www.ogf.org 25July 24, 2006
Specifications Landscape: July 2006
SYSTEMSMANAGEMENT
UTILITYCOMPUTING
GRIDCOMPUTING
Core Services
Web ServicesFoundation WS-Addressing
Privacy
WS-BaseNotification
CIM/JSIM
WSRF-RAP
WSDM
WS-Security
Naming
OGSA-EMSOGSA Self Mgmt
GFD-C.16
GGF-UR
Data Model
HTTP(S)/SOAP
Discovery
SAML/XACML
WSDL
WSRF-RL
Trust
WS-DAI
VO Management
Information
Distributed query processing
ASP
Data CentreUse Cases &Applications Collaboration Multi MediaPersistent Archive
Data Transport
WSRF-RP
X.509
StandardEvolvingGapHole
Warning: Volatile data!Warning: Volatile data!
Open Grid Forum - www.ogf.org 26July 24, 2006
Related Standards Bodies
• OGF (Open Grid Forum)− Overall architecture for grid computing− OGSA, CDDLM, WS-Agreement, …
• OASIS (Organization for the Advancement of Structured Information Standards)− Middleware/Web services focused− WSRF, WS-Notification, WSDM, WS-Security…
• DMTF (Distributed Management Task Force)− Management and Information models (CIM)− Server management− WS-Management
− WS-CIM
• W3C (WWW Consortium)
− WS-Addressing
− WS-PolicyNot a complete list!Not a complete list!
Logos are trademarks or service marks of their respective owners.
Open Grid Forum - www.ogf.org 27July 24, 2006
OGF SCRM-WG
• Formed at GGF14
• Online reference guide of standards− Taxonomy− Glossary
• Live wiki document available
SDOs Collaboration on networked Resourced Management- WG
DMTF
July 24, 2006 Open Grid Forum - www.ogf.org 28
Making It Happen:The OGSA Working Group
Open Grid Forum - www.ogf.org 29July 24, 2006
The OGSA Working Group
• History− Established September 2002− Declared OGF’s “flagship architecture” March 2004
• Getting it done− Over 300 mailing-list subscribers− Twice-weekly teleconferences− Regular face-to-face meetings (at least 3+3 per year)− Collaboration with other working groups and standards organizations
• Contributors− Industry− Government− Academia
Open Grid Forum - www.ogf.org 30July 24, 2006
Aims & Perspective
• Goals− Interoperable solutions for grid-based applications− Addressing loosely coupled distributed computing
• Approach− Standardization at the architectural level
• Understand & describe the elements of grid systems and models• Develop architectural framework for standards in service-oriented
grids• Similar to profiling
− Use existing standards and technology where possible• Validate current standards for applicability in grids
− Use case-driven gap analysis – gaps filled proactively• Provide direction/motivation for new standards activity• Leverage & collaborate with other standards organizations
Open Grid Forum - www.ogf.org 31July 24, 2006
The OGSA Process
• Use case-driven− 21 detailed use cases
• “Design team” working model− Facilitate cross-fertilization within and outside OGF− Focus mind-share (the most valuable commodity!)
• e.g. DAIS and OGSA-Data working groups
− Design teams: Execution Management, Resource Management, Security
• OGSA fellow WGs− Formal independent OGF WG with “OGSA-” prefix− Work on some particular service interface
• Design teams may move up to OGSA fellow WGs
− Tightly collaborate and synchronize with OGSA-WG
• Iterative refinement− Abstract service evolving to concrete specifications
Open Grid Forum - www.ogf.org 32July 24, 2006
OGSA Document Structure
OGSARoadmap
OGSAArchitecture
OGSAUse Cases
Scenario
ServiceDescription
Candidate Profile
Profile
Actualspecs
consistent
inform andguide
inform andguide
feedback
refer
Proposedrecommendation
Informational
OGSA-WG documents
Documents producedby other OGF WGsor other SDOs
Base document
Informationmodels
Modelingguidelines
ProfileDefinition
Guidelines
Open Grid Forum - www.ogf.org 33July 24, 2006
OGSA and Fellow WGs
2002 2003 2004 2005 2006
OGSAdebut
WSRF TC
WSRF
WSRF 1.0
Arch 1.0
WSRF BP 1.0
OGSA-Naming WG
OGSA-BES WG
OGSA-ByteIO WG
OGSA-RSS WG
OGSA-WG
OGSA-AuthZ WG
Arch 1.5
OGSA-Data WG
OGSA-DMI WG
OGSA-HPCP WG
OGSI 1.0OGSI-WG
Roadmap1.0
Use Cases Roadmap1.1
Open Grid Forum - www.ogf.org 34July 24, 2006
Published OGSA Documents
20062005
Base document
Service description
Profile
Specification
2004
Architecture V1.0 (GFD-I.30) Glossary V1.0 (GFD-I.44) Resource Management
(GFD-I.45)
Architecture V1.0 (GFD-I.30) Glossary V1.0 (GFD-I.44) Resource Management
(GFD-I.45)
Use Cases (GFD-I.29) Use Cases (GFD-I.29)
Roadmap (GFD-I.53) Roadmap (GFD-I.53)
JSDL (GFD-R-P.56)
Profile Definition (GFD-I.59) Guideline
Open Grid Forum - www.ogf.org 35July 24, 2006
OGSA Schedule
2005 2006
EMS arch scenarios
HPC Profile
ByteIOBasic Execution Service
Architecture V1.5Architecture V1.5Glossary V1.5Glossary V1.5
Roadmap V1.1Roadmap V1.1Base document
Service description
Profile
Specification
Guideline Modeling guideline
WS-Naming
Public comment start GFD publication
Data architecture
WSRF Basic Profile
Basic Security Profiles
Container info-model
Open Grid Forum - www.ogf.org 36July 24, 2006
The OGSA Roadmap
• Defines OGSA as:− An architectural process− A set of specifications & profiles− OGSA software
• For each OGSA-related document:− Schedule− Dependencies
• Publication:− v1.0 published September ’05 (GFD.53)− Next version v1.1 in progress and plan to
publish Jan. 2007
Open Grid Forum - www.ogf.org 37July 24, 2006
OGSA-Based Open-Source Grids
Numerous grid projects are implementingOGSA components
Open Grid Forum - www.ogf.org 38July 24, 2006
Summary
• An open, service-oriented architecture
• Built on a Web services infrastructure
• Virtualize resources to hide complexity
• An open, service-oriented architecture
• Built on a Web services infrastructure
• Virtualize resources to hide complexity
• Interoperability is key!
• OGSA is defining a set of core services− Designed to work together…− …but with implementation flexibility
• Interoperability is key!
• OGSA is defining a set of core services− Designed to work together…− …but with implementation flexibility
• The current set of services is not the end of the road – just the beginning!
• The current set of services is not the end of the road – just the beginning!
Open Grid Forum - www.ogf.org 39July 24, 2006
More materials
• Architecture and Glossary v1.5− https://forge.gridforum.org/sf/go/doc13553?nav=1 − https://forge.gridforum.org/sf/go/doc13555?nav=1
• OGSA-WG project page− http://forge.ggf.org/projects/ogsa-wg
• OGSA-WG mailing list− Send email to "[email protected]" with the body text
"subscribe [email protected]".
Open Grid Forum - www.ogf.org 40July 24, 2006
Acknowledgement
With special thanks to Jem Treadwell, Andrew Grimshaw,
Jay Unger, Andreas Savva, David Snelling, Tom Maguire,
Dave Berry, Chris Jordan, Frank Siebenlist, Fred Maciel,
Michael Behrens, Allen Luniewski, Andrea Westerinen,
and other OGSA-WG members
Open Grid Forum - www.ogf.org 41July 24, 2006
OGF Full Copyright Notice
Copyright (C) Open Grid Forum (2005, 2006). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the OGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the OGF Document process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the OGF or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and THE OPEN GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."