ogf standards for cloud computing

OGF Standards: An Open Process Applied to Clouds, Grids and Advanced Distributed Computing

Alan Sill OGF President

1

NIST Cloud Computing Forum & Workshop VIII Gaithersburg, Maryland

July 7, 2015

© 2015 Open Grid Forum NIST Cloud Computing Workshop and Forum VIII July 7, 2015

About the Open Grid Forum:

Open Grid Forum (OGF) is a global organization operating in the areas of cloud, grid and related forms of advanced distributed computing. The OGF community pursues these topics through an open process for development, creation and promotion of relevant specifications and use cases. OGF actively engages partners and participants throughout the international arena through an open forum with open processes to champion architectural blueprints related to cloud and grid computing. The resulting specifications and standards enable pervasive adoption of advanced distributed computing techniques for business and research worldwide.

2


History and Background

• Since 2001 (as Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid Alliance --> formation of OGF in 2005.)

• Focused on grid, cloud and other forms of advanced distributed computing

• Works to promote cooperation, information exchange, and best practices in use and standardization.

• Cooperative agreements with other SDOs. • Developed computing, security, storage and network

standards used in science and business (BES, GridFTP, DRMAA, JSDL, RNS, GLUE, UR, etc.).

• Also develops cloud, networking and data standards (OCCI, DFDL, WS-Agreement, NSI/NML, etc.) in use in large-scale production systems world-wide.

3


• DRMAA: Distributed Resource Management Application API Grid Engine (Univa), Open Grid Scheduler: (open source); TORQUE and related products: Adaptive Computing; PBS Works: Altair Engineering; Gridway: DSA Research; HTCondor: U. of Wisconsin / Red Hat;

• OGSA® Basic Execution Service Version 1.0 and BES HPC Profile: BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering;

• JSDL: Job Submission Description Language (family of specifications): BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing; Windows HPC Server: Microsoft Corporation; PBS Works and PBS/Pro: Altair Engineering; Tivoli Workload Scheduler: IBM Corporation;

• WS-Agreement (family of specifications): ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator, LM-Architecture and Framework: (Multiple partners); BREIN SLA Management Framework: (Multiple partners); WSAG4J, Web Services Agreement for Java (framework implementation): Fraunhofer SCAI.

4

OGF HPC Standards In Use In Industry:

http://www.univa.com/products/grid-engine.php

http://sourceforge.net/projects/gridscheduler/

http://www.clusterresources.com/products/torque-resource-manager.php

http://www.pbsworks.com/

http://www.gridway.org/

http://www.cs.wisc.edu/condor/

http://bespp.sourceforge.net/

http://www.microsoft.com/hpc/en/us/default.aspx


http://bespp.sourceforge.net/

http://www.platform.com/workload-management/high-performance-computing

http://www.microsoft.com/hpc/en/us/default.aspx


http://www.apple.com

http://www-01.ibm.com/support/knowledgecenter/#!/SSGSPN_8.5.1.1/com.ibm.tivoli.itws.doc_8.5.1.1/awsbrmst30.htm%23dqx1twsvars

http://www.elasticlm.com/

https://gforge.beingrid.eu/gf/

http://www.eu-brein.com/

http://packcs-e0.scai.fraunhofer.de/wsag4j/


OGF Document Types

• Informational: To inform the community about a useful idea or set of ideas.

• Experimental: To inform the community about a useful experiment, testbed or implementation of idea or set of ideas.

• Community Practice: To inform the community of common practice or process, with the objective to influence the community and/or document its current practices.

• Recommendations: To publish a specification, analogous to an Internet Standards track document. Recommendations are initially designated as "proposed," and following further experience and review may become full recommendations.

• Further information including guidance and advice contained in GFD.152 at: http://ogf.org/documents/GFD.152.pdf

5

http://ogf.org/documents/GFD.152.pdf

© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015

Example: Worldwide LHC Computing Grid

13

~450,000 cpu cores ~430 Pb storage

Typical data transfer rate: ~12 GByte/sec

Total worldwide grid capacity: ~2x WLCG across all grids and

VOs

www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI*InSPIRE/RI*261323

EGI international presenceStorage Value+(yearly+increase)

Disk%(PB)%%235%PB%(+69%)

Tape%(PB) %%176%PB%(+32%)

Value+(yearly+increase)

CPU%cores 361,300%across%53%countries%(1.44%M%job/day)

Standards-based international collaboration

The Role of Standards for Risk Reduction and Inter-operation in XSEDE

XSEDE: The Next Generation of US National Supercomputing

Infrastructure

Cloud and grid standards now power some of the largest academic supercomputing infrastructures in the world!

Blacklight+Shared/Memory/4k/Xeon/cores

Darter/24k/cores+

Nautilus/Visualization/Data/Analytics+

Keeneland+/CPU/GPGPU

Stampede/460K&cores/w.&Xeon&Phi/>1000&users/Upgrade&in&2015

Wrangler/Data/Analytics

Trestles/IO*intensive/10k/cores/160/GB/SSD/Flash/

Gordon+++Data/intensive/64/TB/memory/300/TB/Flash/Mem

Open+Science+Grid/High/throughput/124/sitesBlue+Waters/

Leadership/

SuperMIC+380/nodes/–/1PF/(Ivy/bridge,/Xeon/Phi,/GPU)

Over/13/million/service/units/day/typically/delivered/as/of/2014/across/all/XSEDE/supercomputing/sites/(about/3/million/core/hours/day),/totaling/about/1.6/billion/core/hours/per/year

Yellowstone/Geosciences

US National Cyberinfrastructure

Promote an open, robust, collaborative, and innovative

ecosystem

Adopt, create and disseminat

e knowledge

Extend/ the impact of cyber-

infrastructurePrepare

the current and next

generation

Provide technical

expertise and support services

Collaborate with other CI groups and

projects

FutureGrid*/

Maverick/Visualization/Data/Analytics

Comet/“Long/Tail/Science”/47k/cores/2/PF/High/throughput

ACIDREF&Campus&sharing,&NSF&Cloud&(shared)

Grids

Credit: Irene Qualters, US

National Science Foundation

© 2015 Open Grid Forum OGF 44 - EGI Conference Lisbon, Portugal May 18-22, 2015

Example: Worldwide LHC Computing Grid

13

~450,000 cpu cores ~430 Pb storage

Typical data transfer rate: ~12 GByte/sec

Total worldwide grid capacity: ~2x WLCG across all grids and

VOs

www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI*InSPIRE/RI*261323

EGI international presenceStorage Value+(yearly+increase)

Disk%(PB)%%235%PB%(+69%)

Tape%(PB) %%176%PB%(+32%)

Value+(yearly+increase)

CPU%cores 361,300%across%53%countries%(1.44%M%job/day)

Standards-based international collaboration

The Role of Standards for Risk Reduction and Inter-operation in XSEDE

XSEDE: The Next Generation of US National Supercomputing

Infrastructure

Cloud and grid standards now power some of the largest academic supercomputing infrastructures in the world!

Blacklight+Shared/Memory/4k/Xeon/cores

Darter/24k/cores+

Nautilus/Visualization/Data/Analytics+

Keeneland+/CPU/GPGPU

Stampede/460K&cores/w.&Xeon&Phi/>1000&users/Upgrade&in&2015

Wrangler/Data/Analytics

Trestles/IO*intensive/10k/cores/160/GB/SSD/Flash/

Gordon+++Data/intensive/64/TB/memory/300/TB/Flash/Mem

Open+Science+Grid/High/throughput/124/sitesBlue+Waters/

Leadership/

SuperMIC+380/nodes/–/1PF/(Ivy/bridge,/Xeon/Phi,/GPU)

Over/13/million/service/units/day/typically/delivered/as/of/2014/across/all/XSEDE/supercomputing/sites/(about/3/million/core/hours/day),/totaling/about/1.6/billion/core/hours/per/year

Yellowstone/Geosciences

US National Cyberinfrastructure

Promote an open, robust, collaborative, and innovative

ecosystem

Adopt, create and disseminat

e knowledge

Extend/ the impact of cyber-

infrastructurePrepare

the current and next

generation

Provide technical

expertise and support services

Collaborate with other CI groups and

projects

FutureGrid*/

Maverick/Visualization/Data/Analytics

Comet/“Long/Tail/Science”/47k/cores/2/PF/High/throughput

ACIDREF&Campus&sharing,&NSF&Cloud&(shared)

Grids

Credit: Irene Qualters, US

National Science Foundation

Global Cyberinfrastructure

Millions of cores

Exabytes of storage

World-wide multi-100 Gb/s networks


OGF Published Documents

7

http://ogf.org/documents

217 published formal documents so far

http://ogf.org/documents

© 2015 Open Grid Forum

The Open Cloud Computing Interface

OCCI is a boundary-layer protocol and API suitable for any type of cloud control and operation at any level.

© 2015 Open Grid Forum

Design Philosophy

OCCI is a boundary-layer protocol and API suitable for any type of cloud control and operation at any level.

OCCI is not designed to be just A cloud standard; OCCI is designed to be prototypical of what the most general

design for a boundary-level cloud standard should be.

RESTful by design Extensible Portable Flexible

Interoperable

Designed to be LIKE the cloud to be useful for cloud purposes.

NIST Cloud Computing Workshop and Forum VIII July 7, 2015© 2015 Open Grid Forum

OCCI version 1.1

• Base documents originally published in 2011. • Initial documents include: Core (GFD.183),

Infrastructure (GFD.184), RESTful HTTP Rendering (GFD.185).

• Very stable standard set. • Implemented very extensively. • As per the established (IETF-like) OGF process,

these are Proposed Recommendations awaiting experience documentation and refinement for the group to advance them to full recommendations.

10

NIST Cloud Computing Workshop and Forum VIII July 7, 2015© 2015 Open Grid Forum

OCCI version 1.2

• Derived from experience in the field following OGF “Recommendations-track” process.

• Backwards compatible. Aim: Better completeness. • Changes & new documents in public comment as a

complete set. Publication: 3Q to 4Q 2015. • Updated/new proposed-rec. documents include:

Core, Infrastructure, HTTP Protocol, Compute Resource Templates Profile, JSON Rendering, Text Rendering, Platform, Service Level Agreements, and a new Monitoring informational document.

11


OCCI: Extensively Implemented(Dozens of repositories in GitHub)

12

Google search

Java

Python

Ruby

OpenStack

OCCI is very popular in the developer community. Implementations are in regular use on a production basis.

Github search

• Projects:

OCCI: Extensively Implemented!• Languages:

- Java - Ruby - Python - Javascript - Erlang - Perl

• Implementations: - rOCCI - OpenNebula - OCCIware - OpenStack (occi-os) - Cloudstack - CompatibleOne - ACCORDS - FogBow - EGI Federated Cloud - etc…

13

Source: occi-wg.orgDozens of independent projects.

http://occi-wg.org


Checklist for a Successful Standard:

• Uptake by developers • Well-designed, stable formulation • Open consensus • Community based • Available in many languages • Solves real-world problems • Adaptable, easy to implement • Well documented • Startup companies target its use • Low cost of adoption • Implementable at scale

15

Criterion✓Yes




15

Criterion✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes




15

Criterion✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes✓Yes

EGI Federated Cloud: A successful standards-based international federated cloud infrastructure

Credit: David Wallom, Oxford University and EGI Federated Cloud Task Force

Members •70 individuals •40 institutions •13 countries

Stakeholders •23 Resource Providers •10 Technology Providers •7 User Communities •4 Liaisons

Technologies •OpenStack •OpenNebula •StratusLab •CloudStack (in evaluation)

•Synnefo •WNoDeS

BSC

CNRS

LMU

OeRC

Masaryk

TUD

IFAE

Cyfronet

100%IT

CESNET

RADICAL

SRCE

DANTE

FZJ

GRNET

GWDG

STFC

SARA

KTH

INFNFCTSG

EGI.eu

Imperial

CESGACETA

IFCA

IGI

IPHC

IN2P3

SZTAKI

IISAS SixSq

Standards •OCCI (control) •OVF (images) •X.509 (authN) •CDMI (storage - under development)

(Updated July 2014)

www.egi.euEGI-‐InSPIRE RI-‐261323CloudWatch webinar: Standards ready for prime time

EGI Cloud Infrastructure

17

EGI Core Platform

Federated AAIService Registry

Monitoring Accounting

EGI Cloud Infrastructure Platform

Instance Mgmt

Information Discovery

Storage Management

Help and Support

Security Co-‐ordination

Training and Outreach

EGI

Collabo

ration

Tools

EGI A

pplication DB

Image Re

pository

EGI Cloud

Service M

arketplace

Sustainable Business Models

User Community

Cloud Management Stacks (OpenStack, OpenNebula, Synnefo, …)




17

EGI Core Platform




Instance Mgmt


Storage Management

Help and Support



EGI

Collabo

ration

Tools

EGI A

pplication DB

Image Re

pository

EGI Cloud

Service M

arketplace


User Community


GSIGLUE 2.0

CDMI 1.0

NAGIOS UR

OVF 2.0*

OCCI 1.1




17

EGI Core Platform




Instance Mgmt


Storage Management

Help and Support



EGI

Collabo

ration

Tools

EGI A

pplication DB

Image Re

pository

EGI Cloud

Service M

arketplace


User Community


GSIGLUE 2.0

Cloudinit CDMI 1.0

NAGIOS UR

OVF 2.0*

OCCI 1.1




17

EGI Core Platform




Instance Mgmt


Storage Management

Help and Support



EGI

Collabo

ration

Tools

EGI A

pplication DB

Image Re

pository

EGI Cloud

Service M

arketplace


User Community


GSIGLUE 2.0

CDMI 1.0

NAGIOS UR

OVF 2.0*

OCCI 1.2




17

EGI Core Platform




Instance Mgmt


Storage Management

Help and Support



EGI

Collabo

ration

Tools

EGI A

pplication DB

Image Re

pository

EGI Cloud

Service M

arketplace


User Community


GSIGLUE 2.0

CDMI 1.0

NAGIOS UR

OVF 2.0*

GLUE 2.1

OCCI 1.2




17

EGI Core Platform




Instance Mgmt


Storage Management

Help and Support



EGI

Collabo

ration

Tools

EGI A

pplication DB

Image Re

pository

EGI Cloud

Service M

arketplace


User Community


GSIGLUE 2.0

CDMI 1.0

NAGIOS UR

OVF 2.0*

GLUE 2.1

OCCI 1.2

EGI Cloud Profile



International PartnershipsAggregate Resources

– 12 NGIs provide 21 certified resources – 5 NGIs currently integrating resources – 5 NGIs with interested resource providers

– Worldwide interest & integration • Australia* (NeCTAR) • South Africa* (SAGrid) • South Korea* (KISTI) • United States* (NSF CAC Centres)

* Not shown on map

Usage since launch -‐ 244,913 (397,128) VMs certified (uncertified) -‐ 10.6M (12.3M) CPU hours (wall time) -‐ (As of June 1, 2015)

18Credit: David Wallom, Oxford University and EGI Federated Cloud Task Force


Public Comment process

19

http://redmine.ogf.org/projects/editor-pubcom/boards/

OCCI v1.2 document setis now in Public Comment

http://redmine.ogf.org/projects/editor-pubcom/boards/

© 2015 Open Grid Forum NIST Cloud Computing Workshop and Forum VIII July 7, 2015 20

OGF Cooperative Agreements In Place as of May 2015

OGF and IEEE: • OGF co-sponsors activities at many IEEE conferences;

planning engagements with P2301 & P2302 working groups. OGF and DMTF:

• Joint work register with DMTF on comparison of the OCCI and CIMI specifications has been in place for several years.

OGF and ISO/IEC: • OGF has a Category A liaison with ISO/IEC JTC1 SC38 on

Cloud Computing and is working on several joint activities. OGF and SNIA (CDMI):

• OGF has cooperative agreement w/ SNIA with respect to CDMI and has co-hosted 16 community Cloud Interoperability Plugfests so far (series continues at this conference!).


OGF and OCEAN: • OCEAN has joined the Cloud Plugfest series as a co-sponsor

and as one of the primary sponsors of the previous September 2013 OGF/EGI Cloud Interoperability Week.

OGF and OW2: • OW2 has co-sponsored Cloud Interoperability Plugfests as a

formal co-sponsor and active supporter of open source cloud projects, and supports the OCCIWare project.

OGF and XSEDE: • Extensive XSEDE adoption of OGF standards in production.

OGF and EGI: • Extensive community involvement, interaction and adoption of

OGF specs.



OGF and ETSI: • Cooperative MoU in place; contributing to ETSI CSC effort.

ETSI has joined the Cloud Plugfest series as a co-sponsor. OGF and ITU-T:

• OGF formal liaison in place with ITU-T JCA Cloud group. OGF and TM Forum:

• MoU in place; ongoing cross-SDO work on on End-to-End Management of Cloud Service Agreements, including SLAs.

OGF and CSA: • Cooperative agreement between OGF and CSA in place since

early 2012.



Continuing series…

Oriented towards REAL DEVELOPMENT AND

TESTING

Past eventsco-sponsored by many open source and standards-related

organizations including:OGF, CSCC, DMTF, SNIA,

OASIS, ETSI, OCEAN and OW2

Long-running developer-oriented

in-person standards and

software testing series

Easy to get involved and join

in events as developers or

project researchers.

19 Plugfests held in series so far

http://cloudplugfest.org

Cloud Plugfest Developer Series:

23

More events in planning pipeline.

http://cloudplugfest.org


Conclusions and Future:

• OGF documents support a variety of flexible architectures for advanced scientific, community and business uses.

• The Open Cloud Computing Interface (OCCI), OGF’s main entry into cloud standards, is experiencing good uptake and adoption in both research and production settings and is open for public comment at version 1.2 .

• OGF actively engages many partners and participants throughout the international arena through an open forum with open processes to promote best practices and standards in advanced distributed computing.

• OGF’s experience has enabled distributed computing built on these architectures to provide flexible, efficient and utility-like global infrastructures. Join us!

24

ogf standards for cloud computing

Technology

grid computing

open forum

open grid forum ogf

global grid forum

open grid scheduler

ogf community

open processes

ogf standards