© 2012 ibm corporation openstack ce technology review & demo egan ford ibm distinguished...

© 2012 IBM Corporation

OpenStack CETechnology Review & Demo

Egan FordIBM Distinguished [email protected]

© 2012 IBM Corporation2

PPT’s and Videos: http://xmission.com/~egan/cloud/


Agenda

• IBM SmartCloud and OpenStack• Cloud Taxonomy • Some OpenStack Public Use Cases• What is OpenStack• OpenStack Resources• IBM Resources/Solutions for OpenStack Available Today• OpenStack (Video) Demo


Evolve existing infrastructure

to Cloud

Accelerate adoption with

expert integrated systems

Immediate access to a managed

platform

Common Open Standards-based Cloud Management Services

Open architectures enable real innovation through interoperability


IBM embraces & invests in open source to foster innovation

CloudComputing

ApplicationServers

ServiceOrientation

Service OrientedArchitecture

Systems ofInteraction

SocialBusiness

Open CloudArchitecture

June 1998: IBM enters into an engineering agreement with The Apache Group for development of the open-source Apache HTTP server software eventually becoming the leader of the new Application Server market

June 1998: IBM enters into an engineering agreement with The Apache Group for development of the open-source Apache HTTP server software eventually becoming the leader of the new Application Server market

September 1999: IBM capitalizes on an untapped market trend and begins participating in the community development of Linux with a $60M annual investment

September 1999: IBM capitalizes on an untapped market trend and begins participating in the community development of Linux with a $60M annual investment

November 2001: IBM rallies 150 influential vendors and the development community around a new tools environment with a $40 Million software donation disrupting the leadership of the software development ecosystem

November 2001: IBM rallies 150 influential vendors and the development community around a new tools environment with a $40 Million software donation disrupting the leadership of the software development ecosystem

September 2012: IBM orchestrates the launch of The OpenStack Foundation boasting $10 million in funding and 5,600 members changing the dynamics of the Cloud ecosystem

September 2012: IBM orchestrates the launch of The OpenStack Foundation boasting $10 million in funding and 5,600 members changing the dynamics of the Cloud ecosystem


An open cloud architecture is emerging…built on open technologies

Hardware

OAuthOAuth

TOSCA

OSLC

cloudfoundry.org


OpenStack is a global collaboration of developers & cloud computing technologists working to produce an ubiquitous Infrastructure as a Service (IaaS) open source cloud computing platform for public & private clouds.

The OpenStack Foundation – IBM is a driving force in it’s success

Platinum SponsorsPlatinum Sponsors APR2012

Sep2013150

Contributors2600 Individuals

1021 Contributors11,800+ Individuals

Exponential growth in 1+ YRGold SponsorsGold Sponsors

Software Defined

Environment


IBM is #2 in contributionsCloud Computing

Reference Architecture (CCRA)

Cloud StandardsCustomer Council

(CSCC)

LinkedData, OSLC

TOSCA

IBM’s ecosystem approach to an Open Cloud Architecture

OpenStackOpen Source

Reference

IBM is working to accelerate OpenStack Foundation success

Because an open interoperable Cloud is critical for flexible cloud deployment and customer success…

IBM has 12 core contributors12

IBMers working on OpenStack – from formation of the Foundation to Code Quality & New Function370+

IBMers have signed the CLA

2

90

© 2011 IBM CorporationIBM CONFIDENTIAL

IBM contributions to OpenStack are wide-ranging, contributing to OpenStack success and delivering real value

OpenStack ComputeProvision and manage large networks of virtual machines•Platform integration (CCS)•High Availability enhancements (RES)•Resource optimization (RES)•Live upgrade contributions (LTC)•Enablement for P & Z Systems, DB2 (CCS, SOS)•ESXi support (SOS w/ VMWare)•VM group enablement in scheduler (RES)•CPU allocation for vCPUs (RES)

OpenStack Object StoreCreate petabytes of secure, reliable storage using standard HW•Block & object storage enablement for IBM capability (RES)•Nova blueprints (LTC)•Cinder local storage & local instance clone (CCS)•Efficient clone image in Cinder SVC driver for cFlex (RES)•Nova & Cinder storage blueprints (CCS, SOS)•Storwise/SVC driver update – support iSCSI CHAP auth (SOS, RES)•Wsgi application interface enabling external web server (RES)•Swift / Keystone interface for Keystone v3 API (RES)

OpenStack NetworkingCreate petabytes of secure, reliable storage using standard HW•Support for key emerging networking standards (RES)•Quantum blueprints & migration from Nova (LTC)•FibreChannel support (RES)

OpenStack DashboardEnables administrators and users to access & provision cloud-based resources through a self-service portal.•Globalization and crowd-sourced translation integration (SOS)•Cross hypervisor testing and validation (CCS)

OpenStack Shared ServicesLibraries that provide image management, authentication & security across all OpenStack projects•Security & authentication enhancements (CCS, SOS)•Image activation for OVF (CCS)•Guest level metric collection (CCS)•APIs: Enablement for key emerging standards (SOS)•Membership services enhancements (CCS, RES)•Glance: multiple image locations (CCS)

General OpenStack contributions•Drive IBM value-add capability from SCP (CCS)•Community facing contributions – bug fixing, community building & promotion (LTC, SOS)•QA items (LTC)

CCS – Common Cloud Stack (STG & SWG)LTC – Linux Technology Center (STG)SOS – Standards & Open Source (SWG)RES – Research

Par

tici

pat

ion

K

ey

© 2011 IBM CorporationIBM CONFIDENTIAL

IBM SmartCloud solutions are moving to an OpenStack-based infrastructure layer, enabling smooth migration and upgrade

Related Standards & OrganizationsRelated Standards & Organizations

SmartCloud OrchestratorOrchestrate Services across multiple environments and domains

SmartCloud OrchestratorOrchestrate Services across multiple environments and domains

CIMI &OVFCIMI &OVF

TOSCATOSCA

CCRACCRA OSLCOSLC

Simple 3 tier structure, with increased Client Value at each tier

Using open, common, standards based architecture providing choice, flexibility, interoperability, portability

Clean upgrade paths with progression to fully integrated and factory optimized PureApplication System

Significant customer benefits above and beyond base OpenStack

SmartCloud ProvisioningAutomate Optimized

Workloads


Workloads

SmartCloud EntryAutomate IT Delivery



Workloads


Workloads



Customer integrated hardware

Customer integrated hardware

PureFlex SystemPureFlex System

PureApplicationSystem

PureApplicationSystem

Automate Optimized WorkloadsAutomate Optimized Workloads

Ke

y CommonCloud Stack

FactoryIntegrated

BundleOption

PureApp Server


Cloud Taxonomy

Source: http://it20.info/2012/02/the-cloud-magic-rectangle-tm/


Cloud Value Proposition and Positioning



How You (Provider) Build These Clouds



What You (Consumer) Get with These Clouds:



Policy-based Clouds and Design-for-fail Clouds are purpose optimized Infrastructure Management solutions

Policy-based Clouds

• Purpose optimized for longer-lived virtual machines managed by Server Administrator

• Centralizes enterprise server virtualization administration tasks

• High degree of flexibility designed to accommodate virtualization all workloads

• Significant focus on managing availability and QoS for long-lived workloads with level of isolation

• Characteristics derived from exploiting enterprise class hardware

• Legacy applications

Design-for-fail Clouds

• Purpose optimized for shorter-term virtual machines managed via end-user or automated process

• Decentralized control, embraces eventual consistency, focus on making “good enough” decisions

• High degree of standardization• Significant focus on ensuring availability of

control plane• Characteristics driven by software• New applications


Some OpenStack Public Use Cases

• Internap• http

://www.internap.com/press-release/internap-announces-world%E2%80%99s-first-commercially-available-openstack-cloud-compute-service/

• Rackspace Cloud Servers, Powered by OpenStack• http://www.rackspace.com/blog/rackspace-cloud-servers-powered-by-openstack-beta/

• Deutsche Telekom• http://www.telekom.com/media/media-kits/104982

• AT&T• http://arstechnica.com/business/news/2012/01/att-joins-openstack-as-it-launches-cloud-fo

r-developers.ars

• MercadoLibre• http://openstack.org/user-stories/mercadolibre-inc/mercadolibre

-s-bid-for-cloud-automation/• NeCTAR

• http://nectar.org.au/• San Diego Supercomputing Center

• http://openstack.org/user-stories/sdsc/


OpenStack design tenets focus on delivering essential infrastructure on an available, scalable, elastic control plane

Sources:http://www.openstack.org/downloads/openstack-compute-datasheet.pdfhttp://wiki.openstack.org/BasicDesignTenets

Basic Design Tenets

1) Scalability and elasticity are our main goals

2) Any feature that limits our main goals must be optional

3) Everything should be asynchronous. If you can't do something asynchronously, see #2

4) All required components must be horizontally scalable

5) Always use shared nothing architecture (SN) or sharding. If you can't Share nothing/shard, see #2

6) Distribute everything. Especially logic. Move logic to where state naturally exists.

7) Accept eventual consistency and use it where it is appropriate.

8) Test everything. We require tests with submitted code. (We will help you if you need it)

OpenStack Leadership's vision statement

“essential Infrastructure, support platform”


OpenStack

Source: http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/


OpenStack is comprised of seven core projects that form a complete IaaS solution

Compute (Nova)

Storage (Cinder)

Network (Quantum)

Provision and manage virtual resources

Dashboard (Horizon)Self-service portal

Image (Glance)Catalog and manage server images

Identity (Keystone)Unified authentication, integrates with existing systems

Object Storage (Swift)petabytes of secure, reliable object storage

IaaS

Source: http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/

IaaS


Compute delivers a fully featured, redundant, and scalable cloud computing platform

Architecture

Sources:http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/http://openstack.org/projects/compute/

Key Capabilities:

•Manage virtualized server resources• CPU/Memory/Disk/Network Interfaces

•API with rate limiting and authentication

•Distributed and asynchronous architecture• Massively scalable and highly available system

•Live guest migration• Move running guests between physical hosts

•Live VM management (Instance)• Run, reboot, suspend, resize, terminate instances

•Security Groups

•Role Based Access Control (RBAC)• Ensure security by user, role and project

•Projects & Quotas

•VNC Proxy through web browser


Compute management stack control plane is built on queue and database

Key Capabilities:

• Responsible for providing communications hub and managing data persistence

• RabbitMQ is default queue, MySQL DB• Documented HA methods• ZeroMQ implementation available to decentralize

queue

• Single “cell” (1 Queue, 1 Database) typically scales from 500 – 1000 physical machines

• Cells can be rolled up to support larger deployments

• Communications route through queue• API requests are validated and placed on queue• Workers listen to queues based on role or role +

hostname• Responses are dispatched back through queue


nova-compute manages individual hypervisors and compute nodes

Key Capabilities:

• Responsible for managing all interactions with individual endpoints providing compute resource, e.g.

-- Attach iSCSI volume to phsyical host, map to guest as additional HDD

• Implementations direct to native hypervisor APIs– Avoids abstraction layers that bring least common

denomination support– Enables easier exploitation of hypervisor

differentiators

• Service instance runs on every physical compute node, helps to minimize failure domain

• Support for security groups that define firewall rules

• Support for– KVM– LXC– VMware ESX/ESXi (4.1 update 1)– Xen (XenServer 5.5, Xen Cloud Platform)– Hyper V


nova-scheduler allocates virtual resources to physical hardware

Key Capabilities:

• Determines which physical hardware to allocate to a virtual resource

• Default scheduler uses a series of filters to reduce set of applicable hosts and uses costing functions to provide Weight

• Not a focus point for OpenStack– Default implementation finds first fit– Shorter the workload lifespan, less critical the

placement decision

• If default does not work, often deployers have specific requirements and develop custom


nova-api supports multiple API implementations and is the entry point into the cloud

Key Capabilities:

• APIs supported– OpenStack Compute API (REST-based)

– Similar to RackSpace APIs– EC2 API (subset)

– Can be excluded– Admin API (nova-manage)

• Robust extensions mechanism to add new capabilities


Network automates management of networks and attachments (network connectivity as a service)

Key Capabilities:

•Responsible for managing networks, ports, and attachments on infrastructure for virtual resources

•Create/delete tenant-specific L2 networks

•L3 support (Floating IPs, DHCP, routing)

•Moving to L4 and above in Grizzly

•Attach / Detach host to network

•Similar to dynamic VLAN support

•Support for• Open vSwitch• OpenFlow (NEC & Floodlight controllers)• Cisco Nexus• Niciria

Architecture


Cinder manages block-based storage, enables persistent storage

Key Capabilities:

• Responsible for managing lifecycle of volumes and exposing for attachment

• Structure is a copy of Compute (Nova), sharing same characteristics and structure in API server, scheduler, etc.

• Enables additional attached persistent block storage to virtual machines

• Support for booting virtual machines from nova-volume backed storage

• Allows multiple volumes to be attached per virtual machine

• Supports following– ISCSI– RADOS block devices (e.g. Ceph distributed file

system)– Sheepdog– Zadara

Architecture


Identity service offers unified, project-wide identity, token, service catalog, and policy service designed to integrate with existing systems

Key Capabilities:

• Identity service provides auth credential validation and data about Users, Tenants and Roles

• Token service validates and manages tokens used to authenticate requests after initial credential verification

• Catalog service provides an endpoint registry used for endpoint discovery.

• Policy service provides a rule-based authorization engine and the associated rule management interface.

• Each service configured to serve data from pluggable backend

– Key-Value, SQL, PAM, LDAP, PAM, Templates

• REST-based APIs


Image service provides basic discovery, registration, and delivery services for virtual disk images

Key Capabilities:

• Think Image Registry, not Image Repository

• REST-based APIs

• Query for information on public and private disk images

• Register new disk images

• Disk images can be stored in and delivered from a variety of stores (e.g. SoNFS, Swift)

• Supported formats– Raw– Machine (a.k.a. AMI)– VHD (Hyper-V)– VDI (VirtualBox)– qcow2 (Qemu/KVM)– VMDK (VMWare)– OVF (VMWare, others)References

http://openstack.org/projects/image-service/


Dashboard enables administrators and users to access and provision cloud-based resources through a self-service portal

Key Capabilities:

• Thin wrapper over APIs, no local state

• Registration pattern for applications to hook into

• Ships with three central dashboards, a “User Dashboard”, a “System Dashboard”, and a “Settings

• Out-of-the-box support for all core OpenStack projects• Nova, Glace, Switch, Quantum

• Anyone can add a new component as a “first-class citizen”.• Follow design and style guide.

• Visual and interaction paradigms are maintained throughout.

• Console AccessReferenceshttp://horizon.openstack.org/intro.html


OpenStack Resources

• Forums

• http://forums.openstack.org/

• Wiki

• http://wiki.openstack.org/

• Documentation

• http://docs.openstack.org/

• Mailing Lists

• http://wiki.openstack.org/MailingLists

• OpenStack Project Management

• https://launchpad.net/openstack

• Blogs

• http://planet.openstack.org

• Real-time chat room

• #openstack and #openstack-dev on irc://freenode.net (443 users currently logged in)

• Rackspace Reference Architectures

• http://www.referencearchitecture.org/

• Easy Install

• http://www.hastexo.com/resources/docs/installing-openstack-grizzly-20131-ubuntu-1204-precise-pangolin


IBM Resources/Solutions for OpenStack CE Available Today

• developerWorks• Google: openstack IBM developerworks

• xCAT (FOSS) for 0-day deployment• xCAT OpenStack Paper (CATStack)• Automated qcow2 image creation for Glance• HW control• Bare-metal discovery and bring up

•Firmware, Base OS, etc…

• IBM Intelligent Cluster Solutions (see Matt Ziegler's PPT)• Preconfigured Switches• Rack and stacked and ready to go• Lab Services for 0-day


IBM Resources/Solutions for OpenStack Available Today

• All IBM System Software and Tools can coexist with OpenStack.• Director, ASU, lflash, etc…

• SoNAS for shared file (NFS, SMB)

• XIV, v7000 for block storage (Cinder)

• iDPX for scale-out Nova Compute and Swift

• BNT switches for OpenFlow and Quantum

• GPFS for iSCSI/block (Cinder) or file.


OpenStack Demo Setup

10.0.9.10 10.0.9.11 10.0.9.12 10.0.9.13 10.0.9.X

172.20.249.10 172.20.249.11 172.20.249.12 172.20.249.13 172.20.249.X

os-essex0 os-essex1 os-essex2 os-essex3 os-essexX

Control Nodes Compute Nodes

Private Networks: eth0: 172.20.249/24 vm: 172.20.250/24

Public Networks: eth1: 10.0.9.0/25 vm: 10.0.9.128/25

computenetwork

computenetwork

computenetwork

computenetworkschedulervolumeconsoleglanceapi

computenetworkschedulervolumeconsoleglanceapi

Scale OutHA Active/Passive

VMVM

VMVM

VMVM

VMVM

VM Firewall


PPT’s and Videos: http://xmission.com/~egan/cloud/

© 2012 ibm corporation openstack ce technology review & demo egan ford ibm distinguished...

Documents

openstack foundation

ibm rallies

ibm embraces

openstack cloud taxonomy

openstack success

customer success ibm

open interoperable cloud

cloud ecosystem slide