Marriage of ESX and OpenStack at PayPal

Scott Carlson, PayPal

VSVC4994

#VSVC4994

VMWORLD 2013

MULTI-VENDOR AGILITY

THE MARRIAGE OF ESX AND

OPENSTACK AT PAYPAL

3

PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes.

• 132,000,000 Users.

• $300,000 Payments processed by PayPal each minute.

• 193 markets / 25 currencies.

• PayPal is the World’s Most Widely Used Digital Wallet.

ABOUT PAYPAL

4

• 80% of the PayPal front-end is virtualized on VSphere 5.0u1

• Primary Criteria

− Stability, performance, industry expertise, availability of experts

• Standardized on VCE VBLOCK© for initial implementation

• Fully consumable API

• Load-test harness well understood in industry (specInt & vMark)

− Predictable scaling pattern for horizontally scaled workloads

WHY WE VIRTUALIZED ON ESX

5

CLOUD

6

2012/2013 Shift toward an internal cloud model

• Shift from Enterprise design model to cloud-based design

• Elastically scale and self-heal infrastructure to accommodate unpredictable usage patterns of customers and internet commerce

• Separate rapidly iterating customer experiences from core services

• reduce overall cost per transaction within the environment

PAYPAL INTERNAL CLOUD

7

CLOUD IS THE GREAT ENABLER

ENABLE THE DEVELOPER

Code Deploy Enjoy

ENABLE THE BUSINESS

Payment Delivery

One-Click Developer Self Service

Global Compute & Data Fulfillment

Self-Organizing & Optimizing Infrastructure

System Intelligence Driven Operation

8

• Technology

− Adopt Open Source Solutions where ever possible

− No Vendor Lock-in

− Industry Best Practices

− Leverage Industry/ebay Inc Investments

• Functionality

− Self-Service tool for application life cycle management.

− Robust Automation & Orchestration

− Seamless On-Demand Capacity Fulfillment

PAYPAL CLOUD PLATFORM – GUIDING PRINCIPLES

9

PayPal deploying Openstack in order to help transform our global infrastructure into an agile and open cloud platform.

Agility - time to market for customer facing services

Agility - speed to service developer requests for VM resources

Agility – utilize the engineering culture of PayPal to

deliver specialized cloud services where needed

OPENSTACK

10

INCREDIBLE INDUSTRY SUPPORT

11

TECHNOLOGY STACK

Operations Portal Horizon, Ceilometer

DEVS Deployment Portal

Orchestration Engine

Cloud Formation (Heat)

Nova, Cinder, Swift, Keystone, Quantum, Horizon

Cobbler

ISC DHCP Salt BIND RHEL 6.x Hypervisor Zabbix

x86 Compute Local Storage Network Load

Balancer

User

Interface

Orchestration

Foundational

Services

Software

Infrastructure

Hardware

Infrastructure

Traffic Mgmt Monitoring Metering Stages Workflow Monitoring

LBaaS, DNSaaS FWaaS

PP Specific

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

CLOUD BEFORE INTEGRATION

DATABASE & RESTRICTED ZONE

Clo

ud M

ana

gem

ent Z

one

VC

ente

r M

an

ag

em

en

t WE

B

MID

KVM

Local Disk

“Stateless & Disposable”

ESX 5.0u2

Shared Storage

ESX 5.0u2

Shared Storage

ESX 5.0u2

Shared Storage

Physical

Non-virtualized

KVM

Local Disk

ESX 5.0u2

Shared Storage

Physical

Non-virtualized

FZ = Logical Fault Zones

13

SIDE-BY-SIDE

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

F

Z

CLOUD AFTER INTEGRATION

DATABASE & RESTRICTED ZONE

KVM

Local Disk

ESX 5.0u2

Shared Storage

Physical

Non-virtualized

KVM

Local Disk

ESX 5.0u2

Shared Storage

Physical

Non-virtualized

Clo

ud M

ana

gem

ent Z

one

WE

B

MID

15

But isn’t Openstack a direct replacement for ESX? Why would you keep them both?

ESX/Vsphere != Openstack

NOVA != vSphere || vCenter || ESXi

NOVA =~ vCD, vCAC

KVM =~ ESX

To connect to any hypervisor, the Openstack cloud ‘proxies’ connections to any supported hypervisor via Nova. That abstracts the ‘Cloud’ from the hypervisor

COMPARING

16

• Equivalent functionality on KVM and ESX

• Full birth to death lifecycle management of virtual machines

− Build new, power on, power off, console, rebuild, delete

• Auto-configuration of host resources following t-shirt sizes standards

− CPU, RAM, NIC, IP, OS Version

• IP Address Management

• Build from “Snapshot”/”Template”

• Deploy resources following appropriate fault zone model

• Must work from within single Horizon/Asgard interface

BRINGING ESX ‘INTO’ THE CLOUD

17

• VSphere/ESX 5.1

− 5.0 works but many, many, many back-ports / tweaks

• Single security zone per hypervisor

− No sharing of confidential & non-confidential on same hardware (PCI)

• Openstack management network communication

− This is NOT necessarily the VKERNEL network

HYPERVISOR REQUIREMENTS

18

• Shared storage required

− Data Store Cluster

− Single Data Store support [ bug fix coming ]

• DRS Enabled with auto-placement

• Data Stores must be created in advance

− No Cinder support

STORAGE REQUIREMENTS

OPENSTACK GRIZZLY

OpenStack Object Store

OpenStack Image Ser vice OpenStack Compute

OpenStackDashboard

OpenStack Identity Service

OpenStack Compute API /

Admin API

keystone(ser vice & admin APIs)

nova-api(OS, EC2, Metadata, Admin)

nova-consoleauth

nova-cert/objectstore

nova-consolenova-*proxy

VNC/ VMRC/ Spice

OpenStack Object API

http://www.solinea.com

Queue

nova-compute

nova-scheduler

novadatabase

OpenStackCompute API

OpenStack Image API

Hor izon

OpenStack Image API

ident it y backend

swif t-proxy

objectcontaineraccount

objectstore

accountDB

containerDB

OpenStack Object API

HTTP(S)

OpenStackObject API

OpenStack Identity API

OpenStack Identity

API

OpenStack Identity

API

Internet

OpenStackImage API

OpenStack Identity

API

OpenStack Image API

catalog backend

token backend

OpenStack Identity

API

hyper visor

libvirt, XenAPI, etc.

HTTP(S)

Amazon Web Ser vices

EC2 API

OpenStack Net work Ser vice

glance-api

glance-regist r y

glancedatabase

OpenStack Block Storage

OpenStack Block Storage API

cinder-api

cinder-volume

quant um-ser ver

quant um plugin(s)

OpenStack Identity

API

cinder-scheduler

cinderdatabase

OpenStack Net work API

net workprovider

OpenStack Block Storage API

OpenStack Net work API

policy backend

Queue

OpenStack Net work API

quant umdatabase

quant um agent (s)

nova-conductor

memcached

OpenStack Identity

API

⁃ OpenStack Command Line Tools (nova-client, swif t-client, etc.)

⁃ Cloud M anagement Tools (Rightscale, Enst rat ius, etc.)

⁃ GUI tools (Cyberduck, iPhone client, etc.)

volume provider

cinder-backup

OpenStack Object API

OpenStack Identity

API

Queue

OpenStack Block Storage API

ITS ALL ABOUT NOVA

CONFIG OF NOVA

21 Confidential and Proprietary

#compute_driver = libvirt.LibvirtDriver

compute_driver = vmwareapi.VMwareVCDriver

vmwareapi_host_ip=192.168.20.50

vmwareapi_host_username=root

vmwareapi_host_password=vmware

vmwareapi_cluster_name=openstack_test

vmwareapi_wsdl_loc=https://192.168.20.50/sdk/vimService.wsdl

Vcenter 5.1 Appliance

Can be multiple

clusters now!

Nova is the project name for OpenStack Compute, a cloud computing fabric controller,

the main part of an IaaS system. Individuals and organizations can use Nova to host

and manage their own cloud computing systems.

GLANCE AND IMAGES

22 Confidential and Proprietary

glance add name=”MYMACHINE.vmdk" disk_format=vmdk container_format=bare

is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated"

vmware_ostype="otherGuest" < /path/to/MYMACHINE.vmdk

Rules for Glances images for VMWare

• Saved in VMDK Format

• Imported as VMDK Format

• Thick Provisioned VMDK Required

• No split VMDK allowed (must be merged)

• In a multi-hypervisor cloud, all images are separate

‘per hypervisor’ (no launching KVM VM’s on ESX)

BUILDING AND INSTALLING OS

23 Confidential and Proprietary

• Kickstart

• Build a small root disk

• Use kickstart to image machine

• Post-install with puppet to customize machine and

add additional mount points depending on

application requirements

• Image Deploy

• Currently does not support ‘config-drive’

• Need Guest Tools to ‘duplicate’ functionality

WHAT ABOUT THE NETWORK

24 Confidential and Proprietary

• Quantum requires NVP 3.2

• Cannot talk directly to VSphere API to allocate VDS

Port to NIC

• Implemented via vAPP – integration bridge

• Configured as separate transport zone within Nicira

WHAT’S LEFT

25 Confidential and Proprietary

• Component “at-scale” testing

Currently manage “tens” at a time, need to move

to “hundreds” or “thousands”

• Most fixes scheduled to go into Havanna, every

bug-fix needs to be reviewed and possible back-

ported to Grizzly

• Multiple Data Store enumeration on a cluster

• Full Certification on VCE© VBLOCK with Vision

Intelligent Operations, auto-upgrades, and full

Openstack support of all components

READING MATERIALS

• http://www.solinea.com/2013/06/15/openstack-grizzly-architecture-revisited/ - Ken Pepple

• http://www.slideshare.net/kenhui65/getting-started-with-open-

stack?ref=http://cloudarchitectmusings.com/2013/06/16/getting-started-with-openstack/ - Kenneth Hui

• http://docs.openstack.org/trunk/openstack-compute/admin/content/config-drive.html - config-drive doc

• http://docs.openstack.org/trunk/openstack-compute/admin/content/vmware.html - Openstack VMWARE doc

• http://www.ebay.com - Buy It Now

• http://www.paypal.com - and then Pay for it Here!

Interested? DL-PayPal-Cloud-Hiring@ebay.com

THANK YOU

Marriage of ESX and OpenStack at PayPal

Scott Carlson, PayPal

VSVC4994

#VSVC4994