Arthur Berezin, Sr. Technical Product Manager, Red Hat

Deep Dive into Highly Available OpenStack Architecture

OpenStack SummitVancouver May 2015

Agenda

★ HA Enabling ServicesPacemaker and HAProxy

★ Shared ServicesMariaDB w/Galera, RabbitMQ w/Mirrored Queues

★ OpenStack ServicesKeystone, Nova, Neutron, Glance, Cinder, Horizon

★ TopologiesController, Compute, Network, Storage

cc: Morio2015 Source: https://www.wikiwand.com/en/Scuderia_Ferrari

Losing Your Controller

https://www.youtube.com/watch?v=Kb43Nxuwc4I

High Availability

● Minimize downtime by avoiding SPOF ● Create service redundancy

○ Active-Active When possible■ Stateless services or HA internal support■ Active-Passive if nothing else is applicable

● Scale out Architecture

HA Enabling TechnologiesPacemaker, HAProxy

● Cluster Resource Manager● Uses Corosync for cluster communication● Monitor and Control Resources:

○ Floating Virtual IP Address (VIP)○ SystemD/LSB/OCF Services ○ Cloned Services(Active/Active)

● STONITH - Fencing with Power Management○ Important for ensuring data consistency

Pacemaker

● Virtual IP(VIP)● SystemD Cloned Resource● STONITH Fencing

Pacemaker OpenStack Service

Node 2 - 192.168.1.2Node 1 - 192.168.1.1

pcsd pcsd

Cloned

STONITH STONITH

Service Service

ServiceVirtual IP10.0.0.1

HAProxy Load Balancer

Load Balancing and Proxy for HTTP/TCP● Mature and popular with web applications● Health Checking ● Load Distribution

● Load Distribution○ Round Robin, ○ Stick-Table

● API Isolation● Failure Detection

Node 1

Node 2 Node 3

HAProxy Load Balancer

Service Service

HAProxy

Avoiding SPOFsA day in a Highly Available Service Life

Horizon Controller

Give Me Horizon Web UI NOW!

Horizon Controller

Give Me Horizon Web UI NOW!

Single Point Of Failure

Horizon Controller 1

Horizon Controller 2

Horizon Controller 3

Give Me Horizon Web UI NOW!

HAProxy Controller 1

Horizon Controller 1

Horizon Controller 2

Horizon Controller 3

Give Me Horizon Web UI NOW!

HAProxy Controller 1

Single Point Of Failure

Each Could Fail

Horizon Controller 1

Horizon Controller 2

Horizon Controller 3

Give Me Horizon Web UI NOW!

HAProxy Controller 1

Single Point Of Failure

Pacemaker Cloned Horizon Service

Horizon Controller 1

Horizon Controller 2

Horizon Controller 3

Give Me Horizon Web UI NOW!

HAProxy Controller 1

HAProxy Controller 3

HAProxy Controller 2

Pacemaker Cloned Horizon Service

Pacemaker Cloned HAProxy Service

Pacemaker Cloned HAProxy Service

Horizon Controller 1

Horizon Controller 2

Horizon Controller 3

HAProxy Controller 1

HAProxy Controller 3

HAProxy Controller 2

Give Me Horizon Web UI NOW!

Horizon

VIP

Pacemaker Cloned Horizon Service

Shared ComponentsDatabase, Messaging

Galera with MariaDB

● Active-Active MultiMaster Synchronous Replication

● Auto Node Joining● Row level parallel replication● Native with MariaDB

DB Node 3DB Node 2DB Node 1 GALERA REPLICATION

wsrep

MariaDB

wsrep wsrep

MariaDBMariaDB

RabbitMQ Clustering with Mirrored Queues

RabbitMQ Node1RabbitMQ Node1RabbitMQ Node1

RabbitMQ

RabbitMQ Clustering

RabbitMQ RabbitMQ Mirrored Queue

OpenStack Services Keystone, Glance, Cinder, Nova, Neutron, Horizon

Keystone

Keystone

HTTPD

SQL: Assignments SQL: Identities LDAP: Identities

API Call

Keystone

Service:★ httpd/Keystone

○ API○ Assignments ○ Identities

■ LDAP■ SQL

● Cloned Stateless HTTPD Service

● Same SSL Certs on all nodes

● Cache is local on each host

Node 2Node 1

Cloned

Keystone

SQL: Assignments SQL: Identities LDAP: Identities

API Call

ClonedHTTPd/Keystone

HTTPd/Keystone

HAProxy HAProxy

pcsd pcsd

Keystone

VIP

STONITH STONITH

● Cloned Stateless HTTPD Service

● Same SSL Certs on all nodes

● Cache is local on each host

Node 2Node 1

Cloned

Keystone

SQL: Assignments SQL: Identities LDAP: Identities

API Call

ClonedHTTPd/Keystone

HTTPd/Keystone

HAProxy HAProxy

pcsd pcsd

Keystone

VIP

STONITH STONITH

Glance

Glance

SQLStorage

Glance-API Glance Registry

Service:★ Glance-API

○ API○ Storage Calls

★ Glance-Registry○ Keeps images

registry at the Database

CeilometerNotifications

HTTP

RabbitMQ

● Both services areCloned Active/Active

● Both services are LB and VIP

Node 2Node 1

Cloned

Glance

SQLImages Store

HAProxy HAProxy

pcsd pcsd

GlanceRegistryVIP

ClonedGlance-API Glance-API

ClonedGlanceRegistry

GlanceAPIVIP

GlanceRegistry

STONITH STONITH

● Both services areCloned Active/Active

● Both services are LB and VIP

Node 2Node 1

Cloned

Glance

SQLImages Store

HAProxy HAProxy

pcsd pcsd

GlanceRegistryVIP

ClonedGlance-API Glance-API

ClonedGlanceRegistry

GlanceAPIVIP

GlanceRegistry

STONITH STONITH

● Both services areCloned Active/Active

● Both services are LB and VIP

Node 2Node 1

Cloned

Glance

SQLImages Store

HAProxy HAProxy

pcsd pcsd

GlanceRegistryVIP

ClonedGlance-API Glance-API

ClonedGlanceRegistry

GlanceAPIVIP

GlanceRegistry

STONITH STONITH

Cinder

CinderVolume

★ Cinder-API○ API

★ Cinder-Scheduler○ Volumes placement

★ Cinder-Volume○ Manages Storage

★ Cinder-Backup

SQL

Storage

Cinder-API

CinderScheduler

RabbitMQ

CinderBackup

Storage

Cinder

Driver

VMData Path

● Cinder-API isStateless Cloned

● LB and VIP● Cinder-Volume is A/P

due it potential races

● Cinder-Backup is A/PNode 2Node 1

A/PVolume Volume

Cloned

Cinder

Storage

HAProxy HAProxy

pcsd pcsd

ClonedCinder-API Cinder-API

ClonedScheduler Scheduler

CinderAPIVIP

DriverDriver

STONITH STONITH

Nova

Nova

NovaCompute

★ Nova-API○ API

★ Nova-Scheduler○ VM placement

★ Nova-Conductor○ Updates DB on

Compute’s behalf ★ Nova-Compute

○ Runs VM Instances

SQL

Nova-API

NovaScheduler

RabbitMQ

NovaConductor

libvirt/KVM

VMVM

Compute Controller Services

Nova

NovaCompute

★ Nova-API○ API

★ Nova-Scheduler○ VM placement

★ Nova-Conductor○ Updates DB on

Compute’s behalf ★ Nova-Compute

○ Runs VM Instances

SQL

Nova-API

NovaScheduler

RabbitMQ

NovaConductor

libvirt/KVM

VMVM

Controller Services● Nova-API configured

with LB and VIP● Nova-API,

Nova-Scheduler and Nova-Conductor are Stateless A/A Cloned services

Node 2Node 1

ClonedConductor Conductor

ClonedHAProxy HAProxy

pcsd pcsd

ClonedNova-API Nova-API

ClonedScheduler Scheduler

Nova-APIVIP

Nova

SQL RabbitMQ

STONITH STONITH

Compute Service● Each host is independent● Nova-compute watched locally

by SystemD● VM HA not supported(yet),

Probably Liberty

Nova

Compute2

NovaCompute

libvirt/KVM

VM

Compute1

VMVM

NovaCompute

libvirt/KVM

Compute Service● Probably supported in Liberty● Each host is independent● Nova-compute watched locally

by SystemD● Liberty Blueprint: Mark Host Down

Nova VM HA

Compute1

VMVM

NovaCompute

libvirt/KVM

STONITH

pacemaker_remote

Compute1

VMVM

NovaCompute

libvirt/KVM

STONITH

pacemaker_remote

Neutron

★ Neutron Server○ API and Management

★ Neutron L2 Agent○ L2 Traffic on compute

★ Neutron L3 Agent○ Network Routing

★ DHCP Agent★ LBaaS Agent

Neutron

SQL

NeutronServer

L2 Agent(s)Open vSwitch

RabbitMQ

L3 Agent

DHCP Agent

LBaaS Agent

Controller

NeutronNetworkNode

Compute2

SQL Internet

Compute1L2 Agent(s)Open vSwitch

L2 Agent(s)Open vSwitch

VMVMVMVMVM VM

L3 Agent DHCP Agent

L2 Agent(s)Open vSwitch

RabbitMQ

LBaaS Agent

NeutronServer

R1

NetworkNode2

NetworkNode1

A/P

Cloned

Cloned+

VRRP

L2 Agent

L3 Agent

LBaaS Agent

RPC

DHCP Agent

Compute1

L2 Agent

L3 Agent

Controller1

LBaaS Agent

DHCP Agent

R1pcsd

Neutron APIVIP

Controller2pcsd

Cloned

Cloned

HAProxy

Neutron-API Neutron-API

HAProxy

R1

Neutron

● Kilo○ L3 Agent HA with VRRP○ DHCP Agent HA

● Liberty■ L3 Agent - DVR■ DVR + VRRP

Longer Term■ Distributed DHCP on compute nodes

Horizon

Service:★ httpd/OpenStack-

Dashboard○ Django web app○ Uses services APIs

Horizon Browser

Horizon

CinderAPI

Neutron API

GlanceAPI

Keystone API

NovaAPI

Horizon

● Cloned Stateless HTTPd Service

● Same SSL Certs on all nodes

● Cache is local on each host

Node 2Node 1

Cloned

ClonedHTTPd/Horizon

HTTPd/Horzon

HAProxy HAProxy

pcsd pcsd

Horizon

VIP

STONITH STONITH

TopologiesController, Compute,Network, Storage

Active - Active Controller Cluster

Controller 1 Controller 2 Controller 3

HAProxy

Packemaker

Keystone

Neutron

Cinder

...

HAProxy

Packemaker

Keystone

Neutron

Cinder

...

HAProxy

Packemaker

Keystone

Neutron

Cinder

....

Galera Multi-master replication

MariaDB MariaDB MariaDB

RabbitMQ Mirrored Queues

Controller Cluster

Compute2Compute1

Nova Compute Nova Compute

Controller 1 Controller 2 Controller 3Controller Services

PublicTenantManagement

Controller Services Controller Services

Controller Services

Controller Services

Controller Services

Controller Cluster

Compute2Compute1

Nova Compute Nova Compute

Network Cluster

PublicTenantManagement

Neutron NetworkNode1

Neutron NetworkNode2

Neutron NetworkNode3

Controller 1 Controller 2 Controller 3

Controller Cluster

Compute2Compute1

Nova Compute Nova Compute

Storage Cluster

StorageManagement

Controller 1 Controller 2 Controller 3 CinderGlanceNode1

CinderGlanceNode2

CinderGlanceNode3

Volume Storage Image Store

Resources

Resources RDO HA Ref Arch

https://github.com/beekhof/osp-ha-deploy

Layer 3 High Availability - VRRP DVR DHCP

http://assafmuller.com/2014/08/16/layer-3-high-availability/

DVR

http://assafmuller.com/2015/04/15/distributed-virtual-routing-overview-and-eastwest-routing/

Creating a Highly Available Red Hat OpenStack Platform Configuration (OSP5 and RHEL 7)

https://access.redhat.com/articles/1150463

About High Availability with OpenStack Platform

https://access.redhat.com/articles/1274203

New nova API call to mark nova-compute down

https://review.openstack.org/#/c/169836/

The Different Facets of OpenStack HA

http://blog.russellbryant.net/2015/03/10/the-different-facets-of-openstack-ha/

Implementation of Pacemaker Managed OpenStack VM Recovery

http://blog.russellbryant.net/2015/04/08/implementation-of-pacemaker-managed-openstack-vm-recovery/

HA Talks during Summit

HA Infrastructure Talks

Pacemaker: OpenStack’s PID 1

MariaDB Galera cluster : Best practices

High Availability Architecture

Deep Dive Into a Highly Available OpenStack Architecture

Real World Practices

Highly Available OpenStack: From Theory to Reality

Lessons learned on upgrades: the importance of HA and

automation

Providing OpenStack Service High-Availability Through

Anycast Routing

HA Storage Talks

Keeping OpenStack storage trendy with Ceph and containers

DRBD9 for OpenStack

The Road to Enterprise-Ready OpenStack Storage as Service

Dude, where is my volume

HA Networking Talks

Highly Available, Performant, VXLAN Service Node

IPv6 impact on Neutron L3 High Availability

High Availability and Resiliency Testing Strategies for OpenStack

Clouds

Thank You