neutron scale

#rackstackatl

Justin Hammond - Developer

Andy Hill - Systems Engineer

Chad Norgan - Systems Engineer

Neutron at Scale

#rackstackatl

Rackspace is early in Neutron implementationMigrating from older versions of Quantum/Melange used since the launch of our public cloud

Scope of this talk is primarily Nova Neutron interaction and the challenges we ⬄faced deploying Neutron at scale

Scope of the Talk

#rackstackatl

Tens of thousands of compute nodes

Hundreds of thousands of instances

Most instances have two or more ports

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

What we mean when we say “at scale”

#rackstackatl

Maintain backwards compatibility with existing products

Neutron will be the ultimate authoritative source for network state

IP Address Management (IPAM)

Modular network drivers so Neutron can service heterogeneous port types

Enable new products to easily integrate into our public cloud offering

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Implementation Requirements

#rackstackatl

Quark Plugin: Open source plugin for Neutron v2 API with IPAM

Custom database migration from Melange/Quantum->Neutron/Quark

Wafflehaus middleware collection

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Implementation Details

#rackstackatlRACKSPACE® HOSTING | WWW.RACKSPACE.COM

Rackspace’s Neutron Implementation

Neutron-api nodes running quark plugin with wafflehaus

Active/Passive databasewith slave

Active/Passive Load Balancers

#rackstackatl

Wafflehaus is a middleware for some specific Rackspace requirements

Very simple way to minimize upstream diffs

Upstream efforts better spent on work that benefits the broader community

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Wafflehaus Overview

#rackstackatlRACKSPACE® HOSTING | WWW.RACKSPACE.COM

Wafflehaus - “The API Mullet”

Business logic in the front, party in the back

#rackstackatl

Does the request body contain particular UUIDs

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Wafflehaus Explained

Wafflehaus middlewares

Would this request violate policy?Add this tag to the request header

Quark plugin

Neutron-api

API Request

#rackstackatlRACKSPACE® HOSTING | WWW.RACKSPACE.COM

Wafflehaus Explained

API Request

Wafflehaus middlewares

Quark plugin

Neutron-api

#rackstackatl

Calls to Keystone

Melange/Quantum Neutron (trunk) Wafflehaus + no-auth

Build 0 5 per port 0

Delete 0 5 per port 0

Info Cache Update 0 LOTS 0

TOTAL 0 TOO MANY 0

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

#rackstackatl

Wafflehaus and No-Auth Middleware

Neutron-api withwafflehaus

PTR for 10.1.2.3?

PTR at compute.trusted.domain

A for compute.trusted.domain?

A at 10.1.2.3

DNS Server

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

API Requestx-forwarded-for

#rackstackatl

[composite:neutronapi_v2_0]

use = call:neutron.auth:pipeline_factory

noauth = dns_filter request_id catch_errors extensions neutronapiapp_v2_0

keystone = request_id catch_errors authtoken keystonecontext extensions

neutronapiapp_v2_0

[filter:dns_filter]

paste.filter_factory = wafflehaus.dns_filter.whitelist:filter_factory

whitelist = trusted.domain

enabled = true

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Wafflehaus Explained

#rackstackatl

Call Volume Before & After

#rackstackatl

Call Volume Before & After

#rackstackatl

Nova caches a copy of the instance’s network information (info cache)

Cache is refreshed on instance operations which reach out to Neutron

Callback system is needed

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

On Info Cache Updates

#rackstackatl

Happens on nova-compute restart

Also happens every heal_instance_info_cache_interval (default 1m)

Currently 6 calls to Neutron per port

Set heal_instance_info_cache_interval=0

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

On Info Cache Updates (continued)

#rackstackatl

nova-cells and Info Cache Updates

Child cells periodically sync with parent cells

Migration to Neutron exposed upstream bug that was corrected in rpc network api, not neutron

Cache updates were sent from child cells to global cells faster than global cells could process

Delays other messages from being processed

#rackstackatl

Callback system between nova and neutron

Read-only database slave usage

Cells support

Nova & Neutron: Fewer calls that do more (e.g., 1 API call, many ports)

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

What’s needed

#rackstackatl

Publicly expose neutron

Security Groups extension support through OVS flows

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

What’s next

#rackstackatl

Patches, Blueprintshttps://review.openstack.org/#/c/88484/ (Neutron, Nova and Cells)https://blueprints.launchpad.net/neutron/+spec/nova-event-callbackhttps://review.openstack.org/#/c/57517/ (noauth python-neutronclient)https://blueprints.launchpad.net/neutron/+spec/ovs-firewall-driver (OVS Firewall Driver)

Projectshttps://github.com/rackerlabs/quarkhttps://github.com/roaet/wafflehaus

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Links

#rackstackatl

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM