neutron scale

#rackstackatl

Justin Hammond - Developer

Andy Hill - Systems Engineer

Chad Norgan - Systems Engineer

Neutron at Scale

#rackstackatl

Rackspace is early in Neutron implementationMigrating from older versions of Quantum/Melange used since the launch of our public cloud

Scope of this talk is primarily Nova Neutron interaction and the challenges we ⬄faced deploying Neutron at scale

Scope of the Talk

#rackstackatl

Tens of thousands of compute nodes

Hundreds of thousands of instances

Most instances have two or more ports

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

What we mean when we say “at scale”

#rackstackatl

Maintain backwards compatibility with existing products

Neutron will be the ultimate authoritative source for network state

IP Address Management (IPAM)

Modular network drivers so Neutron can service heterogeneous port types

Enable new products to easily integrate into our public cloud offering


Implementation Requirements

#rackstackatl

Quark Plugin: Open source plugin for Neutron v2 API with IPAM

Custom database migration from Melange/Quantum->Neutron/Quark

Wafflehaus middleware collection


Implementation Details

#rackstackatlRACKSPACE® HOSTING | WWW.RACKSPACE.COM

Rackspace’s Neutron Implementation

Neutron-api nodes running quark plugin with wafflehaus

Active/Passive databasewith slave

Active/Passive Load Balancers

#rackstackatl

Wafflehaus is a middleware for some specific Rackspace requirements

Very simple way to minimize upstream diffs

Upstream efforts better spent on work that benefits the broader community


Wafflehaus Overview


Wafflehaus - “The API Mullet”

Business logic in the front, party in the back

#rackstackatl

Does the request body contain particular UUIDs


Wafflehaus Explained

Wafflehaus middlewares

Would this request violate policy?Add this tag to the request header

Quark plugin

Neutron-api

API Request



API Request

Wafflehaus middlewares

Quark plugin

Neutron-api

#rackstackatl

Calls to Keystone

Melange/Quantum Neutron (trunk) Wafflehaus + no-auth

Build 0 5 per port 0

Delete 0 5 per port 0

Info Cache Update 0 LOTS 0

TOTAL 0 TOO MANY 0


#rackstackatl

Wafflehaus and No-Auth Middleware

Neutron-api withwafflehaus

PTR for 10.1.2.3?

PTR at compute.trusted.domain

A for compute.trusted.domain?

A at 10.1.2.3

DNS Server


API Requestx-forwarded-for

#rackstackatl

[composite:neutronapi_v2_0]

use = call:neutron.auth:pipeline_factory

noauth = dns_filter request_id catch_errors extensions neutronapiapp_v2_0

keystone = request_id catch_errors authtoken keystonecontext extensions

neutronapiapp_v2_0

[filter:dns_filter]

paste.filter_factory = wafflehaus.dns_filter.whitelist:filter_factory

whitelist = trusted.domain

enabled = true



#rackstackatl

Call Volume Before & After

#rackstackatl

Nova caches a copy of the instance’s network information (info cache)

Cache is refreshed on instance operations which reach out to Neutron

Callback system is needed


On Info Cache Updates

#rackstackatl

Happens on nova-compute restart

Also happens every heal_instance_info_cache_interval (default 1m)

Currently 6 calls to Neutron per port

Set heal_instance_info_cache_interval=0


On Info Cache Updates (continued)

#rackstackatl

nova-cells and Info Cache Updates

Child cells periodically sync with parent cells

Migration to Neutron exposed upstream bug that was corrected in rpc network api, not neutron

Cache updates were sent from child cells to global cells faster than global cells could process

Delays other messages from being processed

#rackstackatl

Callback system between nova and neutron

Read-only database slave usage

Cells support

Nova & Neutron: Fewer calls that do more (e.g., 1 API call, many ports)


What’s needed

#rackstackatl

Publicly expose neutron

Security Groups extension support through OVS flows


What’s next

#rackstackatl

Patches, Blueprintshttps://review.openstack.org/#/c/88484/ (Neutron, Nova and Cells)https://blueprints.launchpad.net/neutron/+spec/nova-event-callbackhttps://review.openstack.org/#/c/57517/ (noauth python-neutronclient)https://blueprints.launchpad.net/neutron/+spec/ovs-firewall-driver (OVS Firewall Driver)

Projectshttps://github.com/rackerlabs/quarkhttps://github.com/roaet/wafflehaus


Links

https://review.openstack.org/#/c/88484/

https://blueprints.launchpad.net/neutron/+spec/nova-event-callback

https://review.openstack.org/#/c/57517/

https://blueprints.launchpad.net/neutron/+spec/ovs-firewall-driver

https://github.com/rackerlabs/quark

https://github.com/roaet/wafflehaus/

#rackstackatl

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

neutron scale

Technology

rackstackatl rackspace

rackspace hosting rackspace

rackstackatl wafflehaus

ports rackspace hosting

true rackspace hosting

rackstackatl calls

neutron v2 api

rackstackatl novacells