holistic security for openstack clouds

Holistic Security for OpenStack Clouds

Major HaydenPrincipal Architect, Rackspace

@majorhayden

Photo credit: bastiend (Flickr)

Image credit: Wikipedia

Security feels like this

Image credit: Wikipedia

Securing complex systems createsmore challenges

Securing OpenStack can feel liketaking a trip to the Upside Down.

It doesn’t have to be that way(even with something as complex as OpenStack)

Image credit: Pixabay

The key istaking the right approach to secure a complex system.

Major HaydenPrincipal Architect

● At Rackspace since 2006

● Working on OpenStack since 2012

● Focused on information security for Rackspace Private Cloud

● Fedora Linux contributor; Fedora Security Team and Server Working Group member

● Has a terrible domain name purchase habit(please, no ideas for domain names today)

Holisticcharacterized by comprehension of the

parts of something as intimately interconnected and explicable only by

reference to the whole

-- Oxford English Dictionary

The holistic approach for humans considers a person to be made of a body, a mind, and a spirit.

Image credit: Pixabay

The holistic approach for OpenStack considersa cloud to be made of servers, software, and a business goal.

A holistic approach to security involves people, processes, and technologies working in tandem.

“The whole is greaterthan the sum of its parts,especially in the case of OpenStack.”

-- (partially) Aristotle

Image credit: Wikipedia

How does this apply to securing an OpenStack cloud?

Let’s do a quick security refresher.

Assume that attackerswill get inside eventually.

Image credit: Pixabay

Attackers are on offense.They can be wrong many times.

Defenders can only be wrong once for a breach to occur.

Securing only the outer perimeteris not sufficient.

We must secure our OpenStack cloud.We need to go deeper.

We just bought an expensive firewall for the perimeter. Isn’t that enough?

(no caption necessary)

Build small security improvementsat multiple layers.*

* This is the cornerstone of defense-in-depth.

Individually, these changes may not seem to have much value.

All of these changes create a strong, valuable security strategy when they are added together.

Let’s get to the good stuff.

Image credit: Pexels

Work from the outside in(just like you would at a fancy dinner)

Image credit: Wikipedia

Four layersOuter perimeter

Control and data planes

Control plane deep dive:OpenStack services and backend services

OpenStack services deep dive

Image credit: imageme (Flickr)

The outer perimeter

Image credit: Pixabay

OUTER PERIMETER SECURITY GOAL:Convince your attackers that it’s easier to attack someone else’s cloud

Key concepts

Make it expensive for attackers to breach your perimeter defense

When they do make it through, ensure that you know about it immediately

Perimeters usually have openings on the outside and inside -- secure both of them

Tactical objectives

Require a VPN for access from external networks

Segregate internal networks using a firewall or an internally-facing VPN

Monitor all logins (successful and unsuccessful) for unusual activity

Track bandwidth usage trends using netflow data

Secure the perimeter

VPN

Internet Corporate network

Firewall

Log collector Alert system

Netflow collector

Auth system

Control and data planes

Image credit: Pixabay

Control and data plane

Control planekeystone, nova, glance,

cinder, neutron, horizon, rabbitmq, mysql,

memcached

Data planeHypervisors and

tenant-built items (VMs, containers, networks,

storage)

CONTROL/DATA PLANES SECURITY GOAL:Keep the inner workingsof your OpenStack cloud separated fromtenant infrastructure

Key concepts

Tenant infrastructure should have extremely limited access to the control plane, and vice versa

A misconfigured tenant VM could open a wide hole in your secure network

Protect your cloud from VM exit exploits that allow attackers to gain hypervisor access

Tactical objectives

Separate control plane, hypervisors and tenant infrastructure with VLANs and strict firewall rules (and monitor dropped packets)

Use SELinux or AppArmor on hypervisors to reduce the impact of VM and container exit exploits

Hypervisor

Linux Security Module refresher

Three popular implementations: SELinux, AppArmor, and TOMOYO

sVirt (in libvirt) ensures that all processes are labeled properly (SELinux) or have profiles configured (AppArmor)

VM exit exploits are confined in most situations

Tenant VM

Storage Network

Linux Security Module

Do not disableSELinux or AppArmoron your hypervisors.

(Seriously. Leave it enabled.)

Control plane deep dive:OpenStack and backend services

Image credit: Wikipedia

CONTROL PLANE SECURITY GOAL:Heavily restrict lateral movement and restrict access to the “crown jewels”

“crown jewels” are the databases and message queuesin your OpenStack cloud

Control plane deep dive

OpenStack serviceskeystone, nova, glance,

cinder, neutron, horizon

Backend servicesmysql, rabbitmq,

memcached, syslog

The “crown jewels” are hereThe map to the “crown jewels” is here

Key concepts

Allow the least amount of access possible from the OpenStack services to backend services

Further restrict access to specific ports, sources, and destinations

Deploy services into containers to apply fine-tuned network and process restrictions

Tactical objectives

Use a load balancer or firewall to create a “choke point” between OpenStack and backend services

Monitor messaging and database performance closely to look for anomalies or unauthorized access

Use unique credentials for each MySQL database and RabbitMQ virtual host

OpenStack services deep dive

Image credit: Wikipedia

OPENSTACK SERVICES SECURITY GOAL:Know what valid communication looks like and alert oneverything else

OpenStack has many (predictable) interactions

Key concepts

OpenStack services are heavily interconnected, but the connections are predictable

Limit access between OpenStack services and monitor any invalid questions

Tactical objectives

Use iptables rules to limit access between OpenStack services; alert on any invalid connections

Give each service a different keystone service account (with different credentials)

Monitor closely for high bandwidth usage and high connection counts

Let’s wrap up

Analyze.Isolate.Monitor.Repeat.

These small security changesadd up to a strong defense

Image credit: Wikipedia

Try OpenStack-Ansible

OpenStack-Ansible deploys enterprise-grade OpenStack clouds using Ansible.

Security and reliability are two of the core priorities for the project. Most of the security changes in this talk are already implemented.

Learn more: http://bit.ly/openstack-ansible

RACKSPACE PRIVATE CLOUDPOWERED BY OPENSTACK®

Learn more about ourproven operational expertise,

industry-leading reliability,and OpenStack Everywhere.

Join us at the Rackspace booth (A22)in the OpenStack Marketplace.

RACKSPACE INVENTED OPENSTACK® – NOW WE'RE PERFECTING IT

Thank you!Major Hayden

@majorhaydenmajor.hayden@rackspace.com

Photo credit: bastiend (Flickr)