Securing OpenStack

Chris C. Kemp

About Chris C. Kemp

Chris C. Kemp / Twitter @Kemp

• OpenStack Co-Founder• Former CTO for IT, NASA• Founder and CEO of Nebula, Inc.

Open source software for buildingprivate and public clouds.

OpenStack is a wonderful choice for the security-minded enterprise’s private cloud *if* best practices are followed during

all stages of implementation and operation.

I’m not *only* making outrageous claims, I’m going to make some points to back this up with the rest of my talk.

OpenStack is a *true* cloud platform

1. On-demand … through self-service interfaces2. Elastic … dynamically scale up and down3. Shared … pooled resources4. Metered by use … at high level of granularity5. Accessible … broadly over the network

OpenStack Details

• Multi-tenant, massively scalable, open source cloud operating system.

• Supports various hypervisors, including: Xen/XenServer , KVM, Hyper-V, VMWare/ESX, Linux Containers (LXC), QEMU, UML

• Flexible network and storage options• Apache 2.0 open source license

Why Build Private Cloud?

1. Maturity. Ability to overcome barriers to entry related to culture, process, technology, experience, and tools.

2. Performance. Need to deploy an application near the data and services that are already deployed on premises - lower latency and increased bandwidth.

3. Security. Must keep data inside Company’s security perimeter, where we trust security team, tools, and processes.

4. Cost. TCO much higher for predictable IaaS workloads.5. Architectural Constraints. Application is not architected

to run well in public cloud, or has unique technical requirements.

By the Numbers

• Community includes 2300 people from 153 Companies

• Over 100 active committers with 250K lines of code

Who is Involved?

And many more…

OpenStack Conceptual Architecture

So, now we know a little something about OpenStack..

CIO

“Sounds exciting!”

CSO

“Sounds target-rich..”

…and we’re forming some initial opinions….

Security and the OpenStack project

OpenStack Security Community Highlights

OpenStack project groups– Vulnerability response– Formalizing security– Audit projects– Multiple security-centric

blueprints – ongoing code improvements

Commercial efforts– Professional penetration Testing /

API fuzzing– Sponsored bugfests with growing

participation– Active and ongoing source code

review process

Bringing defense in depth to OpenStack clouds

Assuming you’ve laid the foundation….

Before we begin we’ll need the governance, guidance, and groundwork that will define the requirements..

- CSA Security Guidance - Critical Areas of Focus in CloudMapping the Cloud Model to the Security Control & Compliance Model

• Compliance and Audit

• ERM

• Legal and Electronic Discovery

• Information Lifecycle

• Corporate policy

• Portability

• Interoperability

• Architecture

• Operations “touch points”

….let’s take a closer look at the technologies in play.

OpenStack Conceptual Architecture

OpenStack Logical Architecture

OpenStack - Under the hood…

Everything needs to be hardened and continuously monitored.

…Luckily, we have a few best practices for doing this stuff with open source software.

• Mostly implemented in Python

• REST and WSGI communication between services

• Multiple application choices to implement backend• databases• queue• networking

OpenStack compute service (nova)

• Equivalent to Amazon EC2

• Runs virtual machines on hypervisor of your choice

• Includes support for block volumes external to hypervisor

• The architecture of nova allows for massive parallel scaling, but to get there requires some complexity.

Underlying technologies

• nova-compute– speaks to libvirt, XenAPI,

etc.• nova DB– SQLalchemy & a SQL DB

• queue– Python Kombu+AMQPlib– rabbitmq

Underlying technologies

• Nova-network– Provides connectivity

• Nova-volume– provides volume API– Backended with iscsi,

sheepdog, ceph, etc

• nova-scheduler– Determines available resources– Assigns workloads

OpenStack Compute Security Considerations

• Secure your hypervisor – This is a topic for another talk… but certainly not trivial.

• Choose a database– Consider high availability– Enhanced security configuration

• Message Queue– Harden the queue software’s configuration– Monitor and correlate queue messages

• Choose filesystem(s)– Enable filesystem’s security features– Deploy hardened daemons– Monitor activity

• Monitor API access

OpenStack object store (swift)• Similar to Amazon S3

• Configurable number of duplicate object replicas

• Supports geo-replication of objects

• Internally:– memcache provides caching for

scale and speed – SQLite– rsync– python greenlets/eventlet

OpenStack Object Store Security considerations

• Properly secure underlying technologies– memcached, rsync

• Implement and test RBAC

• Restrict admin read access to objects– Least privilege, is admin read access required?

• Integrated information lifecycle– Automate / integrate IL processes when possible

• Monitor & correlate API access– Record all access to the object store

OpenStack Dashboard (Horizon)

• Standard webapp stuff

• django-based

• Uses keystone for authN/Z

OpenStack Dashboard Security considerations

• Use enterprise authentication behind keystone

• Standard webapp hardening process

• Protect credentials

• Monitor access and correlate

OpenStack image service (glance)

Provides a repository for VM images and snapshots

• SQL for metadata

• Supports for multiple backend filesystems• Ceph• S3 / Swift• Local FS

OpenStack Image Service Security Considerations

• Choose distributed filesystem(s)– Enable filesystem’s security features and

configure hardened endpoints– Monitor activity

• Choose a database– Consider high availability– Enhanced security configuration

• Audit– Automate audit of images for OS controls

• Patch management– Automate patching and configuration updates to

OS images

OpenStack Identity Service (keystone)

• authN and authZ provider for OpenStack

• Rewrite introduced a new architecture – Straightforward integration with commercial / external auth

products and solutions

OpenStack Identity Service Security considerations

• Use backend enterprise authentication provider – – OpenStack is not an identity project– Keystone’s backend API provides easy integration for

authN, and acceptable authZ

• Monitor API access– Attempts– Failures

• Logging - Monitor and correlate– Monitor identities across OpenStack– Debug loglevel is informative but sensitive

Other parts of the OpenStack ecosystem

• OpenStack incubated projects– Two exciting networking projects• Quantum• Mélange

• Other interesting OpenStack projects– Database-as-a-Service– Dashboard enhancements and plugins– Hybrid cloud functionality

(cloudgateway, etc)

…Zooming back out

Enough of the trenches. This is a keynote, after all.

OpenStack seems to be made up of defensible technologies

– Lots of readable python– Databases: sqlite, mysql, postgres– Message queue: rabbitMQ– Distributed Filesystems: gluster, ceph– Hypervisors: Xen, KVM, ESXi, Hyper-V*– memcached– django– authN / authZ API interface – Linux security features

OpenStack Logical Architecture

…But It’s the responsibility of the implementer to turn the “security switches” to “on.”

Harden

Integrate

Monitor

So - OpenStack isn’t a production-ready cloud?

• Most technical security controls required for compliance are NOT built in to OpenStack.

•

• That shouldn’t dissuade you. • The building blocks are all in place.

Hardening OpenStack system environments

• Restrict network and data access to least privilege

• Enable security features of underlying software

• Configure security features of the underlying OS

• Harden the hypervisor

• Use PKI for SSL

• Implement database security

Integration - benefits to even the playing field

Some integration points:• SIM/ SIEM • IR automation / Live Forensics• CMDB / Service desk • Asset mgmt / Patch mgmt• Auditing process automation• IPAM

Integrating the underlying cloud framework into these elements yields huge benefits

Monitoring – Benefits in the open cloud

OpenStack is powerful foundation to build advanced security controls

Building complex solutions becomes relatively simple

• SIEM sees significant benefits• Automated Incident Response• Cloud-wide flow monitoring• Security appliances: IPS-aaS, FW-aaS, …

Defense in depth of workloads in cloud

• An integrated defense in depth strategy can benefit from open source software and from private cloud

• OpenStack is a great example

Back out to “The big picture”

- CSA Security Guidance - Critical Areas of Focus in Cloud Mapping the Cloud Model to the Security Control & Compliance Model

Looking ahead - 2012 and OpenStack security

A few ideas we’re kicking around at :• Interesting security-as-a-service potential

– Quantum provides some of the missing building blocks needed for metered and scalable security controls on demand in OpenStack

• IR process integration offers excellent coverage– Potential for huge efficiency improvements in remediation of incidents and live

response activities• SIM / SIEM benefits

– Coverage over large infrastructure increases value of SIM integration– The visibility and control that IaaS offers eases SIM complexity

The coming year looks to be very exciting for the OpenStack project, and specifically for OpenStack security.

OpenStack-based products could offer powerful security options.

Conclusion

• OpenStack is a flexible foundation– It’s a viable option, but not necessarily right out of the box– It’s not right for every workload or enterprise– Its open-ness is a big plus for security

• Still some significant unanswered security questions– Expect to see commercial OpenStack-based products

bridging this gap

• Exciting new developments improving the security of OpenStack are happening every day

Thanks for listening!

Chris C. KempTwitter: @Kemp