* Blue box, an ibm company 1

What does it take?

Maintaining a Healthy OpenStack Cloud

Tyler Britten @vmtyler

* Blue box, an ibm company

Who is blue box?

•  In Business Since 2003 Deep operational expertise

•  Global Data Center Deployments Now including IBM SoftLayer data centers

•  300+ Blue Box Customers Trusted private infrastructure deployments

•  Blue Box Panel Allows Blue Box Cloud to be managed over multiple environments from one NOC

•  Acquired by IBM in June of 2015

* Blue box, an ibm company 3

Private Cloud as a service – The best of both worlds

§  Fast to deploy §  Scalable §  Easy to use §  Elastic capabilities

§  Irregular performance §  Vendor lock-in

PUBLIC CLOUD

Hosted PCaaS §  Fast and flexible §  Predictable performance §  Dedicated environment §  Deployed ‘as a Service’ §  Security controls §  Private line, VPN or

Cross-connect

§  Security controls §  Cost control §  Open standards §  Integrate to existing IT

§  Challenging to scale §  Capital and staff

intensive

PRIVATE CLOUD

+

+

All of the security, control, and predictability of Private Cloud with the ease of Public Cloud

* Blue box, an ibm company

LEADING SECURITY

RICHEST PARTNER ECOSYSTEM

Why Customers Choose Blue Box

True Private Cloud as easy to consume as Public Cloud

* Blue box, an ibm company 5

* Blue box, an ibm company 6

* Blue box, an ibm company

Are you doing what you're supposed to be doing?

7

* Blue box, an ibm company

If not, what happened?

8

* Blue box, an ibm company 9

* Blue box, an ibm company 10

What should I be monitoring?

•  The Basics •  OpenStack Services •  RabbitMQ

* Blue box, an ibm company 11

Monitoring The Basics

•  CPU •  Memory •  Disk Space •  RAID & disk issues, fstab •  Large Files •  NTP, kernel options, syslog

* Blue box, an ibm company 12

Monitoring OpenStack Services

•  All OpenStack Services •  Don't forget dependent services! •  Percona •  HAProxy

•  Nova Oversubscription •  Watch for TMI- Too Many Instances

* Blue box, an ibm company 13

Monitoring RabbitMQ

•  Rabbit Services •  Rabbit Queues themselves •  Specifically Growing without Draining

* Blue box, an ibm company 14

* Blue box, an ibm company 15

Logging Rules

1st Rule: Don't impact the system 2nd Rule: DON'T IMPACT THE SYSTEM

3rd Rule: Don't lose logs

* Blue box, an ibm company 16

How do we do that?

•  Avoid dependencies on remote syslog within OpenStack •  Python 2.7 SysLogHandler bug in Ubuntu (1081022)

•  Log Locally to disk and use Logstash-forwarder (Lumberjack) •  Build your logging infrastructure to scale out •  Log Alerting - Less is more

* Blue box, an ibm company 17

* Blue box, an ibm company 18

Monitoring tools

•  Sensu for monitoring •  Graphite / Grafana for

visualization •  Pagerduty for alerting • Sensu Plugins •  Bash, Python, & Ruby •  Basics •  Available on Blue Box

Github!

* Blue box, an ibm company 19

Logging Tools

• ELK Stack (Elastic Search, Logstash, Kibana) • Archiving logs older than 30

days to secondary ELK stack • Scaling Out •  DNS Round Robin for Load

Balancing •  Tag logs appropriately when

forwarding - otherwise stack traces can get impossible

* Blue box, an ibm company 20

* Blue box, an ibm company 21

Ursula

•  A tool created by Blue Box for installing, managing, and maintaining OpenStack powered clouds •  Leverages Ansible Playbooks •  Available on GitHub under the MIT license

• Configuring Logging and Monitoring is part of the playbooks!

* Blue box, an ibm company 22

references

• https://github.com/blueboxgroup/ursula • https://github.com/blueboxgroup/ursula-monitoring/ • https://bugs.launchpad.net/ubuntu/+source/

python2.7/+bug/1081022

* Blue box, an ibm company

THANK YOU FOR YOUR TIME!

twitter.com/bluebox linkedin.com/company/blue-box-group

REACH US AT 119 Pine Street, Suite 200 Seattle, Washington 98101

International: +1.206.607.0660 Fax: 253.272.3005 Mail: sales@bluebox.net Web: www.bluebox.net

WHERE WE ARE FIND US AT

23