cloudy operations - oscon 2010

Copyright © 2010 Opscode, Inc - All Rights Reserved 1

John WillisVP of [email protected]/botchagalupe

Thursday, July 29, 2010

mailto:[email protected]

mailto:[email protected]


IT Management PodcastDevopsCafeCloudCafe


Managedhosting

Virtualization

Private Public

SaaS

PaaS PaaS

IaaS IaaS

Slide courtesy Alistair Croll - [email protected], July 29, 2010

Managedhosting

Virtualization

Private Public

SaaS

PaaS PaaS

IaaS IaaS

If you want to

talk clouds,

pick one first.


Infrastructure as a Service(IaaS)

Amazon EC2, Rackspace Cloud, Terremark, Gogrid, Joyent (and nearly every private cloud built on Zenserver or VMWare.)


Cloudy Operations


Copyright © 2010 Opscode, Inc - All Rights Reserved

Pixie Dust!

6Thursday, July 29, 2010


Did They Lie?



Did They Lie?

I did not have “cloudy” relations with that provider



Infrastructure is Hard!



FullyAutomated Infrastructure

ReleaseControl

Orchestration

Dispatcher

Provisioning

Deploy

Config management

OS boot/install

Artifact repository

Build

CI Server Issue tracker

SCM Repository

Model

Asset inventory

Host naming

Identity

CMDBMonitoring

Events

Trending Reporting

Trending

Workflows

Resources

Topology

Configuration

Code

Sources

Scheduler


Network OperationsSystems Administrators

Software DevelopersDatabase Administrators

Storage ManagementProject ManagementChange Management

Continuity PlanningRisk Management

Web DesignPerformance

ComplianceArchitecture

ToolingTestingSecurityReporting

Facilities

* in no particular

order( )



SNAFU11



Cloudy Monitoring



Cloudy Monitoring

12

Performance



Cloudy Monitoring

12

Performance Log



Cloudy Monitoring

12

Performance Log

Alerts



Cloudy Monitoring

12

Performance Log

AlertsEvent



Cloudy Monitoring

12

Performance Log

AlertsEvent

Correlation



Cloudy Monitoring

12

Performance Log

AlertsEvent

CorrelationCapacity



Cloudy Monitoring

12

Performance Log

AlertsEvent

CorrelationCapacityAnalytics



Cloudy Monitoring

12

Performance Log

AlertsEvent


Nagios



Cloudy Monitoring

12

Performance Log

AlertsEvent


NagiosCollectd



Cloudy Monitoring

12

Performance Log

AlertsEvent


NagiosCollectdjcollectd



Cloudy Monitoring

12

Performance Log

AlertsEvent


NagiosCollectdjcollectdGanglia



Cloudy Monitoring

12

Performance Log

AlertsEvent


NagiosCollectdjcollectdGangliaZenoss



Cloudy Monitoring

12

Performance Log

AlertsEvent



JMX



Cloudy Monitoring

12

Performance Log

AlertsEvent



JMXOpenNMS



Cloudy Monitoring

12

Performance Log

AlertsEvent



JMXOpenNMS

Munin



Cloudy Provisioning



Cloudy Provisioning

13

Provisioning



Cloudy Provisioning

13

Provisioning

Configuration



Cloudy Provisioning

13

Provisioning

Configuration

Systems Integration



Nodes

opslb01

opsws01opsws02

opsdm01opsds01opsds02

Provisioning



Roles

loadbalancerwebserverdbmasterdbslave

Configuration Management



Load Balancer

Web Server Web Server

DB Master

DiskDisk

DB Slave DB Slave

Disk

Recipes

haproxyapache2myssql

Systems Integration



name "webserver"description "Systems that serve HTTP traffic"

run_list( "role[base]", "recipe[apache2]", "recipe[apache2::mod_ssl]")

default_attributes( "apache" => { "listen_ports" => [ "80", "443" ] })

override_attributes( "apache" => { "max_children" => "50" })

Role Based Configuration



Load Balancer Example



Load Balancer

Web Server Web Server

DB Master

DiskDisk

DB Slave DB Slave

Disk

Systems Integration



Devops



Devops

•Culture



Devops

•Culture

•Automation



Devops

•Culture

•Automation

•Measurement



Devops

•Culture

•Automation

•Measurement

•Sharing



What Do Developers

Need?


For Developers...


For Developers...

• Self Service Operations


For Developers...


• The infrastructure is the application (and vice versa)


For Developers...



• Minimize Bottlenecks


For Developers...



• Minimize Bottlenecks

• The “Right” Tools



What Does Operations Need?


Operations

http://covers.oreilly.com/images/9780596007836/lrg.jpg

Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft




http://twitter.com/moonpolysoft


Operations• Say “Yes”.









• You never liked rack and stack that much anyway.










• You have never been more critical.










• You have never been more critical.

• Just get out of the way.









Agile Infrastructure

Development Team focusIDE/WorkbenchAgile methodologySource Control



Operations

Individual focusScript VI basedSource control?Waterfall

Agile Infrastructure



Infrastructure as Code

28

http://www.flickr.com/photos/wonderlane/2306082998/





Infrastructure as Code is...

29http://www.flickr.com/photos/kwerfeldein/2634561264/sizes/o/


http://www.flickr.com/photos/kwerfeldein/2634561264/sizes/o/



Infrastructure as Code is...

29

A technical domain revolving around building and managing infrastructure programmatically






Enable the reconstruction of the business from nothing

but a source code repository, an application

data backup, and bare metal resources.


A Tornado Hits Your Data

http://www.flickr.com/photos/gi/518613153/sizes/o/





• Pause your movie







• Sign into your cloud provider








• Upload your offsite backups









• Provision, config and integrate the new servers










• Change DNS to point to “Hit by Tornado” page











• Restore the customer and application data












• Remove the “Hit by Tornado” page













• Unpause moviehttp://www.flickr.com/photos/gi/518613153/sizes/o/












• Unpause moviehttp://www.flickr.com/photos/gi/518613153/sizes/o/

Chapter 5 Infrastructure

as CodeAdam Jacob





In a Cloudy WorldYour Prime Constraint Should Be

32

http://www.flickr.com/photos/visualage/2126833132/sizes/o/





In a Cloudy WorldYour Prime Constraint Should Be

32

The time it takes to

restore your application

data






Infrastructure as Code


RecipiesApplies resources in the order they are specified

http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/





• Can include other recipes.






• Can include other recipes.

• A DSL like Ruby.






(http://radar.oreilly.com/archives/2007/10/operations-advantage.html)

10

20

30

40

50

“Traditional” Operations

# o

f H

our

s

05

101520

1 2 3 4 5 6 7 9 10 11 12

Ser

vers

Week #

10

20

30

40

50

Operations - The “Secret Sauce”

UpkeepConfigOS InstallHardware

05

101520

1 2 3 4 5 6 7 9 10 11 12

Week #

ExistingNew

Tale of Two Startups


http://radar.oreilly.com/archives/10/



(http://radar.oreilly.com/archives/2007/10/operations-advantage.html)

10

20

30

40

50

“Traditional” Operations

# o

f H

our

s

05

101520

1 2 3 4 5 6 7 9 10 11 12

Ser

vers

Week #

10

20

30

40

50

Operations - The “Secret Sauce”

UpkeepConfigOS InstallHardware

05

101520

1 2 3 4 5 6 7 9 10 11 12

Week #

ExistingNew

This is the secret of Cloud Computing.

Every other virtue stems from here.

Tale of Two Startups




A Period of Combinatorial Innovation



• Abstract and fault tolerant components




• Integrated network accessible services




• Integrated network accessible services

• Unlimited infrastructure



Industry Shifts

37

Be bold-and mighty forces will come to your aidBasil King


cloudy operations - oscon 2010

Documents