susecon crowbar presentation
DESCRIPTION
Presentation given by Greg Althaus & Rob Hirschfeld @ SUSEcon 2012 about Dell Crowbar, DevOps and Cloud Deployment.TRANSCRIPT
Crowbar: The Cloud Operations Framework
Greg Althaus Dell Principal Engineer and Crowbar Co-Founder
Revolutionary Cloud & Big Data Solutions
Agenda
• What Led to Crowbar?
• What is Crowbar?
• Crowbar Architecture
• SUSE + Crowbar = A Path to Clouds
• Crowbar 2.0 and Beyond
• Q&A
2
Revolutionary Cloud & Big Data Solutions
What Led to Crowbar?
3
Revolutionary Cloud & Big Data Solutions
Crowbar <3 OpenStack “Boxes to working OpenStack in <4 hours” • Crowbar’s birthright was “an OpenStack installer”
– But we started it knowing there was a much bigger problem
• Like many open source projects, OpenStack is complex and changes
constantly – Want the innovation and afraid of change
• To create an OpenStack project, needed:
– To get customers running quickly – Have a way to cope with frequent updates – Connect a lot of parts together – Be part of the open source community
• Got & get substantial uplift from
the OpenStack community
4 Confidential
Revolutionary Cloud & Big Data Solutions
Problem: Clouds do not operate themselves!
HW
HW
SW
SW
Ops Ops
Operations
Expertise
Servers Networks
Storage Etc.
5 Confidential
Revolutionary Cloud & Big Data Solutions
Ops automation (“devops”) jumpstarts learning, reduces complexity and manages scale
HW
HW
Ops
Ops
SW
SW
Operations
Expertise
Operations Automation
(DevOps)
Servers Networks
Storage Etc.
6 Confidential
Revolutionary Cloud & Big Data Solutions
What is DevOps? Why is it important?
• DevOps extended to infrastructure is “CloudOps”
• Operational Model for Software – “Operations as Code” – Highly automated – Layered deployments (no image based) – Continuous deployment
• Shaped by Amazon Cloud
– API only access to infrastructure – Elastic resources that are easier to create and destroy than change
• Configuration Management Tooling – Chef & Puppet – Builds applications from open source modules/recipes – Not quite coding, but far from manual
7 Confidential
Revolutionary Cloud & Big Data Solutions
OpenStack has a lot of interlocking parts!
8 Confidential
Revolutionary Cloud & Big Data Solutions
…and it matters how they are deployed!
Cookbooks
9 Confidential
Revolutionary Cloud & Big Data Solutions
Images vs. Layers: Lifecycle
Images: Replacement Layers: Upgrade
I+A+U+O/S
Config
OS
U Bar v1 Foo
I
Co
nfi
g
I+A+U+O/S
Config
I+A+U+O/S
Config
I+A+U+O/S
Config
OS
U Bar v2 Foo
I
Co
nfi
g
Bar v2
10 Confidential
Revolutionary Cloud & Big Data Solutions
CloudOps for OpenStack
There are lot of pieces, but they share many elements!
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova API
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova DB
I
Co
nfi
g
OS
U
Nova Cntrl
I
Co
nfi
g
OS
U
Dash Board
I
Co
nfi
g
OS
U
Nagios
I C
on
fig
OS
U
Crowbar Chef
I
Co
nfi
g
11 Confidential
Revolutionary Cloud & Big Data Solutions
CloudOps for OpenStack
Plus interconnections are highly complex and dynamic
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova API
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova Node
I
Co
nfi
g
OS
U
Nova DB
I
Co
nfi
g
OS
U
Nova Cntrl
I
Co
nfi
g
OS
U
Dash Board
I
Co
nfi
g
OS
U
Nagios
I C
on
fig
OS
U
Crowbar Chef
I
Co
nfi
g
12 Confidential
Revolutionary Cloud & Big Data Solutions
What is Crowbar?
13
Revolutionary Cloud & Big Data Solutions
Crowbar, the Cloud Deployer++
• What is Crowbar – Not only reduces the effort for deploying clouds – Also provides an operational model for managing them
• Field proven technologies – Build on locally deployed Chef Server – Raw servers to full cloud in <2 hours, multi-node from the ground up – Hardened with over a year of deployments
• Apache 2 open source (on github/dellcloudedge)
– Multi-apps (OpenStack & Hadoop), multi-OS (Ubuntu, RHEL, CentOS, SUSE), Development “in the open” w/ community engagement
• Hadoop & OpenStack releases (dell.com/OpenStack)
– Cactus, Diablo and Essex!
• Open version is not Dell hardware Limited – This is been an important factor in adoption
14 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar in the Open (Community!)
• Community – Focused on OpenStack, expanding in other areas – Engineering benefits from community test and dev – Dell engineering active participants
• Developed in the Open – Most code development is current & live on github – Needed for collaboration – Allows faster response to customers and partners
• Active Community – 300+ email list users & nearly 500 Github followers – 500+ downloads of the Crowbar image – Partners: Intel, Opscode, SUSE, etc – Customers: telecom, cloud, financial, etc
15 Confidential
Revolutionary Cloud & Big Data Solutions
Cloud Taxonomy
Admin Software Web Services & APIs
Customer Management
Entitlement, rights Billing
Self Service Portal
Metering
Infrastructure Software
Abstraction Software
Physical
Software as a Service IT as a Service Platform as a Service Infrastructure as a Service Everything as a Service
HVAC Power Facility
Network
Environmentals
Compute Switch Storage
OS
Hardware Virtualization
Operating System Virtualization
Application Run-Time Virtualization
Data Store
Monitoring
Intelligent Resource Manager
Workload Lifecycle Management
Platform Provisioning
Ser Gov/Workflow Automation
Orchestration
Info
rma
tio
n S
erv
ice
Ma
na
ge
me
nt
Le
ga
cy
Ma
na
ge
me
nt
Fir
ew
all
Re
po
rtin
g
An
aly
tic
s
IPS
S
ec
uri
ty
LD
AP
/AD
S
SO
Overarching Systems
Revolutionary Cloud & Big Data Solutions
Crowbar Builds the Cloud Taxonomy
Admin Software OpenStack APIs
Customer Management Customer Management
Entitlement, rights Entitlement, rights Billing Billing
OpenStack Dashboard
Metering Metering
Infrastructure Software
Abstraction Software
Physical
Software as a Service IT as a Service Platform as a Service Infrastructure as a Service Everything as a Service
BIOS BIOS RAID RAID NETWORK NETWORK
Quantum
Environmentals Environmentals
Nova Switch Switch Swift
OS OS
Hardware Virtualization Hardware Virtualization
Operating System Virtualization
Operating System Virtualization
Application Run-Time Virtualization
Application Run-Time Virtualization
Data Store Data Store
Monitoring Monitoring
Intelligent Resource Manager Intelligent Resource Manager
Workload Lifecycle Management
Workload Lifecycle Management
Ser Gov/Workflow Automation Ser Gov/Workflow Automation
Orchestration Orchestration
Info
rma
tio
n S
erv
ice
Ma
na
ge
me
nt
Info
rma
tio
n S
erv
ice
Ma
na
ge
me
nt
Le
ga
cy
Ma
na
ge
me
nt
Le
ga
cy
Ma
na
ge
me
nt
Fir
ew
all
F
ire
wa
ll
Re
po
rtin
g
Re
po
rtin
g
An
aly
tic
s A
na
lyti
cs
IPS
IP
S
Ke
yst
on
e
LD
AP
/AD
L
DA
P/A
D
SS
O
SS
O
Overarching Systems
Overarching Systems
Nova Glance Swift
OS
KVM
Crowbar
Revolutionary Cloud & Big Data Solutions
Crowbar Builds Up Clouds in layers
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
Core Components & Operating Systems
Core Components & Operating Systems
Cloud Infrastructure Cloud Infrastructure
Physical Resources Physical Resources
APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners
18 Confidential
Revolutionary Cloud & Big Data Solutions
1) Hardware Configuration
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
Core Components & Operating Systems
Core Components & Operating Systems
Cloud Infrastructure Cloud Infrastructure
Physical Resources Physical Resources
APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners
19 Confidential
Revolutionary Cloud & Big Data Solutions
2) OS and Ops Infrastructure
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
Core Components & Operating Systems
Core Components & Operating Systems
Cloud Infrastructure Cloud Infrastructure
Physical Resources Physical Resources
APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners
20 Confidential
Revolutionary Cloud & Big Data Solutions
3) Cloud Platform Integration
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
Core Components & Operating Systems
Core Components & Operating Systems
Cloud Infrastructure Cloud Infrastructure
Physical Resources Physical Resources
APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners
21 Confidential
Revolutionary Cloud & Big Data Solutions
4) Cloud App Management
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
Core Components & Operating Systems
Core Components & Operating Systems
Cloud Infrastructure Cloud Infrastructure
Physical Resources Physical Resources
APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners
22 Confidential
Revolutionary Cloud & Big Data Solutions
Late Binding Matters! It’s “Just-in-time” Deployment Late binding is a critical component of Crowbar flexiblity
Discover
Inventory Provision
Deploy Layer N
Allocate with System View Allocate with System View
Network Config
• Integrated configuration of RAID, BIOS, provision & networking, so no longer have to do that work in advance.
• Do NOT need to make these decisions early in the process
• Can adapt and change over time.
User App & Ops decisions drive the entire config & deploy process!
23 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar Architecture + Demo
24
Revolutionary Cloud & Big Data Solutions
OpenStack Orchestration: Chef & Crowbar
Server(s)
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
Operating System Operating System
Chef Server Chef
Server
Network Configuration Network Configuration
DHCP DHCP
Cro
wb
ar
Ma
gic
Ha
pp
en
s!
Cro
wb
ar
Ma
gic
Ha
pp
en
s!
Orchestration State Machine Orchestration State Machine
Run List
Config.
Hardware Config (uses “Sledgehammer”)
Hardware Config (uses “Sledgehammer”)
OpenStack Components OpenStack Components
25 Confidential
Revolutionary Cloud & Big Data Solutions
Dell Cloud Installer: Crowbar Barclamps D
ell
“C
row
ba
r”
Op
s M
an
ag
em
en
t
Core Components & Operating Systems Core Components & Operating Systems
Cloud Infrastructure & Dell IP Extensions Cloud Infrastructure & Dell IP Extensions
Physical Resources Physical Resources
APIs, User Access, & Ecosystem Partners APIs, User Access, & Ecosystem Partners
* Crowbar has potential to service other programs beyond OpenStack
Crowbar Crowbar
Deployer Deployer
Provisioner
Network RAID
BIOS IPMI
NTP NTP
DNS DNS Logging Logging
Nova Swift Glance
Nagios Nagios Ganglia Ganglia
enStratus Cloudera
Dashboard
Block
Horizon
Keystone Hadoop
26 Confidential
Revolutionary Cloud & Big Data Solutions
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
What is a Barclamp?
Barclamp Barclamp
Chef Recipes Chef Recipes O/S App
Packages O/S App
Packages Crowbar API & Partial UI
Crowbar API & Partial UI
Components & Scripts
Components & Scripts
• Crowbar API & Partial UI • Adds states and transitions to orchestration • Customizes UI specific to barclamp • Visual extensions to nodes/networks/utils
• Chef Recipes • DevOps description of application • Enforced repeatable configuration
• O/A App Packages • Operating system dependencies (Debs & RPMs)
• Components & Scripts • Ad hoc configuration not covered by Chef
27 Confidential
Revolutionary Cloud & Big Data Solutions
De
ll “
Cro
wb
ar”
O
ps
Ma
na
ge
me
nt
What is a Barclamp?
Barclamp Barclamp
Chef Recipes Chef Recipes O/S App
Packages O/S App
Packages Crowbar API & Partial UI
Crowbar API & Partial UI
Components & Scripts
Components & Scripts
Barclamp Barclamp
Barclamp Barclamp
• Interact as service layers • Have dependencies and pre-requisites • Can operate on multiple operating systems • Have independent version streams • Leverage/reuse Chef cookbooks
28 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar’s Proposal Workflow
• Proposals manage deployments • Proposals are a core design objective to ensure customer flexibility
and system level management • Workflow:
• Templates are default configurations without resources • Proposals all users to choose config and nodes (resources) • Applying a proposal tells Crowbar to “make it so”
Deployed Services
Barc
lam
p Active Proposal
Proposal Proposal
Nodes Apply
Active Active Nodes Nodes Apply Apply
Discovered Nodes
29 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar Orchestration = State Machine
30 Confidential
Revolutionary Cloud & Big Data Solutions
Networking in Crowbar
Server Server
IMPI
eth0
eth1
eth#
VLAN
VLAN
VLAN
VLAN
Teamed Teamed
IP Allocations
DHCP
Network Barclamp Network Barclamp
Chef Client Chef Client
Crowbar Does NOT Configure The Switch! O/S Only* not
mgm’d not
mgm’d
Crowbar Discovers Switches and Ports
* For now
31 Confidential
Revolutionary Cloud & Big Data Solutions
Networking in Crowbar
Server Server
IMPI
eth0
eth1
eth#
VLAN
VLAN
VLAN
VLAN
Teamed Teamed
IP Allocations
DHCP
Network Barclamp Network Barclamp
Chef Client Chef Client
not mgm’d
not mgm’d
1g1
1g2
1g3
Conduit
Conduit
Conduit
Other Barclamps Request “Conduits”
Interface & Conduits Normalize Networks Between Nodes
32 Confidential
Revolutionary Cloud & Big Data Solutions
SUSE + Crowbar = A Path to Clouds
33
Revolutionary Cloud & Big Data Solutions
SUSE Cloud with Crowbar
• Version of Crowbar that runs on SUSE
• Packaged to take advantage of SUSE repositories and methods
• Provides OpenStack Essex
• Extends and builds SUSE Studio • Released and available as of the beginning
of September
34 Confidential
Revolutionary Cloud & Big Data Solutions
Team Work
35 Confidential
• A huge key to the success has been team work
• Joint development
• Design and planning discussion
• Not just limited to SUSE, but definite leader
Revolutionary Cloud & Big Data Solutions
Crowbar 2.0 and Beyond
36 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar 2.0 (currently in progress) http://bit.ly/crowbar2 • Simplify use of Chef, eliminate Crowbar requirements in Chef recipes
– Reduce the initial effort required to leverage Crowbar – Opens Crowbar to a broader audience (see Upstreaming)
• Provide heterogeneous/multiple operating system deployments to
enable: – Multiple versions of the same OS running for upgrades – Different OS operating simultaneously (and deal with heterogeneous
packaging issues) – Accommodation of no-agent systems like locked systems (e.g.: virtualization
hosts) and switches (aka external entities) – UEFI booting in Sledgehammer
• Strengthen networking abstractions
– Allow networking configurations to be created dynamically (so that users are not locked into choices made before Crowbar deployment)
– Better manage connected operations – Enable pull-from-source deployments that are ahead of (or forked from) available packages
37 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar 2.0 (currently in progress) http://bit.ly/crowbar2 • Improve Crowbar's core database and state machine to enable
– Larger scale concerns – Controlled production migrations and upgrades
• Packaging rework
– Lead by some of the SUSE work – Create distribution-based packages
• Upgrade components
– Start the process of enabling security updates – Staged Updates from test to production environments – Start considering application
• Other important items
– Make documentation more coupled to current features and easier to maintain
– Upgrade to Rails 3 to simplify code base, security and performance – Deepen automated test coverage and capabilities
38 Confidential
Revolutionary Cloud & Big Data Solutions
Crowbar 2.0 (currently in progress) http://crowbar.sync.in/crowbar2
• Network models (http://youtu.be/eTSTLecNlGM ) -
http://crowbar.sync.in/crowbar2-network
• Online mode (http://youtu.be/U05apbOqq0k ) - http://crowbar.sync.in/crowbar2-online
• Pull from source (http://youtu.be/Bo4V6Dax3LU ) - http://crowbar.sync.in/crowbar2-pullsource
• Heterogeneous OS (http://youtu.be/xL-2eefqH8s) - http://crowbar.sync.in/crowbar2-multi-os
• Attribute driven recipes (http://youtu.be/wtUNJfrHU1Q) - http://crowbar.sync.in/crowbar2-attributeconfig
39 Confidential
Revolutionary Cloud & Big Data Solutions
Q&A
40 Confidential
Revolutionary Cloud & Big Data Solutions
Thanks for Listening!
• Greg Althaus [email protected] Twitter: @galthaus
41 Confidential