open source tool chains for cloud computing

26
Open Source Toolchains for Managing Clouds Monday, August 9, 2010

Upload: mark-hinkle

Post on 11-May-2015

8.149 views

Category:

Technology


0 download

DESCRIPTION

This presentation was given at LinuxCon 2010. The proliferation of cloud computing is inevitable, hosted apps, software-as-as-service and now dynamic on-demand utility computing is becoming the norm. The session will be a “fire-side” chat style discussion of the types of challenges presented by IT management operations personnel and how they can manage cloud infrastructure using open source tools. The talk will discuss options for deploying and integrating tools that provision, configure, orchestrate and monitor cloud (and physical)infrastructure. The session will appeal to those IT professionals (syadmins, net-ops, developers) who develop and manage infrastructure that resides in hosted environments like Amazon EC2 without disregarding traditionally hosted internal infrastructure.

TRANSCRIPT

Page 1: Open Source Tool Chains for Cloud Computing

Open Source Toolchains for Managing Clouds

Monday, August 9, 2010

Page 2: Open Source Tool Chains for Cloud Computing

Mark R. HinkleVP of Community

Zenoss Inc. [email protected]@gmail.comTwitter: @mrhinkle

John M. WillisVP of ServicesOpscode Inc.

[email protected]@gmail.comTwitter: @botchagalupe

ABSENT

Monday, August 9, 2010

Page 3: Open Source Tool Chains for Cloud Computing

%whoami•Former Linux Desktop Advocate

•Former LinuxWorld Magazine Editor-in-Chief

•Open Management Consortium Conspirator

•Open Desktop Consortium Instigator

•Author - “Windows to Linux Business Desktop Migration” - Thomson

•NetDirector Project - Open Source Configuration Management Project

Monday, August 9, 2010

Page 4: Open Source Tool Chains for Cloud Computing

Today’s

Agenda

Definitions (Toolchains, Systems Management, Cloud Computing)

Bad Jokes

Overview of Open Source Management Tools

Culture Changes

Alien Autopsy Photos

Example Cloud Computing Toolchains

Monday, August 9, 2010

Page 5: Open Source Tool Chains for Cloud Computing

ToolchainA set of programs where the output of one program forms the input of another program.

Monday, August 9, 2010

Page 6: Open Source Tool Chains for Cloud Computing

Open Source Management Tools Adoption

• 98%  of  enterprises  use  open  source  systems  management  tools

• 76%  indicate  they  prefer  to  use  open  source  whenever  possible

• Compelling  factors  for  using  open  source  is  flexibility  followed  by  cost  savings

• 50%  are  already  using  some  form  of  cloud  technology  including  but  not  limited  to  hosted  applica@ons,  Amazon  Web  services  and/or  hosted  storage

• Top  IT  management  priori@es  for  2010:  monitoring,  configura4on  management,  patching  and  

Source: 2010 ZenossOpen Source Management Survey

Monday, August 9, 2010

Page 7: Open Source Tool Chains for Cloud Computing

Cloud ComputingHype is Becoming a

Reality•UBS says Web Services will be a $15 billion

+ market by 2014

• IBM says Cloud Computing will be a $126 billion by 2012

• 20% of Businesses won’t have IT Assets by 2012 - Gartner

Bottom line a large portion of our infrastructure no longer lives in our data center... but

we gotta manage it.

Monday, August 9, 2010

Page 8: Open Source Tool Chains for Cloud Computing

Systems Management Disciplines

Sets the parameters for servers, starts and stops services, rotates logs and other menial task

Monitoring queries the servers for overall health and alerts administrators to problems

Configuration Management and Automation

Installation of operating systems and other software

Provisioning

Monitoring

Monday, August 9, 2010

Page 9: Open Source Tool Chains for Cloud Computing

How to Choose Open Source Cloud Tools

• Open Source (OSI Approved License)

• Manage Legacy Infrastructure and Cloud

• Extensible (Plugins, accept code contributions)

• Vibrant Communities (activity in Downloads, Forums, Extensions)

• Client/server (or at least network aware)

Monday, August 9, 2010

Page 10: Open Source Tool Chains for Cloud Computing

Comparison of Provisioning Tools

Year Started Language License Installation

Targets

Cobbler 2007

Fully Automatic Installation (FAI) 2000

Kickstart ?

OpenQRM 2005

Spacewalk 2008

Viper 2008

Python GPL Red Hat, Fedora, OpenSuSE, Debian, Ubuntu

Perl GPL Debian

Python GPL Most .deb and Fedora based Linux

PHP GPL Linux, Solaris, Windows

Perl, Python, Java GPL Fedora, CentOS

Perl GPL Debian

Monday, August 9, 2010

Page 11: Open Source Tool Chains for Cloud Computing

Comparison of Configuration Management & Automation Tools

Year Started Language License Client/Server Backing

AutomateIT 2009

bcfg2 2003

Cfengine 1993

chef 2009

Puppet 2004

Ruby GPL No None

BSD BSD Yes Argonne National Labs?

Apache Apache Yes Cfengine Inc.

Ruby Apache Chef Solo - NoChef Server - Yes Opscode

Ruby GPL Yes PuppetLabs

Monday, August 9, 2010

Page 12: Open Source Tool Chains for Cloud Computing

Comparisons of Open Source Monitoring Tools

Year Started License Language Monitoring

Type Collection Methods

Cacti

Nagios

OpenNMS

Zabbix

2001 GPL PHP Performance SNMP, Syslog

1999 GPL C/PHP Availability SNMP, TCP, ICMP, IPMI Syslog

2000 GPL Java Availability, Performance, SNMP, JMX, HTTP

2001 GPL C/PHPAvailability,

Performance, and more

SNMP, TCP, ICMP, IPMI, Synthetic Transactions

Zenoss 2005 GPL PythonAvailability,

Performance, Event Management

SNMP, SSH, Syslog, Event Management, Synthetic

Transactions

Monday, August 9, 2010

Page 13: Open Source Tool Chains for Cloud Computing

CloudOps: Change in Culture

GapingVoid - http://www.gapingvoidgallery.com/product_info.php?products_id=1643&osCsid=bc3tdqg6fuh8gato04m9obr0o1

Monday, August 9, 2010

Page 14: Open Source Tool Chains for Cloud Computing

Old Systems Management

Practices•Human Powered, Labor Intensive

•Repetitive Tasks

•“Meat Cloud”

•How many servers do you manage per admin?

Monday, August 9, 2010

Page 15: Open Source Tool Chains for Cloud Computing

Cloud Changes Everything • Hyperscalable

• Hardware Abstraction

• Dynamic Infrastructure

• Geography Independent

• Fast & Flexible

• How many cloud instances do you manage per admin?

Monday, August 9, 2010

Page 16: Open Source Tool Chains for Cloud Computing

The Myth of the NinesAvailability % Downtime per Year Downtime per Month Downtime per Week

99.9% (three nines) 8.76 hours 43.2 minutes 10.1 minutes

99.95% 4.38 hours 21.56 minutes 5.04 minutes

99.99% (four nines) 52.6 minutes 4.32 minutes 1.01 minutes

99.999% (five nines) 5.26 minutes 25.9 seconds 6.05 minutes

99.9999% (six nines) 31.5 seconds 2.59 seconds .0605 seconds

• Average polling interval for monitoring? 5 minutes? • Even super human operations people can’t be alerted and take action in under 5 minutes. • One outage per year could drop service level to three nines or worse.

Monday, August 9, 2010

Page 17: Open Source Tool Chains for Cloud Computing

Copyright © 2010 Opscode, Inc - All Rights Reserved

P

Automate or get run over.

17Monday, August 9, 2010

Page 18: Open Source Tool Chains for Cloud Computing

Cultural ChangesAgile IT and DevOps movements

• Operations and Developers should collaborate with each other to deliver excellent products

• Systems Administrators need to be come Systems Engineers building automated, fault tolerant systems not just maintaining infrastructure

• More frequent changes, more outages (albeit short) to rapidly improve IT products and services

• Process, version control, and automation are important

Monday, August 9, 2010

Page 19: Open Source Tool Chains for Cloud Computing

AutomateITChef

Control TierPuppet

KickstartSpacewalk

CobblerOpenQRM

Configuration Management and

Automation

GangliaNagios

OpenNMSZabbix

Zenoss Core

MonitoringProvisioning

Systems Management Tools for Cloud Computing

Monday, August 9, 2010

Page 20: Open Source Tool Chains for Cloud Computing

“Off the shelf” Open Source Toolchains

• OpenQRM and Nagios

• Cobbler and Puppet

• Zenoss & [Chef, Cfengine and Puppet]

• Spacewalk and Cobbler

• OpenNMS and Rancid

• OpenNMS and Puppet

Monday, August 9, 2010

Page 21: Open Source Tool Chains for Cloud Computing

DevOps ToolChain Project

Project centered around how to automate and improve infrastructure management using Agile/DevOps methodologies

Discussion Topics • Open questions on unified pipe architecture

• Distribution methods: package vs file, rscyn/murder vs yum/rpm vs DFS

• Configuration management: RPMs vs puppet/cfengine/chef tool?

• Rollback methodologies for package and config management tools

• Controlling and timing package release and config management tools

• Log management (aggregating, crunching, charting)Change detection

http://code.google.com/p/devops-toolchain/

Monday, August 9, 2010

Page 22: Open Source Tool Chains for Cloud Computing

Cloud Computing

Changes Everything

•MeatCloud, Can’t Keep up with Cloud Computing

•Devops & Agile IT Philosophy

•Script Repetitive Tasks

•Automate, Automate, Automate

Monday, August 9, 2010

Page 23: Open Source Tool Chains for Cloud Computing

Example Cloud ToolChain•Multiple Cloud Providers

•Mix and Match and match tools

•Portability, Flexibility, and

Monday, August 9, 2010

Page 24: Open Source Tool Chains for Cloud Computing

Example - Geeknet• Servers are automatically built using configuration

management software

• Discovery tool finds infrastructure and populates a CMDB then spits out information to scripts that translate information to BIND configurations for DNS

• Monitoring tool adds hosts to polling tool to check servers for availability

• As infrastructure changes systems are updated automatically

• Servers can be spun up and managed in minutes, not hours automatically with little or no human interaction

Hundreds of servers, serving web, databases, and other infrastructure for some of the world’s most highly trafficked websites – over 40 million visitors per month.

Monday, August 9, 2010

Page 25: Open Source Tool Chains for Cloud Computing

Summary

• Automate to improve service, apply leverage

• Rethink how your operations works

• Choose tools that can extend and adapt to new types of infrastructure (what does cloud look like in 2015?)

Monday, August 9, 2010

Page 26: Open Source Tool Chains for Cloud Computing

Questions? Comments?

Monday, August 9, 2010