a presentation about puppet that i've made at the osspac conference

Infrastructure Automation

Ohad Levy – Infineon [email protected]

Automated

”…a process which may once have been performed manually but has been altered in some way which allows a machine or computer to either wholly or partially manipulate the process to save time”

Infrastructure

“Infrastructure is generally a set of interconnected structural elements that provide the framework supporting an entire structure”

Automated Infrastructure

“Having the basic services necessary for your infrastructure to operate largely without the aid of a keeper.”

Grow – from This:

http://i68.photobucket.com/albums/i2/RealConnections/googlecomputers.jpg

Large Scale Environment

Where does it fit in?

Typical System Lifecycle

Pre/ Installation

DNS/DHCP…

Initial Configuration

KickstartJumpstart

Custom Scripts

FixesUpdatesAudits

System configuration possibilities Manual System Configuration

„log in and „just do it!““¬ + OK for small number of servers¬ + Quick and easy

¬ - Very difficult to ensure similar configurations¬ - No auditing capability¬ - No change history¬ - No release documentation¬ - No way to deploy multiple servers quickly.¬ - No way to rebuild critical servers quickly.

System configuration possibilities Install Time Auto-Configuration

„Deploy using Kickstart, Jumpstart etc“¬ + Ensures that new systems are brought into proper state as part

of installation process ¬ + Possible to deploy multiple servers quickly

¬ - No Validation that settings remain correct¬ - No easy way to deploy changes¬ - Auto-Configuration process is different between the sites leads

to different configurations ¬ - No Auditing¬ - No change history¬ - No release documentation

Typical System Lifecycle

Pre/ Installation

DNS/DHCP…

Initial Configuration

KickstartJumpstart

Custom Scripts

FixesUpdatesAudits

Puppet What is Puppet ?

A product designed to deploy system configurations. It is – ¬ Open source based on Ruby¬ Policy based¬ Runs every 30 minutes¬ An abstraction layer between the system administrator and the

system¬ Capable to run on any UNIX operating system

Our Challenges

Keep our systems "harmonized“ Audit / Know what's going on each system Replace a server if it dies or to be able to add another server

that is exactly like it Similar Applications, different OS's Push out changes to all the servers that need a particular change Stop duplicating effort Go home early

How does Puppet works?

Puppet Types

A Type is a particular element that Puppet knows how to configure

Files (content, permissions, ownership) Packages (installations, updates..) Services (enabled/disabled, running/stopped) Exec (run commands) Full List: cron, exec, file, filebucket, group, host, interface,

k5login, mailalias, maillist, mount, nagios*, package, service, sshkey, tidy, user, yumrepo, zone

Example: Managing sudoers file

file { /etc/sudoers: owner => root, group => root, mode => 600, source => puppet://server/files/sudoer

}

Dependencies

"require" and "before" / "after" settings ensures that types are applied in the correct orderfile { "/etc/sudoers": ... require => Package[sudo]}package { "sudo": ensure => present, before => File["/etc/sudoers"]}

Dependencies - continued "notify" and "subscribe" settings can trigger cascaded updates

Particularly useful in services, exec

file { "/etc/ssh/sshd_conf": ... notify => Service["sshd"]

}service { "sshd": subscribe => File["/etc/ssh/sshd_conf“]}

What is Facter?

Facter gathers information about the client, which can be used as variables within puppet.

You can add custom facts as needed.package {"sshd":

ensure => installed,name => $operatingsystem ? {

solaris => "IFKLssh",default => “openssh-server”

}}

Example Facts$ sudo facterarchitecture => amd64domain => sin.infineon.comfacterversion => 1.3.8fqdn => sinn1636.sin.infineon.comhardwaremodel => x86_64hostname => sinn1636ipaddress => 172.20.89.113kernel => Linuxkernelrelease => 2.6.24-21-genericlsbdistcodename => hardylsbdistdescription => Ubuntu 8.04.1lsbdistid => Ubuntulsbdistrelease => 8.04macaddress => 00:1c:25:14:26:abmanufacturer => LENOVOmax speed => 2000 MHzmaximum memory module size => 4096 MB

maximum total memory size => 8192 MBmemoryfree => 988.30 MBmemorysize => 1.94 GBoperatingsystem => Debianoperatingsystemrelease => 2.6.24-21-genericprocessor0 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHzprocessor1 => Intel(R) Core(TM)2 Duo CPU T7300 @ 2.00GHzprocessorcount => 2productname => 6458B43ps => ps -efpuppetversion => 0.24.5rubysitedir => /usr/local/lib/site_ruby/1.8rubyversion => 1.8.6serialnumber => L3A8632swapfree => 912.23 MBswapsize => 956.99 MBtype => Notebookuniqueid => 007f0101vendor => LENOVO

What is a Class? A named collection of type objects Can include or inherit from other classesclass sudo_class { include foo_class file { "/etc/sudoers": ... } package{ "sudo": ... }}

Class inheritance

class afile { file { "/tmp/foo": ensure => file source => "/src/versionA" }}class another_file inherits afile { File["/tmp/foo"] { source => "/src/versionB" }}

What is a Node ? A configuration block matching a client Can contain types, classes "default" node matches any client without a node block

node "ohad.myself" { include sudo_class include other_class}

External Nodes Node definitions can be defined outside of puppet – e.g. LDAP,

external script… Ideal when you have too many nodes..

Classes and definitions Classes are groups of resources. Definitions are similar to classes, but they can be instantiated multiple

times with different arguments on the same node.

class apache2 {define simple-vhost ( $admin = "webmaster", $aliases, $docroot) {

file { "/etc/apache2/sites-available/$name":mode => "644",require => [ Package["apache2"], Service["apache2"] ],content => template("apache/vhost.conf"),

} }

}

Classes and definitions - Continued

node mywebserver {include apache2

apache2::simple-vhost { "test.example.com": docroot => "/var/www/test“

}

apache2::simple-vhost { "debian.example.com": docroot =>"/var/www/debian“,alias => [“debiantest.example.com”,”debian”],admin => “[email protected]”

}}

vhost.conf template Puppet uses Ruby's ERB template system:<VirtualHost *> ServerAdmin <%= admin %> DocumentRoot <%= docroot %> ServerName <%= name %><% aliases.each do |al| -%> ServerAlias <%= al %><% end -%> ErrorLog "|/usr/bin/cronolog /var/log/apache/<%= name %>/%Y-%m/error-%d" CustomLog "|/usr/bin/cronolog /var/log/apache/<%= name %>/%Y-%m/access %d" sane</VirtualHost>

Template output

more /etc/apache2/sites-available/debian.example.com<VirtualHost *> ServerAdmin [email protected] DocumentRoot /var/www/debian ServerName debian.example.com ServerAlias debiantest.example.com ServerAlias debian ErrorLog |/usr/bin/cronolog /var/log/apache/debian.example.com/%Y-%m/error-%d“ CustomLog "|/usr/bin/cronolog /var/log/apache/debian.example.com/%Y-%m/access-%d“ sane

</VirtualHost>

OS API - It also works theother way around:

$ ralsh user levyouser { 'levyo': groups => ['tidc'], ensure => 'present', password => 'x', shell => '/bin/bash', uid => '49960', comment => 'Ohad Levy, +65 xxxx xxxx', gid => '11100', home => '/home/levyo'}

Puppet Modules Modules allow you to group both the logic and the files for an

application together. Puppet automatically searches its module path to find modules. Modules can contain four types of files, each of which must be

stored in a separate subdirectory: Manifests – must be stored in manifests/, and if you create

manifests/init.pp then that file will be loaded if you import that moudle name directly (e.g. import “mymodule”)

Templates – must be stored under templates/, put in your manifest some where template(“mymodule/mytemplate”)

Files – stored in files/, can be accessed as: source => puppet://”mymodule”/myfile

Plugins – Additional types, provides or facts

File server and File Bucket Puppet also includes a file server which you can use for

transferring files from the server to the client. If you configure it, puppet can also save a backup of each file

that is changed on the client to the server. The backups go in a filebucket and can be retrieved later.

Getting Started Install puppet (yum,pkg-get,…) or manually install ruby, facter

and puppet Setup a puppet server (puppetmaster / puppet-server). Write a manifest for your node (classes etc) Start puppet master on the server Start puppetd on the client

Some more info Puppet uses SSL for all communications, therefor it includes a

CA, you can automatically sign CSR or use puppetca tool to mange them.

Storeconfig, option to save machine states(facts, configuration runs) and special facts (e.g. SSH keys) in a database.

Reporting, puppet has a few reporting options, most common are emails with changes, RRD files, yaml files and puppetshow web interface.

Puppet HA, load balancing etc

What are the benefits ? Costs

Based on open source tools – No license cost Reduced Labor Reduced Cost

Productivity Reproducibility, Accuracy and homogenous environment (Reduce needed knowledge to operate our environment) Allow easy ramp up of new sites and services

Change Management Testing Processes, Auditing Global control over servers Harmonization Consolidation

…

Conclusions We're all stuck on the hamster wheel Makes easy stuff easier, hard stuff possible Harmonize! (not just today, also in the future) Enable the business for consolidation Similar projects

cfengine bcfg2

Additional Resources http://reductivelabs.com/trac/puppet http://reductivelabs.com/trac/puppet/wiki/LanguageTutorial http://reductivelabs.com/trac/puppet/wiki/CompleteConfiguration

#puppet on irc.freenode.org

Backup SlidesAdditional information

Configuration DB - Terminology Module – A bundle of settings related to a single applicationAt Infineon we use 3 types of modules:

Host Type - e.g. Login Servers, Compute nodes… Service - module which is needed by other modules, e.g. ssh, x11 etc. Sites Modules - sites customizations

Versioning – The ability to refer (tag) to a certain state/PiT Host Type and Service modules are associated with a version, e.g.

Login Server v1.0 Site Modules are not tagged in order to reduce complexity

* Every change in a module, requires a check in action, which is identified by a user, and a comment (change log)

* A user access restriction should be applied to each module and to the tagging mechanism

Configuration DB - Terminology Environments – The ability to select a group of modules in a

certain versionEach site is associated with at least two environments – Production

and Testing: Testing – an environment which refer to versioned modules

under a test tag. Production – A selection of modules in a certain version that

has already been approved for Production usage. Development environment – The state where the modules are in

the latest check-in version, but are not tagged (e.g. not a testing candidate yet).

Development Testing Production (Per Host Type)

Configuration DB - Terminology Global modules / owners

Each owner is responsible for his module (or modules) A special attentions is required for Service module as many

other modules might depend on them

Site modules As Site modules are not tagged (versioned) any change in the

site specific settings, will be rolled out immediately.

Risks & Challenges If the right processes are not in place, it would be possible to

implement the wrong changes globally Learning Curve

How our platform works How to use version control How to work in the new processes/workflows

Fear (Am I obsolete?, will I continue to learn?)

Why Did we choose Puppet Open source More flexible than other open source alternatives Really works (Declarative language, Abstracts configuration as

resources, Allows relationships, Transactional, Modular) Provides an API to the OS, making it very simple to support other

OS’s in the future Puppet is now being adopted in many projects For more info see

http://reductivelabs.com/trac/puppet/wiki/CfengineVsPuppethttp://reductivelabs.com/trac/puppet/wiki/WhosUsingPuppet

http://reductivelabs.com/trac/puppet/wiki/CfengineVsPuppet

http://reductivelabs.com/trac/puppet/wiki/WhosUsingPuppet

Datacenters Challenges Harmonization Simplify Servers management Implement and maintain Global standards Accurate Configuration Database Rapid recovery and deployment of services globally Eliminates duplicate / unnecessary effort Same look and feel everywhere (enables consolidation / global

operation) Audits / Compliance

Infineon Puppet Infrastructure

Configuration Database(Puppeteer, SVN, Netapp)

configuration Server

Site A

deployment &monitoring

Managed Servers

Failover

Site B

deployment &monitoring

Managed Servers

configuration Server

• Puppeteer (Puppet Master of site PM’s) • Puppet Master per site (or based on size/load)• SQL Database for Web portal, Inventory• Configuration database (Version control & Ticketing system)

Gini’s Features AD/LDAP Authentication DHCP/DNS Connectors (API, + MS(yacks) DHCP, MS(yacks) DNS and infoblox) Puppet Host type management Disk Layout Management Host Hardware management (needed for Solaris DHCP jumpstart, and general host info) Multi OS management (e.g. RH, CentOs, Solaris.... in their various versions) User role management (who is allowed to install, in which site, which host type etc) Subnet and IP Management (auto assignment of IP address, DHCP reservation DNS records etc) Multi Domain / site management Hands-free auto OS installation (currently Kickstart and Jumpstart (sparc) using DHCP) Management of Puppet Environments, choose an env per host (e.g. we have different production states for

different sites, or for different host types). Management of puppet certificates (also support chained certificates). Import of Machine Facts (we cant use store config due to the large setup of our env) Nice Graphs (with Gruff) Puppet modules selection per host(for those who needs more than the default host type) SP (Service Processor, iLOM etc) management TFTP management Status indications for machine state (pre installation, during puppet first run, etc) ... and probably a lot of other small stuff that I cant think of now

Our Solution

Web Portal

Puppet

Configuration DBInventory

DHCP/DNS..

Where does GINI live?

GINI

DBDNS

TFTP

CERTIFICATES

CMDB

DHCP

LOGS LSF

Future work

a presentation about puppet that i've made at the osspac conference

Documents