puppet camp berlin 2014: advanced puppet design

Advanced Puppet Design

Craig Dunn, Puppet Camp Berlin 2014

Friday, 11 April 14

Hello

• Craig Dunn

• Puppet user since 2008

• Previously worked for Puppet Labs

• Founder of Enviatics

• IT automation engineer and trainer

• Based in Spain but work anywhere

Friday, 11 April 14

My talk

• Avoiding pain

• Writing good modules

• Challenges of codebase design

• Roles / Profiles

Friday, 11 April 14

But first....Everyone loves polls, right?

Friday, 11 April 14

Thinking right

• Business has requirements

• We use technology to fulfill them

• But it’s not that simple!

Friday, 11 April 14

“Business logic does not oftenalign with technology design”

Friday, 11 April 14

Business requirement

“We have 3 applications we

need to deploy using Puppet”

Friday, 11 April 14

Puppetize

• Write 3 Puppet modules for 3 applications

• Because that was the requirement

• Was it?

Friday, 11 April 14

It’s not working

Friday, 11 April 14

Let’s suppose

• Each “application” is a set of shared components implemented different ways

Friday, 11 April 14

The business view

Application Y Application Z

Application X

Friday, 11 April 14

The technical reality

Application YApplication Z

Application X

Friday, 11 April 14

Danger Signs

• Resources being declared in two modules

• You don’t know where your implementation “fits”

• Lot’s of logic at a node level

• Repetition and duplication

• The if statement is your go-to-guy

Friday, 11 April 14

Catastrophic Signsif ($::hostname =~ /^host[0-3].*/) { package { ‘my-special-app’: ensure => installed, }}

Friday, 11 April 14

Catastrophic Signsif !defined(Package[‘httpd’]) { package { ‘httpd’: ensure => installed, }}

Friday, 11 April 14

Catastrophic Signsensure_resource(‘package’,‘httpd’,{‘ensure’ => ‘installed’})

Friday, 11 April 14

Catastrophic Signsensure_resource(‘package’,‘httpd’,{‘ensure’ => ‘installed’})

if function_defined_with_params(["#{type}[#{item}]", params])      Puppet.debug("Resource #{type}[#{item}] not created because it already exists")    else      Puppet::Parser::Functions.function(:create_resources)      function_create_resources([type.capitalize, { item => params }])    end

Friday, 11 April 14

World ending signs

• You use parser=future in production

• You aren’t regretting it yet

• You then implemented order=manifest

Friday, 11 April 14

An unhappy Puppet

Friday, 11 April 14

Stop thinking about what it looks like

• Break everything down into components

• Granularity is the key

• Think about what it actually is

Friday, 11 April 14

Writing modules

• Granularity

• Portability

• Flexibility of implementation

Friday, 11 April 14

A bad moduleclass web {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

package { ‘php5’: ensure => installed, require => Package[‘httpd’], }

package { ‘mysql-server’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Friday, 11 April 14

A bad moduleclass web {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

package { ‘php5’: ensure => installed, require => Package[‘httpd’], }

package { ‘mysql-server’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Granularity

Friday, 11 April 14

A bad moduleclass web {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

package { ‘php5’: ensure => installed, require => Package[‘httpd’], }

package { ‘mysql-server’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Portability

Granularity

Friday, 11 April 14

A bad moduleclass web {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

package { ‘php5’: ensure => installed, require => Package[‘httpd’], }

package { ‘mysql-server’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Portability

Implementation

Granularity

Friday, 11 April 14

Keep your modulesgranular

• Manage only resources in the scope of the module

• Small modules are ok!

Friday, 11 April 14

A granular moduleclass apache {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Friday, 11 April 14

A granular moduleclass apache {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Granularity

Friday, 11 April 14

A granular moduleclass apache {

$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }

file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}

Granularity

Portability

Friday, 11 April 14

Sharing is good

• Re-usable by others in your team

• You can publish to the forge!

• People will collaborate

Friday, 11 April 14

It’s all about sharing!

Friday, 11 April 14

You are gonna shareyour s**t aren’t you?

Friday, 11 April 14

Sharing is not justpull requests

• Share your ideas in blog posts

• What worked? What didn’t?

• Discuss and collaborate on mailing lists

• IRC

Friday, 11 April 14

Making sharing easier

• Data separation (Hiera)

• Allow the user flexibility of implementation

Friday, 11 April 14

defaults params pattern

• Use a parameterized class

• Default from an inherited class

• Allow the user to decide implementation

Friday, 11 April 14

defaults patternclass apache { $packagename=‘httpd’ $docroot=’/var/www’ $listenaddr=‘10.0.1.12’ $servername=‘myweb.foo.com’

package { $packagename: ensure => installed, }

.....

Friday, 11 April 14

defaults patternclass apache { $packagename=‘httpd’ $docroot=’/var/www’ $listenaddr=‘10.0.1.12’ $servername=‘myweb.foo.com’

package { $packagename: ensure => installed, }

.....

In-module private data

No way to override

Friday, 11 April 14

defaults patternclass apache ( $packagename=‘httpd’, $docroot=’/var/www’, $listenaddr, $servername,) {

package { $packagename: ensure => installed, }

.....

Friday, 11 April 14

defaults pattern

class { ‘apache’: listenaddr => ‘10.0.1.2’, servername => ‘foo.example.com’,}

Override data on implementation

Friday, 11 April 14

defaults pattern

# /etc/puppet/hieradata/dev.yaml

---apache::docroot: /var/dev/sites

Overriding from Hiera

Friday, 11 April 14

defaults patternclass apache::defaults {

$packagename=$::osfamily ? { redhat => ‘httpd’, debian => ‘apache2’, default => ‘httpd’, } $docroot=’/var/www’ $listenaddr = $::ipaddress $servername = $::fqdn}.....

Friday, 11 April 14

defaults patternclass apache::defaults {

$packagename=$::osfamily ? { redhat => ‘httpd’, debian => ‘apache2’, default => ‘httpd’, } $docroot=’/var/www’ $listenaddr = $::ipaddress $servername = $::fqdn}.....

Added logic todefaults

Common defaults

Friday, 11 April 14

defaults pattern

class apache ( $packagename=$::apache::defaults::packagename, $docroot=$::apache::defaults::docroot, $listenaddr=$::apache::defaults::listenaddr, $servername=$::apache::defaults::servername,) inherits apache::defaults {

package { $::apache::packagename: ensure => installed, }

.....

Friday, 11 April 14

defaults pattern

Granularity

Portability

Implementation

Smaller modules with smaller scope

No hard coded data in modules

User can decided how / where to override defaults without editing the module

Friday, 11 April 14

Component Modules

Friday, 11 April 14

Now to build something awesome

... but where?

Friday, 11 April 14

Designing Puppet

Component Modules

Node Classification

Friday, 11 April 14

Node-level logicnode basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’]}

Friday, 11 April 14

Node-level logic

node basil inherits base { class { ‘apache’: version => ‘latest’, }}

Friday, 11 April 14

Node-level logic

• Risks duplication and repetition

• No guarantee of consistency

• Pseudo nodes and inheritance trees will get messy, fast.

• TMI!

Friday, 11 April 14

Thinking beyond the module....

• Puppet is a code base

• You need an effective framework

• Gluing everything together

Friday, 11 April 14

Introducing Profiles

Friday, 11 April 14

Profiles

• Wrapper classes implementing component modules

• Define a logical technology stack

• But just one!

Friday, 11 April 14

Profilesclass profile::blog { User <| group == ‘webadmins’ |> class { ‘::mysql::server’: } class { ‘::mysql::bindings’: php_enable => true, }

class { ‘::wordpress’: install_dir => ‘/var/www/wp’, }}

Friday, 11 April 14

Profiles

• Component modules manage the resources

• Profiles provide a layer of implementation

Friday, 11 April 14

Profiles and Components

Resources

Friday, 11 April 14

Profiles and Components

Resources

Components: Resource modelling

Friday, 11 April 14

Profiles and Components

Resources

Components: Resource modelling

Profiles : Implementation

Friday, 11 April 14

Lessons learned

• Granularity is good

• Don’t assume business logic will directly translate to technology

• Abstraction is awesome.... but that’s nothing new....

Friday, 11 April 14

Abstraction is a core principle of coding

• Implementation is abstracted by methods

• Methods abstracted by classes and modules

• They are abstracted with libraries

• Puppet is code!

Friday, 11 April 14

Puppet is all about abstraction

• Providers are abstracted by types

• Resources are abstracted by classes

• Classes are abstracted by modules

Friday, 11 April 14

Puppet is all about abstraction

• Providers are abstracted by types

• Resources are abstracted by classes

• Classes are abstracted by modules

• Modules are abstracted by profiles

Friday, 11 April 14

Focussing on Abstraction

• We’ve turned business logic into a technology stack

• Can we translate that back into business logic?

• Why would we even want to do that?

Friday, 11 April 14

Introducing roles

• Translate to business logic

• Identify the function of a server in human terms

• We never said business logic was a bad thing

Friday, 11 April 14

Configuration model

include profiles::securityinclude profiles::usersinclude profiles::networkinginclude profiles::blog

This is a “acme” server

Friday, 11 April 14

Think about the usersMeet John, Susan and Bill.

Friday, 11 April 14

John is a Sysadmin

• Wants to ensure all servers have kernel hardening, NTP and SSH Server installed

• Wants to manage what packages, services, files and other resources

• Is responsible for maintaining all the components of a any type of server

Friday, 11 April 14

Susan is an application specialist

• Cares that a the node has Wordpress and MySQL implemented properly

• She probably doesn’t care about how sudoers is configured

Friday, 11 April 14

Bill is an IT manager

• Bill cares that the server is an ACME App server

• He probably doesn’t understand what sudoers is

Friday, 11 April 14

What do they care about?

• John cares about modelling all resources

• Susan cares about the technology stack

• Bill cares about the business logic

Friday, 11 April 14

In Puppet

• Resource modelling is done in component modules

• The technology stack is defined in profiles

• Where do we represent the business logic for Bill?

Friday, 11 April 14

Roles

• Represent business logic, not technology

• Define a set of technology stacks (profiles) that make up the logical role

• Allow the business to manage how the infrastructure looks without defining what it is

Friday, 11 April 14

A node can only have one role

• A role can include as many profiles as required to define itself

• If a node requires two roles, it has by definition become a new role

Friday, 11 April 14

A node can only have one role

• A role can include as many profiles as required to define itself

• If a node requires two roles, it has by definition become a new role

• Something couldn’t be a lion and a kangaroo at the same time!

Friday, 11 April 14

It would be a Lingaroo

Friday, 11 April 14

Roles

• One-to-one to nodes

• One-to-many to profiles

• Only implement profiles, that’s it!

Friday, 11 April 14

The Stack

Resources

Friday, 11 April 14

The Stack

Resources

Components: Resource modelling

Friday, 11 April 14

The Stack

Resources

Components: Resource modelling

Profiles : Implementation

Friday, 11 April 14

The Stack

Resources

Components: Resource modelling

Profiles : Implementation

Roles : Business Logic

Friday, 11 April 14

Role classes

class role::acme { include profiles::security include profiles::users include profiles::networking include profiles::blog}

This is a “acme” server

Friday, 11 April 14

Terminology

• Profiles and Roles are Puppet modules

• They are not special

• Everything is a module

Friday, 11 April 14

Classification

• Assigning classes to a node

• You can classify within Puppet code (site.pp)

• You can use an External Node Classifier (ENC)

Friday, 11 April 14

Classification

• You can classify your nodes however you want

• Puppet Dashboard

• Enterprise Console

• Foreman

• Site.pp

• Custom script

Friday, 11 April 14

Classification

node ‘craig.puppetlabs.vm’ { include roles::acme_app}

Friday, 11 April 14

Classification

Friday, 11 April 14

Classification

• With roles and profiles we just classify the role to the node

Friday, 11 April 14

The Stack

Resources

Components: Resource modelling

Profiles : Implementation

Roles : Business Logic

Friday, 11 April 14

The Stack

Resources

Components: Resource modelling

Profiles : Implementation

Roles : Business Logic Classifier

Friday, 11 April 14

Data Separation

If you’re not using Hiera you are goingto break your data!

Friday, 11 April 14

Data Separation

• Use parameterized classes

• Hiera data bindings

• Component modules and profiles can look up data from Hiera

• Roles should NOT

Friday, 11 April 14

Roles and Profilesfor DevOps

• Full props to Laurent Bernaille from D2SI

• Achieving Continuous Delivery and Devops with Puppet

• Puppet Camp Paris, 2014.

Friday, 11 April 14

Roles and Profilesfor DevOps

• Using roles and profiles makes it easier for developers and ops to all collaborate on Puppet

• Developers write profiles for the their apps

• Ops write profiles for their infrastructure

• Roles encompass all of them

Friday, 11 April 14

The Roles/ProfilesStack

Resources

Components: Resource modelling

Roles : Business Logic

Hiera:Data

Classifier Classification

Dev profiles Ops profiles

Friday, 11 April 14

Roles/Profiles/Devopshttp://fr.slideshare.net/D2SI/d2-si-puppetcamp

Friday, 11 April 14

Key benefits

• Reduced node-level logic to a role.

• Gain the ability to be flexible with implementation

• Business logic improves managability by non-Puppet users

• Edge cases are now easy to solve

Friday, 11 April 14

Enough Preaching!

Friday, 11 April 14

This is not the way to design Puppet... It’s a

way.

Friday, 11 April 14

Can I implement this design without roles?

Friday, 11 April 14

Can I implement this design without roles?

• Yes.

• You lose the layer of abstraction that exposes business logic

Friday, 11 April 14

Can my roles be defined in my ENC?

Friday, 11 April 14

Can my roles be defined in my ENC?

• Yes.

• Keeping it in code makes it versionable

Friday, 11 April 14

Can’t I just use Hiera to define profiles?

Friday, 11 April 14

Can’t I just use Hiera to define profiles?

• Technically yes.

• You lose the flexibility to implement code logic in profiles and it may become restrictive

Friday, 11 April 14

The fundamental concepts....

Friday, 11 April 14

The fundamental concepts....

• Abstraction, abstraction, abstraction

Friday, 11 April 14

The fundamental concepts....

• Abstraction, abstraction, abstraction

• Decoupling business logic, implementation and resource modelling.

Friday, 11 April 14

The fundamental concepts....

• Abstraction, abstraction, abstraction

• Decoupling business logic, implementation and resource modelling.

• Separating data and code

Friday, 11 April 14

The fundamental concepts....

• Abstraction, abstraction, abstraction

• Decoupling business logic, implementation and resource modelling.

• Separating data and code

• Reducing node-level complexity

Friday, 11 April 14

Other resources

• http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

• http://www.craigdunn.org/2012/05/239/

Friday, 11 April 14

Danke. Questions?

• Follow me at @crayfishX

• Bug me on Freenode: crayfishx

In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beerand helped me on my journey of discovery learning how to implement Puppet properly. R.I.P

Friday, 11 April 14