designing puppet: roles/profiles pattern

Designing Puppet:

Roles / Profiles Design Pattern

Puppet Camp Stockholm, Feb 2013

Thursday, 7 February 13

Hello

• Craig Dunn

• Puppet user since 2008 as an IT contractor

• Started with 0.24

• Joined Puppet Labs in June 2012

• @crayfishX

• Freenode IRC: crayfishx


Agenda

• How people typically design Puppet

• Real-world case study

• Thinking about components

• Designing Puppet for your users

• Node classification

• Data separation


Background

• Originally a blog post written in May 2012

• Advocated by many Puppet Labs Engineers

• Based on a real world solution

• Several community members have adopted with success


Designing Puppet

• You write awesome modules

• You classify them to your node


Designing Puppet

Modules

Node Classification


Down the road...

• Your infrastructure grows

• Business requirements will change

• Your Puppet code feels bulky and high maintenence

• There will always be edge cases eventually

• You decide it needs refactoring


Danger Signs

• Resources being declared in two modules

• You don’t know where your implementation “fits”

• Lot’s of logic at a node level

• Repetition and duplication

• The if statement is your go-to-guy


Write good modules

• Should manage only it’s own resources

• Should be granular

• Should be portable


Thinking beyond the module....

• Puppet is a code base

• How do I design an effective framework

• Gluing everything together


Node-level logic

• Risks duplication and repetition

• No guarantee of consistency

• TMI!


Node-level logicnode basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’]}


Node-level logic

• What happens when I have 1000 nodes

• Or 10,000 nodes!!

• That’s a lot of code!

• So where should implement this?


Designing Puppet

• Provide business logic to classification

• Provide an abstraction layer for implementation of components

• Make code adaptable to complex requirements

• Reduce node-level logic

• Reduce functionality overlap


What is the worse thing that is

going to happen to your Puppet code?


Business requirements


Business logic does not align with technology


Case study

• Real world problem

• Solved through design


Case study

“We have 3 applications we

need to deploy using Puppet”


The business view

Application Y Application Z

Application X


Go forth and Puppetize!


Go forth and Puppetize!

And we jumped right in...


Things got painful


Problems

• These applications aren’t that different

• They seem to share a whole bunch of similarities

• Implementation differed on different environments and locations

• Writing 3 separate modules creates conflicts and duplication


Our code was hacky


We are trying to code business logic.


Stop thinking about what it looks like

• Break everything down into components

• Granularity is the key

• Think about what it actually is


What we realised

• Each application stack is a collection of a subset of the same Java apps implemented in different ways


The business view

Application Y Application Z

Application X


The technical reality

Application YApplication Z

Application X


We only have one application

Implemented many different ways


So we had an idea!

• Reduce each Java sub application into granular Puppet modules

• Create a code layer responsible for implementation

• Let’s call them profiles


class profiles::x { include tomcat include mysql include componenta include componentb componentb::resource { ‘name’: ensure => present, }}

class profiles::y { include tomcat include mysql include componenta include componentc include componentd}

class profiles::z { include tomcat include mysql include componenta include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’]}


class profiles::application { include tomcat include mysql include componenta}

class profiles::application::x inherits profiles::application { include componentb componentb::resource { ‘name’: ensure => present, }}

class profiles::application::y inherits profiles::application { include componentc include componentd}

class profiles::application::z inherits profiles::application { include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’]}

Use inheritance for abstraction within profiles


Profiles and Components

Resources



Resources

Components: Resource modelling



Resources


Profiles : Implementation


In reality it was worse



• 2 different deployment types made up of over 15 server types each




• 10+ locations




• 10+ locations

• 4 environment types




• 10+ locations


• Every installation was an edge case!




• 10+ locations


• Every installation was an edge case!

• My slides weren’t big enough.


Lessons learned

• Granularity is good

• Don’t assume business logic will directly translate to technology

• Abstraction is awesome.... but that’s nothing new....


Abstraction is a core principle of coding

• Functions are abstracted by methods

• Methods abstracted by classes and modules

• They are abstracted with libraries

• Puppet is code!


Puppet is all about abstraction

• Data is abstracted by Hiera

• Providers are abstracted by types

• Resources are abstracted by classes

• Classes are abstracted by modules


Puppet is all about abstraction

• Data is abstracted by Hiera

• Providers are abstracted by types

• Resources are abstracted by classes

• Classes are abstracted by modules

• Modules are abstracted by profiles


Focussing on Abstraction

• We’ve turned business logic into a technology stack

• Can we translate that back into business logic?

• Why would we even want to do that?


UAT Cluster node

include securityinclude usersinclude ntpinclude ssh::serverinclude customappinclude tomcat::server

class { ‘jenkins’: require => Class[‘tomcat::server’],}

include mysqldatabase { ‘apptest’: ensure => present,}

Our example configuration model:


Think about the usersMeet John, Susan and Bill.


John is a Sysadmin

• Wants to ensure all servers have kernel hardening, NTP and SSH Server installed

• Wants to manage what packages, services, files and other resources

• Is responsible for maintaining all the components of a UAT cluster server


Susan is an application specialist

• Cares that a UAT Cluster node requires MySQL Server, Tomcat Server and Jenkins server installed.


Bill is an IT manager

• Bill cares that the server is a UAT Cluster node


What do they care about?

• John cares about modelling all resources

• Susan cares about the technology stack

• Bill cares about the business logic


In Puppet

• Resource modelling is done in component modules

• The technology stack is defined in profiles

• Where do we represent the business logic for Bill?


Introducing Roles

• Represent business logic, not technology

• Define a set of technology stacks (profiles) that make up the logical role

• Allow the business to manage how the infrastructure looks without defining what it is


A node can only have one role

• A role can include as many profiles as required to define itself

• If a node requires two roles, it has by definition become a new role


A node can only have one role

• A role can include as many profiles as required to define itself

• If a node requires two roles, it has by definition become a new role

• Something couldn’t be a lion and a kangaroo at the same time!


It would be a Lingaroo


Roles

• One-to-one to nodes

• One-to-many to profiles

• Only implement profiles


Example role

class role::uat_server { include profiles::base include profiles::customapp include profiles::test_tools}


Classification

• Node classification simply assigns roles to nodes

• Roles expose profiles


Classification

node ‘craig.puppetlabs.vm’ { include roles::uat_server}


Classification


The Stack

Resources


The Stack

Resources



Roles : Business Logic


Terminology

• Profiles and Roles are Puppet modules

• Components are Puppet modules responsible for modelling resources

• Everything is a module


Naming conventions

• Components should be named after what they manage (apache, ssh, mysql)

• Profiles should be named after the logical stack they implement (database, bastion, email)

• Roles should be named in business logic convention (uat_server, web_cluster, application, archive)


Hiera OverviewLet’s talk about data!


Managing infrastructure

Dev



Dev

QA



Dev

QA

Production



Dev

QA

Production

DC1



Dev

QA

Production

DC1 DC2 DC3


Managing data in Puppet is hard.


Without Hiera?if ( $::environment == ‘dev’ ) { $ntpserver = ‘192.168.2.1’} else { if ( $::fqdn == ‘host4.mycorp.com’) { $ntpserver = ‘127.0.0.1’ } else { $ntpserver = ‘213.21.6.4’ }}


With Hiera?$ntpserver = hiera(‘ntpserver’)


Hierarchical lookups

• Hiera uses facter facts to determine a hierarchy

• Top down hierarchy for overriding configuration values based on roles, environments, locations.... or anything else

• And do this without any coding!


Separation of data from code

• Puppet modules without hard-coded data are easily shared and more re-usable

• Infrastructure configuration can be managed without needing to edit Puppet code


Pluggable Backends

• Source data from multiple locations

• Data source is abstracted from code


Pluggable Backends

• Source data from multiple locations

• Data source is abstracted from code

• hiera-gpg

• hiera-http

• hiera-mysql

• hiera-redis

• hiera-json

• hiera-zookeeper


Data Separation

• Use Hiera to abstract your data from your code

• Components and profiles can source data from Hiera


Profiles and Hiera

• Use Hiera to model your data

• Use profiles to model your implementation


The Stack

Resources





The Stack

Resources




Hiera:Data


Classification

• Assigning classes to a node

• You can classify within Puppet code (site.pp)

• You can use an External Node Classifier (ENC)


Leveraging an ENC

• You can classify your nodes however you want

• Puppet Dashboard

• Enterprise Console

• Foreman

• Site.pp

• Custom script


Leveraging an ENC

• An ENC should classify a node to it’s role

• Nothing else


The Stack

Resources




Hiera:Data


The Stack

Resources




Hiera:Data

Classifier


Key benefits

• Reduced node-level logic to a role.

• Gain the ability to be flexible with implementation

• Business logic improves managability by non-Puppet users

• Edge cases are now easy to solve


Enough Preaching!


This is not the way to design Puppet... It’s a

way.


Can I implement this design without roles?


Can I implement this design without roles?

• Yes.

• You lose the layer of abstraction that exposes business logic


Can my roles be defined in my ENC?


Can my roles be defined in my ENC?

• Yes.

• Keeping it in code makes it versionable


Can’t I just use Hiera to define profiles?


Can’t I just use Hiera to define profiles?

• Technically yes.

• You lose the flexibility to implement code logic in profiles and it may become restrictive

• You could possibly use: https://github.com/ripienaar/hiera-puppet-nodes


The fundamental concepts....



• Abstraction, abstraction, abstraction




• Decoupling business logic, implementation and resource modelling.





• Separating data and code





• Separating data and code

• Reducing node-level complexity


Other Resources

• Adrien Thebos’ excellent blog post http://sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as-legos.html

• My original blog post http://www.craigdunn.org/2012/05/239/

• Module Structure Redux by R.I.Pienaar http://www.devco.net/archives/2012/12/13/simple-puppet-module-structure-redux.php


http://sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as-legos.html






http://www.craigdunn.org/2012/05/239/










Thank you. Questions?

• Follow me at @crayfishX

• Bug me on Freenode: crayfishx

Enjoy the rest of Puppet Camp!

In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beerand helped me on my journey of discovery learning how to implement Puppet properly. R.I.P


designing puppet: roles/profiles pattern

Technology

proles class profiles

toyour puppet code

application stack

puppet labs engineers

puppet realworld case

node level repetition

craig dunn puppet user

code adaptable