puppet camp berlin 2014: advanced puppet design
DESCRIPTION
"Advanced Puppet Design" presented at Puppet Camp Berlin 2014 by Craig DunnTRANSCRIPT
Advanced Puppet Design
Craig Dunn, Puppet Camp Berlin 2014
Friday, 11 April 14
Hello
• Craig Dunn
• Puppet user since 2008
• Previously worked for Puppet Labs
• Founder of Enviatics
• IT automation engineer and trainer
• Based in Spain but work anywhere
Friday, 11 April 14
My talk
• Avoiding pain
• Writing good modules
• Challenges of codebase design
• Roles / Profiles
Friday, 11 April 14
But first....Everyone loves polls, right?
Friday, 11 April 14
Thinking right
• Business has requirements
• We use technology to fulfill them
• But it’s not that simple!
Friday, 11 April 14
“Business logic does not oftenalign with technology design”
Friday, 11 April 14
Business requirement
“We have 3 applications we
need to deploy using Puppet”
Friday, 11 April 14
Puppetize
• Write 3 Puppet modules for 3 applications
• Because that was the requirement
• Was it?
Friday, 11 April 14
It’s not working
Friday, 11 April 14
Let’s suppose
• Each “application” is a set of shared components implemented different ways
Friday, 11 April 14
The business view
Application Y Application Z
Application X
Friday, 11 April 14
The technical reality
Application YApplication Z
Application X
Friday, 11 April 14
Danger Signs
• Resources being declared in two modules
• You don’t know where your implementation “fits”
• Lot’s of logic at a node level
• Repetition and duplication
• The if statement is your go-to-guy
Friday, 11 April 14
Catastrophic Signsif ($::hostname =~ /^host[0-3].*/) { package { ‘my-special-app’: ensure => installed, }}
Friday, 11 April 14
Catastrophic Signsif !defined(Package[‘httpd’]) { package { ‘httpd’: ensure => installed, }}
Friday, 11 April 14
Catastrophic Signsensure_resource(‘package’,‘httpd’,{‘ensure’ => ‘installed’})
Friday, 11 April 14
Catastrophic Signsensure_resource(‘package’,‘httpd’,{‘ensure’ => ‘installed’})
if function_defined_with_params(["#{type}[#{item}]", params]) Puppet.debug("Resource #{type}[#{item}] not created because it already exists") else Puppet::Parser::Functions.function(:create_resources) function_create_resources([type.capitalize, { item => params }]) end
Friday, 11 April 14
World ending signs
• You use parser=future in production
• You aren’t regretting it yet
• You then implemented order=manifest
Friday, 11 April 14
An unhappy Puppet
Friday, 11 April 14
Stop thinking about what it looks like
• Break everything down into components
• Granularity is the key
• Think about what it actually is
Friday, 11 April 14
Writing modules
• Granularity
• Portability
• Flexibility of implementation
Friday, 11 April 14
A bad moduleclass web {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
package { ‘php5’: ensure => installed, require => Package[‘httpd’], }
package { ‘mysql-server’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Friday, 11 April 14
A bad moduleclass web {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
package { ‘php5’: ensure => installed, require => Package[‘httpd’], }
package { ‘mysql-server’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Granularity
Friday, 11 April 14
A bad moduleclass web {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
package { ‘php5’: ensure => installed, require => Package[‘httpd’], }
package { ‘mysql-server’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Portability
Granularity
Friday, 11 April 14
A bad moduleclass web {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
package { ‘php5’: ensure => installed, require => Package[‘httpd’], }
package { ‘mysql-server’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Portability
Implementation
Granularity
Friday, 11 April 14
Keep your modulesgranular
• Manage only resources in the scope of the module
• Small modules are ok!
Friday, 11 April 14
A granular moduleclass apache {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Friday, 11 April 14
A granular moduleclass apache {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Granularity
Friday, 11 April 14
A granular moduleclass apache {
$docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, }
file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), }}
Granularity
Portability
Friday, 11 April 14
Sharing is good
• Re-usable by others in your team
• You can publish to the forge!
• People will collaborate
Friday, 11 April 14
It’s all about sharing!
Friday, 11 April 14
You are gonna shareyour s**t aren’t you?
Friday, 11 April 14
Sharing is not justpull requests
• Share your ideas in blog posts
• What worked? What didn’t?
• Discuss and collaborate on mailing lists
• IRC
Friday, 11 April 14
Making sharing easier
• Data separation (Hiera)
• Allow the user flexibility of implementation
Friday, 11 April 14
defaults params pattern
• Use a parameterized class
• Default from an inherited class
• Allow the user to decide implementation
Friday, 11 April 14
defaults patternclass apache { $packagename=‘httpd’ $docroot=’/var/www’ $listenaddr=‘10.0.1.12’ $servername=‘myweb.foo.com’
package { $packagename: ensure => installed, }
.....
Friday, 11 April 14
defaults patternclass apache { $packagename=‘httpd’ $docroot=’/var/www’ $listenaddr=‘10.0.1.12’ $servername=‘myweb.foo.com’
package { $packagename: ensure => installed, }
.....
In-module private data
No way to override
Friday, 11 April 14
defaults patternclass apache ( $packagename=‘httpd’, $docroot=’/var/www’, $listenaddr, $servername,) {
package { $packagename: ensure => installed, }
.....
Friday, 11 April 14
defaults pattern
class { ‘apache’: listenaddr => ‘10.0.1.2’, servername => ‘foo.example.com’,}
Override data on implementation
Friday, 11 April 14
defaults pattern
# /etc/puppet/hieradata/dev.yaml
---apache::docroot: /var/dev/sites
Overriding from Hiera
Friday, 11 April 14
defaults patternclass apache::defaults {
$packagename=$::osfamily ? { redhat => ‘httpd’, debian => ‘apache2’, default => ‘httpd’, } $docroot=’/var/www’ $listenaddr = $::ipaddress $servername = $::fqdn}.....
Friday, 11 April 14
defaults patternclass apache::defaults {
$packagename=$::osfamily ? { redhat => ‘httpd’, debian => ‘apache2’, default => ‘httpd’, } $docroot=’/var/www’ $listenaddr = $::ipaddress $servername = $::fqdn}.....
Added logic todefaults
Common defaults
Friday, 11 April 14
defaults pattern
class apache ( $packagename=$::apache::defaults::packagename, $docroot=$::apache::defaults::docroot, $listenaddr=$::apache::defaults::listenaddr, $servername=$::apache::defaults::servername,) inherits apache::defaults {
package { $::apache::packagename: ensure => installed, }
.....
Friday, 11 April 14
defaults pattern
Granularity
Portability
Implementation
Smaller modules with smaller scope
No hard coded data in modules
User can decided how / where to override defaults without editing the module
Friday, 11 April 14
Component Modules
Friday, 11 April 14
Now to build something awesome
... but where?
Friday, 11 April 14
Designing Puppet
Component Modules
Node Classification
Friday, 11 April 14
Node-level logicnode basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’]}
Friday, 11 April 14
Node-level logic
node basil inherits base { class { ‘apache’: version => ‘latest’, }}
Friday, 11 April 14
Node-level logic
• Risks duplication and repetition
• No guarantee of consistency
• Pseudo nodes and inheritance trees will get messy, fast.
• TMI!
Friday, 11 April 14
Thinking beyond the module....
• Puppet is a code base
• You need an effective framework
• Gluing everything together
Friday, 11 April 14
Introducing Profiles
Friday, 11 April 14
Profiles
• Wrapper classes implementing component modules
• Define a logical technology stack
• But just one!
Friday, 11 April 14
Profilesclass profile::blog { User <| group == ‘webadmins’ |> class { ‘::mysql::server’: } class { ‘::mysql::bindings’: php_enable => true, }
class { ‘::wordpress’: install_dir => ‘/var/www/wp’, }}
Friday, 11 April 14
Profiles
• Component modules manage the resources
• Profiles provide a layer of implementation
Friday, 11 April 14
Profiles and Components
Resources
Friday, 11 April 14
Profiles and Components
Resources
Components: Resource modelling
Friday, 11 April 14
Profiles and Components
Resources
Components: Resource modelling
Profiles : Implementation
Friday, 11 April 14
Lessons learned
• Granularity is good
• Don’t assume business logic will directly translate to technology
• Abstraction is awesome.... but that’s nothing new....
Friday, 11 April 14
Abstraction is a core principle of coding
• Implementation is abstracted by methods
• Methods abstracted by classes and modules
• They are abstracted with libraries
• Puppet is code!
Friday, 11 April 14
Puppet is all about abstraction
• Providers are abstracted by types
• Resources are abstracted by classes
• Classes are abstracted by modules
Friday, 11 April 14
Puppet is all about abstraction
• Providers are abstracted by types
• Resources are abstracted by classes
• Classes are abstracted by modules
• Modules are abstracted by profiles
Friday, 11 April 14
Focussing on Abstraction
• We’ve turned business logic into a technology stack
• Can we translate that back into business logic?
• Why would we even want to do that?
Friday, 11 April 14
Introducing roles
• Translate to business logic
• Identify the function of a server in human terms
• We never said business logic was a bad thing
Friday, 11 April 14
Configuration model
include profiles::securityinclude profiles::usersinclude profiles::networkinginclude profiles::blog
This is a “acme” server
Friday, 11 April 14
Think about the usersMeet John, Susan and Bill.
Friday, 11 April 14
John is a Sysadmin
• Wants to ensure all servers have kernel hardening, NTP and SSH Server installed
• Wants to manage what packages, services, files and other resources
• Is responsible for maintaining all the components of a any type of server
Friday, 11 April 14
Susan is an application specialist
• Cares that a the node has Wordpress and MySQL implemented properly
• She probably doesn’t care about how sudoers is configured
Friday, 11 April 14
Bill is an IT manager
• Bill cares that the server is an ACME App server
• He probably doesn’t understand what sudoers is
Friday, 11 April 14
What do they care about?
• John cares about modelling all resources
• Susan cares about the technology stack
• Bill cares about the business logic
Friday, 11 April 14
In Puppet
• Resource modelling is done in component modules
• The technology stack is defined in profiles
• Where do we represent the business logic for Bill?
Friday, 11 April 14
Roles
• Represent business logic, not technology
• Define a set of technology stacks (profiles) that make up the logical role
• Allow the business to manage how the infrastructure looks without defining what it is
Friday, 11 April 14
A node can only have one role
• A role can include as many profiles as required to define itself
• If a node requires two roles, it has by definition become a new role
Friday, 11 April 14
A node can only have one role
• A role can include as many profiles as required to define itself
• If a node requires two roles, it has by definition become a new role
• Something couldn’t be a lion and a kangaroo at the same time!
Friday, 11 April 14
It would be a Lingaroo
Friday, 11 April 14
Roles
• One-to-one to nodes
• One-to-many to profiles
• Only implement profiles, that’s it!
Friday, 11 April 14
The Stack
Resources
Friday, 11 April 14
The Stack
Resources
Components: Resource modelling
Friday, 11 April 14
The Stack
Resources
Components: Resource modelling
Profiles : Implementation
Friday, 11 April 14
The Stack
Resources
Components: Resource modelling
Profiles : Implementation
Roles : Business Logic
Friday, 11 April 14
Role classes
class role::acme { include profiles::security include profiles::users include profiles::networking include profiles::blog}
This is a “acme” server
Friday, 11 April 14
Terminology
• Profiles and Roles are Puppet modules
• They are not special
• Everything is a module
Friday, 11 April 14
Classification
• Assigning classes to a node
• You can classify within Puppet code (site.pp)
• You can use an External Node Classifier (ENC)
Friday, 11 April 14
Classification
• You can classify your nodes however you want
• Puppet Dashboard
• Enterprise Console
• Foreman
• Site.pp
• Custom script
Friday, 11 April 14
Classification
node ‘craig.puppetlabs.vm’ { include roles::acme_app}
Friday, 11 April 14
Classification
Friday, 11 April 14
Classification
• With roles and profiles we just classify the role to the node
Friday, 11 April 14
The Stack
Resources
Components: Resource modelling
Profiles : Implementation
Roles : Business Logic
Friday, 11 April 14
The Stack
Resources
Components: Resource modelling
Profiles : Implementation
Roles : Business Logic Classifier
Friday, 11 April 14
Data Separation
If you’re not using Hiera you are goingto break your data!
Friday, 11 April 14
Data Separation
• Use parameterized classes
• Hiera data bindings
• Component modules and profiles can look up data from Hiera
• Roles should NOT
Friday, 11 April 14
Roles and Profilesfor DevOps
• Full props to Laurent Bernaille from D2SI
• Achieving Continuous Delivery and Devops with Puppet
• Puppet Camp Paris, 2014.
Friday, 11 April 14
Roles and Profilesfor DevOps
• Using roles and profiles makes it easier for developers and ops to all collaborate on Puppet
• Developers write profiles for the their apps
• Ops write profiles for their infrastructure
• Roles encompass all of them
Friday, 11 April 14
The Roles/ProfilesStack
Resources
Components: Resource modelling
Roles : Business Logic
Hiera:Data
Classifier Classification
Dev profiles Ops profiles
Friday, 11 April 14
Roles/Profiles/Devopshttp://fr.slideshare.net/D2SI/d2-si-puppetcamp
Friday, 11 April 14
Key benefits
• Reduced node-level logic to a role.
• Gain the ability to be flexible with implementation
• Business logic improves managability by non-Puppet users
• Edge cases are now easy to solve
Friday, 11 April 14
Enough Preaching!
Friday, 11 April 14
This is not the way to design Puppet... It’s a
way.
Friday, 11 April 14
Can I implement this design without roles?
Friday, 11 April 14
Can I implement this design without roles?
• Yes.
• You lose the layer of abstraction that exposes business logic
Friday, 11 April 14
Can my roles be defined in my ENC?
Friday, 11 April 14
Can my roles be defined in my ENC?
• Yes.
• Keeping it in code makes it versionable
Friday, 11 April 14
Can’t I just use Hiera to define profiles?
Friday, 11 April 14
Can’t I just use Hiera to define profiles?
• Technically yes.
• You lose the flexibility to implement code logic in profiles and it may become restrictive
Friday, 11 April 14
The fundamental concepts....
Friday, 11 April 14
The fundamental concepts....
• Abstraction, abstraction, abstraction
Friday, 11 April 14
The fundamental concepts....
• Abstraction, abstraction, abstraction
• Decoupling business logic, implementation and resource modelling.
Friday, 11 April 14
The fundamental concepts....
• Abstraction, abstraction, abstraction
• Decoupling business logic, implementation and resource modelling.
• Separating data and code
Friday, 11 April 14
The fundamental concepts....
• Abstraction, abstraction, abstraction
• Decoupling business logic, implementation and resource modelling.
• Separating data and code
• Reducing node-level complexity
Friday, 11 April 14
Other resources
• http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/
• http://www.craigdunn.org/2012/05/239/
Friday, 11 April 14
Danke. Questions?
• Follow me at @crayfishX
• Bug me on Freenode: crayfishx
In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beerand helped me on my journey of discovery learning how to implement Puppet properly. R.I.P
Friday, 11 April 14