Manageable Puppetinfrastructure

~April 2014 edition~

PuppetCamp Berlin

Ger Apeldoorn - http://puppetspecialist.nl

1 / 44

Freelance Puppet Consultant

Trainer for PuppetLabs Benelux

Who's this?

2 / 44

ScopeAlso... why this talk?

3 / 44

Commonpitfalls

4 / 44

Pitfalls

Cause & effectPitfalls

Lots of WorkaroundsUnmaintainable codebaseCollaboration difficulties

5 / 44

Pitfalls

Cause & effect

Quick Wins

Fix your codebase!Quick wins:

Move data to Hiera

Implement Code Review

Use Puppet-lint in a git-hook

REFACTOR CONSTANTLY

6 / 44

A Manageable DesignApril 2014 edition

7 / 44

RequirementsWhadda we need

8 / 44

Our environment should be:Easy to Use

Easy to Comprehend

Easy to Update

and... Safe

9 / 44

This stuffisn't exactly

easy

10 / 44

But we cán make it safe andmanageable

11 / 44

Requirements

Easy to:UseComprehendUpdate

Safe

SafeUse environments to test everything

Create a huge testing environment

Use Git to promote your code

12 / 44

Requirements

Easy to:UseComprehendUpdate

Safe

Manageable

ManageableKeep a consistent module structure

Using roles for abstraction

Facilitate collaboration

13 / 44

DomainsServer Roles

All things data

Deployment & Workflow

14 / 44

OverviewSoftware Components

15 / 44

Software ComponentsPuppet Enterprise or The Foreman

Hiera and hiera-eyaml (Hierarchical Data lookup)

Gerrit (Code review system)

Git (what else?)

Git Flow, adapted version for Gerrit

R10K (Environment deployment tool)16 / 44

Domain #1:

Server Roles

17 / 44

A layer of abstraction

18 / 44

How to do it?Create roles moduleroot@puppet# puppet module generate gerapeldoorn-role

Create a base-role to cover generic settings# modules/role/manifests/base.pp:class role::base { include users include ssh include motd ...

19 / 44

How to do it? -Cont'd-Put all required resources in the classes# modules/role/manifests/app.pp:class role::app { include apache include tomcat apache::virtualhost { 'default': ...

Include role in node definition# site.pp:node 'app01.autiplan.com' { include role::base include role::app}

20 / 44

Domain #2:

All things Data

21 / 44

HieraHierarchical data lookup tool

22 / 44

Configured Hierarchy:#/etc/puppet/hiera.yaml::hierarchy: - "%{::clientcert}" - "%{::environment}" - common

Node app01.autiplan.com:

environment: testing

Hieradata# hiera/app01.autiplan.com.yaml---examplekey: value for \ app01.autiplan.com

# hiera/testing.yaml---examplekey: value for nodes in \ testing environment

# hiera/common.yaml---examplekey: value for all nodes

It's all about Hierarchy

What will be in $test?$test = hiera('examplekey')

23 / 44

Types of HieradataRegular values# hiera/app01.autiplan.com.yaml---examplekey: value

24 / 44

Types of HieradataArrays# hiera/app01.autiplan.com.yaml---array: [ item1, item2, item3 ]

otherarray: - item1 - item2 - item3

Note: Never use tabs in Hiera files!

25 / 44

Types of HieradataHashes# hiera/app01.autiplan.com.yaml---hash: key1: value key2: value

26 / 44

Types of HieradataCombinations# hiera/app01.autiplan.com.yaml---hash: key1: value key2: value key3: - arrayvalue1 - arrayvalue2 key4: subhashkey1: value subhashkey2: value

27 / 44

Hiera-related functions...and what to use them for

28 / 44

What does it do?Retrieves the first-found value in thehierarchy. (top-down)

What to use it for?Basic variable-lookup.Very easy to create exceptions!

How to use it?

$smarthost = hiera('smarthost')

Example Hieradata# hiera/mail.autiplan.com.yaml---smarthost: smtp.myprovider.nl

# hiera/testing.yaml---smarthost: testsmtp.autiplan.com

# hiera/common.yaml---smarthost: mail.autiplan.com

hiera('key' [, default_value])

29 / 44

What does it do?Retrieves an array or hash valuein the hierarchy, concatinates allfound results

What to use it for?Combining data from allhierarchy levels.

How to use it?

$users = hiera_array('users')

Example Hieradata# hiera/app01.autiplan.com.yaml---users: [ 'user1', 'user2' ]

# hiera/testing.yaml---users: [ 'testuser' ]

# hiera/common.yaml---users: [ 'user3', 'user4' ]

hiera_array('key' [, default_value]) (and hiera_hash)

30 / 44

What does it do?Includes all classes listed in thearray that is loaded from Hiera.Takes elements from ALLhierarchy levels.

What to use it for?Lightweight ENC.Put all classes / roles in Hiera.

How to use it?

node default { hiera_include('roles')}

Example Hieradata# hiera/web01.autiplan.com.yaml---roles: - role::web

# hiera/common.yaml---roles: - role::base

hiera_include('classes')

31 / 44

What does it do?Generates resources from aHASH.

What to use it for?Generate any resource based ondata from Hiera.Can also be used withhiera_hash to create resourcesfrom all levels!

How to use it?

create_resources ('apache::vhost', hiera('vhosts', {}))

Example Hieradata# hiera/web01.autiplan.com.yaml---vhosts: autiplan.com: alias: www.autiplan.com autiplan.dk: alias: www.autiplan.dk docroot: /var/www/html/autiplan.dk autiplan.nl: alias: www.autiplan.nl cdn.autiplan.com: port: 81 docroot: /var/www/html/cdn

create_resources('type', HASH [, default_values])

32 / 44

Data bindingsAuto-loading of Hiera data for parameterized classes.

33 / 44

What does it do?Automatically loads classparameters from Hiera.

What to use it for?Specify all class parameters inHiera.Use all hierarchical benefits forclass parameters.Simplify the use ofparameterized classes.

How to use it?

include mysql::server

Example Hieradata# hiera/web01.autiplan.com.yaml---mysql::server::root_password: m0ars3cr3t

# hiera/common.yaml---mysql::server::root_password: t0ps3cr3tmysql::server::package_name: mysql-servermysql::server::restart: true

Data bindings

34 / 44

Putting it all togetherAnything node-specific should be in Hiera!

35 / 44

A Puppet Run: What calls what?

36 / 44

Domain #3:

Deployment & Workflow

37 / 44

EnvironmentsKeeping the environmentalists happy

38 / 44

EnvironmentsWhat is an environment?

Seperate modulepaths/site.pp.Common environments: development, testing, production.Nodes request a specific environment.

Why?Essential to prevent mistakes.NEVER edit code in production!The workflow helps us to 'promote' our code to production.

39 / 44

Demo!

40 / 44

R10k overview

41 / 44

Final remarksKeep public modules as-is, wherever possible

Create wrapper classes in company-module.Create fork if needed, submit pull request for fixes.

Add forked module (gitrepo) to Puppetfile.

Think aheadAlways try to anticipate future applications.If it feels overly complicated, yer doin it wrong.Refactor!

42 / 44

Questions?

43 / 44

Freelance Puppet Consultant

Trainer for PuppetLabs Benelux

Thank you!A howto of setting up this environment (and the workflow!) is available on my

blog: http://puppetspecialist.nl/mpi

44 / 44