Enjoying the Journey from Puppet 3.x to Puppet 4.xRob Nelson

Who Am I?

Puppet user since 2014 (3.3 era)Vox Pupuli, puppet-lint contributor@rnelson0, https://rnelson0.com/

Agenda• Why upgrade?

• Refactor our codebase for Puppet 4

• Upgrade our Puppet master(s) and agents to Puppet 4.x

• Refactor our codebase to remove Puppet 2- and 3-isms

• Tips, tricks, and tools

• Enjoying ourselves

Why?• Puppet 4 is old! First released March, 2015

• Puppet 3 is really old! End Of Support on December 31, 2016

• Puppet 4 only modules

• Puppet 4 language improvements

• Application Orchestration

• PE first, FOSS eventually; some free implementations (such as choria) appearing

• AIO Puppet and Puppetserver

• Better performance, security; same agent/puppetserver between FOSS and PE

• Puppet 5 is coming!

Who does this apply to?• Puppet Enterprise users

• Puppet Opensource users

• Foreman (1.13+) users

• Master and Masterless

Blueprint• Start with Puppet 3.x

• Read the release notes

• Plan the roadmap

• Validate / create tests

• Refactor until the new version passes all tests

• Upgrade/Replace the master(s)

• Upgrade the agents

• Repeat the Refactor / Upgrade steps until you get to 4.latest

Release Notes• All of them - not just the latest version

• Identify potential issues, deprecated features, etc

• Determine the minimum version required to upgrade to target version

• Stay up to date

Define the Roadmap• Determine the current version

• In-place upgrades or new infrastructure?

• Identify intermediate version steps

• Enable Future Parser [and Strict Variables] before you hit 4.x

• PE: Requires intermediate upgrades or fresh installs (check KB)

• FOSS: Go straight to 4.latest

• Determine how upgrades and interruptions affect ecosystem products – PE Console/puppetboard, SEIMs, monitoring, etc.

FOSS Example Roadmap• 3.6.0 -> 3.8.7

• 3.8.7 w/Future Parser [and Strict Variables]

• 3.8.7 -> 4.7.0

• Today’s example roadmap

PE Example Roadmap• 3.7.2 -> 3.8.6

• 3.8.6 w/Future Parser [and Strict Variables]

• 3.8.6 -> 2015.3.3

• 2015.3.3 -> 2016.2.1

Validate/Create Tests• Tests assure (mostly) predictable behavior

• Determine what kinds of tests you need - unit, acceptance, integration, other?

• Good testing setup in puppet-module-skeleton

• puppet-retrospec generates naive tests that need tuned

• Existing tests must pass before modifying code

• Turn on Future Parser [and Strict Variables] only at 3.8.x

• Use puppet-lint community plugins, esp for v4 transition

• Beyond tests: catalog diffs, personalized tests

Rspec Tests$ cat spec/classes/apache_spec.rbrequire 'spec_helper'describe 'profile::apache', :type => :class do let :facts do { facts_hash } end

context 'with defaults for all parameters' do it { is_expected.to create_class('profile::apache') } it { is_expected.to contain_package('httpd') } it { is_expected.to contain_user("apache") } endend

Rspec Run[rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/apache_spec.rbprofile::apache with defaults for all parameters should contain Class[profile::apache] should contain Package[httpd] should contain User[apache]

Finished in 7.82 seconds (files took 2.49 seconds to load)3 examples, 0 failures

Refactor• Create a new branch for the target version, e.g. 3.8.7

• Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and without Future Parser/Strict Variables

• Identify failing tests, refactor to fix

• Upgrade modules as early as possible. Be aware of the required Puppet version for a module version, and look out for defunct or migrated modules, such as those transferred to Vox Pupuli

• Move forward when tests are green for the next version – previous may be red

Testing with particular Puppet versions$ grep PUPPET Gemfile gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0'

[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='~>3.8.0'[rnelson0@build controlrepo]$ bundle updateInstalling puppet 3.8.7 (was 4.6.0)[rnelson0@build controlrepo]$ bundle exec puppet --version3.8.7

[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='3.8.1'[rnelson0@build controlrepo]$ bundle updateInstalling puppet 3.8.1 (was 3.8.7)[rnelson0@build controlrepo]$ bundle exec puppet --version3.8.1

High level Master(s) upgrade process• Prep for new master/in-place upgrade

• Deploy new/upgrade in testing

• Revert

• Deploy new/upgrade in production

• Start with Master of Masters or other “parent” nodes first

• Update separate PuppetDB node, puppetdb-termini on masters

Replace the Master• Prepare a new operational environment

• Do not serve bad/incorrect catalogs to existing nodes

• Deploy a new master on the target puppet version

• Bootstrap configuration/code

• Test the master against itself, puppet agent -t

• Deploy and test canary nodes in the same operational environment

In-place Master upgrade• Snapshot (or equivalent) the master(s) and canary nodes

• Restrict access to the master:

• Control access with firewall/load balancer

• Disable puppet agent on nodes with orchestration

• Revoke certificates for non-canary nodes

• Revoke the CA, generate a new CA and new agent certs for canary nodes only

• Upgrade the master

• Test the master then canary nodes with puppet agent -t

Troubleshooting• Collect logs from the master and canaries

• Look for changed resources, number of resources in catalog

• Revert production environment

• Analyze cause(s)

• Refactor code and data to address issues

• Try again

• Learn from failures, prevent them in the future

Upgrade the Agents• Can often skip on PATCH versions and some MINOR versions (see rel notes)

• puppetlabs/puppet_agent (requirements) updates agents on next check-in

• Orchestration

• Replace nodes with new instances running the new agent

• By hand

Repeat• Relax, enjoy the success of an upgrade!

• Start working on the next version/feature flags

• Repeat the Refactor / Upgrade steps

Keeping Up

Keeping UpRefactor to take advantage of Puppet 4 language improvements, new tools (ex: r10k -> PE Code Manager), new file locations, etc.

• PE has quarterly upgrades, FOSS more frequent

• The less frequently you do something, the more painful it is. “Upgrade early and upgrade often!”

• Try not to get more than 2 MINORs behind

• Test against puppet version ~>4.0 (latest v4) and run bundle update before manual tests

Puppet 4 Language Improvements• Replace create_resources() with iteration

• Replace validate_*() with data types (including a Sensitive type)

• There is a validate_legacy() helper function available in puppetlabs/stdlib to assist with replacing validate_*() functions (blog)

• Simplified resource wrappers with * and + operators

• Improved default attributes are per-expression

• New template type EPP is available

• Puppet Lookup, Data In Modules, and other hiera improvements

• Use $facts[] instead of global variables to tidy up the namespace and remove ambiguity

Tips & Tricks – Puppet Enterprise• PE includes support, use it for planning/errors

• Puppet Enterprise Upgrade Service to engage Pro Services

• PE Classifier changes over time. Review Preconfigured Node Groups documentation

• pe_puppetserver_gem is out, puppetserver_gem is in

• Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead

• Do not ever do this on your master. EVER!

Tips & Tricks - StringsUnderstand how string conversion works in puppet, hiera, rspec-puppet, and how it has changed:

• rspec-puppet: 'undef' represents an undefined value

• Puppet DSL: it is the string undef! Try :undef, without quotes, instead

• If you have a file resource with a title or path of ${undefvar}/${populatedvar}, rspec will start failing because file { 'undef/etc/app.conf' :} is not valid

• Similar issue with 'true' vs true and 'false' vs false

• Other common issues: input from hiera/ENC, quoted numbers as strings, stringify vs structured facts, unquoted strings in case selectors, etc

• May require acceptance tests/canary nodes to become apparent

Tips & Tricks - Hiera• Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver

• Enable the yaml backend and ensure that the master does not rely on eyaml data

• Run the agent on the master to redeploy the gem (with puppet/hiera or similar) before agents check in

• %{}: used to prevent variable interpolation, as in %%{}{environment} to generate the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4 it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use %{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0

• datadir: some versions expect :: prepends to variables and others do not. Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well

Tips & Tricks - Other• Review modules and their supported versions. May be incorrect or weak

assumptions (>= 3 but should also include < 4 – check tests)

• Upgrades across major versions mean additional troubleshooting

• Upgrade early – but with caveats

• Many tools to assist with automating version upgrades in your Puppetfile

• ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>)

• Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4

• External fact weighting bug: FACT-1413

• Minimize coupled/entangled changes

• Ask for help! Colleagues, social media, etc.

Tools• Puppet Community Slack / IRC and Mailing Lists

• puppet-ghostbuster helps you find "dead code" that you may want to prune before you start on your refactoring journey.

• rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their Puppet 4 support

• A number of catalog diff tools exist (diff generators and a viewer) to inspect the actual catalog differences from active nodes across different versions of Puppet.

LinksAdditional information on Puppet 4 and Migrations

• Official Puppet Upgrade Docs

• Whirlwind Tour of Puppet 4 by R.I. Pienaar

• The Power of Puppet 4 by Martin Alfke

• Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt

Summary• Plan the upgrade roadmap

• Have working tests before upgrading

• Step through the new versions / feature flags

• Refactor code to take advantage of the language and tool improvements

• Keep mowing

• Enjoy the journey!