puppetconf 2016: enjoying the journey from puppet 3.x to 4.x – rob nelson, at&t
TRANSCRIPT
Enjoying the Journey from Puppet 3.x to Puppet 4.x Rob Nelson
Who Am I?
Puppet user since 2014 (3.3 era) Vox Pupuli, puppet-lint contributor @rnelson0, https://rnelson0.com/
Agenda • Why upgrade?
• Refactor our codebase for Puppet 4
• Upgrade our Puppet master(s) and agents to Puppet 4.x
• Refactor our codebase to remove Puppet 2- and 3-isms
• Tips, tricks, and tools
• Enjoying ourselves
Why? • Puppet 4 is old! First released March, 2015
• Puppet 3 is really old! End Of Support on December 31, 2016
• Puppet 4 only modules
• Puppet 4 language improvements
• Application Orchestration
• PE first, FOSS eventually; some free implementations (such as choria) appearing
• AIO Puppet and Puppetserver
• Better performance, security; same agent/puppetserver between FOSS and PE
• Puppet 5 is coming!
Who does this apply to? • Puppet Enterprise users
• Puppet Opensource users
• Foreman (1.13+) users
• Master and Masterless
Blueprint • Start with Puppet 3.x
• Read the release notes
• Plan the roadmap
• Validate / create tests
• Refactor until your current version passes all tests
• Upgrade/Replace the master(s)
• Upgrade the agents
• Repeat the Refactor / Upgrade steps until you get to 4.latest
Release Notes • All of them - not just the latest version
• Identify potential issues, deprecated features, etc
• Determine the minimum version required to upgrade to target version
• Stay up to date
Define the Roadmap • Determine the current version
• In-place upgrades or new infrastructure?
• Identify intermediate version steps
• Enable Future Parser [and Strict Variables] before you hit 4.x
• PE: Requires intermediate upgrades or fresh installs (check KB)
• FOSS: Go straight to 4.latest
• Determine how upgrades and interruptions affect ecosystem products – PE Console/puppetboard, SEIMs, monitoring, etc.
FOSS Example Roadmap • 3.6.0 -> 3.8.7
• 3.8.7 w/Future Parser [and Strict Variables]
• 3.8.7 -> 4.7.0
• Today’s example roadmap
PE Example Roadmap • 3.7.2 -> 3.8.6
• 3.8.6 w/Future Parser [and Strict Variables]
• 3.8.6 -> 2015.3.3
• 2015.3.3 -> 2016.2.1
Validate/Create Tests • Tests assure (mostly) predictable behavior
• Determine what kinds of tests you need - unit, acceptance, integration, other?
• Good testing setup in puppet-module-skeleton
• Never written an rspec-puppet test? puppet-retrospec generates naive tests that need tuned
• Existing tests must pass before modifying code
• Turn on Future Parser [and Strict Variables] only at 3.8.x
• Beyond tests: catalog diffs, personalized tests
Rspec Tests $ cat spec/classes/apache_spec.rb require 'spec_helper' describe 'profile::apache', :type => :class do let :facts do { facts_hash } end context 'with defaults for all parameters' do it { is_expected.to create_class('profile::apache') } it { is_expected.to contain_package('httpd') } it { is_expected.to contain_user("apache") } end end
Rspec Run [rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/apache_spec.rb profile::apache with defaults for all parameters should contain Class[profile::apache] should contain Package[httpd] should contain User[apache] Finished in 7.82 seconds (files took 2.49 seconds to load) 3 examples, 0 failures
Refactor • Create a new branch for the target version, e.g. 3.8.7
• Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and without Future Parser/Strict Variables
• Identify failing tests, refactor to fix
• Upgrade modules as early as possible. Be aware of the required Puppet version for a module version, and look out for defunct or migrated modules, such as those transferred to Vox Pupuli
• Move forward when tests are green for current AND next version
Testing with particular Puppet versions
$ grep PUPPET Gemfile gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0'
[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='~>3.8.0' [rnelson0@build controlrepo]$ bundle update Installing puppet 3.8.7 (was 4.6.0) [rnelson0@build controlrepo]$ bundle exec puppet --version 3.8.7
[rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='3.8.1' [rnelson0@build controlrepo]$ bundle update Installing puppet 3.8.1 (was 3.8.7) [rnelson0@build controlrepo]$ bundle exec puppet --version 3.8.1
High level Master(s) upgrade process • Prep for new master/in-place upgrade
• Deploy new/upgrade in testing
• Revert
• Deploy new/upgrade in production
• Start with Master of Masters or other “parent” nodes first
• Update separate PuppetDB node, puppetdb-termini on masters
Replace the Master • Prepare a new operational environment
• Do not serve bad/incorrect catalogs to existing nodes
• Deploy a new master on the target puppet version
• Bootstrap configuration/code
• Test the master against itself, puppet agent -t
• Deploy and test canary nodes in the same operational environment
In-place Master upgrade • Snapshot (or equivalent) the master(s) and canary nodes
• Restrict access to the master:
• Control access with firewall/load balancer
• Disable puppet agent on nodes with orchestration
• Revoke certificates for non-canary nodes
• Revoke the CA, generate a new CA and new agent certs for canary nodes only
• Upgrade the master
• Test the master then canary nodes with puppet agent -t
Troubleshooting • Collect logs from the master and canaries
• Look for changed resources, number of resources in catalog
• Revert production environment
• Analyze cause(s)
• Refactor code and data to address issues
• Try again
• Learn from failures, prevent them in the future
Upgrade the Agents • Can often skip on PATCH versions and some MINOR versions (see rel notes)
• puppetlabs/puppet_agent (requirements) updates agents on next check-in
• Orchestration
• Replace nodes with new instances running the new agent
• By hand
Repeat • Relax, enjoy the success of an upgrade!
• Start working on the next version/feature flags
• Repeat the Refactor / Upgrade steps
Keeping Up
Keeping Up Refactor to take advantage of Puppet 4 language improvements, new tools (ex: r10k -> PE Code Manager), new file locations, etc.
• PE has quarterly upgrades, FOSS more frequent
• The less frequently you do something, the more painful it is. “Upgrade early and upgrade often!”
• Try not to get more than 2 MINORs behind
• Test against puppet version ~>4.0 (latest v4) and run bundle update before manual tests
Puppet 4 Language Improvements • Replace create_resources() with iteration
• Replace validate_*() with data types (including a Sensitive type)
• There is a validate_legacy() helper function available in puppetlabs/stdlib to assist with replacing validate_*() functions (blog)
• Simplified resource wrappers with * and + operators
• Improved default attributes are per-expression
• New template type EPP is available
• Puppet Lookup, Data In Modules, and other hiera improvements
• Use $facts[] instead of global variables to tidy up the namespace and remove ambiguity
Tips & Tricks – Puppet Enterprise • PE includes support, use it for planning/errors
• Puppet Enterprise Upgrade Service to engage Pro Services
• PE Classifier changes over time. Review Preconfigured Node Groups documentation
• pe_puppetserver_gem is out, puppetserver_gem is in
• Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead
• Do not ever do this on your master. EVER!
Tips & Tricks - Strings Understand how string conversion works in puppet, hiera, rspec-puppet, and how it has changed:
• rspec-puppet: 'undef' represents an undefined value
• Puppet DSL: it is the string undef! Try :undef, without quotes, instead
• If you have a file resource with a title or path of ${undefvar}/${populatedvar}, rspec will start failing because file { 'undef/etc/app.conf' :} is not valid
• Similar issue with 'true' vs true and 'false' vs false
• Other common issues: input from hiera/ENC, quoted numbers as strings, stringify vs structured facts, unquoted strings in case selectors, etc
• May require acceptance tests/canary nodes to become apparent
Tips & Tricks - Hiera • Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver
• Enable the yaml backend and ensure that the master does not rely on eyaml data
• Run the agent on the master to redeploy the gem (with puppet/hiera or similar) before agents check in
• %{}: used to prevent variable interpolation, as in %%{}{environment} to generate the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4 it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use %{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0
• datadir: some versions expect :: prepends to variables and others do not. Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well
Tips & Tricks - Other • Review modules and their supported versions. May be incorrect or weak
assumptions (>= 3 but should also include < 4 – check tests)
• Upgrades across major versions mean additional troubleshooting
• Upgrade early – but with caveats
• Many tools to assist with automating version upgrades in your Puppetfile
• ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>)
• Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4
• External fact weighting bug: FACT-1413
• Minimize coupled/entangled changes
• Ask for help! Colleagues, social media, etc.
Tools • Puppet Community Slack / IRC and Mailing Lists
• puppet-ghostbuster helps you find "dead code" that you may want to prune before you start on your refactoring journey.
• rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their Puppet 4 support
• A number of catalog diff tools exist (diff generators and a viewer) to inspect the actual catalog differences from active nodes across different versions of Puppet.
Links Additional information on Puppet 4 and Migrations
• Official Puppet Upgrade Docs
• Whirlwind Tour of Puppet 4 by R.I. Pienaar
• The Power of Puppet 4 by Martin Alfke
• Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt
Summary • Plan the upgrade blueprint
• Have working tests before upgrading
• Step through the new versions / feature flags
• Refactor code to take advantage of the language and tool improvements
• Keep mowing
• Enjoy the journey!