orchestrated functional testing with puppet-spec and mspectator - puppetconf 2014
DESCRIPTION
Orchestrated Functional Testing with Puppet-spec and Mspectator - Raphaël Pinson, CamptocampTRANSCRIPT
ORCHESTRATED FUNCTIONALTESTING WITH PUPPET-SPECAND MSPECTATOR
RAPHAËL PINSON
Who am I?Raphaël Pinson (@raphink)
■ Infrastructure Developer & Trainer @ Camptocamp
■ Augeas & Augeasproviders developer
■ Various contributions to Puppet & ecosystem
www.camptocamp.com / 2/32
Monitoring vs. Functional TestsComplementary or redundant?
www.camptocamp.com / 3/32
Conformity Tests
■ Check if machines comply to standards
■ Avoid permanent heavy monitoring checks
■ Tests must be inter-dependent
■ Focus on getting sysadmins to fix one thing at a time to convergetoward standards
www.camptocamp.com / 4/32
Treetester■ Back in 2008
■ Written in Perl
■ Orchestrate conformity tests on a 4k+ server fleet
www.camptocamp.com / 5/32
Treetester: modules output■ For all hosts/modules
■ Number of hosts filtered per module
■ Modules dependency tree
■ Colors by priority
www.camptocamp.com / 6/32
Treetester: host output■ For each host
■ Failed steps in the module tree
■ Green: OK, Red: KO, Purple: Ignored
www.camptocamp.com / 7/32
Treetester architecture■ All data in a database (MySQL)
■ Tests scripts output YAML
■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http)
■ Tests are inter-dependent
■ Generate filtered data as a tree
■ Generate graphs (graphviz)
www.camptocamp.com / 8/32
Treetester filters■ For each test/module
■ Based on data in MySQL (joins and additional SQL conditions)
■ Allows to link tests to each other
■ Like multiple sieves
www.camptocamp.com / 9/32
Treetester: future?■ Not open-sourced :'-(
■ Too monolithic/not flexible enough
■ Heavily linked to specific architecture
■ Needed a rewrite
www.camptocamp.com / 10/32
Adding specs to Puppet runs■ Testing the catalog before it gets applied
■ Testing the node after the catalog is applied
Enter the Puppet-spec module
www.camptocamp.com / 11/32
Rspec-puppet■ http://rspec-puppet.com
■ Now the standard to unit test Puppet manifests
■ Generates catalogs in clean environments
■ Asserts catalogs for resources/classes
requirerequire 'spec_helper'
describe 'logrotate::rule' dodo
let(:title) { 'nginx' }
it { should compile.with_all_deps }
it { should contain_class('logrotate::setup') }
endend
www.camptocamp.com / 12/32
Puppet-spec■ Runs tests from within Puppet runs
■ Test catalogs using rspec-puppet
■ Test hosts using serverspec
www.camptocamp.com / 13/32
Puppet-spec: Unit testing■ Catalog exposed by PuppetSpec::Catalog.instance.catalog
■ Uses rspec-puppet matchers
■ Asserts real catalogs
■ Runs on the master or agent side (as catalog indirection terminii)
describe 'puppet' dodo
subject { PuppetSpec::Catalog.instance.catalog }
it { should contain_package('puppet') }
it { should contain_package('ppet') }
it { should include_class('puppet') }
it { should include_class('puppet::client::base') }
endend
www.camptocamp.com / 14/32
Puppet-spec: Unit tests output# puppet agent -tinfo: Retrieving pluginerr: Could not retrieve catalog from remote server: Unit tests failed:F..
Failures:
1) packageFailure/Error: it { should contain_package('augeas') }
expected that the catalogue would contain Package[augeas]# /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3# /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find'
Finished in 0.00092 seconds3 examples, 1 failure
Failed examples:
rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package
info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013notice: Using cached catalog
www.camptocamp.com / 15/32
Puppet-spec: Deploying unit tests■ On the master side:○ Tests are located in the spec/catalog/class directory of the
environment○ Only the directories named after classes declared in the catalog
are tested
■ On the agent side:○ Deploy tests using pluginsync○ Tests are located in the lib/spec/catalog/class directory of each
module○ Only the directories named after classes declared in the catalog
are tested
www.camptocamp.com / 16/32
Puppet-spec: Unit tests limits■ When to apply the tests (currently based on class names)
■ Tests on master, or need to deploy all tests with pluginsync
■ Redundant with existing unit tests, or additional security?
www.camptocamp.com / 17/32
Puppet-spec: Setting up Unit testing■ Tests achieved from catalog indirection terminii
■ Plugins (terminii) deployed with pluginsync
■ Setup done in routes.yaml:
agent:
catalog:
# Either on the agent side
terminus: rest_spec
cache: yaml
master:
catalog:
# Or on the master side
terminus: compiler_spec
www.camptocamp.com / 18/32
Serverspec■ http://serverspec.org
■ Provides RSpec matchers for local functional tests (packages,users, services, ports, etc.)
■ Independant from configuration management tools
requirerequire 'spec_helper'
describe service('httpd') dodo
it { should be_enabled }
it { should be_running }
endend
describe port(80) dodo
it { should be_listening }
endend
describe file('/etc/httpd/conf/httpd.conf') dodo
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
endend
www.camptocamp.com / 19/32
Serverspec backendsAllows to use various means of launching tests:
■ SSH (default)
■ Exec
■ Puppet (RAL, removed from core)
$ serverspec-init
Select OS type:
1) UN*X
2) Windows
Select number: 1
Select a backend type:
1) SSH
2) Exec (local)
Select number: 1
www.camptocamp.com / 20/32
Puppet-spec: Functional testing■ Uses serverspec/specinfra matchers
■ Tests the machine state (not the catalog)
requirerequire 'spec_helper'
describe service('httpd') dodo
it { should be_enabled }
it { should be_running }
endend
describe port(80) dodo
it { should be_listening }
endend
describe file('/etc/httpd/conf/httpd.conf') dodo
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
endend
www.camptocamp.com / 21/32
Puppet-spec: Function tests output# puppet agent -tinfo: Retrieving plugininfo: Caching catalog for foo.example.cominfo: Applying configuration version 'raphink/a2c8e0f [+]'... Applying changes ...notice: Finished catalog run in 59.19 secondserr: Could not send report: Unit tests failed:FF
Failures:
1) augeasFailure/Error: it { should be_installed }
expected "augeas" to be installed# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
2) /usr/share/augeas/lenses/distFailure/Error: it { should be_file }
expected "/usr/share/augeas/lenses/dist" to be file# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
Finished in 0.06033 seconds2 examples, 2 failures
Failed examples:
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeasrspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/dist
www.camptocamp.com / 22/32
Puppet-spec: Deploying functionaltests
■ Tests are run after catalog application
■ Tests can be distributed via pluginsync (in the spec/server/class)directory of each module
■ Tests can be distributed with file Puppet resources, optionallyusing the spec::serverspec defined resource type
www.camptocamp.com / 23/32
Puppet-spec: MCollective agent■ Communicates with distant nodes
■ Sends action and values to specinfra check commands
■ Does not implement serverspec syntax
■ Returns true/false
■ Uses MCollective as transport (instead of SSH)
Examples:
$ mco rpc spec check action=user values=rpinson$ mco rpc spec check action=file values=/etc/passwd$ mco rpc spec check action=resolvable values=google.fr,A$ mco rpc spec check action=listening values=80$ mco rpc spec check action=process values=mcollectived$ mco rpc spec check action=file_contain values=/etc/passwd,rpinson
www.camptocamp.com / 24/32
Mspectator■ https://github.com/raphink/mspectator
■ RSpec matchers
■ Calls MCollective to achieve tests
■ Uses MCollective spec agent (among others)
www.camptocamp.com / 25/32
Mspectator architecture■ Client runs RSpec
■ RSpec calls MCollective
■ MCollective calls distant spec agent
■ spec agent calls specinfra backend
www.camptocamp.com / 26/32
Mspectator syntaxOwn matchers, mapping to specinfra backend methods:
requirerequire 'mspectator'
describe 'apache' dodoit { should find_nodes(100).or_less } # Counts discovered nodesit { should pass_puppet_spec } # Runs the `spec` agentit { should have_certificate.signed } # Uses the `puppetca` agent
context 'when on Debian',:facts => { :operatingsystem => 'Debian' } dodo # Filter by facts
it { should find_nodes(5).with_agent('spec') }it { should have_package('apache2.2-common') }it { should_not have_package('httpd') }it { should have_service('apache2').with(
:ensureensure => 'running') }it { should have_file('/etc/apache2/apache2.conf') }it { should have_directory('/etc/apache2/conf.d') }it { should have_user('www-data') }
endend
context 'when using SSL', :classes => ['apache::ssl'] dodo # Filter by classesit { should find_nodes(50).or_more }it { should have_package('ca-certificates') }
endendendend
www.camptocamp.com / 27/32
Mspectator output$ rake spec SPEC=apache_spec.rb/home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb
apacheshould find nodes 100should pass puppet spec (FAILED - 1)should have certificatewhen on Debianshould find nodes 5 (FAILED - 2)...
when using SSLshould find nodes 50 (FAILED - 3)
No request sent, we did not discover any nodes. should have package "ca-certificates"
Failures:
1) apacheFailure/Error: it { should pass_puppet_spec }
expected that all hosts would pass tests, the following didn't:soekris01.wrk.cby.camptocamp.com:soekris02.wrk.cby.camptocamp.com:
# ./apache_spec.rb:5
...
www.camptocamp.com / 28/32
Mspectator demo
www.camptocamp.com / 29/32
ContributeOn GitHub:
■ puppet-spec: https://github.com/raphink/puppet-spec
■ mspectator: https://github.com/raphink/mspectator
www.camptocamp.com / 30/32
Thank you!■ [email protected]
■ @raphink on Twitter/Github
■ raphink on Freenode
■ Slides: slideshare.net/raphink
www.camptocamp.com / 31/32