puppet troubleshooting
TRANSCRIPT
Latest version
http://goo.gl/b2NISc
If you see something,
say something!Google Comments enabled
Seriously, Trust Me
...Animations Are Good things
Watch in Presentation Mode!
PuppetConf 2013
Mastering Puppet / Puppet Cookbook 4*
PUGS
Me
/r/dadjokes
Outline
network
rest api
certificates
catalog
problem workers
compiling
printing
scope
Failures
- can't find puppet
- can't connect to puppet
- can't get certificate
- can't get/compile a catalog
- can't apply a catalog
- can't upgrade puppet
can't connect to puppet...
can't find puppet
puppet
puppet.domain[main/agent]
serverca_server
DNS
/etc/hosts
nsswitch.conf
can't connect to puppet
nodepuppet
master
network
pixies
8140
masterport
can't connect to puppet
● ping
● mtr (--port 8140)
● netcat (nc)
can't connect to puppet
$ ping puppetping: unknown host puppet$ mtr puppet
My traceroute [v0.75]cookbook.example.com (0.0.0.0) Sat Oct 25 01:26:31 2014Keys: Help Display mode Restart statistics Order of fields quit
Packets PingsHost Loss% Snt Last Avg Best Wrst StDev1. puppet.example.com 0.0% 157 0.7 0.5 0.2 1.6 0.2
$ nc -v puppet 8140Ncat: Version 6.45 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.122.100:8140.
can't get certificate
● already signed (clean)
● dates off - expired CA, expired cert
openssl x509 -in cert.pem -text
puppet cert clean host.example.com
ntpq -p
rm /var/lib/puppet/ssl/*/hostname*
can't get certificate
● don't know your own name
● basic unix permissions
● something else...
SELinux
root@puppet:~# sudo -iu puppetpuppet@puppet:~$ cd /etc/puppet/environments/-bash: cd: /etc/puppet/environments/: Permission denied
REST API
Rest API
nodepuppet
master
8140
GET https://puppet:8140/production/certificate/ca HTTP/1.1
-----BEGIN CERTIFICATE -----MIIFXjCCA0agAwIBAgIBATANBgkqhkiG9w0BAqsFADAcMRowGAYDVQQDDBFQdXdbwFiyvryAxxuETs9KORMwoThDIMd4bHLDbqtAz0q0cJ7W/8w==-----END CERTIFICATE-----
Rest API
GET https://puppet:8140/production/certificate/ca HTTP/1.1
ca_server/server
ca_port/masterport
environment
resource
key
Rest API
GET https://puppet:8140/production/certificate/ca HTTP/1.1
resource
key
resourcecertificatefile_metadatafile_contentnodecatalog
Rest API
● wget
● curl
# curl --insecure https://puppet:8140/production/certificate/ca >puppet-ca.pem% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed101 1923 101 1923 0 0 25506 0 --:--:-- --:--:-- --:--:-- 312k
# openssl x509 -in puppet-ca.pem -fingerprintSHA1 Fingerprint=5A:C2:03:7B:40:44:2C:81:45:81:07:11:D3:AC:29:FB:A5:EC:E3:55
# diff puppet-ca.pem /var/lib/puppet/ssl/ca.pem# echo $?0
Rest API
# curl --cacert puppet-ca.pem \https://puppet:8140/production/certificate/lisa.example.comNot Found: Could not find certificate lisa.example.com
# curl --cacert puppet-ca.pem \https://puppet:8140/production/certificate/cookbook.example.com-----BEGIN CERTIFICATE-----MIIFcTCCA1mgAwIBAgIBCjANBgkqhkiG9w0BAQsFADAcMRowGAYDVQQDDBFQdXBwZXQgQ0E6IHB1cHBldDAeFw0xNDA5MzAwNDM1MDdaFw0xOTA5MzAwNDM1MDdaMB8x...# openssl x509 -in cookbook.pem -text
…Issuer: CN=Puppet CA: puppet
ValidityNot Before: Sep 30 04:35:07 2014 GMTNot After : Sep 30 04:35:07 2019 GMT
Subject: CN=cookbook.example.com…
Catalog
# curl --cacert puppet-ca.pem \--cert /var/lib/puppet/ssl/certs/cookbook.example.com.pem \--key /var/lib/puppet/ssl/private_keys/cookbook.example.com.pem \-H 'Accept: yaml' \https://puppet:8140/production/catalog/cookbook.example.com > cookbook.yaml
% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 26099 100 26099 0 0 17434 0 0:00:01 0:00:01 --:--:-- 18642
Tools (gnutls-cli or openssl s_client)
$ gnutls-cli --insecure --port 8140 puppet.example.com
GET /production/certificate/ca HTTP/1.0Accept: s
HTTP/1.1 200 Server: nginx/1.6.1Date: Thu, 23 Oct 2014 21:03:58 GMTContent-Type: text/plain; charset=UTF-8Content-Length: 875Connection: closeX-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.11Status: 200
can't get/compile a
catalog
Problem workers
node
puppet
worker
8140
puppet
worker
8140
puppet
master
8140
devel
puppet
worker
8140
problem
production
mod_proxy_balancer
problem/bugfixes branches
git branches
● per user
● per problem
● per ticket
Problem workers
puppet
worker
8140
--logdest /var/log/puppet/problem.log--debug--profile
logrotate
Compiling
$ sudo puppet master \--compile problem.example.com \--debug --trace \--logdest /tmp/problem.puppet.log \--environment sandbox
{"data": {
"resources": [{
"title": "main","exported": false,"tags": ["stage"],"type": "Stage","parameters": {
"before": "Stage[post]",
JSON
Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'bucketdir': 'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/bucket"}'Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}'Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}'Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'fileserverconfig': 'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}'
Apply
$ sudo puppet apply \--debug --trace \--environment sandbox code.pp
JSON$date = "+%S"exec {'epoch':
command => "echo \$((`date $date` / 86400))",path => '/bin:/usr/bin:/sbin:/usr/sbin',
}
Info: Applying configuration version '1415729233'Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` / 86400))'Debug: Executing 'echo $((`date +%S` / 86400))'Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully
Catalog
YAML - read it
JSON - jq
Catalog
catalog classes.txt
jq '.data.classes[]' < catalog
Catalog
fails to compile● duplicate resource
● modulepath/bad module name
fails to apply● unpredicable exec
● bad/broken service
● bad/missing variable
fails to compile
duplicate resource
separate into subclass ( package {'httpd'} )
virtual resources ( @user, @package, @service)
modulepath
puppet config print modulepath
root@puppet:~# puppet config print modulepath --environment production
/etc/puppet/environments/production/public:/etc/puppet/environments/production/modules
root@puppet:~# puppet config print modulepath --environment master
/etc/puppet/modules:/usr/share/puppet/modules
fails to apply
Unpredictable exec
#!/bin/bash
echo $JAVA_HOME/home/javadev/.bashrc
JAVA_HOME=/your/face
Broken Service
service provider
hasstatus => true
/sbin/service $service status
/etc/init.d/$service status
/usr/bin/systemctl is-active $service
Bad/Missing Variable
$one = "1"file {"lisaone":path => "/tmp/lisa$one",ensure => 'directory',
}file {"lisa1":path => "/tmp/lisa1",ensure => 'file',
}
Info: Caching catalog for node1.example.comError: Failed to apply catalog: Cannot alias File[lisa1] to ["/tmp/lisa1"] at /etc/puppet/environments/production/manifests/site.pp:34; resource ["File", "/tmp/lisa1"] already declared at /etc/puppet/environments/production/manifests/site.pp:30
Bad/Missing Variable
lisa {'one':place => "/tmp/$LISA",type => "directory",
}lisa {'two':place => "/tmp/$LISA",type => "file",
}
define lisa ($place,$type) {file {"$title":path => $place,ensure => $type,
}}
Info: Caching catalog for node1.example.comError: Failed to apply catalog: Cannot alias File[two] to ["/tmp"] at /etc/puppet/environments/production/modules/lisa/manifests/init.pp:5; resource ["File", "/tmp"] already declared at /etc/puppet/environments/production/modules/lisa/manifests/init.pp:5
Printing - Notify
notify {"$variable": }
chaining
notify {'something': }->exec{'thingthatfails': }->notify{'after': }
Checking
exec{'before resolv.conf': command => '/usr/local/bin/puppet-debug before resolv.conf',require => Class['debug']
} -> file { '/etc/resolv.conf':source => template("dns/resolv.conf"),noop => true,
}
class debug { file {'puppet-debug':path => '/usr/local/bin/puppet-debug',source => 'puppet:///modules/debug/puppet-debug',mode => 0755,
}}
Debug Script… just an example#!/bin/bashLOG=$(mktemp /tmp/puppet-debug.XXXXXX)echo Puppet Debug -- $@ -- $(date) | tee $LOGecho "-- Disk --" | tee -a $LOGdf -h |tee -a $LOGdf -i |tee -a $LOGecho "-- Mem --" | tee -a $LOGfree | tee -a $LOGecho "-- Files --" | tee -a $LOGPUPPET=$(pgrep puppet)for proc in $PUPPETdo
lsof -p $proc |tee -a $LOGdone
Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014-- Disk --Filesystem Size Used Avail Use% Mounted on/dev/mapper/VolGroup-lv_root
6.7G 2.5G 3.9G 39% /tmpfs 246M 0 246M 0% /dev/shm/dev/vda1 485M 80M 380M 18% /bootFilesystem Inodes IUsed IFree IUse% Mounted on/dev/mapper/VolGroup-lv_root
440640 79253 361387 18% /tmpfs 62783 1 62782 1% /dev/shm/dev/vda1 128016 50 127966 1% /boot-- Mem --
total used free shared buffers cachedMem: 502268 415488 86780 0 22176 172036-/+ buffers/cache: 221276 280992Swap: 835580 0 835580-- Files --COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEpuppet 2058 root cwd DIR 253,0 4096 14 /rootpuppet 2058 root rtd DIR 253,0 4096 2 /puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/rubypuppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.sopuppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so
Printing - Template
- scope.to_hash
- reject a few
- sort
- print, one per line
file { "/tmp/puppet-debug.txt":content => inline_template("<% vars =
scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>\n<% end %>"),}
vars = scope.to_hash.reject{ |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) };
vars.sort.each do |k,v|k=v\n
end
Printing - Template
- scope.to_hash
- reject a few
- sort
- print, one per line
file { "/tmp/puppet-debug.txt":content => inline_template("<% vars =
scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>\n<% end %>"),}
vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) };
vars.sort.each do |k,v|k=v\n
end
_timestamp=2014-10-23 22:29:52 -0700architecture=x86_64augeasversion=1.0.0bios_release_date=01/01/2011bios_vendor=Bochsbios_version=Bochsblockdevice_vda_size=8589934592blockdevice_vda_vendor=6900blockdevices=vdacaller_module_name=clientcert=cookbook.example.comclientnoop=falseclientversion=3.7.1concat_basedir=/var/lib/puppet/concatdomain=example.comenvironment=productionfacterversion=2.2.0filesystems=ext4,iso9660fqdn=cookbook.example.comgid=roothardwareisa=x86_64hardwaremodel=x86_64hostname=cookbookid=rootinterfaces=eth0,lo
Scope
The scene:
● roles and profiles
● ntp server
class role::ntp {include ntp
}
class ntp {include ntp::server
}
Scope
The solution:
● fully scope everything
● remember scope
class role::ntp {include ::ntp
}
class ntp {include ntp::server
}
Summary
learn some networking
remember the REST api
read up on SSL / x509
use --trace
make a debug class
remember scope
Questions/Comments?
http://goo.gl/b2NISc
http://ramblings.narrabilis.com