puppet camp duesseldorf 2014: kris buytaert - monitoring (with) puppet
Post on 10-Jun-2015
114 Views
Preview:
DESCRIPTION
TRANSCRIPT
Monitoring (with) PuppetMonitoring (with) Puppet
PuppetCamp Dusseldorf
@KrisBuytaert
KrisKris BuytaertBuytaert
● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Organising too many confs , #devopsdays, Organising too many confs , #devopsdays, #loadays, ... #loadays, ...
● Evangelizing devopsEvangelizing devops
Do you know what your children do at 5 am in Do you know what your children do at 5 am in the morning ?the morning ?
● Sleeping ?Sleeping ?
● Crashing a Party ?Crashing a Party ?
● Why are the cops at your front door ?Why are the cops at your front door ?
● What happened ?What happened ?
● How long have they been gone ?How long have they been gone ?
devops = clamsdevops = clams● CultureCulture
● (Lean)(Lean)
● Automate all the things ... Automate all the things ...
• Build Automation Build Automation
• Test Automation Test Automation
• IACIAC
● Monitoring , Metrics ... Monitoring , Metrics ...
● SharingSharing
Monitoring is usually an Monitoring is usually an aftertoughtaftertought
ENOBUDGET, ENOTIMEENOBUDGET, ENOTIME
#monitoringsucks#monitoringsucks● John Vincent (@lusis)John Vincent (@lusis)
● A sub movement A sub movement
● https://github.com/monitoringsucks/https://github.com/monitoringsucks/
#monitoringlove#monitoringlove• #monitoringlove hacksessions#monitoringlove hacksessions
• #monitorama#monitorama
For years we've tolerated humans to to For years we've tolerated humans to to make structural manual changes to the make structural manual changes to the infrastructure our critical infrastructure our critical applications are running on.applications are running on.
Whilst at the same time demanding Whilst at the same time demanding those critical applications to go those critical applications to go
trough rigid test scenarios.trough rigid test scenarios.
Who let this happen ?Who let this happen ?
Infrastructure as CodeInfrastructure as Code● Model our infrastructure Model our infrastructure
● A fast reproducable platformA fast reproducable platform
● Disaster discovery for free“ ”Disaster discovery for free“ ”
Infrastructure as CodeInfrastructure as Code● Code = Code Code = Code
● Version Control Version Control
● Quality ChecksQuality Checks
● TestingTesting
● Continuous IntegrationContinuous Integration
● Continous DeliveryContinous Delivery
Infrastructure as CodeInfrastructure as Code● Core Infrastructure Core Infrastructure
● Middleware deployment and integrationMiddleware deployment and integration
● Automated continuous application Automated continuous application deploymentdeployment
● Integrated Security enforcementIntegrated Security enforcement
● Host, Service and Application Host, Service and Application Monitoring configuredMonitoring configured
Why #monitoringsucksWhy #monitoringsucks
● Manual config (gui)Manual config (gui)
● Not in sync with realityNot in sync with reality
● Hosts onlyHosts only
● Services sometimesServices sometimes
● Appliccation neverAppliccation never
● ChaosChaos
• Really ?Really ?
• Service,Service,
• FunctionalitiesFunctionalities
• eg. vhosts etceg. vhosts etc
• Single Source of TruthSingle Source of Truth
But tools do AutodetectionBut tools do Autodetection
Monitoring 101Monitoring 101● Deploy a host,Deploy a host,
● Add it to the monitoringAdd it to the monitoring
● Add collection toolsAdd collection tools
● Add check definitionsAdd check definitions
● Update the monitoring tool configUpdate the monitoring tool config
● RestartRestart
Collect Metrics 101Collect Metrics 101
CollectdCollectd● CollectsCollects
● Zillion PluginsZillion Plugins
• Nginx,apache, mysql, diskNginx,apache, mysql, disk
● Graphite Carbon PluginGraphite Carbon Plugin
● Send metrics to graphiteSend metrics to graphite
● https://github.com/KrisBuytaert/puppehttps://github.com/KrisBuytaert/puppet-collectdt-collectd
Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !
● git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
2014 style dashboards2014 style dashboards
Alerting 101Alerting 101
Alert all the thingsAlert all the thingsAnd get alertfatigueAnd get alertfatigue
● We are in Germany, EuropeWe are in Germany, Europe
● It needs to be configurable It needs to be configurable
=> Icinga=> Icinga
But Monitoring config is But Monitoring config is Data ?Data ?● If it can be generated it's not user If it can be generated it's not user generated content anymoregenerated content anymore
● Your computer can generate your Your computer can generate your config !config !
Stored ConfigsStored Configs
Collection and ExportCollection and Export
Export :Export :
@@resource { @@resource {
... }... }
Collect:Collect:
Resource <<| Resource <<| query |>>query |>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node clean puppet node clean
Exporting and Collecting Exporting and Collecting
Default Puppet TypesDefault Puppet Types
Puppet-icinga modulePuppet-icinga module● https://github.com/inuits/puppet-https://github.com/inuits/puppet-icingaicinga
Monitoring a VhostMonitoring a Vhost
Alternative ApproachesAlternative Approaches
● https://gist.github.com/jfryman/5808537https://gist.github.com/jfryman/5808537
● https://github.com/favoretti/puppetdbhttps://github.com/favoretti/puppetdb-external-naginator-external-naginator
● Deploy a new appDeploy a new app
● Add monitoringAdd monitoring
● Add Real application monitoringAdd Real application monitoring
● Both on infra and on app levelBoth on infra and on app level
Monitoring Puppet & FriendsMonitoring Puppet & Friends
Puppet RunsPuppet Runs
PuppetMasterPuppetMaster
PuppetMasterPuppetMaster @@nagios_service{"check_socket_8140_puppet_${::fqdn}":@@nagios_service{"check_socket_8140_puppet_${::fqdn}":
check_command => 'check_tcp!8140',check_command => 'check_tcp!8140',
service_description => 'TCP puppet on port 8140',service_description => 'TCP puppet on port 8140',
host_name => $::fqdn,host_name => $::fqdn,
use => 'generic-service',use => 'generic-service',
contact_groups => $::environment,contact_groups => $::environment,
notification_period => $::icinga::notification_period,notification_period => $::icinga::notification_period,
notifications_enabled => $::icinga::notifications_enabled,notifications_enabled => $::icinga::notifications_enabled,
target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",
}}
PuppetDBPuppetDB
PuppetDB(2)PuppetDB(2)● check_puppetdb_memorycheck_puppetdb_memory
• Java heap memoryJava heap memory
● check_puppetdb_processedcheck_puppetdb_processed
• Nr of reports processedNr of reports processed
● check_puppetdb_populationscheck_puppetdb_populations
• Resources, nodes, resources per nodeResources, nodes, resources per node
● check_puppetdb_queuecheck_puppetdb_queue
• Is pgsql down ? :)Is pgsql down ? :)
Puppet DashboardPuppet Dashboard @@nagios_service{"check_http_puppet_dashboard_${::fqdn}":@@nagios_service{"check_http_puppet_dashboard_${::fqdn}":
check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",check_command => "check_http!-H ${::fqdn} -p 3000 -e 200",
service_description => 'HTTP PuppetDashboard on port 3000',service_description => 'HTTP PuppetDashboard on port 3000',
host_name => $::fqdn,host_name => $::fqdn,
use => 'generic-service',use => 'generic-service',
contact_groups => $::environment,contact_groups => $::environment,
notification_period => $::icinga::notification_period,notification_period => $::icinga::notification_period,
notifications_enabled => $::icinga::notifications_enabled,notifications_enabled => $::icinga::notifications_enabled,
target => "/etc/icinga/objects/services/${::fqdn}.cfg",target => "/etc/icinga/objects/services/${::fqdn}.cfg",
}}
Puppet Dashboard(2)Puppet Dashboard(2)
/usr/share/puppet-dashboard/spool/usr/share/puppet-dashboard/spool
PuppetServerPuppetServer
#MonitoringSucks#MonitoringSucks● Puppetruns break our Icinga boxenPuppetruns break our Icinga boxen
● BadlyBadly
● FrequentlyFrequently
It ain't borkenIt ain't borken● Successful puppet runSuccessful puppet run
● Successful Icinga reconfigureSuccessful Icinga reconfigure
● Disk usage growsDisk usage grows
● FastFast
Be aware of bucketsBe aware of buckets
A Puppet BugA Puppet Bug
Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics
● JMXTransJMXTrans
● Export JMXConfigsExport JMXConfigs
● Configure NRPE CheckConfigure NRPE Check
● Export NagiosCheckExport NagiosCheck
● Collect JMX Exports on Collect JMX Exports on JMXTransNodeJMXTransNode
● Graph EmGraph Em
Collect Nagios Configs Collect Nagios Configs on Nagios Serveron Nagios Server
Triggers on GraphsTriggers on Graphs
Triggers on GraphsTriggers on Graphs
SummarySummary● Honour your parentsHonour your parents
● Don't manually do what machines can Don't manually do what machines can do for youdo for you
● Monitor your puppet infrastructure Monitor your puppet infrastructure too !too !
● Send Pull RequestsSend Pull Requests
● Icinga2 module in the worksIcinga2 module in the works
ContactContactKris.Buytaert@inuits.euKris.Buytaert@inuits.eu
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.eu/http://www.inuits.eu/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221
top related