monitoring your vms at scale
TRANSCRIPT
![Page 1: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/1.jpg)
Beyond VM deploymentBeyond VM deploymentMonitoring your VM's at scale
Kris Buytaert
![Page 2: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/2.jpg)
Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source
Consultant @inuits.euConsultant @inuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops● But mostly, trying to be good at my jobBut mostly, trying to be good at my job
![Page 3: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/3.jpg)
What's different in the cloud ?What's different in the cloud ?
● ScaleScale
● VelocityVelocity
● ChangeChange
![Page 4: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/4.jpg)
ChallengesChallenges● ReproducabilityReproducability
● SpeedSpeed
● AuditingAuditing
● Keeping stuff in sync Keeping stuff in sync
• MonitoringMonitoring
• SecuritySecurity
![Page 5: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/5.jpg)
Case :Case :Using a configuration management Using a configuration management tool to configure, update and keep tool to configure, update and keep your cloudscale monitoring and metric your cloudscale monitoring and metric infrastructure sane and manageable.infrastructure sane and manageable.
![Page 6: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/6.jpg)
ToolsTools● PuppetPuppet
● CollectdCollectd
● GraphiteGraphite
● Nagios / IcingaNagios / Icinga
● Chef CfengineChef Cfengine
● GangliaGanglia
● SensuSensu
![Page 7: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/7.jpg)
Not quite a Muppet.Not quite a Muppet.
● Puppet is...Puppet is...
● OSSOSS
● A DSL languageA DSL language
● Written in RubyWritten in Ruby
● Client/server orientedClient/server oriented
● Contains abstraction layersContains abstraction layers
● Repeatable processesRepeatable processes
![Page 8: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/8.jpg)
Master of PuppetsMaster of Puppets● Puppet masterPuppet master
• CA authorityCA authority
• Hosts ModulesHosts Modules
• Hosts Node descriptionsHosts Node descriptions
• Compare, compile, applyCompare, compile, apply
● Master is not a requirement !Master is not a requirement !
![Page 9: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/9.jpg)
Puppet ClientsPuppet Clients● daemondaemon
● Cron jobsCron jobs
● External orchestration:External orchestration:
• for i in $hosts; do ssh $i “puppetd --test”; donefor i in $hosts; do ssh $i “puppetd --test”; done
• mCollective, Func, …mCollective, Func, …
● Get catalogs, play them, Get catalogs, play them,
● reportingreporting
![Page 10: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/10.jpg)
Puppet EnvironmentsPuppet Environments
● Different code bases on 1 masterDifferent code bases on 1 master
● Dev, Uat, ProdDev, Uat, Prod
● Only break one environment at once :)Only break one environment at once :)
● What about testing your Puppetmaster ?What about testing your Puppetmaster ?
![Page 11: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/11.jpg)
Node definitionsNode definitions● Nodes.ppNodes.pp
cclass defaults {lass defaults {$search = "inuits.be"$search = "inuits.be"$nameservers = ['208.67.220.220', '208.67.222.222']$nameservers = ['208.67.220.220', '208.67.222.222']
include dns::resolvinclude dns::resolvinclude ssh::keysinclude ssh::keysinclude ssh::serverinclude ssh::server
}}
node "ns1.dev.inuits.be" {node "ns1.dev.inuits.be" {include defaultsinclude defaultsinclude dns::powerdns::serverinclude dns::powerdns::serverinclude dns::powerdns::resolverinclude dns::powerdns::resolver
}}
node “web1.dev.inuits.be” {node “web1.dev.inuits.be” {include defaultsinclude defaultsinclude apache2include apache2include mysqlinclude mysql
}}
![Page 12: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/12.jpg)
External Node ClassifierExternal Node Classifier● Fixed hostname ? Fixed hostname ?
● How many nodes How many nodes
● Naming schemas solve some issues Naming schemas solve some issues
● External script that sends back yaml class descriptionsExternal script that sends back yaml class descriptions
• Custom writttenCustom writtten
• Foreman Foreman
• ... ...
![Page 13: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/13.jpg)
Classes vs ModulesClasses vs Modules● Module : Module :
● Abstract definition on configuring a serviceAbstract definition on configuring a service
● ReusableReusable
● Class : Class :
● Specific implementation of your use case of Specific implementation of your use case of such a module such a module
•e.g usernames / passwords / hosts do not belong in e.g usernames / passwords / hosts do not belong in modulesmodules
![Page 14: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/14.jpg)
ModulesModules● FilesFiles
● TemplatesTemplates
● ManifestsManifests
• DSLDSL
• ClassesClasses
• ElementsElements
![Page 15: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/15.jpg)
Parametrized Parametrized ClassesClasses
![Page 16: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/16.jpg)
Stored ConfigsStored Configs
![Page 17: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/17.jpg)
Use Cases:Use Cases:● Ssh keysSsh keys
● Reverse proxy configsReverse proxy configs
● Monitoring resourcesMonitoring resources
● Measuring resourcesMeasuring resources
![Page 18: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/18.jpg)
Collection and ExportCollection and Export
Export :Export :
@@resource { @@resource {
... }... }
Collect:Collect:
Resource <<| Resource <<| query |>>query |>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node clean puppet node clean
![Page 19: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/19.jpg)
Defining a ServiceDefining a Service● Local class that :Local class that :
• Configures service using a standard Configures service using a standard module call with hiera based parametersmodule call with hiera based parameters
• Configures BackupConfigures Backup
• Configures logrotation Configures logrotation
• Configures logshipping Configures logshipping
• Exports Monitoring NeedsExports Monitoring Needs
● Abuse modules for git easeAbuse modules for git ease
![Page 20: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/20.jpg)
Apache Example:Apache Example:
![Page 21: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/21.jpg)
#monitoringsucks#monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love:
● Having my hands tied with the model of host and service bindings. Having my hands tied with the model of host and service bindings.
● Having to set up "fake" hosts just to group arbitrary metrics together Having to set up "fake" hosts just to group arbitrary metrics together
● Having to either collect metrics twice - once for alerting and another for trending Having to either collect metrics twice - once for alerting and another for trending
● Only being able to see my metrics in 5 minute intervals Only being able to see my metrics in 5 minute intervals
● Having to chose between shitty interface but great monitoring or shitty monitoring but Having to chose between shitty interface but great monitoring or shitty monitoring but great interface great interface
● Dealing with a monitoring system that thinks IT is the system of truth for my Dealing with a monitoring system that thinks IT is the system of truth for my environment environment
● Not actually having any real choicesNot actually having any real choices
John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-John Vincent (@lusis) on his blog http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.htmlmonitoring-sucks.html
![Page 22: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/22.jpg)
![Page 23: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/23.jpg)
#monitoringlove#monitoringlove● Puppet Puppet
● Nagios (Icinga)Nagios (Icinga)
● GraphiteGraphite
● CollectdCollectd
● LogstashLogstash
![Page 24: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/24.jpg)
Graphite Graphite ● Graphing at ScaleGraphing at Scale
● Graphing at EaseGraphing at Ease
● Any metric is a graphAny metric is a graph
● echo "somestring $somevalue echo "somestring $somevalue $timestamp" | nc <%= graphitehost $timestamp" | nc <%= graphitehost %> 2003%> 2003
![Page 25: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/25.jpg)
Graphite ComposerGraphite Composer
x
![Page 26: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/26.jpg)
Graphite APIGraphite API
![Page 27: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/27.jpg)
Gdash In actionGdash In action
![Page 28: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/28.jpg)
Puppet and GraphitePuppet and Graphite● https://github.com/KrisBuytaert/vagrant-graphite/https://github.com/KrisBuytaert/vagrant-graphite/
● Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Includes Graphite / Gdash / Jmxtrans / Logster / Collectd / Statsd / Tattle and more modules as submodules ! Statsd / Tattle and more modules as submodules !
● git clonegit clone
● git submodule init git submodule init
● git submodule update git submodule update
● vagrant up vagrant up
![Page 29: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/29.jpg)
CollectdCollectd● CollectsCollects
● Zillion PluginsZillion Plugins
• Nginx,apache, mysql, diskNginx,apache, mysql, disk
● Graphite Carbon PluginGraphite Carbon Plugin
● Send metrics to graphiteSend metrics to graphite
![Page 30: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/30.jpg)
Collectd & GraphiteCollectd & Graphite
![Page 31: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/31.jpg)
Exporting and Collecting Exporting and Collecting
![Page 32: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/32.jpg)
Triggers on GraphsTriggers on Graphs● Export Java MetricsExport Java Metrics
● JMXTransJMXTrans
● Export JMXConfigsExport JMXConfigs
● Configure NRPE Configure NRPE CheckCheck
● Export NagiosCheckExport NagiosCheck
● Collect JMX Exports Collect JMX Exports on JMXTransNodeon JMXTransNode
● Graph EmGraph Em
Collect Nagios Collect Nagios Configs on Nagios Configs on Nagios ServerServer
![Page 33: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/33.jpg)
Triggers on GraphsTriggers on Graphs
![Page 34: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/34.jpg)
Triggers on GraphsTriggers on Graphs
![Page 35: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/35.jpg)
Conclusion:Conclusion:● Reproducable monitoring setupReproducable monitoring setup
● Dynamically generated monitoring Dynamically generated monitoring configconfig
● Code is available at Code is available at github.com/KrisBuytaert/snippetsgithub.com/KrisBuytaert/snippets
![Page 36: Monitoring your VMs at Scale](https://reader034.vdocuments.site/reader034/viewer/2022050613/5549b46ab4c905fc7f8b4849/html5/thumbnails/36.jpg)
ContactContactKris Buytaert Kris Buytaert [email protected]@inuits.be
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Duboistraat 50Duboistraat 502060 Antwerpen2060 AntwerpenBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221